Firmwide observability and batch orchestration
Eleven million queries a day, and the alerting that came with it
Goldman Sachs · Jan 2018 — Nov 2022
An Elasticsearch-backed observability platform powering firmwide search and alerting, and the orchestration layer governing ten thousand scheduled jobs behind it.
- queries indexed daily
11M+
queries indexed daily
- scheduled jobs governed
10k+
scheduled jobs governed
- production incidents
~0
production incidents
The problem
Firmwide search over operational data means every team's worst query is your production load. Index design had to survive access patterns nobody would tell you about in advance.
Alerting sat on the same indices as ad-hoc search, so a heavy analyst query could not be allowed to delay a page.
Ten thousand scheduled jobs with real interdependencies fail in ways that are individually rare and collectively constant.
Constraints that shaped it
- Search and alerting share data but not failure modes.
- Job failures degrade rather than cascade — a late upstream should not take down everything downstream.
- Near-zero production incidents was the actual bar, not an aspiration.
Architecture
Select any box for the design decision behind it.
What I owned
- 01Technical lead on the observability platform indexing 11M+ queries/day.
- 02Fault-tolerant AWS microservices and the batch orchestration layer over 10k+ scheduled jobs.
What I would do differently
Four and a half years on one platform teaches something a project cannot: most reliability work is not clever. It is refusing to let two workloads with different failure tolerances share a resource.
This system is employer-internal, so there is no repository to link. Everything above describes system shape, trade-offs, and results already stated on my résumé — no proprietary implementation detail.