Skip to content
JS
All case studies

Streaming + batch market data on AWS

Ten million ticks a day, and a storage bill that stopped growing

Block Scholes · Sep 2024 — Sep 2025

A Kinesis-to-S3 tick pipeline with a compaction job behind it, because the thing that kills a Parquet lake is not volume — it is object count.

AWS KinesisS3PySparkParquetAthena
records processed daily

10M+

records processed daily

storage cost reduction

40%

storage cost reduction

test coverage

+30%

test coverage

The problem

Market data arrives continuously and in small pieces. Written naively, that produces millions of tiny Parquet objects partitioned by date and instrument.

Athena does not mind large tables. It minds scattered ones. Query time was going into object listing and per-file overhead rather than scanning.

Storage cost tracked object count as much as bytes, and the ingest path could not be slowed down to batch harder without adding latency to the live feed.

Constraints that shaped it

  • The ingest path stays low-latency; compaction happens out of band.
  • Original files are lifecycle-tagged and retained for a recovery window — compaction is never destructive on first pass.
  • Deduplication and sort order are configurable per dataset, not hardcoded.
  • The same script runs locally and as a Glue job, so it can be debugged without a cluster.

Architecture

Exchange feeds10M+/dayKinesisordered shardsS3 rawsmall .zstd.parquetGlue compactionPySpark512MB targetS3 compactedfewer, larger-40% costAthenaMetrics + alerting

Select any box for the design decision behind it.

Hot path on top, compaction out of band below — the two never block each other.

What I owned

  • 01End-to-end architecture of the streaming and batch path on Kinesis and S3.
  • 02The PySpark compaction and partition-tuning job, packaged to run both locally and on Glue.
  • 03Monitoring, logging, and metrics across the whole pipeline.
  • 04Raising test coverage 30% to make the pipeline safe to change.

What I would do differently

Small-file problems look like scale problems and are usually layout problems. The cheapest win in the whole system was a parameterised target file size — one number, argued about properly.

This system is employer-internal, so there is no repository to link. Everything above describes system shape, trade-offs, and results already stated on my résumé — no proprietary implementation detail.