A registry-driven collector pulls 57 CoinGlass + alt-data signals (54 live, 3 staged-off) plus the Binance trade firehose, lands every payload as immutable raw, distils it into a narrow hot RDS tier, and compacts it into an immutable point-in-time research lake. One global rate-limiter governs every REST call; the WebSocket tape runs off-budget. This is the data flow, end to end.
Capture spools every WS/REST payload through the off-loop accept-WAL to the
S3 raw/ prefix, indexed in cg_raw_payload_index. Durable-before-ACK: the raw object
is the system of record. RAW_ONLY & RECONCILE signals terminate here.
The transform producer reads newly-durable raw, normalizes it, builds deterministic
replayable bars, and commits manifest + rows + watermark in one transaction to narrow cg_*
RDS tables (append-only, idempotent). as-of views serve point-in-time reads.
Off the hot path, the compactor reads RDS as-of views and writes
immutable monthly parquet + manifests to compacted/gold/{dataset}/ — and may write
only that prefix (research-compute isolation, fail-closed). This lake feeds validation & training.
The trade / liquidation firehose. Owned by the WS client, never charged to the REST ledger. Durable via accept-WAL before ACK.
Every budget-critical REST series + breadth + index + control call. Sized for N=80 at the soft target; 220 is the hard ceiling.
R1 microstructure + R2 event feeds. Non-blocking breaker — the core never waits on them; their DQ gaps are WARNING, not CRITICAL.
The cheapest research substrate — R2 daily on-chain indices. The breaker defers these first under pressure.
Every WS/REST payload is committed to the off-loop write-ahead log before the frame is acknowledged. Crash-safe: the raw object is the system of record, recovered bit-for-bit on restart.
The lease heartbeat, spool, and S3 upload run off the asyncio loop, so the full-tape firehose can never starve liveness. An autonomous watcher load-sheds and self-recovers under pressure.
The gold compactor may write only the compacted/ prefix — fail-closed. Raw ingestion and research compute can never collide; the lake is immutable, monthly, manifest-verified.
Deterministic, replayable bars hash bit-identical from the same raw window. As-of views serve only data that was knowable at each timestamp — no look-ahead leaks into validation or training.
cg_* series + as-of views · db.t3.medium · private subnettraddea-coinglass-raw · immutable gzip JSON · collection_epoch=N/compacted/gold/*.parquet + models + manifests · lifecycle-tieredcg_* tables to RDS — the hot, point-in-time tier.compacted/ only).