Market data

dyson-market-data runs a configured bundle of imported venue publishers. It is a process supervisor and configuration boundary, not another feed parser.

Source references are repo-relative paths in dyson.

Configuration model

flowchart LR
    DEP["MarketDataDeployment<br/>stable launch input"]
    VER["MarketDataVersion<br/>source commit + config hash"]
    PUB["Publisher[]<br/>id, venue, native JSON config"]
    MKT["Markets<br/>inside the publisher's own config"]

    DEP --> VER --> PUB --> MKT

    classDef store fill:#2e7d32,stroke:#1b5e20,color:#fff
    class DEP store

• The deployment ID is the stable launch input and the only argument the binary takes.
• The selected version carries source and configuration provenance.
• Each publisher has a unique ID, one venue, and its venue-native JSON configuration.
• One publisher may cover multiple markets for its venue.

Startup

flowchart TD
    ARG["Deployment ID argument"]
    MDB["Load deployment from MongoDB"]
    DRV["Resolve driver per publisher venue"]
    CFG["Materialize native config<br/>write .tmp, then rename into place"]
    SPAWN["Spawn one child process per publisher<br/>config path passed by env var"]
    SUP["Supervise the bundle"]

    ARG --> MDB --> DRV --> CFG --> SPAWN --> SUP

    classDef gate fill:#2e7d32,stroke:#1b5e20,color:#fff
    class DRV gate

Driver resolution is the validating step: an unrecognized venue string fails the launch before any child starts. Each config is written to a temporary file and renamed into place, so a publisher never observes a half-written config.

The config path is passed to the child through both MARKET_DATA_CONFIG and the driver's own variable (BINANCE_MARKET_DATA_CONFIG, KURU_MARKET_DATA_CONFIG, and so on), which is what lets unmodified imported executables run under the supervisor.

Path	Default	Override
Publisher executables	/usr/local/libexec/dyson-market-data	DYSON_MARKET_DATA_BINARY_DIR
Materialized configs	/tmp/dyson-market-data	DYSON_MARKET_DATA_RUNTIME_DIR

Supported drivers

Each venue maps to one imported executable (crates/market-data/src/main.rs):

Venue	Executable
binance	binance_market_data_writer
bybit	bybit_market_data_writer
coinbase	coinbase_market_data_writer
drake	drake_backfill
hyperliquid	hyperliquid_market_data_writer
kuru	kuru_historical_market_data_writer
lighter	lighter_market_data_writer
okx	okx_market_data_writer
perpl	perpl_backfill

Market-data coverage is therefore much wider than execution coverage — see System architecture.

Outputs

flowchart LR
    VEN["Venue feeds"] --> PUB["Publisher process<br/>venue-native parsing"]
    PUB --> ARC["Local archive"]
    PUB --> NATS["NATS live packets<br/>when live publication is configured"]
    NATS --> STR["Strategy processes<br/>subscribed by configured subject"]

    classDef store fill:#2e7d32,stroke:#1b5e20,color:#fff
    class ARC,NATS store

Publisher behavior remains venue-native. A publisher can write archived market data and can also publish normalized live packets to NATS when live publication is configured. Strategy processes subscribe to the NATS subjects selected by their strategy configuration.

Lifecycle

All publishers in a bundle share one failure boundary.

stateDiagram-v2
    [*] --> starting
    starting --> supervising: all publishers spawned
    supervising --> stopping: SIGTERM or SIGINT
    supervising --> stopping: any publisher exits
    stopping --> [*]: bundle stopped

Stopping is a graded shutdown rather than an immediate kill:

1. Send SIGTERM to every child.
2. Wait for children to exit, up to a 30-second grace period.
3. Kill any publisher still running when the deadline passes.

An exit by any single publisher takes the whole bundle down the same path. This avoids presenting a partially healthy multi-venue feed set as a healthy deployment. Kubernetes restarts the supervisor according to the workload policy; it does not restart individual publisher children independently.

No feed state survives a restart

The supervisor holds no durable state of its own — configuration comes from MongoDB on every launch and materialized config files are rewritten each time. See State management.

Ownership boundary

• Change bundle selection and publisher parameters in versioned configuration.
• Change venue parsing or archive behavior in the imported algo-trading submodule.
• Change supervision and launch behavior in crates/market-data.
• Change NATS subjects with the publisher and strategy consumers together.