The NMLab Environment

NMLab is a market-analysis environment designed and operated entirely in-house, from data collection through to visualization. Building it means solving demanding engineering problems end to end: storing time series at the scale of billions of points, making them interactive on screen, and streaming them in real time. Here is how.

A rendering engine built for hundreds of millions of points

Displaying tick-level financial series across several years goes far beyond what a conventional web chart can handle. The terminal is built on SciChart, a GPU-accelerated WebAssembly graphics rendering engine — the same visualization technology found on the trading floors of investment banks and hedge funds, in Formula 1 teams, and at NASA. We extended it with a custom-built batch-drawing layer: the primitives of a single object — and even of consecutive objects — are merged into a minimum number of render calls, with direct writes into WASM memory and caching of idle layers. The result: the environment displays close to 200 million points simultaneously without the slightest slowdown. Panning, zooming, and manipulating hundreds of drawing objects hold a steady 60 frames per second, where a naive approach would drop to a handful — and the scene stays fluid up to the scale of a thousand objects.

A data architecture designed for scale

Beneath the visualization, several billion tick and candle records are stored in PostgreSQL + TimescaleDB. The hypertables are partitioned by time as well as by instrument; prices are kept as integers with a fixed scale factor, to preserve both precision and speed; the 1-minute, 1-hour, and 1-day candles are continuously materialized by in-database jobs rather than recomputed on the fly. Every choice targets the same goal: fast analytical queries, whatever the horizon being queried.

From tick to screen, in real time

Fresh data does not wait to be queried. As soon as a record is inserted, the database emits a notification that a FastAPI backend immediately relays to connected clients over WebSocket. The same path feeds both the interactive visualization and the live streams — with no polling and no needless latency.

Finding edges, at scale

This speed is not merely a display convenience: it changes the very nature of research. Because the environment sweeps billions of points in moments and exposes exactly the same data to analysis notebooks, we run large-scale statistical studies — thousands of hypotheses tested across many instruments and long periods — in a fraction of the usual time. Working on raw, complete data — without aggregation or sampling — and replaying each test identically on other instruments and other periods makes it easier to eliminate the statistical biases (aggregation, survivorship, overfitting) that distort so many studies. This is a decisive advantage for detecting exploitable edges: the faster the iteration, the more ideas we explore, and the more we isolate the regularities that truly matter. The approach has not remained theoretical: it has allowed us to isolate previously unseen statistical regularities that feed directly into our own research.

Against conventional methods

Conventional approaches almost always force a trade-off. Mainstream charting platforms and web libraries aggregate history and cap out at a few tens of thousands of points — goodbye microstructure and depth. "Vendor data, spreadsheet, and scripts" setups decouple what you look at from what you test and slow exploration down; the cloud and SaaS, for their part, expose data to a third party and bill per query. The environment requires none of these compromises: the data stays tick-level and on-premise, the screen and the analysis share the same source, and billions of points are traversed in moments. Where the conventional forces a choice between granularity, scale, reproducibility, cost, and confidentiality, the environment brings them together.

An integrated system, optimized end to end

Ingestion, storage, API, visualization, and research form a coherent system, designed and operated on-premise. This vertical integration is not just a matter of control: knowing every link lets us optimize the others accordingly, from the storage format through to the rendering pipeline. It is this technical chain, from raw tick to displayed image, that we master.