Technical Architecture
Lunyn's parsing engine represents a complete rethinking of market data processing architecture. Traditional approaches sacrifice either performance or maintainability. We engineered both.
Core Architecture
Parsing operations occur directly on ingested buffers without intermediate allocations. Message boundaries are identified through pointer arithmetic, and field access happens via calculated offsets.
Benchmark scope: memory-mapped historical data with per-message heap allocation avoided in the parser hot path.
Multi-threaded processing using lock-free queues and wait-free data structures. Each parsing thread operates independently with minimal synchronization. Message distribution uses MPSC (multi-producer, single-consumer) queues for message routing, memory ordering guarantees for safe concurrent access, and per-thread state to eliminate cache contention.
Benchmark result: 8 vCPUs reached 1,419 M msg/s for framing and 499.7 M msg/s for full decode on the tested host.
Critical parsing operations leverage AVX2 instruction sets to process multiple fields simultaneously: Parallel byte-to-integer conversion, vectorized field extraction, and SIMD-optimized timestamp parsing. Specific operations can process 8 message fields in parallel vs. sequential processing.
Implementation detail: direct binary field access reduces intermediate parsing work in supported message paths.
Comprehensive error handling for malformed messages, out-of-sequence data, protocol violations, and edge cases. The parser reports detailed diagnostics for debugging.
System maintains parsing state across errors with detailed diagnostics.
Complete ITCH 5.0 Implementation
| Message Type | Frequency | Parse Latency (framing / full decode) | Memory |
|---|---|---|---|
| System Event | Rare | 5.2 ns / 9.9 ns | 16B |
| Stock Directory | Daily | 5.2 ns / 9.9 ns | 256B |
| Add Order | Very High | 5.2 ns / 9.9 ns | 64B |
| Add Order (MPID) | High | 5.2 ns / 9.9 ns | 72B |
| Order Executed | High | 5.2 ns / 9.9 ns | 48B |
| Order Executed (Price) | Medium | 5.2 ns / 9.9 ns | 52B |
| Order Cancel | High | 5.2 ns / 9.9 ns | 40B |
| Order Delete | Medium | 5.2 ns / 9.9 ns | 32B |
| Order Replace | High | 5.2 ns / 9.9 ns | 64B |
| Trade (Non-Cross) | High | 5.2 ns / 9.9 ns | 56B |
| Cross Trade | Low | 5.2 ns / 9.9 ns | 72B |
| Broken Trade | Rare | 5.2 ns / 9.9 ns | 40B |
| NOII | High | 5.2 ns / 9.9 ns | 128B |
| RPII | Medium | 5.2 ns / 9.9 ns | 48B |
Validated Performance Metrics
Median results across 10 full-dataset runs
Measured on the October 30, 2019 historical ITCH session
Google Cloud c3-standard-8 benchmark host
Median results; disk and network overhead excluded
All metrics reproducible via our benchmark suite. Contact us for evaluation access.
View Full Benchmark ReportDeployment Options
Link directly into your application via provided API. Supports real-time streaming interfaces, batch file processing, and custom message routing.
Deploy as microservice with gRPC or REST API, Kafka/Redis output streams, and monitoring and alerting integration.
Command-line tool for batch conversion (ITCH → Parquet/CSV), historical data processing, and data quality validation.
Evaluate the Architecture
Request access to detailed technical documentation, benchmark methodologies, and integration guides for your evaluation.