All posts
C++ Latency EngineeringHFTLow-LatencyC++20Order BookeBPFio_uringBook Release

Announcing C++ Latency Engineering: Inside the Book, the 11 Labs and the 44ns Capstone

C++ Latency Engineering by shivam kumar: a technical deep dive with 11 runnable labs, 100+ worked problems, and a capstone limit order book cutting latency to 44ns.

shivam kumar4 min read

Most C++ performance advice focuses on arithmetic averages: mean throughput, requests per second, or overall CPU utilization. However, in low-latency systems—such as high-frequency trading matching engines, kernel-bypass firewalls, and real-time audio pipelines—what dictates system viability is the one-in-a-thousand outlier (p99.9p_{99.9} and p99.99p_{99.99}). A single microsecond stall creates queue buffer bloat, missed execution windows, and non-deterministic behavior.

We are announcing the release of C++ Latency Engineering by shivam kumar.


1. Structure: 11 Runnable Engineering Labs

Rather than presenting theoretical models, C++ Latency Engineering is structured around 11 runnable benchmark labs supplied in an assignment archive (.zip). Each lab isolates a distinct physical or kernel source of delay:

LabPrimary Source of DelayImplementation & ToolsLatency Objective
01Clock Measurement OverheadCPU cycle serialization (rdtsc / rdtscp)Sub-2ns Timer
02CPU Cache & Memory HierarchyL1/L2/L3 miss penalties and alignas(64)Zero False Sharing
03Memory Allocation & Page FaultsPre-allocated contiguous slab memory poolsZero Dynamic Allocations
04Branch Predictor InefficienciesBranchless conditional moves and software prefetchPipeline Continuity
05Thread Handoff ContentionSPSC ring buffer with acquire-release semanticsLock-Free Handoff
06Virtual Memory & TLB MissesLinux 2MB and 1GB HugePages configurationDeterministic Page Walks
07Operating System JitterCore isolation (isolcpus), nohz_full and C-state locksZero Context Switches
08Profiling Hardware CountersProfiling with perf, Intel VTune, bpftrace, and flame graphsRoot-Cause Isolation
09Kernel Network DelaysKernel bypass via Linux io_uring and eBPF/XDP hooksWire-Speed Ingress
10Compiler Settings & VectorizationAVX-512 vectorization, Link-Time Optimization (LTO), and PGOCode Optimization
11The Capstone Matching CoreInstitutional Limit Order Book cutting latency to 44nsSub-44ns Execution

2. The Capstone: From 262ns Down to 44ns

In Lab 11, readers construct a deterministic C++20 limit order book matching engine. By replacing standard pointer-heavy container trees (std::map) with a cache-conscious flat ladder and contiguous memory pools, median tick-to-trade time is reduced from 262 ns down to 44 ns.

ImplementationMedian (p50)Tail Latency (p99)Worst (p99.9)Classification
Mutex + std::map Standard Pattern262 ns1,420 ns8,900 nsHigh Jitter
SPSC Lock-Free Queue78 ns142 ns310 nsModerate
AlgoMesh Cache-Aligned Order Book44 ns64 ns88 nsDeterministic

3. Core Architectural Implementations

Cache-Line Alignment with alignas(64)

Modern x86-64 processors retrieve memory from cache hierarchies in 64-byte lines. When an order record spans cache-line boundaries, each access incurs two cache fills. The book demonstrates exact structure packing to ensure order nodes fit completely within one line:

include/order.hppcpp
// filename: include/order.hpp
#pragma once
#include <cstdint>

enum class Side : uint8_t { Buy = 0, Sell = 1 };

/* Fit exactly into a single 64-byte CPU cache line to eliminate false sharing */
struct alignas(64) Order {
    uint64_t order_id;     // 8 bytes
    uint64_t client_id;    // 8 bytes
    uint64_t price;        // 8 bytes (fixed-point integer in ticks)
    uint32_t quantity;     // 4 bytes
    uint32_t filled_qty;   // 4 bytes
    Side side;             // 1 byte
    uint8_t padding[7];    // 7 bytes padding to align intrusive pointers
    Order* next{nullptr};  // 8 bytes (intrusive doubly-linked list)
    Order* prev{nullptr};  // 8 bytes
    uint64_t timestamp_ns; // 8 bytes (ingress clock timestamp)
};
static_assert(sizeof(Order) == 64, "Order struct must be exactly 64 bytes");

Zero-Allocation Memory Pool

In latency-critical paths, dynamic heap memory allocations (malloc / new) are prohibited due to lock contention in standard library allocators and potential page faults. A pre-allocated contiguous memory pool provides O(1)O(1) allocation without system calls:

include/order_pool.hppcpp
// filename: include/order_pool.hpp
#pragma once
#include "order.hpp"
#include <vector>

template <size_t Capacity = 1000000>
class OrderPool {
public:
    OrderPool() {
        pool_.resize(Capacity);
        for (size_t i = 0; i < Capacity - 1; ++i) {
            pool_[i].next = &pool_[i + 1];
        }
        pool_[Capacity - 1].next = nullptr;
        free_head_ = &pool_[0];
    }

    Order* allocate() noexcept {
        if (!free_head_) [[unlikely]] return nullptr;
        Order* order = free_head_;
        free_head_ = free_head_->next;
        order->next = nullptr;
        order->prev = nullptr;
        return order;
    }

    void deallocate(Order* order) noexcept {
        order->next = free_head_;
        free_head_ = order;
    }

private:
    std::vector<Order> pool_;
    Order* free_head_{nullptr};
};

Accurate Cycle Timing with rdtsc

Cycle-level profiling requires hardware pipeline serialization using _mm_lfence() to prevent the processor from executing measurement instructions out-of-order:

src/timer.hppcpp
// filename: src/timer.hpp
#pragma once
#include <x86intrin.h>
#include <cstdint>

// Cycle-accurate nanosecond benchmark timer using CPU serialization
static inline uint64_t rdtsc_start() {
    unsigned int aux;
    return __rdtscp(&aux);
}

static inline uint64_t rdtsc_end() {
    unsigned int aux;
    uint64_t t = __rdtscp(&aux);
    _mm_lfence(); // serialize instruction stream
    return t;
}

4. Reading Environment and Resources

Readers access the book through our integrated browser reader:

  • Reading Modes: Single-Page, Two-Page book spread with 3D turn animations, and continuous scroll.
  • Themes: Paper, Sepia, White, and Dark Mode.
  • Synchronization: Reading position automatically synchronizes across devices.
  • Assignment Archive: Downloadable .zip containing all 11 runnable labs, benchmarks, and test suites.

Comments

Sign in to join the conversation.

Loading comments…