Purpose-built for GPU clusters

NULLHZ

The OS Built for LLM Training at Scale

An operating system engineered from the kernel up for LLM training workloads. Zero scheduler ticks on training cores. Immutable, auditable, and purpose-built for GPU cloud infrastructure.

nullhz — boot sequence
# nullhz v0.9.1 — kernel 6.6.22-rt28-nullhz
# CONFIG_NO_HZ_FULL=y | CONFIG_PREEMPT_RT=y
[ 0.000] CPU isolation: cores 2-127 reserved for training
[ 0.001] Tick suppressed on isolated cores (nohz_full)
[ 0.002] RCU callbacks offloaded (rcu_nocbs=2-127)
[ 0.004] IRQ affinity pinned to cores 0-1
[ 0.005] C-states limited: max_cstate=1
[ 0.008] Huge pages: 16384 × 2MB pre-allocated
[ 0.009] RDMA: MLX5 ConnectX-7 initialized
[ 0.012] BTRFS checkpoint volume mounted
[ 0.014] NVIDIA driver H100 — loaded
[ 0.021] nullhz ready — 127 training cores available
root@nullhz:~$
~20%
Training Efficiency Gain
<100ms
Checkpoint Latency
0
Background Daemons on Training Cores
RT
PREEMPT_RT Kernel — Always

A SCRIPT ON UBUNTU
CAN'T DO THIS

What separates nullhz from a tuning script: kernel recompilation, cluster-scale reproducibility, OS-level fault integration, and a security posture your customers can actually audit.

01
Kernel Recompilation

Ubuntu ships a precompiled generic kernel. You can tune sysctls at runtime, but you cannot enable CONFIG_PREEMPT_RT, CONFIG_NO_HZ_FULL, or strip out netfilter and audit subsystems without recompiling. nullhz is built specifically for training workloads — every kernel byte is intentional, nothing is inherited from a general-purpose distro.

02
Guaranteed Node Identity

Across thousands of nodes, Ubuntu + scripts drift — different kernel patch levels, different package versions, different uptime state. nullhz is an immutable image artifact. Every node is bit-for-bit identical. When a node straggles, you know it's not an OS configuration issue.

03
OS-Level Fault Tolerance

A script triggers checkpoints. nullhz intercepts Machine Check Exceptions at the kernel level, detects training stalls via a hardware watchdog module, and triggers atomic BTRFS snapshots before a node fails — not after. Recovery is built into the OS, not bolted on top.

04
Minimal Attack Surface

Ubuntu runs with hundreds of background processes, apt, snapd, and avahi — even after tuning. nullhz contains exactly what LLM training requires and nothing else. Every package is documented, auditable, and reviewed. A complete manifest ships with every image.

05
Zero Interrupt Interference

The name says it. HZ=0 on training cores. The scheduler timer tick is fully suppressed via nohz_full. All hardware interrupts are pinned to OS-only cores. Training threads run uninterrupted — no kernel, no daemon, no NIC interrupt ever preempts them.

06
Driver Support SLA

Every NVIDIA driver release gets a validated nullhz image within 14 days, with a 48-hour hotfix SLA for critical issues. Your cluster stays current without internal OS maintenance overhead. Kernel CVE patches are included in every release cycle.

WHERE THE GAINS COME FROM

Four optimization layers, independently benchmarkable, cumulatively delivering ~20% efficiency improvement on A100 and H100 clusters.

Jitter Reduction 8–12%

PREEMPT_RT kernel + nohz_full + CPU isolation eliminates scheduler jitter on training cores. Synchronized distributed training stops waiting for the slowest node.

RDMA / Network 5–8%

RoCE v2 tuning, GRO/TSO disabled on training NICs, 256MB socket buffers, and ECN/PFC configuration optimized for NCCL all-reduce patterns.

Memory & NUMA 3–5%

16K pre-allocated 2MB huge pages, strict NUMA pinning to GPU-local memory, vm.swappiness=0, and disabled background writeback interruption.

Minimal OS Overhead 2–4%

No systemd, no apt, no snapd. A minimal init sequence with zero non-essential kernel threads competing for CPU time, cache, or memory bandwidth.

// Iteration Time — ms/step (lower is better)
Ubuntu
247ms
nullhz
200ms
// GPU Utilization Variance — stddev (lower is better)
Ubuntu
±8.4%
nullhz
±1.8%
// Checkpoint Latency (lower is better)
Ubuntu
~4.2s
nullhz
<100ms
Ubuntu 22.04
nullhz

MINIMAL SURFACE.
MAXIMUM TRUST.

nullhz is designed for environments where your customers demand infrastructure audits. Every package is justified. Every kernel module is intentional.

Immutable Read-Only Root

The OS partition is mounted read-only after boot. No package manager, no shell access by default, no drift. What ships is what runs — always. Mutations require a signed image update.

Auditable Package Manifest

Every binary in the image is documented. A full package manifest and kernel config diff vs upstream ships with every release — making SOC2 and infrastructure audits straightforward for your customers.

No Unnecessary Network Services

No avahi, no mDNS, no CUPS, no D-Bus, no systemd-resolved. The only network listeners are RDMA management and your training workload. Every open port is deliberate.

Signed Image Delivery

Every nullhz release is cryptographically signed. Nodes validate image integrity at boot using secure boot chains. Tampered or unsigned images are rejected before kernel handoff.

Kernel CVE Patch SLA

Critical kernel CVEs patched and a new validated image available within 48 hours. Standard CVEs within 14 days. Your cluster stays hardened without internal security engineering overhead.

eBPF-Based Audit Trail

Kernel-level eBPF probes provide an immutable audit trail of system calls, process launches, and network connections — without the overhead of userspace audit daemons. Logs ship directly to your SIEM.

// Attack Surface Comparison — Running Processes
Ubuntu
247 processes
nullhz
18 processes
Exposed process
Required process
// Kernel Modules Loaded
Ubuntu
180+ modules
nullhz
23 modules

THE KERNEL THAT
UBUNTU CAN'T SHIP

These aren't sysctl tunables. They require a purpose-built kernel — which is exactly what nullhz ships.

Ubuntu 22.04 — generic kernel Generic
- CONFIG_PREEMPT_NONE=y
- CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_NO_HZ_IDLE=y
- # CONFIG_NO_HZ_FULL is not set
- # CONFIG_PREEMPT_RT is not set
CONFIG_HZ_250=y
- CONFIG_AUDIT=y
- CONFIG_NETFILTER=y
- CONFIG_BRIDGE=y
- CONFIG_VXLAN=y
- CONFIG_RCU_BOOST=n
CONFIG_NUMA=y
- # CONFIG_CPU_IDLE_GOV_TEO is default
nullhz — training kernel Optimized
+ CONFIG_PREEMPT_RT=y
+ CONFIG_PREEMPT=y
+ CONFIG_NO_HZ_FULL=y
+ # nohz_full=2-127 applied at boot
+ CONFIG_HZ_100=y
+ CONFIG_RCU_NOCB_CPU=y
+ # CONFIG_AUDIT is not set
+ # CONFIG_NETFILTER is not set
+ # CONFIG_BRIDGE is not set
+ # CONFIG_VXLAN is not set
+ CONFIG_CPU_IDLE_GOV_LADDER=n
+ CONFIG_NUMA=y
+ CONFIG_HUGETLBFS=y

FOUR LAYERS.
ONE PURPOSE.

Every component of nullhz exists to maximize GPU utilization and eliminate the OS as a bottleneck.

01
// Layer 1
RT Kernel
  • PREEMPT_RT patch applied
  • NO_HZ_FULL tickless on cores 2+
  • RCU callbacks offloaded
  • C-states locked at C1
  • Performance cpufreq always
  • Audit + netfilter stripped
02
// Layer 2
Memory & NUMA
  • 16K × 2MB huge pages pre-allocated
  • NUMA pinning to GPU-local node
  • swappiness=0, no swap
  • Dirty writeback deferred
  • THP always-on mode
  • OOM killer disabled for training
03
// Layer 3
RDMA Network
  • MLX5 ConnectX-7 native driver
  • RoCE v2 + ECN/PFC configured
  • GRO/GSO/TSO disabled
  • 256MB socket buffers
  • NCCL thread NUMA-pinned
  • Zero userspace network overhead
04
// Layer 4
Fault Tolerance
  • BTRFS atomic snapshots <100ms
  • MCE interception pre-failure
  • Hardware watchdog + auto-restart
  • 3-checkpoint rotation policy
  • eBPF audit trail to SIEM
  • Signed immutable image delivery
READY TO
BENCHMARK?

We'll run nullhz against your current stack on your hardware. You keep the results.