What is ONNX (Open Neural Network Exchange)?

Definition

ONNX (Open Neural Network Exchange) is an open-source format for representing machine learning models as computation graphs. Originally developed by Microsoft and Facebook (now Meta), ONNX defines a common set of operators and data types that allow models to be exported from one framework (e.g., PyTorch) and imported into another (e.g., TensorFlow, or specialized inference engines like ONNX Runtime). The format represents models as directed acyclic graphs where nodes are operations (convolution, matrix multiplication, attention) and edges are the tensors flowing between them.

ONNX serves as a bridge between the diverse ML ecosystem. Different teams may prefer different training frameworks, but deployment environments often have strict requirements — a mobile app might need Core ML, an embedded device might need TensorRT, and a cloud service might use ONNX Runtime. ONNX enables a single trained model to be deployed across all these environments through format conversion, without retraining or manual reimplementation.

For language models, ONNX is particularly relevant for inference optimization. ONNX Runtime, Microsoft's inference engine, applies graph-level optimizations — operator fusion, constant folding, memory planning — that can significantly accelerate inference. These optimizations are applied to the ONNX graph representation, making them framework-agnostic. The ONNX ecosystem also supports quantization tools that can compress models to INT8 or INT4 precision directly on the ONNX graph.

Why It Matters

In enterprise environments, the training framework and the deployment environment are often different. Research teams may develop models in PyTorch, but production systems might require TensorRT for GPU inference, ONNX Runtime for CPU inference, or Core ML for Apple devices. Without a portable format like ONNX, each deployment target would require reimplementing the model — an expensive, error-prone process.

ONNX Runtime specifically has become a key inference engine for production deployments due to its performance. On CPU workloads, ONNX Runtime with optimizations can be 2-4x faster than native PyTorch inference. This performance advantage makes ONNX conversion a standard step in production deployment pipelines, especially for models that need to serve at scale with strict latency requirements.

How It Works

ONNX conversion works by tracing or scripting a model's computation graph. In PyTorch, torch.onnx.export() runs a sample input through the model, records all operations performed, and serializes the resulting computation graph in ONNX format. The exported graph captures both the model architecture (operations and their connections) and the trained weights (as constant tensors in the graph).

The ONNX graph can then be loaded by any compatible runtime. ONNX Runtime applies optimization passes: fusing adjacent operations (e.g., combining MatMul + Add + GELU into a single fused kernel), eliminating redundant computations, and planning memory allocation to minimize overhead. The optimized graph is compiled into executable kernels for the target hardware (CPU, GPU, or specialized accelerators), producing an inference session that processes inputs with minimal overhead.

Example Use Case

A company trains a text classification model in PyTorch for customer intent detection. For production, they need to serve the model on CPU instances to control costs. They export the model to ONNX, apply ONNX Runtime's graph optimizations and INT8 quantization, and achieve 3.5x throughput improvement over native PyTorch CPU inference — enabling them to serve 10,000 requests per second on a single instance instead of requiring three instances.

Key Takeaways

• ONNX is an open standard for portable model representation across ML frameworks.
• It enables training in one framework and deploying in another without reimplementation.
• ONNX Runtime provides graph-level optimizations that significantly accelerate inference.
• The format represents models as directed acyclic computation graphs with operations and tensors.
• ONNX is particularly valuable for CPU inference optimization and cross-platform deployment.