Quantized models and zero-copy buffers: the contracts that make edge AI ship

This is a follow-up to Edge video AI pipelines: queues, scheduling, and the real bottlenecks. That post was about throughput and scheduling. This one is about the model files, memory, and making the same pipeline run on different chips. It works with any inference runtime: TensorRT, Qualcomm's QNN, OpenVINO, or whatever your board ships with.

The idea is simple: treat model export, calibration, and buffer ownership as carefully as you treat CI and deployment.

Contents

1. Three layers, and the contracts between them
2. Post-training quantization vs quantization-aware training
3. What "supports this model" should really mean
4. Zero-copy still needs synchronization
5. Making buffer ownership explicit
6. Describing hardware with capability flags
7. Checklist

1. Three layers, and the contracts between them

I think of the stack as three layers:

• Training: model quality, regularization, and sometimes a quantization-aware training setup.
• Conversion: the export format, which operations are supported, layer fusion, calibration data, and optional sparsity.
• Runtime and engine: tensor layouts, batching, which hardware runs what, how long buffers live, and what happens when an operation isn't supported.

Most bugs show up where these layers meet, not in the export file itself. So what matters most is writing down what each layer expects from the next.

2. Post-training quantization vs quantization-aware training

Neither one replaces testing on the actual device. On the edge, whether a model ships depends on latency, memory use, and worst-case behavior, not just accuracy.

3. What "supports this model" should really mean

Saying an engine supports a model should mean more than "it can load the file." A useful engine also tracks:

• The tensor layout each backend expects, such as NCHW vs NHWC, or packed formats.
• Quantization parameters for each tensor, meaning the scales and zero points.
• The batching rules: fixed batch size, dynamic shapes, and padding.
• What to do when an operation isn't supported: split the graph, fall back to the CPU, or refuse to run.

Without this, you get a model that runs fine on a desktop but can't run reliably on the device.

4. Zero-copy still needs synchronization

Zero-copy means stages share the same memory (shared buffer pools, DMA-friendly buffers, import APIs) instead of copying frames between them. That saves a lot of memcpy time. But it doesn't remove three things:

• Cache maintenance. When the CPU and an accelerator share memory, someone has to flush or invalidate caches so each side sees the latest data.
• Fences or events. The consumer has to wait until the producer has actually finished writing.
• Lifetime rules. You have to decide who may overwrite a buffer while another stage is still reading it.

In a real frame path, the decoder writes into buffers from a shared pool, the NPU reads a tensor view of that same memory, and used buffers go back to the pool. No copies happen, but every handoff still needs an explicit sync point.

5. Making buffer ownership explicit

Simplified C++:

#include <atomic>

enum class Owner { Decoder, Preprocess, NPU, CPU };

struct FrameBuffer {
  void*  y;
  void*  uv;
  size_t stride_y;
  std::atomic<Owner> owner;
};

bool handoff(FrameBuffer& fb, Owner from, Owner to) {
  // Finish the previous owner's work first: wait on its fence and
  // flush or invalidate caches (chip-specific).
  sync_for(fb, from, to);

  // Then transfer ownership atomically, so two stages can't both claim it.
  Owner expected = from;
  return fb.owner.compare_exchange_strong(expected, to);
}

The enum isn't the point. The point is to make an illegal ownership state hard to write, so it gets caught in code review and in runtime checks. Two details matter here. The sync happens before ownership changes, so the next owner never sees half-written data. And the change itself is atomic, because stages run on different threads.

6. Describing hardware with capability flags

Instead of scattering #ifdefs for each chip through your business logic, expose a small set of capabilities that the runtime can check:

struct DeviceCaps {
  bool resize_on_fast_ip;
  bool int8_preferred_path;
  bool dmabuf_importer;
};

DeviceCaps probe(const char* platform_id);

When the engine plans a model, it reads these flags and picks the backends. Product code doesn't change, and supporting a new chip only means adding its capability data at the edge of the system.

7. Checklist

1. Calibration data and QAT artifacts are versioned together with the exact export pipeline used in CI.
2. Every buffer handoff has a named owner and a sync rule.
3. Fallback paths are tested, not just documented.
4. The hardware layer exposes capabilities, not raw vendor headers, at a boundary you control.

This is an architecture sketch. Backend names are examples, so check them against your vendor SDKs and safety requirements.