Edge video AI pipelines: queues, scheduling, and the real bottlenecks

Getting a video AI model to run on an embedded chip is the easy part. Keeping it stable for hours, under real load, on a device that's getting hot, is the hard part. These are the notes I use when designing that kind of pipeline. They aren't tied to any vendor SDK: swap in your own decoder, NPU runtime, and queue rules.

The goal is to treat latency, dropped frames, and jitter the way you'd treat OS scheduling or memory: find the actual bottleneck, then fix the path that matters.

Contents

1. The scenario
2. A pipeline is stages plus queues
3. Which hardware runs which stage
4. What a stage looks like in code
5. The bottlenecks that aren't compute
6. Profiling the critical path
7. Checklist

1. The scenario

Assume one incoming 1080p stream at 30 frames per second (you may add more streams later), and these steps:

• Decode the compressed video, on the hardware video decoder or a DSP, with the CPU as a fallback.
• Preprocess each frame: resize, normalize, and convert the layout the model expects.
• Detect objects with an INT8 model on the NPU, or the GPU if there's no NPU.
• Track objects and run business logic, mostly on the CPU.
• Output an overlay, a re-encoded video, or just metadata.

In production, the problem is rarely model accuracy. It's slow frames at the 95th and 99th percentile, dropped frames, and uneven timing once the chip heats up or memory gets busy.

2. A pipeline is stages plus queues

A diagram of boxes and arrows shows the order of the stages, but it doesn't tell you how the system behaves. For that, every arrow needs three decisions:

• Who owns each buffer: who allocates it, who frees it, and whether two stages can share it.
• What happens when the next stage is slower: this is backpressure.
• What to give up under load: drop the oldest frame, wait, lower the resolution, or run detection only every few frames.

So the thing to build is stages connected by bounded queues, not stages calling each other directly.

A typical order is camera or file, then decoder, preprocess, detector, tracker, and finally overlay or encode. Each queue between stages gets a maximum depth, a drop rule, and metrics such as how long frames wait and how full the queue is. That's where you control the system when things get busy.

3. Which hardware runs which stage

This is a common pattern, not a rule. Every vendor names its hardware differently.

So scheduling isn't just a thread pool. It's deciding which hardware runs each stage, plus the rules for priority, CPU pinning, and batching.

4. What a stage looks like in code

This is simplified C++ to show the structure: buffer handles, queues, and where the policy decisions go. It's not a ready-to-use library.

// An opaque buffer id: a pool index, a dmabuf handle, or a vendor token.
struct BufferHandle { uint32_t id; };

template <typename T>
struct BoundedQueue {
  bool try_push(const T&);   // returns false when full (backpressure)
  bool try_pop(T& out);
  size_t depth() const;
};

struct StageCtx {
  BoundedQueue<BufferHandle>* in_q;
  BoundedQueue<BufferHandle>* out_q;
  int max_push_retries;
};

// Processes whatever is queued right now, then returns to the scheduler.
void run_preprocess(StageCtx& ctx) {
  BufferHandle in, out;
  while (ctx.in_q->try_pop(in)) {
    if (!allocate_output(out)) {
      release(in);          // no free buffer: drop this frame
      continue;
    }
    preprocess(in, out);
    release(in);            // input is no longer needed

    int tries = 0;
    while (!ctx.out_q->try_push(out)) {
      if (++tries > ctx.max_push_retries) {
        release(out);       // next stage is too slow: drop, don't leak
        break;
      }
      yield_or_sleep();     // policy: back off instead of spinning
    }
  }
}

What to look for when reviewing code like this:

• Every buffer has a clear owner at every point, and every path ends in release, including the drop paths.
• There's an obvious place to record metrics, such as depth() and how often try_push fails.
• Waiting on a full queue backs off or drops, so it can't spin forever and burn the CPU the other stages need.
• Real code also needs explicit synchronization between the CPU and the accelerators, which this sketch leaves out.

5. The bottlenecks that aren't compute

The decoder, preprocessing, and the NPU all share the same chip. They compete for memory bandwidth, the internal bus, and thermal headroom, and one of those usually runs out before the NPU's raw compute does.

Memory bandwidth. Large feature maps, poorly aligned strides, or extra color conversions can saturate memory while the NPU still has capacity to spare.

Heat. Under sustained load the chip lowers its clock speed to cool down, and timing gets uneven. A pipeline that looks fine for five minutes can fail after twenty, so always test for longer than feels necessary.

PCIe and the internal bus. This matters most when the accelerator or camera is a separate chip, but even fully integrated chips show bus contention under heavy load.

6. Profiling the critical path

Use the vendor's GPU or NPU timeline tools where they exist, plus a system-wide view such as perf, eBPF, or Nsight Systems on NVIDIA hardware. Then:

1. Find the longest chain of dependent steps for a single frame. That's the critical path.
2. Decide whether the next fix belongs on that path, or in queue rules and memory traffic instead.

Speeding up code that isn't on the critical path is one of the easiest ways to lose weeks.

7. Checklist

1. Every queue between stages has a maximum depth and a rule for what happens when it's full.
2. You can name the single biggest bottleneck for your workload: memory, heat, the bus, or one specific stage.
3. You report latency as percentiles (p95, p99), not just averages.
4. The team agrees on how the system should degrade before it has to happen in the field.

This is an architecture sketch. Tune the numbers and policies to your chip, OS, and product requirements.