Accelerating FCN Segmentation from 2000ms to 32ms (0.5 FPS → 31 FPS)
This post outlines how I optimized a Fully Convolutional Network (FCN) segmentation pipeline — from 0.5 FPS (~2000ms per frame) to 31 FPS in real-time.
The numbers may not be exact (based on memory), but the key takeaways should still be clear. Also, the goal of this experiment was not to optimize segmentation accuracy, but to achieve real-time inference speed.
Model Overview
The model is an FCN with a ResNet-101 backbone, pre-trained on a subset of COCO 2017 using only the 20 classes that overlap with Pascal VOC. Again — the goal is speed, not accuracy.
Pipeline Overview
ROS2 Image Msg → Convert to OpenCV Matrix → Run FCN Inference → Decode Segmentation Mask → Apply Colormap → Publish to ROS2 Topic
Optimization Steps
Step 0: Baseline — Naive CPU Pipeline (~2000ms, 0.5 FPS)
The entire pipeline ran on CPU to establish a baseline. It took over 2000ms per frame, with inference alone accounting for ~1500ms. The inference step was the obvious bottleneck.
Step 1: Move Inference to GPU via LibTorch (~70ms)
I ported the model to GPU using LibTorch C++ and loaded the .pth file directly. Inference dropped to under 70ms — a massive improvement, but still not real-time.
Step 2: TensorRT Acceleration (~30ms)
I converted the model to ONNX, then compiled it into a TensorRT engine. Despite minor precision loss, inference dropped to under 30ms. This was the single biggest speed gain in the pipeline.
Step 3: Memory Optimization & Streaming (Minor Gain)
Two additional optimizations were tried:
- Switched to CUDA async memory copies
- Experimented with TensorRT multi-streaming
Multi-streaming had little effect since I'm processing one image at a time — batched workloads would benefit more. I switched back to single streaming. At this point, inference was highly optimized with little room left to improve.
Step 4: GPU Decode & Colormap (~1.5ms)
The next bottlenecks were decoding the segmentation mask and applying the colormap — both running on CPU and taking ~80ms total. I rewrote both as a custom CUDA kernel, running fully on GPU. This reduced the combined time to ~1.5ms.
Step 5: ROS Image to GPU — Zero-Copy Input
Finally, I optimized the ROS input stage by converting the image directly to GPU memory, eliminating an unnecessary copy step. The entire pipeline now runs on the GPU — from input to output.
Results
| Step | Change | Latency | FPS |
|---|---|---|---|
| Baseline | CPU pipeline | ~2000ms | 0.5 |
| Step 1 | GPU inference (LibTorch) | ~70ms | ~14 |
| Step 2 | TensorRT engine | ~30ms | ~33 |
| Step 3 | Async copies + streaming | ~30ms | ~33 |
| Step 4 | CUDA decode + colormap | ~1.5ms saved | — |
| Step 5 | Zero-copy ROS input | ~32ms | 31 |
Through a series of focused optimizations — CPU baseline, GPU inference, TensorRT acceleration, GPU decode, and zero-copy input — I achieved 31 FPS real-time performance.
Every millisecond mattered. And it all added up.