Deploying RF-DETR: A Transformer-Based Detector in TensorRT

I recently deployed RF-DETR — Roboflow's real-time, transformer-based object detector — into a TensorRT-accelerated ROS2 pipeline. This post covers what RF-DETR is, why I picked it, and how the deployment is structured.

Why RF-DETR

Object detection has long had an awkward tradeoff baked in: YOLO-family models are fast enough for real-time use but tend to give up some accuracy for that speed, while classic DETR-style transformer detectors are more accurate but historically too slow to run in real time. RF-DETR, Roboflow's detector, is an attempt to close that gap — a strong DINOv2 backbone paired with a deliberately lightweight decoder, landing above the YOLO speed/accuracy curve on COCO benchmarks. Since its release it's gotten a lot of attention in the CV community, and Roboflow has kept building on it — a keypoint-detection variant is now available as a preview, alongside the existing instance-segmentation support.

Most of my prior detection work (FCOS, FCN, SCNN) has been CNN-based, so RF-DETR is a chance to bring a transformer-based detector into the same real-time, production-grade pipeline — same TensorRT deployment discipline, same ROS2 integration pattern, different model family under the hood. My port is detection-only: I stripped out the segmentation head, since instance segmentation isn't what I'm after here.

Two-Repo Architecture

Like my other perception components, the deployment splits cleanly into two layers:

• rf_detr_trt_backend — a standalone, framework-agnostic TensorRT inference library. No ROS dependency; it just takes an image in and returns detections out.
• rf_detr_detection — the ROS2 node that wraps the backend: subscribes to a camera topic, runs inference, and publishes vision_msgs/Detection2DArray plus an optional overlay image for RViz.

This separation means the backend is reusable outside ROS entirely, and the ROS layer stays a thin adapter.

The Export Pipeline

Getting from a PyTorch checkpoint to a deployable engine follows the same three stages as my other TensorRT work, and it's fully automated through CMake so a plain colcon build regenerates everything when the source model changes:

1. PyTorch → ONNX (FP32) via the official roboflow/rf-detr export path
2. FP32 → FP16 ONNX using NVIDIA ModelOpt's AutoCast, keeping I/O tensors in FP32 for compatibility with the C++ preprocessing code
3. ONNX → TensorRT engine via trtexec

The same script also supports fine-tuned checkpoints (e.g. a 3-class KITTI fine-tune), auto-inferring the model variant from the checkpoint itself.

Two Details Worth Calling Out

Square input only. RF-DETR's DINOv2 backbone uses windowed attention that has no non-square path — every input has to be exactly square (704×704 for the large variant). Since real camera frames (like KITTI's 1242×375) aren't square, the ROS node letterbox-pads each frame before inference and maps the resulting boxes back to the original image afterward, so downstream consumers never see square-space coordinates.

Sigmoid decoding, not softmax. RF-DETR's classification head is a sigmoid/focal-style head rather than a softmax. That means decoding can't use a simple per-query argmax — every (query, class) pair is an independent candidate, so the correct decode is a global top-k across all query × class pairs. It's a small detail, but it's the kind of thing that silently produces wrong detections if you carry over assumptions from a softmax-based detector.

GPU-Side Preprocessing

Like the rest of my TensorRT backends, preprocessing runs as a custom CUDA kernel that fuses BGR→RGB conversion, HWC→CHW layout, and normalization into a single pass — with the mean/std normalization folded into one multiply-subtract instead of a divide. Unlike some of my other kernels, mean/std are passed as plain kernel arguments rather than baked into __constant__ memory, since RF-DETR backends may be instantiated with different per-checkpoint stats and constant memory would let concurrent instances silently clobber each other.

ROS2 Integration

The node follows the same pattern as my other real-time perception nodes: a bounded processing queue decouples the image subscription from the inference rate (dropping the oldest frame under backpressure rather than blocking), and it runs on the EventsCBGExecutor, which uses noticeably less CPU than MultiThreadedExecutor for this kind of callback-heavy node.

Note: This package targets ROS2 Lyrical on Ubuntu 26.04. If you're on Jazzy or Humble, expect build errors — some of the rclcpp APIs and CMakeLists.txt conventions used here have changed across distros.

Performance

The included unit test suite covers correctness (rejecting wrong-sized or non-contiguous images) alongside timing benchmarks. On an RTX 2080, a 100-iteration benchmark averaged 7.83 ms per inference (~128 FPS), with steady-state single-call runs (including CPU-side decode) landing around 9 ms. On a RTX 5090, the same engine comes in under 3 ms per inference — comfortably fast enough to share GPU time with the other perception models in the stack.

What's Next

RF-DETR closes out a run of 2D perception work — FCOS, FCN, SCNN, and now this. From here I'm shifting focus to 3D object detection. Stay tuned.