Turning Real Driving Logs into Closed-Loop CARLA Scenarios with Cam2Sim

Simulators are great for autonomous driving testing because they're safe, reproducible, and fully controllable — but their rendered worlds usually look nothing like the real world. A perception or end-to-end driving model that performs well in simulation can behave very differently once it sees real camera data. This is the classic sim-to-real gap, and it's the problem a recent paper called Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation sets out to close.

Rather than treating real-world recordings and simulation as two separate worlds, Cam2Sim converts a real driving recording directly into a playable, closed-loop CARLA scenario, using Gaussian Splatting so the simulator's camera output visually matches the original recording instead of CARLA's native rendering. I went through the full replication process myself — here's what the pipeline does, what I found reproducing it, and my replication run:

Why Closed-Loop Matters

A lot of prior neural-rendering work — DriveRecon, for example — focuses on reconstructing realistic scenes or generating novel viewpoints. That's useful, but those reconstructions aren't necessarily executable environments where an autonomous driving system (ADS) can actually drive and interact with the world.

Cam2Sim bridges that gap. The ADS under test receives Gaussian-Splatting-rendered camera frames in real time, and its steering commands feed back into the vehicle's trajectory inside CARLA. That closes the loop between perception, control, vehicle motion, and rendered sensor feedback — instead of just replaying a fixed video, the model's own decisions actually change what it sees next.

How the Reconstruction Pipeline Works

The pipeline takes front-facing camera images and vehicle poses as inputs, optionally pulling LiDAR, odometry, steering, and model-prediction data from ROS bags. It breaks down into five stages:

1. Data extraction — synchronized camera frames, ego-vehicle poses, and optional LiDAR/steering/prediction streams are pulled from the ROS recordings, with timestamp interpolation to align the different sensor streams.

2. Dataset processing — parked vehicles are detected with FCOS3D (monocular) or PointPillars (when LiDAR is available), then clustered and averaged into final poses. Road geometry comes from OpenStreetMap via the Overpass API. This stage also preps the Gaussian Splatting inputs: selecting frames, removing the ego-vehicle hood, generating sky masks with SegFormer, and chunking the route into smaller segments.

3. Simulation asset generation — the processed data becomes CARLA-compatible assets: an OpenDRIVE map, replay trajectories, and spawn poses for the ego vehicle and parked vehicles. Both center-position and rear-axle trajectory representations are generated, along with the WGS84-to-OpenDRIVE coordinate transform.

4. Gaussian Splatting preparation — COLMAP handles camera-pose and point-cloud estimation, and local Gaussian Splatting models are trained per route segment using Nerfstudio (splatfacto / splatfacto-big). The GS coordinate system is then aligned to CARLA's through pose matching.

5. Driving simulation — the scenario runs in CARLA in either trajectory-replay mode (frame-by-frame playback for inspection) or closed-loop mode, where a DAVE-2-style end-to-end model drives using either native CARLA rendering (the simulation-only baseline) or the Gaussian-Splatting-rendered images.

What the Paper Found

The authors evaluated Cam2Sim on a 450-meter urban street in Munich. The reconstructed environment reached a mean semantic IoU of 0.774 against the real scene — 0.847 for road, 0.889 for background, and a noticeably weaker 0.577 for vehicles, mostly due to localization error, occlusion, and the difficulty of estimating vehicle poses from recorded sensor data alone.

The closed-loop numbers are the headline result, though: with the Gaussian-Splatting-enhanced rendering, the driving model completed all three evaluation runs and reproduced behavior close to the real-world runs. Using CARLA's native rendering on the exact same scenario, the same model failed all three runs — driving outside the road boundary and hitting simulated collisions. Same geometry, same scenario, different rendering, completely different outcome. That's a fairly strong signal that the visual rendering gap alone can meaningfully change how an end-to-end driving model behaves.

Reproducing It

The Cam2Sim repo ships both a Quick Start and a full Replication guide. The Quick Start was straightforward. Reproducing the full pipeline was not.

The biggest source of friction was environment drift: the authors built and tested on Ubuntu 20.04 with CUDA 11.8, while my machine runs Ubuntu 26.04 with CUDA 13.3. That gap cascaded into dependency conflicts across nearly every stage of the pipeline. The fix that actually worked was containerizing — I put together a Docker Compose setup that isolated each stage with its own required dependency versions rather than fighting to get everything to coexist in one environment.

Is This Production-Ready?

Going through the full replication made me think about how mature this kind of pipeline actually is outside a research setting. As a research framework, Cam2Sim is genuinely compelling — it offers a real path from raw driving logs to visually realistic, closed-loop simulation scenarios. But the effort involved in reproducing it, the number of research-grade tools it stitches together, and its sensitivity to exact software and CUDA versions all raise a real question about production readiness.

Would autonomous driving companies actually adopt a Cam2Sim-like pipeline as part of their validation and testing infrastructure? More specifically: could this become practical for large-scale regression testing and scenario replay, or is it still best thought of as a promising research tool that needs meaningfully more engineering work before it fits into a production AD validation workflow?