Skip to main content

Rendering Photorealistic Scenes in CARLA with NVIDIA NuRec

· 5 min read
Yi-Chen Zhang
Yi-Chen Zhang
Lead Engineer, AI and Autonomous

If you've spent any time in CARLA, you know its strength and its limit at the same time: it's a fantastic, fully controllable simulator, but its Unreal Engine art assets look like a simulator. Buildings, road textures, foliage — it's all clearly synthetic. For a lot of autonomous driving development, that's fine. But when you're trying to close the sim-to-real gap, that visual gap matters.

NVIDIA NuRec is one way to close it. Rather than hand-modeling every building and texture, NuRec uses neural reconstruction — specifically 3D Gaussian Splatting — to turn real recorded sensor data (camera and LiDAR) into a photorealistic, interactive 3D scene. CARLA now has a first-party integration with it. Here's what that integration actually gives you, and how to get it running.

Reconstruction vs. Rendering: Know Which One You're Doing​

The first thing worth being clear on: the CARLA integration is a rendering consumer, not a reconstruction tool.

NuRec's full pipeline looks like this:

Real sensor logs (camera + LiDAR)
        ↓
   NCore format
        ↓
NuRec reconstruction (Gaussian Splatting)
        ↓
  Neural 3D scene (.usdz)
        ↓
   CARLA / Isaac Sim / AlpaSim
        ↓
Simulated camera data

The reconstruction step — turning raw sensor logs into a trained .usdz scene — happens separately, inside a NuRec Docker container, using NVIDIA's own reconstruction tooling. The CARLA tutorial doesn't cover that step at all. What it covers is the last stage: loading an already-reconstructed .usdz scene and rendering novel camera views from it inside CARLA.

Practically, this means you don't need your own sensor rig or reconstruction pipeline to try this out. NVIDIA publishes a large library of pre-trained scenes through the NVIDIA Physical AI Dataset for Autonomous Vehicles on Hugging Face, and that's what the CARLA example scripts pull from by default.

How the Pieces Fit Together​

Under the hood, the integration is a gRPC handoff between two independent processes:

  • CARLA loads the map and manages actors — vehicle state, world ticking — through the normal CARLA API you already know.
  • The NuRec container holds the reconstructed Gaussian-splat scene and does the actual neural rendering, on request, over the NuRec gRPC API.

When your script asks for a camera frame, CARLA isn't drawing it with its own Unreal Engine renderer — it's forwarding the request to the NuRec container, getting back a rendered frame, and handing that frame to your script. This is worth internalizing because it explains something you'll notice immediately once you run the example: CARLA's own default viewport (the CarlaUE4 window) still shows the flat, familiar Unreal Engine world, completely separate from the photorealistic NuRec camera feeds your script displays. Two renderers, two windows, one shared scenario.

Running the Example​

Setup follows the standard CARLA-ecosystem pattern: install Docker and the NVIDIA container toolkit, set up a Python virtual environment, and run the provided installer script from your CARLA root:

./PythonAPI/examples/nvidia/nurec/install_nurec.sh

You'll need a Hugging Face token to pull the sample dataset, and a couple of environment variables set — most importantly NUREC_IMAGE, pointing at the NuRec gRPC Docker image.

From there, replaying a scenario is a single script:

python example_nurec_replay_save_images.py \
  --usdz-filename PhysicalAI-Autonomous-Vehicles-NuRec/sample_set/25.07_release/<scene-id>/<scene-id>.usdz \
  --move-spectator --saveimages

This spins up a Pygame window showing a grid of camera feeds — by default, a front-facing NuRec camera plus left- and right-cross NuRec cameras, alongside any plain CARLA sensors you've configured — while the scenario replays and the ego vehicle follows its recorded trajectory. --saveimages writes each camera's frames to disk as JPEGs, one subfolder per camera.

Customizing the Camera Rig​

If you're trying to match real sensor hardware, the camera setup is driven entirely by a YAML config, editable independent of the script itself. NuRec cameras support F-theta (fisheye) lens models with precise intrinsic parameters — principal point, distortion polynomials — plus custom transform matrices and rolling shutter simulation. You can mix these with ordinary CARLA sensor cameras in the same config, letting the script route each entry to the right renderer automatically.

If your goal is closing the visual sim-to-real gap for perception model training or evaluation, this integration is a genuinely low-friction way to get photorealistic driving scenes into CARLA without standing up a reconstruction pipeline yourself. The reconstruction side — building your own scenes from your own sensor data — is a separate, heavier undertaking, worth its own write-up. But if you just want to see what NuRec-quality rendering looks like inside a tool you already use, the pre-trained dataset plus the example script gets you there in an afternoon.