Paper Review—CARLA: An Open Urban Driving Simulator

Main Contribution

The research problem in this paper is simulation and sensorimotor control, specifically for autonomous driving. To break the physical constraint for further advances in autonomous driving research and development, the authors proposed a digitalized simulation platform for autonomous driving.

The main contributions of this paper include the introduction of CARLA as an open simulator for urban driving, which supports flexible configurations. Moreover, the authors also showcased CARLA’s ability to train models and validate the performance of different approaches for autonomous driving.

Method

In this paper, the authors first introduced the simulation engine and then utilizing CARLA for analysis of autonomous driving systems with three pipelines.

First, the architecture of CARLA consists of a client-side Python client API and a server-side rendering from Unreal Engine. CARLA allows for a configurable sensor suite, including depth map and semantic segmentation map. The digital assets are custom made, and it supports diverse weather and lighting, making the rendering realistic for autonomous driving research. To show CARLA’s feasibility and usability in the research environment, the authors evaluate three popular approaches for studying autonomous driving, including modular pipeline, deep imitation learning, and deep reinforcement learning. A modular pipeline decomposes this task to modular components as perception, local planner, and PID controller. The perception component is fulfilled by semantic segmentation network and classification to tell if it is at an intersection or not. The local planner is a custom state machine of 5 discrete states to avoid collisions and coordinates low-level navigation.

Secondly, for the end-to-end conditional imitation training, training data (observations and actions) is recorded from human drivers who also provide the corresponding commands. The goal is for the model to learn to predict the expert’s actions with a deep convolutional network, however, the drawback is that there’s no way to control the system, for example turning at an intersection. Thirdly, the authors explored end-to-end reinforcement learning, which takes 12 days to train the model while imitation learning takes roughly 12 hours.

Reinforcement learning doesn’t need human driving, instead, it learns from the interaction with the environment by maximizing the sum of future rewards. Imitation learning maps sensory input to driving commands, while reinforcement learning maps the sensory input to actions with a deep network.

Overall, CARLA targets urban driving scenes specifically and supports end-to-end learning for future research.

What do I think?

CARLA has become a popular choice of simulator for a dynamic urban environment with traffic for its physics integration and visual realism. Due to its open-source nature, it has wide-reaching community support, and it is actively adding new features to its ecosystem on the Github repo. Despite being a virtual simulation, it made possible the CARLA Autonomous Driving Challenge and lowered the entry barriers to conduct research projects for autonomous driving.

In one of my previous projects, I worked on a draining problem for autonomous driving to remove the rain-streak and the fogginess of rainy imagery, so that the system can “see” clearly. Due to the lack of real-world paired images (same road condition with only the weather as a variable), we experimented with CARLA where we drove the same road in rainy and sunny weather with varying light conditions. In the end, we proposed a CycleGan-like network so that it can be trained unsurprised with real-world images, but I find CARLA very useful in the proof of concept for its precise variable controlling and also convenient to install and set up for research.