How Robots Understand 3D Space: Voxels and SDF

In 2D images, the smallest unit is a Pixel. So what is the pixel of 3D space? It is a Voxel. A portmanteau of Volume and Pixel, voxels are 1x1x1 cubes that fill 3D space.

This article explains Voxels, SDF (Signed Distance Fields), and their variants TSDF and ESDF that robots use to understand their environment.

Voxel: A Data Container for 3D Space

A Voxel is a data container with unique 3D coordinates $(x, y, z)$ in a Cartesian coordinate system. Each Voxel can store physical properties of that location:

• Color (RGB)
• Density
• Occupancy status (Occupied/Free/Unknown)
• Distance to obstacles

Point Cloud vs Voxel

Point Clouds obtained from LiDAR or depth cameras are fundamentally different from Voxels:

Comparison of Point Cloud (left) and Voxel (right) representations of the Stanford Bunny model

Aspect	Point Cloud	Voxel
Representation	Surface (Shell) - collection of points reflected from sensors	Volume - includes surface, interior, and empty space
Coordinate System	Camera-relative (varies per sensor)	Fixed spatial grid, unique coordinates
Data Persistence	New points generated each frame, requires coordinate transformation and deduplication	Fixed unique coordinates, easy data merging
Empty Space Information	None	Available (Occupancy Grid)

Key Difference: Point Clouds only tell "where objects are", while Voxels also tell "where space is empty".

Occupancy Grid Map

Storing occupancy information in Voxels creates a 3D map:

State	Meaning
Occupied	Obstacle present
Free	Empty space
Unknown	Not yet explored

Values can be stored as binary 0/1 or probability values.

Limitations of Occupancy Grids

Simple occupancy information alone cannot solve two problems:

Problem 1: Safety Distance Verification

Robots are rigid bodies with volume. They need to know how close they are to obstacles for safe movement.

• Occupancy Grid: Can only check if current position is Occupied
• Limitation: Cannot answer "how close?" - must search surrounding Voxels each time

Problem 2: Gradient Information for Path Optimization

Modern path planning algorithms require gradient information for smooth trajectory generation.

• Occupancy Grid: Discrete 0/1 values - not differentiable
• Limitation: Cannot be used for optimization-based path planning

SDF: Signed Distance Fields

SDF pre-computes and stores the distance to the nearest obstacle surface for each Voxel.

Components of SDF

• Value (distance): Euclidean distance to the nearest obstacle
• Sign:
  • Positive (+): Outside the obstacle
  • Negative (-): Inside the obstacle

Problems SDF Solves

Safety Distance Verification:

Current position SDF value = 3.5cm
-> Nearest obstacle is 3.5cm away
-> Safe if robot radius is 3cm

Safety distance can be verified with a single lookup without searching surrounding Voxels.

Gradient-Based Optimization:

Since SDF contains continuous distance values, gradients (derivatives) can be calculated. This enables:

• Knowing the direction away from obstacles
• Use in optimization-based path planning algorithms

TSDF: Truncated SDF

TSDF is an SDF that truncates distance values beyond a certain range.

TSDF Characteristics

• Example: With a truncation range of 30cm
  • Within 30cm of obstacle surface: Store accurate distance values
  • Beyond 30cm: Store as "maximum value (far)"

TSDF Advantages

Advantage	Description
Noise Removal	Ignores inaccurate data from far distances
Smooth Surfaces	Averages across multiple accumulated frames
Efficient Storage	Only stores precise data near surfaces

TSDF Use Cases

Result of reconstructing an office environment in Isaac Sim using TSDF

Primary Applications:

• 3D Surface Reconstruction
• Sensor noise filtering
• 3D environment mesh generation

ESDF: Euclidean SDF

ESDF calculates the full Euclidean distance for all Voxels. Unlike TSDF, it does not truncate far distances.

ESDF vs TSDF

Aspect	TSDF	ESDF
Distance Range	Accurate only within truncation range	Accurate across entire space
Near Surface	Precise	Precise
Far from Surface	Simplified to "maximum value"	Maintains accurate distance values
Gradient Information	Valid only near surface	Valid across entire space

Key Advantages of ESDF

Continuous Gradient Information: Since ESDF provides valid gradients across the entire space, it is suitable for optimization-based path planning.

$$\nabla \text{ESDF}(x, y, z) = \text{Direction away from obstacles}$$

Use Cases:

• Collision avoidance path planning
• Continuous gradient-based trajectory optimization
• Safety distance calculation during robot movement

TSDF vs ESDF: When to Use What?

Purpose	Recommended Method	Reason
Surface Reconstruction	TSDF	Only needs near-surface data, noise removal effect
3D Scanning/Mapping	TSDF	Efficient storage, incremental updates
Path Planning	ESDF	Needs gradients across entire space
Collision Avoidance	ESDF	Can verify safety distance anywhere
Sim-to-Real	TSDF -> ESDF	Surface reconstruction then path planning

Key Takeaways

1. Voxels are pixels of 3D space, data containers that store physical properties at each location. Unlike Point Clouds, they can also represent empty space information.

2. Occupancy Grids only store occupancy status, so they cannot answer "how close?" or provide "gradients".

3. SDF stores the distance and sign to the nearest obstacle for each Voxel. Positive means outside, negative means inside.

4. TSDF stores precise data only near surfaces, which removes noise and is suitable for surface reconstruction.

5. ESDF stores distances across the entire space, providing continuous gradient information, making it suitable for path planning and collision avoidance.

For robots to safely navigate 3D environments, they need not just simple occupancy information, but distance information and gradients. SDF-family technologies efficiently solve these requirements.