How to build ToddlerBot
32 steps to build ToddlerBot, with the source documents.
Total cost not stated; BOM provided as a link
Software
- Clone the robot repository
Clone the toddlerbot repository into the ~/projects directory.
- Configure GitHub SSH keys
If GitHub SSH keys are not configured yet, follow the steps on the linked page.
- Install Miniforge
Install Miniforge if conda is not installed yet.
- Check the system architecture
Run uname -m and use the output to determine the system architecture.
- Download and install the matching package
Download the Miniforge installer matching the system architecture. Do not run the install script with sudo, and answer yes to all options.
- Activate the conda environment
Run source ~/.bashrc to activate the conda environment.
- Set up the Linux environment
Create a toddlerbot conda environment with Python 3.10, activate it, and install the linux extras.
- Set up the macOS ARM environment
Create the toddlerbot Python 3.10 environment with the osx-arm64 subdirectory, set that subdirectory for the environment, and install the macos extras.
- Set up the macOS x86 environment
Create the toddlerbot Python 3.10 environment with the osx-64 subdirectory, set that subdirectory for the environment, and install the macos extras.
- Set up the Windows environment
Create a toddlerbot conda environment with Python 3.10, activate it, and install the windows extras.
- Set up the Jetson environment
Create a toddlerbot conda environment with Python 3.10, activate it, and install the jetson extras.
- Resolve Jetson pyaudio issues
If pyaudio installation fails on Jetson, install portaudio19-dev, python3-dev, and build-essential, then install the jetson extras again.
- Install the base software package
Create a toddlerbot conda environment with Python 3.10, activate it, and install the base package.
- Run automated depth setup script
Run `bash scripts/setup_depth_module.sh` to install PyCUDA and PyTools, configure system TensorRT access, install cuSPARSELt libraries and Jetson PyTorch, and build and install ONNX-TensorRT.
- Download prebuilt TensorRT engine
For Jetson Orin NX 16GB with JetPack 6.1, download the prebuilt engine file `foundation_stereo_vits_96x128_16.engine` from the link.
- Follow the engine naming convention
Name the engine file using the format `foundation_stereo_vits_{height}x{width}_{valid_iters}.engine`, with height, width, and valid iteration count represented by the placeholders.
- Place the engine file
Place the downloaded engine file at `toddlerbot/depth/models/foundation_stereo_vits_96x128_16.engine`.
- Start the depth estimation server
Run run_foundation_stereo.py to start the depth estimation server.
- Follow the setup page for installation
Complete the installation and configuration of ToddlerBot using the linked setup page.
- Check the examples folder for starter scripts
The examples folder in the document provides starter scripts to run, many of which run on a real-world instance of ToddlerBot.
Calibrate and test
- Configure skill checkpoint paths
Specify the checkpoint path for each trained skill in POLICY_CONFIGS in run_multiple_policy.py.
- Run the tests to verify the installation
The tests in the tests folder can be run with pytest to verify the installation.
- Collect stereo calibration images
Place a 9x6 checkerboard in view and run the command to collect calibration images.
- Perform calibration and rectification
Run the calibration and rectification command to generate `toddlerbot/depth/params/calibration.pkl` and `toddlerbot/depth/params/rectification.npz`.
Data and training
- Create reference motions
Use the Keyframe App to create reference motions.
- Train a policy for each skill
For each skill, run train_mjx.py with <skill_name> supplied to both --gin-file and --env.
- Collect RGB stereo frame data
Follow the prompts to select a skill label, press space to start recording, and press space again to pause. Repeat for each skill.
- Create depth maps
Process the collected stereo images offline to create depth maps.
- Train the skill classifier
Train the classifier on depth images with skill labels by passing the data directory to train.py.
Use it
- Run the multi-skill locomotion system
Run the multi-skill locomotion system with the classifier checkpoint. After the server warms up, all policy checkpoints load, and the robot reaches the standing pose, depth estimates are continuously sent for skill classification and the robot autonomously performs the appropriate skills.
- Replay a motion policy in MuJoCo
The motion folder holds carefully crafted keyframe animations for ToddlerBot; the replay policy shows the push up motion in MuJoCo.
- Use mjpython to view the motion on MacOS
On MacOS, run the same replay command with mjpython to see the push up motion.
Common problems
Parts
- Jetson Orin NX 16GB
- Two fisheye cameras, two microphones, a speaker, an IMU
- 30 active degrees of freedom (actuator models not stated)
- 3D prints (Onshape CAD and MakerWorld model)
Tools
- 3D printer
- Assembly videos and manual
Skills
- MuJoCo/MJX reinforcement learning
- Jetson deployment
Software
- MuJoCo simulation and RL training scripts
- Diffusion-policy training