Connecting to sources…

How to build ToddlerBot

32 steps to build ToddlerBot, with the source documents.

Total cost not stated; BOM provided as a link

Software

  • Clone the robot repository

    Clone the toddlerbot repository into the ~/projects directory.

  • Configure GitHub SSH keys

    If GitHub SSH keys are not configured yet, follow the steps on the linked page.

  • Install Miniforge

    Install Miniforge if conda is not installed yet.

  • Check the system architecture

    Run uname -m and use the output to determine the system architecture.

  • Download and install the matching package

    Download the Miniforge installer matching the system architecture. Do not run the install script with sudo, and answer yes to all options.

  • Activate the conda environment

    Run source ~/.bashrc to activate the conda environment.

  • Set up the Linux environment

    Create a toddlerbot conda environment with Python 3.10, activate it, and install the linux extras.

  • Set up the macOS ARM environment

    Create the toddlerbot Python 3.10 environment with the osx-arm64 subdirectory, set that subdirectory for the environment, and install the macos extras.

  • Set up the macOS x86 environment

    Create the toddlerbot Python 3.10 environment with the osx-64 subdirectory, set that subdirectory for the environment, and install the macos extras.

  • Set up the Windows environment

    Create a toddlerbot conda environment with Python 3.10, activate it, and install the windows extras.

  • Set up the Jetson environment

    Create a toddlerbot conda environment with Python 3.10, activate it, and install the jetson extras.

  • Resolve Jetson pyaudio issues

    If pyaudio installation fails on Jetson, install portaudio19-dev, python3-dev, and build-essential, then install the jetson extras again.

  • Install the base software package

    Create a toddlerbot conda environment with Python 3.10, activate it, and install the base package.

  • Run automated depth setup script

    Run `bash scripts/setup_depth_module.sh` to install PyCUDA and PyTools, configure system TensorRT access, install cuSPARSELt libraries and Jetson PyTorch, and build and install ONNX-TensorRT.

  • Download prebuilt TensorRT engine

    For Jetson Orin NX 16GB with JetPack 6.1, download the prebuilt engine file `foundation_stereo_vits_96x128_16.engine` from the link.

  • Follow the engine naming convention

    Name the engine file using the format `foundation_stereo_vits_{height}x{width}_{valid_iters}.engine`, with height, width, and valid iteration count represented by the placeholders.

  • Place the engine file

    Place the downloaded engine file at `toddlerbot/depth/models/foundation_stereo_vits_96x128_16.engine`.

  • Start the depth estimation server

    Run run_foundation_stereo.py to start the depth estimation server.

  • Follow the setup page for installation

    Complete the installation and configuration of ToddlerBot using the linked setup page.

  • Check the examples folder for starter scripts

    The examples folder in the document provides starter scripts to run, many of which run on a real-world instance of ToddlerBot.

Calibrate and test

  • Configure skill checkpoint paths

    Specify the checkpoint path for each trained skill in POLICY_CONFIGS in run_multiple_policy.py.

  • Run the tests to verify the installation

    The tests in the tests folder can be run with pytest to verify the installation.

  • Collect stereo calibration images

    Place a 9x6 checkerboard in view and run the command to collect calibration images.

  • Perform calibration and rectification

    Run the calibration and rectification command to generate `toddlerbot/depth/params/calibration.pkl` and `toddlerbot/depth/params/rectification.npz`.

Data and training

  • Create reference motions

    Use the Keyframe App to create reference motions.

  • Train a policy for each skill

    For each skill, run train_mjx.py with <skill_name> supplied to both --gin-file and --env.

  • Collect RGB stereo frame data

    Follow the prompts to select a skill label, press space to start recording, and press space again to pause. Repeat for each skill.

  • Create depth maps

    Process the collected stereo images offline to create depth maps.

  • Train the skill classifier

    Train the classifier on depth images with skill labels by passing the data directory to train.py.

Use it

  • Run the multi-skill locomotion system

    Run the multi-skill locomotion system with the classifier checkpoint. After the server warms up, all policy checkpoints load, and the robot reaches the standing pose, depth estimates are continuously sent for skill classification and the robot autonomously performs the appropriate skills.

  • Replay a motion policy in MuJoCo

    The motion folder holds carefully crafted keyframe animations for ToddlerBot; the replay policy shows the push up motion in MuJoCo.

  • Use mjpython to view the motion on MacOS

    On MacOS, run the same replay command with mjpython to see the push up motion.

Common problems

  • Resistance modeling in motor parameter identification is unclear #16
  • Editable macOS installation fails while building PyBullet #9
  • First-time calibration endlessly repeats missing-status warnings #2
  • BOM specifies the wrong Dynamixel servo model #8

Parts

  • Jetson Orin NX 16GB
  • Two fisheye cameras, two microphones, a speaker, an IMU
  • 30 active degrees of freedom (actuator models not stated)
  • 3D prints (Onshape CAD and MakerWorld model)

Tools

  • 3D printer
  • Assembly videos and manual

Skills

  • MuJoCo/MJX reinforcement learning
  • Jetson deployment

Software

  • MuJoCo simulation and RL training scripts
  • Diffusion-policy training

Documents