Bodies in the Block World

Abstract

Omo Space puts AI agents at desks inside Minecraft. Each desk is a terminal, and the agent in it does not move. Over three days in late October 2026 we tried giving agents bodies. We built four. Two are open-source robots simulated in MuJoCo and drawn in the game: Pollen Robotics' Microduck and RoboParty's RP1 humanoid. The third is a GLaDOS-style ceiling AI, a hanging chain of joints moved by scripted behaviour on top of physics. The fourth is OMO, a Mineflayer bot that joins the world as a player and is driven by a Hermes agent with 22 tools. The robots share one pattern: a local process owns the body, a client mod draws it, and a voice backend speaks for it. You talk to the body you are near or looking at. This paper describes the systems, lists what happened in order, and says plainly what broke.

01

Introduction

Omo Space started from one idea: a place is easier to remember than a tab. An agent gets a desk in a room, its terminal hangs on the wall, and you walk to it. The public guide states the limit: "A Hermes desk agent does not act in the world. It cannot walk, place blocks or press buttons." S10

That limit raises a question. If the world is the interface, should the agents in it have bodies? What does a body add that a wall does not?

We think of it as a ladder of embodiment:

1. Terminal. The agent is text in a window.
2. Wall. The same terminal, pinned to a place in a shared world. You find it by walking.
3. Avatar. The agent is a player. It can walk to you, follow you, carry things and fight.
4. Simulated robot. The body is a real robot design, moved by a physics engine and a trained policy, not by an animation.

Omo Space already had the first two rungs. This paper is about the last two. We did not set out to test the ladder. We built one body, it worked well enough to show, and then we built three more. The questions came after. They are in Section 7.

Two caveats. The evidence is one builder's streams, logs and repositories, plus a few friends and viewers who joined in. Nothing was compared against a control. And most of the code was written by coding agents from short prompts typed inside the game: "I just asked Opus55 bring RoboDuck into Minecraft." (T5 02:10:03) S12

02

Background

Agents in Minecraft. Minecraft is a common testbed for language-model agents. Voyager calls itself "the first LLM-powered embodied lifelong learning agent in Minecraft", built on an automatic curriculum and a skill library of code S1. MINDcraft is "an easily extensible platform built to enable LLM agents to control characters in the open-world game of Minecraft" S2. OMO's body uses Mindcraft's skill library S3 on Mineflayer, which offers a "high level JavaScript API" for Minecraft bots S4.

Physics and robots. MuJoCo is a physics engine for model-based control S5. Pollen Robotics publishes the Microduck's MJCF model and meshes S6 and pretrained ONNX policies S7. Community members have published more Microduck policies on Hugging Face. RoboParty publishes the RP1's description files S8 and deploy code with pretrained policies S9.

We did not find prior work that puts a real robot design into the same game world as the people and the desk agents, where you walk up to it and talk. We did not search hard, so treat that as weak.

03

Systems

The three robot bodies share one design, written down first for the duck. The RP1 and GLaDOS contracts say to read the duck's files and "copy freely" S13S16.

• A sidecar: one Python process that steps the simulation in real time and serves poses over HTTP on 127.0.0.1 only, "never 0.0.0.0" S13.
• A client mod for Minecraft 26.2 (Fabric). It fetches the mesh once, polls poses at about 25 Hz on a background thread, and draws each part every frame, smoothed between poses. Chat commands (!duck, !rp1, !glados) are caught on the client and never reach the server.
• An optional voice backend. A Simple Voice Chat plugin sends push-to-talk audio to it and plays the reply from the body's head, as positional sound, with captions.

Sidecars run on one thread at low priority, "so it never steals CPU from the game/OBS" S13. The builder plays and streams on the same machine.

3.1 Microduck

Body. Pollen Robotics' Microduck, about 25 cm tall, with its real CAD, servo model and policies. MuJoCo runs at 200 Hz physics and 50 Hz policy, with the BAM XL330 servo model and 2° gear backlash S11. Each step builds the 61-number observation the policies were trained on S12a. Pollen's velstand policy walks, turns, follows and gets up; alpha_sitstand sits and stands S14. A duck still down after 3 s is reset upright S12a.

Blocks as terrain. Every 2 s the mod scans the blocks near the duck and sends their collision boxes to the sim. Walls stop it, carpets are walkable, ledges drop it. At the default scale a block is 25 cm, the duck's own height, so it cannot climb stairs. The README calls that "physically correct" S11.

A flock. Up to 4 ducks by default, 8 at most, each with a persona, name tag, eye colour and quack pitch: Quackers, Rusty, Zippy, Plato and Newton S11. Each duck has its own MuJoCo model, and the others appear in it as trunk-sized capsules S12a.

More brains. Twelve community policies are offered: sprint, roller skates, forward roll, cartwheel, hop, dance and others. Each was graded in the sim over 3 seeds first. Some were rejected, such as sprinters that "ignore the speed command, so nothing can stop them at the leash" S12a.

Cost. About 7% of one core for one duck and 22% for four, at a steady 50.0 Hz with 0 overruns, and about 470 MB of RAM. The model has 25,536 triangles S11.

Voice. Hold push-to-talk within 16 blocks. The backend picks the listener in this order: a duck named in your words, then the duck you just spoke to (for 20 s), then the duck under your crosshair, then the nearest, then the default S12b. Each duck has its own voice call, at most 2 open at once. Spoken moves are parsed locally into sidecar commands, with no agent round trip; other requests can go to a Hermes agent S20. The public release ships the voice plugin and protocol, not the backend. Stream viewers can walk the duck but cannot reset or remove it S11.

3.2 RP1 humanoid

Body. RoboParty's RP1, 24-joint variant, with RoboParty's own policies: locomotion, a wave learned from a motion clip, and a dance S15. MuJoCo steps at 0.002 s with a 50 Hz policy. Joint PD control uses the gains from RoboParty's training setup. The hardware gains were tried first; with them the robot stood and turned but did "not walk forward (0.0 m/s at cmd 0.4)" S15.

Measured. Walking 0.395 m/s with 4 cm of sideways drift over 10 s. Turns of +86° and −85° in 2.6 s. Two 10-minute runs of random commands: 0 falls, 0 resets. CPU 8.1% of one core standing, 15.5% dancing S15.

In the world. One sim metre is one block, so RP1 stands about 1.5 blocks tall S17. Unlike the duck, "the robot can't see Minecraft blocks" S15. There is no get-up policy, so every fall ends in a reset.

Voice. A persona called Roboto, with his own voice. While he talks, a small waist sway and forearm movement are added to the standing policy, and only while he stands S15S20.

3.3 GLaDOS-style ceiling AI

Body. A large AI hanging from the lab ceiling, about 7.5 blocks tall, in the style of the Portal games. The first model was original procedural geometry with "no Valve assets". Two credited fan-made models were added later; the model switches live in about 0.1 s S16.

Physics without a policy. No trained policy exists for a character like this, so "the 'brain' is a scripted behaviour layer on top of real physics" S16. She is a chain of 13 hinge joints under gravity at 60 Hz. Gravity compensation on the arm is cut to 50%, so it "still hangs like a pendulum and sways from the reaction torques of the head". Inverse kinematics keeps her eye on the player. Behaviours include track, angry, scan, bow, sleep, dance and an 18.6 s "spooky" routine that ends with her eye about 1.5 m from the player's. Tracking error was 0.8–4.7° over 8 targets; the service used 5.6% of one core S16.

Voice. Her persona is written in our own words: a dry lab supervisor who calls the player "test subject" S16. One version re-voices her lines through a local voice-cloning speech server S20.

One listener at a time. Three companions in one room means three possible listeners. Each claims the mic when you are within range and looking at it, or closer to it than to the duck. The best fresh claim wins, so "exactly one companion hears each push-to-talk" S17.

3.4 OMO, the agent player

Body. Not a simulated robot. OMO is a Mineflayer bot that logs into your single-player world as a player. He walks, swims, takes knockback, gets hungry, fights and sleeps S18. Omo Space v1.2.0 opens the world to LAN on port 25566 for agents on the same computer; offline logins from other machines are refused. Agents get the Omo skin and a ChatGPT - <name> tag S10bS18.

Brain and tools. You talk to a voice model in a browser page. Requests to do something go to OMO's own Hermes agent, which plans and calls the body's 22 tools over MCP. Long jobs can be interrupted, paused for a fight and resumed. "Come here", "follow me", "stop", "fight" and "chop some wood" skip the language model entirely S18S19.

What this rung adds. The robots move and talk. OMO changes the world. On stream he went from wood to planks, sticks, a crafting table and a pickaxe, then stone and a stone sword, "including retrying a craft that failed the first time" S18.

04

What we did

Times are Australian Eastern Daylight Time, from commits, file times and CHANGELOGs. Stream tags (T4 = 27 Oct, T5–T7 = 28 Oct, T8 = 29 Oct) are offsets into each recording, not clock times S21.

27 October. - Around midday, four commits to the Omo Space mod opened single-player worlds to local agents over LAN, let a plain bot client join a Fabric world, and added the agent skin and name tag S22. - On stream, OMO appeared carrying a sword and following the builder (T4 00:12:59). Asked to use Hermes to find a bot's-eye camera plugin, he came back with "prismarine viewer" (T4 02:39:45) S21. - A viewer had suggested putting the Microduck in Minecraft: "you were the one that gave me the idea" (T4 02:17:24). One prompt to a desk agent built an underground robotics lab, shown about 30 minutes later (T4 02:52:47). The duck was reported built and installed at T4 03:27:36 S21.

28 October, night to morning. - The duck walked on command in the game (T5 00:11:45). The RP1 was spawned and danced (T5 00:55:46): "Now it's truly online." (T5 00:56:30) S21 - First push-to-talk conversation with the duck (T5 01:35:42). "Microduck? Do a jump." worked (T5 01:44:29) S21. - The GLaDOS project began from the same pattern (T5 01:55:23) S21. Her sidecar got a redesign, then switchable fan-made models (live from about 06:31), then the spooky routine S16S23. - Multi-duck support began around 03:30 (a backup folder marks it) S12a. Soon after: "Yo ducks ducks race to my position" (T5 03:02:26) S21.

28 October, day. - 12:28: Omo Microduck v1.0.0, the first public release S24. 14:26: Omo, brought to life v0.1.0 S25. - On stream, six ducks stood up and walked toward the builder (T6 06:51:55). Ducks answered spoken commands in character ("Quack. We're doing it.", T7 03:20:52), and agents cut stream clips of them into short films S21.

29 October. The builder spoke to the flock in passing: "can two of the ducks sit down and the rest stand up" (T8 01:47:37) S21.

30 October. The RP1 and GLaDOS code went into local version control, "not published" S26.

05

Observations

These are things we saw, not things we tested.

A real robot design reads differently from a mob. The duck falls because a policy lost its balance against simulated gravity, not because an animation played. Its stair falls became a public reel: "It can't climb stairs, so down it goes" S11. The failure was the content.

Proximity and gaze worked as addressing. Push-to-talk "within our 16 blocks" (T5 01:20:59), with the crosshair or a name choosing the listener, was enough to talk to one duck in a flock. "Each microduck will have its own personality." (T5 01:46:32) S21. The question from the same stream is still open: "How does it know where I'm talking to?" (T5 03:03:09) S21a.

Other people joined in. Friends on a call directed the robots ("Scale like five or six.", T5 02:19:50). One asked whether the robots could do real work, "like hold something", or were "just regular robots" (T6 07:52:13) S21. That question marks the gap between the robot rung and the avatar rung.

The game became the lab. The robots got a test room, an obstacle course and a separate dev room. One line on a call was "This is the simulator." (T5 02:15:53); the speaker is uncertain S21.

Only the avatar does work. OMO answered two overlapping spoken requests in one reply, a web lookup and his inventory ("four pieces of rotten flesh") S18. The robots never change the world. OMO does.

06

What broke, and limitations

Speed of building, and its cost. Each new body reused the duck's design and took hours, not days S15S17. We cannot tell how much of that was the pattern and how much was the coding agents.

Physics meets blocks. The duck fell on stairs again and again ("The micro doc is just forming on our Minecraft stairs.", T5 01:34:03) and got stuck (T8 03:32:15). Early on it walked through walls (T5 00:11:58), before terrain was added. The RP1 still cannot see blocks and walked through walls in a demo (T6 07:57:55) S15S21.

Sim gaps. The duck's walking policy reaches about ⅓ of its commanded speed in our CPU sim, so commands are scaled and speeds are "nominal" S11. The RP1 reaches about 0.8× S15. Ducks feel each other only as capsules S12a.

The humanoid would not follow. "rp1 follow me and it's not working but maybe the ducks" (T6 07:56:20) S21. The cause is [unverified].

Voice. The cloned GLaDOS voice ran slower than real time: "I'm rating 9.2 seconds of speech. / Took 15 seconds." (T6 04:45:58), and later "So, she is broken." (T6 05:17:38) S21. OMO's voice runs in a browser tab, not in-game push-to-talk, over an undocumented route that "can change or break without notice" S18.

The agent player is fragile. He would not stay put ("Stay sit sit puppy sit right not working", T4 00:13:26). He talked over the builder (T4 02:49:11). He kept dying to zombies, and the builder worried it was "using up my usage" (T4 03:04:49). One Hermes round trip never came back (T4 03:35:07) S21. A request for 8 birch logs ended with 1 S18.

The machine. The 28 October stream ends in a full PC freeze and restart S21.

Opacity. Asked how the duck was connected, the builder said "I don't even know bro." (T5 02:10:02) S21. Building by prompt is fast. It can also leave the builder unable to debug live.

Rights. The Microduck mod and the agent player are public under MIT, with upstream credits S11S18. The RP1 work is "local personal use only" S15. The GLaDOS work uses a character owned by Valve, fan-made models credited for personal non-commercial use, and a cloned character voice S16S20. It is not published, and should not be.

Evidence. One builder, a few friends and viewers. Quotes come from automatic transcripts, checked against the transcript text but not the audio S21.

07

Next steps and testable questions

1. Addressing. With 3 or more companions in a room, how often does a spoken request reach the intended listener? Compare routing by name, gaze and nearness. The routing is already logged S20.
2. Trust. Does a body that can really fall make people more or less willing to hand an agent a real task than a scripted one does?
3. Robots that work. Connect a simulated robot to a Hermes agent with real tools, as OMO is connected. Can it "hold something"?
4. Terrain for the humanoid. Port the duck's block scan to the RP1. Measure falls per minute on a fixed obstacle course, before and after.
5. Getting up. Add a get-up policy for the RP1 and count resets per 10 minutes.
6. Long chains. Run OMO's stone-sword chain 20 times from a fresh world. How often does it finish, and why does it fail?
7. Cost. Measure game frame time with 0, 1, 4 and 8 bodies loaded, on the streaming machine, while streaming.

08

Sources

[S1] Wang et al., "Voyager: An Open-Ended Embodied Agent with Large Language Models", arXiv:2305.16291. arxiv.org/abs/2305.16291. "the first LLM-powered embodied lifelong learning agent in Minecraft"

[S2] White et al., "Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning", arXiv:2504.17950. arxiv.org/abs/2504.17950. "we introduce MINDcraft, an easily extensible platform built to enable LLM agents to control characters in the open-world game of Minecraft"

[S3] Mindcraft repository. github.com/mindcraft-bots/mindcraft. Credited in S18: "Omo's body skills are Mindcraft's library (MIT), running on Mineflayer"

[S4] Mineflayer. github.com/PrismarineJS/mineflayer. Repository description: "Create Minecraft bots with a powerful, stable, and high level JavaScript API."

[S5] Todorov, Erez, Tassa, "MuJoCo: A physics engine for model-based control", IROS 2012, doi:10.1109/IROS.2012.6386109. Repository: github.com/google-deepmind/mujoco. Citation block in the repo README: "MuJoCo: A physics engine for model-based control"

[S6] Pollen Robotics, microduck_rl. github.com/pollen-robotics/microduck_rl. Cited in S12a: "MJCF + meshes from pollen-robotics/microduck_rl, Apache-2.0"

[S7] Pollen Robotics, microduck-policies. huggingface.co/pollen-robotics/microduck-policies. Cited in S12a: "the pretrained ONNX policies from pollen-robotics/microduck-policies"

[S8] RoboParty, RP1_description. github.com/Roboparty/RP1_description. Cited in S15: "MJCF + meshes from Roboparty/RP1_description"

[S9] RoboParty, roboparty_inference. github.com/Roboparty/roboparty_inference. Cited in S15: "RoboParty's own pretrained ONNX policies from Roboparty/roboparty_inference"

[S10] Omo Space, docs/AGENTS-AND-TERMINALS.md, section 9 "What is not built yet" (public). github.com/omoresearch/omo-space/blob/main/docs/AGENTS-AND-TERMINALS.md. "A Hermes desk agent does not act in the world. It cannot walk, place blocks or press buttons."

[S10b] Omo Space, docs/AGENTS-AND-TERMINALS.md, section 8 (public). github.com/omoresearch/omo-space/blob/main/docs/AGENTS-AND-TERMINALS.md. "the mod opens it to LAN on port 25566 ... Those players may join from this computer only"

[S11] Omo Microduck README. github.com/omoresearch/omo-microduck. "MuJoCo at 200 Hz physics / 50 Hz policy, on one CPU core"; "about 7% of one core for one duck and 22% for four"; "It cannot climb stairs or slabs, and that is physically correct."

[S12] Stream analysis T5 (see S21). "I just asked Opus55 bring RoboDuck into Minecraft." (T5 02:10:03)

[S12a] microduck-sim README (repo, local; published as sidecar/ in S11). "61-D obs = imu gyro (3) + projected gravity ... + last action (14) + 13-D command"; "sprint-fixed/hill-fixed run 1.7/1.4 m/s but ignore the speed command, so nothing can stop them at the leash"

[S12b] microduck-sim CONTRACT.md, "Voice (proto 2)" (repo, local). "a duck's name in the utterance > the current duck (sticky 20 s) > looked-at duck > nearest within 16 blocks > default"

[S13] microduck-sim CONTRACT.md (repo, local). "127.0.0.1:8796 only (never 0.0.0.0)"; "so it never steals CPU from the game/OBS"

[S14] Omo Microduck CHANGELOG, v1.0.0. github.com/omoresearch/omo-microduck/blob/main/CHANGELOG.md. "velstand handles walk, turn, follow and getting up; alpha_sitstand handles sit and stand."

[S15] rp1-sim README and CONTRACT.md (repo, local, not published). "walk (nominal 0.4 → policy 0.5): 0.395 m/s forward"; "the robot stands and turns but does not walk forward (0.0 m/s at cmd 0.4)"; "Only a flat floor; the robot can't see Minecraft blocks."; "Local personal use only"

[S16] glados-sim README and CONTRACT.md (repo, local, not published). "the 'brain' is a scripted behaviour layer on top of real physics"; "so the arm still hangs like a pendulum"; "track aim error 0.8–4.7° over 8 targets"

[S17] rp1-mod README (repo, local). "The best fresh claim among GLaDOS and the RP1 owns omo.voice.focus, so exactly one companion hears each push-to-talk." Also: "default 1: RP1 is ~1.5 blocks tall"

[S18] Omo, brought to life README. github.com/omoresearch/omo-brought-to-life. "These are the 22 tools his body serves to Hermes"; "On stream he ran that whole chain himself, including retrying a craft that failed the first time."; "a request for 8 birch logs finished with 1 collected"

[S19] Omo, brought to life, docs/how-it-works.md. github.com/omoresearch/omo-brought-to-life/blob/main/docs/how-it-works.md. "quickIntent in body.mjs matches short, plain requests ... and runs them on the body directly, before any language model."

[S20] Voice backend README and fish-voice.mjs header (local, not published). "One GPT-Live call per duck ... at most 2 live at once"; "her spoken words are re-voiced by the local fish-speech server"

[S21] Stream analysis, stream-analysis/T4–T8 (science lab, local; companion paper "Rooms, Not Windows"). Quotes are whisper-small transcripts, re-checked against the transcript text: "791/791 verified". Example: "[02:15:53] 'This is the simulator.'" (speaker uncertain)

[S21a] SYNTHESIS.md, counter-evidence (same source as S21). "Addressing in voice: 'How does it know where I'm talking to?' (T5 03:03:09)"

[S22] Omo Space git log, 27 Oct 2026 (repo: github.com/omoresearch/omo-space). "feat(mod): open single-player worlds to LAN for local AI agents (offline join, loopback-only)"

[S23] glados-sim PROGRESS-v3.md (repo, local). "restart glados-sim.service 06:31 (blendswap active ...)"

[S24] Omo Microduck git log and release v1.0.0. github.com/omoresearch/omo-microduck/releases/tag/v1.0.0. "2026-10-28 12:28:44 +1100 feat: Omo Microduck v1.0.0"

[S25] Omo, brought to life git log and release v0.1.0. github.com/omoresearch/omo-brought-to-life/releases/tag/v0.1.0. "2026-10-28 14:26:36 +1100 feat: Omo v0.1.0"

[S26] rp1-sim / glados-sim git log (repo, local). "2026-10-30 00:43:24 +1100 Initial import: rp1-sim as it runs today (local repo, not published)"