Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- VR controllers are significantly better than a SpaceMouse for teleoperation. Even if I already had a SpaceMouse I would buy a Quest 3s and switch to it, if I was doing more than a trivial amount of teleoperation. (As is typical with robot teleop setups, I would not wear the headset on my head. It's merely a dongle to track the controllers).by modeless
- Yeah I can see that. Especially the rotation part is pretty awkward (hence why I currently do the axis parallel to table surface trick and only use the yaw angle to rotate the gripper). The space mouse was super easy to get started with though, so in terms of getting something up and running quickly it has an advantage.by mplappert
- Great article. I'll be following along. Would like to learn more about the robotics space.
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?
by wxw - Glad you like it!
Re your questions:
- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.
- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.
- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that
by mplappert - Nice, I will be following your posts! I just bought a robot arm myself, the seeed studio B601DM (€1500 6+1 axis), it works great and is open source hardware as well and a bit more solid then the so101. I also opted to not use ros, I don't want to give up control by putting another framework in between. Is your plan to see whats possible right now or do you also have ideas on how to improve sota?by thomasikzelf
- Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).
I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)
by mplappert - Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human
by sails - Very cool!
Have you seen the recent nvidia thing? They do this at scale for robotics manipulation: https://research.nvidia.com/labs/gear/enpire/
by mplappert - Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category. Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹? Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.by dracotomes
- I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.
On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.
The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.
by mplappert - This is so cool and looks so fun. I want to play with robots. I just play with software :(
I wanted to build something making bows automatically and damn, pretty complex and expensive to buy the parts.
I was like, I can probably just buy a 3D printer, print all the parts and buy some motors, but it seems it's way more complicated than that.
Like to play with a single hand robot. It looks like you need 10k+$. I wanted to spend max 1k, 3D print parts, buy motorized parts on Alibaba, and code on my Mac + spare GPUs I have access to. I'll have to save a little :(
by Frannky - I’d recommend looking at the SO-101 then. Much much cheaper.
And hardware is very fun! It’s also very frustrating. But to me worth it.
by mplappert - Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc.
Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.
by utopiah - Cool video! What made you stop if I can ask?
And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.
I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.
by mplappert - Really cool. Good move on writing your own stack. I was weighing using ros2 or building our own, but ended up going the ros2 route to “save time”. We are working with autonomous mobility robots in nature, so we figured the ecosystem around ros2 would be worth the compromise. It was not. All the time we saved setting things up initially, we are paying for now.
There should exist a minimal, clear, robotics library like what you’ve built. The Flask of the robotics world.
by delbronski - Yeah I tried to use ROS / Gazebo about 10 years ago (it was still ROS 1) and getting it set up was an immense pain. I remember that creating new modules required writing CMake files. Maybe this is better now, but I decided to skip that.
I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.
So I really like the idea. But being an OSS maintainer these days seems... intense.
by mplappert - Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...
Would like to know your reasoning on not going with LeRobot.
by avilay - I'm very interested in the SO101. I've never done any robotics and that seems a palatable entry level thing to try things out.
How have you found it?
(The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
by nl - Looks very cool! I’m not a huge discord user but how about you shoot me an email and we can figure out how to share notes? (I don’t want to post it directly here but it’s easy to find on my personal website, just google my name)
Re why not SO-101: the article has a footnote about this; I actually bought the SO-101 as well! I want to integrate it into the same setup so I can switch depending on task.
Somewhat surprisingly the xarm was actually much faster to arrive; I got it within 2 days of ordering. I don’t have a 3D printer and getting the SO-101 from the vendor I ordered it at took almost 4 weeks. So partially it just came down to what I had access to more quickly.
Second point is reliability: I think the SO-101 is cool but I’d be surprised if it doesn’t break more quickly than the xarm. I wanted something that’s going to last a long time without headaches. And these industrial arms are really mature hardware wise now.
Hope this helps!
by mplappert - - A single arm is sufficient for validating basic Pick/Place tasks, but more complex scenarios require Bi-arm
- Calibration is not required for VLA models.
- RGB or Stereo RGB inputs are sufficient for ACT, DP, and PI0/PI05.
- ROS2 is not strictly required, but it can be useful for sharing/co-developing codes. For instance, the Stanford team built a custom framework for diffusion policy instead. I also developed similar framework because ROS2 is not optimized for bi-manual manipulation or VLA workloads.
by b89kim - Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature.
Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.
by mplappert - Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!
My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.
I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.
Excited to follow your progress!
by colinator - Wow very cool project and thanks for the Ariel pointer! Added to my reading list.
And yes, having a nice robot makes life so much easier. Very happy with my choice so far!
by mplappert - Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.
>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.
Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.
Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.
My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.
by NalNezumi - Great points and I very much appreciate the input!
Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.
by mplappert