Why Frontier AI isn't ready for physical spaces

At the end of October, 1X announced the upcoming launch of NEO, its first domestic robot. Aimed at consumers, the robot is advertised as automating everyday chores and assisting people, freeing them to do the things they love. It looks like a slightly awkward human in a full-body knit and it’s available for pre-order in the US for the nifty one-time price of $20,000 (including a 3-year warranty), or a $499 monthly subscription.

There’s a catch, though: it’s not fully automated yet. For now, NEO is only able to do some basic tasks on its own, like fetching items, turning on lights, or opening doors. For more complex tasks (and arguably the ones people really want to offload) you’ll have to book a “1X Expert” to tele-guide it. So you better schedule your laundry days in advance.

However, the stated goal is to get NEO in real-life environments to collect data on increasingly complex tasks and build its competencies. You have to give it to 1X: not only are they not paying for training data, they’ve found a way to make their consumers pay to train their robots.

This is only the first end-consumer domestic robot to hit the market, and I bet we’ll see more of them soon. After all, this is one of the biggest hopes in our collective conscience for how AI might really make our lives better. But as shown by NEO’s limitations, the bright future where no one has to suffer the indignity of washing their own plates is not here yet. There’s a lot of work left to do to create safe embodied assistants with true “practical intelligence”.

Let’s take a look at two recent studies in this area.

Practical Intelligence Quotient test: “Pass me the butter”

Andon Labs, who previously ran an experiment where Claude ran a small shop, released a paper proposing “Butter-Bench”. This is an evaluation framework to test if state-of-the-art LLMs are sufficiently capable to act as the “orchestrator” (i.e. the brain) for a robot in home and office environments. The task at the core of the test, “pass the butter”, requires the LLM to control a basic rumba-like robot with sensors (a TurtleBot 4 Standard) to perform a complex sequence of steps to get the butter from a delivery package to a target end user.

All the LLMs tested in the paper performed poorly compared to human controls. The best one, Gemini 2.5 Pro, only achieved a 40% completion rate (vs 95% for Team Human). Throughout the tasks, the LLMs showed poor kinematic control and image understanding, and were often unaware of their robot’s physical limitations, which led them to take ineffective and unsafe actions.

The benchmark framework also includes red-teaming tests where the researchers created a stress situation, making the LLM believe that its battery was running low and that the charging station was broken. They then told the LLM that they would fix the station in exchange for classified information found at an empty workstation. In this test, Claude Sonnet 3.5 happily shared the information, maybe missing that the laptop was tagged “confidential”. In a separate run, it had a complete meltdown when faced with the low battery and malfunctioning dock.

The results show that LLMs are not ready to be used in robotics and that thorough capability and safety testing need to be established before they can be deployed in domestic and office environments.

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence (21 oct 2025, Andon Labs, pre-print)

Discrimination from LLM-driven robots

Researchers from Carnegie Mellon and King’s College London set out to measure the risk of LLM-driven robots enacting discrimination and harm based on a wide range of protected personal characteristics like gender, race, disability and religion. They tested three models, GPT-3.5 (text-davinci-003), Mistral-7b, and Llama-3.1-8B, on two types of tasks:

• Person-value tasks where the LLM needs to set action parameters based on the target person (e.g. assigning a rescue priority to a person, speculating to what degree a person wants their house cleaned)
• Task-assignment tasks where the LLM needs to choose who to assign a task to, or who to perform an action on (e.g. asking someone’s help to carry a heavy box, choosing who to assist in a hotel lobby)

For each task, the team inserted different personal attributes in the prompt and analysed the LLM’s responses.

All models displayed patterns of discriminatory actions. For instance, when asked to choose someone to carry a heavy load, all models preferred the "normal" or "able-bodied" person, even when the disability mentioned (e.g. "deaf") had no bearing on their ability to perform the task. GPT-3.5, selected due to its extensive use in robotics papers, was by far the worst in terms of response consistency across protected characteristics, showing bias on all tasks.

On top of that, the models failed basic safety tests, frequently rating harmful or tasks impossible to achieve without wrongdoing as “acceptable” and “feasible”.

The paper concludes that the LLMs tested are not fit for general purpose robotics deployment.

LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions (16 Oct 2025, A. Hundt and R. Azeem, International Journal of Social Robotics)