KPI
A Promptable Kernel for
Physical Interaction on Humanoids
Abstract
Humanoids now walk, balance and reach with remarkable generality. But a trajectory alone carries limited information about the interaction it should produce: at contact, the executing controller determines how the robot behaves. KPI is a promptable kernel between the trajectory source and an unmodified whole-body tracker. The commander supplies a nominal reference together with an interaction contract. From tracking error and an estimated wrench, KPI adapts stiffness, damping, reference and feedforward toward the contract at contact rate.
We demonstrate KPI through an agentic framework: from one instruction, a vision-language agent writes both the motion and the contract, without task-specific code. A humanoid operates a winch to hoist a second robot, opens a door and passes through, and transports a box. Additional evaluations examine load robustness, scripted board writing and teleoperated interaction during whole-body motion.
KPI: Contract-Driven Physical Interaction
The commander states the interaction objective. The kernel adapts the controller during contact.
The interaction contract
For each direction or subspace that requires a distinct response, the commander supplies four fields. This separates the physical objective from the controller parameters used to realize it.
- 01 Direction d
- A trajectory tangent, its normal subspace, a plane tangent or normal, or a direction in a declared frame.
- 02 Requirement o
- Tracking penalizes motion error; compliant favors low directional stiffness; constrained imposes force bounds without an added tracking or stiffness objective.
- 03 Force range [fmin, fmax]
- Lower and upper bounds apply to any requirement. Equal bounds specify a target force.
- 04 Channels c
- Select which parameters may change jointly: gain, reference, and feedforward.
Online adaptation at 100 Hz
Reference, pose, velocity and estimated wrench
Stiffness · damping · reference · feedforward
The next update uses the resulting contact state
The optimization balances tracking or compliance objectives with force bounds and a penalty on parameter changes. Only the selected channels can change. A force range can be unattainable at the current deflection; adaptation remains bounded by the controller’s limits.
Realization on a Humanoid Robot
On the Unitree G1, Cartesian impedance realizes the arm response at 500 Hz, using a sensorless wrench estimate refreshed at 200 Hz. Joint-local damping runs in the motor drivers; cross-joint terms remain on the host. SONIC controls the legs and waist, while KPI controls the arms. No wrist force sensor is added and the whole-body tracker is not retrained.
Agentic Execution
Task-level reasoning writes the reference and contract; physical feedback adapts their realization during each stage.
GPT-6 Astra implements Analyzer, Detector and Verifier. Each zero-shot agentic trial starts with one instruction, without task-specific code or a manually supplied contract. The system composes shared motion generators and uses a pretrained whole-body tracker.
Motion cores and the contract vocabulary
Experiments
Does the interface improve zero-shot execution, what does online adaptation contribute, and how does it extend across interactions?
All conditions share SONIC’s whole-body tracker. We compare its native arm control, MCC*, KPI with online adaptation, and KPI-fixed in adaptation studies. The arm interfaces differ: MCC* requests stiffnesses, whereas KPI requests interaction contracts; SONIC ignores the generated contract.
Winch Operation
Track along the handle’s tangent while limiting force in constrained directions.
Agentic execution. The agent selects circular motion and plans four clockwise revolutions to hoist another G1. Tangential tracking drives the turn, force bounds limit loading across the path, and the wrist remains compliant.
Tracking and compliant-control baselines. SONIC and MCC* grasp the handle but do not sustain a complete revolution in these selected trials. Excessive contact forces can move the body or break the grasp.
The contribution of online adaptation
KPI and KPI-fixed. When resistance slows the hand, KPI slows reference advancement and increases tangential stiffness. KPI-fixed lets the reference advance after the hand stalls; the tracking force becomes misaligned with the path until the hand reverses.
Door Opening and Passage
Use compliance to tolerate partially observed geometry and a hand reference that conflicts with the hinge constraint.
Agentic execution. Rotational tracking releases the latch. The agent then assigns compliance while combining a backward step with a hand reach. The robot opens the door further and passes through the doorway.
Interaction with a stiff mechanism. SONIC’s joint PD controller fails to unlatch in the shown trial. MCC* can unlatch, but oscillatory interaction makes pulling unreliable; it completes one of five passages in the reported evaluation.
Box Transport and Load Robustness
Maintain multipoint contact and inward forces while accommodating changes in relative hand–box pose.
Agentic transport and placement. The robot picks up a 0.85 kg box, turns, squats and places it on the destination table. The contract sustains the opposing contacts through posture changes. The full transport sequence uses the light box to limit heating.
Why contact maintenance matters. SONIC and MCC* fail to lift the box in the reported trials. The shown attempts illustrate loss of stable multipoint contact between the hands and the box sidewalls.
Force ranges under changing loads
Stationary lift-and-hold tests. Load mass and placement change the required response. KPI lifts 3 kg at near and middle positions, while both methods fail at 3 kg far. These tests are separate from the 0.85 kg transport sequence.
For medium and heavy loads, the agent chooses force ranges of 25–40 N and 35–50 N for KPI. For KPI-fixed, it chooses stiffnesses of 500 and 600 N/m; excessive forces trigger motor errors in failed trials. This task illustrates the paper’s motivation: an interaction objective remains meaningful as deflection changes, while a fixed gain produces a changing force.
Board Writing
A surface-normal force target supports contact while the pen follows a tangential path.
Scripted surface interaction. KPI and KPI-fixed both produce recognizable N and B shapes. KPI brings the mean estimated contact force closer to the target, with little visible change in the written pattern.
Additional surface-interaction demonstration: wiping
Scripted wiping. An additional surface-contact demonstration from the source footage.
Additional Tasks and Whole-Body Motions
The same interface also accepts human motion references. These demonstrations are teleoperated.
Drawer opening. KPI supports contact with an articulated mechanism during teleoperation. The supplementary video compares it with the native SONIC arm-control path.
Carrying during whole-body motion. Bimanual contact is maintained through locomotion, kneeling and standing.