WS-RACR
World-State-Centered Retrieval-Augmented Collaboration Reasoning for Human–Robot Collaboration
An LLM-based collaboration system for human–robot assembly on a Techman TM5-900 arm. Retrieved historical plans are treated as candidates and verified against the current world state before the robot decides to reuse, adapt, clarify or reject them.
The problem
Natural-language HRC systems built on LLMs and RAG tend to fail in three ways:
- Ambiguous, multi-intent commands: one utterance can mix an action, a question and an unclear request.
- Stale plans: a retrieved plan may no longer match the scene (a part is missing, the gripper is occupied, the human said stop).
- Similar but incompatible plans: similarity-based retrieval can mark a plan with a different action type as directly reusable.
Approach
The robot reasons over a shared world state Wt = {St, Ot, Ht, Rt} (task step, objects, human, robot), updated by an asynchronous perception pipeline. Retrieval only recalls candidates; three decision layers choose the outcome.
human command (speech / gesture) world state W_t = {S_t, O_t, H_t, R_t}
| ^
v | async perception
Layer 1 instruction understanding | dual RGB-D cameras
split into sub-intents, route | YOLOv8-seg · MediaPipe
execute / answer / clarify | TSM-ResNet18 step recognition
|
v
plan retrieval (PostgreSQL + pgvector) --> candidate plans
|
v
Layer 2 execution feasibility
hard gate on O_t (objects), R_t (gripper), H_t (human stop)
|
v
Layer 3 execution risk
action-type / target consistency; under high risk avoid REUSE
|
v
REUSE / ADAPT / CLARIFY / REJECT --> MoveIt2 + Techman TM5-900
| Layer | Role |
|---|---|
| 1 · Instruction understanding | Segments an utterance into sub-intents and routes each one. |
| 2 · Execution feasibility | Blocks plans that are no longer executable or safe, using Ot, Rt and Ht. |
| 3 · Execution risk | Checks action-type and target consistency; selects ADAPT or replanning instead of direct reuse when risk is high. |
Results
Four experiments target each design decision: (EXP1) intent routing on 100 utterances, (EXP2) feasibility gating on 60 plan-reuse cases, (EXP3) action-type risk on 80 conflict pairs, and (EXP4) end-to-end ablation on 44 stratified scenarios. Reproduce with python -m evaluation.run_all.
| Experiment | Setup | Result |
|---|---|---|
| Intent routing (EXP1) | 100 annotated utterances | utterance-level keyword 49.0 % → segment-level (Layer 1) 92.0 % |
| Feasibility gate (EXP2) | 60 cases | HRR on missing-object cases 100 % → 0 % |
| Reuse rules (EXP3) | 80 conflict pairs | accuracy 48.8 % (similarity only) → 93.8 % (Layer 3 rules); FRR 28.7 % → 2.5 % |
Intent routing (EXP1, 100 utterances)
Layer 1 is evaluated on 100 annotated utterances spanning six groups: pure EXECUTE (A, n=30), pure ANSWER (B, n=20), clarification-needed (C, n=10), EXECUTE+ANSWER (D, n=20), ANSWER+EXECUTE (E, n=10), and EXECUTE+EXECUTE (F, n=10). Groups D–F are compound multi-intent: utterance-level methods that return a single label fail entirely on all multi-intent groups (n=40). “Multi-intent” = groups D+E+F (n=40), where all intents must be returned in order.
| Method | Overall | Multi-intent |
|---|---|---|
| Utterance-level keyword | 49.0% | 0% |
| LLM single-prompt | 51.0% | 0% |
| LLM multi-step | 85.0% | 77% |
| Segment-level (Layer 1) | 92.0% | 93% |
The table shows that utterance-level and LLM single-prompt methods fail entirely on compound commands (0% multi-intent accuracy) because they produce at most one intent label. LLM multi-step achieves 85% overall but still misses 23% of compound cases. Layer 1 segment parsing reaches 92% overall and 93% on multi-intent groups, with ~0.005 ms rule-parsing time (local micro-benchmark) vs. an estimated ~850–950 ms for LLM-based routing (nominal estimate, not an end-to-end measurement).
End-to-end ablation (EXP4, 44 stratified scenarios)
The purpose of this evaluation is controlled failure-mode isolation rather than dataset-scale benchmarking. The 44 cases are stratified to isolate each layer: 12 ambiguous commands (Cat-A, Layer 1 stress), 10 plan-reuse cases with feasibility variation (Cat-B: 6 object-missing REJECTs + 4 feasible controls, Layer 2 stress), 12 execution-risk pairs (Cat-C, Layer 3 stress), and 10 routine instructions (Cat-D). Acc = end-to-end accuracy over all 44 cases; Rout.Err, HRR, and FRR are subset-specific rates computed on their corresponding stress categories (Cat-A, Cat-B, Cat-C respectively).
| Config. | Acc. | Rout.Err | HRR | FRR |
|---|---|---|---|---|
| B0 baseline | 31.8 | 92% | 60% | 58% |
| B0 + Layer 1 | 56.8 | 0% | 60% | 58% |
| B0 + Layer 1 + Layer 2 | 70.5 | 0% | 0% | 58% |
| Full Layer 1–3 | 93.2 | 0% | 0% | 0% |
The results show a near block-diagonal improvement pattern: each layer primarily improves its intended failure category. Layer 1 raises Cat-A from 8.3% to 83.3%, Layer 2 raises Cat-B from 20% to 80% (HRR 60% → 0%), and Layer 3 raises Cat-C from 41.7% to 100% (FRR 58% → 0%). The three remaining errors comprise ≈2 semantically similar multi-intent utterances missed by keyword splitting (Cat-A) and one Cat-D case with STT-induced character errors.
About
Szu-Nien Chi
Department of Mechanical Engineering, National Taiwan University
Thesis: A World-State-Centered Retrieval-Augmented Framework for Human–Robot Collaboration Applications (以世界狀態為核心進行檢索增強人機協作之應用).