WS-RACR

World-State-Centered Retrieval-Augmented Collaboration Reasoning for Human–Robot Collaboration

An LLM-based collaboration system for human–robot assembly on a Techman TM5-900 arm. Retrieved historical plans are treated as candidates and verified against the current world state before the robot decides to reuse, adapt, clarify or reject them.

The problem

Natural-language HRC systems built on LLMs and RAG tend to fail in three ways:

Approach

The robot reasons over a shared world state Wt = {St, Ot, Ht, Rt} (task step, objects, human, robot), updated by an asynchronous perception pipeline. Retrieval only recalls candidates; three decision layers choose the outcome.

 human command (speech / gesture)         world state W_t = {S_t, O_t, H_t, R_t}
        |                                        ^
        v                                        | async perception
 Layer 1  instruction understanding              |  dual RGB-D cameras
          split into sub-intents, route          |  YOLOv8-seg · MediaPipe
          execute / answer / clarify             |  TSM-ResNet18 step recognition
        |
        v
 plan retrieval (PostgreSQL + pgvector)  -->  candidate plans
        |
        v
 Layer 2  execution feasibility
          hard gate on O_t (objects), R_t (gripper), H_t (human stop)
        |
        v
 Layer 3  execution risk
          action-type / target consistency; under high risk avoid REUSE
        |
        v
 REUSE / ADAPT / CLARIFY / REJECT  -->  MoveIt2 + Techman TM5-900
LayerRole
1 · Instruction understandingSegments an utterance into sub-intents and routes each one.
2 · Execution feasibilityBlocks plans that are no longer executable or safe, using Ot, Rt and Ht.
3 · Execution riskChecks action-type and target consistency; selects ADAPT or replanning instead of direct reuse when risk is high.

Results

Four experiments target each design decision: (EXP1) intent routing on 100 utterances, (EXP2) feasibility gating on 60 plan-reuse cases, (EXP3) action-type risk on 80 conflict pairs, and (EXP4) end-to-end ablation on 44 stratified scenarios. Reproduce with python -m evaluation.run_all.

ExperimentSetupResult
Intent routing (EXP1)100 annotated utterances utterance-level keyword 49.0 % → segment-level (Layer 1) 92.0 %
Feasibility gate (EXP2)60 cases HRR on missing-object cases 100 % → 0 %
Reuse rules (EXP3)80 conflict pairs accuracy 48.8 % (similarity only) → 93.8 % (Layer 3 rules); FRR 28.7 % → 2.5 %

Intent routing (EXP1, 100 utterances)

Layer 1 is evaluated on 100 annotated utterances spanning six groups: pure EXECUTE (A, n=30), pure ANSWER (B, n=20), clarification-needed (C, n=10), EXECUTE+ANSWER (D, n=20), ANSWER+EXECUTE (E, n=10), and EXECUTE+EXECUTE (F, n=10). Groups D–F are compound multi-intent: utterance-level methods that return a single label fail entirely on all multi-intent groups (n=40). “Multi-intent” = groups D+E+F (n=40), where all intents must be returned in order.

MethodOverallMulti-intent
Utterance-level keyword49.0%0%
LLM single-prompt51.0%0%
LLM multi-step85.0%77%
Segment-level (Layer 1)92.0%93%

The table shows that utterance-level and LLM single-prompt methods fail entirely on compound commands (0% multi-intent accuracy) because they produce at most one intent label. LLM multi-step achieves 85% overall but still misses 23% of compound cases. Layer 1 segment parsing reaches 92% overall and 93% on multi-intent groups, with ~0.005 ms rule-parsing time (local micro-benchmark) vs. an estimated ~850–950 ms for LLM-based routing (nominal estimate, not an end-to-end measurement).

End-to-end ablation (EXP4, 44 stratified scenarios)

The purpose of this evaluation is controlled failure-mode isolation rather than dataset-scale benchmarking. The 44 cases are stratified to isolate each layer: 12 ambiguous commands (Cat-A, Layer 1 stress), 10 plan-reuse cases with feasibility variation (Cat-B: 6 object-missing REJECTs + 4 feasible controls, Layer 2 stress), 12 execution-risk pairs (Cat-C, Layer 3 stress), and 10 routine instructions (Cat-D). Acc = end-to-end accuracy over all 44 cases; Rout.Err, HRR, and FRR are subset-specific rates computed on their corresponding stress categories (Cat-A, Cat-B, Cat-C respectively).

Config.Acc.Rout.ErrHRRFRR
B0 baseline31.892%60%58%
B0 + Layer 156.80%60%58%
B0 + Layer 1 + Layer 270.50%0%58%
Full Layer 1–393.20%0%0%

The results show a near block-diagonal improvement pattern: each layer primarily improves its intended failure category. Layer 1 raises Cat-A from 8.3% to 83.3%, Layer 2 raises Cat-B from 20% to 80% (HRR 60% → 0%), and Layer 3 raises Cat-C from 41.7% to 100% (FRR 58% → 0%). The three remaining errors comprise ≈2 semantically similar multi-intent utterances missed by keyword splitting (Cat-A) and one Cat-D case with STT-induced character errors.

About

Szu-Nien Chi
Department of Mechanical Engineering, National Taiwan University

Thesis: A World-State-Centered Retrieval-Augmented Framework for Human–Robot Collaboration Applications (以世界狀態為核心進行檢索增強人機協作之應用).