Robo Claw Pilot and PoC Validation for GovTech Startups
Based on the requirements defined in Refine, this explains how to build Agent, Skill and Tool Policy for one municipality, one department, one program and one resident touchpoint, run normal-path and error-path tests, and judge whether to move to production. It covers test items for risks specific to the public sector, including misclassification, misdirected sends, incorrect updates, duplicate application handling, Prompt Injection, and SaaS/API outages.
Who This Is For
Who This Is For
This is intended for product owners, PoC (proof-of-concept) leads, information security leads, and personal data protection leads who defined requirements in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In the Build & Validate STEP, you build a limited-scope Pilot, run normal-path and error-path tests, and judge whether to move to production.
Pilot Scope
Pilot Unit
Validating multiple municipalities or multiple departments at the same time makes permission design and results evaluation complex. We recommend limiting the first Pilot to the following unit.
One Municipality, One Department
Even when working with multiple municipalities, limit the first Pilot to one municipality and one department.
One Program, One Resident Touchpoint
Narrow the target program or resident touchpoint (inquiries, applications, reservations, etc.) down to one.
One Inquiry/Application Type
Limit the target inquiry/application type to one, making risk easier to assess.
One System Connection
Limit the connected inquiry-management/application-intake system to one.
Industry Challenges
Challenges Often Faced During a Pilot
Test coverage is incomplete
Only normal-path cases get checked, and error-path tests for personal data contamination or misdirected sends tend to be insufficient.
Go-live criteria are not decided in advance
It's common for the project to proceed without a clear picture of what must be satisfied after the Pilot ends before moving to production.
No fallback procedure to manual operation
When a problem occurs during the Pilot, a procedure for switching back to manual operation is sometimes not in place.
Municipality staff get involved too late
Municipality staff are not involved from the test design stage, which can lead to rework later.
Method
Implementation Steps
1. Fix the Pilot scope
Fix the scope to one municipality, one department, one program and one resident touchpoint.
2. Build Agent, Skill and Tool Policy
Build the actual Agent, Skill and Tool Policy based on the design defined in the Refine STEP.
3. Prepare test data
Prepare test inquiry/application data that contains no production resident information or identity-verification information.
4. Run normal-path tests
Confirm that initial classification of resident inquiries, program/FAQ lookup, and draft-reply generation are performed correctly.
5. Run error-path tests
Based on the error-path test items below, confirm behavior under misrecognition, misdirected sends, data contamination and outages.
6. Validate human approval, stop and rollback
Confirm that human approval functions correctly and that the system can stop and roll back when a problem occurs.
7. Validate the switch to manual operation
Confirm that the procedure for switching to manual operation works when the Agent is stopped.
8. Decide on production rollout
The product owner and municipality staff compare test results against the Go/No-Go criteria and decide whether to move to production.
Test Scenarios
Error-Path Test Items
Below are example error-path test items to validate for a GovTech startup Pilot. Add and adjust them to fit your own operations and systems.
| Test Item | Assumed Scenario | What to Confirm |
|---|---|---|
| Misclassification | An inquiry/application type is classified incorrectly | Whether misclassification is detected and human correction functions |
| Incorrect summarization | An error occurs in the summary content of a meeting/interview record | Whether a human can review the differences between the content before and after summarization |
| Resident information misrecognition | Application content or resident information is read incorrectly | Whether misrecognition is detected and a human can correct it before finalization |
| Sensitive information misrecognition | Sensitive information is recognized incorrectly | Whether the design has a human perform finalization |
| Identity-verification information misrecognition | Identity-verification information is recognized incorrectly | Whether the final identity-verification determination is made by a human |
| Oversight of benefit/eligibility-related information | Important information relevant to a benefit/eligibility decision is overlooked | Whether it is reliably escalated to the responsible municipal department |
| Personal/sensitive information contamination | Resident personal/sensitive information is unintentionally included in output | Whether pre-output filtering and human review can remove it |
| Misdirected send | Content is sent to the wrong destination or with the wrong content, to a resident or municipal staff member | Whether pre-send approval, send logs, and a correction procedure for misdirected sends function |
| Incorrect publication | Incorrect content is published to a resident-facing page | Whether pre-publication approval, publication logs, and a correction procedure for incorrect publication function |
| Incorrect update | Application data or resident records are updated incorrectly | Whether pre-update confirmation and change-history retention function |
| Double execution | The same process is executed multiple times | Whether an idempotency check prevents duplication |
| Duplicate application handling | The same application is processed more than once | Whether duplication is detected by checking application history |
| Prompt Injection | An instruction hidden in input data causes the Agent to behave unintentionally | Whether the design ignores suspicious instructions and does not proceed to finalized processing |
| SaaS/API outage | A connected SaaS/API does not respond | Whether the system fails safe and notifies staff on error |
| API specification change | A connected SaaS's specification changes | Whether the anomaly is detected and automatic execution is halted until revalidated |
| Human approval | Finalized processing is about to execute before approval | Whether the design prevents send/update/publish from being finalized without approval |
| Escalation | Output occurs that may affect a benefit, eligibility or identity-verification decision | Whether it is reliably escalated to the responsible municipal department and the information policy/security lead |
| Stop/rollback | Response after incorrect output or processing has occurred | Whether execution can be stopped and a procedure exists to restore the prior state |
| Manual operation switch | The Agent is temporarily unavailable | Whether a procedure for switching to manual operation is in place and operations can continue |
| Production rollout criteria | The rollout decision at the end of the Pilot | Whether the responsible party can decide in line with the Go/No-Go criteria |
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- Review and approval of draft replies and candidate missing-item content
- Review by the responsible party of candidates involving personal/sensitive information
- Content review before sending to residents/municipal staff or publishing to a resident-facing page
- The decision to stop and roll back when an anomaly is detected
Measurement
KPI
Error-path test pass rate
The share of defined error-path test items that met the pass criteria
Human review rate
The share of output for which human review was performed
Misdirected sends / duplicate application handling incidents
The number of misdirected sends and duplicate application handling incidents that occurred during the Pilot period
Pitfalls
Common Pitfalls
Skipping error-path testing
Moving to production after checking only normal-path cases carries the risk of starting operation without noticing sensitive information contamination or duplicate application handling.
Testing with production data
Using production data that contains residents' personal information or identity-verification information as-is for testing creates an unnecessary risk of information leakage.
Deciding Go/No-Go criteria after the fact
If the criteria are not defined in advance, the decision to move to production becomes ad hoc.
Go / No-Go Criteria
Rollout Considerations
- All defined error-path test items have been confirmed to behave as expected on the safe side
- The human approval and escalation paths have been confirmed to actually function
- The stop/rollback procedure and the switch to manual operation have been tried and confirmed to actually function
- The product owner and municipal staff have reviewed the results and agreed to move to production
- There is a clear prospect that the production monitoring/logging setup will meet the Deploy & Operate requirements
FAQ
Frequently Asked Questions
How long should a Pilot run?
It depends on workload and the number of test items, but a few weeks to about one to two months is a common guideline. We recommend prioritizing error-path test coverage rather than proceeding on a fixed timeline.
Can production resident data be used for testing?
We do not recommend it. Prepare anonymized/pseudonymized test data, and avoid testing with data that contains production personal information or identity-verification information.
What should benefit/eligibility-decision testing confirm?
Confirm that the design stops at information organization and escalation, and that the final benefit/eligibility decision is always made by the responsible municipal department.
Who decides whether to move to production?
We recommend that the product owner and municipal staff make this decision by comparing test results against the Go/No-Go criteria. If personal data protection issues are involved, the personal data protection lead and legal counsel also take part in the decision.
Let's work through your Pilot and PoC design together.
Review your scope, test items and Go/No-Go criteria, and discuss your Pilot design on our official landing page.