Robo Claw Pilot, PoC, and verification methods in local governments
Once the requirements are fixed, build a pilot for one department, one business, one data classification, and one connection range, verify both normal and abnormal cases, and then decide whether to move to production.
It is important for the pilot to narrow the scope. Limit it to one department, one business process, one data classification, and one connection range, and make sure to verify scenarios such as misclassification, incorrect summarization, misrecognition of ordinances/regulations information, inclusion of resident information, incorrect transmission of personal information, incorrect response candidates, errors in legal interpretation, insufficient authority, exceeding authority between departments, double execution, prompt injection, and external API failures. Only after confirming that human approval, escalation to legal and personal information protection officers, stopping, rollback, and switching to manual operation function as designed, can you proceed with the decision to move to production.
Who This Is For
Who This Is For
Refine STEP targets the information systems department, the responsible department headers, and the personal information protection officers who have finalized requirements.
What You'll Decide
What You'll Decide in This Step
In the Build & Validate STEP, define the scope of the pilot, build Agent, Skill, and Tool Policy, conduct normal and abnormal tests, and decide whether to move to production.
Pilot Scope
Scope of the Pilot
It is recommended to conduct the Pilot with a limited scope according to the following units.
One department
Even in organizations with multiple departments, verification is first limited to one department.
One business process
Limit the verification to the one business process decided in Discover; do not verify multiple business processes simultaneously.
One data classification
Limit the type of resident information handled to one data classification to make it easier to verify authority and approval flows.
One connection range
Minimize the systems connected and start with read-only or draft-focused operations.
Method
Implementation Steps
1. Build the Pilot Environment
Build Agents, Skills, and Tools using test data instead of live resident information.
2. Implement Tool Policy
Implement the range of Allow/Deny defined in Refine as a Tool Policy. Confirmed updates to resident records will be implemented as Deny.
3. Test normal scenario
Verify whether expected classification, summarization, and draft are generated for the assumed input.
4. Test abnormal scenarios
Verify misclassification, incorrect summarization, misrecognition of ordinances/regulations information, contamination of resident information, incorrect transmission of personal information, incorrect response candidates, errors in legal interpretation, insufficient authority, authority deviations between departments, duplicate execution, Prompt Injection, and external API failures.
5. Confirm human approval flow
Check whether human verification and approval function properly for operations that require approval.
6. Check the escalation route
Confirm whether the route functions in situations that require escalation to the legal or personal information protection team.
7. Confirm stop, rollback, and manual operation switching
Verify whether procedures to stop processing, roll back, and switch to manual operation in case of abnormalities function properly.
8. Decide in accordance with production migration criteria
Determine whether migration to production is possible according to the pre-established Go/No-Go criteria.
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The test personnel will check and approve whether the response candidates and summaries generated in the Pilot can be reflected
- Information system department, responsible department, and personal information protection officer will review the Tool Policy settings
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of erroneous transmission or erroneous updates
Number of incorrect operations or duplicate processes detected during the Pilot period
Staff acceptance evaluation
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If the verification of abnormal cases is omitted, the behavior of the system during failures will be unexpected after production deployment.
Testing with actual resident information in production
Using actual resident information from the verification stage increases the risk of misdelivery and information leakage.
Not preparing rollback procedures
If there is no procedure to revert in case of a problem, recovery will take time and the impact on resident support will expand.
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario testing (misclassification, resident information mixing, misdelivery, double execution, prompt injection, external API failure) has been completed.
- Human approval flow functions as designed
- Rollback procedures are established and actual rollback has been confirmed
- It has been confirmed that operations beyond authorization (including final updates to resident records) are rejected.
- Practical issues have been resolved through staff acceptance testing.
- Procedures for switching to manual operation in case of failure are established.
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It varies depending on the complexity of the target operations and connected systems. A uniform period cannot be presented, so please consult individually taking the target scope into account.
Can verification be done without using production resident information?
It is possible. We recommend preparing test data that imitates real data and conducting it in a verification environment that does not include personal information.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the Pilot design and verification together?
The scope, test scenarios, and Go/No-Go criteria can be concretized through discussions in the formal LP.