Methods for Pilots, PoCs, and Verification of Robo Claw in NGOs Handling Support Logistics
In Build & Validate, a pilot is constructed limited to one region, one category of supplies, and one transport process. Operations start focusing on reading and drafting, followed by tests for normal and abnormal cases to evaluate whether the criteria for going live are met.
Do not use the NGO's actual production data for the pilot. Conduct it in a test environment separated from production with anonymized or pseudo-anonymized test data. Verify abnormal cases such as misclassification, mistranslation, incorrect transmission, duplicate delivery, inclusion of personal information, prompt injection, and communication failures, and confirm that the allocation of aid supplies and safety decisions are not delegated to the agent before making a decision to go live.
Who This Is For
Who This Is For
Targeted at logistics and transport managers, information system personnel (including those with concurrent roles), and local site managers who have finalized requirements in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In Build & Validate, finalize the scope of the pilot, construct agents, skills, and tools, and then conduct tests for normal and abnormal cases to evaluate whether the criteria for going live are met.
Industry Challenges
Challenges specific to NGOs and NPOs in charge of support logistics
Preparing test data is difficult
You cannot use actual delivery destination information or aid recipient information directly in tests, and it takes time to prepare test data that has been anonymized or pseudo-anonymized.
It is easy to overlook abnormal scenarios.
While normal classification and summary flows are easy to verify, abnormal scenarios such as misdelivery, duplicate delivery, inclusion of personal or location information, and prompt injection tend to be overlooked.
Testing for communication failures is often insufficient.
There are cases where production is implemented without conducting failure tests assuming a local site's unstable communication environment.
The criteria for moving to production are unclear.
In many cases, the criteria for whether to move to production based on pilot results are not decided in advance.
Method
Implementation Steps
1. Determine the scope of the pilot.
Design the pilot within a verifiable scope, limited to one region, one category of goods, and one transport process.
2. Build the Agent, Skill, and Tool.
Construct the Agent, workflow Skill, and system operation Tool necessary for the target business.
3. Set the Tool Policy.
Define the range of operations (read/write, Allow/Deny) that the Tool is permitted to perform as a policy.
4. Prepare test data.
Prepare anonymized or pseudonymized test data and build a test environment separated from the production environment.
5. Test the normal case
We will check whether the flow of transport request classification, status aggregation, and draft reporting functions as expected.
6. Test abnormal cases
We will check abnormal scenarios such as insufficient permissions, misclassification, mistranslation, mistaken transmission, duplicate delivery, incorrect delivery destination, inclusion of personal or location information, prompt injection, external API failures, communication failures, and double execution.
7. Confirm human approval and responsible person escalation
We will check whether the verification flow functions as designed and whether it is reliably escalated when a safety decision for emergency transport is required.
8. Conduct user acceptance testing
We will have the staff, volunteers, and local office personnel who actually use it try it out and verify its practical usability.
Test Scenarios
Pilot Evaluation Form
The following are examples of verification items. Judgments are recorded in three levels: 'Pass,' 'Conditional Pass,' and 'Fail,' and are used as a basis for deciding whether to transition to production.
| Verification items | Scenario example | Checkpoints |
|---|---|---|
| Normal case | Creation of a plan for the classification and summarization of normal transport requests | Whether the draft and organization results are created as expected |
| Abnormal case (duplicate delivery) | Duplicate transport requests are generated for the same delivery destination | Does the detection and stop mechanism function? |
| Abnormal case (personal information or location information mixed in) | Personal information of the delivery destination is unintentionally included in the output | Does the detection and removal mechanism function? |
| Abnormal case (communication failure) | Communication with the local site is temporarily interrupted | Can it be switched to alternative means or manual operation? |
| Human verification | Review of draft reports and draft notifications | Are they not sent without verification, and is the verification record retained? |
| Audit trail | Recording of operation logs | Are inputs/outputs, executors, and execution details recorded without omission? |
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Review Is Required
- Test personnel confirm the usability of transport request classification drafts and report drafts generated in the Pilot
- Logistics/transport managers and IT personnel review the contents of the Tool Policy settings
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of misdelivery or duplicate delivery
Number of misoperations or duplicate delivery candidates detected during the Pilot period
User acceptance evaluation
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
Omitting verification of abnormal cases may lead to unexpected responses to duplicate deliveries or personal information mixing after actual operation
Testing with actual delivery destination information
Using the personal information of actual support recipients as-is during the verification stage increases the risk of information leakage
Not verifying alternative measures during communication failures
Omitting tests that assume the communication environment of local bases carries the risk of being unable to switch to manual operation when actual failures occur
Go / No-Go Criteria
Checklist for production migration decision
- Can detect and correct errors
- No authority deviation
- Can control external transmission of personal information and location information
- Human verification functions correctly
- Allocation, prioritization, and safety judgment of support supplies are not delegated to AI
- Can escalate to the person in charge
- Can detect and stop duplicate deliveries
- Can revert to manual operations
- There is an alternative means even in case of communication failure
- Can continue operations with NGO personnel and budget
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It varies depending on the target operations and the complexity of data classification. Since a uniform period cannot be presented, please consult individually considering the scope.
Can verification be done without using actual delivery destination information?
It is possible. In fact, it is recommended. Prepare test data with anonymization or pseudonymization, and conduct the verification in an environment separate from the production environment.
Should exceptional operations during disasters also be verified in the Pilot?
It is recommended to verify it as a scenario as much as possible. However, actual safety and evacuation decisions during disasters are always made by humans, regardless of the Pilot results.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize a Pilot for limited operations together?
The scope, test data, abnormal scenario cases, and criteria for moving to production can be concretized through consultations on the official LP.