Pilot, PoC, and Verification Methods of Robo Claw in Major Logistics Companies
Based on the requirements defined in Refine, build Agent, Skill, and Tool Policies within a limited target scope, and verify including normal cases, abnormal cases, false notifications, duplicate executions, connection failures, and human approvals to check whether the criteria for moving to production are met.
For the pilot, focus on a single target operation and location, and verify not only normal cases but also abnormal cases (system connection failures, incorrect information, duplicate executions, missed approvals). Skipping verification of abnormal cases can make problems such as false notifications or duplicate processing more likely to appear after production deployment.
Who This Is For
Who This Is For
The target participants are the operation management supervisor, information systems department staff, and security department staff, whose requirements were finalized in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In Build & Validate, finalize the scope of the pilot, construct agents, skills, and tools, and then conduct tests for normal and abnormal cases to evaluate whether the criteria for going live are met.
Industry Challenges
Challenges Specific to Major Logistics Companies
Preparing test data is difficult
Actual delivery status data and shipper information cannot be used directly for testing, so preparing test data takes time.
It is easy to overlook abnormal scenarios.
While normal delivery flows are easy to verify, abnormal cases such as accidents, delays, and system connection failures are often overlooked.
Risk of Duplicate Execution and False Notification
Due to regular execution and retry processes, there is a risk that the same notification may be sent multiple times, or that old information may be resent.
The criteria for moving to production are unclear.
In many cases, the criteria for whether to move to production based on pilot results are not decided in advance.
Method
Implementation Steps
1. Determine the scope of the pilot.
The pilot is designed by narrowing down to one location and one or two operations so that it can be verified within a manageable scope.
2. Build the Agent, Skill, and Tool.
Build the necessary Agents for the target business processes, the Skills for business procedures, and Tools for external operations.
3. Set the Tool Policy.
Set the range of operations (Allow/Deny) that the tool is permitted to execute as a policy.
4. Prepare test data.
Prepare test data that simulates actual data and build a verification environment in a way that does not include sensitive information.
5. Test the normal case
Check whether the delivery status, report creation, and notification flow function as expected.
6. Test abnormal cases
Verify abnormal scenarios such as erroneous notifications, duplicate execution, system connection failures, and approval waiting timeouts.
7. Verify human approval and security
Check whether the approval flow functions as designed and whether access permissions are as intended.
8. Conduct user acceptance testing
Have actual operations managers and drivers use it to check usability in practical work.
Test Scenarios
Pilot Evaluation Form
The following are examples of verification items. Judgments are recorded in three levels: 'Pass,' 'Conditional Pass,' and 'Fail,' and are used as a basis for deciding whether to transition to production.
| Verification items | Scenario example | Checkpoints |
|---|---|---|
| Normal case | Normal delivery status confirmation and report creation | Check whether expected drafts and notifications are created |
| Abnormal scenarios (system failure) | Connection failures to TMS/GPS | Check if it stops safely during errors and does not output incorrect information |
| Abnormal cases (double execution) | Overlapping or re-execution of scheduled runs | Check whether the same notification is not sent multiple times |
| Human approval | Pre-approval for sending reports to the shipper | Check whether it is not sent without approval and whether approval records remain |
| Security | Attempting operations beyond authority | Check whether unauthorized updates and access are denied |
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The test personnel check and approve whether the reports for the consignor generated with Pilot can be sent.
- The Information Systems Department and Security Department review the settings of the Tool Policy
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of erroneous notifications and double execution.
Number of erroneous notifications and duplicate processing detected during the Pilot period.
User acceptance evaluation
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If the verification of abnormal cases is omitted, the behavior of the system during failures will be unexpected after production deployment.
Testing using production data.
Using actual consignor information or personal information from the verification stage increases the risk of information leaks.
Decide the criteria afterwards
Trying to decide the criteria for production transition after the pilot tends to result in arbitrary judgments.
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario (system failure, double execution, erroneous notification) has been verified.
- Human approval flow functions as designed
- Confirmed that operations outside of permissions are rejected
- User acceptance tests have resolved practical issues
- Production operation system (responsible for monitoring and failure response) has been determined.
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It depends on the complexity of the target business and system integration. Since a uniform period cannot be presented, please consult individually considering the target scope.
Can verification be done without using production data?
] Yes, it is possible. We recommend preparing test data that mimics real data and conducting it in a verification environment that does not contain confidential information.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the Pilot design together?
The scope of targets, test scenarios, and criteria for production transition can be specified through consultations in the official LP.