Pilot, PoC, and verification methods of Robo Claw in major tourism and accommodation companies
Once the requirements are finalized, build a pilot with 1 facility, 1 region, 1 operation, and 1 connection scope, verify both normal and abnormal scenarios, and then determine whether to proceed to production.
It is important for the pilot to narrow the scope. Limit it to one facility, one region, one operation, and one connection scope, and be sure to verify abnormal scenarios such as misclassification, incorrect summarization, incorrect translation, misdelivery, misrecognition of reservation information, misrecognition of fee/inventory information, misrecognition of itinerary/access information, insufficient permissions, double execution, duplicate reservations, mixing of personal/location information, prompt injection, and external API failures. Only when you can confirm that human approval, escalation to the safety officer, stop, rollback, and switch to manual operation function as designed, you can proceed to decide on the production deployment.
Who This Is For
Who This Is For
This is aimed at the information systems department, reservation department managers, and risk management officers who have finalized requirements in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In the Build & Validate STEP, define the scope of the pilot, build Agent, Skill, and Tool Policy, conduct normal and abnormal tests, and decide whether to move to production.
Pilot Scope
Scope of the Pilot
It is recommended to conduct the Pilot with a limited scope according to the following units.
1 facility
Even if multiple facilities are deployed, start by limiting testing to one facility.
1 Region
Instead of all regions, target one region with distinct characteristics.
One business process
Limit the verification to the one business process decided in Discover; do not verify multiple business processes simultaneously.
One connection range
Limit the connected systems to a minimum to make it easier to verify permissions and approval flows.
Method
Implementation Steps
1. Build the Pilot Environment
Build Agents, Skills, and Tools with test data, without using actual customer or reservation data.
2. Implement Tool Policy
Implement the Allow/Deny scope defined in Refine as a Tool Policy.
3. Test normal scenario
Verify whether expected classification, summarization, and draft are generated for the assumed input.
4. Test abnormal scenarios
Verify misclassification, incorrect summarization, incorrect translation, misdelivery, misrecognition of reservation/fee/itinerary information, insufficient permissions, double execution, duplicate reservations, mixing of personal/location information, prompt injection, and external API failures.
5. Confirm human approval flow
Check whether human verification and approval function properly for operations that require approval.
6. Confirm stop, rollback, and switch to manual operation.
Verify whether procedures to stop processing, roll back, and switch to manual operation in case of abnormalities function properly.
7. Make a judgment based on the production migration criteria
Determine whether migration to production is possible according to the pre-established Go/No-Go criteria.
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The test personnel will check and approve whether the reservation change proposals, fare update proposals, and guidance texts generated by Pilot can be reflected.
- The settings of the Tool Policy will be reviewed by the Information Systems Department, the Reservation Department Manager, and the Risk Management Manager.
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of misclassification and duplicate bookings
Number of incorrect operations or duplicate processes detected during the Pilot period
User acceptance evaluation
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If you skip verification of abnormal cases, the behavior of the reservation system during a failure in production will be unexpected.
End up testing with actual customers and location information
Using actual customer information or location data from the verification stage increases the risk of misdelivery and information leakage.
Not preparing rollback procedures
Without procedures to revert when problems occur, recovery takes time and impacts customer response
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario (misclassification, incorrect sending, duplicate reservations, personal location information mixing, prompt injection, external API failure) has been verified
- Human approval flow functions as designed
- Rollback procedures are established and actual rollback has been confirmed
- Confirmed that operations outside of permissions are rejected
- User acceptance tests have resolved practical issues
- Procedures for switching to manual operation in case of failure are established.
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It varies depending on the target operations and the complexity of reservation management and PMS integration. Since a uniform period cannot be indicated, please consult with us individually based on the scope involved.
Is it possible to verify without using actual customer data?
] Yes, it is possible. We recommend preparing test data that mimics real data and conducting it in a verification environment that does not contain confidential information.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the Pilot design and verification together?
The scope, test scenarios, and Go/No-Go criteria can be concretized through discussions in the formal LP.