Methods for Pilots, PoCs, and Verification of Robo Claw in Travel and Tourism Startups
Once the requirements are finalized, a pilot is built for one region/one facility or one experience product/one operation/one system connection, and both normal and abnormal scenarios are verified before deciding whether to move to production.
It is important to narrow the scope of the pilot. Limit it to one region/one facility or one experience product/one operation/one system connection, and be sure to verify scenarios such as misclassification, incorrect summarization, incorrect translation, misrecognition of reservation/price/inventory information, misrecognition of itinerary/access information, inclusion of personal/location information, incorrect transmission, incorrect update, duplicate execution, duplicate reservations, prompt injection, and SaaS API failure/specification changes. Only after confirming that human approval, escalation to the safety responsible person, stop, rollback, and switching to manual operation function as designed can you proceed to the decision to move to production.
Who This Is For
Who This Is For
Targeted at executives, reservation managers, and CS managers who have finalized requirements in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In the Build & Validate STEP, define the scope of the pilot, build Agent, Skill, and Tool Policy, conduct normal and abnormal tests, and decide whether to move to production.
Pilot Scope
Scope of the Pilot
It is recommended to conduct the Pilot with a limited scope according to the following units.
1 Region
Even if multiple regions are being deployed, verification should first be limited to one region.
1 Facility or 1 Experience Product
Even if there are multiple facilities or multiple products, focus on one facility or one experience product.
1 Type of Inquiry / 1 Business Task
Limit to the one business task and one type of inquiry decided in Discover, and do not verify multiple business tasks simultaneously.
1 Sales Channel / 1 System Connection
Limit the connected SaaS and sales channel to the minimum, making it easier to verify the approval flow. Focus mainly on reading and drafting, leaving human verification in the Pilot.
Method
Implementation Steps
1. Build the Pilot Environment
Build Agents, Skills, and Tools with test data, without using actual customer or reservation data.
2. Implement Tool Policy
Implement the Allow/Deny range defined in Refine as a Tool Policy. Actual booking confirmation and real-time pricing updates should be implemented as Deny.
3. Test normal scenario
Verify whether expected classification, summarization, and draft are generated for the assumed input.
4. Test abnormal scenarios
Verify misclassification, incorrect summarization, incorrect translation, misrecognition of reservation, pricing, and inventory information, misrecognition of itinerary and access information, inclusion of personal or location information, incorrect transmission, incorrect update, duplicate execution, duplicate reservation, Prompt Injection, SaaS API failures or specification changes.
5. Confirm human approval flow
Check whether human verification and approval function properly for operations that require approval.
6. Confirm escalation paths to the safety officer
Verify whether the paths function in situations where itinerary safety decisions or disaster/adverse weather responses are expected.
7. Confirm stop, rollback, and manual operation switching
Verify whether procedures to stop processing, roll back, and switch to manual operation in case of abnormalities function properly.
8. Decide in accordance with production migration criteria
Determine whether migration to production is possible according to the pre-established Go/No-Go criteria.
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The test personnel verify and approve whether the guidance text drafts and response drafts generated by Pilot can be reflected.
- The management and reservation managers review the settings of the Tool Policy.
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of erroneous updates, mistaken transmissions, and duplicate reservations.
Number of incorrect operations or duplicate processes detected during the Pilot period
Evaluation of acceptance by the person in charge.
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If abnormal case verification is omitted, the behavior during SaaS failures may be unexpected after actual operation.
Testing with actual customer and reservation data in production.
Using real customer information, itineraries, and location data from the verification stage increases the risk of mistaken transmission and information leaks.
Not preparing rollback procedures
Without procedures to revert when problems occur, recovery takes time and impacts customer response
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario verification (misclassification, itinerary misrecognition, personal location information inclusion, mistaken transmission, double execution, duplicate reservation, Prompt Injection, SaaS API failure) has been completed.
- Human approval flow functions as designed
- Rollback procedures are established and actual rollback has been confirmed
- Confirmed that operations outside of authority (including confirming reservations and final updates of fees in production) are rejected.
- Practical issues are resolved in acceptance testing by the person in charge.
- Procedures for switching to manual operation in case of failure are established.
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It varies depending on the complexity of the target business and connected SaaS. Since a uniform period cannot be presented, please consult individually considering the scope.
Can verification be done without using actual customer and reservation data in production?
It is possible. We recommend preparing test data that imitates real data and conducting it in a verification environment that does not include personal information.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the Pilot design and verification together?
The scope, test scenarios, and Go/No-Go criteria can be concretized through discussions in the formal LP.