Pilot, PoC, and verification methods of Robo Claw in major TMT companies
Based on the requirements defined in Refine, we build Agent, Skill, and Tool Policies within a limited target scope and verify them, including normal cases, abnormal cases, duplicate execution, unauthorized operations, API failures, and human approval, to check whether the criteria for moving to production are met.
For the pilot, limit the target business and target team to one, and verify not only normal cases but also abnormal cases (API failures, insufficient permissions, secret expiration, operator errors, duplicate execution). Omitting verification of abnormal cases makes it more likely that issues such as incorrect PR creation or duplicate processing will surface after production operation.
Who This Is For
Who This Is For
This is intended for persons responsible in the development department, QA managers, information systems department, and security department, whose requirements were solidified in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In Build & Validate, finalize the scope of the pilot, construct agents, skills, and tools, and then conduct tests for normal and abnormal cases to evaluate whether the criteria for going live are met.
Industry Challenges
Issues specific to major TMT companies
Preparing test data is difficult
Actual customer data and source code cannot be used directly for testing, and preparing test data takes time.
It is easy to overlook abnormal scenarios.
While normal review and response flows are easy to verify, abnormal scenarios such as API failures, insufficient permissions, or expired Secrets are often overlooked.
Risk of double execution or creating incorrect PRs
Due to periodic execution or retry processing, the same Pull Request may be created multiple times, or changes may be made to old branches.
The criteria for moving to production are unclear.
In many cases, the criteria for whether to move to production based on pilot results are not decided in advance.
Method
Implementation Steps
1. Determine the scope of the pilot.
Design the pilot focusing on one product and one to two business processes, within a range that is easy to verify.
2. Build the Agent, Skill, and Tool.
Build the necessary Agents for the target business processes, the Skills for business procedures, and Tools for external operations.
3. Set the Tool Policy.
Set the range of operations the Tool is allowed to execute (Allow/Deny) as Policies for the verification environment.
4. Prepare test data.
Prepare test tickets and dummy repositories that mimic real data, and build the verification environment in a way that does not include confidential information.
5. Test the normal case
Verify whether ticket classification, review comment creation, and notification flows function as expected.
6. Test abnormal cases
Check scenarios for abnormal cases, such as API failures, insufficient permissions, expired Secrets, double execution, and timeout for pending approvals.
7. Verify human approval and security
Check whether the approval flow functions as designed and whether access permissions are as intended.
8. Conduct user acceptance testing
Have developers, QA, and CS staff who will actually use the system try it out to confirm its practical usability.
Test Scenarios
Pilot Evaluation Form
The following are examples of verification items. Judgments are recorded in three levels: 'Pass,' 'Conditional Pass,' and 'Fail,' and are used as a basis for deciding whether to transition to production.
| Verification items | Scenario example | Checkpoints |
|---|---|---|
| Normal case | Normal ticket classification and creation of review comments | Whether the expected classification and comment drafts are created |
| Abnormal cases (API failure) | Connection failures to GitHub or Jira | Check if it stops safely during errors and does not output incorrect information |
| Abnormal cases (double execution) | Overlapping or re-execution of scheduled runs | Whether duplicate Pull Requests or comments are created |
| Human approval | Pre-merge review of Pull Requests | Whether merging without approval is prevented and whether approval records are retained |
| Security | Attempts to access repositories without permission | Whether unauthorized references or updates are rejected |
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The test personnel confirm and approve the adoption of review comments and response drafts generated by Pilot
- The Information Systems Department and Security Department review the settings of the Tool Policy
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of erroneous execution or double execution
Number of incorrect operations or duplicate processes detected during the Pilot period
User acceptance evaluation
Evaluation of usability and business suitability by actual users
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If verification of abnormal cases is omitted, the behavior during API failures after production operation will be unpredictable.
Testing with production data or production repository
Using actual customer data or production code from the verification stage increases the risk of information leakage or erroneous changes.
Decide the criteria afterwards
Trying to decide the criteria for production transition after the pilot tends to result in arbitrary judgments.
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario verification (API failures, double execution, insufficient permissions) has been completed.
- Human approval flow functions as designed
- Confirmed that operations outside of permissions are rejected
- User acceptance tests have resolved practical issues
- Production operation system (responsible for monitoring and failure response) has been determined.
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It depends on the complexity of the target business and system integration. Since a uniform period cannot be presented, please consult individually considering the target scope.
Is it possible to verify without using production code or data?
It is possible. It is recommended to prepare test tickets and dummy repositories mimicking actual data, and conduct the verification in an environment that does not include confidential information.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the Pilot design together?
The scope of targets, test scenarios, and criteria for production transition can be specified through consultations in the official LP.