Pilot, PoC, and verification methods of Robo Claw in TMT startups
Based on the requirements defined in Refine, build Agent, Skill, and Tool Policy in a small pilot for one business and one team, verify normal cases, abnormal cases, cost, rollback, and human approval, and check whether it meets the criteria for production migration.
For the pilot, focus on a single target business and verify not only normal cases but also abnormal cases (API failures, insufficient permissions, secret expiration, operational mistakes, duplicate execution) and rollback procedures. For small teams, it is also important to check costs and operational burden at the same time and determine whether it is sustainable.
Who This Is For
Who This Is For
This guide is for CTOs, development leads, and QA staff who hold dual roles and have finalized requirements in the Refine STEP.
What You'll Decide
What You'll Decide in This Step
In Build & Validate, confirm the scope of the pilot, build Agent, Skill, and Tool, conduct normal and abnormal case tests, check costs and operational burden, and evaluate whether it meets the criteria for production migration.
Startup Challenges
Challenges unique to TMT startups.
Preparing test data is difficult
Actual customer data and source code cannot be used directly for testing, and preparing test data takes time.
It is easy to overlook abnormal scenarios.
Due to limited personnel, they tend to be satisfied with checking only normal flows, and abnormal case verification such as API failures or insufficient permissions is often omitted.
Rollback procedures are not established
Procedures to return to the original state in case of a problem are not established, posing the risk of taking time to revert.
It is difficult to assess costs and operational burden
Even if the effect of the pilot is felt, it is difficult to judge whether the ongoing costs and the operational burden on staff handling multiple roles are worthwhile.
Method
Implementation Steps
1. Determine the scope of the pilot.
We will narrow it down to one task and one team and design the pilot within a scope that is easy to verify.
2. Build the Agent, Skill, and Tool.
Build the necessary Agents for the target business processes, the Skills for business procedures, and Tools for external operations.
3. Set the Tool Policy.
Set the range of operations the Tool is allowed to execute (Allow/Deny) as Policies for the verification environment.
4. Prepare test data.
Prepare test tickets and dummy repositories that mimic real data, and build the verification environment in a way that does not include confidential information.
5. Test the normal case
Verify whether ticket classification, review comment creation, and notification flows function as expected.
6. Test abnormal cases and rollback
Check scenarios such as API failures, insufficient permissions, secret expiration, and duplicate execution, as well as rollback procedures in case problems occur.
7. Check costs and operational burden
Verify whether the usage costs during the pilot period and the operational burden on staff handling multiple roles are sustainable.
8. Conduct user acceptance testing
Have developers, QA, and CS staff who will actually use the system try it out to confirm its practical usability.
Test Scenarios
Pilot Evaluation Form
The following are examples of verification items. Judgments are recorded in three levels: 'Pass,' 'Conditional Pass,' and 'Fail,' and are used as a basis for deciding whether to transition to production.
| Verification items | Scenario example | Checkpoints |
|---|---|---|
| Normal case | Normal ticket classification and creation of review comments | Whether the expected classification and comment drafts are created |
| Abnormal cases (API failure) | Connection failures to GitHub, Linear, etc. | Check if it stops safely during errors and does not output incorrect information |
| Rollback | Rollback after creating an incorrect Pull Request | Check whether it can return to the original state quickly |
| Human approval | Pre-merge review of Pull Requests | Whether merging occurs without approval |
| Costs and operational burden | Usage results during the pilot period | Whether it is within the cost limit and the staff handling multiple roles can continue operation |
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- The staff will check and approve whether to adopt review comments and proposed answers generated by the pilot
- The responsible person will review the settings of the Tool Policy
- The responsible person decides on the feasibility of moving to production based on pre-established judgment criteria
Measurement
KPI
Pass rate of test scenarios
The proportion of passed normal and abnormal scenarios
Number of occurrences of erroneous execution or double execution
Number of incorrect operations or duplicate processes detected during the Pilot period
Operational man-hours per case
The human labor required to process one pilot target task
Pitfalls
Common Pitfalls
Judged as pass with normal cases only
If verification of abnormal cases is omitted, the behavior during API failures after production operation will be unpredictable.
Not preparing rollback procedures
If there are no steps to revert in case of a problem, recovery will take time for a small team.
Moving to production without checking costs
If you move to production without checking usage during the pilot period, you may realize unexpected cost increases too late.
Go / No-Go Criteria
Checklist for production migration decision
- All normal scenario tests have passed
- Abnormal scenario verification (API failures, double execution, insufficient permissions) has been completed.
- Rollback procedures are established and actual rollback has been confirmed
- Human approval flow functions as designed
- The pilot period cost is within the expected cost ceiling
- The workload is manageable for staff who are handling multiple roles
FAQ
Frequently Asked Questions
What is the approximate duration of the Pilot period?
It depends on the complexity of the target business and system integration. Since a uniform period cannot be presented, please consult individually considering the target scope.
Can a small team run a pilot?
By focusing on one business and one team and starting with minimal authority design, it is possible to design a pilot that can operate without dedicated personnel. Please consult with us according to your individual setup.
What happens if a Pilot fails?
Review the Refine requirements and Tool Policy, and conduct the Pilot again. It is recommended to continue verification until the standards are met without forcing the transition to production.
Shall we organize the design of a small-scale pilot together?
The scope, test scenarios, costs, and production migration criteria can be concretized through consultation in the official LP.