How to implement and operate Robo Claw in production at a major TMT company
Organizes the necessary authentication, secret management, access permissions, logs, monitoring, alerts, incident response, and change management when moving target operations that met the standards in the pilot to production.
In production operations, in addition to the normal and abnormal behavior confirmed in the pilot, it is a prerequisite to establish a system for continuous monitoring, alerts, incident response, and change management. In particular, clearly documenting stop conditions and escalation points in advance will determine the initial response in the event of an incident.
Who This Is For
Who This Is For
This is intended for the Information Systems Department, Security Department, and SRE/operations managers to deploy business processes that have been verified in the Pilot environment into production.
What You'll Decide
What You'll Decide in This Step
In Deploy & Operate, authentication, permissions, and monitoring structures in the production environment are finalized, and stop conditions and recovery procedures during failures, as well as the approach for continuous change management, are determined.
Industry Challenges
Issues specific to major TMT companies
Alignment with a 24-hour on-call system
It is necessary to organize how SRE on-call duties and Agent monitoring and alert responses are divided.
Failure response across multiple products and environments
When configurations differ for each product, it takes time to isolate problems when a failure occurs.
Fault tolerance for external APIs (GitHub, Jira, etc.)
In the case of specification changes or temporary failures of external tools, change management and fault-tolerant design are required for the integration parts.
Need for cost monitoring
As the frequency of always-on or periodic executions increases, usage costs accumulate, so continuous cost monitoring is necessary.
Method
Implementation Steps
1. Prepare the Execution Environment
Prepare the production environment (cloud, etc.) and check the differences with the Pilot environment, maintaining separation between development, verification, and production.
2. Finalize authentication and access permissions
Determine the authentication method for production users and Agents, and the access rights by product and team.
3. Organize secret management
Establish a system to securely manage and rotate confidential information such as API keys and tokens.
4. Establish a logging and audit system
Organize the storage locations, retention periods, and reference methods for operation logs during audits.
5. Set up monitoring and alerts
Monitor the operating status of Agents, execution failures, abnormal processing counts, etc., and set alerts.
6. Define stop conditions and fault handling
Define the conditions for automatic stop in case of anomaly detection, and the contact and recovery flow in case of failure.
7. Establish change management
Establish a process to check the impact scope when the destination system is changed or Skill/Tool versions are updated.
8. Conduct cost monitoring and regular reviews
Visualize usage costs and establish a system to regularly review operational status.
Data & Systems
Data and Systems Used
Human-in-the-loop
Where Human Approval Is Required
- Final approval for migration to the production environment
- Approval for issuing and rotating secrets and authentication information
- Approval for Skill/Tool updates due to changes in the destination system specifications
- Approval to execute recovery procedures in case of a failure
Measurement
KPI
Operating rate
Operating rate of Agents/Skills in production
Time from failure detection to recovery (MTTR)
Average time taken from anomaly detection to recovery
Alert response time
Time from alert issuance to first response
Pitfalls
Common Pitfalls
Leaving it entirely to monitoring after introduction
If you operate without deciding the monitoring system after going live, detection of failures will be delayed.
Manage secrets individually
If API keys, etc. are saved on personal PCs or notes, risks remain in the case of resignation or transfer.
Omit change management
If operations continue without understanding changes in the destination system specifications or model updates, errors in outputs or processing may occur.
Checklist
Production Operation Checklist
- Authentication and access rights in the production environment are determined
- Procedures for secret management and rotation are established
- The storage destinations and retention periods for operation logs are determined.
- The monitoring and alert targets and notification destinations are set.
- Conditions for stopping in abnormal situations and recovery procedures are documented.
- There is a change management process when the destination system is changed
- There is a mechanism to visualize usage costs
- The frequency of regular reviews and participants is determined.
FAQ
Frequently Asked Questions
Who is responsible for monitoring during production operation?
Typically, the Information Systems Department or SRE team takes the lead, and depending on the business content, collaborates with the development and QA departments. Specific role assignments are designed individually.
Does the Agent automatically stop in case of a failure?
It is recommended to design it to automatically stop if it meets predefined stop conditions, but as the design of conditions differs depending on the business, it is considered individually.
How do you handle security audits?
Maintaining operation logs, access permissions, and approval records makes it easier to respond to audits, but specific audit requirements vary according to the organization’s internal audit standards.
Shall we organize the production configuration together?
The production operation configuration, including authentication, monitoring, incident response, and change management, can be concretized through consultation in an official LP.