raia Copilot User Guide
Scale Human Expertise 10x Through AI-Human Collaboration
The Complete Guide to Testing, Oversight, and Continuous Improvement
Overview
raia Copilot represents the critical bridge between AI automation and human expertise, enabling organizations to scale their output dramatically while preserving quality, judgment, and continuous improvement. While AI agents can handle vast volumes of interactions autonomously, the most successful deployments recognize that human oversight, feedback, and intervention remain essential—not as a failure of automation, but as a strategic advantage that combines the efficiency of AI with the nuance and judgment of human expertise.
Copilot provides the application that internal teams use to interact with, oversee, and refine AI agents in real time. Through an intuitive ChatGPT-like interface, team members can engage directly with agents, review conversations across all channels, provide feedback and ratings, take over interactions when needed, and run simulations to test agent responses before production deployment. This human-in-the-loop approach creates a continuous feedback cycle where agents learn from human expertise, improving accuracy and effectiveness over time while maintaining the human judgment necessary for complex, sensitive, or high-stakes interactions.
How raia Copilot Works
Unified Omnichannel Conversation View
One of Copilot's most powerful capabilities is its unified view of all conversations across every channel where agents operate. Whether customers interact through Live Chat on a website, SMS text messaging, Email, Voice calls, or directly through Copilot itself, all conversations appear in a single, chronological interface. This omnichannel visibility eliminates the fragmentation that typically occurs when different channels use separate systems, providing complete context and enabling seamless oversight regardless of how customers choose to communicate.
The conversation view displays complete interaction history with rich context including timestamps, channel indicators, user information, agent responses, skill invocations, and conversation outcomes. Users can filter and search conversations by date range, channel, agent, user, topic, or custom tags, enabling rapid identification of specific interactions or patterns. Real-time updates ensure that team members see conversations as they happen, enabling immediate intervention when needed. The interface supports multiple simultaneous conversation views, allowing team members to monitor several interactions concurrently or compare agent behavior across different scenarios.
Admin Mode: Debugging and Insight
Admin Mode represents Copilot's most sophisticated feature for understanding and improving agent behavior. When enabled, Admin Mode provides complete transparency into the agent's internal reasoning, retrieval processes, and decision-making. Team members can see exactly which knowledge chunks were retrieved from the vector store, how the agent interpreted the user's question, which skills were considered or invoked, and how the final response was formulated. This visibility transforms agent behavior from a "black box" into a transparent, debuggable system.
The Admin Mode interface displays retrieval results showing the specific content chunks that informed the agent's response, ranked by relevance score. Users can evaluate whether the right information was retrieved, identify gaps where relevant knowledge wasn't found, and spot cases where irrelevant content influenced the response. The reasoning trace shows the agent's step-by-step thought process, including how it interpreted ambiguous questions, why it chose certain skills over others, and how it structured its response. Skill invocation logs detail every capability used during the conversation, including API calls, database queries, webhook triggers, and function executions, complete with parameters, responses, and error messages.
Human-in-the-Loop Intervention
While agents handle most interactions autonomously, certain situations benefit from or require human judgment. Copilot enables seamless human takeover where team members can step into live conversations, taking control from the agent while maintaining complete context. The takeover process is transparent to users, who experience a smooth transition from AI to human assistance without disruption. Once the human team member resolves the situation, they can return control to the agent or conclude the conversation.
The platform provides intelligent escalation triggers that automatically alert team members when human intervention may be beneficial. These triggers can be configured based on various criteria including specific keywords or phrases (like "speak to a human"), sentiment analysis indicating frustration or anger, conversation duration exceeding thresholds, repeated failed attempts to resolve an issue, or agent confidence scores falling below acceptable levels. When triggers activate, designated team members receive notifications through their preferred channels—in-app alerts, email, SMS, or integration with communication platforms like Slack.
Testing and Simulation
Before deploying agents to production or after making changes to training, instructions, or skills, Copilot's simulation capabilities enable comprehensive testing in a safe environment. Team members can run test conversations that exercise agent capabilities without affecting real customers or production data. Simulations can follow predefined test scripts that cover common scenarios, edge cases, and known problem areas, or can be free-form explorations where testers ask questions and evaluate responses organically.
The simulation environment provides complete control over test conditions including user context (simulating different customer types, account statuses, or permission levels), channel selection (testing behavior across Live Chat, SMS, Email, or Voice), skill availability (enabling or disabling specific capabilities), and knowledge configuration (testing with different training pack versions). Results are captured with detailed logging of all interactions, retrieval operations, skill invocations, and outcomes, enabling systematic evaluation and comparison between test runs.
Key Benefits
Quality Assurance and Risk Mitigation
Deploying AI agents without oversight creates significant risks including inaccurate information reaching customers, inappropriate responses to sensitive situations, compliance violations, brand damage from tone or messaging issues, and missed opportunities where human judgment could close deals or resolve complex issues. These risks are particularly acute in regulated industries, high-value transactions, or customer-facing roles where errors have immediate business impact.
Copilot mitigates these risks through comprehensive quality assurance capabilities. Continuous monitoring enables early detection of issues before they affect large numbers of customers. Human review of flagged conversations catches problems that automated systems might miss. Testing and simulation validate agent behavior before production deployment, reducing the likelihood of surprises. Feedback collection creates systematic improvement processes that address root causes rather than symptoms. The combination of proactive testing, real-time monitoring, and reactive intervention creates multiple layers of protection that dramatically reduce risk while maintaining the efficiency benefits of automation.
Accelerated Learning and Improvement
Traditional AI systems improve slowly through retraining cycles that require collecting data, identifying issues, updating training materials, retraining models, and redeploying—a process that can take weeks or months. Copilot enables continuous learning where improvements happen in near real-time based on human feedback. When team members rate responses, flag issues, provide corrections, or demonstrate better approaches through takeover interactions, this feedback immediately informs agent improvement.
The platform captures rich feedback data including response ratings (thumbs up/down or numeric scores), specific issue identification (inaccurate, incomplete, inappropriate tone, wrong skill used), suggested improvements (better phrasings, additional information, different approaches), and demonstrated expertise (how humans handled situations when they took over). This feedback is analyzed to identify patterns—common issues that affect multiple conversations, knowledge gaps that cause repeated failures, skills that aren't working as intended, or training materials that need updating. The insights drive systematic improvements to agent knowledge, instructions, skills, and configurations, creating a virtuous cycle of continuous enhancement.
Scaling Human Expertise
The fundamental value proposition of Copilot is enabling organizations to scale human expertise far beyond what would be possible through hiring alone. A single expert can directly handle perhaps 20-30 customer interactions per day. That same expert using Copilot to oversee AI agents can influence hundreds or thousands of interactions per day by providing guidance, feedback, and occasional intervention while agents handle routine aspects autonomously.
This scaling effect manifests in multiple ways. Knowledge scaling occurs when expert insights are captured in training materials that inform all agent responses, multiplying the impact of expertise across unlimited interactions. Quality scaling happens when experts review samples of agent conversations and provide feedback that improves all future interactions. Intervention scaling enables experts to focus their time on complex, high-value situations while agents handle routine inquiries. Training scaling allows experts to test and refine agent behavior through simulation, ensuring quality before production deployment. The cumulative effect is a 10x or greater increase in the impact of human expertise without proportional increases in headcount.
Operational Visibility and Analytics
Copilot provides unprecedented visibility into customer interactions, agent performance, and operational patterns. The unified conversation view creates a complete record of all customer engagements, enabling analysis that was previously impossible when interactions were fragmented across multiple systems. Team members can identify trends in customer questions, common pain points, frequently requested features, competitive mentions, and emerging issues before they become widespread problems.
The platform's analytics capabilities track key performance indicators including conversation volume and trends, average handling time, resolution rates, customer satisfaction scores, escalation frequency, skill utilization patterns, and agent accuracy metrics. These insights inform strategic decisions about product development, customer experience improvements, staffing requirements, and agent optimization priorities. The ability to drill down from aggregate metrics to individual conversations enables root cause analysis when issues are identified, connecting symptoms to specific causes and enabling targeted improvements.
Key Features
Intuitive Chat Interface
Copilot's user interface is designed for accessibility and ease of use, modeled after familiar chat applications like ChatGPT or Slack. Team members require minimal training to become productive, as the interface follows conventions they already understand. The chat window displays conversations chronologically with clear visual distinction between user messages, agent responses, and system notifications. Rich formatting support enables display of text, links, images, tables, and structured data within conversations.
The interface supports multiple conversation tabs, allowing team members to monitor or engage with several interactions simultaneously. Quick actions provide one-click access to common operations like taking over a conversation, flagging an issue, rating a response, or viewing conversation history. Search and filter capabilities enable rapid location of specific conversations or patterns. Keyboard shortcuts accelerate common workflows for power users. The responsive design works seamlessly across desktop, tablet, and mobile devices, enabling oversight and intervention from anywhere.
Comprehensive Feedback System
The feedback system in Copilot captures both quantitative and qualitative input that drives agent improvement. Simple rating mechanisms (thumbs up/down or 1-5 stars) provide quick feedback on response quality without disrupting workflow. Detailed feedback forms enable team members to specify issue types, provide corrective information, suggest improvements, and add contextual notes. Tagging capabilities allow categorization of feedback by topic, issue type, severity, or custom dimensions.
The system supports feedback at multiple levels including individual response rating (was this specific answer good or bad), conversation rating (was the overall interaction successful), skill rating (did this capability work as intended), and knowledge rating (was the retrieved information accurate and relevant). Aggregated feedback analytics identify patterns and priorities, showing which issues are most common, which knowledge areas need improvement, and which skills require attention. Feedback loops ensure that submitted input is reviewed, addressed, and tracked to resolution, closing the improvement cycle.
Real-Time Monitoring Dashboard
The monitoring dashboard provides at-a-glance visibility into agent activity, performance, and health. Real-time metrics display current conversation volume, active conversations, average response time, escalation rate, and satisfaction scores. Trend charts show how these metrics evolve over time, enabling identification of patterns, anomalies, or degradation. Alert panels highlight conversations requiring attention based on escalation triggers, extended duration, negative sentiment, or other configurable criteria.
The dashboard supports customization where team members can configure which metrics are displayed, set alert thresholds, define time ranges, and create custom views for different roles or responsibilities. Drill-down capabilities enable clicking on any metric to view underlying conversations or detailed data. Export functionality allows dashboard data to be extracted for reporting, analysis, or integration with business intelligence tools. The dashboard updates in real-time, ensuring that team members always have current information for decision-making and intervention.
Conversation History and Search
Copilot maintains complete, searchable history of all conversations across all channels and agents. The history function enables team members to review past interactions for quality assurance, training, troubleshooting, or compliance purposes. Advanced search capabilities support finding conversations based on keywords, phrases, user information, date ranges, channels, agents, topics, outcomes, ratings, or custom tags. Search results display relevant excerpts with highlighting, enabling rapid evaluation of matches.
Conversation detail views show complete interaction history including all messages, system events, skill invocations, retrieval operations, and metadata. Timeline visualization displays conversation flow with clear indication of agent responses, user messages, human interventions, and significant events. Export capabilities allow conversations to be extracted in various formats (text, JSON, PDF) for analysis, reporting, or record-keeping. Bulk operations enable actions on multiple conversations simultaneously, such as tagging, rating, or exporting sets of related interactions.
Simulation and Test Environment
The simulation environment provides a safe space for testing agent behavior without affecting production systems or real customers. Team members can create test scenarios that exercise specific capabilities, edge cases, or known problem areas. Scenario templates provide starting points for common test types including FAQ coverage (testing whether agents can answer common questions), skill validation (verifying that capabilities work as intended), edge case handling (testing unusual or problematic situations), and regression testing (ensuring that changes don't break existing functionality).
Simulation runs capture complete interaction logs including all messages, retrieval operations, skill invocations, and outcomes. Comparison tools enable side-by-side evaluation of different agent configurations, training versions, or instruction sets, showing how changes affect behavior. Automated test suites can be configured to run on schedules or triggered by changes, providing continuous validation of agent quality. Test results are tracked over time, enabling trend analysis that shows whether agent performance is improving, stable, or degrading.
Integration with raia Ecosystem
Copilot seamlessly integrates with other applications in the raia cX platform, creating a unified workflow for agent development, deployment, and optimization. Integration with raia Command enables direct access to agent configuration, allowing team members to view or modify settings, instructions, skills, and security controls without switching applications. Connection to raia Connect provides visibility into knowledge sources, enabling team members to identify gaps, request updates, or add new training materials based on conversation insights.
Integration with raia Chat ensures that conversations occurring through web chat widgets are immediately visible in Copilot with complete context and control capabilities. Connection to raia Control enables oversight of campaign-driven conversations, allowing team members to monitor outbound engagement and intervene when needed. The ecosystem integration eliminates silos, reduces context switching, and creates a seamless experience where insights from Copilot inform improvements across all platform components.
Role-Based Access and Permissions
Copilot implements granular access control that ensures team members have appropriate permissions based on their roles and responsibilities. Administrators can define roles such as Viewer (read-only access to conversations), Responder (can take over conversations and provide feedback), Tester (can run simulations and tests), Manager (can configure settings and view analytics), or Administrator (full access to all capabilities). Permissions can be granted at various levels including agent-specific (access to certain agents only), channel-specific (access to certain communication channels), or organization-wide.
The permission system supports complex scenarios including time-based access (permissions that apply only during certain hours), conditional access (permissions that depend on conversation attributes), and delegation (temporary permission grants for specific situations). Audit logging tracks all access and actions, providing complete visibility into who viewed or modified what and when. This comprehensive access control ensures that sensitive information is protected, compliance requirements are met, and team members have exactly the capabilities they need without unnecessary exposure.
Getting Started with raia Copilot
Organizations beginning with Copilot typically start by defining oversight and intervention strategies that balance automation efficiency with quality assurance. Common starting approaches include sampling review where team members regularly review random samples of agent conversations to assess quality, trigger-based monitoring where specific conditions automatically alert team members to conversations requiring attention, and scheduled testing where simulation runs validate agent behavior on regular intervals.
The typical implementation workflow involves configuring access and permissions for team members based on their roles, setting up monitoring dashboards with relevant metrics and alerts, defining escalation triggers that identify conversations requiring human attention, creating test scenarios that cover important use cases and edge cases, and establishing feedback workflows that ensure insights lead to improvements. Training for team members focuses on using the interface, understanding Admin Mode, providing effective feedback, and knowing when to intervene versus letting agents handle situations autonomously.
Success with Copilot comes from treating it as an ongoing operational tool rather than a one-time setup. Organizations that achieve the greatest value establish regular review rhythms, systematically analyze feedback and conversation patterns, continuously refine escalation triggers and monitoring criteria, expand test coverage as new scenarios emerge, and use insights from Copilot to drive improvements across the entire agent ecosystem. The platform's accessibility and real-time capabilities make this continuous oversight sustainable without requiring dedicated teams or excessive time investment.
Conclusion
raia Copilot represents a fundamental shift in how organizations approach AI agent deployment and management. Rather than treating agents as autonomous systems that either work or don't, Copilot enables a collaborative model where AI efficiency combines with human expertise to achieve results that neither could accomplish alone. The platform's comprehensive monitoring, testing, feedback, and intervention capabilities create the quality assurance and continuous improvement infrastructure necessary for successful, scalable AI agent deployments.
Organizations using raia Copilot report dramatically improved agent accuracy, faster identification and resolution of issues, enhanced customer satisfaction through timely human intervention, and accelerated agent improvement through systematic feedback. The platform's combination of accessibility, power, and integration creates a sustainable approach to AI-human collaboration that scales with organizational needs. As AI agents become increasingly central to customer engagement and business operations, Copilot provides the oversight and improvement infrastructure necessary to ensure those agents deliver consistent, high-quality results while continuously evolving to meet changing needs.
Last updated