Skip to content
Module 6Human-in-the-Loop & The Feedback Loop

The Human-in-the-Loop Decision

The most important decision you make as an Agent User is knowing when to let the agent handle it, and when to step in.

Seamless Takeover

When you click Take Over where supported, you step into the conversation. Whether the user sees a handoff indication depends on the surface and channel; verify the experience in your deployment. Once you have resolved the complex issue, you have two choices:

  • Close the conversation: Use when the interaction is fully complete and no further follow-up is needed.
  • Hand back to agent: Use when the user has more routine questions the AI can handle, freeing you up to assist someone else.

The Four Types of Feedback

Training an AI Agent is not a one-time event; it is a continuous process. Your feedback is the primary engine that drives the agent's improvement.

Whenever you are reviewing a conversation in Copilot, you may see thumb icons beneath agent replies, depending on the message and your role. Clicking these icons opens the feedback modal:

TypeWhat It DoesWhen to Use
Thumbs UpSignals the answer was accurate, complete, and had the right toneRecord positive feedback to help reviewers identify good responses
Thumbs DownFlags the response as incorrect for Admin reviewWhen the answer is wrong, incomplete, or has the wrong tone
Written CommentA plain-text explanation of why the answer was good or badE.g., "The refund window is 30 days, not 14"
Suggested ReplacementYou write the exact response the agent should have givenWhen you can show the right answer for this situation. It gives the reviewer a clear target to fix and re-test against

From Feedback to Training

Feedback in Copilot does not just disappear into a void. It enters a structured continuous improvement cycle:

  1. You submit feedback on a response
  2. Feedback is immediately logged to the conversation history
  3. An Administrator opens the Feedback Skill's Feedbacks list (date, vote, question, answer, comment, conversation)
  4. Feedback that does not apply is removed with Ignore or Delete
  5. When a fix is needed, the Administrator corrects the agent's Agent Instructions or Documents on the Train tab in Launch Pad
  6. If the Feedback Skill is configured to generate and upload a feedback document, that document can also add to the agent's knowledge. The Period setting chooses which feedback goes in; it is not a schedule
  7. Ask the original question again, plus a few variations, in Copilot to check the fix. Nothing guarantees the change holds forever, so keep checking
Feedback configuration and period selector.
  1. Highlight 1: Configuration holds the settings. Feedbacks lists the feedback itself.
  2. Highlight 2: Period: how far back feedback is included (here, Last 30 days).
Generation, automatic-upload and save controls.
  1. Highlight 1: Generation Mode: Auto generates the feedback document for you.
  2. Highlight 2: Auto Upload Document: when on, the document is uploaded to the AI right after it is generated.
  3. Highlight 3: Click Save.

The Scoring Skill

While you provide manual feedback on specific messages, raia also measures quality automatically using the Scoring Skill.

When enabled, the Scoring Skill uses a scoring prompt your admin can customize. Use the supported 1 to 10 range and describe what low, middle and high scores mean for your task.

Example review threshold: For a 1–10 rubric, your team might prioritize conversations scoring below 6. Choose a threshold that fits your rubric and verify that it identifies conversations needing review.

Scoring prompt and fixed 1–10 range notice.
  1. Highlight 1: Prompt for Conversation Scoring: what each score means.
  2. Highlight 2: “Use scoring from 1 to 10. Values outside this range are not supported.”
Summary prompt and save control.
  1. Highlight 1: Prompt for Conversation Summary.
  2. Highlight 2: Click Save.

To see the results, open the agent in Launch Pad and go to the Report tab. Avg AI Score shows the average score out of 10 and how many conversations were scored, split into High, Mid and Low. Open the relevant conversation and compare its transcript and Sources with the expected answer. A score reflects your scoring prompt, not proof that an answer was accurate.

Further down the Report tab: scores, feedback and tokens. Numbers are from a demo agent.
  1. Highlight 1: Avg AI Score: the average Scoring result (1 to 10) and how many conversations were scored, split into High, Mid and Low.
  2. Highlight 2: Human Feedback: thumbs-up and thumbs-down votes.
  3. Highlight 3: Token Usage for the selected period.

The Continuous Improvement Loop

An AI agent is never "finished." The most successful teams treat agent management as a continuous loop:

  1. Test / Monitor in Copilot — Watch how the agent handles real scenarios
  2. Spot an issue — Identify a bad answer or a missing piece of knowledge
  3. Fix the source — Do not just correct the agent in the chat. In Launch Pad, open the agent's Train tab and update its Agent Instructions or Documents
  4. Re-test — Ask the same question again in Copilot to verify the fix worked
Launch Pad → Train → Agent Instructions: fix the source here, then save.
  1. Highlight 1: Current Instructions: edit the text here.
  2. Highlight 2: Save
  3. Highlight 3: Save version keeps this text as a new version.

Key Principle: A thumbs-down does NOT make the agent instantly stop giving that answer. It flags the response for review. The answer changes when someone corrects the agent's Instructions or Documents (or a configured feedback document is uploaded), and you confirm it by re-testing.

Previous lessonNext lesson