Skip to main content
Two reviewers can apply the same scorecard to the same ticket and land eight points apart. The gap is a disagreement about what the rubric means, and it stays invisible until two people score the same conversation. A calibration assigns reviewers the same tickets and reports the spread. The output is a set of deltas: how far each reviewer sat from the AI, and how far they sat from each other. Those deltas are the session agenda.
Calibrations are switched on under Settings > Features, in the Quality assurance group. If you cannot find them, that toggle is off, or quality assurance itself is. They also have no sidebar item: you reach them from Home > Tasks and from direct links.

​
Before you start

  • Have a published scorecard, and tickets already evaluated in the period you want to sample. Sampling draws on tickets evaluated inside the span you choose, not tickets merely created in it. A quiet fortnight of QA yields a thin sample.
  • Line up at least two reviewers who are members of the organization. One reviewer is legitimate for spot-checking your AI, but reviewer-to-reviewer agreement needs two.
  • Decide the question first. “Are we consistent on refunds?” leads to a very different sample from “is our scoring too harsh at the bottom of the range?”

​
Build the sample

  1. Go to Home > Tasks and find the Calibrations card. Click the + button in its header, or use New > Calibration in the tasks list.
  2. If your organization evaluates back-office work, you are asked what to calibrate first: Tickets or Work items. Pick one and continue. When work item QA is off, the question is skipped.
  3. Choose how the tickets are chosen. Randomise samples from a filtered pool; Specific tickets lets you name ticket IDs yourself, which is the right choice when you want everyone arguing about the same known-hard cases.
  4. Name it. Leave it blank and you get today’s date plus “Calibration”, which is fine for a recurring session and useless in a list of twenty.
  5. Set the Due date. It defaults to a week out and it is what reviewers are chased against.
  6. Set the Ticket span, the date range the sample is drawn from.
  7. Under Filter tickets by, narrow the pool by team, channel, or Agent QA score level, and by brand if your help desk has brands. Sampling only Low-band tickets asks whether your reviewers agree about failure, which is where rubrics usually diverge.
  8. Add your Reviewers. Everyone you add gets their own assignments.
  9. Choose the sampling method. Fixed takes a Ticket split, the number of tickets given to each reviewer. Percentage takes a share of the eligible pool instead.
  10. Click Create calibration.
New calibration dialog with a name, a due date a week out, a ticket span of the last 30 days, a Score level Low filter, three reviewers, Fixed sampling, and a ticket split of five
For the deltas to mean anything, reviewers have to score the same tickets. Open Advanced settings and switch on Require multiple evaluations per ticket, then set Evaluations per ticket to how many reviewers should independently score each one. It cannot exceed the number of reviewers you added. Rotate evaluator pairs varies who overlaps with whom across the sample, so you do not learn only about one pair.

​
What reviewers do

Reviewers reach their work from Home > Tasks, or from Start review on the calibration itself. Each ticket opens the normal evaluation workspace with a counter (“3 of 8 tickets”) and their own completion percentage in the header. They score the scorecard as usual, click Submit evaluation, and move to the next one with the navigation arrows. Nothing about the calibration changes how an evaluation is produced. Each reviewer’s submission is a real manual evaluation on that ticket: Perform a manual evaluation covers the scoring itself, and calibration scores show up in reporting like any other manual score.

​
Read the results

Open the calibration and you land on the Calibration tab: the assignment count, the span, an overall percentage complete, the filters you chose echoed back, and a card per reviewer with their own progress. Chase from here: a calibration where one reviewer has done nothing produces no reviewer-to-reviewer numbers at all. Calibration tab showing 12 assignments, a date range, 83% complete, a Medium score level filter, and three reviewers at 75%, 100%, and 75% complete The Evaluations tab has one row per ticket, with:
  • AI QA score — what automatic QA said.
  • A column per reviewer, holding that person’s score for the ticket.
  • AI-to-evaluators delta — the average absolute gap between the AI score and each human score. Lower is closer agreement.
  • Evaluator-to-evaluator delta — the average absolute gap between every pair of human scores. This column only appears once two or more reviewers have work in the calibration.
  • Status — whether that ticket’s reviews are done.
Evaluations tab with a row per ticket, the AI score beside each reviewer's score, AI-to-evaluators and evaluator-to-evaluator delta columns, review status, and a summary strip of reviewers, tickets, assignments, and average scores Both delta columns sort. Sorting descending on evaluator-to-evaluator delta puts the tickets your team scored most differently at the top. Along the bottom, a summary strip counts reviewers, tickets, and assignments, and shows Avg AI Score, Avg Human Score, and completion rate. A persistent gap between those two averages means your AI is calibrated differently from your people overall; a large per-ticket delta means your people disagree case by case.

​
Take one ticket apart

Click a row and you get the ticket beside a comparison panel: Alignment, Deltas as a count of misaligned checks out of the total, the Evaluators involved, and the Agents on the ticket. Work check by check to turn “we were twelve points apart” into “we disagree about whether verifying the account counts when the customer volunteered it”. Mark individual checks as correct or incorrect as you go. Run the meeting from this view, and record whatever you settle somewhere durable: usually a sharpened criterion description in the scorecard, or an organization-wide rule in Set score bands and the contest window.

​
Adjust a calibration in flight

The Evaluations tab has a search box and a Filter by review status for finding your way around a large sample. You can add more tickets to a running calibration, and select rows to export them or remove them. Removal only applies to tickets nobody has scored yet, which keeps completed evaluations from disappearing out of your numbers.