Human review queue tutorial: low scores and missing labels
Route 5 example results into 3 reviews and save reports for handoff.
Use caseRoute failures, missing information, and weak signals into separate review queues
After obtaining labels and scores, keep missing labels, missing scores and low scores visible. This exercise uses 5 fictional records. At threshold 0.8, three enter human review and two remain for spot checks, with downloadable reports for handoff.
Author: JevLog editorial team · Updated 2026-10-02. This is a local routing exercise. It does not call Jev, calibrate model probabilities or measure accuracy.
Open the review queue generator · Download the input CSV · Download the expected JSON report · Download the expected review queue CSV
Establish what the score means
Section titled “Establish what the score means”The tool accepts stable IDs, suggested labels and numeric scores already normalized to 0–1. The fixture has id,text,label,confidence,source. Here, confidence is a teaching field, not a guarantee about the raw fields of any Jev response. Check a real API’s score definition and normalize through an explicit adapter. Do not silently convert 85 into 0.85 or describe any similarity score as a probability of correctness.
1. Load the fixture and run
Section titled “1. Load the fixture and run”Choose id, label and confidence, keep the practice threshold at 0.8, and run. Each file is limited to 5 MB and processed locally in your browser. No login is required.
2. Verify all five routes
Section titled “2. Verify all five routes”| ID | Suggested label | Score | Route | Reason |
|---|---|---|---|---|
| 1 | billing | 0.91 | Spot check | Meets practice threshold |
| 2 | technical | 0.60 | Human review | Below practice threshold |
| 3 | other | Empty | Human review | Missing score |
| 4 | billing | 0.85 | Spot check | Meets practice threshold |
| 5 | Empty | 0.96 | Human review | Missing suggested label |
Expect 5 records, 3 needing review, 2 for spot checks, 1 missing or invalid score and 1 missing suggested label. Routes are mutually exclusive: 3 + 2 = 5. Issue counters may overlap because one record can lack both a label and a score.
3. Check the threshold boundary
Section titled “3. Check the threshold boundary”A score below the threshold triggers low-score review. A score equal to it remains for spot checks. At 0.6, ID 2 no longer triggers low-score review: expect 2 review records and 3 spot checks. IDs 3 and 5 still need review. Lowering the threshold does not make missing labels, blank scores, nonnumeric values or scores outside 0–1 valid.
Both 0.8 and 0.6 are practice parameters. Choose business thresholds on human-labeled examples while considering missed errors, review cost and performance across categories. A lower review rate is not automatically better.
4. Download the right handoff files
Section titled “4. Download the right handoff files”review-report.csvorreview-report.jsoncontains the complete five-record routing report, reasons and available original text. JSON also contains the review queue.review-queue.csvcontains only IDs 2, 3 and 5, preserving every input column, includingsource, and appendingjevlog_review_reason. If that name exists already, a nonconflicting name with an_2suffix is used.
The visible table recognizes an original text column named text, feedback or message. Other text columns remain in the queue download but may not appear in the preview. Spreadsheet-view CSV exports prefix formula-like cells; retain JSON and the original input when checking exact values.
5. Record human decisions separately
Section titled “5. Record human decisions separately”Maintain a separate handoff table such as id, suggested_label, final_label, review_reason, reviewer, reviewed_at, rule_version. Preserve the original suggestion and text; record why each item is confirmed, corrected or deferred.
The current tool generates a queue. It does not provide online reviewer accounts, shared task assignment or automatic storage of human edits. Save those decisions in your own spreadsheet or system, join them back by stable ID and check unresolved records and total counts.
When the next rule version changes labels, use the run comparison tool and retain the version of the human reference decisions too.
Troubleshooting
Section titled “Troubleshooting”What if a request failed but a label and score still exist?
Section titled “What if a request failed but a label and score still exist?”The tool checks labels, scores and thresholds. It does not inspect a status or error column for technical failures. A real adapter should preserve a separate failure queue so stale labels or scores do not accidentally pass into spot checks.
What blocks queue generation?
Section titled “What blocks queue generation?”Blank or duplicate IDs and inconsistent row widths block reliable routing. Select distinct, existing ID, label and score columns. Use the CSV checker to inspect input structure. Invalid scores and missing labels stay visible as review records rather than being dropped.
Are queues retained after refresh?
Section titled “Are queues retained after refresh?”No. Download the complete report and queue first, then reload the input and threshold to reproduce the result. No login is needed. Spot checks are not execution approval; the page does not message customers, issue refunds or call external systems.
Complete the workflow
Section titled “Complete the workflow”CSV preparation → ticket-classification exercise → this review exercise → label-change comparison.
Fixtures and expected reports use the current local routing rules. They verify the queue-generation process, not Jev’s live performance, score calibration or business automation benefits.