Small-N Detector Agreement Beats “Which Tool Catches More”

Campus chats often ask which AI-writing detector “catches more.” A more useful question for a local review team is how often two detectors agree on the same matched passages—and what disagreement means for process fairness.

This note is literacy, not a product bake-off. It does not promise scores after revision.

Why “more flags” is the wrong leaderboard

Detectors are classifiers over qualifying continuous prose under their own display rules. A higher flag rate can mean a stricter threshold, a different denominator, or a different model—not a truer identity label. Cross-tool percentages are not a common scale.

What a small-N check can still teach

1. Keep source conditions known (human draft / locked LLM draft / documented rewrite).
2. Run ≥2 detectors as raters on the same texts.
3. Report agreement rate and Cohen’s kappa—not a pass/bypass rate.
4. Disclose prompts, models, settings, and conflicts of interest.

Local samples train judgment. They do not replace institutional policy.

How to talk about disagreement

• “These two reports diverge on the same paragraph—what did each denominator include?”
• “We are measuring agreement, not crowning a harsher tool.”
• “Open the full PDF, not the messenger crop.”

If revision is already decided, scope it to highlighted continuous prose on the matching upload. Importing a Turnitin or iThenticate AI report to rewrite only AI-flagged passages—preserving citations and formatting—is the narrow job HumanPen is built for. It does not predict the next digit.