The hidden cost of that is that you might end up shopping for the verdict that you wanted. Your own paragraph gets flagged so that checker looks bad, but the AI draft gets flagged somewhere else so that checker looks good. All of this is understandable, but I would have to admit that I would not have chosen one based on the success it had over that particular pair.
I would have to push back against the notion of choosing a detector as a second opinion.
Before giving it the job, I’d want to see how it did on examples that it wasn’t involved in choosing.
Otherwise, you’d have picked the referee based on which calls they had already seen. I think the free detectors worth testing against would be candidates for a small blind comparison, not tools that I would already have recommended to the team. The access details are useful, narrowing the candidates, but I would not be giving any of them a preferred spot to start with. A manageable approach would be to have a coworker put together a small sample of non-confidential excerpts, keeping the authorship labels away until the end. You could have a mix of voices and subjects instead of excerpts from the same document, and have whoever is running the checks note the results in advance. Part of the batch could be used to pick the candidate, which could be tested against the untouched remainder, without changing your selection rules. That would be the part that matters to me, if I was choosing based on something besides the examples it had already seen, that my choice still looked good when it got fresh examples, not the ones I used to pick it.
I would have to decide what constituted a mistake beforehand as well.
If the concern is accusing a teammate of using AI, a false flag of human writing would have to have a lot of pull in your decision. If you’re just curious about an AI draft, missing it might be disappointing, but it’s not going to change anything. I think I would treat those differently and not consider them equal ‘got this many right’ points. I would put any drafts that were a mix of human and AI writing in a separate bucket unless I had agreed on an answer to look for.
I don’t think you need to turn this into a research project.
You can set a stopping point and accept ‘none earned a place in our process’ as a result.
Your onboarding example gives you a reason to doubt the results, but it doesn’t owe you a better choice. I would be far more comfortable announcing to the team that none of the candidates had passed an initial test than presenting the one they felt most confident about as if it were a clear winner.