If you ever read a claim of an AI outperforming a human, dig into the performance data to check out the human benchmark. The mediocrity of the human is often more salient than the competence of the AI. The AI has an easy job. Here’s an example from a paper by Ayers and friends (2023). They sampled 195 question and response exchanges on Reddit’s r/AskDocs and entered those same questions into ChatGPT 3.5. Responses by the Reddit physicians and ChatGPT were then independently rated for qual...
