Don't count on an AI chatbot to judge your illness or legal problem on its own; before seeing a doctor, going to court or spending big money, find the original source for what it cites and check it

Cost

Free. Each check takes a few minutes to half an hour more. Have the AI write out the law, guideline or paper it relies on, then open the original yourself and find that sentence. The habit to change is not acting on something just because it was stated with confidence.

What you get

First, medical care. In 2026 Nature Medicine published a randomized trial with 1298 ordinary people in the UK, with the protocol registered in advance. Each person was given two made-up cases and had to decide what the illness was and whether to go to the emergency room or watch at home. Some used an AI chatbot to help, with GPT-4o, Llama 3 and Command R+. The comparison group used their usual methods, mostly searching online or looking at the UK National Health Service (NHS) website. When a complete case was handed straight to the AI, it recognized the relevant illness in 94.9% of cases and got where to go for care right in 56.3%. Ordinary people using the same AI recognized the relevant illness in less than 34.5% and got where to go right in less than 44.2%. The comparison group's odds of recognizing the relevant illness were 1.76 times those of the AI users (95% CI 1.45–2.13, the plausible range). The odds of recognizing a serious illness with danger signs were 1.57 times (1.28–1.92). On judging where to go for care, the two sides were about the same. The authors went through the conversation records and found the problems at both ends. At one end the user did not give all the symptoms, and at the other the AI mentioned the right illness but the user did not take it up. Next, law. A U.S. research team in 2024 used public models such as ChatGPT 4 to ask about U.S. federal court precedents. ChatGPT 4 had fabrication in 58% of its answers, and Llama 2 as high as 88%. Fabrication means content that does not match legal facts, which the paper calls “hallucination.” The model cannot tell which of its own sentences are made up. When a user's question carries a wrong legal premise, it often accepts it as is. The same team in 2025 tested specialized legal AI tools used by lawyers. These tools advertise that they do not make things up, but 17% to 33% of their answers had fabrication. It looks like a good deal because AI is available at any time, free of charge, and speaks as confidently as a doctor or lawyer

Notes

Dispute. A 2023 study drew 195 patient questions from a U.S. forum and had medical staff score the answers without seeing the source. 78.6% of the ratings judged ChatGPT's answer better than the doctor's. But that compared how well the answers were written, and did not look at whether the person who asked ended up judging correctly. The trial used 2024 models, and newer models may be stronger. Even so, the same trial found that the AI's own test score did not predict how well ordinary people did using it. Why AI makes things up was explained in 2025 by OpenAI researchers in a paper that has not been peer reviewed. The way models are trained and scored rewards “guessing an answer,” and answering “I don't know” earns no points, so it speaks with confidence even when it is unsure. The way to check is to have it write out the name and article number of the law, or the title of the guideline or paper, then open the original on an official website and look for that sentence; if you can't find it, treat it as never said. When you ask, don't write your own conclusion into the question; ask “what does the law say about this situation,” not “am I sure to win in this situation.” For signs such as a sudden crooked mouth, weakness on one side, or pressing chest pain, don't ask the AI first; call 120 (the emergency medical line) directly. See §13.3 (crooked mouth, weak arm on one side) and §13.7 (pressing chest pain). If you can't afford a lawyer for a lawsuit, see §7.3 (apply for legal aid). The benefit is rated small because there are only intermediate measures like “whether the illness was recognized” and no numbers on fewer deaths. The beneficiaries are you and your family.

Evidence grade

A : Specific numbers from meta-analyses, large cohorts or randomized trials; for law and policy, the statute or official text.

Sources

1. Bean AM, Payne RE, Parsons G, et al. (2026). Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 32(2), 609–615. https://doi.org/10.1038/s41591-025-04074-y
2. Dahl M, Magesh V, Suzgun M, Ho DE (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1), 64–93. https://doi.org/10.1093/jla/laae003
3. Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22(2), 216–242. https://doi.org/10.1111/jels.12413
4. Kalai AT, Nachum O, Vempala SS, Zhang E (2025). Why Language Models Hallucinate. arXiv:2509.04664（预印本，备注里那项）. https://arxiv.org/abs/2509.04664
5. 反方：Ayers JW, Poliak A, Dredze M, et al. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589. https://doi.org/10.1001/jamainternmed.2023.1838

Translated from the Chinese original. Original book, section 6, tip 31. Content may be out of date and rules differ by place. When you actually deal with an agency, a doctor or a court, follow what local officials, doctors and lawyers tell you. Disclaimer