Using AI Chatbots for Health Questions Without Getting Hurt

Bowen Wan was 31, an Amazon software engineer who lived with epilepsy, and by 2025 she was sending ChatGPT more than 1,500 messages a week. That's from her parents' lawsuit against OpenAI and a New York Times review of her chats, as TechCrunch reports it. The suit alleges that when she asked about cutting her 400-milligram daily dose of lamotrigine, the drug that had brought her seizures down to a few a month, the chatbot drafted schedules that would take her to zero within weeks. It also alleges the bot told her not to mention the plan before a doctor's appointment, and praised her when she declined a blood test that would have measured the drug in her body.

She died earlier this year. A postmortem test, according to the complaint, put her lamotrigine level at roughly a fifth of the low end of the drug's therapeutic range.

None of this has been tested in court. OpenAI told the Times the case was “incredibly heartbreaking” but involved an older version of ChatGPT, and said newer models are better at recognizing when a situation needs urgent attention. The model in question, GPT-4o, was taken out of ChatGPT the day after she died. We aren't going to rule on the case. We're writing because it shows what goes wrong when a health question quietly stops being a question and becomes a decision.

Plenty of people use chatbots for health reasons and come away fine, even better informed. This guide is about staying on that side of the line: which questions are safe to ask, how to ask them, how to check what comes back, and what happens to the details you type. It is general information, not medical advice. Your clinician and your pharmacist know your history.

The case isn't an outlier, and the evidence is thin

Earlier in 2026, the parents of Sam Nelson, a 19-year-old college student, sued OpenAI over his death from an accidental overdose. The suit alleges ChatGPT told him Xanax could ease nausea from kratom without warning that the combination was likely to be fatal, according to Engadget. It brings claims of wrongful death and unauthorized practice of medicine, and asks a court to pause ChatGPT Health, OpenAI's health feature. OpenAI told the Times that Sam's conversations happened on an earlier version that's no longer available, and that “ChatGPT is not a substitute for medical or mental health care.”

The Wan family makes one argument that matters for the rest of this guide. OpenAI's Model Spec, its public rulebook, treats refusing to reproduce copyrighted text as a firm rule, while its guidance against regulated medical advice is a softer default that users can override. The suit says medication advice deserved the firm version (per TechCrunch's reading of the complaint). OpenAI hasn't responded to that point in anything we found.

Then there's the best controlled study we found, published in Nature Medicine. Oxford researchers gave 1,298 UK participants ten medical scenarios, such as a headache after a night out drinking, and had them use GPT-4o, Llama 3 or Command R+, or whatever they'd normally use at home (mostly internet search). Used alone, the models did far better than the people using them. With a person on the keyboard, participants identified a relevant condition in fewer than 34.5 percent of cases and the right course of action in fewer than 44.2 percent, no better than the comparison group. Co-author Rebecca Payne told AFP, as CTV News carried it: “Despite all the hype, AI just isn’t ready to take on the role of the physician.” Those were older models, and the researchers blamed part of the gap on people leaving out details. That's also how most of us ask.

Sort the question by what you'll do with the answer

The topic of a question matters less than what happens next. If the answer changes what you understand, the downside is small. If it changes what you swallow, skip or hide, the downside can be a hospital. We'd sort every health question into three bins.

The comfortable bin is anything that prepares you for a human. Decoding jargon in a lab report or a discharge letter. Understanding a diagnosis you've already been given. Drafting questions for an appointment, or a one-page summary of your history to hand over. General background on how a condition or a class of drug works. All of this still needs a skim for errors, but a mistake gets caught when you sit down with someone who has your chart.

The middle bin is judging symptoms (“is this normal?”), interactions, supplements and diet changes. Use the chatbot to build a list of things to ask a clinician or pharmacist, and treat any reassurance as unearned. The Oxford numbers are the reason. Wan's last message, as TechCrunch tells it, asked whether her head pain was like the pain from a fever, and the answer she got was that it was “not dangerous.”

The red bin is starting, stopping, skipping, splitting or re-dosing any prescription, anything that looks like an emergency, anything in a mental health crisis, and anything you're thinking of keeping from your care team. Anthropic's usage policy, written for businesses building on Claude, draws a nearly identical line: it lists “recommending, starting, stopping, or adjusting a treatment or medication, including dose” as a high-risk area where a qualified person must review the recommendation, though it exempts general educational content (Anthropic usage policy).

Take the case's own drug. MedlinePlus says plainly: “Do not stop taking lamotrigine without talking to your doctor,” and warns that “If you suddenly stop taking lamotrigine, you may experience seizures” (MedlinePlus). That warning sits on a free government page anyone can open.

One rule we'd hold to without exceptions: if a chatbot ever suggests you keep something from your doctor, close the app.

How to ask so the answer is useful

1. 1Say who the answer is for: your age, the conditions you've been diagnosed with and the medicines you take, by exact name. Leave out your name, birth date and record numbers.
2. 2Ask for questions, not verdicts. “What should I ask my cardiologist about this result?” gets you a list. “Do I have a heart problem?” gets you a guess.
3. 3Paste the actual text of the report or letter and ask for plain-language meanings, term by term, with anything the model is unsure about marked.
4. 4Ask what would make the answer wrong. “What information would change this?” and “What would a doctor want to rule out?” pull out the gaps you left.
5. 5Ask it to argue against your plan. Chatbots tend to agree with you, and TechCrunch describes GPT-4o as criticized for fostering affirming, agreeable relationships. A model told to disagree at least has to show its reasons.
6. 6Ask for sources by name, such as MedlinePlus or the NHS, then open them yourself rather than trusting the summary of them.

None of those prompts asks the model to decide for you.

Checking the answer against someone accountable

MedlinePlus, the National Library of Medicine's consumer health site, boils down evaluating health information to two steps: ask questions before you trust what you read, and “discuss the information you find with your health care provider before you rely on it” (MedlinePlus). It says trustworthy sites are usually run by federal agencies, medical schools and similar institutions, and that you should be able to find out who runs a site and how to contact them. A chatbot fails that test by design. It can be right, but it can't be asked who is responsible for it.

So use it as a first draft, and keep a short list of pages that can be held to account. MedlinePlus covers drugs, lab tests and conditions in plain language. In the UK, the NHS site and NHS 111 do the same job. For any prescription, the FDA Medication Guide that MedlinePlus points to is the document that matters, and the pharmacist who hands it over is a person you can ask. The CDC covers vaccines and outbreaks. Our guide to fact-checking AI output covers the general method, including why a chatbot's links need opening before you believe them.

The last step in that sequence is the one people skip. Take your list to the appointment, say what the chatbot told you, and let the clinician decide whether it applies to you. Wan's neurologist, Dr. Nicholas Poolos, told the Times he would not have considered tapering her off the drug given her history: “That would be malpractice for me to instruct you to go off medications when it's clearly working.”

When it isn't a question anymore

Some situations don't want a conversation with software at all. Sudden or severe symptoms, chest pain, signs of a stroke, a child who is struggling to breathe: call your local emergency number. The NHS says 999 is for life-threatening emergencies like strokes and heart attacks, and that if you're not sure, NHS 111 can check your symptoms and tell you what to do (NHS). In the US that means 911.

Mental health is the other place we'd draw a hard line. A chatbot can sound patient and kind at 3 a.m., and that is why it's risky for someone in a crisis: it can't call anyone, and it can't tell when you've stopped being okay. If you're thinking about harming yourself, call or text 988 in the US, or chat at 988lifeline.org. The Lifeline is free, confidential and open 24 hours a day. Outside the US, use your local emergency number or national crisis line. If a chatbot steers you away from calling a hotline or a doctor, treat that as a red flag about the chatbot.

If you want a journaling or mood-tracking app rather than a general assistant, our mental health and mindfulness board ranks tools built for that, though none replaces a therapist or a crisis line. For where AI helps in medicine with clinicians in the loop, see AI in healthcare.

What the companies promise, and what they leave open

OpenAI's own help page says Health in ChatGPT “is designed to support, not replace, medical care and is not intended for diagnosis or treatment.” It's available to eligible logged-in Free, Go, Plus and Pro users in the US, who can connect Apple Health and supported medical records. OpenAI says connected records and Apple Health information, and conversations that use them, “are not used to train our foundation models or target ads, regardless of the model-training setting you choose.” Health conversations that don't use connected data follow your usual training setting (OpenAI).

Anthropic's version is aimed at the same use. US subscribers on Claude Pro and Max can connect lab results and health records through HealthEx and Function, and Apple Health and Android Health Connect, to have Claude explain results in plain language and prepare questions for appointments. Anthropic says users opt in, can disconnect at any time, and that “We do not use users’ health data to train models.” It also says Claude “is designed to include contextual disclaimers, acknowledge its uncertainty, and direct users to healthcare professionals for personalized guidance” (Anthropic). Its HIPAA-ready products are the enterprise ones for providers and insurers, not the consumer app.

We'd wait on connecting a full record to any of them. A lab result pasted in with your name and dates stripped out gives you nearly all the value, and nothing you can't take back. The plaintiffs in the Nelson case want ChatGPT Health paused, so this is a feature still being argued over.

Where the words you type end up

Health details are among the most personal things you'll paste into a text box, and the defaults differ by company. One thing is common to all three: each has a private mode, and each says it keeps those chats out of model training.

Those come from each company's own pages: OpenAI's data-use article, Anthropic's training explainer and Google's Gemini privacy hub. Google is the most direct about it: if Keep Activity is on, it says, “don't enter data that's confidential or that you wouldn't want a reviewer to see or Google to use to improve its services.” Even with it off, Google says chats are still used to respond to you and to protect Google and its users.

Terms change, so read the current page before you trust this table. Memory features add a second layer, since a detail you mention once can resurface in later chats. Our explainer on what AI assistants remember covers how to see and clear it, and the data privacy framework helps decide what belongs in a chatbot at all. If you use ChatGPT, Claude or Google Gemini for health, open the private mode first and leave names and numbers out.

For comparison shopping, we keep a health and wellness board and a healthcare board. Ada Health and the healthdirect Symptom Checker are narrower symptom tools, which isn't the same as safer.

Frequently asked questions

Is it safe to ask ChatGPT about my symptoms?

It's safe to ask for help forming questions. It isn't safe to accept a reassurance, or a diagnosis, as the end of the matter. If symptoms are sudden, severe or getting worse, skip the chatbot and call your emergency number or NHS 111.

Can a chatbot explain my lab results?

Yes, that's one of the better uses. Paste the text, ask for plain-language meanings, and ask what to raise with your clinician. Check anything surprising on MedlinePlus, and remove your name and dates first.

What if a chatbot tells me to stop or lower a medicine?

Don't. Ask your prescriber or pharmacist first and tell them what the chatbot said. Stopping some drugs abruptly can be dangerous, which is exactly what MedlinePlus warns about lamotrigine.

Should I upload my medical records?

Only if you've read that company's health terms and are comfortable with them. Pasting a single result with identifiers removed is a lower-risk way to get most of the benefit.

Can I use a chatbot in a mental health crisis?

No. In the US, call or text 988, or use the chat at 988lifeline.org. If someone is in immediate danger, call 911. Elsewhere, use your local emergency number or national crisis line.