AI in Legal Advice: Limitations and Concerns
Key Takeaways:
- Turning to a bot for legal advice could hurt your ERISA claim.
- AI legal advice may contain hallucinations and errors, such as fabricated case citations.
- Conversations with public AI chatbots are not confidential, potentially giving other parties access to details about your case.
- AI cannot provide legal advice, and it can’t understand the nuance of individual cases.
Let’s be honest: More and more people are asking AI tools like ChatGPT or Google’s Gemini how to appeal a denied disability claim before they ever think of calling a lawyer.
I get it. AI systems are fast. They’re available 24/7 and don’t require you to talk with a live person about the difficult details of your health. And, of course, AI is free or low-cost.
But when your financial stability and health are on the line, you need to know the full truth: AI is not a substitute for experienced legal counsel. In fact, trusting AI outputs could do more harm than good.
I’m not here to bash technology. There is much to gain from the efficiency of evolving AI technology, and I am certainly not advocating AI abstinence. I am here, however, to explain AI legal advice limitations, and why having a compassionate, strategic, and knowledgeable attorney is critical for your long-term disability benefits claim. As someone who has spent over two decades representing clients in ERISA disability cases, I can tell you that AI is not cut out for this work.
The Potential Risks of Using AI for Legal Advice
If you type into ChatGPT:
“How do I appeal a denied ERISA disability claim?”
You’ll probably get something like this:
“Request your claim file. Review the denial letter. Submit additional medical evidence. Write an appeal within 180 days.” You might even get general “advice” about including medical records and statements in support of your appeal. However, even AI knows its own limitations. A common bot disclaimer goes something like this:
“Generative AI features are not intended for professional advice. Do not use generative AI features to seek or provide legal, medical, financial, or other kinds of professional advice or any opinions, judgments, or recommendations without conducting your own independent consultation or research. Generative AI features cannot replace advice provided by a qualified professional and do not form any such relationship (e.g., attorney-client relationship).”
Even tech giants know that the disclaimer is necessary because AI advice is not strategic, not confidential, not detailed, and, in fact, is dangerously simplistic.
For example, AI does not, and cannot, analyze whether:
- Your plan’s language requires a de novo or abuse of discretion standard of review in court.
- The facts of your case meet the definition of disability under your plan language.
- The insurer has disclosed all required information in its claim file.
- There is evidence of biased claims handling in your file.
- The insurer has calculated offsets accurately in your case.
- Evidence in the claim file needs to be rebutted in an appeal.
AI also can’t assess the likelihood of settlement, or estimate what your settlement amount might be.
The bottom line: AI gives you non-confidential general advice. Springer Ayeni gives you a legal strategy based on decades of experience handling ERISA long-term disability cases.
AI ‘Hallucinations’ and Fake Data: A Dangerous Trend
Can AI attempt to give advice on legal matters? Yes. But AI tools are known for hallucinating legal citations—that is, making up case law that sounds real but isn’t. I’ve read up on examples of how AI will generate completely fictitious case references and quotes from decisions that don’t exist. This isn’t just sloppy, it’s risky.
First, if you submit an appeal letter to an insurer or judge citing fake cases, you undermine your credibility. Second, insisting that there are cases in your favor when you reach out to an attorney, when those cases are entirely fictitious (unbeknownst to you), will make an attorney hesitant to represent you. Most importantly, you won’t even know when AI has done you dirty with a fictitious citation, because it often sounds right, even when AI cites facts from different cases and presents them as originating from the same case.
Legal writing and research requires advocacy, precision, and ethics. AI is still a baby in the world of law, and cannot be trusted to do actual legal research or give you reliable citations.
AI Is NOT Confidential, and that Can Put You at Risk
There’s no such thing as an AI lawyer. AI doesn’t form an attorney-client relationship, as defined by tech User Guidelines. That means:
- There is no legal confidentiality. Whatever you input may be stored by AI models, analyzed, or even discoverable later.
- You might waive privilege by sharing facts about your medical condition, employment, or insurer. Moreover, AI remembers your input from search to search, so even if you don’t share all the facts in one session, it may know a lot more about you, your condition, your family, your history, your dreams, and your goals from all of the data you have input in previous searches.
- There is no duty of loyalty. AI doesn’t protect your privacy or interests—it just completes a task. For example, if you share a link to a ChatGPT conversation, your conversations could end up appearing in Google search results.
As an attorney, I never input clients’ confidential information into AI platforms. Doing so risks breaches of ethical duties and the security of the case. Some legal professionals are misusing AI and unknowingly exposing client data. That’s not how I work.
If I use AI at all, it’s only for routine tasks — not for case strategy, legal analysis, or communication.
Your trust is sacred. And your privacy is non-negotiable. AI doesn’t have to follow standards of professional conduct and will not protect you and advocate for you like a good attorney will.
A Real Appeal Tells the Full Story—Not Just the Medical One
An AI-generated appeal might focus only on lab results and doctor notes. But I know that winning a disability appeal requires telling your whole story, not just summarizing medical visits. It also requires understanding the law and how it applies to the facts of your case, and being able to spot and rebut flaws in the insurer’s claims-handling and medical reviews.
I listen for and translate the truths that often don’t make it into a medical file, like how your fatigue crashes your productivity by noon, or how your brain fog makes multi-step tasks impossible—not just at work but in your everyday life—or how your anxiety prevents you from completing your tasks efficiently, or even how you have good days and bad days, requiring you to rest for days if you have over-exerted yourself.
I draw out the details of your daily life, your work history, and your limitations, and I explain them clearly and persuasively to the insurance company. This is something a chatbot can’t do. It takes time, training, empathy, experience, and sound judgment.
I’m not just reviewing records. I’m building your narrative, filling gaps in the file, rebutting arguments, and fighting for your future.
AI Can’t Show Up for You. I Will.
Some pundits say artificial intelligence will replace lawyers. I do think that technology can make lawyers more efficient at administrative tasks that take away from time spent on actual cases. However, the best lawyers, those who listen, advocate, strategize, work as a team with their client, and ultimately are successful in cases, can never be replaced by a bot. Technology will evolve, but compassion, reputation, and results still matter.
When you work with me, you’re not getting a script. You’re getting:
- Over two decades of ERISA disability experience
- Strategic, circuit-specific legal knowledge
- A reputation for success and tenacity among colleagues, insurers, and opposing counsel
- A teammate who sees you as more than a claim number
- A fierce advocate who will not ignore you or reduce you to a piece of data that will influence future actions
You deserve someone who understands not just the law, but the weight of what’s at stake. You deserve someone who knows how to get results and treats you with dignity along the way.
Don’t Trust a Bot with Your Livelihood
AI has its place—for planning trips, researching recipes, or learning a simple skill—but not as your attorney. When your disability benefits, your financial survival, and your mental and physical health are on the line, you need a real human advocate on your side.
Before you go down a rabbit hole of chatbot answers, talk to someone who’s been through this hundreds of times, won’t hallucinate case law, will respect your privacy, and knows how to win an appeal, not just write about it.
Contact Springer Ayeni today. Let’s protect what matters most—your income, your dignity, your future.
FAQ
How accurate is AI legal advice?
AI cannot provide generalized legal advice because it is not a state-barred attorney, and there’s always the risk that it may produce inaccurate information. It’s not a substitute for competent representation from an experienced attorney.
Who is responsible if AI gives bad legal advice?
AI has no obligation of professional responsibility. If it gives bad advice, and you act on it, you are solely responsible for the results.
How do law firms use AI?
In the legal profession, AI can be useful for automating routine tasks, helping with non-lawyer tasks like graphic design, or forecasting business needs.
Read More