AI in Legal Advice: Limitations and Concerns
Key Takeaways:
- Turning to a bot for legal advice could hurt your ERISA claim.
- AI legal advice may contain hallucinations and errors, such as fabricated case citations.
- Conversations with public AI chatbots are not confidential, potentially giving other parties access to details about your case.
- AI cannot provide legal advice, and it can’t understand the nuance of individual cases.
Let’s be honest: More and more people are asking AI tools like ChatGPT or Google’s Gemini how to appeal a denied disability claim before they ever think of calling a lawyer.
I get it. AI systems are fast. They’re available 24/7 and don’t require you to talk with a live person about the difficult details of your health. And, of course, AI is free or low-cost.
But when your financial stability and health are on the line, you need to know the full truth: AI is not a substitute for experienced legal counsel. In fact, trusting AI outputs could do more harm than good.
I’m not here to bash technology. There is much to gain from the efficiency of evolving AI technology, and I am certainly not advocating AI abstinence. I am here, however, to explain AI legal advice limitations, and why having a compassionate, strategic, and knowledgeable attorney is critical for your long-term disability benefits claim. As someone who has spent over two decades representing clients in ERISA disability cases, I can tell you that AI is not cut out for this work.
The Potential Risks of Using AI for Legal Advice
If you type into ChatGPT:
“How do I appeal a denied ERISA disability claim?”
You’ll probably get something like this:
“Request your claim file. Review the denial letter. Submit additional medical evidence. Write an appeal within 180 days.” You might even get general “advice” about including medical records and statements in support of your appeal. However, even AI knows its own limitations. A common bot disclaimer goes something like this:
“Generative AI features are not intended for professional advice. Do not use generative AI features to seek or provide legal, medical, financial, or other kinds of professional advice or any opinions, judgments, or recommendations without conducting your own independent consultation or research. Generative AI features cannot replace advice provided by a qualified professional and do not form any such relationship (e.g., attorney-client relationship).”
Even tech giants know that the disclaimer is necessary because AI advice is not strategic, not confidential, not detailed, and, in fact, is dangerously simplistic.
For example, AI does not, and cannot, analyze whether:
- Your plan’s language requires a de novo or abuse of discretion standard of review in court.
- The facts of your case meet the definition of disability under your plan language.
- The insurer has disclosed all required information in its claim file.
- There is evidence of biased claims handling in your file.
- The insurer has calculated offsets accurately in your case.
- Evidence in the claim file needs to be rebutted in an appeal.
AI also can’t assess the likelihood of settlement, or estimate what your settlement amount might be.
The bottom line: AI gives you non-confidential general advice. Springer Ayeni gives you a legal strategy based on decades of experience handling ERISA long-term disability cases.
AI ‘Hallucinations’ and Fake Data: A Dangerous Trend
Can AI attempt to give advice on legal matters? Yes. But AI tools are known for hallucinating legal citations—that is, making up case law that sounds real but isn’t. I’ve read up on examples of how AI will generate completely fictitious case references and quotes from decisions that don’t exist. This isn’t just sloppy, it’s risky.
First, if you submit an appeal letter to an insurer or judge citing fake cases, you undermine your credibility. Second, insisting that there are cases in your favor when you reach out to an attorney, when those cases are entirely fictitious (unbeknownst to you), will make an attorney hesitant to represent you. Most importantly, you won’t even know when AI has done you dirty with a fictitious citation, because it often sounds right, even when AI cites facts from different cases and presents them as originating from the same case.
Legal writing and research requires advocacy, precision, and ethics. AI is still a baby in the world of law, and cannot be trusted to do actual legal research or give you reliable citations.
AI Is NOT Confidential, and that Can Put You at Risk
There’s no such thing as an AI lawyer. AI doesn’t form an attorney-client relationship, as defined by tech User Guidelines. That means:
- There is no legal confidentiality. Whatever you input may be stored by AI models, analyzed, or even discoverable later.
- You might waive privilege by sharing facts about your medical condition, employment, or insurer. Moreover, AI remembers your input from search to search, so even if you don’t share all the facts in one session, it may know a lot more about you, your condition, your family, your history, your dreams, and your goals from all of the data you have input in previous searches.
- There is no duty of loyalty. AI doesn’t protect your privacy or interests—it just completes a task. For example, if you share a link to a ChatGPT conversation, your conversations could end up appearing in Google search results.
As an attorney, I never input clients’ confidential information into AI platforms. Doing so risks breaches of ethical duties and the security of the case. Some legal professionals are misusing AI and unknowingly exposing client data. That’s not how I work.
If I use AI at all, it’s only for routine tasks — not for case strategy, legal analysis, or communication.
Your trust is sacred. And your privacy is non-negotiable. AI doesn’t have to follow standards of professional conduct and will not protect you and advocate for you like a good attorney will.
A Real Appeal Tells the Full Story—Not Just the Medical One
An AI-generated appeal might focus only on lab results and doctor notes. But I know that winning a disability appeal requires telling your whole story, not just summarizing medical visits. It also requires understanding the law and how it applies to the facts of your case, and being able to spot and rebut flaws in the insurer’s claims-handling and medical reviews.
I listen for and translate the truths that often don’t make it into a medical file, like how your fatigue crashes your productivity by noon, or how your brain fog makes multi-step tasks impossible—not just at work but in your everyday life—or how your anxiety prevents you from completing your tasks efficiently, or even how you have good days and bad days, requiring you to rest for days if you have over-exerted yourself.
I draw out the details of your daily life, your work history, and your limitations, and I explain them clearly and persuasively to the insurance company. This is something a chatbot can’t do. It takes time, training, empathy, experience, and sound judgment.
I’m not just reviewing records. I’m building your narrative, filling gaps in the file, rebutting arguments, and fighting for your future.
AI Can’t Show Up for You. I Will.
Some pundits say artificial intelligence will replace lawyers. I do think that technology can make lawyers more efficient at administrative tasks that take away from time spent on actual cases. However, the best lawyers, those who listen, advocate, strategize, work as a team with their client, and ultimately are successful in cases, can never be replaced by a bot. Technology will evolve, but compassion, reputation, and results still matter.
When you work with me, you’re not getting a script. You’re getting:
- Over two decades of ERISA disability experience
- Strategic, circuit-specific legal knowledge
- A reputation for success and tenacity among colleagues, insurers, and opposing counsel
- A teammate who sees you as more than a claim number
- A fierce advocate who will not ignore you or reduce you to a piece of data that will influence future actions
You deserve someone who understands not just the law, but the weight of what’s at stake. You deserve someone who knows how to get results and treats you with dignity along the way.
Don’t Trust a Bot with Your Livelihood
AI has its place—for planning trips, researching recipes, or learning a simple skill—but not as your attorney. When your disability benefits, your financial survival, and your mental and physical health are on the line, you need a real human advocate on your side.
Before you go down a rabbit hole of chatbot answers, talk to someone who’s been through this hundreds of times, won’t hallucinate case law, will respect your privacy, and knows how to win an appeal, not just write about it.
Contact Springer Ayeni today. Let’s protect what matters most—your income, your dignity, your future.
FAQ
How accurate is AI legal advice?
AI cannot provide generalized legal advice because it is not a state-barred attorney, and there’s always the risk that it may produce inaccurate information. It’s not a substitute for competent representation from an experienced attorney.
Who is responsible if AI gives bad legal advice?
AI has no obligation of professional responsibility. If it gives bad advice, and you act on it, you are solely responsible for the results.
How do law firms use AI?
In the legal profession, AI can be useful for automating routine tasks, helping with non-lawyer tasks like graphic design, or forecasting business needs.
Read More
A Conversation with Attorney Cassie Springer Ayeni
Q: When did you know you would pursue a career in the legal field?
CSA: As early as junior high or high school, I felt the need to advocate for others, to make a point to stand up for the kids who were being bullied or ostracized. I also realized around the same time that I truly enjoyed writing, reasoning, and debating. The law seemed a natural fit: I could use my skills to make a palpable difference in people’s lives. I had settled on a career in law by the time I was 16 and never looked back.
Q: Can you explain what types of cases an ERISA Attorney handles and why you chose this area of law?
CSA: I primarily help people with their employer-sponsored disability benefit claims. ERISA governs all private employer benefit plans, not just pension plans. Many employers provide disability benefit plans in addition to health and pension benefits. In fact, 65% of ERISA litigation is over denied disability benefit claims. The plans are often insured, and clients typically come to me after they stop working, apply for disability benefits to the insurance company, and are denied. I then step in to try to get their disability benefit income restored while they focus on their health.
Q: What is your approach or philosophy to winning or representing a case?
CSA: I lead with kindness. I want to understand truly what my clients have gone through medically, economically, and emotionally to get to this point in their lives, and how I can help. In representing a client, I am there for that person: I am responsive and listen with an open heart. I believe this the secret to winning a case as well – if I am doing my job right, I am painting a picture of the person and the case to help the court appreciate why my client is deserving of their disability benefits. I have the paintbrush and paint ready because I have spent months getting to know my clients and researching the law. And I also aim to be the best-prepared lawyer in the room with the most compelling brief … that helps too.
Q: If we interviewed all your past clients … what is “one” common word that comes up when they describe working with your law firm?
CSA: Compassion
Q: What are some of the most popular topics you are asked to lecture on?
CSA: I speak frequently on ERISA topics, especially in my current role as the co-chair of the ABA’s Employee Benefits Committee. I love presenting to newer attorneys about the fundamentals of ERISA litigation, because I am eager to energize attorneys about how engaging ERISA work is. ERISA is fun! The law is always evolving, providing constant intellectual stimulation, and practicing ERISA law is also a great way to help people in need. I also enjoy speaking to women about re-defining what it means to be a successful litigator, and how to challenge traditional law firm standards to achieve a better work-life balance.
Q: What advice would you give to young women who want to pursue a career as an Attorney?
CSA: Any woman who wants to pursue a career as an attorney should make sure that her potential work environment values diversity and inclusion, fosters a sense of belonging, and creates opportunities for women at every turn. If she is unable to find that environment, she should open her own law firm and create those opportunities herself.
Q: How do you maintain a work/life balance?
CSA: Being the owner of my firm allows me to make up the rules: I look to the best practices of progressive companies and pick policies that I believe are imperative to work/life balance for everyone. I figure that everything I need to raise four children and work full-time as an attorney is what everyone at my firm needs too. I offer unlimited PTO, great benefits, four months of fully paid maternity leave, the ability to bring children in to work as needed (in fact I have brought each of my babies into work until they needed another environment), the ability to work from home, and a practical approach to parenting … like closing the office on Halloween and Valentine’s Day so that we can all attend class parties without rushing to or from work. I, of course, avail myself of each of these policies, which is why I created them in the first place. That’s not to say that I don’t wake up early to work for a bit so that I can focus fully on my kids to take them to school, or grab a few hours on the weekend to pound out a brief, but family always comes first, for me and for everyone who works at Springer Ayeni.
Q: What’s one lesson you’ve learned in your career that you can share with our audience?
CSA: I have always learned to be myself. If you are authentic, you are compelling as an advocate and a counselor: you listen and communicate better when you are not spending energy on worrying about how you should come across. I also believe in the power of preparation. If you are committed to doing your best and preparing your utmost, then even a negative outcome cannot be met with regret, but a positive outcome is much more likely. And if you’re nervous before a court appearance or speaking engagement, just take a breath and realize that even in a worst-case scenario, you will survive and move forward … we all do.
Q: What are some of the challenges you feel women face today?
CSA: Looking at women lawyers only, there is much progress to be made. As Joan Williams at the UC Hastings Center for WorkLife Law has analyzed convincingly, women lawyers (and women in other fields too) are asked to “prove it,” then “prove it again.” In other words, women need to do the job before getting the promotion, whereas men are promoted based on potential. This is a major obstacle to women’s promotion and needs to be addressed systematically.
Five Things About Cassie Springer Ayeni
1. If you could talk to one famous person past or present, who would it be and why?
There are so many! But if I had to pick one it would be Harriet Tubman – her courage and vision even in the face of medical problems were and are awe-inspiring.
2. What’s your favorite holiday? Why?
Valentine’s Day! I don’t view it as just a romantic holiday, but as a day to express to those around you how much you care.
3. If you were a superhero, what would your special powers be?
Time-traveling. That would be magical!
4. What app can’t you live without?
Facebook – it has allowed me to make great connections with other lawyer moms – lawmas!
5. Favorite food to eat?
Strawberries. Now, then, forever, and always.
See the interview at https://bayareawomenmag.com/news/view/11665/Her_Firms_Achievements_-_Results_are_Measured_by_the_Satisfaction_of_Her_Clients_A_Conversation_with_Attorney_Cassie_Springer_Ayeni
Read More