[Editor's Note: This is the third in a three-part series on legal education in the age of generative AI. You can read Part 1 and Part 2 here.]
Generative AI has already started to fundamentally reshape the legal ecosystem—even if the impact on the actual delivery of legal services remains limited.
First and foremost, enterprises are actively exploring the use of GenAI in their products and across all their internal functions. Given the myriad legal and compliance considerations bound up in novel uses of novel technology—privacy, data security, intellectual property, etc.—this net new activity puts additional pressure on legal and compliance departments to support the business in making better decisions, faster.
Second, consistent with this exploration, expectations have shifted dramatically. Whether the expectations shift is reasonable merits debate. But that debate should be grounded in the reality that recalibrated expectations are already manifesting as ever more onerous budgetary and headcount constraints for in-house departments—constraints that cascade to their law firms—despite the attendant increase in demand.
That heightened automation expectations might result in considerable reliance on proven but underutilized traditional approaches to workflow and knowledge management—rather than GenAI technology that remains early days—is almost besides the point. The expectation shift is real, and already in motion, regardless of whether GenAI itself fully lives up to the hype.
Still, if you peer beyond the hyperbole and define success in reasonable terms, GenAI appears to be on the path to transforming entry-level legal tasks. Take, for example, the humble deposition summary.
Ford Motor Co. won the 2024 Legalweek Leaders in Tech Law Awards for Best Use of AI and Most Innovative Legal Ops, as well as an individual award for Darth Vaughn, managing director, Legal Ops+ and Litigation Counsel, as the In-House Innovator of the Year. Among the myriad AI projects the Ford team pursued was testing the performance of GenAI models on deposition summaries.
“Our organization handles a number of cases that involve taking depositions. In the past, we’ve paid external attorneys to read through and summarize these deposition transcripts,” Vaughn explained. “To test the effectiveness and quality of AI-only generated deposition summaries, we asked our in-house lawyers to compare them with summaries produced by external attorneys. The AI-only generated summaries were notably faster and more accurate. However, the overall consistency and quality (with respect to specific legal context) varied depending on how well our attorneys crafted the AI prompts. Ultimately, combining AI technology with the expert review of a technologically skilled legal professional resulted in the highest quality work, achieved much more quickly and at a lower cost.”
Ford’s findings are consistent with the academic literature. In Summarization is (Almost) Dead, researchers concluded, “[Large language model-generated] summaries are significantly preferred by the human evaluators, which also demonstrate higher factuality.” The last two words, “higher factuality,” are of particular interest because the legal profession, understandably, appears terrified of GenAI’s propensity to hallucinate. Yet, in Summarization is (Almost) Dead, the researchers also found, “LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations” than human-generated summaries.
One more time: GenAI hallucinated less than the humans.
Less, not zero. Hallucinations remain a genuine concern, important to address. But the propensity to hallucinate is not in and of itself a valid objection to the use of GenAI without answering the corollary question, “Relative to what?”
We’ve seen this movie before in e-discovery. After academic research debunked the myth of human review as the ‘gold standard,’ courts have consistently permitted the use of technology-assisted review despite empirical evidence that, while superior to standard manual review, TAR is still not 100% accurate in terms of precision or recall. Rio Tinto v. Vale, 306 F.R.D. 125, 127 (S.D.N.Y. 2015) (“the case law has developed to the point that it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.”).
Better is better, even if better is not perfect. And the evidence will only continue to mount that imperfect AI is less imperfect at certain tasks than imperfect humans, especially as the application layer matures.
For example, similar to the Ford protocol, the researchers behind Better Call GPT, Comparing Large Language Models Against Lawyers used senior lawyers’ judgment as a “ground truth” to benchmark both junior lawyer and legal contract reviewer efficacy at locating legal issues in contracts. The study concluded, “advanced models match or exceed human accuracy in determining legal issues. In speed, LLMs complete reviews in mere seconds, eclipsing the hours required by their human counterparts. Cost-wise, LLMs operate at a fraction of the price, offering a staggering 99.97% reduction in cost over traditional methods.”
Eventually, the evidence will become overwhelming. And the implications are fairly profound. This is true even if you believe, as the author does, that GenAI will only increase the need for well-trained lawyers augmented by well-crafted systems that serve as force multipliers of the lawyers’ expert judgment. There is much to be unpacked around service models, business models, and training—the latter being the topic of the moment.
This week, Legaltech News ran the first two articles in this series on educational initiatives set against the backdrop of GenAI. First, we covered the ABA Business Law Sectional M&A Committee National Invitational, a mock negotiation competition in which law students substantially upskilled in less than a month of extracurricular training. Second, we covered Goodwin Procter’s new associate training program, which was similarly built around asynchronous learning, instructor-led simulations, and competence-based practice with real-time feedback.
The types of practical, interactive, and experiential training underpinning both initiatives is certainly part of the solution.
As Joe Colucci, founder of the competence-based training company Procertas observed, “We’ve issued over 42,000 micro-certifications to current and future legal professionals despite their pre-training testing scores being abysmal. For example, only 20% of law students can pass a pre-training knowledge check on the basics of using Microsoft Word for legal tasks. But they upskill rapidly when training is mandated. These are smart, talented, motivated people who should not be blamed for not automatically possessing skills they’ve never been taught.”
Michael Bloom, founder of Praktio—which provides interactive training as part of the Goodwin package—concurred. “Praktio also aims to take the blame out of training by encouraging learners to ‘make mistakes’ and learn from them. It’s no secret—problem-based training situated in realistic contexts is more effective than, say, sitting in a lecture. It improves the learner’s motivation and ability to transfer lessons into practice,” he explained. “For example, a senior attorney might nudge a junior to jump into Praktio’s mock data room before doing real-life diligence. Whether the junior will be reviewing an AI-built first pass or doing the initial work themselves, on-demand, simulated ‘at bats’ equip juniors to do their work well, while saving senior nonbillable time (on training juniors and/or redoing junior work).”
The need for practical training will not only be driven by AI, but also extends to AI itself. Hotshot, a video-based learning platform which also underpins Goodwin’s new associate training program as well as the ABA M&A Committee tournament, has recently released a new series of courses that teach lawyers and other legal professionals about AI and its impact on law and practice. As part of this initiative, they interviewed AI leaders and experts from law firms, law schools, corporate legal departments (including Ford), and legal tech companies to be able to share perspectives from across the legal ecosystem.
Hotshot co-founder Ian Nelson explained, “We’ve had overwhelming interest in a series of courses on AI for lawyers, and it was amazing to see how many leaders in the space wanted to work with us to create this content for the legal industry. No matter where an organization is on its journey of adopting AI, everyone at this point has to understand the technology and related issues for lawyers.”
Development takes time, attention, and structure. But, done properly, it takes less than we often think. The traditional method of grinding through lower-level work is wildly inefficient. Hotshot’s training was also utilized to prepare students to participate in the ABA M&A Invitational in which, according to a room full of seasoned practitioners, the students performed at a level they would expect from a third- or fourth-year associate.
“Both the ABA event and the Goodwin program are terrific examples of experiential learning programs that still have the lawyers (or students) learning directly from the lawyers in their organizations for nuance and the unique ‘firm way’ of doing things, while letting the learners level-set and learn key concepts on their own so that the live training time is that much more valuable and interactive,” Nelson added. “Plus, let’s face it—delivering, and listening to, long lectures isn’t fun for anyone.”
Structured experiential training is essential. But, so too, is real-world experience. Here, the AI-enabled uptick in volume may help. There should be more opportunity. After all, it is not only in-house and Big Law that will utilize the new tools. Part of the expected increase in volume is plaintiffs’ lawyers, regulators, and consumers gaining access to “good enough” technology that will enable them to credibly maintain matters that were previously beyond their capacity.
“Neither the court system nor traditional ADR are sufficient to handle even our current dispute volumes. Disputants already need to move many high-volume matter types to more cost- and time-effective resolution methods. And volume will only increase, especially as plaintiffs and defendants alike are equipped with the next generation of tools,” said Rich Lee, founder of New Era ADR, winner of the 2024 Legalweek Leaders in Tech Law Award for New Law Company of the Year. “The legal industry—firms and clients—should treat higher-volume, lower-stakes, more-contained matters as an appropriate arena for younger lawyers to gain real-world experience. With proper supervision, of course.”
Pro bono is another avenue for young lawyers to obtain real-world experience. Many pro bono programs are already designed to provide excellent oversight and guidance, while affording young lawyers real-world opportunities to interact with clients, courts, opposing counsel, and administrative bodies. Kristen Sonday, the co-founder of Paladin, a pro bono management platform, underscored the importance of pro bono work for young lawyers. “Pro bono is a unique way to provide emerging lawyers tangible experiences in dealing with real-world clients, courts, and other legal parties, all with valuable training and supervision,” she explained. “While engaging in pro bono work should always be focused on serving the client in need, it’s important to also recognize it as a tool for professional and personal development.”
Better professional development is a NOW problem. While the road to product is long, it is not infinite. The application layer is maturing, as is the understanding of where GenAI can make a significant impact in terms of quality, leverage, speed, and cost. And even if the impact of this technological inflection point was overhyped, especially in the near term (because, of course, it was), the impact of the expectation inflection point could not be more real, nor more immediate.
Casey Flaherty is chief strategy officer at LexFusion, where he helps accelerate legal innovation. Flaherty is also a legaltech founder and has been a BigLaw litigator, in-house counsel, legal operations consultant, and law firm executive. He is an expert in the shifting dynamics of the legal market, legal technology, and the design of legal services to meet business needs at scale and pace.
Stay Informed, Stay Ahead
Discover the latest trends and insights in the legal industry. Learn more about Generative AI, Delivering Legal Services and Strategy and Scale in a Complex World.
Let’s Forge the Future Together
Interested in joining our roster of legal innovators or simply curious about the world of legal tech? Reach out! Our team is eager to connect, collaborate, and contribute to the ever-evolving legal landscape.
