Legal Futures recently reported that three-quarters of lawyers now believe that AI will increase productivity. Despite this, within the legal sector, the real world impact of this technology is still yet to materialise. As a company with one foot squarely in each of the "Legal" and "Technology" camps we decided to put our money where our mouth is and build "Lawrence": an AI paralegal designed to help the solicitors we work with save time on their legal work.
Testing Lawrence
In order to put Lawrence through his paces, we decided to have him complete SQE1, the first part of the new Solicitor Qualification Exam as run by the SRA. For those not familiar, the purpose of SQE1 is to assess a candidate's functioning legal knowledge. The questions themselves covered a wide range of topics from Property to Landlord & Tenant as well as Probate.
Surpassing our expectations, Lawrence not only passed the exam (exceeding the standard threshold of 55%-60%) but did so with flying colours, boasting an impressive 74% correct answers.
Of the questions answered incorrectly, we studied the responses to see if there were any common themes where Lawrence struggled. It was apparent that the questions he didn’t pass were more complicated involving complex chains of logic and wider context. Lawrence also struggled when two concepts shared similarities, confusing public nuisance versus private nuisance for example.
For those interested, we've shared all of Lawrence's answers including his rationale here.
Client Empathy
As all solicitors know, it's one thing to know the law but an entirely different thing to work with clients.
In a bid to understand Lawrence's effectiveness in client interactions, we presented Lawrence and a human solicitor with a mock legal file to assess tone and empathy. While Lawrence successfully gleaned necessary information from the client, the interaction remained notably transactional and was only half as lengthy as the human counterpart's conversation.
Client feedback was positive for both responses, yet Lawrence received criticism for a perceived lack of empathy. Notably, Lawrence failed to delve into the late relative's spending habits, missing the opportunity to uncover financial liabilities crucial for the solicitor's awareness.
Lawrence's Future at Lawhive
Whilst it's clear that AI will have a monumental impact on practicing the law, our experiments with Lawrence clearly demonstrate there's still no substitute for a qualified legal professional, especially in consumer law, when it comes to in replicating the depth of human connection crucial in emotionally charged legal situations.
While AI streamlines minor legal tasks and enhances efficiency, the indispensable value of human empathy in the solicitor-client experience remains irreplaceable. The potential we see for Lawrence (and his future iterations) is as a sort of "co-pilot", providing support to solicitors on monotonous or repetitive tasks, freeing them up to focus more on the work that they are uniquely placed to do.
Methodology
In order to test Lawrence, Lawhive had him answer a series of questions provided as part of SQE1 – The first part of the Solicitors Qualification Exam, used to assess a candidates functioning legal knowledge.
A total of 90 SQE1 Sample questions provided by the SRA via their website were used.
The SRA states “On these two sets of 45 questions, the pass mark would be likely to be in the range 55% - 60%.” Lawrence correctly answered 67 of 90 questions making his overall mark 74%.
We’ve provided a full list of Lawrence’s answers (including his rationale) via the Lawhive website in CSV format.