Law Firm Innovation Index

Phase 1, Version 1.0 – Methodology Below

Goal

Create an initial measure of law firm innovation.

Scope

This Phase 1, Version 1.0 of the Law Firm Innovation Index measures law firm innovation based on Google Advanced searches for indicators of innovation on law firm websites. This is not a ranking of law firms. As discussed below, this approach has weaknesses, including that it may not capture the most critical aspects of innovation, such as whether a firm has an innovative culture in which everyone from the bottom to top of the organization is engaged in innovation and efforts to continuously provide greater value to clients. That said, given the paucity of information currently available, this research ought to be a helpful addition to the discussion. This is intended to be a “minimum viable product.” I look forward to obtaining feedback and iterating through later versions to improve this product, as discussed in the Introduction. The searches were conducted for all Am Law 200 (American Lawyer), Global 100 (American Lawyer), and Canadian top 30 law firms (Lexpert); a total of 260 law firms.

Introduction & Methodology

As discussed in the Overview, this endeavor responds to Legal Services Corporation President Jim Sandman’s call to action that, rather than rank law firms on revenue and profit, we ought to rank and assess them on their use of technology. The purpose of doing this is to accelerate innovation and technology adoption that will lead to legal-service delivery improvements and increased access to legal services for everyone.

A handful of innovative law firms are well known because they’ve won innovation awards or they’ve been written about. We also learn about innovation in the legal industry through surveys of law firms and legal departments, including by Altman Weil, ALM Legal Intelligence, Thomson Reuters, BTI Consulting Group, and a few other organizations. But I’ve been unable to identify any systematic measure of law firm innovation.

Given the paucity of information, attempting to determine what law firms say on their websites about their own innovation efforts is a helpful starting point. In this Phase 1, Version 1.0 release, rather than manually review each law firm’s website, we have used Google Advanced Search to search each of the 260 law firm websites for certain words and phrases related to innovation in legal-service delivery.

For each law firm, we recorded the total number of hits for each category. The number of hits is an indicator for whether the law firm is innovating in that category–albeit perhaps a weak indicator, particularly in certain categories. These weaknesses are discussed further below. Nonetheless, the fact that a law firm talks about, for example, alternative fees on its website, is a proxy for a measure of the law firm’s willingness to entertain alternative fee agreements. Thus, even in light of the weaknesses of this approach, this information provides some insight into whether and how law firms are innovating.

Searches of 260 Law Firm Websites

We searched 260 law firm websites:

Am Law 200 – American Lawyer
Canadian Top 30 – Lexpert
Global 100 – American Lawyer

The firms that are categorized in our results as “Global 100” only include those firms not already categorized as part of the Am Law 200 or Canadian 30. For this reason we have placed an * next to Global 100 in the Tableau vizzes. Keep this in mind when comparing the Global 100 to other categories of law firms. Additionally, some firms have merged or been acquired since the latest release of the lists above.

Innovation Categories and Search Terms

For this iteration, we performed searches for the following ten categories of innovations, using the search terms identified below. When selecting these categories and determining the search terms within them, I considered not only the terms and innovations that law firms are using, but also the innovations and terms that law firms should be using, based on my research and engagement with leaders in law firms, legal departments, legal startups, alternative legal service providers, and others in the legal industry.

Alternative Fees – “alternative fee” OR “alternative fees” OR “AFA” OR “AFAs” OR “fixed fee” OR “fixed fees” OR “value based billing” OR “value based pricing”
Project Management – “project management” OR “project manager” OR “project managers”
Process Improvement and Innovation Framework – “lean thinking” OR “six sigma” OR “process improvement” OR “design thinking”
Knowledge Management – “knowledge management” OR “knowledge engineering”
Automation Basics – “expert system” OR “expert systems” OR “document automation” OR “document assembly” OR “process automation”
Data Analytics – “data analytics” OR “predictive analytics” OR “decision tree” OR “decision trees” OR “data driven”
Artificial Intelligence – “machine learning” OR “deep learning” OR “artificial intelligence”
Legal Operations – “Legal operations” OR “collaborative disaggregation”
Proactive Law – “proactive law” OR “preventive law” OR “preventative law” OR “promotive law”
Blockchain – “Blockchain” OR “smart contract” OR “smart contracts” OR “computable contract” OR “computable contracts”

The last category, blockchain, has a stronger relationship to substantive legal issues than improvements in legal-service delivery. Nevertheless, it has been included also as a measure of how law firms are responding to emerging legal issues.

Conducting the Google Advanced Searches

In a Google Sheet, I automated the construction of each of the ten search queries for each law firm (a total of 2,630 queries) so that each could be run with one click. This ensured that each of the ten searches were run identically across all of the law firms. Upon submitting each Google Advanced search, we recorded the number of results reported in the first page returned by Google. (A topic that I address briefly below and plan to discuss in future posts is that this first number returned by Google is an estimate of the number of hits.)

We conducted the searches for the Am Law 200 firms on August 1, 2017; for the Global 100 firms and the Canadian top 30, on August 2, 2017. These initial searches were run by the same person, from the same computer, in the same browser, from the same physical location. To audit our information, we again ran the searches of the Global 100 firms and Canadian top 30 on August 15, 2017 and the Am Law 200 firms on August 16 and 17, 2017. We compared the results to identify possible errors in the first round of the searches. The Tableau vizzes above are based on the data recorded from the searches on August 1 and 2.

Weaknesses/Criticisms of Google Searches

The number of times a word or phrase is mentioned on a law firm’s website is a proxy for law firm innovation–although for some categories it might be a weak or poor proxy. But again, the extent to which a law firm talks about an innovation category on its website is meaningful information and worth assessing for discussion purposes.
There may be a significant gap between what a law firm says it is doing and what it actually does. This can cut both ways. For example, a law firm may say it’s using project management, when it’s doing little more than creating budgets before initiating projects. That is a positive step, but it falls far short of implementing project management in a way that will result in learning and improvement and lead to the creation of a collaborative, innovative culture. On the other hand, a firm may have a thriving culture of innovation, in which each person involved in the delivery of services, including each administrator, technologist, paralegal, data scientist, business development professional, junior attorney, and senior attorney, is encouraged to innovate, identify opportunities to better serve clients, and work with clients to co-create value, but they may not be discussing this on their website, at least not with the terms we’ve used for our searches. While our searches would likely pick up some indicators of the innovation likely to be the result of this innovative law firm culture, the searches would not detect many of the underlying characteristics of a truly innovative organization. (I attempted to outline a path for legal-services innovation in: Legal-Services Innovation: A Framework and Roadmap for Leveraging Technology.)
Some law firms host blogs inside of their main website, while many others host their blogs on a separate site under a different domain name. Similarly, some law firms store a large volume of articles and other content on their main website. Law firms with blogs, articles, etc. on their main website have that much more content against which the searches have been run. For law firms that host this content outside of their primary website and talk about these innovation topics, their number of hits has been underreported. (This is not a commentary on where blogs and other content should be located. It is only an explanation of what is excluded with Google Advanced Searches that are directed at the domain of the law firm’s main website.)
The search terms may need to be adjusted for jurisdictions outside of the U.S., including international law firms websites for which English is not the primary language.
Type I Errors (false positives)
1. Some of the searches generate false positives; that is, hits that do not indicate law firm innovation. False positives include:
  1. Hits on content describing what clients are doing that is not related to law firm service. (For example, an article that mentions a client doing project management for a construction project.)
  2. If a firm practice group name or publication contains a search term, it might be repeated across numerous pages, such as in repeated menus of links, which will result in false positives–perhaps many, many false positives if the term appears across all of the firm’s pages, including outdated pages that are still indexed by Google. (Some of these hits bringing up a “404 page” with no content–the page no longer exists.)
Type II Errors (false negatives)
1. Some of the searches generate false negatives; that is, the search terms fail to generate a hit for pages on a law firm’s website that provide evidence of innovation. False negatives include:
  1. We have not used the search term “technology,” for example. Some law firms might describe their use of technology to improve legal service delivery without using any of the search terms above. In that case, our searches would not pick up the hit. But terms like “technology” and “innovation” would generate many false positives, so those terms have not been included.
  2. Law firms that use additional domain names other than the domain for their main website (including for subsidiaries doing project management, for example) will have only the content on the main website searched for hits.
We recorded the number of results reported in the first page returned by Google Advanced Search after submitting the search query. This first number returned by Google is an estimate of the number of hits. (For information about Google search, visit How Google Search Works.) The actual number could be higher or lower. Nevertheless, because the same searches were used for all law firms the resulting information, even if not exact in absolute values, is useful as a relative measure.
Legal-service delivery innovation must include not only cost reduction, efficiency, and improved quality, but also measurably improved substantive legal outcomes. Most industry innovation awards do not distinguish between innovation in legal-service delivery (e.g., building a client-facing expert system) and substantive legal innovations (e.g., creating a new organizational vehicle to complete the acquisition of a company). Legal-service delivery innovation, to flourish, must foster organizational change and embrace the possibilities for greater value creation through improved service delivery, substantive legal innovations, and improved substantive legal outcomes. For now, our searches mostly identify indicators that law firms are embracing foundational disciplines and tools.
There is some double counting across categories.
Big Law firms may not be the ones to bring innovation to the legal industry. If smaller firms are more likely to innovate, then their exclusion from these searches is a problem. The solution right now is that the Catalog of Law Firm Innovations captures some of these firms. All firms may submit proposed innovations for inclusion on the Catalog.

Individual Firms – Charts for “All Hits” versus “Percentile Order per Category”

The chart “Individual Firms – All Hits – NOT a Ranking” is a stacked bar chart of the total number of hits for each category for each law firm. Please review the methodology above to place this information in its proper context. You will notice, as discussed above, that certain search categories returned thousands of hits when run across certain law firm’s websites. In some cases this is due to the way that Google calculates the estimated number of hits. In other cases, it is because a certain search term is reproduced across a law firm’s website on every page. (For example, if a law firm has a blockchain practice group, the term “blockchain” might appear on every page in a sidebar menu.) Given these and other caveats, rather than treat the number of hits as an absolute measure, it seems best to treat it as a relative measure (i.e., comparing each individual result to all other results).

The chart “Individual Firms – Percentile Order – Not a Ranking” is a stacked bar chart of the percentile into which each law firm falls for each category. For example, if the law firm’s number of hits for a particular category falls in the middle compared to the other firms (i.e., about half of the firms had fewer hits, about half of the firms had more hits), it is at the 50th percentile, or 0.50. For each category, the law firms are placed in order based on the number of hits for that category and assigned a score of 0.0 for the lowest to 1.0 for the highest. If a law firm was at the top in all ten categories, it would score a 1.0 in each category and a 10.0 overall. If it was in the middle in each of the ten categories, it would score a 0.50 in each category and a 5.0 overall.