2
September 2012 / The New York County Lawyer
The “CAR” Keys to Defensibility By Dino E. Medina, Esq.
The purpose of pretrial discovery in the American legal system is to promote the full disclosure of facts relevant to a given dispute. To this end, the Federal Rules permit a litigating party to seek all information that is reasonably calculated to lead to the discovery of admissible evidence.1 There was a time when discovery requests would yield only boxes of documents from which to locate relevant material. The societal shift from paper documents to electronically stored information (ESI) over the past ten years, however, has led to sharp increases in discovery data volume, review time and cost. The following statistics illustrate this phenomenon: • Statisticians estimate that only five to seven percent of all data is created outside of a computer system.2 • 73 percent of all production costs are attributable to review of ESI.3 • The cost to review one gigabyte of ESI is between $5,000 and $30,000.4
As discovery in the legal industry continues to evolve in this electronic age, defensible methods to manage increased data volume, speed-up review time and curb review cost have become essential.
Technology’s latest solution to these discovery issues is computer-assisted review (CAR). CAR is a process by which analytics technology leverages the subject matter proficiency of human document reviewers to make conceptually-based coding decisions on all documents within a particular universe or “corpus.” Attorneys with sub-
ject matter expertise must first generate a “seed” or “training” set of documents (e.g., 500 docs), code it consistently using a specified protocol and run an analytics engine against the “seed” set to extrapolate its coding to the remaining documents in the corpus. Next, the analytics software generates a random sample from the corpus, which is then re-coded by the attorneys with subject matter expertise. Once re-coded, the analytics tool will again extrapolate the coding to the corpus. Third, an overturn report is run which, among other things, allows one to see the percentage of computer coding decisions overturned by the attorneys with subject matter expertise. Several iterations of this process are implemented until the computer’s coding is overturned by the attorneys with subject matter expertise at a rate of less than five percent. This process is proven to be significantly less costly and time consuming than traditional manual document review.5
Given all of the recent hype surrounding CAR and the judicial acceptance of this “black box” technology in the Da Silva Moore, Global Aerospace and Kleen Products cases, one might wonder: “can CAR really be as reliable (and therefore, legally defensible) as traditional manual review in identifying relevant data?” This question has become one of primordial importance to large-scale litigators tasked with sorting through huge volumes of ESI to find responsive and privileged material in an efficient and cost-effective manner. As will become evident, CAR technology combined with a robust audit trail can provide a defensible mechanism to meet one’s e-discovery obligations. How does one achieve defensibility in the
context of CAR? The producing party must build a detailed audit trail that satisfies a two-pronged test. The first prong speaks to the “reasonableness” of the CAR process; the selected method must accurately capture a reasonably sufficient number of the responsive, non-privileged ESI pursuant to a given document request, and any remaining unproduced ESI is predominantly not relevant or privileged.6 An audit trail satisfying the first CAR defensibility prong entails reaching agreement with one’s adversary as to the degree of recall and precision the CAR software will utilize in gathering, analyzing and coding discovery documents. Recall is the number of potentially relevant documents the system retrieves relative to the number of documents in the entire collection7 (e.g., as a result of a keyword search or concept-based search). Precision is the number of relevant documents identified in relation to the number of potentially relevant documents previously retrieved. Recall is an indicator of the system’s effectiveness in finding potentially relevant data, and precision is an indicator of the accuracy of the document coding process. Recall is a measure of completeness or quantity, whereas precision is a measure of exactness or quality. With too little recall, potentially relevant information will be overlooked; with too little precision, one’s “relevant” document set becomes riddled with non-relevant documents. The key to defensibility is a memorialized negotiation and agreement on these issues; it would be difficult for one’s adversary to claim production deficiencies in the face of a written recall and precision rate agreement. Additionally important in satisfying the
first defensibility prong as it relates to CAR, is the ability to document the process of coding “seed” documents and subsequent system “training” documents. It is generally understood that the technology used to identify relevant data is only as good as those training it. Because most difficulties in the CAR process stem from human review errors, the “seed” and subsequent “training” phases must be completed accurately and consistently by attorneys with expertise in the particular subject matter. Without accurate and consistent human input, the CAR analytics engine is unable to produce reliable results. Attorneys with subject matter expertise should draft a protocol detailing the criteria for accurately and consistently categorizing documents for defensibility purposes. Finally, the statistical components underlying the CAR process must be scientifically sound and documented to satisfy the first defensibility prong. These statistical components include random sampling, confidence levels, and confidence intervals. Random sampling is a means to obtain a truly representative sample of documents, by enabling all documents within a corpus to have an equal chance of being selected. Random sampling can be used to select the initial “seed” set and must be used to draw samples for each of the iterative training rounds. A confidence level expresses the level of certainty one can expect that a sample set of documents is drawn randomly from the system and reflective of the types of documents in a given population. A confidence interval is the margin of error— expressed as both positive and negative numbers—that a selected sample is truly (See CAR Discovery on Page 15)
But I get all of my business through referrals... storage, article libraries, and interactive calculators. By not offering these online resources, you may eventually lose clients to competitors who are.
By Fred Cohen, J.D. When we give presentations to attorneys, we inevitably hear one or two say, “I get most of my business through referrals, I don’t need a good website (or a website at all for that matter).” What these attorneys fail to realize is that a firm’s website is not only a business development tool but also a big part of the firm’s overall appearance and reputation. Even firms bursting at the seams with new clients need to have a great website. Here’s why:
Referrals will Google You Sure, the accountant down the street may refer all of his clients to you but, in many cases, his endorsement alone is not enough to seal the deal. After hearing your name, prospective clients will Google you. They’re looking for your credentials and your contact information, and if they’re not able to find any of this information easily, the likelihood of them contacting you decreases.
Colleagues and Opposing Counsel will Google You When your name comes up at a business meeting or when you are first introduced to your opposing counsel, they will Google you. First impressions are everything. By not having a website or having a poorly developed, or horribly outdated website, your firm may appear less pro-
Online Reputation Management Even if your firm is not fully embracing the web, other businesses and individuals are utilizing its many benefits. Each day there are new sites on the web devoted to local reviews so consumers can make the best decision when hiring a service provider. You have no control over whether you firm is reviewed and more importantly what reviewers say about your work. You do, however, have control of the online presence that you create. A well-optimized site will help to ensure that Google users are led to a place that you’ve created with accurate information about you as an attorney and your practice as a whole.
fessional. An effective online presence helps to establish your expertise and credibility.
Referral Sources Also Use Google By this point in the article, you should know everyone uses Google. Okay, maybe not everyone but a few hundred million people last year. Prospective referral sources are also using the web to find local attorneys who they might refer clients to. Without a website, it’s more dif-
ficult to connect with new referral sources that can drive business to your firm.
Your Site Should Be a Real Resource for Your Current Clients So you’re not looking for new clients on the web? That doesn’t mean that your current clients shouldn’t have access to firm resources online. As competition stiffens and technology advances, firms are delivering more value to their clients through online tools such as document
Not wanting to attract new clients is no excuse for not having an effective website. Working with an experienced website developer with expertise in the legal arena is the best way to establish a web presence that meets your firm’s objectives.
Fred J. Cohen, J.D., is the founder and President of Amicus Creative Media, an attorney web design and marketing company (amicuscreative.com), and a NYCLA member benefit partner. He can be contacted at 877-269-0076.