Published in Westlaw Today
Failing to provide discovery as required by the Federal Rules of Civil Procedure risks sanctions. The standard for conduct in discovery, however, is not perfection. So, what lies between sanctionable conduct and perfection? More often than not, the answer is quality assurance (QA). QA can help make imperfect discovery defensible, rather than sanctionable, whereas a lack of QA could be a recipe for sanctions.
Take, for example, Bans Pasta, LLC v. Mirko Franchising, LLC, 2015 WL 13861049, at *2-*3 (W.D. Va. Feb. 6, 2015). In that case, the plaintiff subpoenaed documents from a third party after the defendants had produced documents. The third party produced documents that the defendants presumably should have produced but didn’t, such as emails copying individual defendants. In response to plaintiff’s motion for sanctions, the court held that “[t]hese documents should have been found in a proper and thorough search of Defendants' computer systems and email accounts.” Emphasizing the defendants’ obligations under the Federal Rules to conduct a reasonable search and to produce relevant and responsive documents, the court granted the sanctions motion because “Defendants' document review, with no oversight or quality control mechanisms in place was, at best, negligent[.]” Id. (emphasis added). In other words, QA was not optional.
What is QA? It is a system of principles, methods, protocols, and procedures to ensure the quality of a product or process. Within QA, quality control (QC) is the act of inspecting the work product to ensure that it meets specifications. See e.g., American Society for Quality, Quality Assurance and Quality Control (Accessed September 8, 2022) https://asq.org/quality-resources/quality-assurance-vs-control#Perspectives.
The need for QA is not unique to eDiscovery. QA systems and QC measures are not only standard but also are essential in a myriad of industries. For example, QA is a part of software development, automotive manufacturing, aviation, pharmaceutical development, food production, and healthcare, to name a few.
So, what does QA and QC look like in the context of eDiscovery? The Sedona Conference Commentary on Achieving Quality in the E-Discovery Process (15 Sedona Conf. J. 265 (2014) (the “Sedona Commentary”)) describes QA as procedures designed to serve as the basis for certification and reliance, and QC as the safeguards used during the process to ensure high quality throughout. The Sedona Commentary identifies the following five methods to achieve quality in eDiscovery:
- Judgmental sampling, where a more experienced person reviews a sample of another person’s work to assess whether that person is making the correct decisions on the given task. Judgmental sampling does not permit extrapolations about the entire population from which the sample was drawn (e.g., a random sample of 10% of documents coded “Not Responsive” or 5% of documents that hit on a privileged term as “Not Privileged” to assess whether those coding decisions were correct);
- Independent testing,where third-party professionals examine a process to evaluate whether the results can be replicated and confirmed;
- Reconciliation techniques,which involve comparing inputs and outputs (e.g., “comparing what volume of email or ESI enters a process, what remains in a process (after, for example, deduplication), and what exits a process.”);
- Inspection to verify and report discrepancies, similar to judgmental sampling with the added feature of reporting, which can be used to course correct and improve performance (e.g., a tiered approach where reviewers with over a 10% overturn rate have their documents subjected to additional QC review); and
- Statistical sampling, which evaluates a statistically significant random sample of documents that have gone through a process (or been excluded from the process) to determine the error rate. If the error rate is higher than a pre-determined acceptable error rate, informed decisions can be made to improve the process and achieve a more acceptable error rate. Unlike judgmental sampling, statistical sampling can be used to estimate the overall probable error rate for a project.
Quality in eDiscovery can be measured and achieved using any of the above techniques to varying degrees, or potentially in a variety of other ways, depending on the type and volume of the document population, the methods used to cull and review, and the time available to complete the review. But among these methods, sampling – selecting a subset of documents from within a larger population - may be the simplest to execute while also going a long way towards making a search or review process defensible. The Electronic Discovery Reference Model (EDRM), a recognized source of eDiscovery best practices, has identified several discovery phases where sampling may be appropriate. These include:
- Judgmental sampling in identifying and collecting documents;
- Statistical sampling during review as a quality control measure to ensure that the correct coding decisions are being applied; and
- Using sampling to assess the quality of a particular reviewer or an entire review team.
See EDRM Statistical Sampling Applied to Electronic Discovery (Revised February 18, 2015), https://edrm.net/resources/project-guides/edrm-statistical-sampling-applied-to-electronic-discovery/. In terms of knowing when enough QC has been done, the EDRM also helpfully explains:
“The stronger the case that further review would be expensive, fruitless and disproportionate, the better the argument for ending the review. Any decision to end review early needs to be backed up with appropriate facts that justify this choice and generally, no single factor will be determinative on its own. However, demonstrably valid statistics can be one of the factors used to justify this decision.” Id.
Regardless of the how, it is apparent that some level of QA during the eDiscovery process is not only consistent with and supported by the spirit of the Federal Rules governing discovery, but also can be the determining factor in avoiding sanctions. See In re Seroquel Prods. Liab. Litgn., 244 F.R.D. 650, 662 (M.D. Fla. 2007) (“[c]ommon sense dictates that sampling and other quality assurance techniques must be employed to meet requirements of completeness.”). Rule 26(g) also arguably encourages some level of quality assurance in responding to discovery. See Fed. R. Civ. P. 26(g)(1)(b) (by signing a discovery response, the attorney or party certifies that it was made after a “reasonable inquiry”); see also City of Rockford v. Mallinckrodt ARD Inc., 326 F.R.D. 489, 494 (N.D. Ill. 2018) (stating that validation and quality assurance support the reasonable inquiry certification pursuant to Rule 26(g)).
Thus, while perfection in eDiscovery is not required, employing QA is always a good idea to help ensure that your document collection and review processes are defensible should they ever be challenged.
By Gareth Evans, Leslie Melo, and Amanda Mitchell, Redgrave LLP
Gareth Evans is a partner, and Leslie Melo is a counsel, at Redgrave LLP, where they counsel clients on complex matters related to information law, which includes eDiscovery, information governance, and data privacy and cybersecurity issues. They are located in Los Angeles and Chicago, respectively, and can be reached at gevans@redgravellp.com and at lmelo@redgravellp.com. Amanda Mitchell is a senior advisor at the firm, where she manages eDiscovery processes and develops and implements strategies for document review. She is based in Chicago and can be reached at amitchell@redgravellp.com.
The views expressed in this article are those of the authors and not necessarily those of Redgrave LLP or its clients.