Is Process Transparency Required?, Assisted Review Series Part 8

Over the nine years since it first rose to prominence in eDiscovery, technology-assisted review has expanded to include numerous new tools, more potential workflows, and a variety of legal issues

In “Alphabet Soup: TAR, CAL, and Assisted Review,” we discussed TAR’s rise to prominence and the challenges that it has created for practitioners.  In “Key Terms and Concepts,” we discussed the key terms and concepts practitioners need to know.  In “Applications, Aptitudes, and Effectiveness,” we discussed use cases, relative merits, and overall efficacy.  In “Are You Allowed to Use TAR?,” we discussed case law on authorization to use TAR.  In “Can You Be Compelled to Use TAR?,” we discussed case law on being compelling or directed to use TAR.  In “When Is the Right Time for TAR Process Objections?,” we discussed case law on objection timing.  In “What TAR Process Results Are Objectionable?,” we discussed case law on how good is good enough.  In this Part, we review case law discussing process transparency requirements.

Another common question regarding the use of TAR approaches in discovery is whether process transparency is required.  Process transparency in this context refers to transparency regarding how the chosen TAR approach is being deployed in the case, for example: how the chosen tool is being configured, how it is being trained, how its results are being tested, and to what standards of recall and precision it is being held.  In some cases, process transparency has even included disclosure of the actual (non-privileged) documents used in the seed sets and control sets, including non-relevant documents.  But, is any or all of that required?

Although there is some variation in the cases considering this question, the short answer is that process transparency is preferred but is not typically required absent some demonstrated deficiency in the process’s results.

Voluntary Process Transparency

In many of the early cases addressing the use of TAR approaches, the party seeking to use TAR voluntarily offered a lot of transparency into their proposed TAR process.  Given the novelty at that time of using a TAR approach, this voluntary transparency was offered to help overcome objections and secure agreement from their opponent and approval from their judge.

• In da Silva Moore, which we previously discussed, the defendants’ proposal for using a TAR approach incorporated extensive voluntary transparency, including: turning over all non-privileged documents reviewed to create the seed set (both relevant and not), all non-privileged documents reviewed during iterative training (both relevant and not), and all non-privileged documents reviewed during the final random sample to check for missed material (both relevant and not).

• In Global Aerospace, which we also previously discussed, the defendants’ proposed TAR approach involved extensive transparency into the process for the plaintiffs, including production of the “the full set of training documents” (with the exception of privileged documents and “sensitive documents coded as irrelevant”), as well as production of the “statistically valid random samples [] taken from both the relevant and irrelevant document set” to determine the process’s recall (again, with privileged and sensitive irrelevant documents excluded).

• In In Re: Actos, which we also previously discussed, the parties cooperatively negotiated a detailed plan for the use of a TAR approach, which was memorialized in a Case Management Order. It provided that all review and coding of the control set, the training sets, and the validation sets would be done collaboratively (with privileged materials screened out), all relevance determinations would need to be agreed to by both parties, and both parties would need to sign off on the results of each phase in the process – including agreeing on the review cutoff point.

• In Bridgestone, which we also previously discussed, the plaintiff asked if it could switch to using a TAR approach due to enormous volume encountered using the originally agreed discovery plan. The plaintiff volunteered to provide process transparency if permitted to proceed.  The Magistrate Judge allowed the switch to using a TAR approach, but in a subsequent order concerning the ongoing discovery efforts, the Magistrate Judge “remind[ed] both parties that to the extent they use predictive coding, he expects full transparency in how the predictive coding is established and used.”

For a time, the pattern in these early cases made it seem like a high level of transparency might be a requirement for TAR use, despite the facts that: (a) transparency in those cases was often offered voluntarily to help overcome objections to what was then a novel approach and (b) traditional document review approaches were never that transparent (or expected to be).

Attempts to Compel Transparency

Not all cases have involved voluntary process transparency, however, and in cases where transparency has not been offered voluntarily, courts have expressed displeasure but have declined to compel transparency absent some demonstrated deficiency in the results:

• In In Re: Biomet M2a Magnum Hip Implant Products Liability Litigation, No. 3:12-MD-2391 (N.D. Ind. Aug. 21, 2013), the defendants opted to use a TAR approach, and the plaintiffs’ attempted to compel disclosure of the seed set documents the defendants used to train their predictive coding tool, so that the plaintiffs could look for gaps. The Judge did not see a basis for compelling the requested disclosure:

The [plaintiffs] want[] to know, not whether a document exists or where it is, but rather how [the defendants] used certain documents before disclosing them.  Rule 26(b)(1) doesn’t make such information disclosable.  [emphasis added]

Though he concluded he could not compel the defendants to make the requested disclosure, the Judge did describe the defendants’ refusal as “troubling” and “below what the Sedona Conference endorses.”  He also warned the Defendants that there could be consequences for uncooperativeness:

An unexplained lack of cooperation in discovery can lead a court to question why the uncooperative party is hiding something, and such questions can affect the exercise of discretion.  [emphasis added] 

• In Winfield, which we also previously discussed, the plaintiffs alleged that the defendant’s TAR process was deficient and sought to compel the disclosure of both detailed process information and non-privileged samples of the documents deemed not responsive by the process. The Magistrate Judge reviewed the available case law and found it split:

Courts are split as to the degree of transparency required by the producing party as to its predictive coding process.  In some cases, parties have agreed to – or courts have ordered – transparency into the TAR or predictive coding processes.  In other cases, courts have not required production of seed set documents.  [internal citations omitted, emphasis added]

In this case, the Magistrate Judge also split, coming to different conclusions regarding the requested process information and the requested document samples:

• • The magistrate Judge opted to review the requested process information in camera and concluded that those materials were “protected by the work product privilege” and that it was “unclear how this information is even potentially relevant to the claims and defenses in this litigation, as required under Federal Rule of Civil Procedure 26” [emphasis added].

• • The Magistrate Judge did conclude, however, that the plaintiffs had made a sufficient showing of potential issues (“examples . . . suggest there may have been some human error in categorization that may have led to gaps in the City’s production”) to “justify their request for sample sets of non-privileged documents” [emphasis added].

• In Entrata, Inc. (Property Solutions Int’l) v. Yardi Systems, Inc., No. 2:15-cv-00102-CW-PMW (Utah D. Jun. 20, 2018), the requesting party sought to compel the producing party to turn over “the complete methodology and results” of its TAR process so that they might “assess the adequacy” of that process. The Chief Magistrate Judge, however, was not willing to order the requested disclosures without some showing of actual deficiency with the results:

Notably, [the plaintiff] has not provided any specific examples of deficiencies in [the defendant’s] document production or any specific reason why it questions the adequacy of [the defendant’s] document collection and review.  Without more detailed reasons why production of [the defendant’s] TAR information is needed, the court is unwilling to order [the defendant] to produce such information.  [emphasis added]

The Judge ultimately agreed with the Magistrate Judge that the plaintiff was not entitled to discovery of the defendant’s process details:

“The scope of the obligation to search for, and produce, ESI is circumscribed by Federal Rule of Civil Procedure 26(g) . . . .”  Karl Schieneman & Thomas C. Gricks III, The Implications of Rule 26(g) on the Use of Technology-Assisted Review, 7 Fed. Cts. L. Rev. 239, 243 (2013).  But “[n]othing in Rule 26(g) obligates counsel to disclose the manner in which documents are collected, reviewed and produced in response to a discovery request.”  Id. at 254.  [emphasis added]

Upcoming in this Series

Up next in the final Part of this series, we will conclude our discussion of assisted review with a look at some case law discussing the combination of search terms and TAR approaches, as well as a review of some key takeaways from across the series.