In ALIVECOR, INC., v. APPLE, INC., Case No. 21-cv-03958-JSW (N.D. California, Feb. 23, 2023), before the Court was Defendant’s motion for relief from a non-dispositive discovery order.
This dispute arose out of the parties’ decision to utilize technology assisted review (“TAR”) to locate and produce ESI material in response to requests for production. Two of the key metrics used in TAR are “recall” and “precision.” “Recall is the fraction of relevant documents identified during a review; precision is the fraction of identified documents that are relevant. Thus, recall is a measure of completeness, while precision is a measure of accuracy or correctness. The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the ‘value’ of the case.” DaSilva Moore v. Publicis Groupe, 287 F.R.D. 182, 189-190 (S.D.N.Y. 2012).
The parties entered into an ESI Protocol that referenced the possibility that a party might use TAR. After the parties could not agree on a deadline to substantially complete their document production, the judge set a deadline of Oct. 15, 2022, which was subsequently extended to Oct. 31, 2022.
Defendant argued that although Plaintiff certified its document production was substantially complete, Plaintiff overproduced non-responsive and likely underproduced responsive documents. Defendant attributed this to a flawed TAR algorithm. Plaintiff responded that it structured its TAR process to be conservative. In its view, Defendant’s complaints stemmed in part from overbroad search terms and discovery requests. The parties submitted the dispute to the Court and on January 18, 2023, the Court issued a summary denial of Defendant’s motion.
When a party objects to a non-dispositive order issued by a magistrate judge, a district court “must consider timely objections and modify or set aside any part of the order that is clearly erroneous or is contrary to law.” Fed. R. Civ. P. 72(a); see also 28 U.S.C. § 636(b)(1)(A). Defendant argues the Discovery Order is clearly erroneous. (Mot. for Relief at 2:8-9.) A ruling is clearly erroneous if the reviewing court, after considering the evidence, is left with the “definite and firm conviction that a mistake has been committed.” United States v. U.S. Gypsum Co., 333 U.S. 364, 395 (1948).
Rule 26 requires that a party must supplement a discovery response if that “party learns that in some material respect the disclosure or response is incomplete or incorrect, and if the additional or corrective information has not otherwise been made known to the other parties during the discovery process or in writing[.]” Fed. R. Civ. P. 26(e)(1)(A).
In their ESI Protocol, the parties agreed that if they used TAR, they would meet and confer on an appropriate protocol. The parties exchanged drafts of such a protocol and agreed upon a recall rate at 70%.
Defendant asserted it conducted “several tests to assess the accuracy of [Plaintiff’s] TAR algorithm,” which it argues showed the “algorithm cannot reliably distinguish between responsive and unresponsive documents.” However, Defendant did not support those assertions with evidence, such as a declaration from the attorney who performed the test. Defendant also did not introduce examples of documents from those tests that it contended should have been flagged as responsive but were not.
“[P]erfection in ESI discovery is not required; rather a producing party must take reasonable steps to identify and produce relevant documents.” Winfield, 2017 WL 5664852, at *9; accord Reinsdorf v. Skechers U.S.A, Inc., 296 F.R.D. 604, 615 (C.D. Cal. 2013)
The Court also found the following discussion from Winfield apt:
[T]here is nothing so exceptional about ESI production that should cause courts to insert themselves as super-managers of the parties’ internal review processes, including training of TAR software or to permit discovery about such process, in the absence of evidence of good cause such as a showing of gross negligence in the review and production process, the failure to produce relevant specific documents known to exist or that are likely to exist, or other malfeasance.
Winfield, 2017 WL 5664852, at *9.
Accordingly, the Court determined the lower court judge’s ruling was clearly not erroneous and denied Defendant’s motion for relief.