Predictive Coding: Understanding the Wows & Weaknesses - BKD.com

Predictive Coding: Understanding the Wows & Weaknesses Bryan Callahan, CPA, CFF, CFE, CVA

Lanny Morrow, EnCE®

Managing Consultant

Supervising Consultant

Forensics & Valuation Services

Forensics & Valuation Services

[email protected]

[email protected]

experience expertise // Elevate your understanding of key forensics concepts with guidance from BKD Forensics Institute.

Definitions & Aliases “Predictive coding is the electronic coding, organization, and prioritization of entire sets of electronically stored information (“ESI”) according to their relation to discovery responsiveness, privilege, and designated issues before and during the legal discovery process. Lawyers control this process by specifying relevant criteria.” --Forbes “The analysis and identification of relevant ESI, assisted by artificial intelligence.” Some also call it “technology-assisted review.”

2

The Reason • Explosion of data – “Big Data” • 95% of data never leaves digital domain • Considered “greatest eDiscovery challenge” by 2015 • Predictive coding has “greatest potential to improve eDiscovery” by 2015

3

What Is Unstructured Data?

4

Overview of the Process 1. Predictive coding tool selects random sample of documents (called “seed set”) 2. Senior attorney(s) review this set & assign each item as “relevant” or “not relevant” 3. Artificial Intelligence (AI) component performs “find more like this” search 4. Subsequently retrieved documents are again reviewed & assigned a category 5. Process continues until no more items are found to be relevant 5

Under the Hood • Seed set – Relevance is sometimes moving target in a case. Random sampling protects against bias • AI learns from initial items marked “relevant”, then assigns probability to every single item • Applies true “latent semantic analysis” • AI compares attorney decision to its own & continuously “retrains” itself • Better systems will have “game” component • Process is keyword independent

6

Latent Semantics From: To:

The Boss Everyone

I wanted to congratulate Lisa on her really outstanding presentation at the meeting! The update was easy to follow and was very well received by the group. Great job, Lisa! Kim also gave a nice presentation. We appreciate you both.

Lisa

Kim nice

Real Example Token Thank you

To: Briber From: Employee taking bribe --------------------------------------------“Thank you for the “little” token – I park it in the driveway so my neighbors can be jealous! It screams on the open road (wow...!) PS-I promise it’s just between you and me!”

park driveway road

gift

vehicle

Between you and me promise conspire !,!,! Wow Thank you

excitement

Benefits & Best Uses • Forget smoking gun – 85 to 90% reduction in review is primary benefit • Every part of process is documented, repeatable & can be statistically validated • Excellent in early case assessment & investigations for rapid identification of relevant materials • Best use is in tandem with traditional keyword, concept & similarity searches already established • Reduces inherent human error

9

2011 RAND Study on Human Review “Taken together, this body of research shows that groups of human reviewers exhibit significant inconsistency when examining the same set of documents for responsiveness under conditions similar to those in large-scale reviews … Human error in applying the criteria for inclusion appears to be the primary culprit [regarding the lack of accuracy], not a lack of clarity in the document’s meaning or ambiguity in how the scope of the production demand should be interpreted. In other words, people make mistakes, and based on the evidence, they make them regularly when it comes to judging relevancy or responsiveness.”

10

Weaknesses, Myths & Concerns • Not “man vs. machine”—it’s more about “augmented intelligence” than “artificial intelligence” • This is not new technology, just really improved • Vendors jockeying for market position, so lots of “proprietary flavors” of technology • Still under consideration by courts (when both parties are not in agreement) • AI is still far from perfect—proceed with caution

11

Da Silva Moore Decision Da Silva Moore decision in Southern District of New York – appropriateness of predictive coding 1. Parties’ agreement 2. Vast amount of ESI to be reviewed 3. Superiority of computer-assisted review to available alternatives 4. Need for cost effectiveness & proportionality 5. Transparent process proposed by Defendant Da Silva Moore v. Publicis Groupe, 287 F.R.D. 102 (SDNY 2012)

12

Current Stance by Courts • July 2012 – Nat’l Day Laborer Org. Network v United States ICE Agency, 877 F. Supp. 2d 87, 109 (SDNY) • December 2012 – Robacast, Inc. v. Apple, Inc., No. 11-235 (D. Del.) • January 2013 – EORHB, Inc. v. HOA Holdings, Inc., No. 7409-VCL (Del. Ch. Ct.) • March 2013 – Chevron Corp. v. Donziger, No. 11 Civ. 0691, 2013 U.S. Dist. LEXIS 36353 (SDNY) and Harris v. Subcontracting Concepts, LLC, No. 1:12-MC-82, 2013 U.S. Dist. LEXIS 33593 (SDNY) 13

The Future of Predictive Coding “It’s hard to predict things, especially the future” – Yogi Berra • Rise of “evidentiary expert”? • Can attorneys cooperate enough? • It’s not perfect, is that ok?

14

Predictive Coding: Understanding the Wows & Weaknesses Bryan Callahan, CPA, CFF, CFE, CVA

Lanny Morrow, EnCE®

Managing Consultant

Supervising Consultant

Forensics & Valuation Services

Forensics & Valuation Services

[email protected]

[email protected]

experience expertise // Elevate your understanding of key forensics concepts with guidance from BKD Forensics Institute.

BKD Forensics Institute Upcoming Schedule • Valuation Strategies for 2013 & Beyond: Tax & Business Succession Planning After the Fiscal Cliff o o o

Wednesday, May 1, 2013 11:30 a.m. – 12:30 p.m. CST Presented by Teal Dakan, CPA & Carol Lewis, CPA, ABV

• Whistle-Blowers & Fraud Hotlines: Reporting & Preventing Fraud o o o

Wednesday, May 8, 2013 11:30 a.m. – 12:30 p.m. CST Presented by Julia Swafford, CPA, CFE

bkd.com/fi 16

Starbucks Gift Card & Certificate • To help promote the event as associate training tool, we are offering a bonus o

17

Associates who attend all three sessions will receive a Starbucks gift card & Certificate of Participation from BKD

Slides & Webinar Archive • Today’s presentation slides are available at bkd.com/fi. Recorded copy of webinar will be available at the same location at conclusion of webinar series • If you have any questions, please contact Dane Ryals @ [email protected]

18

Thank you for attending. Learn more at bkd.com/fi.