Adventures In AI

ADVENTURES IN AI

VOL 1

Part 1

The Innovation Officer and the Data Scientist

JD Cambell, VP of Innovation at Golden Life Insurance company, is contemplating his future.

what am I going to do? I need a win this year or I’m fired.

AI is the future, but we haven’t accomplished anything.

3 years ago I convinced Susan, our CEO, to invest in AI. we had such high hopes...

3 Years Ago

I’m going To support JD’s Plan.

3

What have I delivered?

I haven’t put a single machine learning application into production.

Nothing except an occasional list of targeted email addresses for marketing— which they hardly use.

I’ve lost all internal support from my business customers.

And now I’ve got 5 months to make an impact with AI—or I’m out.

And yesterday, Susan gave me “the talk.”

She has supported my initiatives these past 3 years and wants to see real results.

I have no ideas.

I’m basically done.

4

Meanwhile, Lori Santos, Senior Data Scientist, is having a conversation with Don Richards, her supervisor.

DOES HE SERIOUSLY EXPECT ME TO JUST STAND HERE AND WAIT FOR A RESPONSE?"

Is he even listening?

So...

as I was saying...

I’ve been testing these NLP transformer models to classify support emails.

The results were really good, and I’ve never seen anything understand text this well.

I think transformers have huge potential for our business.

5

Yeah, NLP, we’ve been doing it for years. I think Amy tried transformers and they didn’t work too well.

I know...

I did a code Review ON AMY'S Project...

Remember?

She just downloaded the BERT model and ran it against call center data before she got sucked into the data clean-up project.

We didn’t give transformers a serious look.

And those call transcripts were crap anyway.

So...?

Yeah, do you have the list of emails for marketing?

We’ve got a deadline next Friday so they can run their next campaign.

Não acredito que estou trabalhando para estes idiotas preguiçosos....

Sure, I’ll get right on it!

6

Well, my 15 minutes of actual data science work for this week is done.

Yipee, I ran another random forest classifier against the 3 useful features and made another marketing email campaign.

Good thing I got my PhD in computational astrophysics.

My adviser will be so proud!

I’ve got to get out of here.

It isn’t Don. He’s not a bad boss....

he’s just retiring soon and keeping the seat warm for the next leader.

7

I just expected to be doing more than cleaning data and sitting in meetings all day.

5 Years ago.

When JD Campbell spoke at a python meetup 5 years ago, he really sold me on the AI future here at Golden. And I’ve spent the past 2 years lying to myself about how everything is going to change.

Nothing here is going to change.

So much is happening in AI, and it’s time to start looking for my next gig.

O PLUT

VENU Y CUR MER

8

S

RN SATU

S MAR

H EART

R YTE JUP

NUS URA

NEPT

UNE

9

Hey JD, you look like I feel.

Oh, sorry.

Yeah, long week.

Huh?

Good grief, what am I saying? It’s only Tuesday.

How are things going in data science?

Ha. Yeah, I get it. Will I see you at the next Python meetup?

Nah. I stopped going since most of the talks are about Django.

Good.

Well, you know, the usual.

I’m doing a talk at the machine learning meetup tomorrow.

You going?

10

I guess I am now. What’s your talk about? Anything I’ll understand? or is it super-technical?

APACHE

NLP transformers.

It’s a state-of-the-art technology that has made several NLP breakthroughs.

I just started playing around with it, and so far I’m seeing some pretty amazing results.

What time is your talk?

11

The next evening at the Machine Learning Meetup.

So those are the 3 steps:

(1) Download a pretrained NLP transformer model,

(2) Fine-Tune it on your data,

And (3) swap out the final neural network layer to do the text classification.

Any questions?

12

You performed the 3rd step by classifying a movie review in the IMDB data set,

so you’re starting with a set of labeled text data.

But in the real world we don’t have labeled data, and it seems like it would be a whole lot of work to get experts to hand-label documents.

Excellent points!

At Golden Life we have a lot of complex text that requires expertise to understand, such as contracts, policies, claims, and customer emails.

Yep, you nailed it.

That’s the biggest concern.

There’s just no way we can ask our lawyers, for instance, to spend a month reading and handlabeling documents.

NLP transformers are impractical without a way to solve the document-labeling problem....

13

I think you can bootstrap document labeling through weak supervision.

But I’ve got some ideas on ways to overcome this.

I’ve seen a few references to this approach, but I haven’t had a chance to try it out.

Any suggestions for test projects?

14

Thanks everyone.

Lori, as always, that was awesome.

We’re out of time, but Lori can hang out for a few minutes to chat further.

In the meantime we’ve got refreshments in the back.

Hey rockstar, you want a lift home?

Sure.

Beats waiting for an Uber.

15

That was awesome. Are you using NLP transformers for any work projects? I never heard about any of this.

Nope. I’m happy to tell you more, but you’ll need to indulge my ranting.

Ha. From you I expect nothing less than brutal honesty. Rant away.

I know we’ve got a lot of work on our plate ... yada yada yada.

I’ve tried talking about the potential of NLP transformers, but nobody cares.

Innovation is one of our corporate values, but it’s just talk.

This technology will revolutionize our industry. by ignoring it, we’re going to get crushed by an insurtech startup or a competitor.

Nobody listens to me—or cares.

Ah. Ok, I get your frustration, and maybe I can help.

Here’s the deal.

16

Our industry hasn’t changed much in 100 years.

The leaders at Golden have a hard time imagining how something like NLP transformers will change our industry.

I’ll put myself in that category....

I get that you’re excited about this technology, but I still don’t see the connection between what you presented tonight and our business.

Fair enough.

Perfect. My office at 9:30?

We’re almost at my place, but my morning is wide open if you want to discuss.

Maybe we can do something with NLP transformers.

Deal. See you tomorrow, and thanks for the lift.

17

it's a long shot, but at this point I’ve got nothing to lose and no better ideas.

Part 2

The potential of NLP transformers to transform Golden Life ... and save JD’s job.

So this is the basic process as I explained last night. but as we discussed, the real challenge is step 3—

1. Download pretrained transformer.

2. Fine tune.

3. Build categorization.

Here’s the deal.

You’re brilliant and 100x smarter than the rest of us.

Lori, I’m lost.

But you have a vision about this technology that I don’t understand.

And if I don’t understand it, nobody else will.

So what?

What can NLP transformers do for Golden Life?

So let’s take a step back.

What’s new?

Why now?

19

Most of our business still largely runs on unstructured text—

Ok, sure.

Here’s how I see it. ..

conversations, emails, forms, chats, contracts, etc.

Conversations between agents and applicants.

Calls, emails, and chats to our support center.

Applications that go to our underwriters, and the notes they take.

Resumes and cover letters submitted to HR. Attending-physician statements received by our underwriters.

Claims submissions we process and check for potential fraud.

Policy documents we read and summarize.

Reinsurance contracts our lawyers process.

20

we’re drowning in unstructured text data.

Most of our people do nothing but ingest, sort, summarize, and generate text—

Let’s take claims, for instance.

I see people reading a claim submission,

matching it to a policy, CLAIMS SUBMISSION

reading the policy terms, processing this information,

and summarizing for a supervisor.

Suppose you could automate 50% of that work in the next 4 years—what happens?

21

Well, using your example of claims,

POTENTIAL FRAUD

CLAIMS

CLAIM PROCESSING

I imagine a smaller department. claims operators and data scientists can create algorithms to automatically process most claims.

MANUAL REVIEW

And a few people can process exceptions or investigate potential fraud.

Well, our costs would obviously be lower.

We would win more business by processing applications faster. We’d retain more customers by being more responsive.

COST

RISK 2021

2022

2023

2024

2025

2026

And we could finally begin mining our customer conversations for up-sell opportunities.

Exactly.

And what happens if competitors start automating these processes—and we don’t?

Yeah, I get it. And this isn’t a hypothetical situation.

These insurtech startups don’t have our legacy infrastructure and can leap ahead of us with automation.

And Amazon or Wal-Mart ... I don’t even want to think about them.

22

Don told me we’re already doing NLP.

What’s so new about your idea?

But I’m still missing the connection to NLP transformers.

Why now?

Ah, Well NLP is a very hard computer science problem. and traditional techniques never worked very well.

Don is correct. we’re doing some simple NLP using word vectors and bag-of-words, neither of which can be used to automate our business.

Everything changed in 2016, when Google demonstrated state-of-the-art NLP results by using neural networks for Google Translate.

Many traditional NLP techniques were instantly obsolete.

unfortunately, building SYSTEMS TO ADOPT THE NEW TECHNOLOGY took Google-sized budgets, which are unrealistic for most companies.

23

That is, until now.

Companies like Google and Facebook invest millions to develop and train NLP transformers.

These neural networks understand how language works, and they achieve human-level performance on some tasks.

In step 1 I download a pretrained language model to leverage Google’s investment to build a custom, state-of-theart NLP application for Golden.

In step 2 I customize the transformer for our language: insurance terms, our product names, etc.

And in step 3 I teach it a task, like classifying customer conversations.

24

Ok I get it. Automating unstructured-text processing will shift the cost/speed curves in our industry. While it may provide an initial advantage, this tech will soon become table stakes.

NLP transformers are a breakthrough technology that can process tech as well as people can.

And any data scientist can use them to build a state-of-the-art custom text-processing application.

We do still have to address the practical problem of labeling the training data.

But I have ideas for solving it with weak supervision.

Any??? I didn’t say any data scientist.

That is, I could solve it, but instead I’m cleaning data, sitting in meetings, and rerunning the same simple model to generate An email list that marketing doesn’t use....

But seriously, I give up.

I’m tired of trying to get my colleagues to understand.

We talk about innovation, but people here just want to float along and pretend that we won’t get crushed like retailers and automakers did.

I didn’t come here to be another corporate drone, and I damn sure didn’t get a PhD to do data science 101 and watch us get run over.

Anyway, thanks. This was fun, but I’m 10 minutes late for the weekly marketing meeting.

Oh my, what will they do without me?

If I can just deliver initial value to one of my business customers with these NLP transformers,

I know I can use it to sell a bigger vision. I need to find Lori a useful challenge, because I know she won’t give up until she solves it.

Later that week JD organizes a meeting between Lori and Diondre Williams, VP of contracts in Golden Life’s Legal Department.

Thanks for making the time, Diondre. A few months ago you asked me about automatically sorting our contracts by their legal risk. Can you tell us a bit more?

Sure. We have thousands of contracts with clauses describing situations where 3rd parties have the right to decrease payments to us for things like

failure to comply with reporting requirements, cash reserves, acts of God, etc. Maybe 10 different scenarios.

When a scenario arises—like a hurricane or a big change in interest rates—we need to review our outstanding contracts for potential legal exposure.

Got it. So why can’t you just digitally search the contracts and read the relevant section? And couldn’t you just do this once?

We read through as many PDFs and paper copies as we can to sort them into buckets of risk.

27

Simple keyword searches won’t work.

If I search for something like “payments” I get too many results.

“PAYMENTS”

3000 RESULTS

So the contractual language could be in many different places and worded differently?

A more specific search like “decrease payments in the event of” will miss sections if the wording isn’t perfect.

Exactly.

Some contracts date back to the ‘80s and we’re always getting new ones.

legal terms change and we can’tjust categorize them once.

After the meeting.

It looks like a good application for NLP transformers.

I can prototype it in a Jupyter notebook, but first I need an hour with someone on your team who can show me a few example categories.

What do you think? Will it work?

I have no doubt that NLP transformers can do it. But the real problem is training data.

Diondre’s team doesn’t have weeks to sit around labeling contracts for me, so I’m going to try weak supervision to label contracts programmatically.

28

Ok. When do you think you’ll have results?

About 8 weeks, assuming Diondre’s team can give me occasional feedback.

I’ll give you an update in 2 weeks.

29

4 months until I brief our senior leadership team on our innovation program.

I’ll need Diondre’s boss to stand with me and endorse this project.

If Lori can’t solve Diondre’s business problem I will have nothing substantive to present to the senior leadership team.

I will have to present another prototype, and Susan will probably cancel the innovation program and fire me.

Because we’ve got nothing else, and Susan won’t go for more happy talk about prototypes and potential.

The funding for the innovation program—and my job—are in Lori’s hands.

30

Part 3

Can Lori save the day by using weak supervision to generate training data?

2 weeks later.

Well, I made some progress.

Legal provided 3,000 contracts as PDFs.

How’d it go?

They showed me how they would sort the contracts manually.

the first step is converting the PDFs to text files via OCR.

and our IT department can’t do that for at least 2 months.

Oh man... you... you should have seen your face.

Do you really think I would sit around waiting for these lazy guys?

Really? You just had to do that to me?

32

AAAANyWAy... Fortunately I found a way to do the OCR myself using Tesseract, an open source solution. And I made some good progress on the training data by using weak supervision.

Diondre’s team showed me how they manually search and sort these contracts for payment risks.

I followed their approach to create a label function that uses regular expressions.

The label function creates a set of imperfect training data. This is why we call it weak supervision.

The training data is messy, but it didn’t require much work from our lawyers. I used it to train the model, and now I’m feeding all of the contracts into the NLP transformer to see how it categorizes them.

I still don’t understand how you’re using weak supervision to create training data.

I get that you’re making a set of messy training data, and you’re doing this because Diondre can’t manually label the data. But what happens next?

33

The NLP transformer I fine-tuned to our contracts understands the language of these contracts. my hope is that it can generalize the categorization by using a few examples even if it isn’t trained on every possible example.

Suppose we train the Because The model should classify model to classify a document a document it understands that that mentions severe that mentions a hurricane in a a hurricane is a type weather in the same way specific risk category. of severe weather.

Category 1: Training

Category 1: Predicted

right to delay payments in the event of a hurricane

and in the case of a severe weather event such payments can be delayed by up to 2 months.

Ah, I get it. So... Using the label function to sort them all would be too complicated? The label function would get too messy and too hard to maintain? well done, you must have a head for attention to focus on what is most relevant. That’s exactly it.

You could use traditional NLP techniques instead, but they would be too expensive to build and support.

We want the neural network to handle this workload.

So when will you know if it works?

34

We need feedback from Diondre, because I don’t understand all of the legal jargon, and the categories are a judgement call. We’re meeting next week.

1 week later.

So here are the initial results from the NLP transformer I customized for your team.

It reads each document and categorizes chunks of text based on the legal risk associated with payments.

Most of these look right.

Perfect.

Number 4 is wrong.

It should be “moderate risk” because of the clause in the 2nd sentence.

It looks like this issue causes a few other contracts to be misclassified.

I can make a quick update to the label function and fix it for all—

Wait. What is 24? Which contract is that? Can you show me the full paragraph?

35

That’s new.

I didn’t know any of our contracts had an act-of-God clause that includes drought.

With climate change in mind, I need to dig into this and learn about our exposure. And you correctly categorized this as “high act-ofGod risk.”

How?

How did you do this? We didn’t know this example was in our contracts.

How did the NLP transformer find it?

Well THAT’s the power of NLP transformers.

They understand the meaning of language and can generalize situations even if we don’t have the exact wording.

It looks like the model correctly identified “drought” when I told it to categorize contracts based on severe weather.

Honestly I wasn’t expecting this to work. these contracts are so complex. What’s our next step?

And a supportive email from your boss to me and Don would help us get more resources on the project.

I need you to tell me which contracts are correctly or incorrectly classified so I can improve the results.

Done and done.

I’ll put them into a spreadsheet so you can give me feedback.

36

So tell me.. .

I didn’t know.

The examples were in random order.

how did you know to put that clause about drought on the first slide?

Total luck, my friend.

You know that you just hit the data science jackpot, right?

Total luck.

A win.

Finally.

5 years of hard work with lackluster results.

And at the last moment I’m saved by a brilliant Brazilian astrophysicist and a generous helping of good-old-fashioned luck.

3 months later at the innovation program brief.. .

And so that’s how the contract classifier works.

Eliza, what’s the feedback from your team?

Diondre can’t stop talking about it.

It looks like it’s already saving them time, and the model found additional financial exposure from climate change that we didn’t know about.

We need this.

37

I’ve been in this business for 30 years.

This NLP technology is also going to change everything for the reasons Lori and JD explained.

When the Internet arrived I instantly knew this was a long-term fundamental change in our industry, and so I bet on it.

After the meeting adjourns Susan has a private chat with JD.

Nice job.

I want a business case and budget for investing further in this NLP technology.

I want to see quantified cost savings or revenue targets.

Get the IT department involved so we can make this part of our cloud migration.

And don’t give me a rosy unrealistic timeline and budget.

38

Let me share a secret with you.

In financial services we primarily spend money on two areas: lawyers and support.

And both involve a whole lot of unstructured text data. If we don’t invest in this tech we’re roadkill.

I need to tell you about Lori.

I’m worried she is frustrated and—

39

Already resolved.

Epilogue EPILOGUE

I Hope my story helps you understand the power of NLP transformers and why they will change everything about your business.

BUDGET CLOUD

ROADMAP

But I’m sure you have A LOT questions about getting started.

WORKFORCE TRANSFORMATION

42

Unfortunately, I can’t help you because I’m just a cartoon!

But let me introduce you to my friend Russ Rands of Prolego.

His company can help you accelerate your adoption of AI, and he is happy to answer any questions you may have.

New Message To russ@prolego.com Subject Russ,

just email him at russ@prolego.com

or connect at linkedin.com/in/russrands

43

ABOUT ADVENTURES IN AI Most people don’t understand what AI can do, and people who do understand the technology often struggle to explain its value. Adventures in AI bridges this gap through examples and illustrations. Although the stories are fictional, the challenges, opportunities, and technology are based on actual case studies.

Episode 1 helps anyone understand how natural-language processing (NLP) transformers can automate tedious business processes.

ABOUT PROLEGO Prolego is the AI acceleration partner for Fortune 1000 companies. We develop AI strategies, build models, deploy solutions, transform workforces—and do anything else our clients need to realize their AI goals faster.

PROLEGO.COM