While machine learning might be as close to a household name as any AI capability in 2023, any serious historian will tell you the technology itself is nothing new. The term ‘machine learning’ was coined in an IBM study on computer gaming and AI dating back to 1959.
Looking at this history holistically, legacy financial institutions have been traditionally best positioned to reap the benefits of machine learning – that is, if they’ve been investing in developing these capabilities in tandem with their organizational growth over decades. Few organizations were as visionary in adopting machine learning as American Express.
For a comprehensive perspective on American Express’s decades-long digital transformation and how advanced their models are at detecting credit card fraud, Emerj spoke in 2021 with the credit card giant’s then-Head of Machine Learning Centre of Excellence, Dmitry Efimov, on the AI in Business podcast. (Dmitry has since departed American Express and currently serves as Senior Manager of Research Science at Amazon.)
Dmitry began working at American Express in 2014 as their fraud prediction models began undergoing significant changes that – as he explains at length on the program and we analyze in depth throughout this article – place them at the forefront of credit card detection today. This article will also examine two key insights Dmitry shares with business leaders on the podcast about how American Express approaches model development to address fraud:
- Asking the right questions: Looking closely at what imminent questions and formula changes to models are downstream from asking, “how can we help our customers reduce fraud risk while improving customer experience?”
- Human oversight in stopping transactions: Determining the appropriate role of human administration even in the decisions that models can deliver with maximum certainty.
- Leveraging legacy data in self-learning models: How neural networks are helping models to teach themselves to better detect fraud by integrating customer and merchant data.
Listen to the full episode below:
Guest: Dmitry Efimov, then-Head of Machine Learning Centre of Excellence, American Express; current Senior Manager of Research Science at Amazon.
Expertise: Machine learning, computer science, data science, research science
Brief Recognition: Dmitry Efimov is a machine learning data science leader with nearly two decades of experience, stretching from his current role at Amazon to his time at American Express we cover in today’s podcast. Dmitry also has spent time teaching applied data mining and machine learning at American University of Sharjah and Lomonosov Moscow State University, respectively.
Asking the Right Questions
Dmitry begins by spelling out the big difference between how American Express used machine learning before and after his arrival at the company in 2014. In essence, the technology being used was essentially the same as today, but that technology wasn’t answering the right business questions for their customers.
“What happened before? In fact, that was machine learning for probably already more than 50 years,” Dmitry tells Emerj CEO Daniel Faggella on the AI in Business podcast about the essential capabilities of machine learning when he arrived at American Express in 2014. “It’s just the technique that was used at that time was not so advanced.”
The ‘technique’ Dmitry is referring to underscores the success of all predictive models that has nothing to do with being able to code or having any advanced knowledge of math, which is: what kind of questions are you asking that are using the model to answer?
The difference in technique Dmitry cites above came when American Express began using machine learning to answer a series of specific questions that he outlines during his podcast appearance in the following order:
“The first question you should ask is, ‘What is the specific business question you want to answer?’ Let’s take the example of fraud risk within American Express. The first question we want to ask is how we can help our customers and how we want to improve the customer experience and reduce fraud risk for them?’
In fact, even if we go deeper in this question, we could split it into different types of questions. We can ask, for example, ourselves, how we can decrease the fraud rate on the transactional level. And in this case, the model becomes on the transactional level.”
– Head of Machine Learning Centre of Excellence at American Express, Dmitry Efimov
In looking at the results of their model’s attempts to answer these questions, Dmitry and his team decided the principle metric they needed to find to make their model more effective was the probability of fraud for each transaction.
Realigning their focus begged another vital question, Dmitry explains: “Another question we can ask could be, can we predict for – on the customer level – if fraud appears for the specific customer in the customer history?”
Finding answers to that question meant having to not only ask during investigations whether or not customer activity was indicative of their fraud but also whether or not customer transaction histories could show they had also been victims of fraud.
“And in these cases, actually, the machine learning models become very different,” says Dmitry. “So that’s why, in fact, all the time when you work on [a] machine learning problem, you should first start from the equation [asking] what business problem you want to solve.”
In developing their model to answer these double-sided questions from even before the wave of credit card and other financial fraud that would come with the COVID-19 pandemic, American Express was able to apply predictive analytics to their model and begin predicting where and when fraud would occur – both from the transactional and customer perspectives.
Dmitry explains that another significant difference between American Express models a decade ago and the generation they use today is that, as far back as 2014, the company’s models were developed by a method of logistic regression – a more primitive process of defining likely outcomes.
He tells Emerj that, up until that point, models predicting fraud in each of American Express’s major customer segments were developed in silos with logistic regression. To integrate them, Dmitry says that American Express created thousands of decision trees that the model and human teams use to decide what data is relevant to specific consumer segments.
As of 2021, when the episode was recorded, Dmitry indicated that the 10th generation of the American Express fraud model (referred to both in the podcast and at American Express as “Gen X”) consolidated both international and US markets.
By that point, American Express was then able to single out all prior siloed customer segments in automatic permutations based on whatever geography, industry, and personal demographic delineations they want to calculate to answer essential business questions for their customers.
Human Oversight in Stopping Transactions
As advanced as models are for detecting fraud, Dmitry states unequivocally, they are never allowed to single-handedly decide if a transaction is fraudulent at American Express. Furthermore, he tells Emerj that the decision to halt a transaction remains in the hands of an oversight team composed of human beings in American Express’s Strategy department.
Dmitry outlines a three-step process for investigating and possibly halting transactions based on suspicions of fraudulent activity:
- A department called Decision Science is responsible for building the model.
- An oversight team in American Express’s Strategy division then takes the model and, after much deliberation and analysis, decides on a score by which a transaction will be categorized as most likely fraudulent and not allowed to proceed.
- The oversight team then decides how they want to use the score in business rules, which produces a final decision.
Dmitry tells Emerj that American Express uses thousands of business rules to determine the score of potential fraud in a transaction. Furthermore, the standard for deciding high scores and significant fraud incidents worthy enough to halt a transaction is very high and ultimately very rare so as to not unnecessarily disturb the customer experience.
He also clarifies that the decision to stop a transaction can rely on extraneous factors and variables too long for a brief article to cover. However, there are some situations where a single business rule is simple and – if broken – triggers an investigation to see if it’s fraud and stops the transaction if it is.
That is as close as any process related to American Express’s fraud detection model comes to being fully automated. Ultimately however, human beings still decide whether to follow through with the investigation based on how the business rule was broken. Dmitry also emphasizes that team members from American Express’s Strategy division are still involved throughout every stage of the process.
Leveraging Legacy Data in Self-Learning Models
When asked what he sees as the future of credit card fraud modeling, Dmitry cites an application currently in development at American Express in “self-learning” models.
Essentially, in developing 10 generations of fraud models, American Express’s Decision Science team noticed repetitive, highly automatable decisions in updating models and teaching them to predict fraud in increasingly more complex customer segments and delineations.
In observing these repetitive processes, they were able to develop ways of automating those decisions that essentially amount to the model being able to teach itself predominantly how to predict fraud activity better.
Dmitry cites what he sees as another indicator of the future of fraud detection models in integrating “unique” legacy customer and merchant data that American Express has been able to collect from their decades-long standing as the foremost legacy US-based credit card brand:
“I know that data that are available at American Express are very unique because we add two sides,” he tells Emerj. “So we have customer information, we also have merchant information. The uniqueness is because we can combine this information together and create very powerful features.”
While these applications are not nearly mature enough to make common practice – Dmitry tells Emerj that, in combining this data, Strategy and Decision Science teams at American Express show significant improvement at predicting what transactions will be fraudulent than with prior iterations of the model.
