How this top 10 US bank is rethinking loyalty with better data and LLMs

The Need

One of the largest FI-s and credit card providers in the US with close to half a trillion in assets under management came to us needing better coverage and merchant information to manage their payments and loyalty experience. This will help them create sticky and useful journeys for end consumers and increase the LTV-s. 

Why does Spotify know the exact right song for me at the exact right time, while my bank is clueless that I am about to purchase a house ? This is a question we tried to answer for them using information from the bank transactions of their customers.To do this, they benchmarked  Ntropy’s Transaction Enrichment API, powered by our LLM orchestration infrastructure and GPT-4-0314.

They needed a solution that was beating their internal models , which was making errors in over 30% of transactions.

Every mistake, particularly a consumer facing one , is costly in this setting. It either results in frustration and churn, poor NPS or calling customer service which in turn is operationally expensive.

This FI needed Ntropy to

• Provide names and information for well-known and obscure merchants equally well
• Enrich the merchant with additional information such as websites, addresses, phone numbers

Evaluation

As a benchmark, we ran 20,000 hard transactions through the transaction enrichment API.

96.75% of all merchants, and 89% of intermediaries were correctly recognized, including 94.1% correctly assigned merchant websites. 

The distinction between intermediaries and the merchants was key for this FI to be able to action the transactions in a loyalty context and issue merchant rewards. 

Mixing intermediaries with the merchant is a common problem we encounter in the industry and a hard one to solve, particularly as the number of intermediaries and how they appear in transaction descriptions continues to grow. 

ROI

As a result, they can boost the AUC of their internal ML models that are built on top of enriched transactions by a significant amount. This ML stack is responsible for customer loyalty, underwriting, fraud and user recommendations. This also means less need for extra workarounds and consumer-facing UI tweaks to cover for model failures.  

As a further testament to the quality of the data produced via the Ntropy enrichment engine, the results even outperformed human labelers trained on this specific task. 

Leveraging the data quality stemming from the enrichment allows many more teams beyond loyalty at this bank to benefit from accuracy improvements across their ML pipeline which includes various key functions from general risk and underwriting to fraud detection and recommendations. 

Numbers

Since we talked accuracy boosts and how it lifts the whole pipeline, now lets talk cost and latency. This FI is processing around 60M credit card transactions per day. Enriching these transactions directly with GPT4-0314 , would cost $4.5M per day. With Ntropy, this costs $9k per day, a 500x reduction, while maintaining the same accuracy.  

The reduction in latency is also very significant. Using GPT-4 directly, the enrichment pipeline runs at p50 = 24s and p95 = 66s, and involves multiple calls to LLMs, databases and search engines. With the Ntropy stack, latencies are: p50 = 90ms and p95 = 245ms, depending on variables like the length of the transaction, number of entities involved, etc. This amounts to a ~260x reduction.