How this business bank is changing the game for SMB finance with LLMs + Ntropy AI

The Need

When GPT4 launched, one of the first demos that came out was the finance copilot for businesses. Brex released this in collaboration with OpenAI. It was a bot that could answer all your finance related questions seamlessly, based on your internal data. 

“The natural language AI-features will further enhance Brex Empower’s live budget capabilities, and will be able to answer a wide range of questions related to utilization on various budgets, insights on spend patterns, vendor trends, and more” said the initial press release. 

This sounded fun at the time, unclear in terms of usability in practice. 

The reality is that AI has a ton to do with improving SMB lives, as the latter lack time and have the data, which is typically a ripe spot for workflow automation. This is what Brex, Ramp, Navan and others are focused on delivering today, using financial data, aka live transactions, to capture the pain of finance teams and founders right at the root. It is not just about moving money fast or accessing money fast, it is about understanding your business better and your unit economics to make better decisions. It is also absolutely about saving time on the repetitive processes, such as closing your books, paying bills, assigning and policing budgets, and reconciling receipts. 

To improve and end-to-end automate these workflows, you do not need to be able to speak to a copilot. You instead need LLM-s that completely replace and unify, as well as commoditize high value and high throughput mini-tasks, from Named Entity Recognition in business onboarding and KYB, to OCR on receipts and bank statements, as well as invoices, and finally transaction classification and enrichment. 

Traditionally these tasks were solved with old-school ML models that would fail on edge cases, which were accommodated for with workarounds and humans in the loop. For every one of these tasks you would have a few vendors, an internal solution and human processes. These are expensive and hard to maintain and never get you 100% of the way there with the end-user experience. 

Recently, one of the major up-and-coming commercial banks came to us, recognizing this problem and being very keen to replace their existing models and vendors with a single and reliable approach that would allow their customers to see their own transactions auto-categorized and reconciled with the accounting ledger without additional hassle. 

They had internally tested LLM-s and knew they were the way forward, yet were struggling with the orchestration part. The LLM output was costly for the size of the model that would accurately perform on their task. They were rate-limited, and finally needed to build internal support infrastructure to do everything from continuous evaluations to validation. 

They started testing Ntropy’s transaction enrichment on top of Ntropy's AI platform, as a potential solution for the vendor management functionality they were building out.

Evaluation

We ran the evaluation in less than 12 hours with our human validation team involved as well, double-checking the model output and pinpointing the discrepancies, if any. 

The results were 10-15% accuracy percent improvement over the previous system, with less maintenance, configuration, vendor management and operational overhead.

- Labels - 94.8%

- Merchants - 91.4%

- Intermediaries - 97%

The set that was sent to benchmark was a handpicked collection of hard cases and transactions with a lot of ambiguity that are tricky even for a human to understand.

ROI

LLM-s are here to completely change the lives of SMB-s and what it means to run an SMB. 

We saw 15% accuracy improvement, faster closing of books and reconciliation, better receipt matching and finally hours saved by finance teams and founders. This helped the commercial banking provider avoid building the orchestration part in-house, involving many complex vertical-specific steps such as model routing, benchmarking, prompt engineering, evaluations that have to be continuous, as well as cost and speed optimization.