The insurance industry has always operated in an environment of uncertainty. However, evolving risk dynamics, increasing data volumes and growing competitive pressure are challenging insurers to make faster and more informed decisions. While AI is often presented as a transformative solution, many carriers find that the real challenge lies not in accessing technology, but in embedding it effectively into day-to-day operations.
While many of the themes discussed in this article have broader relevance across insurance, the examples highlighted are drawn primarily from Property & Casualty (P&C) insurance, particularly in pricing and claims.
Based on our work with insurers, we believe the key to closing this gap lies in moving toward an "insurance-native" application of AI—one that respects the unique mathematical and regulatory constraints of the industry while helping organisations overcome some of the legacy limitations that have historically slowed innovation.
Breaking the "statistical corset"
For decades, insurance analytics has operated within what we might call a "statistical corset". We have been highly effective at predicting outcomes based on structured data—the neat rows and columns of policyholder age, postcodes, and claim frequencies. However, this focus has left a vast wealth of information untouched.
80%+of insurer data is unstructured—hidden in notes, reports, and images.
It is estimated that over 80% of an insurer’s data is unstructured—hidden in claims adjuster notes, legal documents, medical reports, voice recordings, and site photographs. Until recently, the "corset" of traditional modelling made this data nearly impossible to utilise at scale.
The emergence of Transformer architectures represents a significant advancement in insurers' ability to derive value from unstructured data. While insurers have long used techniques such as OCR, text mining, natural language processing and other AI approaches, transformers have significantly expanded the scale, context and sophistication with which unstructured information can be analysed. Unlike previous iterations of AI that processed information sequentially, transformers use "attention mechanisms" to analyse relationships between many pieces of information simultaneously, allowing them to understand context much more effectively.
To visualise this, consider how a human scans a professional document. We do not read every word with equal weight from top to bottom; our brains subconsciously focus our "attention" on keywords, headers, and specific clauses that hold the most meaning. Transformers replicate this process at a massive scale, allowing an AI to "read" a 50-page policy document or analyse a satellite image of property damage with a level of context that traditional models simply cannot match.
Deep-Rooted Insights: Using retrieval-augmented generation for industry precision
While the public discourse around AI has been dominated by Large Language Models (LLMs) like ChatGPT, these general-purpose tools are often insufficient for the high-stakes environment of insurance. An AI can be fluent without being accurate, and in insurance, a "hallucination" in a claims decision or a pricing model represents a significant regulatory and financial risk.
When a claims handler asks an AI to evaluate a complex industrial claim, the RAG-enabled system does not just guess based on general knowledge.
To solve this, we are seeing the rise of Retrieval-Augmented Generation (RAG). Instead of relying solely on the data the AI was originally trained on, RAG "augments" the model by connecting it to a specific, private knowledge base—such as an insurer’s internal underwriting guidelines or a specific country’s case law.
When a claims handler asks an AI to evaluate a complex industrial claim, the RAG-enabled system does not just guess based on general knowledge. It retrieves the specific policy wording, verifies the current local regulations, and cross-references them against the evidence provided. This helps ensure that AI functions not as a replacement for human judgement, but as a high-precision research partner that provides a grounded and auditable basis for recommendations. Used within appropriate governance frameworks and oversight processes, it can improve consistency and efficiency while supporting expert decision-making.
The pricing paradox: Model instability and the Winner’s Curse
Perhaps the most sophisticated application of data science today is occurring in the pricing space, where the move toward more complex machine learning models—such as Gradient Boosting Machines (GBMs)—has introduced a "hidden" form of risk: Model Instability.
In highly competitive markets, particularly those driven by price comparison websites, insurers often fall victim to the "Winner’s Curse". This occurs when an insurer wins a policy not because they recognised a "good" risk, but because their model accidentally underpriced it.
Our research into model stability reveals a critical competitive differentiator. Two different GBMs can have identical "predictive power" (the same Gini coefficient or loss-cost accuracy on a test set), yet they can exhibit wildly different levels of stability. An unstable model is one where a tiny change in the training data—simply removing a few random rows—causes the predicted price for a specific individual to swing by 10% or 20%.
In a live market, this instability manifests as "price noise". If your model is noisy, you will inevitably be the cheapest for the risks you have accidentally underpriced, leading to a deteriorating loss ratio that your high-level metrics will not explain until it is too late.
Conversely, our simulations show that insurers who prioritise Model Stability can achieve a superior "Loss Ratio vs. Market Share" frontier. By reducing the noise in their predictions, they can strategically discount on the risks they truly want to win, while avoiding the "accidental" wins that erode profitability. In this context, data science is not just about being "more accurate"; it is about being more certain.
From insights to impact: Reimagining the claims journey
When these technical concepts—Transformers, RAG, and stable modelling—are combined, they enable a total reimagining of the insurance value chain. Nowhere is this more evident than in claims handling.
The goal of a modern claims transformation is to remove the "donkey work" that consumes the majority of a handler’s day. In a reimagined workflow:
An autonomous "agent" scans an incoming home insurance claim, instantly summarising the policy history and flagging any discrepancies in the narrative.
The system analyses photos of the damage (e.g., an escape of water), estimating the severity and comparing it against the "average" for that property type.
By analysing the sentiment and timing of the claim—unstructured data that humans might overlook—the AI provides a "fraud propensity score".
The handler receives a "next-step" recommendation supported by clear, explainable evidence.
An unstable model is one where a tiny change in the training data… causes the predicted price for a specific individual to swing by 10% or 20%.
By augmenting these data-intensive stages of the claims process with AI, we have seen carriers achieve more than $40 million in annualised benefits, driven not by reducing headcount, but by enabling their experts to make better, faster decisions on the claims that truly matter.
$40M+annualised benefits achieved by carriers through AI-enabled claims transformation.
The path forward
The transition to an AI-enabled insurance model is not a single leap, but a series of deliberate, insurance-led steps. To succeed, organisations must move past the "innovation lab" phase and focus on how these models function within a live, regulated environment.
In our experience, success requires a three-pronged approach:
Ensuring that data scientists work hand-in-hand with actuaries and claims experts.
Building "guardrails" around AI output to ensure explainability and fairness.
Moving models out of silos and into production environments where they can impact the "point of sale" or "point of claim" in real-time.
Insurance is inherently a business of data and trust. By leveraging modern data science to navigate complexity, insurers can build a lasting foundation of confidence—for their business, their stakeholders, and ultimately, their policyholders.
Authors
