Burke Publishes Framework to Assess Synthetic Data

In Cincinnati, Ohio, MR consultancy and training provider Burke, Inc. has launched the FAR Framework, a new approach for evaluating synthetic data quality and decision reliability; and published the results of its own research on the subject.

Burke's research compared multiple synthetic methodologies, examined how generative data models perform relative to synthetic panels, and tested 'whether the quality of the underlying human data remains critical in a synthetic-data world.' Results suggest that LLM-based synthetic panels can be valuable for early exploration, but are 'not yet reliable for decision-grade applications that rely on quantitative insight.' Burke says that at the 'commonly cited' 80% accuracy level, LLM-based synthetic data produced false conclusions in roughly 60% of tested business scenarios, with methods grounded in validated respondent-level human data - 'generative data models' - performed substantially better.

The FAR Framework evaluates synthetic data quality across three dimensions: Fidelity - Whether synthetic data aligns with the underlying source of truth; Authenticity - Whether synthetic responses reflect realistic variation rather than simply reproducing existing data; and Resolution - Whether relationships between variables, segments, and business conclusions are preserved.

'Organizations are hearing increasingly strong claims about synthetic data,' comments Burke VP Data Strategy Eli Moore (pictured). 'The important question isn't whether synthetic data sounds like your customer. It's whether it leads to the same conclusions you would reach by talking to your customer. That is the standard that we believe matters most.'

Web site: www.burke.com .

All articles 2006-23 written and edited by Mel Crowther and/or Nick Thomas, 2024- by Nick Thomas, unless otherwise stated.