How to measure the lift from buyer intent data with a holdout test

To measure the lift from buyer intent data, a sales and marketing team compares accounts it worked with the data against accounts, chosen at random, that it worked without. A comparison of conversion by score shows that the score sorts accounts, and leaves open what the data adds. This post gives you the holdout test that settles it before an annual contract: the design, the size of each group, the arithmetic of lift and its uncertainty, a worked example and a checklist for the pilot.

What it means to measure lift from buyer intent data

Lift is the difference between two rates over the same period: the share of accounts that reach an outcome, such as an opened opportunity, when your team works them with the intent data, and the share when it works them without. Both groups come from one list, split at random, so the data is the only systematic difference.

Intent data changes the order in which reps and campaigns reach accounts, and no outcome by itself, so what you test is a rule: what your team does differently when a score is high. The lift belongs to the data and the rule together, and a pilot that leaves the rule unwritten measures neither.

Of three comparisons called lift, one measures it.

Run the first comparison as a screen: a score that does not sort your past opportunities has little to offer a pilot. The guide to prioritising accounts with intent scores shows that look-back.

Google Ads describes its Conversion Lift as an incrementality tool: a controlled experiment that separates an audience into people who see the ads, the treatment group, and people who do not, the control group, and, for studies based on users, reports incremental conversions as absolute lift beside a relative conversion lift. The third comparison is the same experiment with accounts in place of people, and the assignment is yours to make.

The holdout test: incrementality measurement for B2B intent data

A holdout test is the incrementality measurement for B2B intent data. Split your target accounts at random into a group your team works with the data and a holdout it works as usual, with the scores hidden. Fix the outcome, the window, the group sizes and the end date before the data arrives, then compare the two rates once.

The UK Cabinet Office's 2012 paper Test, Learn, Adapt sets out nine steps required to set up any randomised controlled trial, for example deciding the outcome in advance and allocating the intervention at random. A pilot of intent data needs seven decisions, written into a test plan you and the provider both hold.

1. The outcome. One yes-or-no outcome per account your CRM records with a date: a first meeting held, or an opportunity opened at a stage with written entry criteria. Pipeline value and closed revenue are secondary: a few large deals move them, and most arrive after the pilot ends.
2. The window. A fixed number of days from assignment, the same for every account, long enough for a typical account to reach the outcome.
3. The unit. The account, assigned one by one, which the sizes in this post assume. Assigning whole territories or teams is a different design with its own calculation.
4. The assignment. Random, from a recorded seed, inside blocks. The NIST handbook's rule for randomised block designs is to block the factors you can control and randomise against the rest. Put accounts into blocks by segment, fit tier and owner, then split each block at random, so every rep carries the same number of accounts on each side.
5. The rule. What the team does in the treatment group, in the provider's own fields: for example, an account flagged as an intent surge on one of three named topics goes to the top of its owner's queue that week.
6. Equal effort. The same reps and the same budget of touches on each side. If attention moves from the holdout to flagged accounts, the holdout's rate falls and the test reports a transfer of effort as lift. Log touches by group.
7. The analysis. Every account counts in the group it was assigned to, whether or not it was ever flagged or contacted, and the result is read once, on the end date in the plan.

Assign the whole list for the figure the contract depends on: the net lift per target account, with effort displaced from unflagged accounts counted in. When the list is small, assign at the flag: each account that first crosses the threshold is released to its owner or withheld, at random, and the flag is recorded either way. Accounts never flagged then do not dilute the effect: if one account in five is flagged, a ten-point lift among flagged accounts is a two-point lift across the list.

How many accounts each group needs

The size of each group follows from four numbers: the baseline rate, the rate to detect, the significance level and the power. The NIST handbook gives the calculation for one proportion: a critical value for the significance level times one standard deviation, plus a critical value for the power times another, divided by the difference to detect, and squared. The formula below for two groups of equal size is not on that page: it is the same calculation with the standard deviations of a difference between two rates, pooled under equal rates, as in the test below, and unpooled under the lift to detect, as in the interval.

accounts per group =
  ( 1.96 × sqrt(2 × p × (1 - p)) + 0.8416 × sqrt(p1 × (1 - p1) + p2 × (1 - p2)) )^2
  / (p2 - p1)^2

p1      baseline rate, expected in the holdout
p2      rate to detect in the treatment group
p       (p1 + p2) / 2
1.96    critical value for a two-sided test at 5% significance
0.8416  critical value for 80% power

In words: multiply 1.96 by the square root of twice p times one minus p, where p is the average of the two rates. Add 0.8416 times the square root of the sum, over both rates, of the rate times one minus the rate. Divide by the difference between the rates and square the result. The formula assumes accounts assigned one by one, one yes-or-no outcome per account, groups of equal size, and groups large enough for the normal approximation to the binomial.

From that formula, the accounts needed in each group for a two-sided test at 5% significance with 80% power:

At a 5% baseline, 300 accounts in each group will not reliably show even a 50% relative lift, which takes 1,471. When your list is too small, three changes help. Measure an outcome higher in the funnel, since doubling the baseline more than halves the size. Assign at the flag. Keep the groups equal: the variance of the lift carries one over the size of each group, so holding out a fifth of the accounts in place of half needs roughly 1.6 times as many in total (1/0.8 plus 1/0.2 is 6.25, against 4 for two halves).

Incremental conversion rate and incremental pipeline from an intent data A/B test

The incremental conversion rate of an intent data A/B test is the treatment group's conversion rate minus the holdout's, in percentage points. Multiply it by the number of accounts you will work with the data to get incremental opportunities, and by your average opportunity value to get incremental pipeline. Report each with its confidence interval.

treatment rate  = treatment conversions / treatment accounts
holdout rate    = holdout conversions / holdout accounts

lift, in points = treatment rate - holdout rate
relative lift   = lift / holdout rate

standard error  = sqrt( treatment rate × (1 - treatment rate) / treatment accounts
                      + holdout rate × (1 - holdout rate) / holdout accounts )
95% interval    = lift ± 1.96 × standard error

pooled rate     = all conversions / all accounts
z               = lift / sqrt( pooled rate × (1 - pooled rate)
                             × (1 / treatment accounts + 1 / holdout accounts) )

incremental opportunities = lift × accounts to be worked with the data
incremental pipeline      = incremental opportunities × average opportunity value

The interval is the normal-approximation interval for a difference between two proportions in NIST's Dataplot reference. In words: for each group take the rate times one minus the rate, over the group's size; add the two and take the square root; then go 1.96 of those standard errors either side of the lift. The same page gives an adjusted interval that adds one success and one failure to each group and improves coverage. Use it when a group has few conversions.

The test is the large-sample test for two proportions in the NIST handbook. In words: divide the lift by the square root of the pooled rate, times one minus the pooled rate, times the sum of one over each group's size. A value beyond 1.96 in either direction rejects equal rates at the 5% level. Interval and test both assume accounts assigned one by one and large groups, and for small samples that page gives the Fisher exact test.

Read the interval against a figure fixed in advance, the break-even lift: the cost of the contract for one outcome window, divided by the expected revenue of one opportunity, divided by the number of accounts you will work with the data. Agree the action for each of three readings.

Break-even can sit far below the smallest lift the pilot is sized to detect. An interval that spans break-even then says the test was too small to tell, which is a different finding from no lift.

A worked example: Acme Robotics tests an intent data provider

Acme Robotics is an invented company, and every figure in this example is illustrative.

Assumptions:

• Acme has 5,000 target accounts, all matched to the provider's company records.
• In its CRM history, 5% of target accounts open a qualified opportunity in a 90-day window.
• The smallest lift Acme wants to be able to detect is two points, from 5% to 7%.
• The test is two-sided at 5% significance with 80% power.
• Accounts are assigned one by one, at random within blocks of segment and owner. The same reps work both groups with the same budget of touches, and the result is read once, 90 days after assignment.
• An opportunity is worth 40,000 US dollars of pipeline on average and one in five is won. The provider quotes 60,000 US dollars a year.

The working:

Size of each group
  p               = (0.05 + 0.07) / 2                             = 0.06
  first term      = 1.96 × sqrt(2 × 0.06 × 0.94) = 1.96 × 0.33586 = 0.6583
  second term     = 0.8416 × sqrt(0.05 × 0.95 + 0.07 × 0.93)
                  = 0.8416 × 0.33556                              = 0.2824
  accounts        = ((0.6583 + 0.2824) / 0.02)^2 = 47.035^2       = just over 2,212, so 2,213

Result, 90 days after assignment, with 2,500 accounts in each group
  worked with the data: 180 opened an opportunity                 = 7.2%
  holdout:              125 opened an opportunity                 = 5.0%

Lift
  lift            = 7.2% - 5.0%                                   = 2.2 points
  relative lift   = 2.2 / 5.0                                     = 44%

Interval
  standard error  = sqrt(0.072 × 0.928 / 2,500 + 0.05 × 0.95 / 2,500)
                  = sqrt(0.0000457264)                            = 0.6762 points
  95% interval    = 2.2 ± 1.96 × 0.6762 = 2.2 ± 1.3254            = 0.87 to 3.53 points

Test
  pooled rate     = (180 + 125) / 5,000                           = 0.061
  z               = 0.022 / sqrt(0.061 × 0.939 × (1 / 2,500 + 1 / 2,500))
                  = 0.022 / 0.006769                              = 3.25

Pipeline in one 90-day window, across 5,000 accounts
  opportunities   = 0.022 × 5,000                                 = 110
  interval        = 110 ± 0.013254 × 5,000 = 110 ± 66.3           = 44 to 176
  pipeline        = 110 × 40,000                                  = 4.4 million US dollars
  at the low end  = 44 × 40,000                                   = 1.76 million US dollars

Break-even
  one opportunity = 40,000 × 1/5                                  = 8,000 US dollars of expected revenue
  one window      = 60,000 / 4                                    = 15,000 US dollars of contract
  break-even      = 15,000 / 8,000 = 1.875 opportunities          = 0.04 points on 5,000 accounts

Each group needs 2,213 accounts, and Acme puts 2,500 in each, which raises the power to about 85%. The lift is 2.2 points with a 95% interval of 0.87 to 3.53, and z is 3.25, beyond 1.96, so Acme rejects equal rates at the 5% level.

Across the list that is 110 more opportunities in a window, and 44 at the low end of the interval, against a break-even of 1.875. The interval lies wholly above break-even, so the reading is to sign. The result holds on one further assumption: the rule and the effort stay as tested.

Had the holdout opened 160 opportunities, the lift would have been 0.8 points with an interval from minus 0.6 to 2.2. That interval spans break-even, so the reading would have been a lift not shown at this size.

How to pilot test a B2B intent data provider before an annual contract

To pilot test a B2B intent data provider before an annual contract, agree a written test plan before any data arrives: the accounts, the outcome, the size of each group, the rule your team will follow and the end date. Then store every delivery as it arrives, run the holdout for one full outcome window and read the result once.

Send the provider your own target accounts as domains, count the matches by segment, and treat that coverage as the first result, because an unmatched account cannot be in either group. The ten questions for a company data vendor cover the rest of the review.

A score that carries dated evidence makes the pilot auditable, in three checks.

• The order of events. For each converted account in the treatment group, the evidence should be dated before your first touch, and the touch before the opportunity. An opportunity that opened before its signal cannot be credited to the data.
• No hindsight. A score in the holdout has to be the score as it stood that week. A delivery you stored should match a fresh export of the same week, the test set out in why we write each table once.
• One definition. The scoring version on the rows should not change during the pilot.

Fokals Intent Scores support each of these checks. Each is a weekly score from 0 to 100 for one company and one topic, and every score comes with the dated evidence behind it, while Company Signals delivers dated company signals daily. Weekly rows are written once, after the week closes, and are never revised, and a score keeps the definition it was written under. The intent dataset delivers both, the methodology describes each dataset, and the post on what a Fokals intent score gives you shows how to read one. Sample data for the companies you name is sent on request with the data dictionary and the methodology, so the match and the evidence audit can run on your own list.

An evaluation checklist for an intent data vendor pilot

An evaluation checklist for an intent data vendor pilot has three parts: what is fixed in writing before the data arrives, what is logged while the pilot runs, and what is checked before the result is believed. Each line below names the record that shows the step was done.

Read splits by segment or topic as questions for a second test: the pilot was sized for one comparison.

Frequently asked questions

The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.

What this page says about the products it names was checked against their public documentation on 7 October 2026. Product and company names are trademarks of their owners. Fokals is not affiliated with them or endorsed by them.