An A/B Test Variant Reached the Wrong Approved Audience: Reconstruct Assignment and Quarantine Results

A LinkedIn outreach A/B test can fail even when every delivered message was approved. The failure is assignment: one valid variant reached prospects who belonged to a different approved cohort.

Treat the incident as a data-quality problem first. Do not compare reply rate or connection acceptance rate, as well as booked calls until the prospect-level mapping explains who should have received each variant and what they actually received.

For consent-based business outreach, the immediate control is simple: pause new sends from the affected logic while preserving the raw records, then separate affected prospects from unaffected prospects. Do not resend a message merely to make the dataset look balanced.

Which prospects received a valid variant in the wrong approved cohort?

Build one incident census with one row per prospect and campaign exposure. Start from the approved cohort rules that existed when the test launched, not from the cohort definition someone edits after the incident.

For each prospect, calculate an intended variant from the frozen approved rule. Then join that record to the delivered variant, send timestamp, reply events, connection events, and booked-call events. A mismatch between intended and delivered variant is the affected population.

Keep the cohort rule readable. Optimizely documents audiences as logical conditions that can be nested with the AND/OR/NOT operators. That structure makes a rule review possible when an exclusion or inclusion was mapped incorrectly.

• Freeze the approved cohort rule and its effective time window.
• List every prospect evaluated by the test during that window.
• Mark the intended and delivered variants, plus whether exposure evidence exists.
• Flag every intended-to-delivered mismatch for quarantine.

What evidence reconstructs intended and observed assignment?

Use records that identify the prospect, experiment, variant, timestamp, and the rule version. A warehouse assignment source can explicitly define a timestamp column, experiment ID column, group ID column, and mappings from experiment units to source columns, as documented by Statsig.

Observed assignment needs a separate delivery or exposure record. LaunchDarkly describes exposure events as records of a context evaluated and assigned to a variation, with its context key and first evaluation time available for review alongside the variation. That is the shape of evidence to preserve, even if your outreach system uses different field names.

Do not let a CRM status stand in for exposure evidence. A prospect marked contacted may have been queued or retried, or handled outside the affected test. The census should state which system produced each field and whether the record is direct evidence or a later operational label.

Can any observations survive after the assignment reconstruction?

Yes, but only at the row level. An observation can remain in a limited analysis when the prospect was eligible under the frozen rule, received the intended variant, has direct exposure evidence, and has a metric event inside the predeclared window.

Exclude a row when the prospect received the wrong valid variant or if either the intended cohort cannot be reconstructed or delivery evidence is missing. Those rows do not become usable because a reply or booked call occurred later.

Keep a third state for prospects with valid assignment but uncertain metric attribution. For example, a booked call may be real but outside the chosen measurement window. Preserve it in the incident log without using it to compare variants.

• Salvage: matching intended and delivered assignments with evidenced exposure and an intact metric window.
• Quarantine: intended and delivered assignment conflict, or the intended assignment cannot be proved.
• Rerun: the affected cohort or the test-level comparison no longer supports a decision.

When does the full test need a rerun?

Rerun the affected comparison when the wrong mapping changes who was eligible for a variant or makes the remaining groups incomparable. Do not average the issue away by moving affected rows into the variant they should have received.

A rerun should use a new experiment record with a frozen cohort definition and documented start time. Preserve the incident census beside it so later reviewers can distinguish the clean test from the quarantined one.

Holdout design can protect a defined portion of an experimentation audience from included experiments. LaunchDarkly recommends creating a holdout before the experiments it will include, so it is a planning control for the corrected run, not a repair for prior exposure.

How do you test the corrected assignment logic before sending?

Test both sides of each cohort boundary. Positive tests confirm that a prospect who meets the approved rule receives the intended variant. Negative-mapping tests confirm that a prospect who fails a rule cannot receive that variant.

Use explicit fixtures for exclusions, overlapping conditions, missing fields, and rule-version changes. Review the fixture result against the frozen approval, then have someone outside the rule change review the mismatches.

Optimizely's documented condition grammar supports nested Boolean logic. That makes nested-condition fixtures important: a simple test of one inclusion rule can pass while either an OR branch or a NOT condition still routes the wrong prospect.

• Run one expected-match test for every approved cohort.
• Run one negative-mapping test for every exclusion that previously mattered.
• Log the expected variant, returned variant, rule version, reviewer, and timestamp.
• Block launch when any fixture returns an unexpected variant.

What should the incident record say to the outreach team?

State the operational fact without turning it into a performance claim: an approved outreach variant reached an approved but incorrect cohort under the test plan. Name the send window, affected campaign, frozen rule version, and whether new outreach is paused.

State the measurement boundary: replies, connection acceptance rate, booked calls, or any other recorded outcome from quarantined rows will not decide the variant. If a corrected run follows, report it separately from the incident.

For Well Met campaigns, keep the distinction clear. Comment-led outreach and warm connects, alongside follow-up conversations, can be legitimate activity, but an A/B assignment error still makes the affected comparison unfit for a decision until the evidence supports it.