Rule: do NOT make "do X, conversions rise Y%" promises to clients — only ~10% of A/B tests produce a positive primary-metric win (Optimizely meta-analysis); ~10-20% at Google/Bing (Kohavi); published wins are survivors

Rule: Do not make "do X, conversions rise Y%" promises to clients. Treat any specific percentage lift sourced from a vendor case study as survivorship-biased until proven otherwise on the client's own site, with their own traffic, with adequate sample size.

Why: Optimizely's meta-analysis of ~20,000 experiments found only ~10% reached a significant primary-metric win (Optimizely meta-analysis of ~20,000 experiments (per Thomke & Ghosh, Harvard Business Review): only ~10% reached a statistically significant primary-metric win); Kohavi reports Google and Bing see only ~10–20% positive (Ronny Kohavi (ex-Microsoft/Amazon): Google and Bing see only 10–20% of A/B tests produce a statistically significant positive result on the primary metric — published CRO wins are survivors). Published wins are survivors of a process where the large majority of ideas fail (Survivorship and file-drawer bias in CRO — published "psychology win" case studies are surviving winners; failed tests are rarely reported; specific lift figures are not corroborated science). The mechanism-to-conversion mapping is not a fixed quantity (Genuine unknown — none of the lab effects license a numeric conversion prediction for any specific site; the mapping from "50 ms appeal" to revenue is NOT a fixed quantity).

How to apply: When discussing CRO with a client, frame as: "the mechanism (halo, fluency, etc.) tells us which direction to lean — toward visual professionalism, clarity, working functionality. The magnitude of any specific change can only be measured on your own site, with your own traffic, and most tests don't produce a significant win even at elite shops." When a vendor pitch quotes a specific lift figure, ask for the denominator (out of how many tests was that the published win?) and the base rate they're comparing it against.