A/B testing
A test can tell you what works.Only if it has enough visitors.
Most A/B tests on small stores end too early or never had a chance. We work out first whether your traffic can give a real answer. If it can, we set the test up and read it properly. If it cannot, we tell you and fix the obvious problems instead.
- Hypothesis first
- Sample size from your traffic
- Whole weeks
- No early calls
Why tests mislead
The dashboard said winner.Sales did not move.
Called after four days.
The variant looked ahead, so the test stopped. Checking early and often makes false winners far more likely.
Not enough visitors to know.
A low-traffic page cannot separate a small real change from luck, however long you stare at it.
Counting the wrong thing.
Clicks went up, orders did not. Without one primary metric chosen up front, any result can look like a win.
Also on our list
- Testing app slowing every page
- Traffic split that came out uneven
- Variant URLs indexed by Google
A/B testing that gives a real answer.
A/B testing on Shopify means showing two versions of a page, or of a change, to two random groups of visitors at the same time, then comparing how many in each group did the thing you care about, such as placing an order. It is the most reliable way to know whether a change caused a result. It also needs a lot of visitors. If your store does not have them, the test cannot give you a real answer, and the honest move is to not run one.
Can you A/B test on Shopify?
Yes. Shopify has a built-in option called Rollouts, managed from Markets > Rollouts in your admin. One rollout type, an Experiment, compares a treatment against a control. It can include changes to your theme, checkout and accounts settings, catalogs or discounts. New experiments split traffic 50/50 by default and you can change that. Shopify says Experiments need the Grow plan or higher, only apply to the online store, and do not work on vintage themes.
Third-party testing apps are the other route. They differ in what they can test (theme sections, prices, whole templates), how they split traffic and how they count results. Whatever you use, check where it gets its numbers and how it handles visitors who switch devices.
How much traffic do I need to A/B test?
Enough to tell a real difference from luck. Four things decide the number:
- Your baseline: how often visitors convert today, on the page you are testing.
- The smallest change worth detecting: the minimum detectable effect. Small changes need far more visitors to see.
- Significance level: how often you accept calling a winner when there is no real difference. Five percent is a common choice.
- Power: how often the test catches a real difference of that size. Eighty percent is a common choice.
Only visitors who actually reach the changed element count. A change in the cart drawer is only seen by people who open the cart.
A worked example (an illustration, not a benchmark)
Say the page converts at 2.0%, and you want to detect a move to 2.2%, a 10% relative change. With a 5% significance level and 80% power, the standard two-proportion formula gives about 81,000 visitors per version, so roughly 162,000 in total. If 20,000 visitors a month reach that page, the test needs about eight months. These inputs are made up to show the math. Plug in your own numbers in a sample size calculator before you plan anything.
The lesson is not the exact number. It is that small stores testing small changes often wait months for a result that still says "no clear difference".
Why do low-traffic stores struggle to run valid tests?
Because there are not enough orders to separate signal from noise. A few lucky days can make a version look like a winner. The usual mistakes make it worse:
- Stopping when the dashboard turns green. Checking again and again and stopping at the first good-looking result makes false winners much more likely.
- Running for a few days. Weekday and weekend shoppers behave differently, so we run tests in whole weeks.
- Testing tiny changes. A new button color needs a huge sample to show anything.
- Changing other things mid-test: a sale, a new ad campaign, a theme update.
What should I do instead of A/B testing?
Work in this order:
- Fix obvious problems. Broken add-to-cart on a phone, a slow product page, a confusing shipping message. You do not need a test to fix a bug.
- Do research. Watch session recordings, read support emails and reviews, and ask a few customers why they almost did not buy.
- Then test, if you can. Test bold changes based on what the research found, on pages enough people see.
Ship small, low-risk improvements without a test, and watch the numbers before and after. That is not proof, but it is honest about what your traffic can tell you.
How do you set up an A/B test properly?
Before launch, write down:
- The hypothesis: what you change, for whom, and why you expect it to help.
- One primary metric, such as orders per visitor, decided before the test starts.
- Guardrail metrics that must not get worse, such as refund rate or page speed.
- The sample size and test length in whole weeks, worked out from your own traffic.
- How traffic is split, and the check that the split came out as planned.
- The decision rule: what result means ship, what means do not ship.
If the test uses separate URLs, follow Google's guidance: no cloaking, a canonical tag pointing at the original, and temporary 302 redirects, not 301s.
What we do
Plan the test.Then trust the result.
Plan
2 services
- 01Can you test? checkVisitors per page, current conversion and the smallest change a test could detect in a sensible number of weeks.Visitors per page, current conversion and the smallest change a test could detect in a sensible number of weeks.
- 02Hypothesis and metricsOne hypothesis, one primary metric, guardrails and a decision rule, written before launch.One hypothesis, one primary metric, guardrails and a decision rule, written before launch.
Build
2 services
- 03Variant buildBoth versions built on a duplicate theme or in Shopify Rollouts and checked on real devices.Both versions built on a duplicate theme or in Shopify Rollouts and checked on real devices.
- 04Tracking and split checksEvents that match Shopify’s order data and a traffic split that comes out as planned.Events that match Shopify’s order data and a traffic split that comes out as planned.
Read
2 services
- 05Honest readoutRead at the planned end against the rule set up front, including "no clear difference".Read at the planned end against the rule set up front, including "no clear difference".
- 06Ship and clean upThe winning version made permanent, test code and extra URLs removed.The winning version made permanent, test code and extra URLs removed.
How we work on it.
How we decide what to test, if anything
We start with the numbers you already have. How many visitors reach the page? How many convert? From that we work out the smallest change a test could detect in a sensible number of weeks. If that change is unrealistic, we tell you, and we do not run the test.
Then we rank ideas by evidence, not opinion. A problem we can see in recordings, support emails or the funnel goes ahead of a hunch. Bugs and clear friction get fixed straight away. Only ideas that are both uncertain and important become tests.
We also check that measurement is solid first. If orders and add-to-cart events do not match what Shopify reports, we fix tracking before any test.
What a typical first month looks like
- Week one: we check your tracking, your traffic per page, and any tests already running. We give you a plain answer: can this store test right now, and on which pages?
- Week two: we review the research you have and do a little more. We write a short list of hypotheses, each with a primary metric and the sample it needs.
- Weeks three and four: if a test is feasible, we build both versions on a duplicate theme or in Shopify Rollouts, check them on real devices, confirm the traffic split, and launch. If no test is feasible, we fix the top problems from the list instead.
- On the 1st: your report says what is running, how long it has left, and what we changed.
When the test ends, we read it against the rule written before launch. A result of "no clear difference" is a real answer, and we say so.
What we will not do
- Call a test early because it looks good. We wait for the planned sample, in whole weeks.
- Run a test your traffic cannot finish. We would rather fix things and tell you why.
- Promise a lift. A test tells you what happened, not what you hoped.
- Test fake urgency, fake stock counts, pre-ticked add-ons or other dark patterns.
- Report a win from a single metric when a guardrail got worse.
- Name or push a testing vendor. We use Shopify's tools where they fit, or the app you already have.
Related work
Testing is the last step of conversion work, not the first. A CRO audit or landing page audit finds what to fix. The upsell and AOV audit covers offers in the cart and after purchase. Checkout tests on Shopify Plus may involve checkout and Functions work, and a slow variant can sink a test, so speed work often comes first. It all runs inside our care plans. The free health check tells you where your store stands today.
How it works
From first look to a monthly rhythm.
Health checksample
yourstore.com- Speed
- Script tags
- Apps
Step 1: Free health check
We scan your store and record a short walkthrough of what we find.
#Priority fixessample
- 01Remove unused app codeHigh
- 02Compress hero imagesHigh
- 03Fix cart drawer on mobileMed
Step 2: Three priority fixes
You get the top three fixes and a plan recommendation. Keep the report either way.
Collaborator accesssample
- Themes
- Apps
- Orders
Live theme backed up
Step 3: Safe access
Collaborator access you control. We back up your live theme and work on a duplicate.
Report · the 1stsample
- Done
- Cart drawer fixed
- Found
- Unused app script
- Next
- Image compression
Step 4: Report on the 1st
What was done, what we found, and what is next. In plain English.
Fit
What this is, and what it is not.
A good fit if
- Your key pages get steady traffic and you want to know, not guess, whether a big change helps.
- You have run tests before and are not sure the results were real.
- You want someone to tell you plainly when a test is not worth running.
Not a fit if
- You want a test that finishes this week on a low-traffic page. The math does not allow it.
- You want to test fake countdowns or hidden extras. We will not run those.
- You want a guaranteed winner. Most honest tests end with no clear difference.
Pricing
Testing inside a care plan.
Test planning, builds and readouts run on the monthly hours in any care plan: 30 on Care Essential, 80 on Care Growth, 160 on Care Plus. Care Growth adds a quarterly speed and conversion review, and Care Plus covers Functions and checkout extension work. A one-off change can be a One-Off Fix from $149 per task, priced before we start.
Care Essential
Small stores, under about $500K a year
$399/ month
- Reply
- 2 business days
- Monitoring
- Monthly health report
- 30 hours of fixes and tweaks a month
- Reply within 2 business days
Not included
- Hosting, security patches and core platform updates (Shopify handles these)
Care Growth
Most pickedGrowing stores, about $500K to $5M a year
$1,200/ month
- Reply
- 1 business day
- 80 hours of development a month
- Reply within 1 business day
- Quarterly speed and conversion review
- Shopify deadline migrations handled
- Monthly AI answer check: ChatGPT, Perplexity, Gemini
Not included
- Hosting, security patches and core platform updates (Shopify handles these)
Care Plus
Shopify Plus brands
$2,900/ month
- Reply
- Same day, critical issues
- 160 hours of development a month
- Same-day reply on critical issues
- Functions and checkout extension work
- Monthly planning call
- Monthly AI answer check: ChatGPT, Perplexity, Gemini
Not included
- Hosting, security patches and core platform updates (Shopify handles these)
Not ready for a plan?
Free Store Health Check
Every store. The best place to start.
Free
- Automated scan of your storefront
- Short recorded walkthrough of what we found
- Speed check
- Legacy script tags ahead of the March 1, 2027 cutoff
- App load
- Accessibility basics
- Broken tracking
Not included
- The fixes themselves. You choose who does them.
One-Off Fix
Small stores with one problem
from$149/ per task
- Fixed price agreed before work starts
- Done on a duplicate theme
- 7-day warranty
Not included
- Ongoing monitoring or a monthly report
- Hosting, security patches and core platform updates (Shopify handles these)
All prices in USD. Month to month.
Questions
What people ask about this
How much traffic do I need to A/B test on Shopify?
It depends on your conversion rate and the size of change you want to detect, not on a fixed number. Small changes on low-traffic pages can need months of visitors. Work it out with a sample size calculator using your own numbers before you start. If the answer is longer than you can wait, fix and research instead.
Can I A/B test my Shopify theme without an app?
Yes, on the Grow plan or higher. Shopify Rollouts includes an Experiment type that compares a treatment against a control, set up from Markets > Rollouts in your admin. Experiments only apply to the online store and do not work on vintage themes.
How long should an A/B test run on Shopify?
Until it reaches the sample size you planned, in whole weeks. Running in full weeks covers both weekday and weekend shoppers. Shopify Rollouts sets a default end date 90 days after an Experiment starts, but that is a default, not a recommendation.
Why shouldn’t I stop an A/B test as soon as it shows a winner?
Because checking again and again and stopping at the first good-looking result makes false winners much more likely. Decide the sample size before you start and read the result only when the test is done.
Does A/B testing hurt my Shopify SEO?
Not if it is set up properly. Google asks you not to show search engines a different page than shoppers, to point variant URLs at the original with a canonical tag, and to use temporary 302 redirects. Remove test code and extra URLs when the test ends.
What should a small Shopify store do instead of A/B testing?
Fix obvious problems first, such as bugs, slow pages and unclear shipping costs. Then research why people do not buy, using recordings, support emails and short customer surveys. Ship low-risk fixes and compare before and after, and keep testing for bold changes on pages with enough traffic.
Get started
Find out if your storecan run a real test.
Book a call with your traffic numbers and we will tell you, live, whether a test can finish. Or start with the free health check.
Sources
- 1. Shopify Help Center: Rollouts (checked 2026-11-17)
- 2. Shopify Help Center: Types of rollouts and changes (checked 2026-11-17)
- 3. Shopify Help Center: Requirements and considerations for using rollouts (checked 2026-11-17)
- 4. Evan Miller: How Not To Run an A/B Test (checked 2026-11-17)
- 5. Google Search Central: Minimize A/B testing impact in Google Search (checked 2026-11-17)