You’re three proposals deep, every agency deck has a “results” slide, and every slide has an impressive-looking number on it. “+187% revenue.” “3x conversion rate.” “212% ROAS.” At some point you have to ask the obvious question: how much of that is real, how much of it is the agency’s work specifically, and how much of it would have happened anyway?
This matters more with Shopify agencies than it might for other services, because Shopify work spans so many disciplines, theme development, apps, SEO, paid media, CRO, email, and a single case study can quietly blend credit from all of them, or from a general trading tailwind, into one headline number. A vague or unverifiable case study isn’t automatically a red flag on its own, but it should change how you weight everything else the agency tells you.
This post gives you a practical framework for reading agency case studies properly: what questions to ask about attribution, what proof to actually request before you sign, and what a genuinely credible case study looks like versus one built to impress rather than inform.
Why Shopify Case Studies Are Easy to Fudge
Ecommerce results are noisy by nature. Revenue moves with seasonality, ad spend, competitor activity, currency shifts, stock availability, and general market conditions, completely independent of whatever the agency did. A Shopify store that grew 40% year-on-year during a period when the agency rebuilt its theme might have grown 35% anyway just from a stronger paid media budget the client funded separately.
On top of that, Shopify projects are rarely single-variable. If an agency ran a Shopify Plus migration, redesigned the theme, fixed technical SEO, and launched a Klaviyo flow in the same quarter, attributing a conversion rate lift to any one of those is genuinely difficult, even for the agency itself, working in good faith. The problem isn’t always dishonesty. It’s often that nobody isolated the variable being claimed.
That’s exactly why you, as the buyer, need a framework rather than relying on the headline metric.
The Core Attribution Questions to Ask
Before you take any number at face value, ask the agency these directly. A credible partner will answer specifically; a shaky one will get vague or defensive.
1. What was the baseline, and over what period?
“Conversion rate increased 24%” means nothing without knowing the before-and-after window and whether it’s comparing like-for-like periods (same season, same traffic mix, same promotional calendar). Ask for the actual before/after dates.
2. What else changed at the same time?
Did the client increase ad spend? Launch a new product line? Get a PR mention? Change pricing? If the agency can’t tell you what else was happening in the business during that period, they likely haven’t controlled for it, which means the number is a correlation, not a causation.
3. Is this the agency’s number or the client’s number?
Ideally, results should be pulled from Shopify Analytics, GA4, or Google Search Console, tools the client controls, not just self-reported in a testimonial. Ask whether the figures can be traced back to a dashboard.
4. Was this a full-funnel project or one lever?
If the case study covers “SEO,” check whether organic traffic and organic conversions moved, not just overall revenue. Overall revenue could easily be driven by paid media running in parallel. A trustworthy case study isolates the metric that maps to the service being sold.
5. What was the agency’s actual scope?
Sometimes an agency shows a client’s full transformation but was only responsible for one workstream, say, the theme migration, while a different vendor ran the SEO and ads that produced the headline growth number. Ask plainly: “What specifically did your team do, and what did other vendors or the in-house team handle in the same period?”
6. Can you speak to the client?
This is the single highest-value verification step and the one agencies most often try to avoid or delay. A reference call, even a 15-minute one, tells you more than any case study PDF.
A Practical Checklist Before You Sign
Use this as a working checklist when you’re down to your shortlist of two or three agencies:
- [ ] Ask for the raw before/after dates behind every headline number in their case studies
- [ ] Ask what else was happening in the client’s business during that window (spend, promotions, seasonality)
- [ ] Ask whether the metric is from a source you could independently verify (GA4, Shopify Analytics, GSC)
- [ ] Confirm exactly what the agency’s scope was versus other vendors involved
- [ ] Request one live reference call with a past or current client, ideally in a similar category or size bracket to your store
- [ ] Ask to see one example of their actual deliverable, a Liquid template, an SEO audit, a build spec, not just a results slide
- [ ] Check whether the case study is recent and reflects the current Shopify platform (a 2019 theme build case study tells you little about how they work with Online Store 2.0 or Shopify Plus today)
- [ ] Ask what didn’t work in a past project, and how they handled it
That last point is worth dwelling on. Any agency that has run enough Shopify projects has had one go sideways, a migration that slipped its timeline, an app integration that needed rework, an SEO change that temporarily dipped rankings before recovering. If a prospective partner has zero examples of something not going to plan, that’s less reassuring than it sounds. It usually means they haven’t done enough volume of work to have hit friction yet, or they’re not being fully candid.
What a Genuinely Credible Case Study Looks Like
The best case studies read less like a highlight reel and more like a technical brief. They typically include:
- A clear starting problem, stated specifically (e.g. “mobile conversion rate was 40% below desktop” rather than “the store wasn’t performing well”)
- The scope of work, broken down by what was actually built or changed
- A defined measurement window, with the metric tied to a specific, checkable source
- Context on what else was happening, so you can judge how much of the movement is attributable
- Honest caveats, such as “organic traffic recovered within six weeks after an initial dip during migration,” which is a completely normal and expected pattern, not a failure
If you’re doing this evaluation properly, you’ll naturally end up wanting to see a body of work, not just one or two hero examples. That’s the point of a proper case studies library, it should let you filter by industry, project type, and scope so you can compare like-for-like rather than judging an agency on whichever single result they chose to lead with.
Red Flags vs Green Flags in a Case Study
It helps to have a quick mental checklist you can run through while an agency is talking you through their portfolio, rather than trying to remember every question above on the spot.
Red flags:
- The number is a single headline figure with no dates, no baseline, and no source mentioned
- “Revenue” or “growth” is used without specifying whether it’s overall store revenue, a specific channel, or a specific product line
- The case study never mentions what the agency’s actual scope was, only the client’s overall outcome
- Every case study in the portfolio shows a spectacular result, with nothing showing a modest, realistic, or slow-building outcome
- The agency gets evasive, changes the subject, or drags out timelines when you ask for a reference call
- Screenshots of dashboards are cropped tightly enough that you can’t see the date range or the metric definition
Green flags:
- Specific before/after dates and a stated measurement window
- A clear breakdown of what the agency was responsible for versus what other vendors or the in-house team handled
- A mix of results, some strong, some modest, which tends to signal an honest sample rather than a curated highlight reel
- Willingness to set up a reference call without excessive delay or conditions
- Case studies that describe the actual technical approach (e.g. “we rebuilt the collection filtering using Shopify’s native Search & Discovery app instead of a third-party filter app to cut page weight”) rather than only the outcome
- A results claim tied to a metric that logically maps to the service performed, an SEO case study citing organic sessions and organic revenue, not just total site revenue
None of these signals is conclusive on its own. But if you’re seeing three or four red flags across an agency’s portfolio and proposal, that’s a pattern worth taking seriously, even if the headline numbers look appealing.
Why This Matters More for Shopify Plus and Migration Projects
The stakes for getting attribution right scale with the size of the engagement. A small CRO tweak that underdelivers is a minor loss. A Shopify migration or a move to Shopify Plus that’s sold on inflated case study claims is a much bigger problem, because you’re often locked into a platform decision, a rebuilt tech stack, and a multi-month timeline before you find out whether the agency’s claimed expertise matches reality. For projects at that scale, the attribution questions above aren’t optional diligence, they’re closer to a prerequisite before you commit a budget that size.
When to Bring In a Specialist to Sanity-Check a Proposal
If you’re not confident reading Shopify-specific technical claims, whether a theme migration timeline is realistic, whether an app stack makes sense, whether an SEO claim is technically sound, it’s worth getting an independent second opinion before you sign anything substantial. This is particularly true for larger projects like a Shopify Plus migration or a full replatform, where a bad agency fit is expensive to unwind. A short consulting engagement or an independent audit of the proposal itself can pay for itself many times over if it stops you from committing to the wrong partner.
FAQ
How many case studies should I expect an agency to show me?
There’s no fixed number, but you should expect at least two or three in your general category (similar store size, similar industry, or similar project type) rather than a single flagship example repeated across every pitch. If an agency only has one strong case study, ask directly how many similar projects they’ve completed and why the others aren’t showcased.
Is it reasonable to ask for a client reference call?
Yes, and any established Shopify agency should be able to arrange one, sometimes with a short delay to check with the client first. If an agency refuses outright or keeps deferring indefinitely, treat that as a signal worth weighing against everything else in the proposal.
What if the agency’s case studies don’t mention specific numbers at all?
That’s not automatically disqualifying. Some agencies avoid publishing numbers for confidentiality reasons tied to client agreements. In that case, lean more heavily on scope descriptions, technical specifics, and a reference call rather than dismissing the agency outright.
Should I be suspicious of very large percentage results, like “300% increase in conversions”?
Large percentage jumps are more believable from a very low base (a store converting at 0.4% jumping to 1.2% is a 200% increase but still a modest absolute number) than from an already healthy baseline. Always ask for the absolute before-and-after figures alongside any percentage claim.
Does a lack of case studies mean an agency is inexperienced?
Not necessarily, some capable teams are newer, work under NDA-heavy enterprise contracts, or simply haven’t invested in marketing their past work. In that situation, weight the reference call and a technical interview with the actual people who’d work on your project more heavily than published case studies.
Next Step
If you’d like a second opinion on a Shopify agency proposal you’re evaluating, or want an independent read on your current store before you commit budget to a new project, book a call with our team or start with a Shopify audit. Either way, you’ll go into the decision with a clearer picture of what’s realistic.
