Shopify Flow / Improvement (Apps)

With "real shop data,"
you can now test your workflows

You can now specify real records, such as past orders and customers, as test events. On top of that, Sidekick uses AI to analyze branch paths and automatically generate test cases.

What's on this page
  1. What's actually changing (in 30 seconds)
  2. How it works: the flow up to running a test
  3. What you can do with "real-data testing"
  4. Before vs. this update
  5. How it works in practice (4 steps)
  6. 5 points developers should know
  7. 3 practical use cases
  8. A one-line summary for your pitch

1What's actually changing

Shopify Flow workflows can now be tested using "real shop data.".
And with the "Generate test events" button, Sidekick analyzes the workflow's branch paths and automatically generates test cases.

Before: run with dummy data

To test, you had no choice but to hand-write plausible data and run it — never quite sure it would really work.

Now: pick real orders and customers

You can pick specific records to test, like "last week's fraudulent order" or "a particular customer." Sidekick gathers them automatically and covers every branch, too.

2How it works: the flow up to running a test

Workflow Branch logic Analyze Sidekick Extract branch paths Shopify AI Search Real shop data order #1023 (fraud) order #1024 (standard) customer A-201 Past orders / customers Candidate generation Test event list case 1: #1023 → expected: blocked case 2: #1024 → expected: passes case 3: A-201 → expected: tag applied You can edit, delete, and add them
Sidekick reads the if branches / conditions in the workflowand finds "which real data would take this path."
The generated cases are reviewable, editable, and deletable, and you can also add your own cases. You can run the test right after generation (no extra setup needed).

3What you can do with "real-data testing"

Regression testing

Confirm "will it stop next time"

Replay last week's fraudulent order to confirm that the block workflow you built actually fires.

False-trigger check

Confirm "it doesn't stop the others"

You can add a test to check that the block doesn't misfire on normal orders.

Branch coverage

Cover paths automatically

Sidekick analyzes the branches in the workflow and pulls candidate data to exercise each one.

4Before vs this update

ItemBeforeThis update
Test data Hand-built Enter dummy values each time Real data Picked from the shop's past records
Identifying branch paths A person reads the workflow and decides Sidekick analyzes automatically
Creating test cases Build them one by one by hand Bulk-generate with "Generate test events"
Editing after generation — Edit, delete, or add your own
Setup Prepare separate test data no further setup required Run instantly with a button

5How it works (4 steps)

1

Open the workflow

Open the Flow workflow you want to test.

2

"Generate test events"

Sidekick analyzes the branches and automatically generates cases populated with real data.

3

Review the cases

Delete the cases you don't need and add the ones you do. Manual selection happens here too.

4

Run the test as-is

Verify behavior instantly with no extra setup.

As the changelog states, "You can test it immediately, no further setup required." — being able to run it as-is right after generation is a major operational improvement.

65 points engineers should know

AI

1. Sidekick is the core engine

Test case generation is handled by Shopify's AI assistant, Sidekick. It reads the workflow's branching structure and finds, from your real data, the records that exercise each path.

2. Tests run against "real data"

You can specify actual shop records such as orders/customers. While this inherently makes near-production validation possible,referencing production datais something to keep in mind in your operational policy.

3. Generated results can be edited and added to

It explicitly states, "Review the generated cases, edit or remove any that don't apply, and add your own." It's designed on the premise that you won't leave everything to the AI, but will add and remove the cases you need from an operational standpoint.

4. You can build both the "fires" and "doesn't fire" sides

Both "fraudulent order → block fires" and "normal order → no block" positive and negative casescan coexist in a single workflow. This matters from a regression-testing standpoint.

5. No mention of API or supported conditions

The changelog body supported plans, supported regions, and whether it can be operated via APIis not mentioned. For needs such as automation via Admin GraphQL or invoking it from CI, confirm separately in the documentation(the article provides reference links to the documentation and the Shopify community).

73 use cases you can apply to your work

!
USE CASE 1

Setting up regression tests for a fraudulent-order-blocking workflow

Challenge
We built a "block the next fraudulent order" workflow, but it was hard to confirm whether it actually fires and whether it was catching normal orders too.
Solution
Test by specifying past fraudulent orders, plus add a "doesn't fire on normal orders" test → keep bidirectional test cases permanently on hand within Flow.
Impact
Reduces verification cost when revising workflows. Sidekick's generated cases also make it easy to ensure branch coverage.
Technical notes
By deliberately pairing positive cases (block fires) and negative cases (passes through), it becomes easier to catch 'unnoticed false triggers' when operations change.
Sidekick
USE CASE 2

Comprehensive testing of complex branching workflows (VIP detection / restocking / return triage, etc.)

Challenge
Workflows with many conditional branches (VIP customer detection, auto-replenishment, return routing, etc.) are prone to missed paths when you rely on imagination alone.
Approach
Use 'Generate test events' to auto-generate cases per path → then keep or discard the resulting cases based on whether they 'happen in reality or not.'
Impact
Cuts the labor cost of test design while still surfacing gaps. Effective against the cold-start problem when building something new.
Technical notes
Sidekick's suggestions are not meant to be adopted as-is (the article itself says 'edit or remove'). Build a review step into your operational flow.
Delivery documentation
USE CASE 3

Incorporate test events into deliverables as 'evidence of guaranteed behavior' for client projects.

Challenge
Even after building and delivering a Flow, the client had nothing to judge whether it 'really works as intended,' so they had to rely on verbal explanations.
Approach
Register representative cases in Flow using Sidekick generation plus your own additions, and hand over the set of test events together with the workflow.
Impact
Makes it easy for clients to verify and decide on revisions themselves after delivery. Can also be reused as reproduction data when reporting bugs.
Technical notes
The test events themselves become a 'living sample of the spec.' They can be positioned as documentation that complements the requirements definition.

8One-line summary you can use in a proposal

"You can now test Shopify Flow workflows with real shop data, and
on top of that, Sidekick analyzes branch paths and auto-generates test cases.
— letting you verify both 'fires / doesn't fire' against real data, with no extra setup and instant on-the-spot runs."