Plan one A/B test before buying experimentation software
Turn one product question into a decision-ready experiment brief with two free local tools.
A one-page experiment brief with a hypothesis, control, one treatment, primary metric, guardrail, exposure and conversion notes, and a pre-agreed readout decision.
Start with the decision, not a platform subscription. It does not randomise traffic, collect events, calculate a winner or claim a tested result. The illustration is a FreeStack concept, not a provider screenshot.
About this guide & its examples
This original FreeStack exercise plans a fictional signup-button test in Excalidraw and Markdown. Official sources checked 22 October 2026.
Illustrations are FreeStack concepts, not provider screenshots. Check the free limits below and test your own files before delivery.
Free boundaries before you start
What’s free, whether you need an account, and when we last checked.
Excalidraw · checked 2026-10-21
The free whiteboard remains free forever with an infinite canvas, full editor, unlimited scenes, compressed image uploads, browser-stored personal libraries, link collaboration and PNG, SVG and editable JSON exports. The current comparison also lists 10 AI requests a day. Automatic cloud sync, workspace organisation, comments and managed teams are paid Plus features. Keep an independent source-file copy.
The repository does not explicitly establish current account requirements for every hosted sharing action. Do not infer no account from the MIT licence.
Visual Studio Code · checked 2026-10-21
Free for private and commercial core editing. The Microsoft distribution licence differs from the MIT Code-OSS source licence. Local Markdown editing and preview need no AI plan. Copilot, model services and extensions have separate accounts, licences and usage limits; no free hosted compute is included.
Official FAQ explicitly says installing and using core editor features needs no account. Connected AI or sync services have separate requirements.
A small example to aim for
Would a benefit-led label help eligible visitors complete signup?
Primary metric: completed signup rate
Guardrail: validation errors
Decision: use the agreed readout rule
One change.
One metric. One guardrail.
FreeStack experiment-brief concept · not a provider screenshot or tested result.
Illustrative example · adapt it to your project.
Jump to a step
Name the decision and the change
Write the product decision in one sentence: should the signup page use a specific benefit-led button label instead of the current generic label? Record a falsifiable hypothesis, the audience eligible for the test and the single visible change. Keep pricing, layout, form fields and traffic source unchanged in this first brief. If the treatment changes several things, split it before continuing so a result can still inform a decision.
Sketch control and one treatment
In Excalidraw, draw two small frames at the same size. Label the current version Control A and the proposed version Treatment B. Use fictional copy and no customer information. Add a note naming what is deliberately held constant. Export a PNG for the brief and keep the editable scene file. The sketch is a planning artefact, not production UI or evidence that either variation performs better.
Choose one primary metric and a guardrail
Write one primary metric that answers the decision, such as completed signup rate among eligible exposed identities. Add a guardrail that would stop a superficially positive result from hiding harm, such as validation errors or support contacts. Define the event names, denominator and time window in plain language. Do not copy a universal sample-size number from another project; traffic, baseline rate, minimum useful effect and risk determine the plan.
Specify identity, exposure and conversion
In a Markdown file in VS Code, state when an identity becomes eligible, when exposure is recorded and which event counts as conversion. Use the same stable, privacy-reviewed identifier across exposure and conversion records. Note exclusions such as staff or automated traffic. Flagsmith's current experimentation documentation joins exposure and conversion events by identified user and currently requires remote evaluation; that is useful source evidence, not a requirement to use Flagsmith for this planning exercise.
Write the readout and stop rules before launch
Define who will review the result, which pre-agreed evidence can trigger rollout, and what happens if the result is inconclusive or a guardrail worsens. Add checks for traffic allocation, sample-ratio mismatch, missing events and identities receiving more than one variation. Do not promise a date until the owner of the production system has estimated eligible traffic. A plan should make it possible to stop for bad data, not only to celebrate a winner.
Upgrade only when the running requirement is real
This brief is complete without an experimentation membership. FreeStack editorial upgrade triggers: pay when you need a named capability such as production traffic allocation, governed remote flags, reliable event ingestion, warehouse integration, statistical analysis or support with an agreed service level. Compare that requirement with a self-hosted route and its operating burden. Do not buy a platform merely to store a hypothesis that can fit in one reviewed page.
Your workflow, connected.
Follow the steps and carry the output into the next stage. These are suggested handoffs, not automatic integrations.
- 02Make it visible
Excalidraw's core editor is free and MIT-licensed; Excalidraw+ collaboration is a separate paid service.
A/B concept + editable source - 04Plan the evidence
VS Code core editing needs no account; extensions, sync and hosted AI services have separate terms.
Instrumentation notes
Provider information
Check current plans and documentation before committing to a workflow.
View 6 provider links
- Excalidraw: current repository and MIT core licence
- Excalidraw: free core editor and separate Plus service
- VS Code: free commercial use and no-account core editor
- Flagsmith experimentation overview: variations, metrics and event flow
- Flagsmith: exposure and conversion instrumentation boundaries
- Flagsmith: result quality checks and analysis terms
A couple of useful questions
Is this a real A/B test?
No. It is the decision and instrumentation brief that should exist before production traffic is split. A real test needs approved implementation, assignment, exposure and conversion logging, quality checks and analysis.
How much traffic do I need?
There is no honest universal number. Estimate it from the baseline rate, minimum effect worth acting on, allocation, chosen error or decision criteria and eligible traffic. Get statistical review when the decision carries material risk.
What stays free?
Excalidraw's core editor is open-source MIT software and VS Code core editing is free for private and commercial use. Your device and storage remain your responsibility. Managed collaboration, hosting, analytics, warehouses and experimentation platforms are separate.
What does Save this toolkit keep?
The existing FreeStack flow saves Excalidraw and VS Code to your private stack after sign-in. It does not store the experiment brief, create a feature flag, instrument events or analyse a result. Keep the editable files in your approved project location.
What would you try?
Share your result or ask a specific question about this workflow.
Join the conversation