Plan a landing-page A/B test before using a skill

Keep five decisions separate

The source spans planning, implementation, monitoring, result interpretation and promotion of winners. These are different gates, not one workflow approval.

No customer brief, analytics, eligible traffic or event data was available here. There is no feasibility label, sample target, live experiment, result or winner.

What the source supports—and where it stops

Quick table: arithmetic check, not power plan

The source’s two tables agree on overlapping cells. An independent conventional equal-allocation, two-proportion normal approximation found all 25 rounded expanded-guide cells within 5% of its outputs. For example, 1% baseline with 20% relative lift gives about 42,693 per arm versus the source’s 43,000; 5% with 20% relative lift gives about 8,158 versus 8,200. These are diagnostics of the source table, not estimates for a real campaign.

The actual calculation needs a defined eligible assignment unit and event denominator, baseline period and uncertainty, commercially justified MDE in absolute and relative terms, α, power, tails, arms and comparisons, allocation, conversion window, exclusions and statistical/stopping method. Traffic by calendar and exposure rate then inform collection time. Pageviews are not automatically independent participants. A larger MDE changes the effect a test can detect; it does not make the underlying page effect larger. The statsmodels power API makes allocation ratio and alternative explicit; the Miller calculator also asks for baseline, MDE, power and α. Neither was used to set a sample here.

Stopping and decision language need a plan

The selected file warns against peeking at a fixed-horizon primary result, then suggests stopping on a “significantly negative” guardrail. Weekly significance checks are repeated looks unless a safety or sequential rule, multiplicity and analysis treatment were specified in advance. Operational harm may justify a pause independent of a p-value, with a named owner and escalation threshold. Miller’s methodological note explains the false-positive risk of unplanned repeated testing; the ASA statement says p-values alone do not measure effect size or warrant a business decision. A non-significant result is not proof of equivalence. Record the effect interval, guardrails and uncertainty for analyst review, then make any release or rollback decision separately.

A first worksheet for one page

This is a proposed local review route, not a required contract in the skill.

1. Owner brief: exact page and one variant, eligible audience, primary conversion event, approved hypothesis and data access. If the change combines several promises, clarify the causal claim first.
2. Measurement: event numerator and denominator, unique assignment unit, attribution window, baseline and tracking QA owner. Defer quantitative testing if assignment or events cannot be verified.
3. Analyst design: business-worthy MDE, α, power, tails, arms, allocation, comparisons, guardrails, method and predeclared stopping rule. Missing values mean no sample, duration or feasibility claim.
4. Traffic and calendar: eligible unique units by day and source, exposure, seasonality, delays and opportunity cost. Analyst calculates a sensitivity range rather than applying quick-guide thresholds as rules.
5. Action owners: separate approval for implementation, tracking QA, operational pause, final inference and shipping. A memo is not a deployment instruction.

Why the worksheet comes first

The source offers convenient prompts and templates, but its package also reaches local context reads, deployment, monitoring and program cadence. An owner-maintained worksheet can include the same planning fields plus source provenance, units, missing-input states and named review without granting that wider scope. This is an inference from inspected instructions, not a head-to-head outcome: neither route was run. The source’s ICE/weighted priority schemes differ, and its experiment cadence, win-rate and “cumulative lift” figures were not supported as general norms in the inspected passages.

Revisit the skill only with an approved real brief, an immutable selected payload and host permissions reviewed. Compare it with the worksheet on omissions in event/unit definition, power and stopping, unauthorized reads or actions, correction time and analyst effort. Material, reliable improvement without added risk could justify a planning-only conditional review. No such evidence exists yet.

Provenance and limits

The mutable main selected file displayed version 2.0.3 on 28 October. The directly linked sample-size guide was returned with a same-day crawl label; the test-template page had a last-week label, so their exact revision alignment is unknown. These labels do not establish a selected-file SHA/date, immutable pin, complete package, installed behavior or current host permissions. No skill was installed or invoked; no local context, account, analytics, customer data, live page, test result or outcome was used.