# Skill: test-architect

## Purpose

Takes a performance hypothesis — "I think the hook is wrong", "I want to test a different offer", "should I try a shorter video?" — and produces a **complete A/B test structure**: what to change, what to hold constant, budget split, run duration, success metric, and an explicit decision rule for calling a winner. The output is a test brief the marketer can hand to anyone and have executed correctly.

This skill does not run the test or monitor its progress. It designs the test and defines when to call it.

> **Read-and-recommend only.** This skill never creates or modifies ads. All test setup is applied manually by the user.

---

## When to Use This Skill

Use when the user says:
- "I want to test [X] vs [Y]"
- "Should I try a different [hook / offer / format / audience]?"
- "How do I know if [ad / campaign] is the problem?"
- "Set up a test for me"
- "What should I be testing this week?"

---

## Step 0 — Hypothesis Clarification

Before designing the test, the hypothesis must be precise. A vague hypothesis produces an uninterpretable test.

Ask the user:
1. **What is the single variable being tested?** (Only one variable per test. If they want to test two things, design two tests.)
2. **What is the current state (control)?** (The thing being run now that you're comparing against.)
3. **What is the proposed change (variant)?** (The one thing that changes.)
4. **What is the account's current performance context?** (Spend rate per day, approximate CPA or ROAS target, conversion volume per week.)

If the user says "I want to test everything," clarify: good testing changes one thing at a time. Ask them to rank which variable they believe is most likely to be the bottleneck, based on the current data.

---

## Variable Classification

Only the following variables are valid test subjects for this skill. Anything outside this list requires a different approach (e.g. campaign-level restructuring, audience strategy).

| Variable | What Changes | Platform |
|---|---|---|
| Hook | First 3 seconds of video, or first frame of image | Meta / Google Display |
| Body / middle | Seconds 4–20 of video, or body copy of image/carousel | Meta / Google Display |
| Offer | Discount amount, free trial length, lead magnet, risk reversal | Meta / Google |
| CTA | Call to action text, button, spoken phrase | Meta / Google Display |
| Format | Video vs image, 30s vs 15s, carousel vs single | Meta |
| Angle | Entirely different message (pain point vs aspiration vs social proof) | Meta / Google Display |
| Audience | Broad vs LAL vs interest, or new vs retargeting | Meta |
| Match type | Broad vs phrase vs exact on the same keyword | Google Search |
| RSA headline | One headline variant vs another in a Responsive Search Ad | Google Search |

**If the test involves changing more than one of these at once → split into two tests and note the dependency.**

---

## Sample Size Requirements

The test must run long enough to reach statistical confidence. Use these minimums before calling a winner.

**For conversion-based tests (ROAS, CPA, purchase):**

| Spend rate | Minimum events per variant before calling | Minimum duration |
|---|---|---|
| Low (< $100/day) | 30 conversions per variant | 21 days minimum |
| Medium ($100–$500/day) | 50 conversions per variant | 14 days minimum |
| High (> $500/day) | 100 conversions per variant | 7 days minimum |

**For engagement-based tests (CTR, Hook Rate, Hold Rate):**

| Test type | Minimum events per variant before calling |
|---|---|
| CTR test | 500 clicks per variant |
| Hook Rate test | 1,000 video plays per variant |
| Hold Rate test | 1,000 video plays per variant |

**Important:** If the account cannot reach these thresholds in a reasonable time (< 21 days), note this in the output and suggest either increasing the budget allocated to the test or narrowing the audience to concentrate spend.

---

## Budget Split Logic

| Scenario | Split | Rationale |
|---|---|---|
| Neither variant is live yet | 50 / 50 | Equal exposure, cleanest data |
| Control is currently performing at or above target | 70 / 30 (control / variant) | Protect revenue while testing |
| Control is clearly underperforming | 50 / 50 | No value in protecting a loser |
| Testing audience only (not creative) | Must use separate ad sets with identical creatives | Never test audience and creative simultaneously |

**Platform-specific setup:**
- **Meta:** Use Ads Manager A/B test feature (preferred) OR duplicate the ad set and run both in isolation. Do NOT run variants in the same ad set — Meta's algorithm will skew delivery to its predicted winner before statistical significance is reached.
- **Google:** Duplicate the ad, run in the same ad group for RSA/creative tests. Use Campaign Experiments for bid strategy or budget tests.

---

## Success Metric Selection

Pick exactly one primary success metric. The decision rule is based solely on this metric.

| Campaign type | Primary success metric |
|---|---|
| Ecommerce (purchases) | ROAS or CPA, based on which the user optimises toward |
| Lead gen | CPL (cost per lead) |
| App | CPI (cost per install) or cost per activation event |
| Awareness / top-of-funnel | CTR or Hook Rate |

Secondary metrics (e.g. CPM, frequency, Hook Rate for a conversion-focused test) are diagnostic only — they explain the result, they don't determine the winner.

---

## Decision Rule Construction

The decision rule must be explicit before the test starts. It prevents cherry-picking and post-hoc rationalisation.

**Decision rule format:**

```
WIN:          If [variant primary metric] beats [control] by ≥ 10% after [minimum events] → declare variant winner, pause control
LOSE:         If [variant primary metric] underperforms [control] by ≥ 10% after [minimum events] → pause variant, test new hypothesis
INCONCLUSIVE: If difference < 10% after [minimum events] → extend by 7 days, then call inconclusive — do not keep both running indefinitely
EARLY STOP:   If [variant] underperforms by ≥ 40% after 3 days AND minimum 100 impressions per variant → pause variant early (clear loser)
```

The 10% threshold is the minimum meaningful difference for budget decision-making. Tests that show < 10% difference are not worth acting on — the variables are likely not the bottleneck.

---

## Output Format

```
TEST BRIEF

Hypothesis: [what you're testing, in one sentence]
Variable: [the ONE thing changing]
Control: [current ad / current setting / current audience]
Variant: [what changes — be specific]

────────────────────────────────
SETUP
────────────────────────────────

Platform: [Meta / Google]
Test type: [Ad-level A/B / Ad set A/B / Campaign Experiment / RSA variant]
Budget split: [X% control / Y% variant]
Budget to allocate: [daily $ to the test — based on minimum sample requirement]
Minimum run duration: [N days]
Minimum events before calling: [N conversions / N clicks / N video plays] per variant

────────────────────────────────
WHAT TO HOLD CONSTANT
────────────────────────────────

[List every element that must NOT change between control and variant. This is as important as what changes.]
- Audience: [same targeting]
- Budget: [same daily budget per variant]
- Objective: [same campaign objective]
- Bid strategy: [same bid strategy]
- Landing page: [same URL]
- [Any other elements]

────────────────────────────────
DECISION RULE
────────────────────────────────

Primary metric: [ROAS / CPA / CPL / CTR / Hook Rate]
WIN:          If variant [metric] beats control by ≥ 10% after [N] [conversions/clicks/plays] → pause control, scale variant
LOSE:         If variant [metric] underperforms control by ≥ 10% after [N] events → pause variant
INCONCLUSIVE: If difference < 10% after [N] events → call inconclusive, form new hypothesis
EARLY STOP:   If variant underperforms by ≥ 40% after 3 days AND ≥ 100 impressions → pause now

────────────────────────────────
NEXT TEST (IF VARIANT WINS)
────────────────────────────────

[What to test next, building on the result of this test — e.g. if hook wins, test two hook variants; if offer wins, test offer format]

────────────────────────────────
NEXT TEST (IF VARIANT LOSES)
────────────────────────────────

[What to test next if this hypothesis is wrong — the next most likely bottleneck to investigate]
```

---

## Guardrails

- Only one variable per test. Refuse to design a test that changes two things simultaneously — the result will be uninterpretable.
- Do not design a test during learning phase (Meta: < 50 optimisation events in the last 7 days on the ad set). Wait for the existing campaign to exit learning before adding a test.
- Do not recommend testing on an account with < $30/day spend — the minimum sample size will take too long and the data will be noisy.
- Do not call a test winner before the minimum event threshold is reached, even if the early numbers look strong.
- Do not design audience tests and creative tests in the same test structure. They require different isolation setups.

---

## Minimum Required Data

| Source | What You Need |
|---|---|
| MCP or CSV | Current control ad/ad set: daily spend rate, current CPA or ROAS, current primary metric value, conversion volume per week. Needed to calculate minimum run duration and budget allocation. |
| Manual input | User's hypothesis (what they want to test and why), platform, campaign type, and current performance target. The test structure can be designed without account data if the user provides spend rate and target manually. |