AI ad copy: paste your past winning headlines or write from the brand brief?

When you ask an AI to write ad copy, do you paste in your best-performing headlines and descriptions, or just give it the brand brief and product facts? I lean toward the brief first. Past winners tend to get copied almost word for word, so every new batch sounds like the last one and you lose the variety you need for testing. Say a dental clinic pastes ten headlines that all lead with "Same-day appointments." The model will keep that angle even when the offer changes. Showing two or three winners still helps with tone, though, so maybe use them only as style references and tell the model not to reuse their phrasing. Has anyone found a cleaner way to split those two?

Agree with the split. I'd treat the winners as a list of what the audience actually responded to (a number, a price, a named service) rather than as wording to copy. Have the model pull those out first, then write fresh copy around them, and the "same-day appointments" trap mostly goes away.

Pulling out the "what worked" elements first is a good move. One rule of thumb I'd add: give the model only the winners for the first pass, then use its new batch as the reference for the next round, so the winners never anchor more than one cycle. Otherwise the best of the old copy quietly turns into the template for everything that follows.

Agreed on the one-cycle rule. When you pull out "what worked," log the element and the metric it moved (say, "price in headline, CTR up") instead of just the phrase. That way the next batch can test a different angle on the same lever instead of reshuffling the old words.

I'd push back on logging CTR as the metric. A price in the headline can lift CTR and still attract people who never book, so the lever looks like a winner when it isn't. Log the element against cost per conversion or conversion value instead, otherwise the next batch optimises for clicks rather than money.

Agreed on cost per conversion. For e-commerce I'd log value per click or ROAS as well, because a price in the headline often pulls in bargain hunters with smaller baskets. CPA can look fine while margin quietly drops. A decent rule of thumb is that a lever only goes in the "what worked" log if it beat the control on cost per conversion and each variant had roughly 30 conversions or more. Below that you're mostly logging noise, and the next batch inherits it.

I don't buy the 30-conversion floor as a hard cutoff. At 30 conversions the count alone carries about ±18% noise at one standard deviation, so a 15% CPA gap between variants can easily be luck. The reverse problem is worse: a lever that clears 30 but wins by a thin margin gets logged as a winner and then anchors the next batch.

I'd log the gap together with its range instead, something like "price in headline, CPA 12% lower, range roughly -30% to +6%," and mark it tentative until a second test confirms it. A dental clinic doing eight bookings a week will never hit 30 per variant in a month, so a fixed number would mean nothing ever gets logged. Treat the threshold as a rough direction to check, not a gate.

The noise number looks too small to me. ±18% is the uncertainty on a single count of 30 conversions. A CPA comparison is the difference between two counts, and their errors add up, so the uncertainty on a 15% gap is closer to ±25%. At 30 conversions per side, a 15% gap could easily be zero in either direction, and the range you'd log (-30% to +6%) is probably too narrow.

I'd also log the raw conversion counts for both arms and the dates they ran, next to the range. That way anyone reading the log, including the model writing the next batch, can see how thin the evidence was instead of trusting a percentage. A dental clinic doing eight bookings a week can still record that. It also stops a tentative lever from being treated as settled just because it has a tidy number next to it.

Agreed on the widened range. Your quadrature point matters most, since a 15% gap sits well inside one sigma. I'd also add a sample-size check to the log, because it tells you what the next test has to be. At the usual 95% confidence and 80% power, a 15% CPA gap needs roughly 800 conversions per arm before it's readable, and a 30% gap needs about 230 per arm. For a clinic doing eight bookings a week, that's around 100 weeks for the small gap and about 30 for the big one. So most "winners" from tests at that volume should be logged as "too small to call." Bookings also cluster by weekday, so run both arms over whole weeks and record the start and end dates, like you suggested. Otherwise a Monday spike can look like a lever.

Your per-arm counts of 800 and 230 check out for 15% and 30% gaps. But if eight bookings a week is the total, a 50/50 split gives each arm four a week, so the timelines double to roughly 200 weeks for the small gap and about 58 for the big one.