ChatGPT Ads as a test channel: how much budget before you can judge it?

I keep running into the same problem with ChatGPT Ads. We have years of benchmarks for Google and Meta, but almost nothing for a surface where the user is in the middle of a conversation instead of scrolling a feed or typing a search query.

My hypothesis is that the intent signal could be strong while the path to conversion is harder to see. A click might be the end of a long research chat, so last-click could undervalue the channel, or credit it for something it only nudged.

So the real debate is about setup, not performance. One option is a small fixed test budget with a written stop rule, like a set cost per qualified lead or a set number of conversions by a date. The other is waiting until you have a proven baseline on another channel and then comparing. Waiting feels safer, but you may never get clean data if you only look at the channel after everyone else has already made up their minds.

How do you define a kill threshold for a channel where you don't yet trust the attribution? What would you measure first, and what would you ignore?

Elena, I don't think undervaluing the channel is the bigger risk. More often it goes the other way: a new surface takes credit for conversions that were already coming in through branded search or direct, and it looks great for a month. Check for that before you settle the budget.

Compare conversions from the new channel's landing page against branded query volume over the same dates. If both climb together, the channel may just be relabelling demand you already had instead of creating any.

On the kill threshold, a fixed sample size tells you more than a fixed date. At a 3 percent conversion rate you need roughly 1,000 clicks before cost per conversion means much more than noise. Size the test budget to get there, and write the stop rule before launch.

I'd keep that rule on cost per qualified lead, and decide now what to ignore. Assisted and view-through numbers should stay out of the judgement until the baseline is stable.

Theo, I like the branded-query check, but the sample rule needs tightening. At a 3 percent conversion rate, 1,000 clicks gives you about 30 conversions, and the 95 percent range around 30 is roughly a third either way. So a cost per lead that looks 40 percent better than target could still be noise. A strict stop rule should be based on conversions reached, not just clicks.

I'd also treat the first test as a creative test before it's a channel test. Say you run a dental clinic and your ad is a generic "book a consultation" line. Someone asking an assistant about clear aligners is in a different frame from someone scrolling a feed, and the message has to answer the question they're already asking. Judge the surface on one weak message and you might kill a channel that would have worked with a better hook. Split two or three angles inside the same budget, and only call the channel verdict once you know which message the traffic responds to.

Mira, the 30-conversion rule matters, but for a local service business the sample problem is harder than the maths suggests. A small clinic can take months to collect 30 qualified leads, and by then the season has moved on. I'd pair the conversion target with a hard spend cap so the test ends on whichever comes first, and accept that the verdict is only directional.

Theo, for local businesses the branded-query check needs adapting. The closest equivalent is usually Google Business Profile activity: calls, direction requests and website clicks from the profile. If those rise right alongside paid-channel calls, the new channel may just be relabelling demand you already had.

Before launch, give the new channel its own call-tracking number that isn't used anywhere else, and log how many calls you can't attribute. If that unattributed share is large, your cost per lead is closer to a guess than a measurement, and the stop rule should say so in writing.

Marco, on the unattributed calls, I'd go further than counting them: ask. One question at the start of each call, like "where did you first hear about us?", logged against the call, gives you a rough split between paid, profile and word of mouth. It's imperfect, but it turns a blank into a directional number you can argue about.

I'd also write the definition of "qualified" into the stop rule before launch and freeze it. If sales tightens what counts as qualified halfway through, cost per qualified lead moves for reasons that have nothing to do with the channel. Decide up front who's allowed to change the definition, and when.

Last, make the rule asymmetric. Killing the channel should be easy once the hard cap or stop condition is hit, but scaling it should be harder. One good stretch shouldn't be enough; promoting it should take a second window of data showing the same result, not one lucky month.

Mira, agreed that the stop rule should be asymmetric, but there's a check that should come before any kill decision: did the channel actually get to run? A new surface can under-deliver for reasons that have nothing to do with message or audience, like thin inventory for your topic, low eligibility, or a bid that never wins. If spend stays well below plan, a weak cost per lead tells you almost nothing. Log delivery next to the stop metrics each week, say spend against planned spend and the share of eligible requests you appeared in, and write the rule so a verdict only counts once delivery hits a set level.

Marco's "where did you first hear about us?" question is useful, but I'd take the answers with a grain of salt. People tend to name the last thing they remember, which is often the most recent touchpoint rather than the first. It still beats a blank field, as long as nobody reads the split as precise.