Deprecated: Function set_magic_quotes_runtime() is deprecated in /home/mwexler/public_html/tp/textpattern/lib/txplib_db.php on line 14
The Net Takeaway: Stats are Meaningless, Pt II


Danny Flamberg's Blog
Danny has been marketing for a while, and his articles and work reflect great understanding of data driven marketing.

Eric Peterson the Demystifier
Eric gets metrics, analytics, interactive, and the real world. His advice is worth taking...

Geeking with Greg
Greg Linden created Amazon's recommendation system, so imagine what can write about...

Ned Batchelder's Blog
Ned just finds and writes interesting things. I don't know how he does it.

R at LoyaltyMatrix
Jim Porzak tells of his real-life use of R for marketing analysis.






Stats are Meaningless, Pt II · 06/11/2004 12:03 PM, Analysis Marketing

After my previous rant, I realized that I had said lots of this stuff before, and in a better way… So here is that repost. This one also focuses on email, but it is germane to almost any interactive marketing out there. It was in response to yet another request for “the industry open or click average”.


Ok, provocative title line to get your attention. No, stats aren’t meaningless.

But “Industry Averages” are often completely useless, at least for current e-mail marketing. Here’s why, in a collection of parts:

1) Are you measuring the right variable? When asking for clicks, folks often assume it as a proxy for conversion. But you need to know clicks:conversion ratios for it to be useful. So, don’t focus on the wrong metrics. And remember, it may take multiple messages to drive a conversion, so don’t try to tie a conversion to a single “do or die” message in every case. Finally, remember that different folks have different standards: some companies are upset with “only” a 10% conversion, while others are ecstatic with a 0.25%.

“OK, but I really do want to know clicks. Why can’t you just add them up and tell me, on average, what you get?”

2) Well, its that old apples and oranges thing again. There are so many factors to manipulate in email that I can create lots of different types of mails. Rolling all these different types of mails together in one pot doesn’t give you email, it gives you fruit salad, when what you really wanted was to understand apples vs. oranges.

Just off the top of my head, here are some factors differentiating mails. There are more, but just to get started…

Ah, but still some naysayers. “Who cares? Just lump it all up, it all comes out in the wash, and give me an average, you meany!” some cry. Well, not so easy.

3) Averages are highly impacted by outliers. This means that if I throw in my customized newsletters to my membership audience with a 30% click rate, then I can make all my other bad mails look great, “on average”. Should I take unique clicks and divide by net mailed? Take total clicks and divide by net mailed? Should I just average the click rates? Should I weight smaller mails? Should I weight larger mails? Perhaps a Mode or a Median?

Yes, we can make an “average” or some other Measure of Central Tendency, but it will have little to no meaning. Why? Because I will calc it differently from some other group, and some groups will include or exclude mails based on factors above. So, we will all have different answers… should we just average those averages? I think not.

The confusion is due in part to the fact that most folks aren’t asking the right question about their business. The question should be something like:

“For my industry, mailing the type of list and type of message I will be mailing to a certain type of list, what types of opens, clicks, and overall responses have others seen with this specific type of offer and this ‘conversion process’ (buy something, fill out a form, whatever)”?

That starts to get somewhere… but rarely do I hear folks ask that. And the overall answer isn’t some average, it might be a range of medians or a list of modes, showing comparisons across various factors of importance. For me, the answer is part of some ongoing research we have to group mails by type and then track them across our variety of clients. I know others are doing the same, and soon, we will start to be able to share and merge this data (I hope!). Look, even the DMA publishes a big book splitting their response rates by industry, media, and lots of other variables like the ones I mention above. Why would we (as an industry younger than direct mail) want to not learn from them?

And what’s the followthrough? Work with your own metrics to make them better. What can you change that is under your control to refine them? What specific groups are high or low, and what levers can you use to change their behavior? Don’t worry about what everyone else is doing; fix your own programs first and then everyone else will worry about what you are doing!

So, there is no industry average (and thank goodness). But over time, with the right questions and lots of cooperation with each other, we will start to better understand expectations for folks… and maybe, just maybe, we’ll have that correctly framed and calculated “Industry Average” (for your industry, business goal, etc.) after all.

* * *


  Textile Help
Please note that your email will be obfuscated via entities, so its ok to put a real one if you feel like it...

powered by Textpattern 4.0.4 (r1956)