Deprecated: Function set_magic_quotes_runtime() is deprecated in /home/mwexler/public_html/tp/textpattern/lib/txplib_db.php on line 14
The Net Takeaway: ExactTarget still observing...


Danny Flamberg's Blog
Danny has been marketing for a while, and his articles and work reflect great understanding of data driven marketing.

Eric Peterson the Demystifier
Eric gets metrics, analytics, interactive, and the real world. His advice is worth taking...

Geeking with Greg
Greg Linden created Amazon's recommendation system, so imagine what can write about...

Ned Batchelder's Blog
Ned just finds and writes interesting things. I don't know how he does it.

R at LoyaltyMatrix
Jim Porzak tells of his real-life use of R for marketing analysis.






ExactTarget still observing... · 04/26/2006 02:17 PM, Analysis

Chris Baggott, of ExactTarget, responded to my post pointing out the myriad issues in his company’s recent study. (I don’t think I can cram any more useful links in that sentence).

I appreciate that he encourages folks to read my post, but at the same time, he doesn’t fully communicate an understanding of the issues I raised. He runs the company, so its no surprise he chooses to defend the work instead of acknowledging its potential issues. But as you will see after finishing this post, if I were him, my response would have resembled the following: “We recognize that the industry is moving past the need for observational studies, so we release this as a first and last baseline. We intend to invest heavily is a rigorous round of testing with our clients, and we invite the industry to do the same. Its time to start learning instead of just observing”.

Instead, he says “This guy makes some decent points about some of our industry studies, but he is dismissing the value of any learning that comes from macro level studies. There is tremendous value in looking at macro-level information, there is also value in conducting controlled experiments.”

Ok, if you haven’t yet read my post If you can’t do it right, add more sample!, now is a good time.

So, do I dismiss the value of observational studies? It really all depends on what question you are trying to answer. If your question is, “Are there differences in behavior across mailers such as when they mail and their average performance of their mails?”, well then yes, an observational study gets you that answer.

But you know, I think we’ve milked that one dry. That is, I think we know that different mailers do things differently and get diferent results. So, while the general public will continue to buy any magazine with Britney on the cover, even if it has the same news over and over again… I expect the email industry to be better, and to not accept “yet another observational study”. To me, this study is more SSDD.

I think the industry is at the point of wanting to know why the differences occur. Certainly the collection of clients working with e-Dialog do. They ask questions around causation: What did different mailers do differently? What was it about those differences which most impacted performance? What specific things cause, either on their own or in interaction combos, a change in performance? If these are the research questions of the time, why do we waste our time with “yet another observational study”?

Baggott says “We always advocate testing.” If so, why do an entire observational-only study? Why does the advocation occur in exactly 1 line of the 10 page PDF, “Individual organizations still must conduct their own testing to determine which day of the week works best for them, but any test should consider Friday and Sunday as viable challengers.” on page 6… and nowhere else?

I have higher expectations for the remaining independents. I expect companies like e-Dialog, ExactTarget, and Responsys to be more nimble, more smart, more ahead than the big ol’ database companies. They are full of smart folks thinking about the issues, and they need to educate the industry about how to use email effectively. I don’t think more observational studies are the way to do this.

Ok, lots of whining, big mouth… but if this “observational” study is not as helpful as I want, what other kind of studies are there? For those who aren’t familiar with basic experimental design (probably most of you with a life, unlike me), I will summarize.

Usually, we perform research to answer a question. We phrase the question often in an expected answer, a prediction of the outcomes (yes, it does sound like an inverted Jeopardy). This is known as a Hypothesis. The study is an attempt to support or disprove (if possible) your hypothesis. The hypothesis is usually stated in a way to reflect that one thing impacts another, i.e., “a more forceful call to action will result in an increase in conversion” or even ”...a 10% increase in conversion”. (By the way, sometimes we hypothesize that nothing will happen with the hope that we’ll be wrong, so its really all in how you phrase the question.) So, the whole point of the research is to test that assumption of causation (or, if we can’t, we fall back to “relationships”).

There are two main types of studies out there: Observational (which Mr. Baggot calls “macro-level”, a mis-use of the economic term referring to “between groups or organizations”) where one observes behavior and groups it by descriptions of the behavior and individuals performing it (say, by gender or age, or by “heavy spend” or “light spend”), and Experimental, where most conditions are controlled or randomly distributed, and specific aspects of interest (such as offer, promo level, etc.) are manipulated. If our hypothesis is around causation, then the experiment is really the only way to provide strong support (and to be honest, multiple experiments are often required). Its the only process which controls for as many other factors (factors that aren’t of interest to the question) as possible to make sure that if a cause is present, we can see its effect. While we can lend some support or remove some support for our hypothesis with an observational study, we can never be really confident that what we think caused the changes really did, b/c we didn’t take the necessary steps to isolate the effects.

Now, there are certainly many, many situations where we can’t really run an experiment. We can’t manipulate if someone is male or female, or black or white, or rich or poor. But those aren’t really the issues in most email marketing. Instead, most marketers really do have some levers under their control, and they want to understand how they impact performance. This imples that a useful thing to run would be an Experimental study.

So, certainly, both types of studies have value in the world, depending on your question and the research possiblities to answer it. And yes, most good research programs include a mix of observation and experimentation. But given what I said earlier, about us wanting to answer the why question and moving past the “differences exist” re-runs, it would appear that observational studies are answering questions that, as a lawyer might say, are “asked and answered, let’s move on”.

Yes, the observational studies are easier. You just look at all the data you’ve collected already on behalf of clients, and just aggregate it. That’s why there are so many of them. Anyone with enough volume can pop one of these out.

Look, as an industry, feel free to do all the observational studies you want. But are they really answering the questions people are asking? And are you helping people to ask the right questions?

Its time to grow up. Email is now an accepted part of almost every major marketer worldwide. You do yourselves and the industry a disservice if you continue to put out purely observational work. Its not wrong, its not illegal… but I just expect better. Soon, the rest of the industry will as well. My suggestion: be on the right side of that expectation or risk looking like you “don’t get it”.

Again, this should not be seen as an attack on ExactTarget or Chris B or any of the other companies I’ve pointed out over the years. Its a request to say, “Ok, we get it. People do things differently. Which differences have impact, and which don’t? Let’s start to find out.”

P.S. Some comments to me have said things like “Observational studies are useful for laying out what conditions to include in a manipulation: for example, if free shipping appears to be the most popular offer, we now know to make sure we include it in the test, right?” Sort of. If free shipping fits your business model and allows you to make a profit, then test it. If not, whether or not others are doing it, maybe you shouldn’t. But yes, the observational study does show some ideas of what others have tried, no argument there. Didn’t want those folks to think I was ignoring them.

* * *


  Textile Help
Please note that your email will be obfuscated via entities, so its ok to put a real one if you feel like it...

powered by Textpattern 4.0.4 (r1956)