Deprecated: Function set_magic_quotes_runtime() is deprecated in /home/mwexler/public_html/tp/textpattern/lib/txplib_db.php on line 14
The Net Takeaway: Page 28


Danny Flamberg's Blog
Danny has been marketing for a while, and his articles and work reflect great understanding of data driven marketing.

Eric Peterson the Demystifier
Eric gets metrics, analytics, interactive, and the real world. His advice is worth taking...

Geeking with Greg
Greg Linden created Amazon's recommendation system, so imagine what can write about...

Ned Batchelder's Blog
Ned just finds and writes interesting things. I don't know how he does it.

R at LoyaltyMatrix
Jim Porzak tells of his real-life use of R for marketing analysis.







Time of Day and Observational Studies · 07/27/2004 10:55 AM, Analysis Marketing

Recently, 2 studies have been released which try to claim that the time of day or day of week that one emails has real impact on performance. Jumping to the end, these observational studies are both flawed: Past controlled research (by my team) shows little impact for day of week, though we’ve performed little research with time of day (mostly because email delivery time is controlled by the ISP, not the sender, and we’ve seen delays as high as 3 days ore more for mails to get through an ISP’s systems).

But ok, enough foreshadowing. ReturnPath says that mails sent at certain times are more likely to get delivered; Direct Magazine reports here. eROI, a small email agency, reports that it’s found that performance on certain days of the week appear to be better. MarketingSherpa has the exclusive, but its only available for a short time here.

Both are interesting, somewhat provocative… and are completely flawed. (Caveat: I don’t have the full report or data for either study, so maybe they mention these flaws… but they certainly didn’t mention them to the press, so I doubt they emphasize them enough in their reports.)

Why? They are both observational studies. They both detect a change in time, but do not do any of the necessary controls to understand why these changes are happening.

Let’s start with ReturnPath: Like every other observational study, they throw around large numbers with the belief that every study sounds better if the sample size is large enough. In addition, they mention that they calculate an “index per company” as if that controls for the fact that they are measuring a variety of factors and controlling for none.

They did the standard thing: took the results of a mailing (in their case, some measure of deliverabilty, though unclear if its Inbox or just ISP acceptance of the message) and grouped by time.

The problems are manifold. Lots of different people mailing here. Some B2C, some B2B. Some have a large list, some have small. Some have high quality, others less so. Did RP randomize these elements so every time frame has the same distribution of mailing types? No, they just group by time.

(Less important: Did they wait to see if mails got accepted after time, so that a mailing sent at 5 am got delivered at 10 pm? Unclear in the blurb, but I doubt it. They probably just counted it as a 1and moved on… but not sure about this)

This is a huge flaw in the study. Yes, there is evidence that more spammers mail in the wee hours when traffic is light on the wire; you can send faster that way, and spammer make money on volume. (However, this appears to be changing; see stats to see a pretty even distro of spam across the day.) But does that mean that sending in the early morning is less likely to be delivered… or that the people who choose to mail in the early morning have larger and lower quality lists? Can’t tell from this study.

The eROI study has the same huge gaping hole. Its purely observational. The fact that the “best day” changed so much since the last time (Weds used to be the incorrectly assumed best day to mail) implies that its not stable, meaning (perhaps) that it wasn’t really correct in the first place.

But yes, like the RP study, it doesn’t control for its main variable of interest (day of the week) but simply observes when their collection of clients chose to mail. This doesn’t make it “wrong”, but some of the claims made are not supported by this type of study. Sure, some observations and “directional learning” is possible, but without a true experiment, observations are about all you are left with.

A controlled study would take the same mail, send it out on each day, and measure a rolling window of 7- or 10- or 30-days-out results. Since that was not done here, we can’t really say anything about which day is the “best” or the “winner”. For example, it may be that certain types of mails are sent on that day and therefore get opened/clicked.

By saying that its over “6k marketers” and “a wide range of industries”, they hope that they can hide the fact that there are tons of possible confounds. Are all the marketers of the same type? Are they sending similiar types of messages, with similar calls to action? Are they selling, inviting to webinars, sending informational newsletters, etc? And are all these equally distributed over every day? What is their window, is it 30 days from mail or just 30 days of results rolled up (so that the last week has fewer days in which to count, while the first week has 3 more weeks to drive response)? Without either controlling for or randomizing across these conditions, one cannot really say that there is anything going on. There are too many correlational factors for anyone to feel confident that any day (or any time) is the correct one.

Now, at the marketing company where I work, we have run these studies as well… but we run them as controlled experiments. That is, we use the same email, making sure that there are no confounds like “sale ends tommorrow” or other time-based content. We mail it at about the same time each day (knowing, of course, that arrival time is not really controllable, but we can at least send at the same time). We then measure out the same time window (varies for different clients, but always at least 7 days) to examine all behaviors (clicks, opens, forwards, conversions, unsubs, bounces, etc.)

What have we found? There is an immediate effect no matter when you send: People online get alerted and open. Sending at noon east coast during a weekday does result in higher immediate response than 4:00am on a Saturday morning… but not often huge, just clearly more. Then, by the time 24 hours have rolled around, we see people logging on and checking their mail and looking over the last 24 hours… and seeing the mail. Then we see a bump on the weekend when weekend surfers check their mail for the week. This same pattern results no matter when the mail is sent: after the first day, the usage pattern reflects when users are online, and nothing we do can change that. And if we try to guess when each person is online… well, maybe that is when the mail should be sent. But the mail response curve quickly resembles a site usage curve, reflecting how users go online, not what day is their favorite for reading emails.

So, on average, no day is better than any other. Certain types of mails may do better on certain days, esp if consumers tend to shop in a specific way or products are distributed in certain ways (New CDs and DVDs are released on Tuesdays, movies hit theaters on Thursdays)... but on average, every test we’ve run has shown no effect for day of the week after 24 hours, across a collection of industries, mailtypes, and response vehicles (simple click, conversion, etc.). People open mails either when they arrive, or when they are online, and this doesn’t change per day (with the exception of weekend vs. weekday as mentioned above): controlled studies show this time after time.

So, focus on relevance, high quality content, smart personalization. Beat your baseline. Don’t worry about trying to take a shortcut and mail on a certain day or time and hope for stable improvements in performance. And, in the interest of personalization, perhaps the right thing to do is to see if segments consistently open on certain days, and mail that way. We can even do this by time of day. But given all the factors that really play into a quality message, such as relevance, customized offerings, and an understanding of the trust and relationship you have with the reader… it seems less important to play with day of the week and hope that makes a difference.

Certainly, as I point out, for certain cases, it might help (not as much has having a more relevant message, but…). And if, after doing a controlled test, you see that it does for your specific mailings, great! But don’t assume it will be of much help in the long run. Focus instead on understanding customer needs and expectaions, and that will make your mails much better than a Tuesday vs. Thursday send.

All of these players (ReturnPath, eROI, Sherpa and Direct Magazine) are great sources of info and work, so no slight on them. Its just that correlation does not imply causality, and these observational studies are just bad science. If we want to see the effect of time of day or day of week, let’s do the controlled experimental study. But to toot one’s horn for an observational study, imply you’ve found something new, and not examine if its actually an artifact of the “research” process is just sloppy, and I have higher expectations for the legitimate email marketing industry.

Otherwise, we really will just be the spammers people think we are.


* * *


SPSS to stay listed on NASDAQ... · 07/23/2004 12:35 AM, Analysis

As mentioned here, SPSS has received a determination from the NASDAQ Listing Qualifications Panel indicating that they can stay on the NASDAQ if they meet some requirements such as meeting deadlines. But basically, the review is over with, and SPSS gets to stay traded. That’s a good thing.


* * *


SPSS and Hyper-Threading · 07/14/2004 01:21 PM, Analysis

Stumbled on these posts about SPSS crashing on Hyper-threading machines.

1st Post


Update Post

For those who don’t know, modern Intel processors can pretend to be 2 processors to the OS. The OS and/or running program can then split off pieces of itself to run “in parallel” (this works on single-proc boxes as well, but not as cleanly). Its not really in parallel, but the chip can make it seem pretty close. In some cases, this can perceptibly impact performance… and in other cases, programs are written in such a way so as to not benefit at all from multiple processors. In the past, SMP or “symetric multi-processor” machines were very expensive, but this “Hyper-Threading” brings some of that capability to the user in an very affordable way.

When a program splits off parts of itself (often called “threads” or “processes”, hence a “multi-threaded” program), having multiple processors (or the illusion of such) can really help. Now, its really, really hard to do this well, so no surprise it took SAS 9 versions to do it. But it really needs to be done in SPSS (yes, and Clementine!), given the new size of data that we are all dealing with.

These postings exemplify a couple of things:

Yes, these things are hard. Hey, its hard for MS themselves to keep track of the variety of OS and Hardware combinations out there. But SPSS has a special mandate: They need to calc fast. Any hardware or OS features which support that goal should be tested at SPSS headquarters. If a bug is found affecting performance increases, then that should be patched, because if its affecting one guy loud enough to pipe up, you can fix it before it affects lots of people who won’t pipe up, and will instead switch from the crashing software to another product.

Dumb Joke:
A pilot is flying around Seattle in the usual misty fog and gets a bit lost. He hovers as close to the ground as he can to look for landmarks, and sees a guy on top of nearby building.

Hovering close to him, our pilot yells “Where am I?”

“You are in a helicopter!”, yells back the rooftop guy.

Immediately, the pilot turns 30 degrees to the right, flies high speed for 10 minutes, and then cuts power to drift right onto the center of the pad.

A ground crew rushes out to him, “How did you do that in the fog?”

“Simple”, the pilot replies. “I asked a guy a question, and he gave me a technically correct but useless answer. So, I knew I was above Microsoft, and that Sea-Tac was 30 degrees to the right, exactly 20 miles!”

Update:Mr. Turner posted that he has written a program to force an exe onto a specific processor. It’s linked in his recent post at his blog.


* * *


Welcome! · 07/01/2004 02:57 PM, Personal

Samuel Anatol Wexler was born on June 30, 2004, at 11:20 am. Weighing in at a somewhat slight 5 lbs, 5 oz., he nevertheless has the lungs of a boy twice his weight, and delights in sharing his skills with the entire hospital.

So, expect light blogging for a bit while I try to remember the sounds of silence.

PS: Pics here

Comments? [1]

* * *


Quote of the day... · 06/29/2004 02:56 PM, Analysis

“Never try to walk across a river just because it has an average depth of four feet.”
—Milton Friedman

Six Sigma is a great effort on many fronts. One of them is the attempt to change the business mindset about that one number or one set of numbers as metrics. Six Sigma “black belts” preach that no average should be presented without some measure of spread, and even better would be a mini distro or histogram. They are also pretty good about using Taguchi for experimental design, and about beating your own baseline, not some arbitrary industry average.

I suspect Milton would agree with this school of thought. I know I do.


* * *


Do User Conferences count? · 06/23/2004 02:23 PM, Analysis

So, I complain that SPSS has no “reach out” program. But, they do have a user conference (more info here) in October (24-27th). Its the traditional tech way; IBM has had user confabs since the 60s, and companies large and small throw user group meetings or parties. Some try to make a profit out of them, others accept it as part of the cost of doing business with a specialized audience. In the past, you could latch them onto other more general shows, but most of those shows have dried up (though some of the BI shows are starting to pick up steam again). (As an aside, note that Comdex 2004 has been canceled. Talk about the drying up of the old guard).

So, does this conference make up for the lack of community efforts that I’ve whined about elsewhere on this blog?

Well, frankly, no, in my opinion, though its a start. Here’s why:

To be fair, this conference does have some very interesting speakers. I suspect the SPSS product managers will be in attendence so we can beat them up (nicely, now!) about features and bugs. It might be nice to have a hard-core tech session so we can talk about memory management, row-to-row time to speed up counts, and hard core byte-twiddling tricks to get the most speed out of SPSS; I don’t see much mention of that in the invitation materials I looked at. The training classes look about “middle”: most are not aimed at SPSS-wizards, nor at beginners, so that’s probably about right. So, I can’t really complain about what I can see so far about the conference; its a pretty basic large show in the middle of the desert.

Now, SAS also has big shows as well… but SAS also has regional user groups who put on smaller conferences. What’s nice is that SAS supports these groups, helping them put on pretty professional get togethers. The materials are often published on CDs or made available on a site, so if you didn’t attend, you can still get access to them. There is often a mix of practitioners and academics, and often some “SAS official” is there to spill the beans on some sort of new offering, or get feedback on things.

In fact, speaking of SAS, they have their Data Mining conference (guess where? guess when? Ok, just before SPSS’s show. One could stay and make a long few weeks out of it.) Some sharp folks there as well, good list of speakers.

I am not going to either of them. No, mohammed will not be going to the mountain; instead, in this world of Webex and pdfs, I expect to be able to access the infomation to help me use the product (and therefore keep paying for it) on a web site, virtually to my desk, and at rates affordable for small companies as well as large ones.

So, easy for me to complain; I’ve never been to either SAS’s nor SPSS’s large conferences. I would love to hear from those who have attended to let me know if I’m totally missing it; perhaps these shows are the perfect way to start building a sense of community, and perhaps physicality and hand-shaking is a necessary part of building a group. You see the comments link, so let ‘em fly.


* * *


Finding a Web Analytics vendor · 06/22/2004 11:34 AM, Analysis

While I could go on for hours and hours about this (and probably will in later postings), there is a nice series at NetworkComputing about their experiences with this very problem. Web Analytics Review: Searching for Results is a multipart article from March through July. It’s worth keeping an eye on…


* * *


Judoscript Re-awakens... · 06/21/2004 04:12 PM, Tech

James mentioned offhand that we should check out Javaworld… and a quick peek shows a great new introduction to Judoscript, phrased as a “JDBC Scripting Language”

Expect to see Part 1 this week, and Part 2 next week.

Also, if you are so encouraged, check out the book James is writing to improve documentation. The JudoScript Language 0.9

Red blocks are not done; yellow are partially done, and green has some real content. From the mailing list, James mentioned:

“I’ve also rectified the language a bit, and enhanced the Java scripting syntax. Please refer to The old ways still work but will be deprecated.”


* * *


Links with Spaces in Outlook · 06/18/2004 10:52 AM, Tech

Ever paste a link into Outlook with spaces in it? Notice how “smart” Outlook is about parsing? Try making a quick mail and typing in (or copy/paste)

"Hey, I stuck this on the server; please review! Load up file://x:\Our Stuff\Important\Client Presentation\x.ppt".

Note how Outlook only hots the part up to the first space? Congrats; you’ve got a broken link.

The trick is a simple one. Put < and > around the entire link. Try

"Sorry about the last mail, it's at <file://x:\Our Stuff\Important\Client Presentation\x.ppt>"

and see how the entire link is active now.

One of those great tricks you never know you need til you need it.


* * *


Stats are Meaningless, Pt II · 06/11/2004 12:03 PM, Analysis Marketing

After my previous rant, I realized that I had said lots of this stuff before, and in a better way… So here is that repost. This one also focuses on email, but it is germane to almost any interactive marketing out there. It was in response to yet another request for “the industry open or click average”.


Ok, provocative title line to get your attention. No, stats aren’t meaningless.

But “Industry Averages” are often completely useless, at least for current e-mail marketing. Here’s why, in a collection of parts:

1) Are you measuring the right variable? When asking for clicks, folks often assume it as a proxy for conversion. But you need to know clicks:conversion ratios for it to be useful. So, don’t focus on the wrong metrics. And remember, it may take multiple messages to drive a conversion, so don’t try to tie a conversion to a single “do or die” message in every case. Finally, remember that different folks have different standards: some companies are upset with “only” a 10% conversion, while others are ecstatic with a 0.25%.

“OK, but I really do want to know clicks. Why can’t you just add them up and tell me, on average, what you get?”

2) Well, its that old apples and oranges thing again. There are so many factors to manipulate in email that I can create lots of different types of mails. Rolling all these different types of mails together in one pot doesn’t give you email, it gives you fruit salad, when what you really wanted was to understand apples vs. oranges.

Just off the top of my head, here are some factors differentiating mails. There are more, but just to get started…

Ah, but still some naysayers. “Who cares? Just lump it all up, it all comes out in the wash, and give me an average, you meany!” some cry. Well, not so easy.

3) Averages are highly impacted by outliers. This means that if I throw in my customized newsletters to my membership audience with a 30% click rate, then I can make all my other bad mails look great, “on average”. Should I take unique clicks and divide by net mailed? Take total clicks and divide by net mailed? Should I just average the click rates? Should I weight smaller mails? Should I weight larger mails? Perhaps a Mode or a Median?

Yes, we can make an “average” or some other Measure of Central Tendency, but it will have little to no meaning. Why? Because I will calc it differently from some other group, and some groups will include or exclude mails based on factors above. So, we will all have different answers… should we just average those averages? I think not.

The confusion is due in part to the fact that most folks aren’t asking the right question about their business. The question should be something like:

“For my industry, mailing the type of list and type of message I will be mailing to a certain type of list, what types of opens, clicks, and overall responses have others seen with this specific type of offer and this ‘conversion process’ (buy something, fill out a form, whatever)”?

That starts to get somewhere… but rarely do I hear folks ask that. And the overall answer isn’t some average, it might be a range of medians or a list of modes, showing comparisons across various factors of importance. For me, the answer is part of some ongoing research we have to group mails by type and then track them across our variety of clients. I know others are doing the same, and soon, we will start to be able to share and merge this data (I hope!). Look, even the DMA publishes a big book splitting their response rates by industry, media, and lots of other variables like the ones I mention above. Why would we (as an industry younger than direct mail) want to not learn from them?

And what’s the followthrough? Work with your own metrics to make them better. What can you change that is under your control to refine them? What specific groups are high or low, and what levers can you use to change their behavior? Don’t worry about what everyone else is doing; fix your own programs first and then everyone else will worry about what you are doing!

So, there is no industry average (and thank goodness). But over time, with the right questions and lots of cooperation with each other, we will start to better understand expectations for folks… and maybe, just maybe, we’ll have that correctly framed and calculated “Industry Average” (for your industry, business goal, etc.) after all.


* * *


On a previous episode...

powered by Textpattern 4.0.4 (r1956)