Deprecated: Function set_magic_quotes_runtime() is deprecated in /home/mwexler/public_html/tp/textpattern/lib/txplib_db.php on line 14
The Net Takeaway: Scales, or how to think about a weighty subject


Danny Flamberg's Blog
Danny has been marketing for a while, and his articles and work reflect great understanding of data driven marketing.

Eric Peterson the Demystifier
Eric gets metrics, analytics, interactive, and the real world. His advice is worth taking...

Geeking with Greg
Greg Linden created Amazon's recommendation system, so imagine what can write about...

Ned Batchelder's Blog
Ned just finds and writes interesting things. I don't know how he does it.

R at LoyaltyMatrix
Jim Porzak tells of his real-life use of R for marketing analysis.







Scales, or how to think about a weighty subject · 11/18/2008 02:36 PM, Tech

Scales are a fun thing. There is a ton of research on how these things work, mostly in Psychometrics.

For example, does it matter if the scale has 5 points or 7 points? Should you anchor the endpoints (Very Dissatisfied 1 2 3 4 5 Very Satisfied)? How about the midpoint? I won’t even go into whether you should treat these as categorical or continuous or ordinals, nor when a scale is a Likert vs. not.

Survey analysis, psychometrics, all sorts of studies have given some answers, but as in so many cases, the answers tend to be “it depends”.

But there are some cases where scale choices were just dumb.

For example, the Vista Windows Experience score. This is a tool that you can run to see if your PC is ready for Vista, and it’s also built into vista. It basically takes 4 components of performance, picks the lowest, and makes that your score.

So, my new PC scored a 5.5, and I was upset. For what I paid, I expected more than the middle of the range. And MS didn’t provide any guides as to what was a good or bad score. I tried various tweaks, but still, I was hampered by my memory speed.

Running out of options, I looked for configs of people who had scored higher… and discovered that lo and behold, the range is from 1 to 5.9 (not 6, 5.9).

Hmm. So there is no mention of this on the tool, and only a few sites mention it. How many rules of information design, usability, and psychometrics does this break? Lots.

Why did MS choose this odd range? They were leaving some headroom. Of course, this is a silly reason. A score is either a sum of attributes, or its a scaled representation of a state. So, even though you got, so 20 out of 37 right, we might give you a scaled score of 7 out of 10. The scale is a conversion of units. So, it will get to 10… and then 11 (requisite Spinal Tap reference).

So, why not just make the scale 1 to 5 now? Or 1 to 10? As soon as you hit 10, you either rescale (ie, make a new scale) or just extend it again and now the scale is 1 to 13 or whatever. So, leaving headroom keeps the endpoints arbitrary, and makes comparisons really strange. Also, assigning score by picking the lowest seems strange, compared to how we actually make composites (ie, see factor analysis).

This really doesn’t help. Arbitrary scales confuse people, and they make comparisons difficult. I happen to know MS has a bunch of really sharp psychometricians on their staff (Hello, Maria!). Perhaps they should start understanding more about how scales work instead of picking arbitrary limits, confusing users, and demonstrating yet again just how messed up Vista really is.

* * *


  Textile Help
Please note that your email will be obfuscated via entities, so its ok to put a real one if you feel like it...

powered by Textpattern 4.0.4 (r1956)