Marketing Pilgrim's "Research" Channel

Sponsor Marketing Pilgrim's Research Channel today! Get in front of some of the most influential readers in the Internet and social media marketing industry. Contact us today!

Why I Think the New “Pop-Up” Study is Flawed




Ars Technica has details of a new study from the excellent (yes, I’m biased) Psychology Department of North Carolina State University. Basically, they created a series of “pop-up” windows designed to look like system messages, but could easily have been viruses.

The results?

Of the 42 students, 26 clicked the OK button for the “real” dialog. But 25 clicked the same button for two of the fakes, and 23 hit OK on the third (the one with the status bar showing). Only nine of them closed the window—two fewer than had closed the real dialog. In all cases, a few of the users simply minimized the window or dragged it out of the way, presumably leaving the machine’s next user at risk.

Now, I’m not a psychology major, but I did stay in a Holiday Inn Express study it for two years back in Ole Blighty–so I’d like to think I have some insight on experiments like this.

OK, here’s why I think the data is flawed. It appears the experiment was carried out on the college’s computers, likely in some research lab. With that in mind, why would anyone (especially a college kid) care about what might happen to a computer they don’t own? In that same situation, I’d assume that perhaps the college had virus protection installed or simply not take ownership of my actions.

If the above assumption is correct, I’d consider the data useless until the study is conducted again–this time using the student’s own computer, in their own home, with perhaps more than 42 participants. I suspect (“hypothesize” in psych speak) that they’d be a lot more cautious in their clicks. ;-)

Let me know your observations (now that I’ve biased them with my analysis) ;-)

(image credit)

  • http://www.jozsoft.com/blog/ Joe Hall

    I think you are right, people tend to take more care over their own machine. Especially, when compared to the machines in public university labs. I have personally seen people do things on the keyboards in universies’ labs that I would never do anywhere. I have seen students repeatedly down load software that is overtly adware/spyware. The general assumption that many students have is that the IT department will fix it.

    Joe Hall’s last blog post..Links are making people go crazy!

  • http://shardex.com David Sharek

    In research like this, it’s important to take a ‘first stab’ to see if the topic is worth investigating further. Since we got significant results in this study, the next step would be to add personal affect, to see if there was an effect of a diffusion of responsabilty. My guess is that we will find similar results, but you are absolutely correct in thinking that the research needs to be taken a step further. To say it is flawed, though is not quite accurate.

  • http://keekeespeaks.com KeeKee

    I don’t think the fact that it’s not the students own computer makes much difference. Even if the research were to be done on their own computers, a lot of students have the mentality that ‘Mommy & Daddy’ will just buy me another computer so they still hold no responsibility.

    KeeKee

    KeeKee’s last blog post..Madrid Spain

  • http://www.marketingpilgrim.com Andy Beal

    @David – thanks for providing some official feedback. As I said, I’m not an expert, but in addition to what I stated, here are some other observations:

    1. I’m assuming the study group knew they were taking part in some kind of study? That would likely skew the results.
    2. 42 is a small sample. Again, I’m assuming these were all college students. Not exactly a representation of web users.
    3. In addition, there were 4 different scenarios, correct? Divided-up by 42, means that only 10 students per scenario.

    I could go on, but here’s the thing. I haven’t read your paper, so I don’t know what your conclusions are. You may only claim that college kids are dumb enough to click on fake pop-ups and make no effort to extrapolate that to the general population. Also, your follow-up study may well have the same result. However, based on the information I have to hand, I do believe the study is flawed–at least, when we try to apply its results to the general web population.

    Just my thoughts.

    Go Pack! ;-)

  • http://cssdigitaldesign.co.uk sheppy

    After reading the article, I’m a bit confused. So the users were put in a situation where they were told they were going to be asked questions about how a website loaded?

    If I was on an unfamiliar computer with a different objective to using the website, I’d probably move the box out of the way, if I thought it was just something that wasn’t supposed to happen.

    I know that my statement above doesn’t qualify, as you’d tend to ignore what people say they’d do rather than what people actually do!

    sheppy’s last blog post..The Eagle has landed…

  • http://www.grahamjones.co.uk/ Graham Jones – Internet Psychologist

    Andy, I think the study is only partially flawed. If the study was designed to show that people ignore potential indications of malware, then, yes, it would have been more appropriate to do that on computers in which the participants had some kind of stakeholding. So doing such a study on departmental PCs would invalidate the results.

    However, the study does not seem to have had this as its aim. What the study appears to be be aimed at is the dislike people have for the standard windows dialogs. The study was mainly looking at response to dialog boxes.

    The “give-aways” as to potential malware are actually only give-aways to technically savvy people.

    What the study showed is that dialog boxes are largely ignored – and that’s why the malware producers succeed. It’s actually a study that tells us that Microsoft’s products are too focused on “engineers” and not on real people. Those of us who are not engineers are not in the least interested in the messages – so we just click OK to get rid of the interruption.

    That’s how the malware producers have tapped into us – they are much more aware of human behaviour than Microsoft seems to be.

    In essence, the study tells us what we already know – we hate interruptions to what we are doing (which explains why most advertising doesn’t work as well).

  • http://www.marketingpilgrim.com Andy Beal

    @Graham – thanks for your thoughts. Have you read the full study? I’d like to know the actual hypothesis they were testing. Do you know, or are we all just guessing? :-)

  • http://haiku-poems-blog.blogspot.com/2008/09/captivating-capiz-haiku-poem.html kouji

    i think it would also depend on what they were doing on those computers. after all, if i was working on a paper or thesis, i’d be very careful, even if i wasn’t on my own system.

    kouji’s last blog post..haiku poem: dry

  • http://factualsatisfaction.freehostia.com/cell.html brando the Great

    Can we drag these people out in the street and shoot them?