Is Automated Sentiment Analysis Reliable?
The New York Times takes a look at the developments in automated sentiment analysis. I was intrigued by one company’s analogy:
“This is a canary in a coal mine for us,” said John Whelan, StubHub’s director of customer service.
Which is about where I stand on automated sentiment analysis: it’s a blunt tool at best that’s still many years away from fulfilling its potential.
I’m not entirely against sentiment analysis–70% accuracy is better than 0%–but I continue to be concerned that businesses are lulled into a false sense of security by it. After all, would you walk into a coal mine with a bird that has a 30% chance of getting it wrong about dangerous gas levels? I know I wouldn’t.
The problem is that most sentiment analysis algorithms rely on us using simple terms to express our sentiment about a product or service. If it were as easy as identifying “I love BestBuy” or “I hate the iPhone” then we could all build a database of keywords and sentiment analysis would be 100% accurate. Unfortunately, the English language–or any language for that matter–isn’t that simple.
“Sentiments are very different from conventional facts,” said Seth Grimes, the founder of the suburban Maryland consulting firm Alta Plana, who points to the many cultural factors and linguistic nuances that make it difficult to turn a string of written text into a simple pro or con sentiment. “ ‘Sinful’ is a good thing when applied to chocolate cake,” he said.
I recently spoke to a very large technology firm that had tried just about every social media measurement tool available and they expressed dissatisfaction at the accuracy of such automated guesswork.
Still, it’s not all bad news. Once coal mining companies realized the importance of canaries, they started developing better gas detection technologies. And the same will happen with sentiment analysis–it will get better! It has too! There are now hundreds of millions of us willing to share our sentiment online and there are thousands of companies starting to listen-in to these conversations. Those two factors will lead to better accuracy in automated sentiment analysis.
Until that day comes, I stand by my assertion that the most accurate sentiment analysis continues to be of the human variety. Only you can determine–with 100% accuracy–if a blog post about your company is to the benefit or detriment of your reputation. Until technology improves, we’ll have to continue acting as our own canaries.