Thanks for writing this post, as we have to agree with you on certain points and were very interested in the NYT piece. While automated analysis has its benefits, it is not as much an option yet for an international monitoring system or anyone who wants to monitor their brand outside of English-speaking countries.

While there may be certain software programs that have developed an impressive sentiment analysis accuracy rate, sourcing in countries like China and Russia, such as we do, cannot be left to an automated program.

Humans are the most accurate for working out the intricacies of a language and determining its sentiment as related to a product, brand, or company. While we are developing certain automated functions, we ultimately use native speakers of the countries in which we monitor.

Phil made a great point about putting in the elbow grease to find out where the pearls are hiding. We would like to make everything automated, wouldn’t we? Would we?
Thanks again, Andy, it was great to see all of the responses.

Best,
Michelle
@Synthesio

Interesting post. I think Tom from MotiveQuest and Christine from Lexalytics did a nice job at articulating things. I would agree that at this stage in evolution of the space automated sentiment is certainly not 100% accurate but it certainly aids in the dissection data for action by brands that have a lot of data to sift through. Human analysis is also very important because their is an interpretive layer of insight that varies depending on the desired intentions of what brands maybe trying to get consumers to evoke. With our technology we have seen accuracy in 65 to 85+% range but also variations and unique differences by vertical, affinity, B2C and B2B. Sentiment is a very important piece of the puzzle for folks interacting in the social space and I am certain that will continue to see technology improvements but human reasoning/interpretation would be disappearing anytime soon either.

@bcahill
@visible_tech

@Simon In regards to customization: this is also something I’ve noticed is lacking in most sentiment analysis tools. Dealing with our own clients we find differences in their community and editorial standards ( the same goes for PR standards for corporations). You have to be able to customize the tool to these standards in order to achieve the best results.

As for accuracy, we’re at 90% and climbing…testament to what we call principled training.

First let me congratulate the NYT for noticing that we are doing this. When we started up MotiveQuest about 6 years ago – BuzzMetrics and Cymphony were already up and running.

Second, @Joe has it right above. Use computers for what they are good at (pattern recognition) and use humans to define the patterns the computers look for.

Our software tools are designed so that the strategist adjusts the linguist model for every context and project. Sick (obviously) doesn’t mean the same thing in online gaming as it does in osteoporosis. This makes the tools quite complex to master – but much more effective at generating accurate, relevant and useful sentiment scores.

@Mikko has a great point with this comment:

Volume (how many) > Meaning (what is it) > Intent (why is it) > Results (why should I care) > P&L (how much)

Our work is predicated on getting to the why and then measurement of real world results. Otherwise you are just counting number of mentions and sentiment which in and of themselves are meaningless.

Tom O’Brien
MotiveQuest LLC
.-= Tom O’Brien´s last blog ..Advocates are more important than influencers =-.

I agree to some extent with what you are saying in your post.
You argue that “would you walk into a coal mine with a bird that has a 30% chance of getting it wrong about dangerous gas levels? I know I wouldn’t.” Knowing that the humans interpret information about 80% accurate the gap between humans and the “computer” is not that big.

For a machine to understand and interpret text (Twitter, blog, article, comment about a product) it needs to have deep semantic knowledge (like a human) which is why Saplo (www.saplo.com) base their technology on leading (academically accepted) research on how the human brain process, interpret and recall information. In other words, Saplo imitates how the human brain interprets information.

This is not to say that we can leave it all to the computer. Joe Hall (above) said it nicely “The key is to not create and offer tools for automated sentiment analysis, but rather to create and offer tools that aid human analysis.”

Best
Peter Larsson

@saplo.com

PS. Saplo have documented results, in comparison with human analysts (80% accuracy), that shows 80-90% accaracy.

And why bother, it’s just one step in much longer online media metric stairs:

Volume (how many) > Meaning (what is it) > Intent (why is it) > Results (why should I care) > P&L (how much)

A good guideline for tech developers working with computing is to automate things that machine can do better than man, and leave everything else for man. When the time is ready for sentiment analysis by machines in a sensible way, we’ll be way ahead up the stairs and it wont even matter anymore.

Have fun!
.-= Mikko Kotila´s last blog ..mikkokotila: @r2r0 “I’m just working”-> “I’m learning the strategy” -> “I understand the mission” -> “It’s my mission” = self managing teams =-.

Thanks for the post. I agree that sentiment can be semantically tricky and something is better than nothing. What is even more important in this age is that having all of these new data research capabilities forces companies to be user/customer centric. As you know, reputation may be one of the most valuable assets a company can have these days especially if you’re in the business of serving customers, which is everyone last time I checked.

Beyond reputation, I think its important for companies to understand how to measure their effort by using social media monitoring tools. For instance, how is revenue, reach and return on operations being measured in the various tools on the market? Are they at all? I’d love to see one.

Phil, I was thinking the same thing this morning re: metal detectors. It still takes a human to dig in the dirt and it should always since we are using these tools to connect with and influence public/people perception as Joe alludes to. It’s supposed to be authentic, right? =)
.-= Jason Cronkhite´s last blog ..How Do You Measure Social Media? =-.

I agree that things are getting better – and cheaper – and faster – so time will tell where this all leads us because you are correct, there’s a whole lot of sentiment out there these days that companies need to know about.
Best,
Christine
Lexalytics
@christinelexa

Therefore I absolutely agree with you Andy, no matter how much automation you build in, you’re going to need a human eye to make a final determination in most cases