A long time ago, I interviewed at a company called Affectiva. The company spun out of MIT’s Media Lab and was developed to help identify emotions for and from children with Autism. Fortunately or unfortunately depending on how you look at it, people realized that this could be used for marketing purposes. By analyzing the user’s facial expressions while watching an ad, the application could identify at which points the user felt specific emotions. The insight gained from this was a useful metric for how user’s felt about the ad, at which points they felt it, etc. This real-time sentiment analysis was very helpful but it had limitations and was not as scalable as scraping all comments about a product from a few different social media platforms.

In this post we will be covering what sentiment analysis is, the role of different technologies, and some of the ways it can be used.

What is Sentiment Analysis?

Sentiment analysis is the process of identifying and extracting subjective information like opinions, emotions, and attitudes from data (text) using natural language processing (NLP). Generally speaking it allows businesses to determine customer sentiment toward products, monitor brand reputation, and understand public opinion. Overall it aims to determine the attitude, opinion, or feeling behind text. For example, a product review can express a positive, negative, or neutral sentiment. Sentiment analysis provides valuable insights that can inform business strategy and decisions. It does not have to be limited to just products though, it can be “how are my employees feeling about the work?” or “how is this company perceived?”.

Traditionally, sentiment analysis involved manual annotation of training data which was time consuming. With advances in machine learning and the availability of large datasets, AI can now automatically learn to carry out sentiment analysis with high accuracy.

Methods for Sentiment Analysis

Machine Learning (ML)

Modern sentiment analysis uses machine learning algorithms like convolutional and recurrent neural networks. These algorithms are trained on large datasets of text labeled with their sentiment. The algorithms can capture semantic and linguistic nuances in language and understand context better than rules-based systems.

• Machine learning classifiers: Naive Bayes, SVM (Support Vector Machines), Logistic Regression, Random Forests, etc. These require labeled sentiment data for training.

Neural networks can analyze sentiment at different levels of granularity. Aspect-based sentiment analysis classifies sentiment towards specific aspects like product features. Emotion detection recognizes different emotions like joy, sadness, anger, etc. Sentiment analysis systems can be designed for domain-specific data like social media, movie reviews, etc. by training on in-domain datasets. Some of the more popular Neural Networks are: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNNs), LSTMs (Long Short Term Memory Networks) are widely used. These are important to note because they can capture semantic and linguistic features.

Transformer models

Deep learning methods like BERT (Bidirectional Encoder Representations from Transformers) have led to highly accurate sentiment classifiers. BERT uses attention mechanisms to learn contextual relations between words in a text sequence. Fine-tuning pretrained BERT models on task-specific data produces truly robust sentiment analysis models.

Honorable Mentions

• Lexicon-based methods: Sentiment lexicons contain words mapped to sentiment polarities and intensities. Lexicon scores can be aggregated to determine overall sentiment.
• Aspect-based modeling: Identifies opinion towards specific targets like product features. Uses techniques like dependency parsing.
• Emotion detection: Detecting specific emotions requires labeled data of emotions. Deep learning models are commonly used.
• Multimodal analysis: Combines textual data with audio, visual and physiological cues for more nuanced sentiment analysis.

Applications of Sentiment Analysis with AI

Finance

Sentiment analysis of financial news, earnings calls, and social media discussions can provide valuable insights into investor sentiment and movements in the stock market. AI can extract market sentiment signals from text and audio data such as earnings reports, news articles, blog posts, podcasts, and tweets. These sentiments can indicate how investors feel about particular stocks or the overall market direction.

For example, sentiment analysis can determine if management sentiment in earnings calls is positive or negative. This can signal future stock price movements when sentiment diverges from earnings numbers. Analyzing investor discussion forums and social media using NLP can provide a pulse of the retail investor community. Sentiment analysis is increasingly being used by algorithmic trading systems for stock signal generation and price forecasting.

Products

Sentiment analysis is invaluable for companies to understand customer opinions and feelings towards their products. By analyzing user generated content like product reviews, forum discussions, social media posts, and survey responses, brands can gain product sentiment insights.

Aspect-based sentiment analysis can detect sentiment towards specific product attributes like design, features, price, and quality. This granular analysis provides actionable feedback for improving products. Sentiment can also be analyzed across customer journeys to identify pain points.

Monitoring product sentiment over time gives insights into how people react to new product launches, features, and campaigns. Sudden swings in product sentiment may indicate emerging issues that brands need to address. With large volumes of textual data, AI-based sentiment analysis allows brands to listen to product voice-of-customer at scale.

Employees

Sentiment analysis of internal employee communications can provide organizations with valuable insights into workforce morale, culture, and dynamics. By analyzing emails, meeting transcripts, internal forum discussions, and employee engagement surveys, sentiment analysis can identify trends and patterns in employee attitudes, emotions, and perceptions.

For example, sentiment analysis could detect negativity or uncertainty during times of change like layoffs, mergers, or new policies being implemented. Sentiment shifts may correlate with organizational factors like leadership changes, workplace issues, or macroeconomic conditions.

This allows human resources and management to gauge employee sentiment, address concerns, and take corrective actions. Analyzing employee communications over time can unveil long-term sentiment trajectories related to engagement, satisfaction, and retention.

Additional applications for sentiment analysis

• Social media monitoring: Companies can track brand mentions, product feedback, and complaints on social media in real-time to understand customer sentiment.
• Customer support: Sentiment analysis of customer conversations can help prioritize support and identify pain points.
• Market research: Product and brand sentiment analysis from surveys, reviews, and social media provides actionable consumer insights.
• Fraud detection: Sentiment analysis can identify suspicious patterns like fake positive reviews.
• Political analysis: Analyzing sentiment around policies, issues, and candidates from media provides insights into public opinion.

Sentiment Analysis enables brands and companies to understand the state of their customers, users, and employees at scale. With continued advances in NLP, sentiment analysis will become even more accurate and linguistically nuanced. As neural network architectures evolve, sentiment analysis will approach human-level understanding of language without a doubt.

What about the data?

The most common thread when it comes to leveraging AI for your company is the data, the same applies to sentiment analysis. Proper planning and data capture are key to getting the best insights. When you enter the world of sentiment analysis you are going to need some manpower to help scrape that data, connect with social media API’s, etc. We are currently working with a few companies to help identify the best ways to get the data that is required for sentiment analysis as well as which methods provide the best results. Our findings will be published in a few weeks.

Vatché

Tinker, Thinker, AI Builder. Writing helps me formulate my thoughts and opinions on various topics. This blog's focus is AI and emerging tech, but may stray from time to time into philosophy and ethics.