The Player XP sentiment system uses a state-of-the-art neural network architecture trained on a dataset of over 100 million pieces of feedback, making it one of the largest datasets for domain-specific data ever compiled.
Player XP exists entirely in the cloud, running on AWS infrastructure, we utilise the latest advances in cloud computing to run all 3 pillars of our service: data aggregation, data processing, and data analysis.
Continue reading for answers to all questions you may have about Player XP’s sentiment system…
How Can Our System Tell When Data is Positive, Negative or Neutral?
The AI model we use for identifying sentiment is known as a “supervised learning” algorithm. This means that it learns from being shown labelled examples of what is considered positive, negative, and neutral feedback. It uses these examples to learn patterns within the different classifications of review.
Of course, our system will not agree with a human assessment all of the time, and we can only assess the accuracy of our sentiment with our analysts or against pre-analysed data sets. When we perform sentiment analysis, we aim for greater than 90% accuracy against a manual review by an analyst.
How Accurate is Our Sentiment?
There are two core metrics for how accurate sentiment analysis is, precision and recall. Precision is how often a system categorises correctly when it makes a prediction. Recall is how often our system makes a prediction when there is one to be made. Our sentiment performs better or worse against different sets of data, so we cannot give a single number for the precision and recall of our system. We expect a minimum of 80% accuracy for both metrics for any given game’s data set, however, we strive to have greater than 90% accuracy in all data segments.
To ensure our sentiment system remains accurate, for every new game we add to Player XP, an analyst will manually review a significant sample of data to ensure it’s performing as expected. If we do not meet our baseline criteria we will take corrective measures such as fine-tuning our existing model or creating a dedicated model for the specific type of data to ensure that we meet our quality standards.
Managing Different Sources and Data Points
We have built a trailblazing data pipeline that is highly scalable and extremely flexible and allows us to offer our service to clients. We have a suite of integrations for all of our data sources that bring data into our data processing pipeline. This gathers the data using these integrations and processes it with our industry-leading sentiment and semantic processing. The data is finally stored for aggregation and query purposes by our Player XP dashboard.
How Does Sentiment Work For Translated Data?
As some of you may have read in a previous blog article, we have built our own multilingual translation engine that has been trained on game domain data. The Player XP translation engine is used to translate non-English data into English data, so we can then perform our routine sentiment and semantic analysis on the data. The impact of machine translation on aggregate data such as that in Player XP is very minimal so we can still return high-quality insights.
Flattened Sentiment
Our dashboard allows the louder voices to impact your sentiment more so than quieter voices, giving you a more accurate insight into the discussion happening within your community as a whole. However, if you want to avoid giving more credence to the louder voices, Player XP provides a form of sentiment calculation called “Flattened Sentiment”. This reduces each individual person within the data to a single sentiment value before averaging, therefore normalising the strength of their voice.
Sentiment Impact
People can often have extreme opinions, but it matters how often this opinion is reflected as a proportion of your active community. All of our sentiment calculations are presented alongside community sizes to ensure you can contextualise extreme opinions appropriately. You can use this to identify the topics of discussion in your community that are being talked about by many players, and which ones may be one player having a louder voice.
Elements of our dashboard such as this and many other features are how Player XP can help you find the answer to why your game may be underperforming in certain areas. The age of your game and the size of your community are a couple of potentially many factors to this, but you can use our dashboard to get critical insights and gain a better understanding of your community’s feedback. Player XP also provides “Analyst Support” as an extra service to help you find those answers quickly and easily.
Should You Trust Automated Systems?
There are many automated systems currently that don’t do their job well enough, so it is tough to blindly trust an automated system. However, we can ensure that our system demonstrably works. It is built on current industry-leading Natural Language Processing AI technology and uses best-in-class cloud technologies to service the rest. We liaise with you to identify the Player XP services you need and show you how you can use the dashboard to get key insights into your game’s community that will benefit your team.
Check out the use cases on our website to see the real-world scenarios where we have helped our clients. We want you to trust that the team at Player XP are dedicated to making sure our tool is a trailblazer in the industry and you can use it to get the clarity into your community that you desire. Feel free to contact us via email or Slack if you have any more questions about our sentiment analysis.
