Is there was a relationship between a skill’s ratings and its ranking on the Alexa marketplace’s categories?
People must use the category pages to discover new skills, so it would be beneficial for a skill to be in one of the top pages. Also, Amazon likely wants to promote good skills on the first few pages of their marketplace to showcase good user experiences. I took a look at our data to see if this latter assumption was right and found something surprising.
First, some stats…
On the y-axis (vertical), we have the number of reviews a skill have received. Note that this axis is not linear (the space between 1 and 2 on the graph and 11 999 and 12 000 is not the same). Instead, it’s logarithmic (as the numbers get bigger, the space between numbers on the graph gets smaller). On the x-axis (horizontal), we have the rank of the skill in its category.
The blue dots are average skill review counts for each rank while the grey dots are an average count of skill reviews for each rank (1 to 8000). The grey dots are honestly a bit useless in this graph, so don’t worry about them too much.
Back to the fun part….
What we see is that there are some very well liked, in terms of number of reviews and rating, in the back pages of the Amazon marketplace. This also brings into question how Amazon ranks its skills. No one outside Amazon knows how skills are promoted, but looking at this, it seems like there is a randomness to the process. If these “page demotions” were based on a sudden decrease in skill performance, we’d expect to a poor rating next to a decent amount of ratings and reviews.
With our data, it’s impossible to truly know why these skills are so poorly ranked. However, it does give us a bit of insights as to how the Alexa marketplace behaves (though for now it seems as though it’s randomness). As we move forward with voice tech, it is almost certain that there will be something similar to ASO (App Store Optimization). People will start to write more optimized skill descriptions to gain a better ranking under the different categories, leading to better discovery by users. Understanding why some of these skills are currently so harshly “page demoted” might be the first step to cracking that code and getting a skill on the first page!
Check out some of the best rated, but lowest ranked, skills on the US Alexa store. Honestly, these skills seems pretty good! It’s unfortunate no one will ever find them! (there is a limit of 500 pages per category)
204th page:
CompliBot by 3PO Labs – 4.1 out of 5, 389 ratings, 60 reviews
CompliBot has got your back!It sees the best in you, and loves to share it!
Random review: love it especially since it will use your name (5 out of 5)
216th page:
Happy Days by heats – 4.1 out of 5, 331 ratings, 29 reviews
Are you having a bad day? Would you like to hear an inspirational positive quote? This is the skill for you! Simply open the Happy Days skill and sit back and smile.
Random review: I just felt sad and opened Happy Days. Hearing a quote from Dr. Seuss made me laugh and actually feel better. (5 out of 5)
262nd page:
Meme Review by LiamSorta – 4.8 out of 5, 85 ratings, 48 reviews
Bring the joy of Meme Review to your Alexa-enabled device. Wonder at the sea of featured memes and cast your vote to score them!
Random review: Subscribe to PewDiePie (5 out of 5)
What Amazon definitely takes into account is the “freshness” of the review. Our assumption as well is that they take the “diversity” of the reviews as well. Meaning that if all the reviews come on the first week after the skill launches and come from the same city/state, then they are less valued than if those are spread over time and geography.
Interesting… It seems like the perfect hypothesis for us to test (at least the time bit)!