Alexa Champions is “a recognition program designed to honor the most engaged developers and contributors in the community”. In a previous post, we revealed that there were some very well rated posted hidden several hundred pages deep in the Alexa marketplace. Eric Olson, co-founder of 3PO-Labs and Alexa Champion, pointed out that all 3 of these skills were by Alexa Champions. It seems odd for skills from champion developers to be rendered essentially undiscoverable by the very company that features them. This raises the question of whether Alexa Champions have any objective advantage over regular developers. We already looked at skills’ ratings and skill ranking on the marketplace. Today, we will look at the number of ratings for Alexa Champion skills.
Analysis
Below are results for 13 Alexa Champions. Our analysis did not include Invoked Apps because their wildly successful skills make them an extreme outlier. We excluded VoiceXP because their 36 skills makes it hard to match the developer with an industry average (there is only one other developer with 36 skills). We also excluded 9 developers for only having 1 or 2 skills. Mark Tucker’s skills are distributed across a few developers, so unfortunately, his were also excluded from the analysis.
In the graphs, you will see the name of the Alexa Champion developer next to “n=# average”. This represents the average of all developers with “#” number of skills (same number that the Alexa Champion published). The black lines represent the variability of the average. If the bar is small, that means most data points were close to the average. A larger bar means that the data points were far from the average (either more or less).
Number of ratings
The number of ratings is my favorite metric. Because it’s such a quick thing to do, it’s the metric that gets the most input, with the flip side of getting more “meaningless” votes. “Meaningless” votes are from people who try a skill once and think it’s neat. It’s a binary vote (1 or 5) because they haven’t given the app much thought and just want to leave feedback. They either liked it or they didn’t. A large majority of votes are either 1s or 5s. Alexa can also ask user to leave feedback as they exit, but it only does this for popular skills. This means that popular skills have an opportunity to boost their meaningless votes (hopefully in the 5/liked direction). Skills with larger numbers of votes seem better and attract more users, leading to more reviews from this exit poll, and so on.
Below we see the average number of ratings per skill for the Alexa Champions and other developers with the same number of skills. A long black line (variability) means that this developer has at least one skill that is very far from the average. The developers with the largest variability are Opearlo with Find my phone (25k+ ratings), Philosophical Creations with Big Sky (5k+ ratings), and labworks.io with True of false (1k+ ratings). These skills vary so much from the average, that just showing them makes the graph totally useless!
Looking at this data without the variability????, we can see that three Alexa Champions (Opearlo, Philosophical Creations, labworks.io) have average rating counts way above everyone else. We can tell they make skills people like (or have at least one super popular skill). Even now, these three Alexa Champions make it hard to see how the majority of Alexa Champions perform compared to the average matched developers. Averages have the disadvantage of being misleading. These three “above average” developers seem like they’ve developed several oft rated skills, but that’s not necessarily the case.
Once we removed Opearlo, Philosophical Creations, and labworks.io, the 10 remaining Alexa Champions and their matched averages become clearer. We were wondering whether Alexa Champions consistently receive more ratings, which could suggest a marketplace advantage given by Amazon to developers it chose to promote. Looking at these averages, there doesn’t seem to be a clear trend. Alexa Champion rating count averages can be larger, smaller, or about the same as non-Champions. You might expect Amazon to select one “top skill” from the Alexa Champions to promote, but only a small minority of them have at least one very popular skill, so that doesn’t seem to be the case either.
