Normally humans judge media based on a rating out of 10, which is intended as a proxy for the value they ascribe to the work. There are some websites that aggregate ratings of various individuals and rank the anime, such as myanimelist:
There are several reasons why these rankings may not be reliable indicators of the amount of “value” the work holds to the general public:
To attempt to correct for some of these biases, I scraped the rankings of 20 individuals who rate anime on myanimelist as well as the mean rating the anime gets on myanimelist. These individuals are mostly well-known anime reviewers, the full list of which is available in the Appendix. This is to correct for some of the bias caused by some people being to stupid to recognize value, as the average IQ of an anime reviewer is (probably) reasonably higher than that of the average viewer. While some of these reviewers no doubt have low IQs, they are still probably better than the general public.
To correct for the rater bias, I determined whether some raters give systemically higher or lower ratings than others using a mixed effects meta-analysis which adjusted for the mean rating each rater gave. The ratings the raters gave were also normalised to avoid biases arising from differences in variance as well. In addition, anime with less than 10 ratings were excluded, as their ratings are not reliable enough - this results in the average rating to have an omega reliability of .93. This could potentially ignore underwatched but great series, but, for what it’s worth, reducing this to 5 ratings only adds a few sequels to the top 10.
Adjusting for the bias caused by mapping is difficult - I opted for averaging the percentage of people who gave an anime a 10 and the means to yield a somewhat more accurate ranking I call the ‘Greatness Index’. Of course, these metrics were scaled to avoid one of them from overtaking the other.
There are also some issues with outliers - I removed ratings more than 3 SD away from the overall mean.
According to this method, the top 20 series are:
The top 10 movies are:
The top 5 sequels are:
Unfortunately, given the reliability of the ranking metric was 0.93 and that there were 307 anime that could potentially have scraped the top 10, the exact top 10 should not be that accurate, and most of the top 10 will probably drop to 10-20 after I do the next revision with 69 judges. Because of that, I do not expect Monster to occupy the “true” top 10 slot.
There are several reasons why these rankings may not be reliable indicators of the amount of “value” the work holds to the general public:
- Sequel bias: most people who watch sequels were already fans of the original work, so their scores are inflated - note that 3 of the top 5 are sequels. Despite that, FMA:B and Steins;Gate have somehow managed to survive the sequel menace and have claimed their places as the top anime.
- Rater bias: people who give systematically higher ratings may watch certain anime more than others. For example, more experienced viewers may give lower ratings across the board, which inflates the ratings of anime with high levels of popularity.
- Unrecognized value: some people may not be good at recognizing the value that a work has, because it requires a degree of attention or cognitive ability that some members of the audience do not possess.
- Signalling: raters may take into consideration that promoting certain anime may change their social status, and may subsequently overrate “high status” anime.
- Bad mapping: most people rate titles with a mean of 7 with a standard deviation of about 1.5-2. Subjectively speaking, a ‘10’ is much more distant from a ‘7’ than a ‘4’ in value. Try to think of it this way - a world with 100 anime that are ‘10’s and 100 that are 4s is much better than a world with 200 ‘7’s.
To attempt to correct for some of these biases, I scraped the rankings of 20 individuals who rate anime on myanimelist as well as the mean rating the anime gets on myanimelist. These individuals are mostly well-known anime reviewers, the full list of which is available in the Appendix. This is to correct for some of the bias caused by some people being to stupid to recognize value, as the average IQ of an anime reviewer is (probably) reasonably higher than that of the average viewer. While some of these reviewers no doubt have low IQs, they are still probably better than the general public.
To correct for the rater bias, I determined whether some raters give systemically higher or lower ratings than others using a mixed effects meta-analysis which adjusted for the mean rating each rater gave. The ratings the raters gave were also normalised to avoid biases arising from differences in variance as well. In addition, anime with less than 10 ratings were excluded, as their ratings are not reliable enough - this results in the average rating to have an omega reliability of .93. This could potentially ignore underwatched but great series, but, for what it’s worth, reducing this to 5 ratings only adds a few sequels to the top 10.
Adjusting for the bias caused by mapping is difficult - I opted for averaging the percentage of people who gave an anime a 10 and the means to yield a somewhat more accurate ranking I call the ‘Greatness Index’. Of course, these metrics were scaled to avoid one of them from overtaking the other.
There are also some issues with outliers - I removed ratings more than 3 SD away from the overall mean.
According to this method, the top 20 series are:
- Monster
- Fullmetal Alchemist: Brotherhood
- Cowboy Bebop
- Hunter x Hunter (2011)
- Legend of the Galactic Heroes
- Steins;Gate
- Death Note
- Gankutsuo
- Mushishi
- Fate/Zero
- Neon Genesis Evangelion
- Tengen Toppa Gurren Lagann
- Houseki no Kuni
- Mahou Shoujo Madoka★Magica
- Ghost in the Shell: Stand Alone Complex
- Cardcaptor Sakura
- Violet Evergarden
- Shouwa Genroku Rakugo Shinjuu
- Baccano!
- Tatami Galaxy
The top 10 movies are:
- Perfect Blue
- Princess Mononoke
- A Silent Voice
- Spirited Away
- Totoro
- Kara no Kyoukai Movie 5: Mujun Rasen
- K-On! Movie
- Kara no Kyoukai Movie 7: Satsujin Kousatsu (Go)
- Kimi no Na wa
- Cowboy Bebop: Tengoku no Tobira
The top 5 sequels are:
- Fate/Zero: 2nd season
- The End of Evangelion (this is a sequel, fight me)
- Clannad: After Story
- Aria the Origination
- Ghost in the Shell: Stand Alone Complex 2nd GIG
Unfortunately, given the reliability of the ranking metric was 0.93 and that there were 307 anime that could potentially have scraped the top 10, the exact top 10 should not be that accurate, and most of the top 10 will probably drop to 10-20 after I do the next revision with 69 judges. Because of that, I do not expect Monster to occupy the “true” top 10 slot.
