Welcome back, dear reader! It’s been nearly two months since our last update and while we’re a bit delayed from a monthly schedule, release delays and a lack of events meant relatively few shifts over the last month. However, we now have a few important changes, with Blood Angels and Death Guard out and January FAQs released, and more data to look at their impacts.
The Impact of the January 2021 FAQs
The January 2021 FAQs changed a number of factors for Warhammer 40k, particularly for competitive play. While there were a number of points changes, we’re chiefly concerned with the structural changes to the missions in the GT 2020 Missions pack. Specifically:
- Players who win the roll-off may no longer opt to go second
- The player going second scores primary objectives at the end of their Turn 5 as opposed to during the Command phase
- The Abhor the Witch and Bring it Down Secondary Objectives both changed to reward fewer points
Additionally, we’ve also had more time to look at the impact of the Death Guard and Blood Angels Codex Supplements, and start to get an early idea of how Codex: Death Guard may impact the faction’s ability to win games.
Thanks to the wonderful efforts of tournament organizers and app developers around the world, we have access to what is essentially every meaningful piece of data around competitive games of 40k. The data in this month’s study comes primarily from The ITC Battles App, a brilliant app for tracking games both in and out of tournaments. With tournament activity currently slowed in most areas thanks to the pandemic, we were still able to analyze data on nearly 4,000 games of Warhammer 40k played in January.
A New Way to Look at Faction Rankings
Ranking factions is difficult; on any small scale, player skill has a massive effect on faction outcomes and while larger numbers of games will reveal certain trends it’s still difficult to account for things like faction vs. faction strengths and weakness, or order of games. One example we’ve seen recently is Chaos Knights, a faction that posts high win rates but does not win tournaments – they score low with regard to 40kstats’ Tournaments in Winning Position (TiWP) metric. This indicates that Chaos Knights will frequently lose early on to better factions but then clean up in the losers bracket.
Instead another way to consider relative faction strength is to calculate ratings for our factions. Glicko is a rating system similar to ELO (it was designed as an improvement on ELO, in fact), that can be used to gauge performance. As a 40k faction in our sample wins or loses games, they gain or lose points from their score, gaining more points from beating a higher-ranked faction and fewer from winning against a lower-ranked faction (likewise losses to lower-ranked factions have harsher penalties). There are some additional factors to the model, such as how much you want it to react to each new game result, but for the most part, those are the basics. In our model, a score of 1500 is about average.
Plotting Glicko scores for faction over time creates the following chart:
Overall Glicko scores give us a picture that confirms a lot of our prior findings - typically a very good thing when creating new metrics - while also showing us a few interesting new things - also very good when creating new metrics (otherwise what would be the point)? Here are some things to note from our Glicko scores trended over time:
- Harlequins have been, and continue to be, insanely strong
- Death Guard have seen a significant bump over the last week, winning more games with their new Codex
- Custodes have also been on the rise since the January points update
- Monofaction Genestealer Cults appear to be the game’s worst faction, though T’au aren’t far behind
- Imperial Knights have been making significant improvement, likely a result of the Imperial Armour improvements for the Magaera getting backed up by cuts to Armigers.
- Deathwatch continue to struggle more than we expected. Their codex appears to just be less than the sum of its parts overall, and new units and kill teams don’t make up for the loss of Special Issue Ammo on most units
Considering a potential handicap
We recently received reader mail from a TO in Australia looking for advice on handicapping the field or creating soft scores for players based on a faction’s perceived strengths. While we aren’t a fan of this - handicapping factions takes agency away from the players and a simple points value ignores things like good/bad matchups - it was an interesting question to consider. Is there a way to use our ratings data to slightly level the playing field?
Using the rating difference from each game to predict the primary victory point difference, we can create the following handicaps by faction, in bonus VP allocated to the underdog:
These handicaps are cumulative - a T’au army facing Harlequins would begin the game with 10 bonus VP (though would still have a maximum possible score of 100 points). This is also a helpful way to visualize what the win rate discrepancies actually look like in terms of a game being played. On that note, we can also quantify the value of the first turn with this model - going first should cost you on average about 4 VPs according to this model, slightly less than our prior predictions, but still pretty close!
Secondary Objectives Update
Last time around we looked a deep look at secondary objectives - average scores, frequency taken, and impact on win probability. At the time we were still pretty light on faction-specific secondary data, and since that update we’ve had major changes to the way two secondaries worked, so it was time to revisit that analysis.
Looking at scoring shifts for secondary objectives since the FAQ dropped shows two major shifts: Bring it Down is now taken 15% less often (a statistically significant drop) and players taking it score an average of 0.8 fewer points than they did before, and Abhor the Witch is now being taken half as often - only 4% of the time - but tends to be worth 1.4 points more on average. Note however that unlike the drop in Bring it Down, these results are not statistically significant at the 95% confidence interval and may continue to shift over the next month. Still, this is an interesting result to consider - the number of games in which you'd take Abhor is now significantly less (and likewise you may be more likely to take a psyker in your list), but when you're up against lists where it's still easy to max, you're more likely to be making the right call.
Also of note here is that now the Warpcraft category of secondary objectives are only chosen around 6% of the time, making them by far the least most popular secondary category. When you factor in the clarification that psychic actions can now be denied by any ability that denies psychic powers, the outlook for them is pretty grim. I don't typically like to make suggestions here but in this case I'm going to go ahead and say that the Warpcraft secondaries need a rework.
On top of this we have the faction-specific secondaries. Here we can see that there are some real standouts - almost all of the Necron faction secondaries are amazing, while Space Marines' Oaths of Moment is the clear stand-out for their faction and Warrior Pride looks like a very solid choice for Space Wolves while Relentless Assault appears to be emerging as the build-around pick for Blood Angels, though data on Blood Angels secondaries is very sparse. The interesting thing about this is that it suggests that non-Codex factions are at a real disadvantage if Necrons are consistently taking a secondary from their codex - something that happens in more than half of Necrons' games - and consistently scoring higher on that objective than they would with one out of the GT Missions Pack.
While it's still much, much too early to make any real determinations on the brand-new Death Guard secondaries these results aren't surprising to me based on the games I've played. Our review tagged Fleeing Vectors as the consensus worst of the bunch and we thought both Spread the Sickness and Despoiled Ground had potential, though I'm finding in my games that Despoiled Ground is much more difficult to actually accomplish than I originally expected and probably needs a specific build to work. Both of these have so far just resulted in middling average scores, however. On the other side of the "newer book" spectrum Deathwatch still don't have a ton of data but it doesn't appear that their secondary objectives are doing much for them. The good news is that every one of the Space Marine factions can also just take Oaths of Moment.
Is First Turn Advantage Gone?
Finally we move on to the biggest question of our analysis and probably the hardest to answer. We don't yet have a ton of data for post-FAQ play, so some of these results may change as we get more data in. In our Metawatch article on Warhammer Community last week, we worked to answer the question: Is the large advantage conferred by having first turn gone?
Well, it doesn't seem like it.
Looking at our results from the first 1,500 or so games played since the FAQ dropped - with the caveat that not all of them might have used the new rules (though we did remove from our sample any games where the player who won the roll-off to go first went second), it seems like there hasn't been any statistically significant change in win rate for the player going first.
So right now there doesn't seem to be a statistically significant change in win rates for players who have the first turn, but this may change over time as we accumulate more data. However, it may not - it's worth investigating why we might be seeing this result at this stage, and explore some of the nuances of what the data can tell us about scoring in games. Let's run through some potential reasons why the two changes to mission structure may not have produced the intended result:
- Not being able to opt to go second just may not matter. As we saw in our prior articles, every faction in the game had a higher win rate going first than second, and even if a faction had relatively close win rates, it was likely they'd be playing against a faction or army that would benefit more from going first than second. Additionally, the winner of the roll-off only chose to go second in about 8 percent of games in the sample last time we looked, so the number of games where this would even have an impact to begin with is likely very small. It may take significantly more time to identify whether this makes any difference.
- Turn five scoring doesn't matter in many games. Games that see one player with a massive lead coming into the fifth battle round, or where one player has already definitively won aren't uncommon. At that point players may choose to talk the game out or concede, rendering the adjustment to end-of-turn scoring for the second player completely moot.
- Turn five end-of-turn scoring may not generate enough VP to generate real results. We've identified the impact of going first to be about 4-5 primary VP through various analyses, so a go-second bonus needs to essentially be worth that many points to counteract the first player's primary objective plays. Our prior analysis showed that the gains players get from going first are typically realized on turn two, when both armies are likely to still be relatively whole. A turn 5 army will look completely different, and may too often not have the units necessary to score hold three or hold more while still accomplishing key secondary objectives.
Let's go through these. We've already addressed the first - very few players regularly chose to go second before the FAQ. However identifying the impacts of this rules change on close games or games where the shift in scoring may make a difference is a bit trickier.
Examining Game States Going Into Battle Round 5
Let's start with the turn five issue. Figuring out how many games "end" before turn 5 is a difficult task, but we've got a few solid indicators we can look at, with the first being "how many games are blowouts?" We've seen before that a player's chances of winning hit 80% when they score 80 VP, but we haven't looked at the average final score differentials. The average score difference between winning and losing players is about 26 points, with a median of 21 points. The standard deviation on points differential is about 20 points, so I'd say that a blowout in a game of 40k is something probably in the area of outscoring the opponent by more than 40 points. Those make up about 23% of the games in our sample, giving us another quarter of so of games for which T5 scoring changes likely do not matter.
Statistically speaking, about 86% of the games in our sample went into the fifth battle round with one player having a higher primary score than the other. One good thing to know is that most games are pretty close with regard to primary VP scoring. More than half of games - 58% - see players coming into the fifth battle round with a primary VP score that has a closeable gap (15 points or fewer).
What about games that aren't massive points blowouts but whose conclusion is still certain, however? Examining game state, even with ITC Battles App data, is difficult. Secondary objective scoring by turn won't tell us the endgame story and so can be a bit unreliable. We do have primary VP scored by round however - if we look at the primary VP for both players coming into the fifth battle round, we can look at how that predicts the final outcome of the game. As you might expect, T5 primary differential and final score differential re highly correlated - there's a near one-to-one correlation between the two (r=0.94). This is partly because the final primary score differential only changes on turn five in about 47% of games played. If we look at this for on a per-score basis, we can put together the following chart:
A few things to note here:
- Coming back from being down 45 points on primaries is impossible unless you've got a painting score advantage over someone, but apparently coming back from being down 40 points on primary objectives has happened 3 times for players using the ITC Battles App to record games. You three players are legends.
- Pretty much any game with a 25-point or more primary VP differential going into battle round 5 is a lost cause. Players turn those around less than one percent of the time and a player going into round 5 up 25+ point will typically win by an average of 39.4 victory points, really close to our blowout threshold.
- Player 2 can't influence player 1's primary scoring on their turn 5, so if player 1 maxes out their primary VP, the best player 2 can hope for is to match it - this is one reason primary VP score differentials may not change as often as we'd suspect on Battle Round 5, even with endgame
So yeah, being down 30 or 40 points on primary objectives is a great way to lose the game and anything more than a 20-point discrepancy tends to be pretty hopeless. But we've seen recently during the LVNo that comeback wins as a result of the new scoring system can and do happen. So what does the impact of the new scoring look like? Well if we look at the chances of a comeback win based on the difference in primary scores between the two players coming into battle round 5, we can see some effects. Specifically, the player going second may now be more likely to make a comeback after coming into the fifth battle round down 15 points, though not necessarily more likely to snag a win if they're only down 5 or 10 points. This jump - nearly double - is statistically significant despite the small sample size and worth keeping an eye on. The reason for this may be because the second player is now more likely to be able to score all three primary objectives on the final turn of the game, making it more likely they can overcome a 15-point deficit.
So on the whole it looks like the FAQ has had a slight effect on game outcomes - about 2-3 percent of games so far, but that could be as high has 9-10% if more data shows the result for games with a 5- and 10-point differential going into the fifth battle round. With only 2-3% of games influenced however that means that the FAQ change also hasn't necessarily made a significant impact on the turn 1 advantage, if only because the opportunity for the scoring change to matter doesn't come up often enough. This is worth continuing to monitor for additional shifts and is something you can count on us covering again one month from now.
Next Month: More Death Guard and Dark Angels
That wraps up our meta analysis for February but join us next month when we come back and take an updated look at these results with more data - it's important to note that these results are preliminary and likely to change at least a little. It may be that as players acclimate to the new T5 scoring methods we see more of a shift that leads to a lower first turn advantage. Or we may see no change, but either way we'll be showing our work. We'll also be taking a deeper look at the impacts of the Death Guard codex as well as the new Dark Angels and how they impact the meta. Until then, if you have any questions or feedback, drop us a note in the comments below or email us at firstname.lastname@example.org.