Update Quiz Update: Outlaws of Thunder Junction Quiz Closed and Modern Horizons 3 Quiz Open

The Outlaws of Thunder Junction update quizzes just closed, so as has become tradition, I’m going to take a moment to walk through some questions from the quizzes and talk about a few of the questions. You can read the wrapups of the Lost Caverns of Ixalan and Murders at Karlov Manor quizzes on our blog. 

I’m also announcing today that the Modern Horizons 3 Quizzes are available now for all Level Two+ Judges. Note that this is a much longer delay after set launch than usual, and it was not the fault of the Update Quiz team – Steven and the rest of the team were ready before the set launched, and some unrelated factors kept the quiz from going live as scheduled.

As usual, I’ll start by asking you to join me in thanking the volunteers who created these exams. These judges put in time to help keep all of us up to date, generating high-quality questions in the short window between when cards get revealed and when players get them in their hands – no one on this project gets advance information about the upcoming set, so there’s a period of intense work as cards get previewed to put together a quiz that’s fair and educational. On Outlaws of Thunder Junction’s quiz, Steven Zwanger led the team, development work on JudgeApps was done by Paul Baranay and Dan Collins, and questions were written by Nathan Long, Steven Zwanger, Tobias Vyseri and me. If you’re interested in joining the update quiz team and writing questions (which now counts toward your maintenance requirements), please email Steven. 

A Fistful of Exam Stats

The quiz was five questions long (four rules, one policy), and judges needed a 60% or better score to pass. The advanced question pool had 11 questions (enough for two attempts), while the standard pool had 17 questions (enough for three attempts). This set’s quiz was also unique in that we published it before the Prerelease, instead of waiting for Toby’s article to come out. We’ll be posting the MH3 quiz early as well, and posting a poll on JudgeApps over the summer to gauge judge opinions on timing.

The standard quiz was attempted 133 times by 109 judges. That’s 41% of eligible test-takers – if you’re a Level Two or Level Three Judge who wasn’t in the 41%, remember that update quizzes are the easy way to fulfill a maintenance requirement. Here’s a chart on how folks did:

93% of test-takers passed on their first attempt, and 98% passed within two tries. 

The advanced quiz was taken 20 times by 16 judges, 55% of Judge Foundry’s L4s and L5s. Here’s the chart for the advanced quiz:

Did an L5 actually get none of the questions right? I doubt it, and I expect we’ll get another comment like this one after this article goes live.

The Good, the Bad and the Ugly Questions

I talked a lot about question relegation in the MKM wrapup. I don’t want to repeat what I wrote there, but I will talk a little about practice quizzes. Judges will consume a functionally infinite amount of content, and then ask for more content. Writing good exams is tough – you want questions that are challenging but fair, and you want them to be appropriately tuned for each level. 

But sometimes, we have to look beyond how a question is performing to see if it’s tuned correctly. We’ve added questions to exams that are a little easier than our target, because we think it’s that critical that every judge knows them that we’re willing to give a free point to a lot of folks so those who didn’t know will learn it. Similarly, there are questions where the pass rate is way lower than our target, but we keep them on the exam because this is something we expect judges of a given level to know it, we’ve already created the most straightforward way of testing that knowledge, and even though the question has a low pass rate, that’s a sign that we need more education, not that the question is too difficult.

On this exam, here’s how relegation broke down:

Specific Questions

I usually (read: the last two times) only walk through three questions per exam, but there were four questions on this exam where results were interesting enough to merit deeper discussion. It’s very possible that I’ll be kicking myself in a few sets when I have nothing to talk about, since I’ve already used up a lot of the good question-specific topics, but that’s a problem for future me. 

Answer Spree

Our first question is Q2739, a rules question on the Standard Update. This question’s pass rate is 34%, way under our target for the evaluation exams.

Alexis casts Getaway Glamer. Which of the following are true? (Select all that apply.)

1. If Alexis controls Helm of Awakening and she chooses both modes, she pays {2}{W} to cast Getaway Glamer.
2. Alexis may cast Getaway Glamer off of The Key to the Vault and has the option to pay the spree costs. If she doesn’t pay for any additional costs, Getaway Glamer resolves but doesn’t do anything.
3. If Alexis chooses both modes, the mana value of Getaway Glamer is 1.
4. If Alexis chooses both modes, the mana cost of Getaway Glamer is {3}{W}.
5. None of these are true.

This question is a multi-select, and it boils down to “Tell me everything you know about Spree.” It’s easy to write these questions, a lot easier than writing single-select questions, because you don’t have to come up with a series of plausible wrong answers and splits. However, they’re a lot harder on the test-taker, because you can’t use splits to help you answer the question and because they require more comprehensive knowledge.

We try not to use multi-select questions on lower-level tests, and to moderate their use on high-level exams. They’re a way to test knowledge on multiple aspects of a rule at once, which is valuable when you only get so many questions, but their pass rate is always lower. That said, players expect us to get every part of their questions right, and they don’t often give us easy splits to use, so some multi-select questions are okay. 

Almost everyone correctly selected A (25/28) and C (27/28). And almost no one selected E (1/28) and B (6/28). But D is where the wheels fell off the wagon stagecoach. 14/28 judges, half of the judges who saw this question, selected D. That might be a sign that we should do more testing on mana cost with Spree, in a new question that just tests that aspect. So keep an eye out for a question like this on a future exam. As for Q2739, it’s being put out to pasture on the Hard Rules exam.

Saddling Down

Our next question is Q2702, another rules question on the Standard Update. This question’s pass rate is 87% – that would normally make it a great candidate for the L2 exam, but read on.

Armando controls Bridled Bighorn, The Gitrog, Ravenous Ride and three 1/1 Bird tokens. Which of the following actions can he take during his precombat main phase? (Select all that apply.)

1. Tap Bridled Bighorn to saddle itself.
2. Tap two 1/1 Bird tokens to saddle Bridled Bighorn.\
3. Tap Prairie Dog that entered the battlefield this turn to saddle Bridled Bighorn.
4. Saddle Bridled Bighorn after its saddle ability has already resolved that turn.
5. Tap The Gitrog, Ravenous Ride to saddle Bridled Bighorn.
6. Tap three 1/1 Bird tokens to saddle Bridled Bighorn.

We’ve got another multi-select here, and another question that’s “Tell me all the things you know about [set mechanic].” So why did judges do so much better here than on Q2739? I’ve got a few theories.

First, there wasn’t one killer answer here. All five of the correct answers received 41-44/44. And the incorrect answer only received 1/44.

Second, everything you need to know to answer this question correctly is in the reminder text. Spree messes with some fundamental building blocks of Magic, and requires you to know how to cast a spell. Saddle mostly just requires you to read the reminder text, and if that isn’t enough, you can just think of how Crew works and you’ll be mostly fine.

Despite this question being in the sweet spot of difficulty, I don’t think it would make a good candidate for an evaluation exam, because it’s all about one non-evergreen mechanic and because it’s a multi-select. While we want you to know everything about Spree before you judge a Sealed OTJ RCQ, in a year we won’t expect every L2 to know all these things about it off the tops of their heads. So this question is heading to Easy Rules Practice.

Reading, the Key to Everything

Our third question is another rules question from the Standard Quiz. Meet Q2736, with a pass rate of 62%. 

Archie controls Loot, the Key to Everything, Scene of the Crime, Kraum, Violent Cacophony, and Glittering Frost. During Archie’s upkeep, how many cards does he exile to Loot, the Key to Everything’s triggered ability?

1. Two.
2. Three.
3. Four.
4. Five

This question has a lot in common with the previous question, which is why I’ve sequenced them this way. Specifically, they’re alike in that everything you need to know to answer it correctly can be found on the card. We call these RTFC questions, and we sometimes include them in update quizzes because some cards can be tricky, and we want you to know their tricks before you see them on the floor.

Now, you do need to know what is a type, and how supertypes and subtypes aren’t types. But the most common incorrect answer here is C (8/26), which most likely means that people were counting the Scene of the Crime, so they missed the word “nonland” in Loot’s ability. 

RTFC questions are good to occasionally appear on update quizzes, but they’re bad for evaluation exams, because they don’t really test whether judges will read the cards carefully on calls. I would love for every judge to carefully read the cards on every call… I would even love for me to carefully read the cards on every call (sorry, players at SCG Baltimore with a Yasharn, Implacable Earth). But judges are going to be far more invested in passing an exam then they are in every judge call, and they’re going to read, then read, then read again and get this question right on a test when they might not on a call (or in an update quiz). For that reason, this question is moving to Easy Rules Practice.

Cold Case

Since we haven’t done a policy question or a question from the Advanced Quiz yet, let’s do Q2726, which is both of those things. It has a pass rate of 90%.

Alma casts Stop Cold on Nilda’s Bird Token, but doesn’t tap it. On Nilda’s turn when she goes to declare attackers, Alma mentions the token should be tapped. A judge determines that Alma has committed Game Play Error–Missed Trigger. What is the correct penalty and fix?

1. Alma receives a Warning. The triggered ability resolves immediately, tapping the Bird token.
2. Alma receives a Warning. Nilda chooses whether the ability goes onto the stack now or is skipped.
3. Alma receives no penalty. The triggered ability resolves immediately, tapping the Bird token.
4. Alma receives no penalty. Nilda chooses whether the ability goes onto the stack now or is skipped.

This might seem like an odd policy question for the exam. There’s no new policy here, and the situation is pretty straightforward. Why test for it?

Part of the point of an update quiz is to refresh judges on policy. We’ve seen judges forgetting about this corner of Missed Trigger fairly regularly, mostly because it primarily comes up at Limited events where Claustrophobia effects are either good or plentiful. Judges’ experience about policy at events is a good way for us to gauge what parts of policy could use more testing, and this one definitely fits.

The question is on the Advanced Quiz not because we think it’s particularly hard, but because we want our Team Leads and Head Judges to be aware of it, and putting it on the update quiz was a way to do that. With that being done, the question will move to Policy Practice – it’s too easy for the Advanced Rules and Policy, there’s a better version of it on the Level Three Policy Exam, and it’s too niche for the Level Two Policy Exam. So to the practice pool it goes!

Riding off into the Sunset

That’s it for our wrapup of the Outlaws of Thunder Junction quiz – I hope this was useful! You can take your Modern Horizons 3 Quizzes in the Exam Hub on JudgeApps now. The MH3 Quiz will be open until July 24th – see you next update!