Introduction

In the first part, we looked at some common myths and facts about how the GRE is scored. In this second part, we start taking a deep-dive into the intricates of the test and how it is scored. This part focuses on the questions themselves.

This blog post was made by working through various sources, including LLMs. However, a staff member wrote this post by hand. This blog post might be a bit simplified at places for readability.

How a test is made

To create a test, you need questions. And each question has a bunch of parameters that ETS uses when constructing the test (which can well be automated). Some of them are obvious, such as

• “Difficutly” level (we’ll explain more below)
• Topic (you don’t want half the test to cover the same concept, for instance)

But also some that are less obvious, such as

Figure 1 This is a common example of a question that exhibited DIF issues and would not show up on the GRE of today as a result. ^[1]

• Differential item functioning (DIF). This refers to how likely a question (or test) is going to vary amongst different groups with the same level of ability. For example, if a question turns out to be harder for blacks as compared to whites, could it be because the question is testing something that favours whites^[2]? That would be unacceptable, obviously. This does not mean that the scores across different ethnic groups must be the same, but rather that someone of the same ability should have an equal chance of getting the question correct. It’s a little like finding a verbal RC question easier to answer because the question is on a topic that you like.
• Test exposure. There is a risk of too many students seeing the same question - the more a question is seen, the greater the chance of the question being leaked. Hence questions are rotated and retired (never used again) regularly.

But we’re mainly interested in something else, which we’ll see in the next section.

What’s in a question?

To begin with, here are some examples^[3]:

Figure 2 a=0.818254, b=0.670479

Figure 3 a=0.502194, b=-0.303289

Figure 4 a=1.008065, b=-0.727798

Notice that we provided two real numbers, a and b, for each of the three questions. These are really important numbers; the test is primarily scored using these two numbers for each question.

The discrimination parameter (a)

Let’s have a small experiment. If you ask 100 GRE students to solve, say, 5 + 5, pretty much everyone will get it right. That question didn’t separate anyone at all. Similarly, if you give them an esoteric, out-of-GRE-scope question, almost no-one (barring those that know that concept) will get it right. Again doesn’t tell anything about GRE. Conversely, a grade 9 algebra question will give you some indication about the students’ GRE potential, because such a question is likely to come in the GRE as well. Some will get it right, while some won’t, but you’re getting a signal about how good the student is.

The discrimination parameter of a question tells how good the question is in separating candidates at a certain level (the higher, the better). Consider a question with a high discrimination parameter that is meant for a 155. A 160 will almost always get it right, while a 150 will almost always get it incorrect. And the chance a student gets it right rises steeply at 155. On the other side, if the discrimination parameter is low, then as the expected score of a student increases towards 170, the chance the student gets it right won’t rise as quickly. It’ll be slow. Especially with only 27 questions for each measure of the test, that’s not good.

The difficulty parameter (b)

This simply tells how hard the question is. Higher the b, harder the question. b is typically scaled to a standard normal distribution (i.e, mean 0 and standard deviation 1). So b = 2 is a very hard question, b = 0 is average-level, and b = -2 is a very easy question.

The ability parameter (θ)

It tells how good the student is, and is connected to a and b. Like b, it is scaled to a standard normal distribution, and hence θ

• θ = 2 is a student that’s going quite well
• θ = 0 is an average student
• θ = -2 is a weak student (i.e, will get a low score)

Given a certain value of a and b, for a certain θ, we can find out the likelihood a student at that θ will get the answer correct. To give an example, consider the first of the three ETS examples (a=0.818254, b=0.670479):

• An average-ability student (θ = 0) has about a 36.6% chance of getting it right
• An high-ability student (θ = 2) has about a 74.8% chance of getting it right
• A weaker student (θ = -2) has about a 10.2% chance of getting it right

For a student to have a 50% chance of getting it right, θ = b ~ 0.67. So this question is targeted for an above-average student, though not only for the top students.

For many tests, where you cannot skip questions, θ directly decides your score and simply scaled. For example, we can scale θ to a mean of 150 and a standard deviation of 10. So a student with θ = 2 will get a 170, θ = -2 a 130 and θ = 0.6 a 156.

Getting a and b

Given that every question must have an a and b, how does ETS get those values? Before the shorter GRE, almost every student had an (hated) experimental section, where the questions in that section do not count towards the score, but the student wouldn’t know which section that was, and hence had to attempt every section to their best. However, this is how ETS determined the difficulty level, and hence a and b, of a new question, alongside checking whether the question had any flaws or DIF issues (see above). The pre-Focus GMAT did the same; there were a few questions scattered throughout the test that were unscored.

Given that the shorter GRE no longer has an experimental section, how are a and b determined? We’re not sure. Some theories are that

1. They are deriving the new question from an older question that does have a and b - but generally that isn’t good practice.
2. They are not developing new questions for the GRE.
3. They are doing some post-test analysis to find a and b. This makes sense in tests like the SAT where you do not get an unofficial score after the test. We find this unlikely as well because you do get an unofficial GRE score after the test.

GMAT Focus is frustratingly coy about this; it appears that they do have experimental questions^[4] but they do not indicate this on the score report at all (and hence it is not even clear whether every student gets experimental questions). The percentiles seem to be the only clue.

Bringing it all together

Notice that in the previous section, we found out the probability of a student that gets the answer correct for a certain question given a and b. This does have an equation, and it’s called the 2PL IRT model. PL being parameter logistic (parameter - number of variables and logistic being the model), and IRT being item response theory (the core approach: predicting a student’s performance based on the questions they have attempted). Here’s the equation: 

$$\mathbb{P}(A) = \frac{1}{1 + e^{-a(\theta - b)}}$$

BTW: e is a constant, about 2.7, and is not covered on the GRE quant.

This tells us the probability of a student at θ (or scaled to a certain score) getting that question correct. Higher the θ, higher the chance. It is also important to note that everything is a probability. It is of course possible that a 170 (i.e, a high θ) will get a b = -3 question wrong (such as a careless slip), just like a very weak student getting a question aimed for a 170 right. What the model cares about is the chance. It’s not like the weak student will get all hard questions right by luck, so the impact is minimal.

You can play around with this yourself using this embed link (which is the same as the above image). Notice what happens when you change a and b; when a increases, so does the steepness. When b increases, the graph goes "towards the right" and vice-versa. 

How it usually works

Figure 5: A simulated example of a 30-question quiz. If we use the previous example and scale it with a mean of 150 and a standard deviation of 10, the student would end up with a 155. 

1. The test starts with an average-difficulty problem (say b = 0). At this point it knows nothing about the student.
2. If the student gets it right, the test updates what it thinks is the student’s ability (from θ = 0 to say θ = 1). It then gives a harder problem (b = 1) to the student.
3. If the student gets that right as well, the test finds an even harder question.^[5]
4. If the student gets the first question wrong, the test gives an easier problem (b = -1) as the second problem.
5. This keeps on going till the end of the test as the student gets problems right and wrong.
6. The final θ is converted to a scaled score and that’s what the student sees.

The difference in the GRE is that the student can skip questions in a certain section, which means that θ cannot be used directly, as the score can only depend on the number of questions attempted correctly (and crucially not which questions were correct/incorrect as is normally the case like the GMAT). ETS came up with a pretty smart alternative instead, which deserves its own blog post.