# The Trouble with Practice Exams

#### by John Cunningham, W1AI

## Learn not memorize

As volunteer examiners we see people getting their ham radio license by memorizing the answers using practice exams, learning little in the process, then dropping out of ham radio because they don’t know where to begin. If you’re going to invest all those hours studying for the exam, most people want to learn something about ham radio in the process.

Practice exams do serve a useful purpose, proving your readiness for the exam *after *you have completed a course of study. However, they should not replace education.

This article explores some of the issues with getting a ham license using only practice exams. We begin with an examination of how the actual exams are structured.

## How Exams are Generated

The National Conference of Volunteer Examiner Coordinators (NCVEC) maintains the question pools used for all U.S. amateur radio examinations. The FCC regulations require at least ten times as many questions in the pool as appear on the actual exam, but the NCVEC generally exceeds that requirement by a considerable margin. As required by the FCC, the NCVEC publishes the question pools to the general public well in advance of their effective date.

Each U.S. question pool is divided into 10 topics, each topic into between 1 and 8 groups, and each group contains between 10 and 24 questions. While it is not required by the FCC, in all cases the number of groups is exactly the same as the number of questions on the exam.

Question pool | Effective dates | Topics | Groups | Questions on exam | Questions pool size |
---|---|---|---|---|---|

Technician | 7/01/2014 - 6/30/2018 | 10 | 35 | 35 | 426 |

General | 7/01/2015 - 6/30/2019 | 10 | 35 | 35 | 462 |

Extra | 7/01/2016 - 6/30/2020 | 10 | 50 | 50 | 712 |

The rules for generating exams only require that every exam have the same number of questions from each topic as there are groups in that topic, but the actual exams are always generated by taking one question, at random, from each group.

This creates an interesting phenomenon: questions in larger groups are underrepresented on the exams. Consider the following two groups:

- Group G9C (Directional antennas) contains 20 questions, so each question in the group has a 1 chance in 20 of appearing on your General exam.

- Group G8B (Frequency mixing; multiplication; bandwidths of various modes; deviation; duty cycle) contains 10 questions, so each question in the group has 1 chance in 10 of appearing on your General class exam.

While you will always get exactly one question from group G9C and one from G8B on the General class exam, any one *specific *question in group G8B is twice as likely to appear than any one *specific *question in group G9C.

Since the free online practice exams are generated in the same fashion as the actual exams, this phenomenon increases the number of practice exams you must take to have a high probability of seeing all the questions in the pool.

## How many random exams?

Assuming you decide to memorize the answers using random exams, just how many do you need to take? The answer depends on how good your memory is and how confident you want to be in passing. Some people are happy to squeak by on the second or third attempt, while others aren’t happy unless they score 100%.

Let’s assume your memory is pretty good, and you’re aiming for a score around 85%. After all, you’re looking for the *cheapest *way to get your license, so you don’t want to risk paying the $15 exam fee (varies by VEC) again.

So you decide that you want to see 90% of the questions at least two times. With a fairly good memory, as long as you cram most of your study into a few days immediately prior to the exam, you should be able to remember most of them. Your long-term retention will be poor, but that doesn’t matter, as you’re just memorizing answers, anyway.

Because of the way the exams are generated, the easiest way to determine the solution was to write a computer program that generates many random exams using precisely the same rules used to generate the actual exams. I averaged the results of 500 trials to eliminate the effects of random fluctuations. For the mathematicians in the crowd, the result is a binomial distribution.

This table shows how many practice exams you need, and estimates how long it will take, assuming you spend 30 seconds per question:

Question pool | Number of exams | Questions per exam | Time per exam | Total "study" time |
---|---|---|---|---|

Technician | 43 | 35 | 18 minutes | 13 hours |

General | 56 | 35 | 18 minutes | 16 hours |

Extra | 60 | 50 | 25 minutes | 25 hours |

In all cases, when you reach the point where 90% of the questions have been seen at least twice, there are still approximately 2% of the questions which have been seen zero times and 8% which have been seen just once. Unfortunately, at least 1/3 of the questions have been seen six or more times!

For those aiming for a higher score, practice exams are frustrating. The closer you get to 100%, the more it repeats questions you’ve already learned. To understand this, consider the case where you’ve memorized 90% of the answers. Since practice exams pick questions at random, that should mean that 90% of the questions it picks are ones you’ve already memorized.

The exam groups exacerbate the problem, especially since the largest groups tend to have the hardest questions. You take a 35-question practice exam, and you might only see 2 questions you haven’t yet memorized. On average, you might see one new question every 5-10 minutes.

While these statistics are useful predictors of total effort, you don’t need to count the number of practice exams you take. You’ll know you’re ready when you consistently pass them.

## How many to see them all?

Some people aren’t happy unless they’re sure they will get 100% on the exam, so they ask us how many exams to be *guaranteed *to see all the questions in the pool at least twice. Unfortunately, there is no number which absolutely guarantees it. It’s a matter of probabilities.

It’s like flipping a coin many times. Flip it once, and there’s a 50% chance you’ve only seen tails. Twice, it’s a 25% chance. Three times, it’s 12.5%. It keeps going like that forever, never quite reaching zero. After 100 flips, there’s still a .00000000000000000000000000008% chance you’ve only seen tails. There is *no finite number *of coin flips that *absolutely *guarantees you will see at least one heads.

However, it is possible to determine the number of random exams you have to take to have a 90% *probability *of having seen every question at least twice:

Question pool | Number of exams | Questions per exam | Time per exam | Total "study" time |
---|---|---|---|---|

Technician | 118 | 35 | 18 minutes | 34 hours |

General | 189 | 35 | 18 minutes | 55 hours |

Extra | 199 | 50 | 25 minutes | 83 hours |

## Not all created equal

Alert: As of this writing some websites are offering practice exams using expired question pools! Those are not the questions you will see on the exam!

If you are going to use practice exams, we think * HamTestOnline*™ has the best. Here’s why:

- Ease of use – You’re going to be answering several thousand questions, so you want it to be easy. You don’t want to have to maneuver the mouse to click a tiny circle every time.

- Immediate feedback – While not knowing your score until the end more closely simulates the real exam, immediate feedback is better for achieving your memorization goal. After you make a guess, you should see the correct answer while the question is still fresh in your mind.

- Save session – If you don't have time to finish the exam, you can sign in later, even from a different computer, and continue where you left off.

- Tracks history – It’s handy to be able to look back and review the questions you missed on prior exams.

- Integrated question pool display – A great way to accelerate memorization is to review the questions that give you trouble.

Here’s how the most popular practice exam sites stack up. (Green is good, red is bad.)

Website | Ease of use | Immediate feedback | Save session | Tracks history | Integrated question pool |
---|---|---|---|---|---|

™ HamTestOnline |
Click anywhere on the answer. | Yes | Yes | Yes | Yes |

QRZ.com | Have to click a small circle | Yes, but doesn’t show correct answer when you get it wrong | No | No | No |

eHam.net | Have to click a tiny circle. | No, have to wait until the end. | No | No | No |

AA9PW.com | Have to click a tiny circle. | No, have to wait until the end. | No | No | No |

W8MHB.com | Have to click a tiny circle, then click another button to advance. | No, forces you to answer (not skip) all questions first. | Yes | Yes | No |

KB0MGA.net | Have to click a tiny circle, then click another button to advance. | Yes | No | No | No |

There’s one additional feature we at * HamTestOnline*™ like the most (because it’s how we stay in business). At any time you can change your mind and switch to our subscription-based online courses. And, since we've been tracking your question history all along, you don't have to start from scratch. We know to focus on those topics and questions where you have trouble.

Hey, you might make our dream come true and actually learn something about ham radio after all!

Why our practice exams are the best

(This article is partly based on a paper I published in the *Journal of the Association of Computing Machinery (JACM)*: "Introducing Computer Simulation through Graphics," J. Cunningham, G. Summers, T. Honneycutt, 1976.)