Behind every question, or item, as they are referred to in the certification industry, is a collaborative process involving subject matter experts (SMEs) from across EMS, including Clinicians, Educators, physicians, and State EMS Officials representing diverse regions, practice settings, and perspectives. Together, they ensure examination content is fair, clinically accurate, and reflective of real-world practice.
“It’s you in the field who makes the exam, not us,” said Matt Ozanich, senior examinations program manager at the National Registry. “People don’t always understand that. There’s a perception that the exam is just written in an office somewhere, but that’s not the case. It’s you in the field who creates the content.”
From Field Experience to Examination Item
Before a question ever reaches a Candidate, it begins with EMS Clinicians and Educators who write examination items based on current practice and established references. These items then move through multiple rounds of internal review at the National Registry before ever reaching what is known as an “item review panel.”
“So, before we talk about the panel process, I'll back up a little and talk about where these items came from,” said Brian Hamilton, manager of content development at the National Registry. “These are items written by working Paramedics, EMTs, and Educators with whom we've contracted. Then we refine them internally before they ever go to a panel.”
Panel Review: Collaboration in Action
Once internal review is complete, a panel of EMS Clinicians from across the United States is assembled to evaluate each item in detail. “We have a representative sample from across the country,” Hamilton said. “They come in and look at the items, and we have dialogues about them. We modify them as needed, make sure they're clinically sound, appropriate for the audience, and measure the knowledge and skills that matter in practice.”
A typical panel may review hundreds of examination items over several days, with each question carefully evaluated, debated, and refined, and preparation for these sessions begins months in advance. "We're developing items throughout the year to prepare for the panels,” Ozanich said. “Then there's recruiting panelists, training, travel arrangements and all the preparation that happens before anyone even sits down to review a question.”
Before reviewing examination content, panelists receive training on item development, common testing pitfalls, and what to look for during the review process.
Every Voice Matters
Panelists are selected from diverse backgrounds, regions, and practice settings to ensure examination content reflects the national EMS profession.
Once discussions begin, no question is considered off-limits. Panelists are encouraged to raise concerns, challenge assumptions, and carefully examine each item. “If one panelist had a concern over the question, it was something that we all dissected and talked about, no matter how big or small,” said Jamie Earp, an EMS educator and AEMT from Decatur County Fire and Rescue in Georgia and a first-time participant in an AEMT Item Review Panel held May 19–21 at the National Registry headquarters in Columbus, Ohio. “It was a very respectful, very well-developed area to stop, think, and work together for the goal of making a comprehensive, fair test.”
Ozanich said one of the biggest surprises for new panelists is realizing how much influence they have. “So many eyes are on every item, and their feedback fully matters,” he said. “The people in that room have real influence over what happens to a question.”
Panel discussions often explore regional differences and whether items reflect consistent national practice. “We have to strike that middle ground,” Hamilton said. “It's a national certification. It's not a Texas certification, a Georgia certification, or an Alaska certification. The only way we can really do that is to have that many perspectives and views.”
That diversity is essential to the process. “I think that diversity is what brings about understanding,” Earp said. “You have people with different experiences, educational levels and perspectives. Having the opportunity to sit across from somebody who has experienced the same thing differently provides a perspective that I probably wouldn't have had otherwise.”
Looking Behind the Curtain
For many participants, the panel process reshapes their understanding of how examinations are built. “As an Educator, I wanted to peek behind the curtain,” Earp said. “Are these questions aligned with the textbooks? Are they aligned with the way we teach? Are they good, solid questions within the parameters of what students have learned?”
What he discovered exceeded expectations. “It was actually more in-depth than what I thought it would be,” Earp said. “I was very impressed with how the material is modeled and used. National Registry was very particular about the level at which material gets used and the evidence supporting it.”
William Lee, an AEMT from Vermont, arrived with a different assumption. “Before I came here, I literally knew nothing,” Lee said. “I just figured it was a bunch of people sitting in cubicles typing away.”
That perception quickly changed. “The National Registry doesn't actually write their questions the way I thought they did,” Lee said. “You've got people coming from across the country, and everybody is reviewing the questions to make sure they're accurate, make sense, and are fair for people taking the test.”
Maria Barcena, an AEMT from Utah, came away with a greater appreciation for the rigor behind the process. “The experience was great,” Barcena said. “I appreciate how we get to learn about what makes a ‘good’ question and how thorough the National Registry is about the citations for each question. It’s interesting to know how one question moves through an extensive process before even seeing the candidates.”
The opportunity to collaborate with EMS Clinicians from different regions also stood out to her. “Textbooks can be so obscure sometimes, so it was refreshing to discuss procedures, treatments, and considerations that take place with other states and areas,” Barcena said. “That way, we can evaluate every Candidate without regional bias.”
Ozanich noted that misunderstandings often stem from the stress of testing itself. “When you're taking a test, you're stressed out,” he said. “Sometimes people remember one unfamiliar word or concept and assume that's what the entire exam was focused on.”
He encourages Educators and Candidates to rely on official testing resources rather than informal interpretations of exam content.
Built on Evidence and Collaboration
According to Hamilton, nearly every item generates discussion. “Most items have some sort of discussion, some sort of refinement,” he said. “By the time it gets to the panel, you'd think there'd be nothing left to talk about, but there always is.”
Those conversations often lead to meaningful improvements in clarity and fairness. “We take the feedback very seriously,” Hamilton said. “It's a continual feedback loop of learning for everybody involved.”
For Ozanich, the process reflects the nature of medicine itself. “I love my job because I get to debate about medicine with people all day long,” he said. “Medicine isn't black and white. You're constantly evaluating different perspectives, justifying decisions with rationales, and making sure the question is testing the right thing in the right way.
The process is grounded in evidence-based references and current EMS practice.
“I learned that it's a priority,” Earp said. “National Registry is hyper-focused on doing the right thing and finding information that supports growth in both testing and provider development. Their focus is making sure that we are providing the highest level of providers that we possibly can in the fairest way possible.”
For Brian Wilson, a physician and EMS Medical Director, the impact is direct. “The quality of the Clinician that comes out and walks into our station to start working for us matters,” Wilson said. “Each question is thoughtfully curated and gone over in excruciating detail to make sure it's assessing what we're trying to assess and doing it in a valid way.”
More Than a Test
By the end of a panel, many participants leave with a new perspective. “A lot of people come in with a view of the National Registry's tests that might not be 100% positive,” Hamilton said. “But it's pretty rare that someone leaves these panels and doesn't have a new respect for the amount of work, care, judgment and expertise that goes into every test question.”
That sentiment is echoed by the participants themselves. “I think it's a testament to the good work that's being done,” Earp said. “National Registry is serious about making sure the material meets students' needs, educators' needs and the profession's needs.”
At its core, the process ensures Candidates are evaluated fairly while protecting the public and supporting the EMS profession. “It's not a machine up there spitting out questions,” Hamilton said. “It's just a whole lot of work and a whole lot of caring.”
The next time someone asks where National Registry examination questions come from, the answer is simple, they come from the field.
Through the item writers and panelists, who are working EMS Clinicians, Educators, and physicians from across the country, EMS professionals help shape the certification examinations used to assess the next generation of Clinicians. Behind every question is a team of people committed to ensuring the examination is fair, relevant, and reflective of real-world practice, helping prepare future EMS Clinicians to serve their communities with confidence.