Study evaluates concordance between internal medicine MOC exam and common office and hospital diagnosis codes

Questions on the internal medicine Maintenance of Certification (IM-MOC) exam had a 69% concordance rate with conditions seen in general internal medicine practices, a recent study from the American Board of Internal Medicine (ABIM) has found.

Researchers from ABIM looked at IM-MOC exams from 2010 to 2013 and used the examination blueprint, described as a specification table “describing and examination's scope of content, clinical tasks (eg, diagnosis, treatment), and functional requirements (eg, item and examination statistics),” to calculate the percentage of questions that covered each of 186 categories of medical conditions. They also estimated the conditions that general internists see in clinical practice by looking the primary diagnosis for 13,382 office visits included in the 2010-2013 National Ambulatory Medical Care Surveys and 108,472 hospital stays included in the 2010 National Hospital Discharge Survey.

The study's main outcome measure was concordance between percentages of MOC exam questions and percentages of the 186 medical condition categories seen in practice (office visits or hospital stays). A condition category was considered concordant if the absolute differences between the percentages of examination questions and the percentages of office visit or hospital stay conditions were lower than the defined relevant concordance thresholds (0.74% for office visits and 0.51% for hospital stays). The study results were published June 13 by JAMA.

A total of 3,600 questions were administered during the 2010 to 2013 IM-MOC examination periods, with 3,461 (96.1%) mapped into the 186 study conditions. When percentages of these conditions seen during office visits and hospital stays were compared with the percentages of questions seen on the IM-MOC exam, 2,389 exam questions (69%) involving 158 conditions were considered concordant. When office visits only and hospital visits only were compared with exam questions, 2,010 questions (58.08%) involving 145 conditions and 1,456 questions (42.07%) involving 122 conditions were considered concordant.

The study authors stated that data on conditions seen could have been affected by errors in the medical record, that data from the National Hospital Discharge Survey covered only 2010 and do not link hospital stays to physician specialties, and that the study did not use the revised ABIM exam blueprint developed in 2015, among other limitations. They noted that ABIM's examination committee considered the concordance data in this analysis when reviewing the 2015 blueprint and that the committee also made changes of the exam content based on physician feedback regarding importance and frequency of conditions seen. “Future research is needed to track whether changes in the IM-MOC examination that were informed by this and other analyses resulted in discernible improvement in the relevance of the IM-MOC examination content,” the authors wrote.

The authors of an accompanying editorial said that the current study “represents an important effort to provide evidence on the validity of the recertification process and its relevance for practicing internists” but called for additional analysis and replication of the results by independent investigators to confirm validity.

“The ABIM must strive for continuous improvement and is most relevant and useful when it asks for help and listens to its members, a process that it has been involved with for the past several years,” the editorialists wrote. “Open, ongoing evaluation of the recertification examination will be essential, along with frank and productive discussion about how to ensure the continuous improvement, excellence, and relevance of the recertification process.”