National Evaluation of Specialty Selection: final report

The National Evaluation of Specialty Selection (NESS) covers the selection processes for 71% of 4,060 training posts in England and 2,751 General Practice (GP) posts in the UK available at ST1/CT1 level in 2009, making it the largest evaluation of specialty selection in the world. Designed as a cross-sectional study using quantitative and qualitative methods, the report describes and analyses the effectiveness and suitability of selection processes across 13 specialties. Taking account of the GMC’s requirement that ‘processes for recruitment, selection and appointment must be open, fair and effective’, our principal analytical model examined the acceptability, reliability, validity and cost-effectiveness of the selection processes.

Our analysis of personal characteristics as determinants of selection scores raises concerns about fairness, particularly with respect to candidates trained overseas. Shortlisting scores were generally good predictors of final selection scores. However, internal consistency and pass mark reliability in the majority of selection systems were undesirably low given the high-stakes nature of specialty selection. Consequently, we recommend: having more stations but with a single assessor; more routine data analysis and sharing results between specialties; greater attention to assessor training and the piloting and evaluating of alternative training models; and using rigorous standard setting processes. Recent moves towards integrating selection within each specialty at a national, rather than a Deanery level, and eliminating shortlisting have improved the cost-effectiveness of selection.