Practice Analysis and Test Specifications Development

A practice analysis, also known as job analysis or role delineation study, is a critical first step in building a sound and legally defensible credentialing program. It is used to define the constructs that are critical to assessing performance relevant to the credentialing program. The results of the practice analysis are used to develop test specifications and examination content. By establishing the link between real world practice and test content, the practice analysis provides empirical evidence that your examination is job-related. This forms the foundation of examination validity and legal defensibility.

Unmatched Expertise and Experience

PES has conducted practice analysis studies for a broad range of professions, and understands the subtleties of these investigations, including those related to professions with evolving practice or standards and/or with an international dimension. We bring creative expertise to solving the unique problems posed by each study, and are accustomed to working very closely with credentialing program sponsors and their stakeholders.

PES has extensive experience developing certification and licensure examination programs based on practice analysis studies. We understand the complexities involved in building and maintaining a credentialing program that meets the needs of diverse stakeholders, including state boards, members of the workforce, contractors, employers, and the public.

“PES’s knowledge and insights relative to the role delineation process, and expertise in facilitating our SME committees contributed significantly to the richness of the PMP and PgMP studies, the results of which will serve PMI very well going forward.”

Nathalie Collautt, Exam Content Supervisor
Project Management Institute

The PES Approach

The PES approach to practice analysis is to deliver results that go beyond the basic development of test specifications. We design practice analysis studies that enable you to achieve multiple organizational and credentialing goals. We partner with you to apply the results of a practice analysis to:

• Develop recommendations and strategies for implementation and enhancement of credentialing programs.
• Develop and implement eligibility (experience and education) and examination requirements.
• Develop and implement written and/or practical examination programs based on the test specifications.
• Explore differences in practice, to guide outreach activities.
• Identify emerging areas of practice and new credentialing opportunities.
• Evaluate the meaning and value of your credential to stakeholders.
• Develop pre- and post-professional educational materials and/or training programs.
• Augment your professions competency framework.
• Develop self-assessment materials to enhance continuing competence.
• Create/enhance guidelines for continuing professional development.

The PES Process

The PES process for practice analysis and test specifications development adheres to all legally recognized and professionally accepted standards for ensuring the content validation of credentialing examinations, and truly represents industry best practice. Within that context, we tailor each study to the unique needs of a credentialing program and profession. Our process typically includes the following key elements:

• Delineation of Practice—We partner with content experts—recognized authorities in the field, who are carefully chosen to represent the diversity of practice—to construct a detailed content- or process-based description of practice that includes the major performance domains of the profession, key tasks within the performance domains, and the knowledge and skills (KSs) required to perform those tasks. Alternately, we may structure the delineation around the identification of foundational and functional competency areas, specific competencies, and behavioral exemplars describing evolving levels of competency. We gather information using techniques such as daily logs, phone-based critical incidents interviews, independent reviews, focus groups, and assembly and review of relevant documentation such as job descriptions and curricular materials within the profession or industry.
• Validation Survey—We validate the description of practice through a web-based survey of a representative sample of current practitioners, locally, nationally, or internationally. Survey participants review domains, tasks, and KS statements and rate them using a set of validation scales designed specifically for your program. The validation scales address such parameters as frequency and criticality of task performance. We analyze the survey ratings to refine the detailed description of practice and to identify any differences in practice related to professional or demographic variables. We then prepare a written technical report documenting the results of the practice analysis study. Upon request, we can also present those results at a national conference, or prepare non-technical articles for publication in a professional journal. PES staff will work with you to facilitate a high return rate and ownership of the results by your key stakeholders.
• Test Specifications Development—Working with content experts, we take the results of the practice analysis to develop a set of test specifications that are based on empirical evidence and subject-matter expert judgments. We use the ratings from the validation survey to establish weightings for the different task and knowledge areas within the practice. Combining the task and knowledge areas and their relative weightings, we then develop a set of test specifications that will determine the content of your examination. PES is experienced in guiding subject-matter experts in reviewing the empirical results of a practice analysis and filtering these results to create legally defensible test specifications that guide your examination development process.

Frequency of Practice Analysis

Industry standards recommend that a practice analysis be conducted every 5 to 7 years, depending on the rate of change of occupational practice. At PES, we help to determine an appropriate timeframe for conducting periodic practice analyses based on the rate of change within your profession. We also conduct targeted practice-analysis update studies designed to focus on specific areas of performance where work practices may be changing rapidly within your profession.