Most of the resume review and assessment updates #241
Conversation
…usds/SME-QA-Website into reorg-interviews-to-assessments
…usds/SME-QA-Website into reorg-interviews-to-assessments
add Will's build command
|
I'm merging this to see if it builds during off hours so I have time to fix it tomorrow if need be. Can always revert. |
|
It worked! |
| {{ page.intro }} | ||
| </p> | ||
|
|
||
| Before you [schedule training](training.md) for the first round of interviews, divide the SMEs into two groups based on which round of interviews they will conduct. Consider how many applicants passed resume review, as well as any scheduling conflicts you may be aware of at this time. Send SMEs calendar invites for the days or weeks they'll be conducting interviews as soon as possible. These invites should just be a placeholder notification of when interviews will generally take place. To solicit the specific times that SMEs will make available for interviews during this period, send them the <a href="{{site.baseurl}}/toolkit/assessments/email-template-sme-availability/">SME Availability Email</a>. |
There was a problem hiding this comment.
should we say here- decide if you're having 2 interview rounds or one first or something? this page assumes 2 rounds which may confuse folks.
| section: hiring-phases | ||
| title: Notifying Applicants After Interviews | ||
| sidenav: hiring-phases | ||
| phase: phone-assessment-interviews | ||
| phase: assessments | ||
| sub-phase-order: 040 | ||
| intro: After SMEs have completed interviews, you should send notifications to all applicants indicating their status. Use these NOR codes and notification messages to communicate with applicants. |
There was a problem hiding this comment.
it seems like we're still implying there must be an interview after the demonstration project/written assessment. is that right?
There was a problem hiding this comment.
Good catch. I didn't update this and have here 497f103
| sub-phase-order: 030 | ||
| title: Reviewing SME Assessments | ||
| intro: As SMEs submit their interview assessments, check their work and email them with any necessary feedback. | ||
| intro: As SMEs submit assessment feedback, check their work and provide necessary feedback. |
There was a problem hiding this comment.
strange to use feedback twice in the sentence? maybe it's fine
There was a problem hiding this comment.
Edited to "intro: As SMEs submit their assessment reviews, check their work and provide necessary feedback." and added a few more edits here: 99f87d2
| @@ -17,7 +17,7 @@ intro: As SMEs submit their interview assessments, check their work and email th | |||
| <div class="usa-alert__body"> | |||
| <h3 class="usa-alert__heading">Do not conduct a separate minimum qualification review</h3> | |||
| <p class="usa-alert__text"> | |||
| After SMEs have completed interviews, do not conduct a separate minimum qualification review. For more information, see "<a href="{{ site.baseurl }}/about/differences/">How our process is different</a>." | |||
| After SMEs have completed assessments, do not conduct a separate minimum qualification review. For more information, see "<a href="{{ site.baseurl }}/about/differences/">How our process is different</a>." | |||
There was a problem hiding this comment.
maybe change "do not" to "you do not need to" ? less prescriptive
There was a problem hiding this comment.
This is previously existing text.
I'm ok to change it, but I think it might be better if left - if HR does something separate, it negates what the SME did, right?
| @@ -31,8 +31,8 @@ Verify that every core competency has a rating of Does Not Meet, Meets, or Excee | |||
|
|
|||
| ### Calculating an Applicant's Final Assessment Rating | |||
|
|
|||
| After SMEs have addressed your feedback for their interview assessment forms, calculate and save the applicant's final assessment rating. | |||
| After SMEs have addressed your feedback for their assessment forms, calculate and save the applicant's final assessment rating. | |||
There was a problem hiding this comment.
"in their assessment forms" not "for their assessment forms"
There was a problem hiding this comment.
maybe link to the toolkit and say "Assessment Form (coming soon)" there
There was a problem hiding this comment.
Also addressed 99f87d2
I don't know how long Patrick will take so didn't add the (coming soon) if that's ok.
| title: Training SMEs to Conduct Interviews | ||
| intro: While phone interviews are being scheduled with applicants, facilitate a mandatory two-hour interview training for all SMEs who will conduct the next round of interviews. The training should explain how to conduct structured interviews, and then assess the applicant’s proficiency with the required competencies based on their interview responses. | ||
| title: Training SMEs to Conduct Assessments | ||
| intro: Facilitate a mandatory two-hour training for all SMEs participating in an assessment round. The training should explain how to review the assessment (including conducting an interview, and how to assess the applicant’s proficiency levels. |
There was a problem hiding this comment.
how to review the assessment against the required proficiency levels (structure interview, written prompt, or demonstration project)
There was a problem hiding this comment.
Also, good catch on me never closing the parenthesis there.
| phase: assessments | ||
| sub-phase-order: 020 | ||
| title: Written Assessments | ||
| intro: SMEs may conduct one or two rounds of written assessments in the assessment phase. |
There was a problem hiding this comment.
I'd rather say "During job analysis SMEs and HR may decide to turn the first structured interviews into a written assessment or demonstration project instead."
There was a problem hiding this comment.
Edited to "During job analysis, SMEs and HR may decide to do a written assessment or demonstration project instead of a first-round interview."
| {{ page.intro }} | ||
| </p> | ||
|
|
||
| You may choose to do a written assessment for one or both assessment phases. |
There was a problem hiding this comment.
i dont like this line bc it assumes no structured interview which i dont love recommending w/o a team conversation. i think we've always said an interview is mandatory part of SMEQA. we can always revisit in the name of flexibility and agencies can do whatever they want.
There was a problem hiding this comment.
Edited to this:
If your SMEs opt for a written assessment or demonstration project instead of a first-round interview, you will need to prepare questions and training for each different assessment round.
The assessment type should be appropriate to evaluate the applicant's proficiency level within the examined competencies
In this commit: a553fb2
| @@ -27,5 +27,5 @@ It's critical for you to identify a project manager to manage the deliverables r | |||
| - **Resume Review Training:** Before SMEs begin reviewing resumes, HR will conduct a mandatory two-hour resume review training, followed by 1 hour of reviewing actual resumes in their list with HR present to answer questions. | |||
| - **Resume Review:** HR will divide applicants evenly among groups of two SMEs. They will have four days to review all the resumes in their list. Each resume requires two reviews. | |||
| - **Resume Review Tiebreaker:** Over a two-day period, an additional SME will be assigned to each divided list of applicants to break any ties between the first two assessments. | |||
| - **Phone Assessment Interview Training:** While phone interviews are being scheduled with applicants, HR will conduct a mandatory two-hour interview training for all SMEs conducting the first and second phone assessment interviews. This training explains how to conduct structured interviews and assess an applicant’s proficiency with the required competencies based on their responses. | |||
| - **Phone Assessment Interviews:** Over a 2-3 week period, SMEs will contribute up to 20 hours per week conducting interviews with applicants. This assumes each interview is one hour long. This includes a recommended 45 minutes after each interview for completing the interview rating sheet and adding their feedback to the talent acquisition system. The time required will vary depending on how many applicants moved forward past the resume review stage and the first phone interview. | |||
| - **Assessment Training:** While phone interviews or other assessments are being set-up with applicants, HR will conduct a mandatory two-hour training for all SMEs conducting assessments (such as, but not only, interviews). This training explains how to conduct an interview or written assessment review, and assess an applicant’s proficiency with the required competencies based on their responses. | |||
There was a problem hiding this comment.
i dont like "such as but not only interviews" i'd rather say "structured interviews, written assessments, demonstration projects, etc"
| - **Phone Assessment Interview Training:** While phone interviews are being scheduled with applicants, HR will conduct a mandatory two-hour interview training for all SMEs conducting the first and second phone assessment interviews. This training explains how to conduct structured interviews and assess an applicant’s proficiency with the required competencies based on their responses. | ||
| - **Phone Assessment Interviews:** Over a 2-3 week period, SMEs will contribute up to 20 hours per week conducting interviews with applicants. This assumes each interview is one hour long. This includes a recommended 45 minutes after each interview for completing the interview rating sheet and adding their feedback to the talent acquisition system. The time required will vary depending on how many applicants moved forward past the resume review stage and the first phone interview. | ||
| - **Assessment Training:** While phone interviews or other assessments are being set-up with applicants, HR will conduct a mandatory two-hour training for all SMEs conducting assessments (such as, but not only, interviews). This training explains how to conduct an interview or written assessment review, and assess an applicant’s proficiency with the required competencies based on their responses. | ||
| - **Assessments:** Over a 2-3 week period, SMEs will contribute up to 15 hours per week reviewing assessments or conducting interviews with applicants. This assumes each interview is one hour long. This includes a recommended 45 minutes after each interview for completing the interview rating sheet and adding their feedback to the talent acquisition system. The time required will vary depending on how many applicants moved forward past the resume review stage and the first round of assessments. |
There was a problem hiding this comment.
how can we possibly give an hour estimate here without knowing the number of applicants or SMEs? i think it's over to say 2-3 weeks for 2 SME based assessments but refer them to the time commitment page for more info on estimate hours
There was a problem hiding this comment.
I had the same question! Maybe I should have just taken out the time piece?
I guess it was a maximum cap based on past pilots - I don't think we've had any SMEs do more than 7 interviews in a week.
I edited here and got rid of the info in that section, but it sounds like we should do more to clean up the time commitment page (this page has more prose than that one): 51d7704
| @@ -30,11 +30,11 @@ intro: The SME-QA process requires time commitments from subject matter experts, | |||
| <dd class="chp-time-commitment__phase">Resume review training (3 hours) and interview training (2 hours)</dd> | |||
| <dt class="chp-time-commitment__hours">5 hours (over 1 week)</dt> | |||
| <dd class="chp-time-commitment__phase">Resume review</dd> | |||
| <dt class="chp-time-commitment__hours">20 hours split over three weeks</dt> | |||
| <dt class="chp-time-commitment__hours">14 hours split over three weeks</dt> | |||
There was a problem hiding this comment.
why are we saying 3 weeks for resume review? am i misreading this? maybe we should just say- am for x number of weeks and the time per SME depends on how many SMEs and applicants you have.
There was a problem hiding this comment.
This text is for the assessments, not the resume review. Resume review is listed at five hours above: (150 applicants = 330 reviews, or 41.25 per SME. 7 min/review = 289 minutes = 4.81 hours.)
It always said three weeks. I just changed to "two weeks"
I modified to say that the hours are variable depending on the # passing resume review and round one: f672e37
| </h2> | ||
| <p> | ||
| SMEs conduct up to two rounds of phone interviews to determine which applicants meet the minimum qualifications. | ||
| SMEs conduct up to two rounds of assessments to determine which applicants meet the minimum qualifications. |
There was a problem hiding this comment.
should we say "verify" vs "determine"?
| sidenav: hiring-phases | ||
| phase: job-analysis | ||
| sub-phase-order: 060 | ||
| intro: Finish your workshop by breaking participants into groups to write questions for both the first and second phone assessment interviews. Use your defined competencies, proficiencies, and required experience to create two rounds of interview questions and good example responses. | ||
| intro: Finish your workshop by breaking participants into groups to write questions for both the first and second assessment rounds. Use your defined competencies, proficiencies, and required experience to create guidance for SMEs reviewing applicant responses. |
There was a problem hiding this comment.
"finish your workshop by" sounds like the very end vs the entire second half. Maybe "Dedicate the second half of your job analysis workshop to working with SMEs to develop the material for both assessment rounds. The SMEs should decide which assessment types make sense to evaluate each competency and then create assessments (interview questions and/or written prompts" to assess the identified proficiency levels."
| --- | ||
|
|
||
| <p class="usa-intro"> | ||
| {{ page.intro }} | ||
| </p> | ||
|
|
||
| As a best practice, applicants should pass through multiple hurdles with multiple SMEs before an agency considers them qualified. Two rounds of structured phone interviews, each conducted by a single SME, assess the breadth and then the depth of applicants' knowledge about your required competencies. Any applicant who doesn't receive a high enough score during the first interview will not progress to the second. | ||
| As a best practice, applicants should pass through multiple hurdles with multiple SMEs before an agency considers them qualified. Two rounds of assessments can assess the breadth and then the depth of applicants' knowledge about your required competencies. Any applicant who doesn't receive a high enough score during the first assessment will not progress to the second. |
There was a problem hiding this comment.
how about:
"As a best practice, applicants should pass through multiple hurdles with multiple SMEs before an agency considers them qualified. In order to eliminate unqualified applicants between the two rounds, they should be measuring different things. For example, one interview can measure breadth of knowledge and the second interview can measure depth of knowledge. Or a written demonstration project could measure technical competencies while a phone interview could measure competencies more easily assessed through oral communications. Therefore, any applicant who doesn't receive a high enough score during the first assessment will not progress to the second."
There was a problem hiding this comment.
awesome, I put this in prettymuch verbatim with a few tense changes and moving down a line from the intro:
As a best practice, applicants should pass through multiple hurdles with multiple SMEs before an agency considers them qualified. In order to eliminate unqualified applicants in each round, the two rounds should measure different things. For example, one interview can measure breadth of knowledge and the second interview can measure depth of knowledge. Or, a written demonstration project can measure technical competencies while a phone interview measures competencies more easily assessed through oral communications. Therefore, any applicant who doesn't receive a high enough score during the first assessment will not progress to the second.
Use your defined competencies, proficiencies, and required experience to create guidance for SMEs reviewing applicant responses.
|
|
||
| ### Assessment Options | ||
|
|
||
| - Portfolio review |
There was a problem hiding this comment.
the DE handbook has a list of assessment hurdles- maybe we cut and paste and reference that rather than establishing our own list (this sort of thing pisses of OPM)
There was a problem hiding this comment.
also portfolio review should be part of the initial application phase as a work sample not assessment hurdle.
There was a problem hiding this comment.
This actually has been on the site since January in another format from this PR: https://github.com/usds/SME-QA-Website/pull/107/files
I checked the DE handbook and it says on 2-18:
Examples of assessment tools are:
• Structured Interview
• Test
• Assessment Center
• Work Sample
• Occupational Questionnaire (Crediting Plan/Rating Schedule)
But there's a link that took me to this page with this list:
- Accomplishment Records
- Assessment Centers
- Biodata
- Cognitive Ability
- Emotional Intelligence
- Integrity/Honesty Tests
- Job Knowledge Tests
- Personality Tests
- Reference Checking
- Situational Judgment Test
- Structured Interviews
- Training and Experience
- Work Samples
I'm going to edit that down on our site to this:
- Biodata
- Cognitive Ability
- Emotional Intelligence
- Job Knowledge Tests
- Situational Judgment Test
- Structured Interviews
- Training and Experience
- Work Samples
Saved in this commit:
2e93703
|
|
||
| ### Assessment Options | ||
|
|
||
| - Portfolio review |
There was a problem hiding this comment.
also portfolio review should be part of the initial application phase as a work sample not assessment hurdle.
| @@ -5,15 +5,15 @@ | |||
| sidenav: hiring-phases | |||
| phase: job-analysis | |||
| title: Job analysis workshop | |||
| intro: Conduct a multi-day job analysis workshop with SMEs to professionally develop your assessments. You'll help SMEs and hiring manager(s) identify and define the required competencies and proficiency levels for the position, as well as write two sets of structured interview questions that can assess the required proficiency levels. SMEs will use these during interview assessments to determine whether applicants meet the technical qualifications for the job. | |||
| intro: Conduct a multi-day job analysis workshop with SMEs to professionally develop your assessments. You'll help SMEs and hiring manager(s) identify and define the required competencies and proficiency levels for the position, as well as write two assessments that can assess the required proficiency levels. SMEs will use the assessments to determine whether applicants meet the technical qualifications for the job. | |||
There was a problem hiding this comment.
i think this is redundant - maybe remove the last sentence?
There was a problem hiding this comment.
Edited to the below in this commit: a46bcf0
In a multi-day job analysis workshop with SMEs, you'll help SMEs and hiring manager(s) identify and define the required competencies and proficiency levels for the position. The group will then write two assessments that can assess the required proficiency levels. SMEs will use the assessments to determine whether applicants meet the technical qualifications for the job.
| intro: Before SMEs begin reviewing the resumes, remove applicants who are ineligible for federal employment or who did not include a resume with their application. SMEs will independently review each remaining applicant’s resume to determine whether they adequately reflect the core competencies and proficiencies to warrant the first of two phone assessment interviews. | ||
| phase: resume-review | ||
| title: Resume Review | ||
| intro: SMEs independently review each remaining applicant’s resume to determine whether they adequately reflect the core competencies and proficiencies to warrant entry to the assessment phase. | ||
| roles: [hr, sme] | ||
| duration: Approximately 4 days |
There was a problem hiding this comment.
i think we say 5 days for this not 4
|
@stephaniefaith all comments are addressed in this PR: #243 |
Should be ready now!
for issue #181