When I first started at Cockroach Labs, the founders and I had a candid conversation about diversity. Studies have shown that diverse companies have a greater likelihood of success through higher employee performance and financial returns [2015 McKinsey report]. We agreed that it was important for us to attract a diverse workforce and concluded that the best way to do so was through creating an inclusive environment. In addition to internal initiatives, we set out on a mission to remove bias from our interview process or more realistically, challenge the bias that our interviewers face when assessing candidates.
Initially, we removed resumes from our interview process. Through removing resumes from the process, we challenge the interviewer not to assess a candidate based on associations that they have with the candidate through a piece of paper. Instead, we ask the interviewer to focus on the direct application of the candidate’s skills during the interview. We believe that by challenging our unconscious bias, we are creating a fairer process, leading to better hiring outcomes.
When starting to hire for non-technical roles, our CEO, Spencer Kimball grew frustrated. As we were looking at the behavioral and situational questions for the position, he asked, “Why can’t we do something like our coding interviews for our non-technical candidates?” Our coding interviews at Cockroach Labs focus on questions that our engineers apply in their positions. Also, they provide the interviewee with an environment that is reflective of the real-world application of their skills.
My answer to Spencer was, “we can.” And we have. Instead of focusing on behavioral questions based on past performance or situational questions based on hypothetical future situations, we focus on exercise based interviews with an emphasis on applied skills required for the role. Exercise based interviews can include case studies, group exercises involving role play or discussions, individual exercises, or presentations. They allow for our interviewers to witness the direct application of a candidate’s abilities to competencies required for the role. For example, our Technical Writers are asked to complete an information architecture exercise, which focuses on the candidate’s ability to break down and structure information with an audience in mind. Since each interview focuses on different areas, collectively, the interviews are mini snapshots of the position. Furthermore, we believe they lead to a stronger signal since they are likely to be strongly correlated with actual job performance.
The added benefit? The candidate receives a clear understanding of what it would be like working in the position and with our team on a day-to-day basis. Furthermore, the interviews also require a high degree of engagement, which results in the interviews being more collaborative and overall more pleasant than traditional interviewing where they are responding to questions.
Most importantly for us in this effort is that we strongly believe that exercise based interviewing reduces bias and subjectivity because the interviews are focused on providing candidates with real-world simulations where they can display their ability.
The Nitty-Gritty: How to construct exercise based interviews?
Once a candidate applies to roles at Cockroach Labs, we use the following process:
Recruiter Phone Screen
Take-home Test or Coding Interview (via CoderPad)
In-Person Interview
The Recruiter Phone Screen is the only stage that the candidate’s resume will be reviewed. Once the assessment portion of the interview process has concluded, the resume is hidden from the interview team.
For non-engineering positions, candidates receive a take-home exercise, instead of a traditional phone interview with the hiring manager. These exercises typically take 1-2 hours to complete and allow for the hiring manager to review a candidate’s output of skills directly related to the position before proceeding to the onsite interview. The take-home exercise takes the place of one phone interview and one onsite interview, equaling two hours of engagement. For example, candidates for our recruiter position are asked to provide an overview of how they would kick-off a search for a position and provide three qualified candidates for the role.
For engineering positions, candidates connect with a member of our engineering team and spend one hour on a coding exercise, leveraging coderpad.io. The coding exercises are directly related to the same concepts our engineers apply on a daily basis, covering areas like coding/debugging, algorithms, and data structures.
During the in-person interview, candidates meet members from the team that they will work with. The slate includes members of the direct team or cross teams. As mentioned before, these interviews can include case studies, group exercises involving role play or discussions, individual exercises, or presentations. The interviews are structured as follows:
Interviewer Presents the Exercise - The goal of the exercise is explained.
Candidate Collects Data - The interviewer has a back and forth discussion with the candidate that allows the candidate to complete the exercise.
Execution - The candidate works on the exercise.
Discussion - The interviewer and candidate have a conversation about how the candidate arrived at the solution. The discussion involves a prompt of the initial questions, follow-ups that allow furthering the conversation, and probes, should clarifying questions be needed.
Example Interview
Onsite Exercise for Administrative Roles - Client Meetings Set-up: 15 minute warm-up/present problem/data collecting, 15 minute work time, 15 minute discussion.
Goal: Schedule two one-hour client meetings for Co-Founder and VP of Engineering Peter Mattis. Peter has not met with these individuals before so please be sure to prepare a short bio on both clients by researching publicly available information. Please also be sure to include any information that may be useful for Peter to know (i.e. areas of overlap, current focus in roles, past experience, etc.).
Client Meetings:
Oscar Health - Alan Warren
Goldman Sachs - R. Martin Chavez
Considerations:
Peter typically arrives at the office by 9am. He is traveling from his home in Connecticut via train and takes 4/5/6 from Grand Central.
Peter prefers commuting via car, not public transportation for meetings.
Peter likes to have 10 minutes of buffer time before the meetings so he does not feel like he is rushed.
Peter’s LinkedIn: https://www.linkedin.com/in/peter-mattis-46549144
For the comprehensive exercise, click here.
How does this influence the candidate experience?
At the end of every onsite interview, we solicit feedback from our candidates to uncover what went well and what didn’t go well. From this survey, we can see if the candidate feels that exercise based interviewing provides the same benefits that we feel they do.
Of 19 people who responded to the survey, 79% preferred the interview format being focused on exercise based interviewing. The remaining 21% felt neutral towards the interview format while there were no unfavorable responses in the data set. As Amruta Ranade discussed in her blog, she “was pleasantly surprised by the “show, don’t tell” aspect of the interviews,” which she felt “was truly an unconventional, yet insightful interview process.”
As always, our interview process is living and breathing. Like everything we do, we have to iterate to improve. When we are kicking off a new requisition, we sit down after each interview to internally discuss what needs to be improved. Often, these improvements have been a direct result of feedback we get. For example, we have learned that for some of the exercises context is important. If there is information that we need to prepare the candidate with, we will send it to them before the interview.
Another piece of feedback we received from a candidate is that they would have preferred to come prepared with information to apply to the exercises as they felt the time constraint of the interview didn’t allow for a strong representation of their work. If they could come prepared, they could have gathered more information needed for the exercise (e.g., read a blog post, etc.).
Conclusion
The positive candidate feedback that we have received regarding our process, along with our desire to challenge how companies approach recruiting has encouraged us to open source our interview process. In the upcoming month, we will be sharing all of our interview questions on Github.
In doing so, we hope to encourage:
Companies to think outside the box regarding their interview process. What else can we be doing to provide candidates with a process that is fair and inclusive?
Candidates to come prepared! If candidates take the time to research the questions and comes onsite ready to have an engaged discussion, we will be thrilled.
Collaboration. Let’s all do better than we have before. We can only do better through engaging with ideas from one another. We welcome feedback on our process and encourage candidates and recruiting professionals to contribute exercises as well.
Look forward to future announcements of us open sourcing our interview process.
Illustration by Lisk Feng
