Aline Lerner is the Founder and CEO of interviewing.io, a business that gives individuals access to anonymous technical mock interviews with engineers from other top companies. With extensive experience in interviewing, Aline discusses how to hire engineers and the ONE thing you should micromanage during the process.
Hi Aline, and welcome to the CTO Craft Spotlight Q & A. We loved your presentation at the CTO Craft May 2022 Hiring MiniCon and wondered if you could tell us more about the challenge of hiring engineers?
You have one main job as an engineering leader regarding hiring: facilitating chemistry. Hiring engineers is hard, and it’s getting harder, and only you and your engineering team can facilitate genuine chemistry.
You might be surprised to learn that way more engineering time is spent per hire than a recruiter’s time. This is why you should micro-manage hiring because even though it’s recruiting, it’s fundamentally an engineering investment. We should accept that and then we should own it.
In my career and later at interviewing.io, we’ve helped tens of thousands of engineers find jobs at hundreds of companies ranging from huge ones like Dropbox and Amazon to small start-ups who are maybe hiring their third engineer. We’ve posted over 100,000 interviews, and it’s been a treasure trove of really cool interview data.
So, is it possible to make better use of engineering time but still create as much chemistry?
I think that every interaction with a candidate should be an opportunity to create chemistry between them and your team.
The hiring funnel usually has these steps:
- Recruiter Call
- Technical Phone Screen
- Onsite
- Offer
- Hire
At every stage, you should ask yourself, ‘What can I do here to create a memorable experience for the candidate?’ For instance, during the phone screen, you can make sure to ask questions that are practical and unique to what you actually do. You can also handpick the right engineers who are better at building rapport with candidates.
At the offer stage, you can make sure that the candidate has the chance to spend an hour with an eng leader.
I have more specific examples of how to create chemistry later on.
In addition, you should look to your funnel with the same critical eye that you look to the rest of your processes and cut inefficiencies while getting the most bang for your buck.
But how do you actually ‘micromanage’ hiring to maximise chemistry and minimise engineering hours in reality?
The first thing is to get a handle on your funnel metrics and determine which sources have the best outcomes. You would be shocked by how many engineering leaders are bad at this and don’t know what their funnel metrics look like.
For example, you know the adage about how you can’t fix what you can’t measure? Well, most hiring managers I speak to don’t know what their funnels look like and yet they’re engineers everywhere else but somehow forget to be engineers when it comes to recruiting? However, once you figure out what your funnel metrics look like, you might see some unexpected things.
Can you give us an example?
For instance, you might notice a very high technical phone screen pass rate and a very low onsite pass rate. It might mean that you might want to move some of the questions from your onsite to your technical phone screen etc., but you won’t know stuff like this until you pull the numbers and spend some time with them.
What would you suggest the next step is after an engineering leader has that hiring funnel visibility?
Once you do that, you’re going to get visibility into which of your sources actually have the best outcomes.
So, when you know which of your sources have the best outcomes, you should hop in and do some intro calls with candidates from those best-performing sources instead of recruiters doing it. This is your first chance to create chemistry, as first impressions are real, and this is also a great way to build relationships with your recruiting team.
Therefore, ask your recruiters to rope you in with high-potential candidates and make them feel good about asking you to do that intro call. You can scope it to a few hours a week, but even this will significantly impact your conversion rates down the funnel.
Do you have any top tips for creating relevant interview questions?
You need to create unique interview questions that sell as much as they vet. In this hiring climate, every interaction with a candidate should be about creating chemistry, and one of the best and most genuine ways to do that is to ask interesting interview questions that will stick in the candidate’s head after they leave.
So many companies don’t spend a lot of time figuring out what to ask. For example, one failure mode I’ve seen a lot is going to lead code and asking lead code questions verbatim. By doing this you’re losing such a huge opportunity to build rapport with the candidate to show them how cool what you’re doing is. Your aim is to make them think about you long after the interview is over.
That’s such a good point. Do you have any suggestions on how engineering leaders can keep track of these interesting and unique interview questions?
I usually suggest starting a shared doc in your engineering team, and every time an engineer on your team does something that they’re proud of (e.g. it can be one line of code, it can be a function, it can be a big project etc.) ask them to note that in the document.
Then, over time you’ll create a repository of these little kernels you can turn into interview questions. These interview questions will be practical, they’re going to be specific to what you do and they’re going to give candidates a genuine view into what their day-to-day might look like if they work for you.
How do you suggest delegating individual team members to each part of the hiring process?
Firstly, figure out who your ‘superscreeners’ are, put them on phone screens, and take them off onsites. Superscreener is a word I made up, but it’s an interviewer whose candidates end up getting offers.
If you actually do an analysis of your funnel and look at who all your interviewers are and how their candidates are performing, I guarantee you that you will find a pocket of interviewers who have probably similar technical phone screen to onsite conversion rates but whose candidates for whatever reason tend to do better in onsites and tend to potentially even accept offers more often.
And why is that? Because given enough interviewers, there will be some interviewers on your team who are not only well-calibrated but also really good at selling. These people aren’t just well-calibrated for the phone screen, but they have some intuition for who will do well at the onsite assessment.
The concept of ‘superscreeners’ is brilliant! But, is there a way to find these people in a team?
I’ve done a lot of work to figure out what makes these people magic, and there are a lot of factors, but regardless of what that is, they exist! So, find out who these people are and make sure that you put them on phone screens because if putting these people on phone screens doubles your onsite to offer rate all of a sudden, you’re spending way fewer engineering hours to make the same number of hires.
You can find my formula for figuring out who your superscreeners are in my recent CTO Craft MiniCon presentation.
What else is important in the interview process to maximise success?
Ensure your interviewers give constructive feedback at the end of interviews, especially for strong candidates. One of the things we ask people on interviewing.io after their interview is:
- ‘How well do you think you did?’ (before they find out how well they did)
- ‘Would you want to work with your interviewer again?’
We found a statistically significant relationship between whether people think they did well and whether they want to work with the interviewer. So, let’s say you’ve done an interview and you’re unsure about how you did – it really could go either way and after the interview, you go through this kind of self-flagellation gauntlet where you overanalyse and say things like:
- ‘I should have said this.’
- ‘That interviewer didn’t interview me well.’
- ‘That question wasn’t good.’
Over the next few days, you’ll be talking yourself out of working at this company even though you did well in the interview. By the time they get back to you and say they want to see you again because you did really well, you’ll have done so many mental gymnastics to talk yourself out of it that you’re going to be over it.
So how do you combat this?
It’s pretty straightforward. If somebody did very well in an interview, tell them as soon as possible because the longer you hold off on giving good feedback, the more likely you are to lose them.
Is there anything else you think engineering leaders should do to maximise chemistry and ensure the best candidates get through to the offer stage?
I think that engineering leaders should do a call with every candidate who’s getting an offer. Internal or external recruiters might extend the offer and handle negotiations (because it can be weird to negotiate with your future bosses’ boss!) but make sure that you as the engineering leader do a call with every candidate getting an offer.
What benefits do you think a call could have at the offer stage?
The candidates who you offer to are about 15 times as valuable as those at the top of the funnel. So, if you do nothing else but improve your close rate from 33% (which is a pretty good funnel standard) to about 50, you’re going to save 30 engineering hours per hire and one and a half times your hires if you do nothing else.
If you do all of these things (doing some napkin maths!) I think you will save about 45 engineering hours per hire, you’ll get three times your funnel throughout, and you’re just going to make hiring more human.
Because everybody hates being a candidate, don’t they? Everybody hates feeling like a statistic, but this is how you start, and I’m hopeful that hiring will just suck way less for our industry if more people adopt this strategy over time.
Can you tell us a bit more about your background and experience in recruiting engineers?
I’ve been on all sides of the hiring table. Many years ago, I was an engineer myself and then an engineering manager and that was the first time really I had to deal with hiring and discovered it was much harder than I thought and ended up making a left turn in my career and working as a recruiter. I really wanted to understand why it was so hard.
I ended up running recruiting at a few companies including Udacity then I started my own recruiting firm and finally that windy path led me to start a company called interviewing.io.
What exactly does interviewing.io do?
Interviewing.io is an anonymous technical interviewing practise platform so engineers come to us to practise technical interviewing and top performers can then connect with companies. We’ve hosted over 100,000 technical interviews in our lifetimes split between practice interviews and real ones.
***
The next CTO Craft Conference will be held later this year in November. Stay tuned to find out more about it soon.
If you or your CTO / technology lead would benefit from any of the services offered by the CTO Craft community, use the Contact Us button at the top or email us here and we’ll be in touch!
Subscribe to Tech Manager Weekly for a free weekly dose of tech culture, hiring, development, process and more.