What makes a great Engineering Manager?

“What gets measured gets managed,” is one of those office-isms we may have heard quoted without context. Originally written by Management Consultant Peter Drucker, Consultant and Writer William Hennessy calls it “the most evil and destructive Drucker quote of all time” because it’s so often misapplied.

And one place it’s misapplied is Engineering Management. The uncomfortable truth is that the most important parts of a manager’s job can’t be easily quantified. Measuring engineering performance is already hard to measure. Most engineering metrics leave a lot to be desired. Engineering productivity and what makes up "good engineering" isn't always easily quantifiable. Engineering Manager performance is that much harder.

The metrics for Engineering Managers tend to be subjective, according to Nathan Feger, Director of Engineering at Schoology. “We don’t necessarily have great metrics for us,” Feger said. “The higher up you go, the more subjective it is.”

“There are a million articles about how to tell whether Software Engineers are doing a good job,” Charity Majors, CTO at Honeycomb.io said on a recent webinar about making the transition from an Engineering Manager to a Manager of Engineering Managers. “But what about their managers?” It’s easy to rattle off good-sounding answers, “But the reality is so hard.”

And yet, measurement remains a necessary evil. If you manage Engineering Managers, you need a basis on which to promote and compensate them. There has to be a way to identify and reward great managers who exceed expectations in software engineering. Here are some tips for doing just that.

Subjective outputs: managing down and up

Effective Engineering Managers need to be effective at managing not just their engineering teams, but their managers as well. Here’s how to tell whether they’re hitting the mark.

Managing down

A pretty easy-to-measure sign of a successful Engineering Manager: “I’ll know you’re doing well if other people are queuing up to join your team,” Rich Archbold, Sr. Director of Engineering at Intercom, said. Is the team so functional, the operations so under control, and the work so exciting that everyone wants to work for them? Looking at manager reviews and team satisfaction can help reveal this.

“Once that happens, you know you’ve hit success,” Archbold said. “Engineers are great at sniffing out bullsh*t.” They’ll ask other Engineers what it’s like to work for a Manager. “Does he or she have their sh*t together? How many times are you getting paged out of bed at night? What’s your roadmap like? Is it credible? Does this person know what they’re talking about? They are the best judge of this stuff.”

But Archbold also notes that these metrics are vague and open to interpretation. Unless you’re using well-crafted surveys on a regular cadence, “How do you know if they’re going up or down?” And if they are moving, “Is it because I’m doing a good job or because the market is doing something or because the weather’s been really bad?” That’s why you need to combine these measures with others, which we’ll get into later in the post.

Managing up

Another thing to consider is whether an Engineering Manager is good at managing up. “I’m usually very good at managing down,” Majors said. “It’s taken me a long time (I’m still not very good at it) to learn how to manage up. It feels like sucking up. Or it feels like currying favor. But, if you don’t have a good relationship with the people above you, you’re not going to be able to effectively advocate for your team when it counts.”

A good manager will learn and grow over time, which includes managing up. Archbold also adds stakeholder management to the mix of metrics for Engineering Managers. “For me, that managing up thing is highly correlated with success and growth,” Archbold said. “And managing out. I need to know that if we give you more and more scope and more and more teams that I can actually trust you to send up the flag, manage up, and notify me about only the stuff I need to know.”

One more sign of someone who's going to manage Software Engineers well: They take care of things as much as they can, but they also know when they need to ask for support. “I can give you autonomy, but there’s a huge difference between autonomy and expecting to do absolutely everything by yourself,” Archbold said. “Some of my best managers and some of my best managers of managers send me either a weekly status report or biweekly status report.”

These reports will have their own unique formats and headings, but the point is that great managers are able to analyze, assess, and communicate just the right amount of information to their bosses.

Below, let’s dive into questions you can ask Engineering Managers, their reports, and their peers to get a fuller picture of their performance.

Questions for Managers

Archbold likes to ask Engineering Managers questions that reveal how well they understand their team, goals, and mission.

These questions include:

• What’s actually the most important project your team is working on?
• Why?
• How does it work?
• What are the actual design decisions you’ve made?
• What are the key technology choices and why?

Project management is part of the day-to-day of the role, certainly. This is along with code reviews, evaluating team members, enhancing engineering productivity, doing performance reviews, handling pull requests, helping engineers up their technical skills, and creating an environment where the whole team can flourish.

“I don’t expect a manager to know about every single piece of technology or every single project or every single piece of software that’s going on in their team,” Archbold said. “But I do expect them to know something about the most important thing their team is working on at the moment.”

Questions for reports

To Majors, a good Engineering Manager has teams that feel happy and productive. She recommends skip level meetings with their reports to ascertain this information. “Skip levels are helpful with this,” Majors said. “Although, you really have to listen for patterns almost more than individual answers. Because a team that’s good can make any manager look good for a long time.”

Schoology also uses skip levels to get a sense of whether the Engineering Manager in question:

• Makes sure their team understands what’s going on in the company
• Is setting a roadmap and sticking to it
• Can identify underperforming teammates and get them on PIPs
• Is aware of any skill gaps on their team that their director can help them get budget for

Archbold also measures Engineering Managers through softer metrics like organizational health, rate of engagement inside teams, and how Engineers feel about the mission. In the team's answers, Archbold is looking for whether the manager is really engaging with the team. Are they playing an active role in helping the team move forward or are they missing in action?

Questions for peers

Then of course it can be helpful to talk to a Manager’s peers to try to understand how influential and well-networked the manager is. HBR found that the Managers who regularly worked with the largest number of people across the organization had employees with up to 5% higher engagement scores.

For Ian Nowland, VP of Engineering at Datadog, it’s not enough for the Manager to be well-known. They also need to be well-liked. When a Manager’s team loves them but all their peers hate them, it’s a sign that “These guys are always looking out for their team at the expense of other teams,” Nowland said. “I spend a lot of time in one-on-ones just asking, ‘How do you think X is doing?’ It’s telling you a lot about how people are perceived. It’s far more situational judgment than anything that is objective. But I don’t know any other way to do it.”

It’s also telling to discover how influential a Manager is among his peers. “Often, very, very early on as a Manager of Engineers, you can get away without really influencing,” Nowland said. “People will just look up to you and really listen to you.” But the standout Managers are “actually influencing their peers, they’re often visible upwards, and those are great characteristics. Because that’s what, really, a lot of the job is gonna become.”

Archbold agrees. “People need to be excellent at the job at hand and be recognized as a role model by their peers,” Archbold said. “Their peers have to look up to them and be like, ‘That person does it really well. I feel like I could learn something from them.’”

Going forward

Evaluating an Engineering Manager is far more difficult than evaluating an Engineer. It’s a lot more nuanced, subjective, and time-consuming. But it’s also incredibly high-leverage when done well. For help managing your schedule as an Engineering Manager, check out our AI calendar assistant, Clockwise.