zakinno vallana reginta

[Matthias Briggs]

I've been at Netflix now for several months, and have found it to be an amazing place to work. What has surprised me most is the culture, how different it is to other companies, and how well it works.

In this post I'll describe my experience at Netflix: starting with recruiting, the culture, my work as a performance architect, and finally our mission. I'm excited by what we are doing as a company, and I hope this can (or can continue to) inspire positive changes across our industry.

My first direct experience with Netflix was the hiring process. Netflix recruiting is outstanding. I was scheduled quickly and with the right people: my would-be manager, co-workers, and higher management. The focus was on finding out if we were a good fit for each other.

The question of compensation was also handled differently at Netflix. In many hiring negotiations, both parties try to bluff their way around this topic. With Netflix, I was encouraged to find out my market worth and discuss it with them, so that we could both agree on what constituted a good salary. They also did their own research by collecting data from candidates and other companies throughout the hiring process, to ensure the offer was top of market. I think some other companies are terrified of staff discovering what they are really worth! But this openness and honesty was characteristic of the Netflix recruiting process.

Many companies talk about about how great their culture is, but this is more aspirational than reality. After joining the company, the real culture is learned by word of mouth, or trial and error. However, with Netflix, the culture deck is true. I think its emphasis in the recruitment stage reinforces this, as everyone learns how to act before they walk in the door.

But how does this work? It's not freedom gone wild. You could introduce a new technology, for example, provided you plan who will service and maintain it, how it's debugged when it fails, how fault tolerance works, etc. Freedom and responsibility.

This works because Netflix hires professionals: people who have good judgement to start with and who use freedom wisely. People with self-discipline. This doesn't mean we're always right: freedom and responsibility includes having the courage and curiosity to try risky projects, when it makes sense to do so, as these can lead to important innovations. The focus is always on making a positive impact for the company.

Apart from professionalism, Netflix also aims to hire high performers. People who are self-driven, highly productive, and who also work well with others. This means no "brilliant jerks", who are explicitly not welcome (this has been described as a polite version of the "no asshole rule" [1]).

There are a few companies doing important work in performance, and Netflix is one of them. With over 50 million subscribers, the largest cloud environment, and handling over a third of the US Internet traffic at night, there are numerous opportunities for performance engineering. This involves not just applying existing practices, but developing the state of the art.

I'm working on many technologies: AWS, Linux, FreeBSD, Java, Node.js, Perl, Python, Cassandra, Nginx, ftrace, perf_events, and eBPF to name a few. There are many technologies we've developed ourselves, which are typically open sourced, including the rxNetty reactive framework, Atlas for performance monitoring, and (coming up) Vector for instance analysis. There's also hardware performance work (for the FreeBSD appliances) and capacity planning.

My day-to-day work includes resolving immediate issues of poor performance, and short- and long-term projects. We're using Linux on our cloud, where advanced performance analysis tools have historically been lacking. I have the freedom to figure out what best to do, which has included developing ftrace and perf_events performance analysis tools, for short-term wins (perf-tools); Java hotspot hacking, for short- and long-term wins; and eBPF testing, for long-term wins. I'll be putting some time into the other tracers (SystemTap, ktap, LTTng), to find wins from them, too.

The FreeBSD Open Connect Appliances, our CDN which streams the actual content, are amazing to work on. FreeBSD provides the most advanced performance analysis environment, including many standard tools as well as pmcstat and DTrace (I summarized these in my MeetBSDCA 2014 talk: slides, video). I was delighted to recently develop CPI flame graphs on the OCAs, using pmcstat.

I'm part of a great team, Performance and Reliability Engineering. I'm not just applying my skills, but learning more from other colleagues, and developing those skills in our environment. We're also expanding our team, and hiring performance and systems engineering roles (if interested, contact Coburn, our manager).

We aren't building a technical facade, where the real mission is to sell the company. We aren't winning by unsavory sales, legal, or marketing tactics. And our customers are not unwittingly themselves the product (we don't read their private emails). We want to win by making a product so good that people choose to buy it. We're an honest company.

In this post I described my opinions of Netflix after ten months at the company. I'm motivated to write because I think the topic of positive company cultures is worth discussing. For more about the Netflix culture, apart from the culture deck, you can read how Netflix reinvented HR, behind the slides, and the woman behind the Netflix culture doc. Maybe you can experience this culture directly (we are hiring), or perhaps more of our industry can adopt or engineer their own positive cultures.

Update: Many people have been emailing me their resumes. I'm glad they are interested in working at Netflix, but I'm an engineer, not a hiring manager. Please use jobs.netflix.com, which will send your details to the right people.

I've now worked at Netflix for over three years. Time flies! I previously wrote about Netflix in 2015 and 2016, and if you are interested in what it's like to work here, I already covered much in those posts. As before, no one at Netflix has asked me to write this, and this is my personal blog and not a company post.

When I joined Netflix in April 2014, we had over 40 million subscribers in 41 countries. We are now in 190 countries and just crossed 100 million subscribers! It's been thrilling to be part of this and help Netflix scale.

You might imagine that at some point we had a major scaling crises, where it looked like we'd fail due to an architectural bottleneck, and engineers worked long nights and weekends to save Netflix from certain disaster. That'd make a great story, but it didn't happen. We're on the EC2 cloud, which has great scalability, and our own cloud architecture of microservices is also designed for scalability. During this time we did do plenty of hard work, rolling out new technologies and major microservice versions, and fixed many problems big and small. But there was no single crisis point. Instead, it has been a process of continual improvements, by many engineers across the company.

Most of my day is a 50/50 mixture of proactive projects, and reactive performance analysis. The proactive projects usually take weeks or months, and are where I'm developing a new technology or helping other teams with performance analysis or evaluations. Most of these projects aren't public yet, and some of them involve working with other companies on unreleased products. My work with Linux is different in that it is mostly public, and includes my perf-tools and bcc/eBPF tracing tools.

It's a good balance. Too much reactive work and you don't have time to build better tools and general fire proofing. Too much proactive work and you can become disconnected from the current company pain points, and start building solutions to the problems of yesteryear.

About one hour on average each day is meetings. Some of these are regular meetings: we have a team meeting once every two weeks where everyone discusses what they are working on, and I have a one-on-one with my manager once every two weeks. At a lower frequency, I have scheduled meetings with my manager's manager, and their manager. All these manager meetings keep me informed of the current company needs, and help connect me to the right people and projects at Netflix.

Then there's some random events that happen during the year. We have offsites, where we plan what to work on each quarter, and team building events. There's also unofficial recreational groups at Netflix, including movie clubs (for good movies, and for bad ones), a karaoke group (which includes some Hamilton fans), and various sports teams. I'm on the Netflix cricket team (if you're at Netflix and didn't know we exist, join the cricket chatroom). I also usually speak at some conferences each year.

The biggest difference I've found working here is still the culture. We are empowered to do the right thing, and believe in "freedom and responsibility".This is documented in the Netflix culture deck, and after three years I still find it true.

Before joining Netflix, you're told to read it and see if this company is right for you. Then while working here, staff cite the culture deck in meetings for decision making advice. It's not nice-sounding values that are printed in the lobby and people forget about. It's an ongoing influence in the day to day running of Netflix. Having it online also beats learning the culture through word of mouth or trial and error.

I know people in tech who are burned out but stay in lousy jobs, assuming every workplace is just as terrible. Jobs where there is little to no freedom, no responsibility or accountability, and where dumb office politics is the norm. I wish everyone could have a chance to work at a company like Netflix. Little to no bureaucracy. You can focus on engineering and getting stuff done, with awesome staff who will help you.

I've noticed a widespread cynicism about successful companies, especially US corporates, where it's assumed that they must be doing something shady to be really competitive. Like selling customer data, or making it difficult to terminate membership. It's been amazing and inspiring to see how Netflix operates, contrary to this belief. We don't do anything shady, and we're proud of that. We're an honest company.

90f70e40cf