43 stories

Dodging garbage fires

1 Share


Trigger warning: harassment, non consensual sex, porn, and all other sorts of gross things I have dealt with.

You might have heard about “the Google guy“. I won’t rehash it again. Instead, I want to focus on the questions that people ask me: “have you seen that?” and “how come I’m not outraged?”

Short answer: because I’ve been seeing this, in one way or another, since I was in kindergarten and I’m very, extremely, incredibly tired.


  • I’m in kindergarten. I want to climb trees, play football and explore the playground. A mean bully boy surrounded by its group of coward groupies declares I can’t, because I’m a girl, and also because I am wearing a dress.
  • I’m in primary school and I want to play with construction games. I am told that is what boys would do. Do I not want to play with Barbie?
  • Later I want to attend computer classes after school. I am told that is something boys would do.
  • In secondary school, I want to enter the drawing contest in my school. I am told nope, girls can’t enter the drawing contest, they have to do aerobic or be cheerleaders during the (male) football match.
  • I’ve been accepted at uni to study Computer Science, I proudly tell one of my class mates. He looks at me with a troubled face and asks/declares “that isn’t something women would study, is it?”
  • I’m at one of the practice labs in the uni, and a male teacher makes a sexist comment about women being really good at cleaning up, when I say I’m tidying up our assignment before submitting it.
  • Later, the mentor where I’m interning describes in a very nonchalant and clinical way his proposition: that I allow him to perform some sexual acts on me, so he could decide if he really likes his girlfriend enough to go live with her forever. In parallel, one of my intern colleagues declares he’s in love with me and gets upset that I refuse him and his gifts. I quit, because jesus freaking christ, I just want to build videogames and my curriculum, not have sex with my coworkers.
  • Elsewhere, we built a website, then I’m told that it’s best if we don’t say that a woman built it, because then people would try really hard to hack it.
  • I’m at a demoscene party and I’m surrounded by men watching porn. I’m confused because a demoscene party is meant to be a computer event and it’s supposed to be about technical excellence, so… why are they watching this here? What is wrong with people?
  • At another demoscene party a guy asks me if I’m “a real demoscener”. I am…
    • releasing a demo which I coded myself with my own scripting system based in Lua and C++,
    • plus a song,
    • and I’m being interviewed and joining a panel or two (I don’t even remember),
    • and also teaching a seminar
    • … and I still need to prove myself somehow. Do I need to dress like Trinity from the Matrix?
  • On my first startup I’m the only woman developer out of 8 developers, and I accidentally get cc:ed on an email in which the CTO refers to me as “the girl in the office”, and wonders if I know any programming language.
  • On the first day of another job, I’m introduced to one coworker, who asks what I’m going to do there. His reply: “Oh. I didn’t think women could program”. Perhaps not ironically, he was terrible at programming. Also at being human.
  • In conferences, meetups, and similar gatherings, I’m warned when technical discussions are happening, just in case I want to refill my drink or retouch my nail polish, I guess.
  • In another startup, the CTO tells me “in confidence” that the VC of the startup where I work is “slightly against women in tech” and so was “slightly reluctant” to me being hired, but he “is fine now”. I “slightly wonder” if I need to give him a medal or send flowers in appreciation.
  • Someone apologised, in retrospect and “in confidence”, for assuming I was a man when they read my CV.
  • A very well known speaker told me people were very interested in seeing me speak… because “I am a very good woman speaker”. Not because I have interesting things to say, it seems.
  • At the after party at a conference where I spoke, a stranger holds me by the waist, then gets very upset that I immediately free myself from his embrace and ask him what the heck is he doing. Later, another guy tries really hard to hook up with me, despite my absolute lack of interest and continuous “no” and refusals. His colleagues assume that he’s going to be successful anyway, and ask him whether they should just leave already, since this is pretty much “a done deal”. I am the one who leaves in a whim because I’m done with all this shit; I just wanted to enjoy a drink and talk to interesting people, not be harassed by randos.
  • And the list goes on, and
  • on, and
  • on, and
  • on forever

And so it looks like I have ‘toughened up’, but not really, because what happens is that you develop this kind of automatic system that detects garbage from a distance, and when I see a barrel of garbage in flames like all these guys rolling my way, I just dodge it as fast as I possibly can, because if you get too close you’re going to get burnt out. There’s no extinguisher for this kind of fires; the only solution is to let them consume themselves to ashes.

But it is tiring. I am expending all this energy that men do not have to expend, and that’s simply in order to exist and do my job without being run over by somebody else’s prejudices or taken advantage of.

I would prefer if I could focus on being my authentic, efficient and greatly productive me.

Because the truth is that I’m actually freaking good at all those things I’m told I can’t or mustn’t do. I’m great at climbing, and at drawing, and playing with construction sets, and telling weirdos to get lost (I’m very good at this), and developing software and systems (I’m even better at this), and fixing and building stuff.

And it’s not only that I’m good, but that I’m actually way better than those self-appointed gurus of biology and gender studies who diagonally read a Wikipedia article once and then decided to go around “weighing in” unasked and telling people what they can or cannot do.

And so are all my friends who have had to put up with all this bullshit already, and are freaking good at their jobs… if only you would let them do them!

Many, who are not as stubborn as I am, have left towards fields in which they are treated like human beings, not sexual accessories, and aren’t questioned every single day. Even I consider leaving from time to time.

And that’s why we’re not “reacting” to this. Because we’re tired of “reacting”.

That said, if the rest of people wanted to “act”, instead of expecting the minorities to “react” as if it were a live entertainment show, it would be appreciated. Tell them they’re being sexist. Tell them they are dressing up facts made out of thin air as science, to support their own discriminatory narrative. Or just tell them to STFU and grow up. It would be nice to not to have to do it ourselves, just the once.

Mulder, this is the field in which I grow my fucks. You will note that it is barren.
Mulder, this is the field in which I grow my f****. You will note that it is barren.

flattr this!

Read the whole story
11 days ago
Share this story

On “soft” skills and senior engineering

1 Share

Like many people, I was disappointed but not surprised to see that anti-diversity “manifesto” that’s been making the rounds on the internet over the past week. There has been quite a lot written about the whole thing already, and I’m not trying to duplicate any of that – click on these links here for some of the excellent conversations that have been started so far.  But these conversations do tie in with some that my team happened to be having last week around career growth, what it means to be a senior engineer, and how hiring and promotion processes can impact team and company culture, so I guess now seems like as good a time as any for me to finally write down some of my thoughts on the subject.

I’ve linked to this post by John Allspaw before and I’ll probably link to it again – being a senior engineer is really about being a mature engineer. Being a mature engineer means so much more than just technical skills, but it seems to me that in most cases in the tech industry, we see having some certain level of tech/coding skill as being the only real prerequisite to being called a senior engineer, with any so-called “soft” skills being deemed optional or an added bonus.

-senior- engineers

This focus on only tech skills leads to organizations where the so-called “brilliant assholes” are tolerated. You know, the sort where “nobody can stand working with him but ZOMG HIS CODEZZZ IS SOOOO GOOD”. In my experiences at least, white men are much more frequently able to get away with this sort of skill imbalance, while women and people of color much more often can’t. I’ve seen this play out on too many interview panels in too many organizations, with female candidates being passed up because they “weren’t easy to talk to” while similarly-qualified men advanced through the interview processes with the same behavior.

We need to get out of this situation where white men are held to a different standard of behavior in the workplace than other groups of people. We need to realize that so-called “soft” skills like collaboration, communication, and empathy are not optional, but are in fact key to engineering. As Yonatan Zunger said,

Essentially, engineering is all about cooperation, collaboration, and empathy for both your colleagues and your customers. If someone told you that engineering was a field where you could get away with not dealing with people or feelings, then I’m very sorry to tell you that you have been lied to. Solitary work is something that only happens at the most junior levels, and even then it’s only possible because someone senior to you — most likely your manager — has been putting in long hours to build up the social structures in your group that let you focus on code.

I think we need to start referring to these types of skills as “core” skills rather than “soft” skills, because they are the core of successful, mature engineering. We need to stop viewing them as nice to have and start seeing them as mandatory, and we need to stop hiring and promoting people who don’t have them.

-senior- engineers 2

People who don’t have some minimum level of collaboration, communication, and empathy skills are not senior engineers. It doesn’t matter how well they can code, if they aren’t also mature and empathetic people capable of working well as part of a team and a larger organization. Being able to do something clever with a B-tree on a whiteboard during an interview process does not mean that a person is a mature, collaborative, and empathetic engineer. Engineering is so much more than just code. Tolerating antisocial, biased, or abusive behavior from people just because they can write good code does this industry and everyone in it a disservice.

We as an industry need to start making these core skills explicitly part of our interview and promotion processes. Skills matrices for promotions (you do have those, right? Because you shouldn’t be promoting people based on just “gut feel” and who happens to be best bros with their manager) should include things like the ability to work on diverse teams, the ability to empathize with others, having effective communications skills, and so forth. If someone has a lot of technical skill but isn’t demonstrating empathy or collaborative skills, they should have to grow those skills in order to advance to more senior positions.

Hiring processes should include questions and methodologies that try to assess these skills as well. I love asking candidates to tell me about a disagreement they had with a coworker, and to tell me about a time when they were wrong about something – if someone can’t come up with a single example of when they were wrong, that’s a huge red flag. Every single person has been wrong about something at some point in their lives, and failing to recognize or admit that is not a sign of a mature engineer.

I like using diverse, multi-person interview panels for this reason. More than once I’ve been co-interviewing a candidate and watched them completely ignore me and speak only to the man who was conducting the interview with me. If I had been doing those interviews alone, that behavior might not have been so obvious, but doing multi-person interviews can do a lot towards finding out how people behave in group settings. Asking candidates about their best and worst experiences working in a group can help suss out how they relate to and interact with other people. Asking them about an experience they found unfair in the workplace might help ascertain if they have any underlying feelings of entitlement.

And as an industry we, especially those of us with more privilege and more systemic power, need to be working on these things proactively rather than reactively. Don’t wait until someone in your organization publishes their own manifesto, because if your culture has gotten to the point where they feel safe publicly saying that they think women are just biologically worse at engineering, it is probably toxic beyond repair. If you have influence over hiring and firing, over promotions, over interview and review processes, it is your responsibility to make sure that those processes are fair. You can help make collaboration and empathy something you require of the senior engineers within your organization, not just something that’s sometimes an added bonus.

If you’re a white man in tech and you don’t agree with the writer of the manifesto, it is your responsibility to stand up and say something. And don’t just say something once. As long as women, people of color, LGBTQ people, keep having to defend our right to even be in this industry, you have to keep speaking up as well. Say something to that problematic dude on your team. Speak up when you see microaggressions. Don’t advocate for the hiring and promotion of people who lack basic empathy. I’ve had plenty of men tell me in private over the years, “oh, that sucks, I’m sorry you have to deal with this”, but very very few willing to stand up and say the same thing in public. The standard you walk by is the standard you accept, and if you sit silently by in public when things like that happen you are saying (whether you intend to or not) that you are ok with that sort of behavior.

Don’t be ok with this sort of thing. Don’t stand for “senior” engineers who don’t have basic collaboration, communication, and empathy skills. Realize how core these sorts of skills are to mature engineering, and start fixing your hiring and promotion processes to require these skills of all your employees, not just those from underrepresented groups. Help to make this industry and the world a better place by not tolerating any more so-called “brilliant assholes” who end up doing harm to the people, teams, and organizations around them.

Read the whole story
12 days ago
Share this story

Bun Alert

5 Comments and 14 Shares
Since buns range from crepuscular to nocturnal, it's recommended that you enable the scheduled "Do Not Disturb" mode on your phone to avoid being woken by alerts about Night Buns.
Read the whole story
19 days ago
Share this story
4 public comments
12 days ago
18 days ago
I want this to be a real thing.
Greater Bostonia
19 days ago
Since buns range from crepuscular to nocturnal, it's recommended that you enable the scheduled "Do Not Disturb" mode on your phone to avoid being woken by alerts about Night Buns.
19 days ago
/attn @haloedrain
Mountain View, CA

CodeSOD: The Wrong Sacrifice

1 Share


Folks, you need to choose a different sacrificial animal for your multithreading issues. Thanks to this comment Edward found in a stubborn bit of Java code, we now know the programming gods won't take our goats.

public static boolean doSomeOSStuff() {

        // Forgive me father for I have sinned - with the bag of crap written in the lines below.
        // When running multiple threads (~500) concurrently, the damn OS commands fail at a 1% rate for no apparent reason.
        // There is no error message, or any indication of failure, they just don't happen and come bite you in the ass.
        // After much frustration, I decided to try 3 things:
        // 1. sacrifice a goat. - did not go as planned.
        // 2. semaphoring this shit. did not change anything...
        // 3. after each failed command, I will try and verify if it succeeded  or not. If not I'll try it 4 more times.
        // I am now going home to take a hot shower, and cry laying on the shower floor. and /or sacrifice a few more goats.
        //                                          ...
        //                       s,                .                    .s
        //                        ss,              . ..               .ss
        //                        'SsSs,           ..  .           .sSsS'
        //                         sSs'sSs,        .   .        .sSs'sSs
        //                          sSs  'sSs,      ...      .sSs'  sSs
        //                           sS,    'sSs,         .sSs'    .Ss
        //                           'Ss       'sSs,   .sSs'       sS'
        //                  ...       sSs         ' .sSs'         sSs       ...
        //                 .           sSs       .sSs' ..,       sSs       .
        //                 . ..         sS,   .sSs'  .  'sSs,   .Ss        . ..
        //                 ..  .        'Ss .Ss'     .     'sSs. ''        ..  .
        //                 .   .         sSs '       .        'sSs,        .   .
        //                  ...      .sS.'sSs        .        .. 'sSs,      ...
        //                        .sSs'    sS,     .....     .Ss    'sSs,
        //                     .sSs'       'Ss       .       sS'       'sSs,
        //                  .sSs'           sSs      .      sSs           'sSs,
        //               .sSs'____________________________ sSs ______________'sSs,
        //                                    ...         sS'
        //                                     sSs       sSs
        //                                      sSs     sSs
        //                                       sS,   .Ss
        //                                       'Ss   sS'
        //                                        sSs sSs
        //                                         sSsSs
        //                                          sSs
        //                                           s
        // snipped OS manipulation code with several cases of aforementioned ritual
[Advertisement] High availability, Load-balanced or Basic – design your own Universal Package Manager, allow the enterprise to scale as you grow. Download and see for yourself!
Read the whole story
126 days ago
Share this story

So you want to be a wizard

1 Comment and 5 Shares

Today I did the opening keynote at SRECon. This talk was a little less technical than my normal talks: instead of talking about tools like tcpdump (though tcpdump makes an appearance!), I wanted to talk about how to make a career where you’re constantly learning and how to be good at your job whether or not you’re the most experienced person.

Here’s the talk abstract, then the slides & a rough transcript. I’ve included links to every resource I mentioned.


I don't always feel like a wizard. I'm not the most experienced member on my team, like most people I sometimes find my work difficult, and I still have a TON TO LEARN.

But along the way, I have learned a few ways to debug tricky problems, get the information I need from my colleagues, and get my job done. We're going to talk about

  • how asking dumb questions is actually a superpower
  • how you can read the source code to programs when all other avenues fail
  • debugging tools that make you FEEL like a wizard
  • and how understanding what your _organization_ needs can make you amazing
At the end, we'll have a better understanding of how you can get a lot of awesome stuff done even when you're not the highest level wizard on your team.

So you want to be a wizard

(this transcript is nowhere near totally faithful; there’s a fair amount of “what i meant to say / what I said, kind of, I think” in here :) )

You can click on any of the slides to see a bigger version.

This talk is called "so you want to be a wizard". The main problem with being a wizard is that, of course, computers are not magic! They are logical machines that you can totally learn to understand.
So this talk is actually going to be about learning hard things and understanding complicated systems.
I work as an engineer at Stripe. (this is the job description for my job).
My team is in charge of a ton of things. Every so often I find out about a new thing that we're in charge of ("oh, there's a GPG keyserver we depend on? Okay!!")
What this means is that I (like many of you!) need to know about a ton of different systems. There are about a million things to know about Linux & networking, the AWS platform is really complicated and there's a ton to know about how it works exactly.

And there's a seemingly neverending amount of new technology to learn about. For instance we're looking at Kubernetes, and to operate a Kubernetes cluster you need to operate etcd, which means that you need to understand a bunch of distributed systems concepts to make sure you're doing it right.
So to do my job effectively, like many of you, I need to constantly learn new things. This talk is about how to do that, and why I like it.
Here are the wizard skills we're going to be discussing in this talk!
In software engineering, I think it's really important to understand both the systems that are a little higher-level than you and lower-level systems.

In reliability engineering, what lives below us is typically "systems stuff" like operating systems & networking. Above us is stuff like business requirements & the programs we're trying to make run reliably.

This talk is mostly going to be about understanding lower-level systems, but we're also going to talk a little about humans and how to make sure you're actually building the right thing :)
As a quick aside, I think understanding computer networking is so important that I wrote a whole zine about it, which you can pick up at the end of this talk.
So -- why is it important to understand the systems you work with?
I think there are 3 main important reasons:

First, understanding jargon is really useful. If someone says "hey, this process got killed by the OOM killer" it's useful to know what that means! (we're going to talk about what an OOM killer is later)

second, it lets you debug harder problems. When I set up a web server (Apache) for the first time, maybe 8 years ago, I didn't understand the HTTP protocol very well and I didn't understand what many of the configuration options I was using meant exactly.

So I would normally debug by Googling things and trying random fixes. This was a pretty viable strategy at the time (I got my webservers working!) but today when I configure webservers, it's important for me to actually understand what I'm doing and exactly what effect I expect it to have. And now I can fix problems much more easily.

rachelbythebay is a great collection of debugging stories, and it's clear throughout that she has a really deep understanding of the systems she works with.

The last reason is -- having a solid understanding of the systems you work with lets you innovate. I think Docker is a cool example of this. Docker was not the first thing to ever use namespaces (one of the kernel features that people call "containers"), but in order to make a tool that people loved to use, the Docker developers had to have a really good understanding of exactly what features Linux has to support isolating processes from others.
A system like Linux seems really intimidating at first, especially if you want to understand some of the internals a little bit. It's like 4 million or 10 million lines of code or something.
So let's talk about how to break off pieces of knowledge one at a time so that you can tackle the challenge!
My first favorite thing to do is learn fundamental concepts.

This is incredibly useful -- in networking, if you know what a packet is and how it's put together, then it really helps to tackle other more complicated concepts.

Let me tell you a quick story about how I learned what a system call was.
The Recurse Center is a 12 week programming retreat in New York where you can go to learn fun new things about programming. I went 3 years ago. RC is about learning whatever interests you (low level stuff? fancy frontend tricks? functional programming? making cool art with programming?), and I went partly with the goal of understanding operating systems better.

When I got to RC, I learned about the concept of a "system call"! (here's the blog post I wrote the day I learned that). System calls are how applications talk to the operating system. I felt kind of sad that I didn't know about them before, but the important thing was that I learned it! That's exciting!
This is the only piece of homework in this talk :)

TCP is the protocol that runs a lot of the internet that we use day to day. Often it "just works" and you don't need to think about it, but sometimes, well, we do need to think about it! So it's helpful to understand the basics.

The way I started learning about TCP was, I wrote a TCP stack in Python! This was really fun, it didn't take that long, and I learned a ton by doing it and writing up what I learned.
I also like to do experiments.
You can make your laptop run out of memory on purpose! I would show you what happens (remember the "OOM killer"? it's a system in the Linux kernel that starts just killing programs on your computer!), but I think it might not be a good live demo for a talk :).

I think doing this kind of experiment is awesome because servers run out of memory in production, and it's cool to see what that looks like and how to reason about it in a safer environment. (hint: if you run "dmesg" and search for "oom" it will show you OOM killer activity)
Also at the Recurse Center, I decided I wanted to write a tiny operating system in Rust. It turns out that writing an OS in 3 weeks when you don't know Rust or operating systems is hard, so I ended up writing a keyboard driver.

I learned SO MUCH by doing this -- you can read more about it here
These are the programming experiment rules I like to follow.
Sometimes I like to read books! Two books I've learned from in the last couple years are Networking for System Administrators & Linux Kernel Development. Networking for System Administrators is written for system administrators who want to be able to do basic networking tasks without having to ask their networking team. I'm not a system administrator, and I don't have a networking team, but I learned a ton by reading this book.
Another thing that I find super helpful is to try to read things or watch talks that are too hard for me at the time. For example, aphyr has an amazing series of posts about distributed systems failures (the ones called "Jepsen"). When I started reading these posts, I honestly didn't understand them very well. I didn't understand what "linearizable" meant, and I'd never worked with distributed databases. So sometimes I'd read a post and only understand maybe 20% of it.

As I learned more and came back to his writing, I was able to understand more of it! I'm still not a distributed systems expert, but I'm happy I tried to read these posts even when I didn't understand them well.

That Linux kernel development book I mentioned is kind of similar. Its goal is to give you the tools you need to become a Linux kernel developer. I am not a Linux kernel developer (or at least not yet!). But I've learned a few interesting things by reading this book.
Another maybe obvious tactic is to work with the thing in your job. Recently I needed to add some logging to a HTTP proxy we had. This was a relatively mundane task, but I learned more about how HTTP proxies work exactly by doing it! That was cool!

It's useful for me to remember that I can learn something even when I'm doing work which is sort of routine.
This is the last, and maybe most important thing. I have models of how a ton of systems work in my head. Sometimes what happens on the computers work with does not match what my model says!

As a small example -- recently we had a computer that was swapping even though it had 16GB of free memory. This did not match my mental model ("computers only swap memory to disk when they're out of memory"). Obviously there was something wrong with my model. So I investigated, and I learned a couple new things about how swap works on Linux!
It turns that there are actually at least 4 reasons a Linux box might swap.

1. You could be actually out of RAM.

2. You could be "mostly" out of RAM. The "vm.swappiness" sysctl setting controls how likely your machine is to swap. This isn't what was happening to us, though.

3. A cgroup could be out of RAM, which was what was happening to us at the time (here's the blog post I wrote about that).

4. There's also a 4th reason I learned about afterwards: if you have no swap, and your vm.overcommit_ratio is set to 50% (which is the default), you can end up in a situation where only half your RAM can be used. That's no good! here's a post about overcommit on Linux
So it turns out understanding swap isn't actually that simple. In fact, there's a cool 200-page book Understanding the Linux Virtual Memory Manager. It also has a bunch of annotated kernel code that handles memory management, which is awesome.

I'm happy I dug in a bit because now I understand how this part of Linux works better!
So even getting to understand something that seems relatively basic like "when does a computer start swapping?" can take a while! There's a lot to know, and it's totally okay to not know it all right away.
A quick story on my Linux journey:

In 2003, when I was 15, my mom bought me a shiny new computer. I was really excited about Linux, so I installed a ton of different Linux distributions. (also, thanks to my mom!! I'm super lucky to have had a computer that I could bork repeatedly and the time & space to do tons of experiments)

Around 2009, in university, I was one of the sysadmins for a small lab of 7 Linux & Windows computers. The old sysadmin said "hey, want to help out?", gave me the root password, and we muddled our way through getting the computers to work for a bunch of math undergrads.

In 2013 I learned what a system call is and a bunch of basic things about operating systems! This was super awesome. (here's everything I learned at the Recurse Center
< And now I'm still continuing to learn.
The next wizard skill I'm going to talk about is asking great questions!
A (great) situation I end up in a lot is where I have a coworker who knows something that I want to know, and they want to help me, and I just need to figure out the right questions to ask to get the answers I want!

Asking good questions is really important because people in general cannot just magically guess what I want them to tell me.
One of my favorite tricks is to state what I know, as a way to frame my question.
Stating what I know is awesome because it helps me organize my thoughts, reveals misunderstands (me: "I know X", them: "that not quite right!"), and helps me avoid answers that are too basic (yes yes yes i know that!) and too advanced (NO PLEASE BACK UP 30 STEPS FIRST).
When asking a question, it's pretty natural to want to ask the most experienced person around your question. They will probably know the answer, which is good! But I don't think it's the best strategy.

Instead, I instead try to remember to ask a less experienced person, who I think will still know the answer.
This is awesome because it reduces the load on the more-experienced person. But there's more reasons this is great! I'm not the most experienced member of my team. I love it when people ask me questions because -- if I don't know the answer to their questions, then I can find out, and I can grow my own knowledge.

So not asking the most experienced person is actually a cool way to show trust in less experienced team members, reduce the bus factor, and spread knowledge around.
Doing research is great! It lets me ask more complicated and interesting questions!
I really like to ask questions that are relatively easy to answer. yes/no questions are a really good way to accomplish this! And often an interesting yes/no question can lead to a great discussion.
When debugging or fixing things, often you can end up in a situation where someone who's super experienced knows how to Do A Thing, and other people on the team don't know how.

And often they have trouble remembering all the details to document them! So I like to (right after someone did something) to ask them to explain exactly what they did, or to ask if I can watch while they do it.
The last thing I have to say about asking about questions, especially to senior engineers / managers / leaders is -- please ask questions in public. I find that it's much easier for senior people to admit that they don't know something (because everybody knows you're competent already!), and doing that really creates space for everyone to ask questions.
Okay, let's talk about reading code!
Sometimes error messages are not particularly helpful. If you go read the code around where the error message got printed, sometimes you can get a better clue about what's going on!
What's more exciting to me, though, is to read the code when software is poorly documented (which happens all the time, especially when it's changing frequently or isn't used by very many people)
I want to emphasize that reading code isn't just for small projects that you're familiar with, though.

In my first job, I was writing plugins to make websites with Drupal, a PHP content management system. Once I remember I had a really specific question about how some Drupal thing worked. It wasn't documented, and there were no results on Google when I looked.

I asked my boss at the time if he knew and he told me "julia, you just have to go read the code and find out how it works!". I was a bit unsure about how to approach it ("there's so much code") but he pointed me to the relevant part of the Drupal codebase, and, sure enough, I could see the answer to my question there!

Since then I've looked at the code for a bunch of large open source questions to answer questions (nginx! linux!) and even if I'm not a super good C programmer, sometimes I can figure out the answer to my question.
Now we're going to talk about one of my favorite things! Debugging!
Let's tell a story! One day we had a client that was making a HTTP request, and it wasn't getting a response for 40 milliseconds. That's a long time!
Why is that a long time? The client and the server were on the same computer. And I expected the server to be fast, so there was no reason for a 40ms delay.
As an aside, 40ms synchronously is 25 requests per second, which is really not a lot. It's easy to see how this kind of delay could become a problem quickly.
I captured some packets with Wireshark to figure out who I should be blaming -- the client or the server!

We found out that the client would send the HTTP servers, wait 40ms, and then send the rest of the request. So the server wasn't the problem at all! But why was the client doing this? It's written in Ruby, and initially I maybe thought we should just blame Ruby, but that wasn't a really good reason (40ms is a very long time, even in Ruby).
It turned out what was happening was a bad interaction between two TCP optimizations -- delayed ACKs, and Nagle's algorithm. When the client sent the first packet, the server would wait to send an ACK (becaues of the delayed ACKs algorithm), and the client was waiting for that ACK (because of Nagle's algorithm).

So they were stuck in this kind of passive-aggressive-waiting situation.

I wrote a blog post about this called Why you should understand (a little) about TCP if you want to know more.
When we set the TCP_NODELAY socket option, it stopped the client from waiting, and then everything got fast!
A while ago I realized I felt like I'd gotten a lot better at debugging since my first job, and I came up with some reasons I think it got easier!
Sometimes when I hit a bug, especially a nondeterministic and difficult to reproduce bug, it’s tempting to think “oh you know, things just happen, who knows”. But everything on a computer does in fact happen for a logical reason (however much the computer may try to convince you otherwise). Reminding myself of that helps me fix bugs. Also known as “OK JULIA IT IS NOT FAIRIES WHAT ACTUAL REASON COULD BE CAUSING THIS?”
Next up, confidence!
A while ago I dealt with a performance problem in a Hadoop job at work that took me 2 weeks to fix (see a millisecond isn’t fast). If I hadn’t been able to fix it, I would have felt pretty bad and like it was a waste of 2 weeks.

But we were processing a relatively small number of records, and it was taking 15 hours to do it, and it was NOT REASONABLE and I knew that the job was too slow. And I figured it out, and now it’s faster and everyone is happy.
From that, I learned that floating point exponentiation is slow, and that 1000 records/second isn't really a lot.
The job was processing 1000 records/second. I found this hard to think about at the time though -- was that a lot? not a lot? How was I supposed to know?

So I decided I wanted to take some time to train my intuitions about how fast different computer operations should be.
I made this game called computers are fast with my partner Kamal. You can go play it online, and we're going to play it now a little bit!
Suppose you have an indexed database table, with 10 million rows in it. How long does it take to select a row from that table? How many times per second can you do that?

The goal isn't to know exactly, but I think it's useful to be right up to an order of magnitude. So can you do it 100 times in a second? 10,000? 10 million times?
It turns out the answer on my laptop is 55,000 times! (or, it takes about 20 microseconds, in Python)
It's also been incredibly helpful for me to have better tools for answering questions about your programs!

When I started out, I didn't have very good tools! But now I know about all kinds of profilers! I know about strace, and tcpdump, and way more tools for figuring out what's going on. It makes a huge difference.
I wrote a whole zine about debugging tools that have helped me answer questions. You can read it here: Linux debugging tools you'll love
But maybe the most important thing is that I learned to like debugging! I used to get grumpy when I ran into bugs. I felt like they were just getting in my way!

But these days, when I run into a mysterious bug, I think it's kind of fun! I get to improve my understanding of the systems I work with, which is awesome!
Now we're going to zoom out a bit from talking about networking and microseconds, and talk about how to design engineering projects. This is something that's really helped me a lot!
There are a lot of words for design document, but they're basically all the same idea -- you write down words about what work you're going to do before doing the work.
When I started at Stripe I thought writing stuff down was kind of dumb. Why couldn't I just start working on the project?

But since then, I've learned to find them really useful! (learning to like design documents)
I was worried that if I wrote a document, I'd either get WAY TOO MUCH feedback, or total silence.

One thing I learned is that it's helpful at first to just share a design I'm working with a few people. Like I'll show it to a couple of other people on my team, see what they think, and then make changes! It's not always necessary to ask every single person who might have an opinion what they think.
I also learned to like designing small projects. Recently I worked on a tiny project that just took about a week. My team lead asked me if I could quickly write up what we were going to do.

It took me maybe 45 minutes to write up the plan (super fast!), I showed it to a manager on another team, he had a couple of things he asked me to do differently, and he was SO HAPPY I'd written down a plan so that he understood what was going on. Awesome!

The small project went super smoothly and I was really happly I wrote up a thing about it first.
How do you know what to write in a design document? I really like to start by writing an announcement email, as if we just finished the project.

This is great because it forces me to articulate why the project is important (why did we spend all that time on it), how it's going to impact other teams ans what other people in the organization need to know about, and how we know that we actually met our goals for the project

The last thing is really important -- more often than I'd like to admit, I get to the end of a project and realize I'm not quite sure how we can tell whether the project is actually going to improve things or not. Planning that out at the beginning helps make sure that we put in the right metrics!
It's also useful to talk about risks! I actually haven't done this yet, but a cool idea I heard recently for figuring out risks was to do a "premortem" for your project. This is kind of the opposite of an announcement email -- instead, you imagine that the project failed 6 months down the line, and you're making sure you understand why it failed.
When I started writing designs, I used to worry a lot that my design would be wrong because things were going to change. It turns out that this is totally true -- designs rarely survive contact with the real world. Priorities change, you run into technical challenges you didn't expect, all kinds of things can go wrong.

But this doesn't mean it's not worth designing at all! I like writing down my assumptions explicitly because when things do change, I can go back and see what assumptions we had are no longer true, and make sure that we update everything we need to update. Having a record of changes is useful!
We've arrived at the last wizard skill!
Sometimes I'm working on something kind of boring, and I wonder like.. why am I doing this?

I usually find it possible to stay motivated if I can remember "ok, I'm spending hours working on configuring nginx, and this is boring, but it's in service of this really cool goal!"

But if I *don't* remember the goal (or what I'm working on actually doesn't make sense), it sucks.
The solution I'm working on to this right now is to approach project planning with the same kind of excitement and curiosity you might bring to a gnarly bug!

I'm trying to get better at saying "okay!!! this project! it has some slow and difficult pieces, so why is it so important? why are we going to feel awesome when it's done? which parts are the most important?"
I have a lot of autonomy in about what I get to work on, so when someone asks me to do something, I like to make sure I understand why it's important. Usually if I don't understand, the right thing to do is to just find out why it's important (usually it actually is!).

But sometimes the task I'm being given is only maybe 80% thought through, and when I go to understand the exact reason for doing it, it turns out that we don't need to do it at all! (or maybe we should actually be doing something completely different!)
And understanding the big picture helps me make better technical decisions!
Like a lot of people, I think a lot about the impact my work has and what I'm really doing here. Kelsey Hightower had a really amazing series of tweets today. here are a couple of quotes.

> I’ve yet to find the perfect job or thing to work on, but I have found a way to live a more meaningful life in tech.

> I now put people first. Regardless of the technology involved I gravitate towards helping people.

> People provide a much better feedback loop than computers or the abstract idea of a business.

> Everything I work on has a specific person or group of people in mind; this is what gives my work meaning; solving problems is not enough.
during this conference, I hope you ask a ton of questions to understand what's going on with this "SRE" thing better. There are so many amazing people to learn from!
I handed out fun networking zines at the end of this talk. If you want to read the zine, it's here: Networking! ACK!
Read the whole story
153 days ago
159 days ago
Melbourne, Australia
Share this story
1 public comment
160 days ago
Julia Evans still favorite person on the inernet
Brooklyn, New York

Ultimate language threat

1 Comment

The news these days, I find, seldom merits a smile. But at one news story I heard at lunchtime today I actually laughed out loud, alone in my kitchen. Michel Barnier, charged with heading the EU side in the complex forthcoming negotiations that will set the terms for the UK's exit from the European Union, has found a way to hurt the British more deeply, and put them more at a disadvantage, than I ever would have thought possible. It is so fiendish it ought to be illegal, yet it violates no law or basic principle of human rights. It is simply wonderful in its passive-aggressive hostility. I  passive-agressive hostility. I have to take my hat off to him. He has announced that he wants all the negotiations with the British team to be conducted in French.

However, spoilsport Paul Foord has pointed me to Barnier's Twitter feed, where the man himself, writing in English, denies even expressing a view on language preference, and stresses that the linguistic medium for the future talks will be agreed between the participants. "Work as often in EN as FR," he says. "Linguistic regime to be set at start–to be agreed btw negotiators."

So that's killed that little moment of laughter. Thanks for nuthin', Paul. My pleasure lasted for slightly more than 70 minutes.

Read the whole story
304 days ago
Share this story
Next Page of Stories