Spotify’s Helios in a Nutshell


I work at Spotify on backend infrastructure. In this context, infrastructure is the shared plumbing and platform on which various Spotify systems run. In particular, I work on an open source tool called Helios. This project and, more importantly, my team are pretty awesome.

What is Helios?

Helios is a Docker orchestration framework. This means it’s a tool used to manage your Docker containers across a fleet of hosts.

Why did Spotify create Helios?

Let’s say you have 20 hosts, and you want to run a Docker container named “hello-world” on each of them. You’ve installed the Docker daemon on each. Now you SSH into each of them and run docker run hello-world. Doing this 20 times is tedious.

So you use cluster SSH or fabric and run it once. That works.

Your application grows and you need environmental variables, exposed ports, and mounted volumes. You Docker command becomes longer:

docker run --env="FOO=BAR" --publish=80:8080 --volume="/etc/default/config:/etc/default/config"

You need to remember this long command so you save it to a file in your code repository. This works OK.

You notice your Docker containers sometimes crash when you start them. You run watch 'docker ps' to see which containers have crashed on which hosts. You have to tail the logs on those hosts to figure out what went wrong. Hm, this is becoming hard to manage.

One day, one of your hosts restarts. You don’t notice that the container is no longer running on that host until several days later. Maybe it’s time to think of a better solution.

These and many other reasons are why Spotify created Helios. Helios makes it easy to deploy to multiple hosts, and Helios keeps track of which containers are running where and will restart containers if they crash.

Semester at Sea


I first heard of Semester at Sea (SAS) from my friend Alyssa who attended the University of Virginia, the official sponsor of the program. SAS is a study abroad program for college students. They live on a ship that circumnavigates the globe while they take courses and visit various regions. It attracts both youth struck by wanderlust and affluent collegiate party-goers alike.

I was envious that I didn’t spend an entire college semester partying aboard a luxurious ship that docked in exotic locations throughout the world. What was I doing drifting around Manhattan’s Morningside Heights when I could’ve explored Mauritius, Malaysia, and Malta? But as I asked Alyssa about her experience, I learned the cruise wasn’t always luxurious and while partying at sea was an allure for many students, it wasn’t the only one.

Confessions of a Noob Landlord I - How to Fix a Gas Range


  1. Have a tenant email you asking you to check out a strange chemical smell coming from her oven when she uses it.

    From: Tenant

    Date: January 20, 2015

    Subject: Oven problems

    To: David

    Hi David,

    I’m sorry to keep email [sic] you with problems, but over the past few weeks the oven has increasingly emitted a chemical smell. Any time the oven is turned on there is a strong burning/chemical odor from the back of the unit. The rear is unusually hot and the smell is so strong that it burns our eyes and noses, so we’ve stopped using it entirely.

    Thanks for your attention,

How to Troll Your Roommates With a Shared Router


This article will show you how to troll your roommates or family by sending their browsers to this amazing website where He Man sings “What’s Up” by 4 Non Blondes (flash required) whenever they try to visit

What you’ll need:

  • administrative access to a router (I used a Netgear router with firmware WNR1000v3 leased from Time Warner)
  • router firmware that lets you configure static routes, port forwarding, etc
  • a server connected to the router (I used jarvis which runs Ubuntu 12.04)
  • DNS software running on that server (I used bind)

What you won’t need:

  • access to your victims’ devices (we will be doing something more sophisticated than simply editing /etc/hosts)

How the prank will work

We will intercept the DNS queries unsuspecting devices make to the router for and reply back with the IP address for He Man. The router will send those DNS queries to your server instead of legitimate DNS servers and your server will reply back with the IP for He Man. Your roommates will be confused.

Can I Please Have an Unsubscribe Option for Physical Mail?


I hate email spam and fanatically guard my inbox from it. I try to not give out my real email address unless it’s for an account I need or a service I find useful. These include bank accounts and billing accounts like credit cards and utility bills.

But I inevitably receive spam. Gmail does a great job of filtering out most of them. But for ones that get through, I appreciate organizations that add an unsubscribe link in their email footers. They are doing themselves a favor because I’m going to click that link instead of marking their email as spam which might hurt their email reputation and actually make it harder for their emails to get into inboxes overall.

Almost all my physical mail is junk, but none of them tell me how to unsubscribe. It’s rather unfair how companies collect and use my email in the first place. They ask me for it when I buy something for the first time without explaining how they’re going to use it, or in the off-chance they do explain, it’s usually about following up on my purchase.

In the months that follow, they send me with sales offers and catalogues. Why would a merchant waste the goodwill I gave them by sending me unsolicited mail? I should be asked to opt-in to promotional and marketing mail. Okay, maybe they’re too lazy or their employee forgot. Then at least give me an option to unsubscribe.

Not only does it crowd my mailbox, I now have to worry that when I move, some sensitive piece of mail might find its way into a strangers hands and be exposed or used maliciously.

Here’s an example of a letter I received from Mount Sinai. I’ve only visited Mount Sinai once to ask a surgeon about what my options were for two herniated disks in my lumbar. I can’t think of any other way Mount Sinai got my mailing address.

The more I think about the letter in the photo above, the more I start to wonder. How and why did my email from a surgical consultation get passed on to Mount Sinai’s fundraising department’s mailing list? Does this violate HIPAA?

They should at least provide me a method to unsubscribe myself. Something like a website, phone number, even a mailing address. Instead they don’t even provide a postage paid envelope for the check they expect me to write.

Corporate Lingo


From elementary school through college, I learned standard English writing and speaking. But when I started working, I noticed there was something different about the language people spoke and wrote in the corporate world and the one I was taught in school. I couldn’t put my finger on it, but the jargon felt impersonal, verbose, bombastic, and loved nouns rather than verbs.

Here are some examples of sentences I’ve heard people utter with a straight face at work:

  • What’s the ask?
  • Our win of the week was…
  • After doing an A/B test, our learnings are…
  • The deliverable is forthcoming.
  • We are waiting for sign-off.

Why do we pervert perfectly good verbs into dreadful nouns? What’s wrong with

  • What does he want?
  • Our achievement this week was…
  • After doing an A/B test, we learned…
  • I will send the result to you soon.
  • Do you approve?

To fit into a corporate setting, be verbose and bombastic.

  • “We can get the deck turned around tomorrow.” instead of “We can give you the updated deck tomorrow.”
  • “The layout will be leveraged for all other workstreams.” instead of “Everybody will use the same f**king layout.”
  • “Please flag anything you might want to escalate.” vs “Tell us if you’re worried about anything.”
  • “Let’s talk offline.” (People have said this to me in person. What do they even mean? Aren’t we already offline?)

Blatantly misuse words.

  • “Please revert with comments.” vs “Please comment.”

And finally, just make up words that sound like Newspeak.

  • “Let’s future-think our mobile design.”

How about “buy-in” or “ideation” or “react” as a noun? People also say “we” when they really mean “you” as in “We should do this.” ie “You should do this.” This usually occurs when speaking to subordinates.

I’m reminded of George Orwell’s “Politics and the English Language” essay in which he criticized political language for being vague and meaningless. He called this kind of language a “contagion” and encouraged people instead to write and speak with concreteness and clarity.

I’m also reminded of the Java programming language which loves nouns instead of verbs. Here’s a fun explanation by Steve Yegge titled “Execution in the Kingdom of Nouns.”

How to Connect to Freenode IRC via SSL


It took me a while to figure out how connect to freenode’s IRC servers via SSL today. Here’s a reference for my future self and anyone else who might find this helpful.

N.B. I use weechat as my IRC client.

  1. Pick a nick that’s not already taken. Weechat kept retrying unsuccessfully when I used one that was already taken.
  2. freenode has instructions here on how to connect via SSL
  3. I run weechat on Ubuntu 12.04 and had to install the intermediate and root certificates mentioned in the link above. Otherwise, weechat complains that it doesn’t recognize the certificate.
  4. sudo mkdir /usr/share/ca-certificates/extra
  5. Copy the Gandi and InstantSSL certificates to that directory.
  6. Let Ubuntu add the new certificates’ path relative to /usr/share/ca-certificates to /etc/ca-certificates.conf by running sudo dpkg-reconfigure ca-certificates.

This should be all. Here’s the weechat IRC configuration for freenode which should be written to ~/.weechat/irc.conf.

freenode.addresses = ""
freenode.ssl = on
freenode.ssl_dhkey_size = 1024
freenode.autoconnect = on
freenode.nicks = "davidxia"
freenode.realname = "David Xia"

New Yorker Redesign


The New Yorker recently redesigned their web site and made all articles free for a limited time. Everything looks great except for that giant sticky header that takes up at least an inch of your screen. I hope they get rid of it.

Amsterdam 2014


I visited Amsterdam after coming going to Stockholm for work recently. It was my first time there. Amsterdam’s canal, bike paths, and green spaces are beautiful.

Indonesia 2014


Nanette and I traveled to Indonesia and visited Borobudur, the site of the world’s largest buddhist temple, and hiked Mount Rinjani, an active volcano.