While stuck inside during social distancing, I’ve been making my way through LiveOverflow’s awesome Youtube playlist “Binary Exploitation / Memory Corruption.” His videos are structured around a well known series of exploit exercises here called “Protostar.” I took the time to truly understand each one before moving onto the next as the exercises build on each other. For the past several days I’ve been trying to understand the “Heap3” level, a relatively complex level that requires manipulating the heap to redirect code execution to an arbitrary function. After rewatching the video many times and reading numerous other online explanations, I finally understand! That moment of understanding feels so gratifying.

Many other resources already explain the exploit well, but I’m writing my own explanation to reinforce my understanding and to celebrate.

In my previous post I wrote about how to load test GKE Workload Identity. In this post I’ll describe how to get metrics from gke-metadata-server, the part of Workload Identity that runs on your GKE clusters’ nodes. This solution is a temporary workaround until GKE provides a better way to get metrics on gke-metadata-server.

Gke-metadata-server runs as a K8s DaemonSet. It exposes metrics about itself in Prometheus text-based format. I want to have an external scraper make HTTP requests to periodically collect these metrics. Unfortunately, the Prometheus HTTP server only listens on the Container’s localhost interface. So how can we expose these metrics, i.e. make the HTTP endpoint available externally?

tl;dr lessons learned

• socat is awesome.
• If something you need is running on a computer you control, you can always find a way extract info from it if you’re resourceful enough.

I manage multitenant Google Kubernetes Engine (GKE) clusters for stateless backend services at work. Google recently graduated GKE’s Workload Identity (WI) feature to generally available (GA). When my team used WI during its beta stage, it seemed to fail when there were more than 16 requests per second (RPS) on one GKE node to retrieve Google access tokens.

Before we knew about this low RPS failure threshold, we told many internal engineering teams to go ahead and use the feature. In hindsight, we should’ve load-tested the feature before making it generally available internally especially since it wasn’t even GA publicly.

My efforts to load test WI have grown more sophisticated over time. This post describes the progression. It’s like the “4 Levels of …” Epicurious Youtube videos. The goal here is to find out at what RPS WI starts to fail and to try to learn some generalizable lessons from load testing vendor-managed services.

tl;dr lessons learned

• always load test new features above and beyond what you expect your production load will be
• use proper load testing tools and not bash for loops

At the beginning of this year I set a goal of becoming a better public speaker and more visible in both tech and other broader causes I believe in. I’m happy to say that in the last two months I gave three talks! Two were prepared talks with slides at tech conferences. The other was an unprepared conversation on a podcast. These were all technical and related to my work at Spotify. Outside of Spotify, I spoke for one minute at a mock political town hall in front of about 30 people and at a public policy forum for ~15 minutes in front of roughly the same number of people. But more on that later. Here are my technical talks. These talks wouldn’t be possible without the help, feedback, and moral support from my Spotify colleagues.

1. Keynote at KubeCon + CloudNativeCon Europe 2019 in Barcelona on May 22, 2019

“How Spotify Accidentally Deleted All its Kube Clusters with No User Impact”

2. Kubernetes Podcast from Google on April 23, 2019

“Spotify, with David Xia”. Listen on Spotify here.

3. Joint talk with Google at Google Next SF on April 11, 2019

“GKE Usage Metering: Whose Line Item Is It Anyway?”

This is a post about Nginx’s DNS resolution behavior I didn’t know about but wish I did before I started using Kubernetes (K8s).

Nginx caches statically configured domains once

Symptoms

I moved a backend service foo from running on a virtual machine to K8s. Foo’s clients include an Nginx instance running outside K8s configured with this upstream block.

1
2
3
4
5
6
7
8
9
10
11
12

upstream foo {
  server foo.example.com.;
}

server {
  ...

  location ~* /_foo/(.*) {
    proxy_pass https://foo/$1;
    ...
  }
}

K8s Pods can be rescheduled anytime so their IPs aren’t stable. I’m supposed to use K8s Services to avoid caching these ephemeral Pod IPs. But in my case because of interoperability reasons I was registering Pod IPs directly as A records for foo.example.com.. I started noticing that after my Pod IPs changed either because of rescheduling or updating the Deployment, Nginx started throwing 502 Bad Gateway errors.

Root Problem

Nginx resolves statically configured domain names only once at startup or configuration reload time. So Nginx resolved foo.example.com. once at startup to several Pod IPs and cached them forever.

Solution

It’s been a while since I blogged about something tech related, but I had some free time today.

Recently, I’ve been trying to refactor an internal Spotify deployment tool my team built and maintains. This deployment tool takes Kubernetes (k8s) YAML manifests, changes them, and essentially runs kubectl apply. We add metadata to the k8s manifests like labels.

Right now this tool receives the input YAML as strings, converts them to Jackson ObjectNodes, and manipulates those ObjectNodes. The disadvantage of this is that there’s no k8s type-safety. We might accidentally add a field to a Deployment that isn’t valid or remove something from a Service that’s required.

My refactor uses upstream k8s model classes from kubernetes-client/java which are themselves generated from the official Swagger spec. Here’s a helpful Yaml utility class that deserializes YAML strings into concrete classes and can also serialize them back into YAML strings. So helpful.

Unfortunately, there’s some bugs in the YAML (de)serialization that prevent me from finishing this effort.

• kubernetes-client/java issue 417
• kubernetes-client/java issue 431
• kubernetes-client/java issue 340

Nonetheless, it’ll be much nicer to change k8s resources in a type-safe way instead of parsing and rewriting raw YAML strings.

I’m compiling a list of Internet meme role models. Here’s what I have so far. These people — they must be real human beings — must have either gone out of their way to do the right thing in a smart manner, something courageous with bonus points for being funny, or just be ridiculous. And they must be memeified. Exceptions will be made for exceptional but not memeified people like Hilde Lysiak.

Who else deserves to be on this list?

• Snack Man defused a violent fight on the subway between a couple before anyone got hurt. He did it by standing in between them and munching on Pringles and Gummy Bears.
• Ben Innes asked the EgyptAir Flight 181 hijacker who comandeered the airplane with a fake explosive belt if he could take a selfie with him. “I thought, why not? If he blows us all up it won’t matter anyway.”
• Hilde Lysiak is ten years old and the writer and publisher of Orange Street News. She doesn’t take shit from anyone.
• Salt Bae is…well you just have to see for yourself.

I work at a music company but am more interested in politics and history. Artists visit our office often.

So I, often being ignorant of their fame, have casually interacted with them or criticized their milk steaming techniques when they’re using the office’s $20K espresso machine.

Only later am I told by their posse, “Did you know that was Mark Ronson/Bebe Rhexa/Max Martin/etc?”

“It’s OK,” I say. “Now Max knows how to make a real latte.”

Whenever I go back home to my parents’ house near Boston, if my maternal grandparents are there, they make hundreds of dumplings for me. I try to help out. We make everything from scratch including the skins. I’m good at rolling the skins but have much to learn on all other parts of the process. I’m becoming better at packing and closing the dumplings now though.

I’ve come to cherish this little tradition more and more. I need to plan my next trip to Boston!