More About Nginx DNS Resolution Than You Ever Wanted to Know

| By

This is a post about Nginx’s DNS resolution behavior I didn’t know about but wish I did before I started using Kubernetes (K8s).

Nginx caches statically configured domains once

Symptoms

I moved a backend service foo from running on a virtual machine to K8s. Foo’s clients include an Nginx instance configured with this upstream block.

1
2
3
4
5
6
7
8
9
10
11
12

upstream foo {
  server foo.example.com.;
}

server {
  ...

  location ~* /_foo/(.*) {
    proxy_pass https://foo/$1;
    ...
  }
}

K8s Pods can be rescheduled anytime so their IPs aren’t stable. I’m supposed to use K8s Services to avoid caching these ephemeral Pod IPs. But in my case because of interoperability reasons I was registering Pod IPs directly as A records for foo.example.com.. I started noticing that after my Pod IPs changed either because of rescheduling or updating the Deployment, Nginx started throwing 502 Bad Gateway errors.

Root Problem

Nginx resolves statically configured domain names only once at startup or configuration reload time. So Nginx resolved foo.example.com. once at startup to several Pod IPs and cached them forever.

Solution


Using Upstream Kubernetes Java Models Is Much Better Than Raw YAML

| By

It’s been a while since I blogged about something tech related, but I had some free time today.

Recently, I’ve been trying to refactor an internal Spotify deployment tool my team built and maintains. This deployment tool takes Kubernetes (k8s) YAML manifests, changes them, and essentially runs kubectl apply. We add metadata to the k8s manifests like labels.

Right now this tool receives the input YAML as strings, converts them to Jackson ObjectNodes, and manipulates those ObjectNodes. The disadvantage of this is that there’s no k8s type-safety. We might accidentally add a field to a Deployment that isn’t valid or remove something from a Service that’s required.

My refactor uses upstream k8s model classes from kubernetes-client/java which are themselves generated from the official Swagger spec. Here’s a helpful Yaml utility class that deserializes YAML strings into concrete classes and can also serialize them back into YAML strings. So helpful.

Unfortunately, there’s some bugs in the YAML (de)serialization that prevent me from finishing this effort.

Nonetheless, it’ll be much nicer to change k8s resources in a type-safe way instead of parsing and rewriting raw YAML strings.


Internet Meme Role Models

| By

I’m compiling a list of Internet meme role models. Here’s what I have so far. These people — they must be real human beings — must have either gone out of their way to do the right thing in a smart manner, something courageous with bonus points for being funny, or just be ridiculous. And they must be memeified. Exceptions will be made for exceptional but not memeified people like Hilde Lysiak.

Who else deserves to be on this list?

  • Snack Man defused a violent fight on the subway between a couple before anyone got hurt. He did it by standing in between them and munching on Pringles and Gummy Bears.
  • Ben Innes asked the EgyptAir Flight 181 hijacker who comandeered the airplane with a fake explosive belt if he could take a selfie with him. “I thought, why not? If he blows us all up it won’t matter anyway.”
  • Hilde Lysiak is ten years old and the writer and publisher of Orange Street News. She doesn’t take shit from anyone.
  • Salt Bae is…well you just have to see for yourself.

Now Max Knows How to Make a Latte

| By

I work at a music company but am more interested in politics and history. Artists visit our office often.

So I, often being ignorant of their fame, have casually interacted with them or criticized their milk steaming techniques when they’re using the office’s $20K espresso machine.

Only later am I told by their posse, “Did you know that was Mark Ronson/Bebe Rhexa/Max Martin/etc?”

“It’s OK,” I say. “Now Max knows how to make a real latte.”


Making Dumplings With My Grandparents

| By

Whenever I go back home to my parents’ house near Boston, if my maternal grandparents are there, they make hundreds of dumplings for me. I try to help out. We make everything from scratch including the skins. I’m good at rolling the skins but have much to learn on all other parts of the process. I’m becoming better at packing and closing the dumplings now though.

I’ve come to cherish this little tradition more and more. I need to plan my next trip to Boston!


Four Fascinating and Weird People

| By

Here are the stories of four fascinating and weird people that will make you laugh, be inspired, or cringe. Chang and Eng Bunker were conjoined twins who married two sisters and were slave-owners on the side of the Southern Confederacy. Rose Wilder Lane is the daughter of the author who wrote the Little House childrens books, a founding member of the American Libertarian movement, and just all around boss ass bitch. John Harvey Kellogg was the inventor of corn flakes, doctor, zealous anti-masturbation campaigner, and eugenicist.


Useful Site for TLS Server Test

| By

My home server’s hard disk’s partition map was somehow corrupted. So I’m serving this website from Digital Ocean for now instead of my apartment. While rewriting the nginx server configs, I found this useful site that tests your server’s TLS configuration. It’ll give you a grade and warn you of weak encryption, key exchange protocols, cipher suites, etc.

Mozilla’s TLS configuration generator is useful for providing secure defaults.

I’m proud to say this site has an A.



British Virgin Islands Sailing Trip

| By

I learned the basics of sailing on a week-long sailing trip in the British Virgin Islands aboard a 48-foot catamaran we called Millenium Falcon.

January 30

We moored off the coast of Virgin Gorda and Saba Rock. President Obama was wind surfing at neighboring Necker Island as a guest of Richard Branson.

The dinghy race at Virgin Gorda.

How to Test Your Home’s Water for Lead

| By

I read the New York Times article “How New York Gets Its Water” a while ago. The end of the article described how you could order a free lead testing kit.

The EPA offers some guidance on how and when to obtain an in-home testing kit through your local water supplier. In New York City, residents can obtain one at no cost through 311.

For more information, visit www.epa.gov/safewater/lead, or call the Safe Drinking Water Hotline at 1-800-426-4791.

I read this article in the aftermath of the Flint water crisis and wanted to test my apartment’s drinking water. I filled out the online form, received a package in the mail containing two large plastic bottles, a pre-paid mailing label, and detailed instructions. The instructions said to fill one bottle with water after not using any faucet in my apartment for six hours. I filled the second bottle with water after having run the tap. After dropping the bottles now full of water into a mailbox, NYC’s Nureau of Water Supply sent me the results only week or so later.

We have received the results of your recent tap water lead test. Both samples were under the federal action level of 15 micrograms per liter (μg/L). Your specific results were:

First draw: 0 μg/L 1-2 minute flush: 0 μg/L