Simplicity can be Imaginary

There’s a comic about simplicity: how an Apple product has one place to touch, a Google product has one search field, and “your company’s app” has dozens of fields with interrelated requirements, obscure codes, strange highlighting, and “…” buttons.

The thing is, for internal or even b2b apps, the user probably knows what kind of thing they have, that they would like to search on.  If they are trying to look up a customer ID, then matching to a PO number is irrelevant; it will just take time and produce extraneous results.  If they can tell the computer directly, “Find customer #33448” then jump straight to the customer record, it saves them an extra round-trip through a search result page they didn’t need.

“Your company’s app” from the comic comes across as more of a data-entry page than the main point of interaction.  One might still organize the form along required/optional dimensions, and put auto-loaded fields in proximity with what will automatically update them.  However, to make the business happen, there’s a minimum amount of data that is genuinely required, that shouldn’t be crammed down to one textarea and parsed back out.

The Enterprise’s Goals

When I was a n00b on the internets, I heard whisperings about awful, over-complex “rule based systems” out there, somewhere.  Programmers scoffed at them for essentially being programs that were being written by non-programmers; nebulous “managers” allegedly dreamed of replacing expensive programmers with cheap office staff.  I did not understand at the time where these systems came from, if everyone seemed to think they were so bad.

Part of that answer is simple.  “Programmers aren’t ‘everyone.’” Oops.

The other part of that answer turns out to be review and auditing. Anything that exists in code is opaque to the business staff; they largely have to trust the programmers on it, or demonstrate defective outcomes. (And at that point, they need to wait for the necessary programming and deployment for it to be fixed.  If it is a big enough problem that customers or clients are exploiting in the meantime, that delay can become costly.)

Functionality that is exposed as ‘configuration data’ to the office staff becomes reviewable by other office staff, such as managers, and errors can be corrected more quickly.  External auditors can use the same review capability for their own work.  The next problem is that this data might not be flexible enough, which pushes toward the development of conditions and actions, and the rule-based system is born.

It was never about the programmers; it was about the business being able to view its own source code.

Every Change Might Be Breaking

We originally had the “automatic minor version upgrade” option active at Amazon RDS.  This option simply does not work very well.  Sometimes, for no clear reason (and without notification), it would stop applying upgrades, and require manual updates to get moving again.  We mostly lived with it, and then we hit the worst case scenario: it did perform the upgrade, and then one of our scripts stopped working.

Not only that, it managed to break while I was on vacation.

(Obligatory xkcd about spacebar heating.)

Since then, we don’t use that option.  When I’m good and ready, I peruse the changelogs, then schedule the update to happen when I will be in the office to handle unexpected issues.

For their part, AWS recommends testing the app against the new version of the database before performing any upgrades.  This is implicitly a recommendation against using automatic minor upgrades, because there is no automated process to test the upgrade first.

One knows an analysis tool is looking at AWS with a security-first paradigm when it recommends switching the automatic upgrade option back on for the database.  It is technically correct that new releases MAY contain security fixes, but upgrading to them MAY cause an automated denial of service.  It is not a simple, inconsequential task.

Some Notes from Fixing my Server’s IPv6 / SLAAC

I had a hard time getting IPv6 to work properly on my VPS. It has a static address, which I published to DNS (ages ago), but it wasn’t fully operational. It wasn’t obvious, because it was able to accept and respond to incoming IPv6, but it was not able to generate outgoing IPv6 connections. Thanks to Happy Eyeballs, the system cheerfully fell back to IPv4 and left me none the wiser.  Probably for years. (Since inbound traffic could be responded to, the IPv6 network-transfer graph looked plausible, too.)

Spammers Get Confused About Temporary Errors

When I wrote Everything Needs Rate Limits, I mentioned in passing that the disk-full state prevented receiving email.  The MTA was returning a temporary “try again later” code, but the clients weren’t responding.  I got several session transcripts emailed to me that were of the form:

> EHLO some-host-name
< 200 OK + capabilities listing
> MAIL FROM sender-address
< 200 OK
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> DATA …
< 500 No recipient given
X Connection lost

The client is recognizing that they’re getting some sort of error, but their idea of “later” was milliseconds later, and the disk space problem was not being cleared at CPU speeds.  After enough rejections of their recipient-address, they YOLO’d it and sent the email body, to no avail.

(I also think it’s pretty interesting that the MTA is happy to tell everyone else ENOSPC, yet deliver these error emails through to me.  I suppose it purposely stops accepting email with some reserve disk space, so that it can continue to deliver critical errors for a while.)

Finding When You Pushed and Pulled a Git Repository

I found myself needing to know if I had pushed something before or after some operational event. The commit itself was dated late on Friday before the event; could it have been deployed then, or did I wait until Monday to push?

It turns out that git reflog has the answers; as it's a log of what the branch tip pointed to, it logs commits, amends, and push/pull activity.  It doesn't normally include a date in the output, but we can convince it to give us one by using the --pretty formatting option.

git reflog --pretty='%cd %h %gd %gs'

This format means “commit date”, Thu Nov 28 17:56:08 -0500 style; the hash; the “shortened reflog selector,” like HEAD{1}; and finally, the reflog summary, which has information like commit (amend): Awesome subject line.

Within the reflog, for push/pull events, the “commit date” is actually the date of the push/pull completing its task.

The major limitation here is that the reflog is local; we can only look at our own activity with this method.

However, in my situation, that was enough to exonerate my commit, allowing us to turn our attention elsewhere for an explanation.

Everything Needs Rate Limits

For reasons of “anything else would cost more,” my web server and email MTA are running on the same VPS. Which apparently means, if a known issue in some web-side software fills the disk, email quits flowing in.

Fortunately, I was at liberty to investigate immediately. The root cause was a bot making things up and filling a cache with negative responses. The actual bot was promptly banned. Not only via robots.txt (which used to return text "# 200 OK Have Fun" so that I wouldn’t get 404 errors for it) but via User-Agent in the Web server, since it had obviously already seen the permissive robots.txt contents.

(Also, the only email that failed to be received were spam messages.  “Lucky” me!)

It worried me, though.  What if another bot did this? Am I going to play Whac-A-Mole® with it? (Don’t get me wrong—that’s not a bad game, but this version doesn’t give me any tickets redeemable for prizes at the front counter.)

To give me more runway to respond to future problems, I added per-source rate limits. This should prevent the cache from filling in units of MB/s. Concurrently, there is also a custom disk-usage alarm that will activate if free space falls below a “normal” amount, giving me a chance to catch problems before the MTA starts refusing service again.

A global rate limit for new connections is still being considered. The current bot menace has been beaten for today, and future bots from a single IP will also find themselves running into limits, but someone running a crawler network could still cause plenty of trouble. The problem is that a global rate limit is probably something that would be HTTP-oblivious, responding with an RST packet instead of an application-layer 429 message.

I know the “everything needs rate limits” is common wisdom, at least in some circles, but I ran public sites for at least 14 years without them. Sadly for the nostalgia, it seems those days are gone.