Adventures with kASLR

Things I discovered recently about kASLR:

• Linux added kaslr a while ago, in 3.14.
• It was enabled via kaslr on the kernel command line.
• kaslr wasn’t compatible with hibernation, originally.  This appears to have changed in 4.8.
• It was enabled by default in 4.12, with nokaslr to disable it.
• kaslr support is compiled in by CONFIG_RANDOMIZE_BASE=y.
• Ubuntu 18.04 LTS (bionic) has that setting enabled in the generic and aws kernels, which are based on 4.15.
• /etc/default/grub may be useless on Ubuntu for setting command line flags.

Under the hood, there are “up to” 512 base addresses for the kernel (depending on the specific machine’s memory map), and kaslr decompresses the image into a random available one during bootup.  This puts the base kernel “somewhere” in a 1 GB-sized area, aligned at 2 MB.

The kernel command line is available in /proc/cmdline.  However, it didn’t have my kaslr customization, which sent me on a quest.  I discovered that Debian/Ubuntu configure a bunch of scripts to produce the final grub configuration file, using /etc/default/grub.d/*.cfg.  These are processed after /etc/default/grub.  There turned out to be a “cfg” file that unconditionally replaced GRUB_CMDLINE_LINUX_DEFAULT, which is where I had put our kaslr flag.  This affected both of our instance types: VirtualBox appeared to have one unintentionally left over from the installer, while AWS had one placed there as part of the cloud image build.

But given that kaslr appears to be default, instead of setting up a local configuration file, I ended up removing the code that was trying in vain to set kaslr.

A Debugging Session

With the company-wide deprecation of Perl, I’ve been rewriting memcache-dynamo in Python 3.  The decision has been made to use a proxy forever.  All of the other options require effort from programmers.  Worse, failures would be inconsequential in dev, but manifest in production.

I’d like to take a moment to walk through the process of debugging this new system, because I built it, I added tests, I felt good about it, and it simply didn’t work with either PHP or Perl clients.

asyncio, new_event_loop, and child watchers

My test suite for memcache-dynamo blocks usage of the global event loop, which was fine, until now. Because aiomcache doesn’t have the “quit” command, and I’m not sure I can duct-tape one in there, I decided to spawn a PHP process (as we’re primarily a PHP shop) to connect-and-quit, exiting 0 on success.

This immediately crashed with an error:

RuntimeError: Cannot add child handler, the child watcher does not have a loop attached

The reason was, the loop didn’t have a child watcher.  Only the subprocess API really cares; everything else just doesn’t run subprocesses, and therefore doesn’t interact with the child watcher, broken or otherwise.

Anyway, the correct way to do things is:

def create_loop():
    asyncio.set_event_loop(None)
    loop = asyncio.new_event_loop()
    asyncio.get_child_watcher().attach_loop(loop)
    return loop

asyncio requires exactly one active/global child watcher, so we don’t jump through any hoops to create a new one.  It wouldn’t meaningfully isolate our tests from the rest of the system.

(Incidentally, the PHP memcached client doesn’t connect to any servers until it must, so the PHP script is really setup + getVersion() + quit(). Without getVersion() to ask for server data, the connection was never made.)

systemd: the house built on sand

Once upon a time, supervisord got the service management done, but I never got the logs stored anywhere sensible.  Eventually, I got tired of being tapped to debug anything that had an obvious error, but where the message was only logged by supervisord.

Thus began a quest for running things through the distribution’s init system, which has given me some experience with upstart and a lot of experience with systemd.  Like most software that reaches success, systemd has not been carefully designed and implemented.  It has only accumulated, organically.

This is nowhere more obvious than in the configuration system.  I can’t just read documentation online, write a .service file, and expect it to work; I have to use the online search to find which man page they hid the relevant directives in, and spin up a VM to read it.  Once I find the directives that apply, it’s obvious that we have an INI system crying out to be a more imperative, stateful, and/or macro-driven language.

Those are related; because the configuration is underpowered, new capabilities require new tweaks.  Consider the number of “boolean, or special string value” options like ProtectHome and ProtectSystem: these were clearly designed as booleans and then extended later.

Because the website doesn’t keep a changelog—everything changes so fast, systemd just has a major version and every release is breaking—it’s not easy to build a cross-platform service definition file that takes advantage of the features systemd offers.  You know, the things that make it unique from other init systems.  The things that were supposed to be selling points.

It’s because everything changes at the whim of the developers.  Stable? Backwards-compatible, at least?  In a fundamental system component?

Big nope from the systemd team.  There are at least a few directives that were superseded, and so it’s impossible to make a portable service description for a service that is otherwise portable. And the lack of past-proofing tells us about future-proofing.  What you write today may simply not run tomorrow.

systemd was supposed to be the obvious, easy choice: in theory, it embraced Linux and cgroups so that administrators could use cgroups to provide isolation without a separate containerization layer.  But in practice, the separate layer is looking ever more like a better choice.

Idea: Type Propagation for Gradual Typing

Regarding this paper recently featured on Reddit, I got to thinking.

Perhaps it’s best to add type information starting in high-level modules; intuitively, having a low-level leaf function (especially one that is frequently called) checking and re-checking its type arguments on every call would certainly be slower than a higher-level function that gets called only a few times throughout the course of a run.

For instance, for a program that does data processing, added type checks in “once per file” functions would have less effect on the execution time than type checks in “once per line” functions.

But maybe we’re missing something, here.  The paper adds complete type information to one module at a time, but does nothing about inter-module calls at each step.  That is, a module may define that it accepts a string argument, but callers in other modules won’t be declaring that they are passing strings until that module has types added.

asyncio not handling SIGCHLD? callback never called?

I wrote a process manager into the new memcache-dynamo.  Maybe I shouldn’t have, but it happened, and I’ve had to fix my bugs.

The problem is, the parent would never notice when the child exited.  Other signals were being handled fine, but the SIGCHLD handler was never called.

This is because, although it says “add” signal handler, the API is really more of a “set” signal handler, replacing any that are already there.  Also, the Unix event loop needs to know about exiting children in order to clean up the subprocess resources, so it sets its own handler.

As it turns out, the correct way to go about this is to use a “child watcher” to allow outside code to react to SIGCHLD.  One should call get_child_watcher and then, on the returned object, add_child_handler. This takes a PID argument, so it can only be done once the child has been created.  At minimum:

proc = await asyncio.create_subprocess_exec(…)
watcher = asyncio.get_child_watcher()
watcher.add_child_handler(proc.pid, onChildExit)

This “onChildExit” is the name of a callback function, which be called with the PID and returncode as arguments.  If more positional arguments were given to add_child_handler, then those will also be passed to the callback when it is called.

The other signals can be handled in the usual manner, but SIGCHLD is special this way.

(This applies to Unix/macOS only, as Windows doesn’t have POSIX signals. Maybe the shiny new subsystem does, but in general, it doesn’t.)

I still don't understand Python packaging

Since we last talked about this subject, I've tried to use pipenv with PIPENV_VENV_IN_PROJECT=1 for the project in question. Everything was going pretty well, and then… updates!

I'm using a Homebrew-installed version of Python to test, because it's easier and faster on localhost, and the available Python version was upgraded from 3.6 to 3.7. As usual, I ran brew cleanup -s so the Python 3.6 installation is gone.

It turns out that my python_version = "3.6" line doesn't do what I want—pipenv will be unable to do anything because that binary no longer exists—and I haven't been able to figure out a way to ask Pipenv to use "3.6 or above" to both:

1. Express the "minimum version: 3.6" requirement
2. Allow 3.7 to satisfy the requirement

pipenv seems pretty happy to use the system Python when given a version requirement of ">=3.6" but it's also acting like that's a warning only. pipenv check doesn't like this solution, and it's not clear that a system Python 3.5 would cause it to fail as desired.

In PHP, this is just not that hard. We put "php": "^7.1.3" in our composer.json file, and it will install on PHP >=7.1.3,<8.*. It will fail on <7.1.3 or on 8.x or on an 8.0 development version. It's all understood and supported by the tool.

So anyway: right now, we have a deployment process which is more or less "read the internet; build in place for production; swap symlink to make updated code available to the web server."

The end goal is to move the production deployment process to "extract a tarball; swap symlink." To do this, we need to create the tarball with "read the internet; build in place; roll into tarball" prior. And AFAICT, building a virtualenv into a tarball will package everything successfully, similar to Composer, but it will also bake in all the absolute paths to the build process's Python installation.

Pipfile and Pipfile.lock look like what I want (deterministic dependency selection in the build stage, and with the environment variable, in-project vendoring of those dependencies) but it seems like it's fundamentally built on virtualenv, which seems to be a thing that I don't want. I obviously want dependencies like aiobotocore vendored, but I don't necessarily want "the python binary" and everything at that layer. I especially don't want any symlinks pointing outside the build root to be put into the tarball.

Overall, I think pipenv is trying to solve my problem? But it has dragged in virtualenv to do it, which "vendors too much stuff," and it has never been clear to me what benefit I'm supposed to get from having a bloated virtualenv. And also, virtualenv doesn't fully support relocatable environments, which is another problem to overcome. In the past, it has been fairly benign, but now it has turned adversarial.

(We have the technical capability to make the build server and the production server match, file path for file path, exactly. But my devops-senses tell me that tightly coupling these things is a poor management decision, which seems to imply poor design on the part of virtualenv at least. And that contaminates everything built on top of it.)

Configuration as a Point of Failure

Recently, I happened to restore service for our /giphy Slack command.  I originally had things set up poorly, so we needed a ProxyPass configured in every virtual host that wanted to run PHP.  That changed at some point.

I updated all of the configurations in git, of course.

But our giphy endpoint is a separate project, meant to be shared with the world, which therefore doesn’t follow our standard layout.  Its config is not in its git dir; I missed it in the update pass.

I had also just dealt with all the config file merging and changes in the Ubuntu 18.04.1 LTS update.

It really got me to thinking: anything configurable is a point of failure. Whether it’s a process/knowledge failure like with our giphy endpoint, or merge conflicts in *.lock or webpack.config.js files, or a system package changing its configuration semantics or defaults between versions:

The presence of configuration allows broken configurations.

We should try to avoid unnecessary configurability in a system.  This is also very difficult to achieve—a system that only has defaults absolutely needs to have correct, working defaults.  The problem only gets harder in open source, where a project often serves many diverse users.

Also, one of my greatest weaknesses is throwing in easy-to-build options “just in case.”

Finally, it’s bad practice to bake credentials into a git repository, but how shall they be provided without configuration?  EC2 has IAM Roles, but they don’t work in VirtualBox… it really does seem like some configuration is a necessary evil.

Version Locking

We have a few lovely workarounds, arrived at through a small measure of blood, sweat, tears, and research, embedded in our cpanfile these days.  Here are my two favorites:

requires 'Email::MIME', '<= 1.928';
    # 1.929+ screws up Content-Type sometimes
requires 'DateTime::Format::Strptime', '<= 1.57';
    # DynamoDB v0.1.16 (current) buggy with rewritten Strptime

Nobody wants to come back and check these, to see if later releases of the packages work again.  (It’s possible I’m the only one who remembers these hacks are here.)

Email::MIME was especially nasty because it only failed sometimes, and now, how do I prove that the fixes (which may have occurred in 1.931 or 1.933) actually solve the problem?  I can’t prove the negative in test, and I can’t trust the package in production.

As for the other fun bug, “DynamoDB v0.1.16” refers to the package whose full name is Net::Amazon::DynamoDB and released on November 6th, 2012.  I think this one was detectable at install/test time, but it was still No Fun.  A lot of work was expended in finding out its dependency changed, and I’m not excited to redo it all to find out if 1.67 (or 1.63) fixed the issues.

Especially since use of Perl was deprecated company-wide, and we want to get it all ported to a new language.

Editor’s Notes: this was another old draft.

Since it was written, we accidentally introduced an updated version of Email::MIME into production, and it still failed.  We fixed the bug that allowed the update to occur; clearly, upstream’s process is broken.  I don’t think we could be the only people hit, across multiple years and multiple versions of the entire software stack.

I’m not entirely sure what happened with Net::Amazon::DynamoDB—but we may have been able to use a newer version with it.

Python, virtualenv, pipenv

I heard (via LWN) about some discussion about Python and virtualenvs.  I'm bad at compressing thoughts enough to both fit Twitter and make sense at the same time, so I want to cover a bit about my recent experiences here.

I'm writing some in-house software (a new version of memcache-dynamo, targeting Python 3.6+ instead of Perl) and I would like to deploy this as, essentially, a tarball.  I want to build in advance and publish an artifact with a minimum amount of surrounding scripts at deploy time.

The thing is, the Python community seems to have drifted away from being able to run software without a complex installation system that involves running arbitrary Python code.  I can see the value in tools like tox and pipenv—for people who want to distribute code to others.  But that's not what I want to do; I want to distribute pre-built code to myself, and as such, "execute from source" has always been my approach.

[Update 2018-09-06: I published another post with further thoughts on this problem.]

Stream Programming, Without Streams

Editor’s note: Following is a brief draft from two years ago.  I’m cleaning out the backlog and faking HUGE POST COUNTS!! for 2018, I guess.

I wrote before that I don't “use” stream programming, but I've come to realize that it's still how I think of algorithms and for loops.  Input enters at the top, gets transformed in the middle, and yields output at the bottom.

It's like I look at a foreach loop as an in-line lambda function.  The concept may not be explicitly named, and the composition of steps built into lower-level control flow… but inside my head, it's still a “sequence of operations” on a “stream of data.”

There doesn't seem to be much benefit to building up “real” streams in a language that doesn't have them either built-in or in the standard library. It creates another layer where the things that have been built can only interoperate with themselves, and a series of transformations can no longer share state.  And, a PHP array can be passed to any array_* function, where (last I even checked) our handmade streams or Iterators cannot.

Get and Set

Editor’s note: I found this two-year-old draft hanging around with all my other posts on my PC.  What follows is the original post, since it appears to have aged fairly well, although it could use a conclusion…

It happens that having to call methods of a class in a particular order—unless there’s an obvious, hard dependency like “don’t invoke Email::send() until all the headers and body have been added”—leads to fragility.  There’s no hint from the IDE about the dependencies or ordering.

Worse, if a class has a lot of methods that must be called after setting some crucial data, every one of those methods has to check that the necessary data is set.  It’s better to pass that into the constructor, and have the created instance always contain that data.

Given a class that wants to maintain a valid state for all its methods at any given time, how do setters fit into that class?  It needs to provide not individual setters, but higher-level methods that change “all” relevant state at once.

So, I’m starting to see get and set methods as an anti-pattern.

Is Elegance Achievable?

Editor’s note: This is a three-year-old draft that I found.  I’m bad at coming back to things. Regardless, what follows is the post in its near-original form; I reworked some “recently” wording because it is certainly not recent anymore.

The best solutions are “clean,” “elegant,” and “simple,” right?  But what does that mean, and are those things even achievable?

Why we have a memcached-dynamo proxy

Previously, I mentioned that we have a proxy that speaks the memcached protocol to clients and stores data in dynamodb, “for reasons.”  Today, it’s time to talk about the architecture, and those reasons.

Ubuntu Bionic OVA size optimization (reduction)

For Ubuntu 18.04 LTS “Bionic Beaver,” Canonical has released a new installer on the server image, the one with live in the ISO’s filename.  (The old installer, with full feature support, is available as an alternative download.) The new installer is named Subiquity.

We have a set of scripts that take “an Ubuntu install” and turn it into a VM image pre-installed with PHP, Perl, and dependencies for our code.  This creates a “lite” VM build.  (We used to have a “full” build that included all our git repos and their vendored code, but that grew large.  We’ve moved to mounting the code to be run into the guest.)

Our standard build scripts produced an 850 MB OVA file when starting from the new Subiquity-based installer, compared to around 500 MB on previous releases.  The difference?  Previous releases used the “Install a minimal virtual machine” mode option in the old installer.  In fact, after checking I had used zerofree correctly, using the minimal VM mode in the alternate installer for 18.04 produced a 506 MB OVA.

But maybe that alternate install won’t be around forever, and I should fix the bloat sooner rather than later.  Where do those 344 MB come from, and how do we find them?

MySQL data copying optimizations

Recently at work, I optimized a script.  It copies data from a live system to a demo system, anonymizing it along the way.  We used GUIDs as contract and customer identifiers, but in the pursuit of something shorter and more URL-friendly, we created a “V2 format.”

The V2 format secretly hides a client ID inside; the demo importer used to copy the IDs and not worry too much, but now it has to change them all. Making that change took it from running in about four minutes to 15 minutes against a local MySQL instance hosted in the same virtual machine.

Unfortunately, it took 75 minutes in production, and the ID rewriting it was doing was causing our infrastructure to raise alerts about data stability, response times, and IOPS rate limiting while a rebuild was running.

Update 2018-05-18: the production run takes around 9 minutes and 3 seconds after optimization, with far less disruption. I got the local instance rebuild down to 2 minutes and 15 seconds.  I want to share a bit about how I did it.

Effectively Using Future in Perl

I’ve been working a lot with the Future library and IO::Async in Perl recently.  There was a bug in our memcache/dynamo proxy, so I ended up doing a lot of investigation about Futures in order to simulate the bug and verify the fix.  So, I want to talk about how Futures work and why it was so difficult for me.

PHP 7.2 exposed our bad code

We upgraded to PHP 7.2, and then something horrible happened.

A “thanks for submitting client $x!” message was distributed not only to the agent who did the submission, but to all agents registered in the system.  A lot of drama ensued as confused messages started coming in, and the account rep forwarded the message to basically everyone in the company saying “what happened!?”

We stared at the code, but it was so simple that it couldn’t possibly be wrong, and besides, it hadn’t changed in months.  But the switch over from PHP 7.1 to 7.2 had happened mere hours before this event.

Commit Logs vs. External Systems

There’s a couple of schools of thought on commit logs: should you make them detailed, or should they be little more than a pointer to some sort of external pull request, ticketing, or ALM system?

When people have external systems, the basic problem they face is that those systems seem to be redundant with the information in the commit log.  Why write the same thing twice, when the commit log can just point to the ticket, which undoubtedly has a lot of rationale and reasoning already written into it?

But in my experience, it’s far more useful to have good commit log messages. When hunting in the code for a problem, the commit log is right there and available to tools like git log --grep, which can also print the diff alongside the log messages.

And while some tools like GitHub pull requests offer line-by-line commentary on diffs, the other interesting thing about commit logs is that they’ve proven much more resilient over time.  Our ticket tracking has migrated from my boss’s inbox, to Lighthouse, to a bespoke ticketing system… that we integrated into our support team’s workflow as well, which has become something we want to split back out.  And then we might replace the “remaining half” of the bespoke system with some off-the-shelf solution.

Meanwhile, our commit logs have been preserved, even across a move from subversion to git, so those go back to the point in time where the founders realized they should set it up.  But the references to “Lighthouse” have been broken for years, and if we succeed in killing a huge pile of custom code nobody wants to maintain, all those “ticket #16384” references are also going to be useless.

But the commit messages will live on, in a way that ticketing systems have not.  And so, I don’t really trust that we’d stick to GitHub and have their issue and pull request systems available for every change, forever.

Aside from that, I think a succinct summary of the ticket makes a good commit message.  I try to avoid repeating all the ticket babble, status updates, and dead ends in the commit:

Contoso requested that their cancellations be calculated on a 90-day cliff instead of days remaining in term.  Add cancel_type=cliff and cancel_cliff_days=90 to the client settings.  Ticket 18444.

This gives the big-picture outlook on what happened and why, and lets the diff speak for itself on whether the change suits the intention.  If there are questions about whether the true intention was understood, then the ticket is still linked, so it can be examined in further detail.

Supposedly Readable Code

There are two hard problems in Computer Science: cache invalidation, and naming things. —Phil Karlton

The problem with long-term evolution of a codebase is that the compatibility requirements end up creating constraints on the design.  Constraints that may be felt for a decade, depending on how history unfolds.

What my company now refers to as “Reserve,” which seems to be fairly universally understood by our clients and b2b partner soup, was initially called “Cost.” That was replaced by “Escrow” because the “Fee” is also a cost, just a different kind.  But escrow didn’t sit right among people who make deals and sign contracts all day, because it wasn’t necessarily being held by a third party.  (Depending on what kind of hash the salesmen made of it, it was held by either the first or second party.)

The point is, before coming up with a universally acceptable term, we needed some term, so Cost and Escrow got baked into the code and database structure to a certain extent.  Along with Reserve.

When someone new comes along, their first instinct is to complain about how “confusing” it is.  And I can see that.  It’s a single concept going by three names.

You get used to it, though.  As you work with it repeatedly, the concept gets compressed in your brain.  Here it’s Cost, there it’s Reserve, it’s the same thing in both places.

But, getting used to it is a symptom of the “ignoring weak signals” problem.  (Is there a better name for that?  “Normalization of deviance” is heavy, too.) If we hired enough people, it would be a clear source of suckage that we’d really want to fix.

On the other hand, I’d love to do a cost-benefit analysis and find out just how important it really is to get fixed.  Unfortunately, that depends on measuring the “loss of productivity” from the multiple names, and measuring productivity to begin with is difficult.  I think the experimental design would also require fixing the problem to get a decent measurement on the single-name productivity.

Therefore, it ends up easy to ignore weak signals because they’re weak, and we don’t know what we’re missing by doing so.

Another justification for ignoring them is that we can’t act on them all.  We have to prioritize.  After all, developers tend to be disagreeable.  I know—whenever I’m making a “quick bugfix” in some code I don’t own, I have to suppress strong urges to “fix” all the things, from naming to consistency to style conventions to getting rid of the variable in $x = thing(); return $x;.  I’m pretty sure the rest of the team does the same for my code.

The funny thing is, I bet each one of us on the team thinks we write the most readable code.  I’ve been doing this longer than anyone, and I put a lot of effort into it.  I standardized my table alias names, and I wish everyone else followed that, because the code was a lot easier for me to read when “clients” was just “C” and not a mix of “C”, “c”, “cl”, or “cli” depending on which SQL statement one happens to be reading.

Between synonyms and the irritating slipperiness of natural language, then—is there such a thing as “readable code?”  There’s certainly code that’s been deliberately obfuscated, but barring that: can we measure code readability? Or is it just doomed to be, “my latest code is best code,” forever?

Linux’s Hazing Ritual

What happens the instant a user wants to install some non-graphical software on their system?  What if they want to tweak something that happens to be considered an “advanced” configuration setting?

All the advice for these situations begins, “Go to a terminal, and type sudo ...”

Whatever is being solved is a common enough problem that there’s advice for it all over the internet, but it’s not common enough that anyone can build a graphical interface for it?

I think about this every time I have to install the VirtualBox Guest Additions.  The GUI package managers have shifted from showing all software on the system to only showing “graphical” software, insisting there are no results for packages like build-essential.  The package is there, it’s just being “helpfully” hidden from me, even when asking by its exact name.

So instead, I am expected to go off to the command line, and either paste a string from a web page, or carefully type it out in its entirety, with exact spelling.

The idea of exploiting web pages comes up from time to time—styling allows additional text to copy but not display—but the general response from the community is “Don’t do that, then.”  Paste it into vim if you’re not sure? How does a user even know to do that in the first place? Can anyone, including expert users, ever be “sure” about it?

I understand why directions are given as terminal commands.  They’re easy to write, and if the copy/paste mechanism is working and trustworthy, they’re efficient to execute that way.

But the underlying cultural problem—that everyone must be able to use the command line as a basic prerequisite to managing their system—is an obstacle to creating a more user-friendly design.  That in turn becomes an obstacle to getting widespread usage.

People tell themselves it doesn’t matter, but consider how hard Microsoft worked on getting “a tablet PC” going for a decade before Apple showed up with the iPad.  The tech circles predicted a quick death, just like they did for the iPod and perhaps the iPhone, but it defined the segment instead.

What was the difference?  The iPad didn’t require stabbing at mouse-sized bits with a delicate stylus.  It was designed around the finger, with no fallback, producing internal pressure to make sure the finger interface worked, worked well, and worked for everyone.

In Linux, the terminal exists by default and remains the only way to do some things, making it inescapable.