The Image's Revenge

Long ago, some programming languages supported image files, which were the running code and data, placed in a file for reloading later.  Wikipedia is vague on the details, but I believe the major benefit was load speed—an image could restore already-compiled code from a single, more-linear file instead of needing to read and evaluate the sources again.

In modern DevOps, we use images all the time.  We even call them images.  The only difference is that these images are built at the level of the filesystem and designed for an external OS/hypervisor to run.  Or if you're willing to give up some of the "can run anything" capability, they can be built without the filesystem.  Just your app and a hypervisor.

TLS Cipher Suites in 2013

Things I learned digging around the RFCs and testing with the ssllabs server test:

1. SSLv3/TLSv1 ciphers listed in the original specifications with forward secrecy are all DES (very weak) or 3DES (less weak).
2. AES ciphers for TLSv1 were originally specified in RFC 3268 in 2002, and did not make it into XP's SChannel implementation.  I believe this is why they aren't present in IE8/XP.  If the link is trustworthy, the only ciphers available on IE/XP are RC4 and 3DES, neither of which can be considered excellent choices today.
3. IE/XP does not truly offer forward secrecy because it doesn't support RSA modes with DHE, only DSS which has a 1024-bit limit; much of the TLS infrastructure has moved to require 2048-bit keys recently as 1024 is no longer considered adequate.
4. TLS specifications require one mandatory cipher, presumably to ensure minimum interoperability.  In TLSv1.2 (RFC 5246, published in 2008), that cipher is AES128-SHA.  Earlier versions including TLSv1 and TLSv1.1 listed 3DES-SHA.
5. Camellia cipher suites were defined in RFC 4132 in 2005.  That RFC has been superseded by RFC 5932 in 2010; the latter uses the SHA-2 hash family instead of SHA-1.
6. Camellia itself is expected to be of similar security to AES; current attacks on AES are better, but it's likely that AES is more widely studied.
7. SHA-2 hash family MACs were introduced in TLS v1.2.  It is believed that the older MD5/SHA-1 functions are still secure as used in TLS.  To the best of my knowledge, SHA-2 suites are only available when using TLSv1.2.
8. ECDHE suites are defined RFC 4492, from 2006; including a ClientHello extension to indicate support.
9. ECDHE is not required (though it is listed) in TLSv1.2, nor is usage of TLSv1.2 a requirement for using ECDHE.  Recent versions of Firefox support ECDHE suites in conjunction with TLSv1.  Older versions—Firefox 21/Fedora 19 for instance—do not.
10. Some browsers prefer alleged key length over other considerations, and prioritize 3DES with its "168" bit key over AES128.  It's actually designated by NIST as providing 80 bits of security, though it could be providing up to 112 bits.
11. The default cipher policy provided by AWS Elastic Load Balancers enables only SSLv3/TLSv1 and an extremely limited set of ciphers: AES128 and 256, 3DES, and RC4, with no forward secrecy suites.  (Suites are configurable in the AWS Console, but not cipher preference order, nor some renegotiation options.  Renegotiation settings are implicit in the choice of supporting, or not, the more-recent TLS versions.)

The Lavabit Problem: the Universal TLS Key

Lavabit refused to hand over the key to the whole kingdom for the U.S. Government to ostensibly capture the traffic of one user.  The judge ruled in the prosecution's favor in the resulting court case:

Judge Claude Hilton said that it was effectively Levison's fault that sites have only a single private SSL key.

Ars doesn't evaluate the truth of that, but technically, it could be accurate.  The Perfect Forward Secrecy modes of TLS—DHE and ECDHE—effectively generate a key per connection, authenticating the parameters used to create it with the server's certificate (which is in turn authenticated by the CA system.)  However, if Lavabit's servers were set up to allow RSA-only modes, then that configuration introduces the weakness.  Under RSA, the server's private key becomes the master private key, capable of decrypting any connection traffic in those modes, because that private key is used to encrypt all the traffic.

Go Slices and the Hash API

I finished my first little program in Go months ago.  One of the mysteries was "Why does Hash.Sum take a byte slice if it doesn't write into it?"

Well, the question is broken.  Let's look at some code that attempts to hash a file's content and save the result into a file-info structure, but which doesn't actually work:

sumbuf := make([]byte, hsh.Size())
hsh.Sum(sumbuf)
fi.hash = sumbuf

What happens?  This allocates a byte slice with both len and cap equal to the output size of the underlying hash, filled with 0x00 bytes.  It then passes into hsh.Sum (which, in this particular case, was crypto/sha512) which copies the internal state, finalizes the hash, then wraps everything up with return append(in, digest[:size]...).

Since sumbuf didn't have any room (make(T, n) is equivalent to make(T, n, n)) the append didn't have anywhere to put it.  So it did what it had to do, allocating a new backing array of sufficient size, then copying both in (aka sumbuf) and the elements from the digest slice into it.  The expanded slice backed by this new array then gets returned... only to be discarded by me.  fi.hash ends up being the slice I allocated containing all zero, which made the program think all files were duplicates.  Oops!

What works?

sumbuf := make([]byte, 0, hsh.Size())
fi.hash = hsh.Sum(sumbuf)

First, the return value must be assigned: the original slice's backing array is shared, but append creates a new slice with a larger length to hold the data.  fmt.Printf's %p will show that fi.hash and sumbuf live at the same address, and they have the same capacity as well, but the len of each differs.

Second, if we don't want append to allocate a fresh array, we need a slice with enough free space after its length to hold the data it wants to copy there.  A slice with nothing in it (yet) and the capacity of the hash length is precisely the smallest thing to fit this description.

Now that we have some working code, let's reflect on the point of passing in a slice to Hash.Sum in the first place.  The goal is to avoid allocating inside Sum, if the caller has space for it.  But Sum already allocates and copies a slice: it needs to finalize the hash for Sum, without disturbing the original state so that writers can still add data.  By working in a temporary buffer on the stack and copying at the end, it doesn't make a discrete allocation on the heap, but it still needs to ask append to copy it.

Why not begin d := append(in, _____) and then work inside the caller's buffer directly?  My guess is that working in a private buffer prevents Sum from leaving partial state visible to the rest of the program.  I don't know if it is, but I would not be surprised if append is atomic from the point of view of running goroutines, and clearly Sum needs to allocate in order to be re-entrant.

The Forgotten Gem: LDAP

I keep wanting a "configuration lookup service" with the following properties:

• Optimized for a read-heavy workload (config stays the same for months).
• Hierarchical structure (by data center, by server group, by specific host if necessary).
• Queryable at any level.

From what I understand, it looks like this LDAP was practically designed for this.  But I've never seen anyone use it for anything but user accounts, which seems like kind of a waste.

Pen and Paper

One of the surprises I had when I wrote some golang code was that static typing was a real bear to wrestle with again.  Once I had the type errors sorted out, though, programs ran fairly well.  By the time I could get it past the compiler, I had been forced to think about it deeply enough that there were far fewer bugs in the final result.
Of course, it's not magic.  My code has still deadlocked.

The mindset that Go enforces, by virtue of checking the types, has led me to try being more careful in my regular work.  I now try to make notes on everything that I need to come back to as I'm exploring a problem (such as: "what code interacts with this variable?") so that I can check my plan against all relevant cases.  Likewise, a multi-step change will end up with a few bullet points like...

EmailBounce (#3351)

• set in SES bounce processor
• clear when email changes
• react in notifier code

This also helps in event of interruptions, such as realizing the office manager's radio is playing the same annoying pop songs again today, for the 110th day in a row.  There's a list of stuff to help re-establish concentration, knowing I'm not forgetting anything.

For complex issues and solutions, doing the design on paper has an extra benefit.  Paper encourages brevity.  There's only so much room within the confines of a page, and it takes a while to write.  Consequently, pseudo code tends to stay pseudo code, at an appropriate level of abstraction, when writing out steps.  It's the exact opposite of the temptation to switch to writing the real code when trying to write pseudo code in a text editor. It looks like a convenient shortcut to combine the two coding steps, but then the faults in the overall plan don't get noticed until (often) most of the way through—then a random amount of already-written code is made irrelevant or wrong.

Modifying half-implemented approach A to become approach B (hopefully not with the same last-lap change of course) adds a lot of mental state to juggle.  Now there are four states complected: original code, usable code from A, unusable but not yet changed code from A, and code from B.  It can be quite useful, and a bit simpler, to work out some of those false starts in advance, on paper.  Then there's both a record of where my thoughts have been, and a clean body of current code to implement the final result.

As useful as paper can be, there are also times when it's at a disadvantage.  For picking apart tricky, complex control flow with nested ifs, repeated conditionals, and the like, I find it easiest to copy the code into a new window and then start deleting lines and replacing them with pseudo-code comments.  Often, hundreds of lines can be reduced to a single window, which makes it trivial to get a higher-level overview of what's being accomplished.

Paper's other major disadvantage is that it lacks undo.  I've learned by the number of cross-outs on the page just how frequently I rename things as I get a better sense of the problem and what those parts represent in the whole.  (I have even been known to choose a name, cross it out in favor of an alternate, then cross out the alternate to restore the original.)

Overall, though, for the appropriate tasks, it's been a great advantage these past few months to put more effort into paper and less into backtracking in vim.

Moose Alternatives

I once wrote about Mouse when I had discovered it.  Amongst all this recent research, I have also discovered Moo.

Moo strives to be even lighter and faster to load than Mouse, which was already Moose without antlers.

I swear, next year I'm going to find an even lighter module called "Mc".

More FastCGI: Apache 2.4, PHP-FPM, PSGI, and hot deployment

Driven by mod_fcgid's failings at graceful restart and preforking, I've been looking hard for alternatives.

tl;dr

External FCGI is best done via mod_fastcgi on Apache prior to 2.4, or mod_proxy_fcgi on 2.4.  mod_proxy in general is super awesome on 2.4.

php-fpm ships with an init script and reloads gracefully via SIGUSR2, so that's all you really need there.

For Perl, gracefully restarting an FCGI app is difficult, so the better approach for PSGI apps is to run them in an HTTP server with Server::Starter support (e.g. Starman or others).

Pre-compiling Twig Templates

To be able to use option 'auto_reload'=>false in production with Twig, I need to be able to force a recompilation.  The easy way would be to clear the cache directory in the deployer when pushing an update, but if I'm micro-optimizing, why not ship built templates as part of the distribution?

The next obvious question is, "How do I make twig compile templates without rendering them?" and it turns out there's a simple answer.  The long form $twig->loadTemplate("name.twig")->render($vars) is the secret: loadTemplate writes to the cache if caching is enabled when the template is loaded.

Conceptually, then, it's dead simple: configure Twig how you normally would, list out all your template files, and call $twig->loadTemplate() on each of them.  I believe the minimal solution on POSIX platforms would look like this:

Smarty vs. Twig (a benchmark done poorly)

Note added 14 Feb 2014: I ended up settling on Twig without too much more rigorous testing.  This used to be titled "Part 1" but there is, and is never expected to be, a Part 2.  Now back to your original post....

I'm pretty cranky about what my templates do (I have been known to roll my own), so I have a small list of must-have features.  Listing in order by increasing likelihood that a template engine satisfies them:

• Extensible functions for adding convenience from the host environment
• Built-in functions for date formatting
• No annoying <? ?> all over the place
• Auto-escaping of variables when building HTML
• Conditionals, expressions, and loops (if-elseif-else chains, boolean logic)
• Access to array elements and object properties

That's pretty much it: I'm willing to let "presentation logic" get pretty heavyweight, as long as it involves read-only access to the variables given to the template.  Templates should in no way mutate their model or context (excepting functions they initiate, like cycle, for hopefully obvious reasons.)

With that, it looks like the contenders are Smarty and Twig, currently at versions 3.1.14 and 1.13.2, respectively.  How do they compare in a benchmark showdown?

Working from Home

Since I've radically changed my work VM configuration, I needed to change my home PC in order to connect properly to the new setup.

Before we dive in, a brief review of the old setup: the host at work, which we will call `audia`, ran a VirtualBox guest named `dev`.  dev had its own set of configs and configuration knobs so that I could write 127.0.0.1 dev.audia.pc dev into /etc/hosts, point Firefox to dev:10080, and code running on dev would be able to recognize "this is development," not require TLS, and know that http://dev:10080/ is the URI root of the app.  I configured VirtualBox's port forwarding to listen on 0.0.0.0 from audia, and when I connected to the VPN, my home computer (call it `starlet`) had its own /etc/hosts listing that used the internal IP of audia for dev.  The same URI from home would then get connected via port-forwarding to the dev VM.

This all changed when I wanted to test a SAN certificate and some redirection rules.  I didn't want certificate mismatch errors in Firefox, and I needed dev to think it was the production server so that it would use the production ruleset.  That became the underlying justification for devproxy.  After that point, dev had the production certificates, hostnames, and rules.  It had a host-only network added, which audia now used, and I took out the port forward.  (This was the problem behind getting Lost in the Complexity.)

But without the port forward, starlet can't reach dev anymore, only audia.  Of course, I came up with a solution:

(Despite what may be implied, the VPN isn't connected to starlet.  I just didn't feel like redrawing the whole tube.)

SNS driven deployment (architectural overview)

3 Sep 2015: a note about how we're using php-fpm instead of suEXEC now has been published here.  Now back to your original post...

One of the problems I faced with getting Auto Scaling up and running was, "If I can't just ssh into the one live server and issue a git pull by hand, how do I push code out?"

Normal people would use capistrano, or Paste Deploy, or something, extended to access their auto scaling group to locate the hosts to use.  Or build on Heroku or Elastic Beanstalk to begin with.  I, however, am clearly not normal.

Bootstrapping an EC2 Instance, 2013 Edition

In the interests of "not implying old posts on my blog are still state-of-the-art", here's a quick overview of how I manage EC2 instances at work, modernized.

Controlling Internal Iteration

Once upon a time, I read this post on iterators.  Yesterday, a solution struck me for absolutely no reason.

How do you stop internal iteration without non-local returns?  Channels!

TLS: all those DH modes and PFS

Advice is easy to come by, explanation less so.  What are all these acronyms, and what makes them secure, or not?

devproxy

Today, I'm pleased to announce the release of devproxy!  It's an HTTP proxy server written in Go (atop github.com/elazarl/goproxy) for QA/testing purposes.  It intercepts requests directed to your production servers and transparently connects them to a staging server of your choice—including HTTPS (CONNECT) redirection.

This lets your staging server perfectly mirror production, including the hostnames and certificates in use, without needing to elevate permissions to edit /etc/hosts every time you want to switch between production and staging.  Instead, you can switch the proxy on/off in your browser; I use FoxyProxy.

I'd like to thank Elazar not only for writing goproxy (it made my life a lot easier) but also for modifying it to support what I needed to do in devproxy.  I'd also like to thank my boss for letting me release this as open source.

A Look at Failing Polymorphism

First, a brief recap of Steve Yegge's "When Polymorphism Fails" in case it moves again. Let's skip to the part where he builds an example in which polymorphism doesn't help the programmer, using a user-extensible online game as the premise.  A user wants to add an OpinionatedElf monster; how does she do it?

Let's say the OpinionatedElf's sole purpose in life is to proclaim whether it likes other monsters or not. It sits on your shoulder, and whenever you run into, say, an Orc, it screams bloodthirstily: "I hate Orcs!!! Aaaaaargh!!!" (This, incidentally, is how I feel about C++.)

The polymorphic approach to this problem is simple: go through every one of your 150 monsters and add a doesMrOpinionatedElfHateYou() method.

Gosh. That sounds insanely stupid.

Yegge then compares that to an alternative, implemented within the OpinionatedElf (who is the one doing the judgement in the first place):

public boolean doesElfLikeIt ( Monster mon )
{
    if ( mon instanceof Orc ) { return false; }
    if ( mon instanceof Elf ) { return true; }
    ... <repeat 150 times>
}

There's also the viable-in-Ruby approach of opening all the classes and adding the doesElfLikeIt method to each of the 150 monsters. But still, there's a lingering problem with all of the approaches so far:

If someone adds a Gremlin, your elf will be stuck screaming something like "Gosh, what's THAT?" until you can update his code to include a case for gremlins.

I ran into my own instance of this problem recently.

Tor for VirtualBox Guests (idea only; no code)

To give a guest only one network card with host-only networking, yet still let it access the Internet, we can let it connect to an HTTP proxy running on the host.  If this proxy is polipo, we can configure it to connect to tor’s SOCKS server as its upstream:

Then what happens?  Any and all Internet traffic from the guest VM is delivered via Tor.  Since the guest doesn't have Internet access of its own, any software which doesn't cooperate with the proxy cannot communicate.  Although malware on a compromised guest could still exfiltrate data, it hides the host's true external IP address from the malware.  (Assuming, dangerously, no security bugs in polipo nor VirtualBox that would allow a compromise of the host.)

I said that first, but maybe not very clearly, on twitter.

But… given an appropriate proxy, traffic can be forwarded over any transport.  A proxy could accept data from the guest and transmit it via VPN.  On the other hand, building a VPN client into VirtualBox to offer a VPN network type would let a client connect to a VPN without necessarily allowing other host processes access to it, nor requiring the VPN to be mediated by an additional (dual-homed) guest.

Polipo doesn't have to use tor as a backend, either; it's also perfectly capable of forwarding using ssh's SOCKS proxy.  (This is known as "dynamic tunnel mode" in some clients.)  Compared to the amount of software and configuration needed to set up the average VPN, ssh is just as secure and much easier to get running.

Sending traffic via proxy is an effective way to apply further modifications to the destination stream, without needing the cooperation of software connecting to the proxy.

Random Idea: BBU SSD

Given that RAM doesn't seem to be that expensive anymore (there are plenty of choices for 2GB DDR3 sticks around $20 at NewEgg right now), why not put a bit more onboard an SSD along with a small rechargeable battery?

The extended RAM would be dedicated to a large page cache, which would do its best to hold frequently-written data (to extend the flash life by avoiding writes to it for as long as possible.)  A landing zone the size of this cache would be reserved in the NAND, and in event of sudden power loss, the pages pending in the DRAM would be dumped into the landing zone under battery power.

Presumably, the 400+ MB/s that current SSDs quote for sequential writes involve the overhead of the OS, host interface, and wear-leveling scheme, and represents a lower bound on performance of the panic save.  On a 2A drive writing 2.00 GB, also intended to be conservative numbers, 5.12 seconds of power is required, for consumption of 170 mAh.

(If the manufacturer quoted their transfer speed in SI and I'm actually writing GiB of data, those numbers change to 382 MiB/s yielding 5.37 seconds of transfer time and 179 mAh of power.  No big deal.)

Lost in the Complexity

I was working on a post about VirtualBox’s networking capabilities and how none of the modes provided for what I wanted out-of-the-box.  But the tirade was interrupted by a simple thought: VirtualBox allows up to four virtual network cards per guest.  I could simply configure a guest with two of them—one connected to NAT for reaching the outside world, and the other connected to host-only networking so I could reach it without having to set up port mapping rules.  (Bridge mode is unsuitable because I want the machine to be externally invisible; also, the LAN is DHCP and I want the machine to have a static IP without involving anyone else.)

That turned out to work, by the way.  The machine still has access to the Internet but also nmap against its static (host-only) IP can see all the open services at their native, unmapped ports.

In the moment I realized that a dual-card configuration would work, I was also struck by the amount of time I had spent coming up with a single-card solution to the external access problem… only to have it turn out to be the wrong problem to be solving.  Or, since it wasn’t technically infeasible, a problem made over-complex by the accidental assumption of a single network.

This illuminates one of the main problems of programming: the tension between breadth and depth.  To determine if a plan is technically feasible, one needs to dive deeply into all the details and try to fit the final product together in their mind.  But, the feasibility alone is not a fitness function.  One must avoid getting so lost in the details that this becomes the only approach visible, and actively “back out” to search for hidden assumptions and gratuitous decoupling.

As a younger programmer, I spent some happy hours working on database abstraction layers, and the projects never changed database.  These were all in-house projects for in-house purposes where all available (and foreseeable) DBA knowledge was built on MySQL.  Building systems that “could” be changed to other ANSI compliant systems was both irrelevant and unnecessarily limiting.  I didn’t allow any MySQL specific optimizations, so that all queries could be represented faithfully on any DBMS.

However, the Serendipity weblog system can run on MySQL or Postgres and for them, it isn’t gratuitous.  Their software is externally distributed and not every admin using the software will necessarily be either conversant with or favorable toward MySQL.  Thus, Serendipity’s user base becomes larger if it has support for other engines.  The same decoupling, but no longer gratuitous, and they probably implemented it better than me anyway.

When the VirtualBox Network Quest began, I made the assumption that I wanted one network, and because that assumption was invisible to me, I chased the details down to completion before spotting the alternative.

OTOH, thinking so deeply about it led to a couple of other interesting observations, but they'll have to wait for another post.

PHP unpack()

In Perl, unpack is pretty easy to use.  You unpack something with the format string used to pack it, and you get a list of the values that were packed.  I'm not sure the historical reasoning behind PHP's version of unpack, but they certainly made it as horrible as it could possibly be.

To get Perl-like behavior, the simplest path appears to be:

<?php list(...) = array_values(unpack("vlen/Vcrc", $header)); ?>

Instead of the simple "vV" it would be in Perl, you give each format code a unique name and separate them with slashes.  You have to provide a name and not an index because PHP interprets a number as the repeat count.  There's nowhere to place an index in the unpack format.  Then, array_values() gives you back the items in the order specified in the unpack string, since PHP associative arrays maintain their ordering.  Finally, the field names must be unique, or else unpack will happily overwrite them.

If you try to use "vV" as the format code, there will only be one value unpacked... named "V".  If you try "v/V", there will be second value... at index 1, where it overwrote the first value.

If you're unpacking just one value, you might try to write list($x) = unpack(...) but this won't work—pack inexplicably returns a 1-based array.  PHP will generate a notice that index 0 doesn't exist and assign NULL to $x.

Theming Drupal 7: Block vs Node vs Field et al

What is the difference among pages, regions, nodes, blocks, and fields?

There's one page that may contain multiple regions.  Regions are containers for nodes and/or blocks.  Multiple nodes can appear within a region if the URL being accessed is a listing view, in which case the nodes will be rendered in teaser form with "Read More" links leading to the full view.  Multiple blocks can also appear in a region.

Blocks seem to correspond to 'widgets' in other systems: chunks of content that can be dropped into the sidebar and remain fairly static on many pages.  For instance, Drupal's search box is a block, and it lands in the sidebar_first region of both default node types.  Blocks do not have fields.

Nodes, on the other hand, correspond to the content that actually gets added to the CMS.  Blog posts, static pages, whatever.  Nodes get URLs assigned to them, either through the URL aliasing features, or the default node/4 style path.  (Or through any SEO friendly URL generation modules you may have installed.)  Nodes have a type and node types have fields.  Fields receive values per-node that are displayed on the node type.

Everything related to a node is rendered inside a single container, corresponding to the page content variable.  Overall, the URL, node with its type and fields, and page content variable are all one interrelated thing when viewing a single node.  Pages and regions are related to the theme as a whole.  Blocks are strongly related to a theme, but also customizable based on node types through the block's Configure link.

Blocks, nodes, and fields are fairly customizable and appear in the admin interface under the "Structure" menu item.  Controlling blocks is done with the Blocks item in the menu (straightforward enough); controlling nodes is done through Content Types.

I should probably note that the paths through the admin interface given are for the default setup where Bartik is the main theme and Seven is the administration overlay theme.  Those paths are not guaranteed for other themes, since themes are PHP and can do nearly anything they want.

If you want "a content area" with several "pieces" to it, the path of least resistance is to construct node.tpl.php with the contents of that content area inside, using fields to display each individual "piece" desired.  Then in your admin interface, establish the fields so they show up when editing the page.

To make that clearer, let's say MyCorp wants a video on their front page with a blurb to the side, a graphical separator, and another couple paragraphs below.  I could make a mycorp_video content type, and add two fields (field_video_embed and field_video_blurb), then create a node--mycorp-video.tpl.php file with the container divs, central bar, and calls to <?php render($content['field_video_embed']); ?> inside their respective containers.  Then I could leave the couple of paragraphs as "body" content and print that below the separator.  Once the template is ready, the node type can be created in the admin interface, the node actually added, and finally set to be the front page of the site.

Controlling something outside of the content area based on the content (node) type is not possible by default, but can be done with an override in the template.php file for the theme:

function themeName_preprocess_page(&$vars, $hook) {
  if (isset($vars['node'])) {
    /* If the node type is "blog_madness" the template suggestion will be "page--blog-madness.tpl.php". */
    $vars['theme_hook_suggestions'][] = 'page__'. $vars['node']->type;
  }
}

The above code was posted by JamieR at drupal.org/node/1089656.  By default, drupal's page renderer only knows about specific nodes (like page--node--4.tpl.php) and not content types (aka node types) in general, which is what this override adds.

A second approach is to use the CCK Blocks module to convert fields into blocks.  This allows them to appear on the block layout and be placed in regions in spite of being node-specific.  The blocks are then made visible in region templates with a  cck_blocks_field prefix, for instance cck_blocks_field_video_embed for a video_embed field.

The latter approach is actually the one I ended up taking.  I needed to handle several optional areas in various combinations.  Instead of a big list of node types and duplication of the markup for any fields shared between types, I have two basic node types and regions handle sharing the markup and displaying the available fields (or nothing, when no fields are set.)

EC2 utilities vs. $AWS_CREDENTIAL_FILE

Most of the AWS command line tools accept their login credentials from a file named in the AWS_CREDENTIAL_FILE environment variable and formatted like so:

AWSAccessKeyId=AKIAEXAMPLE
AWSSecretKey=Base64FlavoredText/Example

The EC2 tools predate this scheme and still refuse to use it, preferring the credentials to be set directly in the environment.  I decided to over-engineer it and pull the EC2 environment variables from the file:

export AWS_ACCESS_KEY=$(grep '^AWSAccessKeyId' "$AWS_CREDENTIAL_FILE" | cut -d= -f2)
export AWS_SECRET_KEY=$(grep '^AWSSecretKey'   "$AWS_CREDENTIAL_FILE" | cut -d= -f2)

(Those will probably wrap on blogger; in code, they're two lines, each beginning with "export".)  Now I can put the credentials in one place, and they're available to all of the tools.

Notes on FastCGI and webservers

This post is a distillation of what I've learned over the past couple of months.  There's both new information here, and links to everything else on the FastCGI topic that I've written so far.

Contents

1. mod_fcgid
2. php-fpm (nginx)
3. other things I tried to use
4. concluding remarks

Minimal, Working Perl FastCGI Example, version 2

This is an update to a previous post.  File layout remains the same: "site" is a placeholder for the actual site name, and /home/site/web is the actual repository of the project.  Static files then appear under public, and Perl modules specific to the site in lib/Site (i.e. visible in Perl as Site::Modname when lib is put in @INC).  I am still using mod_fcgid as the FastCGI process manager.

The major improvement: This version handles FCGI-only scripts which have no corresponding CGI URL.  I discovered that limitation of the previous version when I tried to write some new code, where Apache or mod_fcgid realized that the CGI version didn't exist, and returned a 404 instead of passing it through the wrapper.  As a consequence of solving that problem, FcgidWrapper is no longer necessary, which gives the dispatch.fcgi code a much cleaner environment to work in.

Everything I liked about the previous version is preserved here: I can create Site/Entry/login.pm to transparently handle /login.pl as FastCGI, without requiring every other URL to be available in FastCGI form.  It also stacks properly with earlier RewriteRules that turn pretty URLs into ones ending in ".pl".

Apache configuration:

# Values set via SetEnv will be passed in the request;
# to affect Perl startup, it must be FcgidInitialEnv
FcgidInitialEnv PERL5LIB /home/site/web/lib
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/+(.+)\.pl$ /home/site/web/dispatch.fcgi [L,QSA,H=fcgid-script,E=SITE_HANDLER:$1]
<directory /home/site/web/fcgi>
    Options ExecCGI FollowSymLinks
    # ...
</directory>

Again, the regular expression of the RewriteRule is matched before RewriteCond is evaluated, so the backreference $1 is available to test whether the file exists.  This time, I also use the environment flag of the RewriteRule to pass the handler to the dispatch.fcgi script.  Since I paid to capture it and strip the leading slashes and extension already, I may as well use it.

That means the new dispatch.fcgi script doesn't have to do as much cleanup to produce the module name:

#!/home/site/bin/perl
use warnings;
use strict;
use FindBin qw($Bin);
use Site::Response;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SITE_HANDLER});
    $base =~ s#/+#::#g;
    $base =~ s#[^\w:]##g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = Site::Response->new($base, "$Bin/templates");
    eval {
        eval "require $mod;"
            and $mod->invoke($q, $r);
    } or warn "$mod => $@";
    $r->send($q);
}

I remembered to include the $r->send call this time.  I pass the CGI query object so the response can call $q->header.  That's not strictly necessary—FCGI children process one request at a time and copy $q to the default CGI object, meaning header should work fine alone, but I didn't know that yet.

I also remove non-{word characters or colons} from the inbound request for security, since my site uses URLs like /path/somereport.pl.  You may need to carefully adjust that for your site.

Site::Response is initialized as a generic error so that if the module dies, the response written to the client is a complete generic error.  Otherwise, the template is selected and data set, so the send call ships the completed page instead.

The only thing left that I'd like to do is make this configuration more portable between web servers instead of dependent on Apache's mod_rewrite and mod_fcgid, but since Apache isn't killing us at work, it probably won't happen very soon.

mod_fcgid and graceful restarts

I see plenty of this in my logs when the server needs reloaded to pick up fresh Perl:

(43)Identifier removed: mod_fcgid: can't lock process table in pid 3218

tl;dr: this appears to be harmless in practice.

The leading portion corresponds to EIDRM (see errno(3)) which comes back out of pthread_mutex_lock and cheerfully gets logged as the failure code of proctable_lock_internal.  The proctable is in turn locked during request handling.

My best guess for the order of events is that the Apache parent receives a graceful restart, unloads and reloads mod_fcgid, which destroys the mutex as a side effect.  After old-generation children tie up their requests, they try to notify their parent that they're available again, only to discover that the mutex is gone.  The child then exits, but it doesn't hurt any clients because they've already been served at this point.

This problem is not fixable in Apache 2.2 because there aren't any hooks for graceful-restart.  It just unloads DSOs without warning, and their first clue anything happened is that they start receiving config events.  By then, the mutex and process table are gone, so the newly-loaded master can't communicate with old-generation children.  Someone did make an attempt to fix this for 2.4 (along with modifying mod_cgid to test their infrastructure) but AFAICT nobody has made this available in mod_fcgid for 2.4 yet.

Fun, Work, Puzzles, and Programming

Some programming tasks are just more fun than others. The same thing extends to languages—why are Perl and Ruby so much more fun to work with than Python?

I suspect that the answer lies in the scope of the solution space, in a sweet spot between “too straightforward” and “too complex.”

PHP's debug_backtrace: a compact guide

Every time I need to use this function, I can't remember how it works.

1. The array includes all call sites leading up to the current stack frame, but not actually the current one.  (Everything in the current frame is still in scope to you, so you can use __FILE__ and __LINE__ or your current variables directly.)
2. The array is indexed with 0=innermost to N=outermost frame.
3. Each array index gives you information related to the call site of the next frame inward / earlier in the array.  That is, $bt[0] gives you the immediate caller of your current point of execution.  $bt[0]['function'] refers to the function or method invocation that called you, e.g. if the main code executes foo(1), then inside function foo, $bt[0]['function'] is foo.  The file and line point to the file/line containing the call.
4. When a 'class' key is present, it is the class of the line of code actually executing the call, i.e. what __CLASS__ is at the 'file' and 'line'.
5. When an 'object' key is present, it has the actual object being used for dispatch; i.e. get_class($bt[$i]['object']) may return either the same value as 'class', or any descendant of that class.
6. The 'type' key, when present, is either -> or :: for dynamic or static calls, respectively.  The latter means that the 'object' key won't be set.
7. There is no way in my PHP (5.3.3-14_el6.3 from CentOS updates) to view the invoked class of a static call, e.g. if SubThing::foo is called but Thing::foo is executed because SubThing didn't override foo.  Per the rules above, 'class' will still report Thing.

I needed to know this (this time) because I wanted to write a rough equivalent to Perl's carp in PHP:

<?php
function carp () {
  $msg = func_get_args();
  if (empty($msg)) $msg = array('warned');
  $bt = debug_backtrace();

  // find nearest site not in our caller's file
  $first_file = $bt[0]['file'];
  $end = count($bt);
  for ($i = 1; $i < $end; ++$i) {
    if ($bt[$i]['file'] != $first_file)
      break;
  }

  if ($i == $end) {
    // not found; try the caller's caller.
    // otherwise we're stuck with our caller.
    $i = ($end > 1 ? 1 : 0);
  }

  error_log(implode(' ', $msg) .
    " at {$bt[$i]['file']}:{$bt[$i]['line']}");
}

Obviously this is a bare-bones approach, and could be adapted to pick different (or report more) stack frames, etc.  But, it Works For Me.™