REST and RPC: Not Actually Antonyms

Last month found me writing a rant about REST and the shortcomings of interpreting it as "REST = HTTP+HATEOAS".  I submerged myself into some writings of Fielding, and took some time for reflection, and I've found one of the sources of my problems with "REST".  (Another.)

This problem is that there's too much writing on the web that attacks "RPC systems" as the logical opposite of REST, and I took this assumption unquestioned.

State Transfer

The very foundation of REST, the part that puts "ST" in "REST" and relegates the "RE" to a mere Latinate adjective, is state transfer.  This is, as far as I can tell, the basic property that makes REST distinct from other styles, and also interesting.  The whole thing about state management in REST is: do not keep hidden state; transfer it all.

You know, "shared nothing."  Like NFSv2 or "modern" scalable architecture.

RPC Isn't About State

Wikipedia focuses on the definition of "RPC" as a means of invoking a service on some server in a manner indistinguishable at point-of-use from some local function call. It just happens to be a special function (the RPC stub) that packs up its arguments and sends them over the network.

Since the server doesn't have access to the process-local memory, the RPC call must also be a pure function: all of the state it needs to perform its action is carried in its arguments, and your program is not affected beyond receiving the return value.  This is state transfer.  Can it be called REST?  It probably can, if the function is duck-typed—when any representation can conform to the interface and be accepted.

What about HATEOAS with our pure-functional RPC?  As I understand it, the rationale for HATEOAS is that the server should be in control of its namespace.  The systems that I am familiar with and labeled RPC are available on the network at a single URL, just like a "proper" RESTful server.  Depending on the exact protocol, there may be a separate file specifying what methods, parameters, and return types are available. Providing a definition file allows the server to retain control of its namespace, as long as clients don't cache a stale definition for too long.

Of course, nothing prevents an RPC system from implementing its own definition method, or proxying individual methods to other servers on the backend.

But the Internet Says a "Method" Parameter is Definitely Not RESTful?

I know, but I don't think that RPC prevents a RESTful design from occurring.  If the server can specify what methods it has, and also use that specification in results (such as listCustomers() being able to indicate "customer id 1, details at getCustomer(id=1)") to form a hyperlink, then it can be RESTful.

The implementation probably doesn't work out very well with RPC stubs in languages like C, where the function names to be called have to be fully known at compile time, well before talking to the server.  But a language that allows for running code when a function name isn't found is perfectly capable of the trick.

However, most RPC systems probably don't have the hyperlink aspect to them, in which case they would indeed not be RESTful.  This is what makes for the above rule: a "method" parameter generally indicates RPC, which is generally not RESTfully designed, therefore the method parameter is strongly correlated with non-REST architecture.  That correlation is simply not a guarantee.

Batch Operations

Most of the RESTful API design advice I can find seems to be centered around basic CRUD, using the familiar GET, POST, PUT, and DELETE verbs to act on a single object in the data store at a time: DELETE /users/1 If-Match: SomeETagValue and so forth.

Actually, I lied.  Most of the design advice doesn't even cover If-Match.  However, they do ignore searching, as well as anything that you would approach with a transaction in the average non-RESTful system.  I ran across a comment (#22 at this page) by Fielding on the batch operation matter:

People perceive a need for batch operations because they don’t understand the scope of resources.

Resources are not storage items (or, at least, they aren’t always equivalent to some storage item on the back-end). The same resource state can be overlayed by multiple resources, just as an XML document can be represented as a sequence of bytes or a tree of individually addressable nodes. Likewise, a single resource can be the equivalent of a database stored procedure, with the power to abstract state changes over any number of storage items.

If you find yourself in need of a batch operation, then most likely you just haven’t defined enough resources.

Emphasis mine.

It seems that being equal to a stored procedure is simply another implication of state transfer: all the necessary state to perform the operation is sent in a single request.  HTTP support like If-Match offers the ability to use the server in a lock-free style.  Request some bits, do work, and send updated bits with some sort of version indication so that the server can tell whether they're out of date.  If they are, the server returns 409/Conflict and the client tries again.

URLs as Identities

If you're not familiar with Clojure: identities are a series of states (actual stored data) over time.  These are very similar to variables, but Clojure offers identities as specific kinds of variables, each with unique semantics for updating them.  A value is simply the state of an identity at some point in time.  Conceptually, it's a copy, so that once you have the value, it doesn't change, even if the state of the identity is updated.

If we access /users/3 in our RESTful API, we expect to get the most recent version of user 3.  Since the data at this URL can change over time, it's clearly not a value, so the URL must be an identity instead.  The URL+ETag together form a state, and associated data is the value.

The lock-free style must identify a specific state, so that the Conflict error can be generated if the state operated on doesn't match the current state.  That's what atomic compare-and-swap instructions offer in client-side lock-free programming, and what If-Match provides when we PUT /users/3; but how does it carry over to batch operations, where the resource may need to identify several entities' states?

Apparently, the URLs need to include the ETag to indicate the state rather than the identity.

Refining REST

Now that I defined state as the value of the entity at a URL at some point in time, we can use that definition to get a deeper understanding of state transfer.  In its basic form, when we PUT /users/3, we have both an identity (the request URL) and value; the request itself is asking to associate the two, as of the current time.  Once time is involved, we can say there's a state: the PUT request is defining a new state of the identity.

Therefore, "state transfer" means that we're sending a complete state along the wire, either literally or by reference through URIs to other states.  This is how we can have idempotent requests and caching: if a request is conditional upon a state, has no effect when the condition does not hold, and changes the condition when successful, then duplicate requests will fail if any others succeed.  An RPC-based system may or may not make this guarantee.  Systems implemented over HTTP must follow the guarantees that HTTP defines for its methods, or else use POST for everything, which is an anti-pattern in its own right.

SOAP and WSDL Don't Guarantee RPC or a Non-RESTful Architecture

I want to wrap this up with an example.  QuickBooks offers an SDK, which allows for scripting the QuickBooks GUI.  (Literally so: you cannot interact with company X if the GUI is logged into company Y.)  They also provide a QuickBooks Web Connector (QBWC), which connects to a server from the client machine and makes requests.  The server (eventually) responds with chunks of qbXML that represent QuickBooks operations, which the web connector passes to QuickBooks on the server's behalf.  Finally, QBWC gets another qbXML document from QuickBooks in response, which it returns to the server.

This server is actually a web service, available over SOAP, using the WSDL that Intuit provides with the SDK.  However, HTTP and SOAP are simply building a tunnel for the exchange of qbXML, and the design of those files is RESTful.

To build an audit log and/or prevent conflicting changes in multi-user mode, QuickBooks can only allow the most-recent state to be updated.  Therefore, every object in QuickBooks has both a unique identity and state: ListID and EditSequence.  Writes MUST provide an EditSequence, and the EditSequence value MUST match; otherwise, a conflict has occurred, and the write is denied.  This rule applies to all calls, user-initiated and SDK-generated, and therefore naturally applies to QBWC as well.

Since the server's only point of contact with QuickBooks is qbXML, and it doesn't support constructing a change over the course of multiple calls, all of the data to complete a request must also be included in a single qbXML request.  Once again, this is a signature of REST.

As far as I can tell, the stack involves SOAP and WSDL to provide a guide to the interface between the client and server.  (Even so, the WSDL indicates certain methods are optional, but QBWC will not proceed if they are missing.  I think there's a grumpy comment in our source about that, regarding clientVersion and serverVersion.)  In other words, the stack tunnels over SOAP and WSDL, and uses XML as its resource format, because things like WSDL-to-code generators and XSD schema validators exist.

Using SOAP and WSDL means that Intuit didn't have to create any sort of WSDL equivalent for services exposed directly on HTTP, nor do people implementing their SOAP service have to worry about URL routing.  Implementors can put their service at any URL, and that's all that the tunnel needs to work.  Using this to build the layer that RESTful qbXML travels on butters the bread on both sides: Intuit gets the benefit of WS-* and XML tooling for setting up the requests, and also the benefits of REST when processing them.

Closing Remarks

Fielding has also remarked that it's not possible to determine whether an application is using an interface in the REST style by scanning its network traffic.  If the server provides URLs, but the client ignores them and requests hard-coded URLs that happened to be in the response, then the network traffic is identical, but the decision hasn't been made by actually chasing the hypertext pointers.

It's important to remember that REST is a style.  It's not about integrating with HTTP.  It's not about "not being RPC" or "not being SOAP".  If a service were implemented as EXI sent over SCTP just to be exotic, the result could still be RESTful.

I am now of the opinion that similar things befell the terms "object oriented" and "REST", at least if we take Smalltalk as 'defining' OO: some important, useful, and somewhat mind-bending concept went mainstream in the programming world, and the difficult parts were lost in translation.  This makes it rather annoying to come along later and learn about it, because it seems silly or half-baked instead of deeply insightful, and I begin to wonder why everyone is so enamored with it.  At least until I find the True Meaning.™

Now get out there and rock.