ETags for the Uninitiated

Yet, ETags are one of the features that are the hardest to get right. Sometimes it’s not even clear how they work and while there’s a lot out there on the subject, it can also be difficult to put it all together. Developers frequently play either client and server roles in this exchange, which can make the responsibilities even more confusing.

In this series of blog posts, we’re going to look at ETags from both perspectives: First, a client trying to consume an ETag-enabled API. By focusing on the client side, we can focus on the features ETags offer and learn how these are supposed to look in a perfectly implemented world. In a later post, we’ll look at the gory details of how that API implements ETags and does the appropriate checks.

So, let’s get started with the client point of view by beginning with the natural question:

What is an ETag?

Simple: it’s a string the API generates that represents the current version. When you read or update a record, the ETag is a value you can compare with other versions of that same record to see if they’re the same. The HTTP spec says that ETags are an “opaque quoted string”, meaning they don’t need to be any particular format. In fact, you shouldn’t count on there being a format because there likely won’t be one.

On almost all APIs you’ll encounter, ETags look like an md5 or sha1 hash and for good reason: that’s usually what they are. Even if the ETag has some particular format that you could regex information out of, you shouldn’t do it. When the spec says “opaque” they mean it literally: you can’t see into the ETag. The strings can change quickly and the API is under no obligation to keep it in the same format.

So, how do you get an ETag from the API? When you send requests to a server, the response will place the ETag in its own header. An abbreviated example might look like:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: "72972b69942c42b59bc76bec59436f7c"
{ "text": "lorem ipsum” }

The presence of this header is your hint this resource supports ETags. Not all APIs do, so check carefully. Sending ETag related headers to APIs that don’t support them will just cause the extra headers to be ignored, so there are no dire consequences but it won’t make debugging any easier.

Don’t Cross The Streams

An important thing to note: we said that this resource supports ETags. Even though an API might support ETags, it doesn’t mean that every individual URI in the API will. For reasons known only to itself, the API could theoretically generate ETags for /article/42 but not for /article/43. While uncommon, this is something clients should watch out for.

When you’re working with ETags on the client side, it’s also important to remember that they have a scope. Specifically, they’re only good for a particular URI. If /article/42 and /article/43 both return the ETag “b6b689f”, that doesn’t mean they’re identical. Comparing ETags from across different URIs isn’t valid in any way.

This is especially important for clients or proxies that do some form of local caching. Depending on how the API generates its ETags, this kind of collision can occur often. There’s a simple way to overcome this though; when generating your client side cache key, make sure to include the URI as well as the ETag. This should prevent you from ever accidentally serving the cache of one page as the body of another, as this can result in… terrible mishaps, to say the least.

We’ll discuss it further in a future article but you should also try to avoid using ETags from different representations of the same URI (say, the XML vs the JSON versions of /article/42). A well formed service should generate a different ETag for each representation but this is one of the parts that are often implemented incorrectly. It’s not the client’s responsibility to sort these differences out but if the web service is incorrect, you might need to watch out for this.

Enough already! What can I do with the darn thing?

There’s two main things you can do with ETags: caching and conflict detection. Caching is the most common use case, so we’ll start there.

Let’s consider a hypothetical use case: We’re writing a javascript application that refreshes a news feed every 10 seconds. We take the body of this request and render it for the user. That said, repeatedly redrawing the content over and over causes an annoying flicker in the screen. Even worse, we’re really running up a bandwidth bill by requesting a large JSON blob every ten seconds. How can ETags turn this around?

To begin, we make an initial request to the API. This is just a regular GET request, there’s no funny tricks or headers:

GET /news HTTP/1.1
Host: api.example.com
Accept: \*/\*

The server then gives the same response we saw above:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: "72972b69942c42b59bc76bec59436f7c"

{ “text”: “lorem ipsum” }

Included in the response is the ETag for this version. From this point on, we can store the response and use the ETag for caching. However, there are different ways you might use an ETag so it’s not enough to just send it back using an ETag header in the request. To use it for caching, we need a more specific header to inform the API of our intention.

That header is If-None-Match. The “If” at the beginning of the header is a tip off that this header allows you to make conditional requests. A conditional request tells the server “Hey, I’m going to send you this request but I only want you to actually process it if the given condition is true.” There are two headers specifically for making conditional requests based on ETags: If-Match and the aforementioned If-None-Match.

You can picture If-None-Match in pseudo code as:

if ($ifNoneMatchValue !== calculateETagForResponse($request)) {
    createResponse();
}

To put this into practice, let’s imagine our 10 seconds are up and we’re sending a second GET request to the server. The only thing different is that we’re also sending the ETag value that we acquired from the first response.

GET /news HTTP/1.1
Host: api.example.com
If-None-Match: "72972b69942c42b59bc76bec59436f7c"
Accept: \*/\*

There haven’t been any amazing announcements in the meantime, so theoretically the content of the feed is exactly the same. And so, the API can now return:

HTTP/1.1 304 Not Modified
Date: Sun, 17 Mar 2013 17:51:42 GMT
ETag: "72972b69942c42b59bc76bec59436f7c"

If you compare that to the previous version we received, two things jump out.

First, instead of the 200 OK status code, we receive a 304 Not Modified. This status code speaks for itself, it’s the server informing us that this resource is exactly the same as when we got the ETag. For our simple javascript application, it means we don’t need to rerender the news feed. In the case of your browser, this tells it that it can reload the content from its cache.

Secondly, there’s no body attached. Since we already have the content locally, why should the API send it again? This can be a win-win for both parties: the API doesn’t need to waste time formatting and sending data that’s already known. The client wins by not wasting time downloading or CPU cycles manipulating data it’s already processed (depending on how smart the client wants to be).

So, what happens when the record is modified when a new item comes into the feed? Well, we can just keep sending the same request every 10 seconds and when the feed is updated, we’ll get a new response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: "f4275dd1216aa6cf99926b675c19d1d7"

{ "text": "foo bar” }

Now we receive a 200 OK response with the complete, updated body. With this new version also comes a new ETag to send on future requests and the entire process can start over. We’ll start sending this new ETag on subsequent requests and the server will return 304 responses until a new version is available again.

And in a nutshell, that’s caching with ETags.

Hoo. Wah. What is it good for?

Now that we’ve got it up and running, let’s talk about when and where you’d use this. The main benefit of ETag caching is the savings in bandwidth and the explicit confirmation that the data is unchanged. If the resource you’re downloading is fairly large, the reduction in download and client parsing time can be substantial (especially from a usability perspective). Naturally, all major browsers have supported ETags for years. ETag caching is also great for pages that can change often but are also hit often, the front page of a news site would be a good example. Combined with a reverse proxy that supports ETags, you could devise a high performance caching strategy that also tries to conserve bandwidth.

On some APIs, you might even see an overall decrease in response time as the server can potentially skip some work (template processing, data fetching, etc). That said, it depends on how the API is built; sometimes the extra ETag processing could actually make the request slower. In short, you can’t count on there being any speed gain from this and even if there is, it likely isn’t much.

The main issue with ETag caching is that every cache check requires you to hit the wire. The network request to look for a new version is “slow”. Personally, I tend to try a time based cache (i.e. Cache-Control/Expires) before using ETag caching. As these caches are local, the lookup time is far faster than querying a remote server could ever be. However, for situations where the business logic doesn’t allow doing that, ETags are still an excellent addition.

But wait, there’s more!

In this post, we’ve looked at the basics of ETags from the client’s point of view and seen how we can use them to make simple conditional requests for caching. In the next part of this series, we’re going to move on to using ETags with POST and PUT requests and talk about what’s arguably the killer feature of ETags: helping prevent version conflicts when multiple clients are working with the same resource. We’ll also take a look at “weak” ETags and how they differ from what we’ve seen so far.