Implementing leaner conflict resolution using standard ETAg/If-Match

Question

I have a REST service for which I need to implement safe updates of resources. A standardized way of doing this is to utilize ETag and If-Match headers, so that when a client attempts to PUT a resource with a given URI, it will say "only proceed to update it if its ETag equals this ETag here, which is the source version I modified; otherwise throw a HTTP 412 error".

In my case however, I want to allow some overlapping updates of resources; e.g. some resources contain deep JSON documents where there's no problem to update two independent properties at the same time, but two requests modifying "nearby" data concurrently should not be allowed. "Nearby" data could for example be two clients modifying the same nested array, where overwrites could happen.

So for example, let's say a resource looks like this:

/persons/xxx

{
  name: 'John',
  children: ['yyy', 'zzz', 'www']
}

(ETag: v1)

Here, I don't mind if two clients both update name - the last one will simply win - or if two clients modify completely different attributes. But two clients simultaneously replacing the children array in order to append a new element, remove the first element etc, could cause conflicts. So I want

PUT /persons/xxx
If-Match: v1

{
  name: 'John2'
}

or

PUT /persons/xxx
If-Match: v1

{
  age: '25'
}

to return HTTP 200 even if the ETag is no longer v1; but I want

PUT /persons/xxx
If-Match: v1
{
  children: ['yyy', 'zzz']
}

to return 412 if the current version v2 also modified children (by somehow determining that the changes made between v1 and v2 conflict with the attempted changes - e.g. by checking if any overlapping document paths are arrays or objects and not simple scalar values).

Question: Is it a violation of the HTTP spec to implement a conflict resolution strategy using Etag and If-Match which sometimes allows modifying a resource even if the resource's ETag doesn't match the client's expected value (match meaning strict string comparison)? The spec is quite vague here, but differentiates between strong and weak ETag values, where weak ones aren't even supposed to be used for modifications (only for best-attempt caching). But for strong ETag values, the spec actually says:

[...] if the resource matches one of the listed ETag values. If the conditional does not match then the 412 (Precondition Failed) response is returned.

Is "match" here meant to be literally interpreted as "strings are equal"?

I know that I'm "free" to interpret the spec the way I want to :) But I also want to try to follow best practices and not have my API behave in a way that will make clients very confused.

The alternative I'm thinking of is basically just another way of including a version for retrieved resources, and signalling that a modification operation applies to a certain expected version, e.g. by including a version in the request/response bodies. This is however basically same exchange of information as the standardized ETag and If-Match just in my own proprietary format, so in a way it feels nicer to use the existing well-known mechanism (but the question is then if it's a bad idea to somewhat change its behaviour).

Answer 1

Given that your intent is to follow the best practices, this answer will do its best to quote relevant information from the applicable RFCs themselves and the background that you have provided in your question.

I have a REST service for which I need to implement safe updates of resources.

By that, I assume you have "a server on which a given resource resides or is to be created", which is the definition of "origin server" as defined RFC 2616, Section 1.3 Terminology, at page 10. This RFC is quite relevant here because it is listed as an informative reference as per Section 10.2 Informative References (even if RFC 7232 obsoletes RFC 2616), and that the term "origin server" will come up a lot in RFC 7232 itself. Also, unless otherwise noted, all occurance of "Section" generally refers to RFC 7232 (well, direct links to the relevant sections for all RFCs are provided on every instance in any case). With all that out of the way, we move straight to the main question:

Question: Is it a violation of the HTTP spec to implement a conflict resolution strategy using Etag and If-Match which sometimes allows modifying a resource even if the resource's ETag doesn't match the client's expected value (match meaning strict string comparison)? The spec is quite vague here, but differentiates between strong and weak ETag values, where weak ones aren't even supposed to be used for modifications (only for best-attempt caching).

By HTTP spec, I assume you mean the relevant RFCs. So, I would like to direct your attention to RFC 7232 Section 3.1. If-Match, specifically the following statement:

An origin server MUST use the strong comparison function when comparing entity-tags for If-Match (Section 2.3.2), since the client intends this precondition to prevent the method from being applied if there have been any changes to the representation data.

The RFC is very cut and dried here. Any notion that some form of weak comparison may be used here is absolutely squashed (given the use of MUST and what you got is an origin server), as the idea that "if two clients both update name - the last one will simply win" is most certainly considered non-compliant with the RFC as a specification - given that if the client supplied an If-Match header, "the client intends this precondition to prevent the method from being applied if there have been any changes to the representation data."

Okay, so you might still be thinking that there's could be some other wiggle room elsewhere as you've asked:

Is "match" here meant to be literally interpreted as "strings are equal"?

Now please let me draw your attention to Section 2.3.2 Comparison, which states:

• Strong comparison: two entity-tags are equivalent if both are not weak and their opaque-tags match character-by-character.

Well, there goes that other wiggle room, as that describes a situation that is essentially "strings are equal".

Before I go into how might you actually wiggle out of this while remaining conformant with the specification, I thought I should draw your attention to Section 2.1. Weak versus Strong, as that contains a paragraph I find relevant to your situation:

There are a variety of strong validators used in practice. The best are based on strict revision control, wherein each change to a representation always results in a unique node name and revision identifier being assigned before the representation is made accessible to GET. A collision-resistant hash function applied to the representation data is also sufficient if the data is available prior to the response header fields being sent and the digest does not need to be recalculated every time a validation request is received. However, if a resource has distinct representations that differ only in their metadata, such as might occur with content negotiation over media types that happen to share the same data format, then the origin server needs to incorporate additional information in the validator to distinguish those representations.

While they were listed as examples, the authors of this RFC really want to hammer in on the point about uniqueness of the strong validators. So how might we wiggle ourselves out of this mess? Especially given the following being the main goal that spawned your entire question:

I want to allow some overlapping updates of resources; e.g. some resources contain deep JSON documents where there's no problem to update two independent properties at the same time, but two requests modifying "nearby" data concurrently should not be allowed.

Well, the latter bit is basically spec conformant, given that any "changes made between v1 and v2 conflict with the attempted changes", your application will reject by responding with HTTP 412.

Now to address the problem of allowing some overlapping update of resources and how one might get around that. A more keen observer of Section 3.1 If-Match may have noticed this example:

If-Match: "xyzzy", "r2d2xxxx", "c3piozzzz"

Wait, so maybe there is a way for the user agent to generate ETags specific for each of the fields and they can just throw them all with the PUT request and hope one of them sticks? Not so fast. As per the RFCs, there is actually not a way to actually send multiple values for ETags as the ABNF definition for the ETag header has not been specified to permit multiple values as per Section 2.3 ETag, and that the RFC 2616 Section 4.2 Message Headers has this to say about headers in general:

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma.

So it might be thought that multiple headers like ETag may be provided (notwithstanding doing so would inevitably require a out-of-band method to couple any given ETags to their specific fields, which most certainly violate RESTful principals due to this out-of-band couping, but we really don't need this additional digression here), this is absolutely NOT the case (thank goodness, we don't have to go deep into what is RESTful here) as the paragraph explicitly stated "if and only if the entire field-value for that header field is defined as a comma-separated list", which ETag is NOT defined as such as per the documented ABNF, as it was not a field that got defined using the #rule as hinted by "#(values)". Wait, what is the #rule? Helpfully, RFC 2616 Section 2.1 Augmented BNF explained it as: "a construct "#" is defined, similar to "*", for defining lists of elements." As for an example of message-header fields that permit multiple values are readily found in the relevant RFCs, and one was literally run under our noses even - revisiting Section 3.1, we got this definition for the If-Match header:

If-Match = "*" / 1#entity-tag

Back to ETag, and we see that its rule has no such # line, therefore any attempt to provide multiple values within a ETag header (or to provide more than one such header per message) would be non-conformant with the RFC. Darn. As a matter of fact, this particular question has been asked before on this very site, in case you want some additional reading on top of this already grotesquely long answer for a fairly simple question... Actually, what I explained about If-Match is explained in much less words in this answer, but I thought you wanted a lawyer version of this answer because you tried to lawyer your way out in relation to the specifications with the thinking that you might be "free to interpret the spec the way I want to :)"

Okay, so what can be done? Easy! I would say - just have dedicated end points for each of the fields! Given that each Person (as per the question) have name, age and children, just include additional end points for them, e.g. /person/{id}/name and so on.

This will also easily address the unrelated updates even if the top level Person item got updated - e.g. if the /person/1 contains the following:

{
    "name": "John",
    "age": 25,
    "children": ["yyy", "zzz", "www"]
}

Doing a HEAD /person/1 might produce the following header:

ETag: "ef03ad39"

Doing a HEAD /person/1/age might produce the following header:

ETag: "fa1a82ad"

Now, someone comes in and does a PUT /person/1, that goes something like this (only including the relevant headers at play):

PUT /person/1 HTTP/1.1
Content-Type: application/json
If-Match: "ef03ad39"

{"name":"John Smith","age":25,"children":["yyy","zzz"]}

The request should succeed as the value of the ETag for the original JSON can be calculated to ef03ad39 (it's the CRC32 of the compact representation of that JSON string, in case you want to test). Now, if some other user agent were to PUT anything else but this exact representation (e.g. could even be the same user agent retrying because they didn't receive the success response due to network issues), but with that now very much stale If-Match: "ef03ad39" header, they should be rightfully responded with HTTP 412 as the ETag now has the value of 25b5a131. However, even after that update via that PUT request, a new PUT request to update just the age might come in like so:

PUT /person/1/age HTTP/1.1
Content-Type: application/json
If-Match: "fa1a82ad"

42

This can in fact succeed and would not be invalidated by the previous update (that changed everything else but this "age" attribute), because the calculated ETag has not changed for /person/1/age despite everything else have changed already. Now, doing a GET /person/1 may produce in the following response (as an aside, ETag headers must be double-quoted):

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "f2dce9f9"

{
    "name": "John Smith",
    "age": 42,
    "children": ["yyy", "zzz"]
}

In short, it is completely possible to do partial modification to resources, e.g. modifying only specific fields of a given resource, in a way that fully take advantage of ETag and If-Match for invalidation of conflicting updates at a fine-grained manner in a specification compliant manner, when each of the specific fields have a dedicated endpoint to facilitate the update through HTTP operations.

---

Now, you may argue that the operations you wanted to do is actually patching a subset of field(s), so this brings up PUT vs PATCH, so would something like PATCH /person/{id} with a subset of values get around this. Well, I would argue that the ETag should still be derived from whole target resource to avoid the ambiguity (as that typically requires out-of-band details to resolve which runs counter to REST), unless there is a way provided to query for ETag specific to a particular attribute from the main item - which I've just provided a viable solution above through subpaths (query string is another option if you want, but I personally find paths more friendly).

Expanding that further, this related question asked what if the client does have the ETags for the sub-parts, a PATCH could just use that? Probably, but this does not get around the need to provide this ETag in the first place, which goes back to needing the unique endpoints (or a way to query) for each attributes. Makes sense to have those endpoints accept PUT (or PATCH) requests while at that, and embracing this idea leads to the next issue.

If PATCH on /person/{id} is implemented, inevitably the desire to update multiple attributes would come up. While multiple values for If-Match: may be supplied, the thinking goes it might be possible to submit all of them in the order of the attributes being updated. Well, this actually raises more questions/confusion than answers/clarity. First, no specification exists how to correlate ETags to each attribute (remember what I wrote about needing out-of-band data), second, if one of the attribute has been overwritten, does the endpoint accept or fail the change? The specification states "the condition is false if none of the listed tags match the entity-tag of the selected representation", so if one of them is true maybe the request can be accepted?

Well, let's see what else the RFC says, so back to Section 3.1 - "the If-Match header field makes the request method conditional on the recipient origin server either having at least one current representation of the target resource" - the keywords being "representation" and "resource" - showing just two attributes of the given resource is a representation of it. Thus using the ETags that were generated for unrelated representations (i.e. one generated for a single attribute vs. one generated for two) would be in violation, because "the ETag header field in a response provides the current entity-tag for the selected representation", and there is no way to magically combine two ETags from two separate representation and use it for a third representation (the one with two attributes).

See how the attempt to allow PATCH with multiple attributes while being wishy-washy about ETags gets confusing very fast for every party involved (not to mention we are starting to ask the same question, with PATCH verb instead of PUT). Given the intention is to patch the underlying resource, the ETag for the whole of the resource at the given URI should be considered, and we are back to dedicated endpoints for attributes again. Really, just have a dedicated end-point for each field/attribute/subset of the top level resource, it removes the ambiguity with what the service offers, decreases complexity for having to compute all the different combinations of attributes, and brings clarity to everything and everyone involved.