Subject: [Sip] Question on subnot-etags

From: Eric Rescorla <[email-masked]>
Date: Mon, 03 Mar 2008 10:32:25 -0800

I'm sure I'm wading into a swamp here, but:

Say I have a resource identified by URI http://www.example.org/X.

1. Initially it has value A. I do a SUBSCRIBE and get value
   "A" and Etag 1
2. I do a subsequent get and get "B" with Etag 2.
3. I do a third get and get "A".

Can the Etag in event 3 be 1?

Thanks,
-Ekr

_______________________________________________
Sip mailing list  https://www.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: [email-masked]
Date: Mon, 3 Mar 2008 15:50:40 -0500

From: Eric Rescorla <[email-masked]>

If event 3 has exactly the same value as event 1, I don't see why not.

Dale

---

From: Dean Willis <[email-masked]>
Date: Mon, 3 Mar 2008 15:14:56 -0600

> Can the Etag in event 3 be 1?

I would think not -- it would need to be "3". Otherwise we have a  
synch failure.

--
Dean

---

From: Aki Niemi <[email-masked]>
Date: Tue, 04 Mar 2008 13:08:05 +0200

> Can the Etag in event 3 be 1?

What we currently say on the subject is this:

        [An entity-tag is] guaranteed to be unique across all versions
        of all entities for a resource and event package.

If the version of the entity, in this case "A", is identical to a
previous version, which is "A" in this case, then the entity-tag can
certainly be the same as well.

We would have the exact same result were we to generate the entity-tag
value using a hash function over the entity itself.

This is also unimportant for the subscriber, since it needs to store the
entity-tag "statelessly", and cannot defer any meaning from its value.

Cheers,
Aki

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 04 Mar 2008 07:19:05 -0800

At Tue, 04 Mar 2008 13:08:05 +0200,
> certainly be the same as well.

It's not clear to me that the text you are quoting implies the
interpretation that you just provided. In fact, I would note that you
are referring to "the version" and "the previous version" which
suggests that they are separate versions and must have different
ETags.

Given that when I asked this question I got two different answers from
Dale and Dean, I submit that the draft might be improved by directly
addressing this point. Note that I don't care about the resolution,
but I do care that the draft be unambiguous.

> entity-tag "statelessly", and cannot defer any meaning from its value.

I'm not sure what that means, but I'm not convinced that this is 
meaningless to the client.

I also wonder whether it's a good idea to call this feature
ETags. My understanding is that ETags has been a very confusing
feature of RFC 2616. Even if we think that the semantics here
are the same, perhaps we should consider giving it a different
name.

-Ekr

---

From: Dean Willis <[email-masked]>
Date: Tue, 4 Mar 2008 10:59:33 -0600

> meaningless to the client.

I look at it in terms of sequential application of differential  
transforms.

If we start with a given object O1, then there exists a differential  
transform D1 that when applied to that object produces a second  
object, 02. In other words,  D1(O1)->02. Clearly O2 has a different  
etag from O1.  If we then apply a second differential D2 that produces  
O3 such that D2(O2)->O3==O1, we have a new object O3. Is it reasonable  
for O3 to have the same etag as O1? Mathematically, it's not the same  
version of the object -- it's another version of the object that just  
has the same value. So it instinctively irks me to give it the same  
entity tag as that other version of the document.

Consider the aspect of state -- if I'm sending the whole object every  
time, it doesn't matter. If I only work from the current version of an  
object, it doesn't matter. But if I'm doing version control and I need  
to make sue I've stored the entire sequence of transformations (or at  
least know whether I've missed any steps), it DOES matter. Working as  
Aki proposed, any cycle (any sequence of transformations that takes us  
back to a state we've seen before) could be undetectably dropped out  
of the history.

Do we have a requirement to keep our history straight? Perhaps not.  
After further reflection, I think that for the common use case  
(keeping track of the current version of a document with full document  
transmittal) the technique as proposed by Aki works just fine. Since  
etags aren't really sequential, we don't really gain anything by the  
ability to detect dropped cycles, do we?

Somebody needs to think this through from the perspective of partial  
notification.

--
Dean

---

From: Aki Niemi <[email-masked]>
Date: Tue, 04 Mar 2008 22:07:07 +0200

> suggests that they are separate versions and must have different

Then I'd appreciate any suggestion to make it clearer.

> but I do care that the draft be unambiguous.

Can you point out the unambiguity?

> meaningless to the client.

It means the only thing the client is ever going to do is offer that
same entity-tag back to the notifier in a Suppress condition. The
entity-tag, as defined, is an opaque token that has no meaning to the
subscriber beyond serving as a tag, or reference to the state it
received in the NOTIFY.

If the draft says something contrary, we'll change it.

> are the same, perhaps we should consider giving it a different

The header field is actually called SIP-ETag just for this purpose.
Unlike RFC2616, we don't have weak and strong entity-tags, nor do we
allow a list of entity-tags in SIP-ETag. Especially the former is what
makes entity-tags confusing in HTTP.

This is also discussed somewhat in RFC3903 where SIP-ETag is defined. I
can add some text to the introduction that refers to that RFC with
regards to entity-tags.

Cheers,
Aki

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 04 Mar 2008 12:20:05 -0800

At Tue, 04 Mar 2008 22:07:07 +0200,
> Then I'd appreciate any suggestion to make it clearer.

Explicitly state that in the case I described that it's permissible
to deliver the same Etag in the first and third versions.

> Can you point out the unambiguity?

I think Dean did a good job with that, namely that one could
view "versions" as instances, and that two versions with
different content are different versions and so need
different Etags.

-Ekr

---

From: Aki Niemi <[email-masked]>
Date: Tue, 04 Mar 2008 22:31:45 +0200

> Somebody needs to think this through from the perspective of partial  

In S 4:

   With partial event notification, the NOTIFY message only carries the
   delta state, or the set of changes to the previous version of the
   entity.  In that case, implementations MUST consider the full event
   state as the version of the entity to which the entity-tag in the
   NOTIFY message applies.

In other words, you're not tagging the individual deltas, but the
resulting state after the delta has been applied.

Cheers,
Aki

---

From: Aki Niemi <[email-masked]>
Date: Tue, 04 Mar 2008 22:44:01 +0200

> to deliver the same Etag in the first and third versions.

Something like in S 5.3:

   The subscriber MUST be
   prepared to receive a NOTIFY with any entity-tag value, including a
   value that matches any previous value that the subscriber might have
   seen.

> different content are different versions and so need

That's exactly what the model already says.

Your original question was whether "A" and "A" could have the same etag.
According to the draft, yes. ALso according to the draft, if you decided
to give them a different etag, fine. Nothing would break. Everything
would work exactly the same way.

I'm apparently missing what practical problem the draft is not
addressing.

Cheers,
Aki

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 04 Mar 2008 12:55:16 -0800

At Tue, 04 Mar 2008 22:44:01 +0200,
>    value that matches any previous value that the subscriber might have

Yeah, I don't think that's at all clear. For all I know, that
could mean the server is running in a VM and was rolled back.

> That's exactly what the model already says.

Sorry, that should have said "the same content".

> According to the draft, yes.

But I don't think that's clear as you seem to think it is.

> ALso according to the draft, if you decided

I'm not saying there's a practical problem the draft is not addressing.
I'm saying the draft isn't unambiguous and it's simple to make it
so.

-Ekr

---

From: Aki Niemi <[email-masked]>
Date: Wed, 05 Mar 2008 15:44:49 +0200

> could mean the server is running in a VM and was rolled back.

It could also mean the Lakers just fired Koby in order to make room in
their salary cap to hire me as their point guard. 

It just doesn't matter; the subscriber is not supposed to even attempt
to infer such a meaning from the entity-tag value.

Perhaps you're looking for something like this:

        The entity-tag is an opaque token to the subscriber.  The
        subscriber MUST NOT attempt to infer any meaning from the value
        beyond a simple reference, or assume a specific formatting.  The
        relationship between the current entity-tag and any future or
        past entity-tags is undefined.
        
> I'm saying the draft isn't unambiguous and it's simple to make it

Therein lies the problem. To make it unambiguous would mean we either
mandate different entity-tags for identical entities, or we mandate
identical entity-tags for identical entities.

But currently this is *intentionally* undefined. If my notifier
implements MD5 hash over the entity as the entity-tag value for event
package X, identical entities would indeed receive identical
entity-tags. That will work.

But if my notifier implements a timestamp as the entity-tag value for
event package Y, identical entities would receive different entity-tags.
That will also work.

Bottom line is, the subscriber doesn't care, and the notifier can do
what it wants. We require uniqueness of entity-tags as a friendly
suggestion not to shoot self in foot. If the notifier dishes out the
same entity-tag for different entities, it can cause the notifier to
suppress a NOTIFY that should have been sent.

If this is inline with how you think this should work, I can try to
sketch a paragraph or two into section 6.1 basically saying the above in
normative language.

Cheers,
Aki

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 05 Mar 2008 06:06:49 -0800

At Wed, 05 Mar 2008 15:44:49 +0200,
> their salary cap to hire me as their point guard. 

Yes, precisely. It's not clearly written.

>         past entity-tags is undefined.

But that's not true. You require that if value(A) != value(B),
ETag(A) != ETag(B). that's not undefined at all.

> identical entity-tags for identical entities.

No, that's not true. I'm perfectly happy to have it be unambigous
that you may or may not have identical etags for identical
entities. What I'm not OK with is the current state in
which it is not clear whether this is legal or not and
so the subscriber doesn't know whether he can rely on
it. 

I appreciate that you believe that the current text clearly
allows both designs, but I don't. I think it can
be interpreted either way (as either (1) allowing both
designs or (2) prohibiting one).

> suppress a NOTIFY that should have been sent.

That's not a friendly reminder. It's a hard requirement of the
spec.

> sketch a paragraph or two into section 6.1 basically saying the above in

I'm fine with what you describe above, but what I'm looking for is
to have you *explicitly* state that if there are two instances
with the same value they MAY have the same etag, even if there
was an intervening instance with a different value. Not MUST,
MAY. And the reason I think this is important is that IMO,
the current text can be interpreted as forbidding this.

-Ekr

---

From: Dean Willis <[email-masked]>
Date: Wed, 5 Mar 2008 14:59:11 -0600

>        past entity-tags is undefined.

But we DO infer something -- if the entity tag has changed, then the  
object has changed. So we infer that the object has changed when we  
see a different entity tag.

But if the object changes (and we miss the notification), then changes  
back (returning to the same entity tag value) then we can no longer  
infer that the object changed, because we have the same entity tag. We  
don't know that the object ever changed, and would assume that it had  
just always been the way it is. Thus our inference would be incorrect.

While this may not matter to the state of the object, it might matter  
to our understanding of how the object arrived at that state.

On the other hand, if the entity tag were unique for each version of  
the object, we'd at least know that there had been some state change  
in the object (which could have been a missed notification, or might  
have been a null-operation that just updated the timestamp).

I have a vague, queasy feeling that this might turn out to be  
important for some applications. But perhaps I'm looking at this in  
the wrong way.

For objects where this kind of state awareness matters, perhaps one  
could just require  that the object in question include a timestamp,  
or better yet a serial number.

---

From: [email-masked]
Date: Wed, 5 Mar 2008 23:42:52 -0500

It seems like there are a number of open issues:

1. Is the only thing that is important from the subscriber's point of
view the current value of the event package, or is it important to
know something of the history of that value?  (People have been
discussing this.)

2. The interaction of ETags with partial notifications.  The draft
reads:

That is, the ETags carry information about the composed full state
that the subscriber is maintaining by integrating the partial
notifications that it receives.

But it is problematic to use partial notifications in combination with
ETags:  When the notifier determines that the subscriber's current
state ETag is different from the current ETag of the resource, it does
not know what the subscriber's current state *is* (unless it maps the
presented ETag to a previous package value), and so cannot update the
subscriber with a partial notification.  Use of ETags seems to
effectively require the use of full notifications at all times.

3. Some event packages (e.g., dialog and resource-list) specify that
the package values within any individual subscription be sequence
numbered via a 'version' attribute in the XML.  This reduces the
chances of an ETags match between requests that are part of different
subscriptions, because a new poll-type SUBSCRIBE will necessarily
fetch a value with 'version="1"', and its ETag will not match the ETag
of any value that had a different version attribute, even if the rest
of the XML is the same.

This is particularly unfortunate because the version attribute does
not carry information that is important to the application; it is
control information that is largely redundant with the ETags mechanism
itself.

4. Conflicting assignments of ETags to values in notifiers that are
not co-administered, but are alternative forking destinations of some
URI in the universe.  (People have been discussing this.)

Dale

---

From: Aki Niemi <[email-masked]>
Date: Thu, 06 Mar 2008 09:39:14 +0200

> see a different entity tag.

No, we really don't. 

The semantic for the subscriber is simply: ah, a notification! That'd be
the current state of the resource. Oh, there's also a reference tag on
it. Great. I'll store them and render the state to the user. 

The semantic is not: hmm, a notification. I know they're supposed to be
generated when state changes, but I'm not convinced the state has
changed. I better match the etag with my stored history of etags to see
whether the state *really* changed. Ha! A match! The notifier farm
must've rebooted or the Lakers indeed fired Koby! I better calculate the
nth decimal of pi, launch doom and busy-loop for the rest of the day.

There *really* is no comparison done; the subscriber throws away the old
state when a NOTIFY is received, and stores the new state. (In the case
of partial notification, there's also patching done, but still no
comparison of etags). 

This is how notifications work without conditional notification and
we're not changing that in any way.

> or better yet a serial number.

Assuming this application currently doesn't use etags, how is it *now*
handling missed notifications? If it requires etags to handle them
correctly, then it's using etags in a way they're not meant to be used.

I would rather see the event package itself sport a mechanism that
allows it to carry this additional information about recent state
transitions, if necessary. IMHO, that is the only correct place for such
features.

Cheers,
Aki

---

From: Aki Niemi <[email-masked]>
Date: Thu, 06 Mar 2008 09:53:07 +0200

> know something of the history of that value?  (People have been

IMHO, this is not a use-case that conditional notification needs to
address.

> effectively require the use of full notifications at all times.

Correct. 

The draft also talks about this. If the notifier wants to do journaling,
it not only needs to generate unique etags, it also needs to *remember*
them, along with the entities. Once it has this history, it can generate
a diff based on the entity-tag in the Suppress-If-Match and the journal
it has stored.

> control information that is largely redundant with the ETags mechanism

Um, the 'version' parameter is largely redundant with the CSeq.

Cheers,
Aki

---

From: Dean Willis <[email-masked]>
Date: Thu, 06 Mar 2008 12:09:00 -0600

Aki Niemi wrote:
> the semantic is not: hmm, a notification. I know they're supposed to be

So if a changed etag doesn't mean the content changed, what the heck are
we sending an etag for?

---

From: [email-masked]
Date: Thu, 6 Mar 2008 16:10:30 -0500

From: Aki Niemi <[email-masked]>

And my original suggestion was that it not even attempt to address
it.  But rather that it provide a warning that under some
circumstances it could produce paradoxical results.  (I'm assuming
that you agree with me that these problems could arise.)

Unfortunately it's not.  But that isn't relevant.  The 'version'
parameter is *mandatory* for some event packages, and it makes ETags
(as they are now defined) nearly useless for those packages.

Dale

---

From: Aki Niemi <[email-masked]>
Date: Fri, 07 Mar 2008 08:49:03 +0200

> (as they are now defined) nearly useless for those packages.

Huh? I don't understand. Let's say I subscriber to some resource's
dialog state. The first NOTIFY carries the current state, and comes with
an etag. I refresh the subscription, and use Suppress-If-Match with that
etag. If the state has not changed, I won't get a notification. If I
choose to poll, the same thing will happen. I.e., I won't get a NOTIFY
but a 204 response to the SUBSCRIBE.

It seems I'm constantly paraphrasing the draft as we discuss this; there
are example message flows in the draft that show exactly how this works.
Can you walk me through an example of how etags won't work, please? 

Cheers,
Aki

---

From: Aki Niemi <[email-masked]>
Date: Fri, 07 Mar 2008 08:53:13 +0200

> So if a changed etag doesn't mean the content changed, what the heck are

We're sending it so that the next time the subscriber sends a SUBSCRIBE,
it can say: "Here's the last entity I've seen. Suppress-If-Match,
please". 

Cheers,
Aki

---

From: Dean Willis <[email-masked]>
Date: Fri, 7 Mar 2008 15:27:02 -0600

> it can say: "Here's the last entity I've seen. Suppress-If-Match,

But in the case we're discussing, the last entity the subscriber has  
seen is not the last entity that that the server has seen. And the  
entity that the subscriber currently has isn't the same entity that  
the server has -- it's another entity that just happens to have the  
same etag as the entity that the subscriber currently has.

That's a close enough match for some purposes (perhaps all of our  
purposes), but it's not the same thing.

--
Dean

---

From: Aki Niemi <[email-masked]>
Date: Sun, 09 Mar 2008 13:00:35 +0200

On pe, 2008-03-07 at 23:27 +0200, ext Dean Willis wrote:
> the server has

What makes "seen" and "has" different here?

> -- it's another entity that just happens to have the 

This is against what the draft says. We require unique etags "across all
versions of all entities for a resource and event package". 

Now I understand the text needs a little explaining, which I'll add,
about what "version" really means here, i.e., can two versions with
identical content also have identical etags? 

This is the special case that we need to explicitly call out. For
instance, an MD5 sum over the entity is a good enough entity-tag, even
though it will give the same entity-tag to different "versions" of
entities, albeit they're entities with identical content. 

> purposes), but it's not the same thing.

I still fail to see what significance this has for the subscriber? The
subscriber is at the mercy of the notifier, and it's the notifier's
responsibility to keep its entity-tags straight.

Cheers,
Aki

---

From: Dean Willis <[email-masked]>
Date: Sun, 9 Mar 2008 23:56:21 -0400

> What makes "seen" and "has" different here?

see below . . .

> versions of all entities for a resource and event package".

But that's not what you said earlier.

1) Let's say I have an resource that consists of the string "abcd" and  
the eTag "H".

2) The resource's string  value changes to "defg" and its eTag changes  
to "89"

3) It's value changes again, this time to "abcd". Can it have the eTag  
"H" again, or does it have to have an eTag that is unique across all  
versions of the resource?

Coming back to the "has" vs "seen" question.

if a watcher misses the "defg" eTag "89" phase at step 2for some  
reason. It sees state 1, then it sees state 3. If the eTag is allowed  
to repeat, it doesn't know it missed a state change. If the eTag is  
required to be unique, then the watcher knows SOMETHING happened, just  
not exactly what that was.

My current belief is for applications in which this matters, the  
object should contain a serial number or timestamp so that the  
object's value never repeats. This pretty much makes the question moot.

> identical content also have identical etags?

Yes! That is exactly the question.

> responsibility to keep its entity-tags straight.

It all depends on whether the application requires the subscriber to  
know about ALL transitions to the notifier state or perhaps requires  
knowing about null-transitions -- that is, where the object was  
"written" but its value did not change.

---

From: [email-masked]
Date: Mon, 10 Mar 2008 09:58:31 -0500

From: Dean Willis <[email-masked]>

   if a watcher misses the "defg" eTag "89" phase at step 2for some  
   reason. It sees state 1, then it sees state 3. If the eTag is allowed  
   to repeat, it doesn't know it missed a state change.

Well, there is a design question:  Is the goal to allow the subscriber
update its current knowledge of the state to match the current state,
or do we in addition want the subscriber to know reliably that there
have been state changes (even if their exact content cannot be known)?

You seem to think that knowing of state changes is essential, whereas
the draft considers that to not be a design goal.

I think to progress this discussion, we have to determine whether
reliable knowledge of the existence of state changes is a useful
feature.

Dale

---

From: [email-masked]
Date: Mon, 10 Mar 2008 10:17:48 -0500

The treacherous aspect is that etags will *work*, technically, but the
nature of the situation will prevent it from achieving any savings
(even though naively one would think that savings could be achived),
and that some attention should be paid to see if we can eliminate this
contradiction.

But let me walk through an example:

The critical fact is that some event types (esp. dialog and
resource-list) contain a "version" attribute.  Within any one
subscription, the first value is required to have version 1.
As the value changes, each successive value is required to increment
the version by 1.

(One consequence is that several subscribers to the same resource can
receive values that are logically the same, but which are textually
different, since each subscriber must receive a correct version within
his sequence of values.)

(I suspect that this mechanism was invented to support the "partial
update" facility, so that the subscriber could detect that he'd missed
an intermediate partial update.  But it's remarkably troublesome in
practice. E.g., in the sipX open-source system, we have one global
version number for all subscribers to a resource, and just violate the
rule that the first NOTIFY in a subscription must have version 1, and
hope that this won't trigger bad behavior in subscribers.)

(Dialog events have version numbers, but presence events do not.  But
resource-list events containing presence events do have version
numbers.)

Consider this sequence of events:

    Subscriber establishes a subscription, receives data A with version 1.

    Subscriber receives data B with version 2.

    Subscriber receives data C with version 3.

    Subscriber receives data D with version 4.

    Subscriber receives data E with version 5.

    Subscriber terminates subscription.

    Subscriber establishes a new subscription, receives data E with
    version 1.

The problem is that etags does not suppress sending of the data in the
last SUBSCRIBE, even though the subscriber already knows the *data*,
because the *value* contains the version 1, whereas the previous value
contained version 5, and so had a different etag.

One way to circumvent this problem is for the subscriber to always use
poll-type (zero-length) SUBSCRIBEs, so that all NOTIFY bodies would be
required to have version 1.  Then if the data is unchanged, the etag
would be unchanged.

IMHO, etags is part of the solution, but it overlaps with the existing
problems of the notification version number, and that a clean solution
is going to require thinking about these in an integrated manner.

Dale

---

From: Dean Willis <[email-masked]>
Date: Mon, 10 Mar 2008 13:42:22 -0400

> the draft considers that to not be a design goal.

I'm thinking that it might be essential for some applications, and  
that if the mechanism doesn't support it, then we simply need to  
document that the mechanism doesn't support it because there is an  
easy workaround by encoding version labels in the object itself.
> reliable knowledge of the existence of state changes is a useful

Well, that would help, but only if we think we need to change the spec  
to support the feature. I'm currently thinking that we do NOT need to  
change the spec to get this behavior -- we just need to clarify in the  
spec that it does not. This may be as simple as a one-sentence add  
(although I suspect it's more like a paragraph).

--
dean

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 10 Mar 2008 16:47:18 -0400

Lets contrast this to how this need to know all changes plays out 
*without* subnot-etags.

If you maintain an unbroken subscription without etags, you will receive 
every notification. (In the absence of throttling - see below.) If the 
state changes from A to B to A then you will know that. Using etags 
makes no sense.

If you instead poll rather than maintaining a subscription, and don't 
use etags, then if the state changes from A to B to A and your polls 
miss the time when the state is B, you have no way of knowing you missed 
anything. Adding etags does not change this. It just minimizes the data 
transfer on the second poll.

Note also that in the past this has come up in the context of event 
throttling. When event notification is throttled you also can miss 
certain states.

So it seems there has never been any intent to support the requirement 
to be aware of every state transition.

	Paul

---

From: Dean Willis <[email-masked]>
Date: Mon, 10 Mar 2008 18:12:21 -0400

> to be aware of every state transition.

Works for me. I talked with Aki this afternoon, and have asked him to  
just add the or three needed to make sure that readers don't assume  
that eTags give them something they didn't already have -- versioning.

I believe this will clear up EKR's comment as well. I'd like him to  
review Aki's new text when it becomes available.

So I think the issue is effectively closed for now, barring some  
wordsmithing that I do not expect will require meeting agenda time.

--
Dean