Subject: Re: [alto] New Incremental Update draft

From: Hans Seidel <[email-masked]>
Date: Mon, 2 Nov 2015 17:43:25 +0100

Wendy,

while working on our Incremental Update implementation is was thinking 
about the 3 solutions and decided to prefer option 1. I also like it for 
its simplicity while it still keeps start and stop separated.
However,if I read your mail correctly the stream-id field is only 
allowed in stop updates. That means adding resources to an existing 
stream is not supported. So, if a client wants to add resources it needs 
to create a new stream requesting the existing + the new resource(s)?

Furthermore, I updated our ALTO server 
(alto.benocs.berlin:8000/directory) and it now supports the 
incremental-updates draft -01. However, since we use the interop network 
to generate the maps
there are no changes (yet). Hence, every client will only receive the 
start event and the first resource data but no updates since there 
aren't any changes in the network.

Due to the lack of updates from our ALTO server I was also thinking 
about adding some (optional) incremental update tests to the interop 
draft. What do you think?

Hans

On 01.10.2015 17:53, Wendy Roome wrote:
> Hans,
>
> Thanks for pointing that out!  First, I assumed they were exclusive: a 
> request can have start-updates or stop-updates, but not both. But the 
> draft does not say that. My implementation does not support both in 
> one request, but does not return an error if a request does. (My 
> server just ignores the stop-updates field. My culpa!)
>
> But I just realized there is a use for having both in one request. 
> Suppose a client wants to stop a previous update-stream, and replace 
> it with a new stream. Combining both in the same request is more 
> efficient. On the other hand, is that going to happen often enough to 
> matter?
>
> To complicate things, a request with start-updates, stream-id and 
> stop-updates is ambiguous. The stop-updates part is clear. The 
> ambiguity is for the new resources in start-updates. Should the server 
> return updates for those on the new tcp connection? Or should the 
> server add the start-updates resources to the *other* stream - - the 
> one with the stream-id?
>
> I see several ways to fix that:
>
> 1. Say that start-updates & stop-updates are mutually exclusive. That 
> is, a request has start-updates, or stop-updates and stream-id, but no 
> other combination. And say the server must return an error if the 
> client violates that rule.
>
> 2. Allow both in one request. But then we have to decide whether the 
> start-updates resources are to be added to the existing stream, or to 
> be returned in a new stream.
>
> 3. Define two separate requests: start-new-update-stream and 
> modify-existing-update-stream. They would have different parameter 
> types, and hence different accepts media-types. That is unambiguous, 
> and is very general, and would allow a client to add resources to an 
> existing update stream. But that approach is more complicated, and I 
> do not know if the extra complication is worth it.
>
> What do the rest of you think?  My vote is for #1, just to keep it simple.
>
> - Wendy Roome
>
>
> From: Hans Seidel <[email-masked]>
> Date: Thu, October 1, 2015 at 10:45
> To: "Y. Richard Yang" <[email-masked] <mailto:[email-masked]>>, 
> Wendy Roome <[email-masked] 
> <mailto:[email-masked]>>
> Cc: "[email-masked]" <[email-masked] <mailto:[email-masked]>>
> Subject: Re: [alto] New Incremental Update draft
>
> Wendy, Richard,
>
> great work.
>
> One question that came up while reading:
>
> 1. Is it allowed to use "start-updates" and "stop-updates" in one HTTP 
> request? I found nothing specific that neither supports nor 
> contradicts that.
>     Intuitively, I would say no, it is not allowed to have both in one 
> request but I am not entirely convinced.
>
> Hans

---

From: Wendy Roome <[email-masked]>
Date: Mon, 02 Nov 2015 12:55:23 -0500

Comments inline.

> Date:  Mon, November 2, 2015 at 11:43
> To:  Wendy Roome <[email-masked]>, "Y. Richard Yang"
> <[email-masked]>
> Cc:  "[email-masked]" <[email-masked]>
> while working on our Incremental Update implementation is was thinking about
> the 3 solutions and decided to prefer option 1. I also like it for its
> simplicity while it still keeps start and stop separated.
> However,if I read your mail correctly the stream-id field is only allowed in
> stop updates. That means adding resources to an existing stream is not
> supported. So, if a client wants to add resources it needs to create a new
> stream requesting the existing + the new resource(s)?

That is my suggestion. Adding resources to an existing stream is messy. And
it creates a new class of ambiguity. Eg, suppose creates a update stream for
changes to an ECS query. Then suppose the client modifies the stream by
adding another ECS query. Update events are tagged with the ECS resource id
- - which is the same. So does the added ECS query replace the previous one?
Or should the server merge the query somehow? Or flag it as an error??

I suggest revising the request Update Stream request syntax slightly. The
client request has only one top-level field, start-updates or stop-updates.
start-updates is as before. stop-updates is an object with two fields,
stream-id and resources. stream-id is a string, and resources is an array of
ids of resources to stop. If resources is empty or omitted, stop all
resources.

That makes it clear that a client cannot add a resource to an existing
stream.
> 
> Furthermore, I updated our ALTO server (alto.benocs.berlin:8000/directory) and
> it now supports the incremental-updates draft -01. However, since we use the
> interop network to generate the maps
> there are no changes (yet). Hence, every client will only receive the start
> event and the first resource data but no updates since there aren't any
> changes in the network.

When I tried your server I immediately get EOF on the stream. However, I did
that from work, where I have to go through a very strict (and probably
ancient) proxy, and the proxy might not handle SSE. I will try tonight from
home.
> 
> Due to the lack of updates from our ALTO server I was also thinking about
> adding some (optional) incremental update tests to the interop draft. What do
> you think?
> 
> Hans

Yes, we would need that if we want the interop test to be a permanent
feature. Quick idea: define a new set of network specifications. Say one
network map and associated cost maps. Define several different cost maps,
which differ by a few cost points. The server would randomly switch between
those maps. The client would verify that the map is always one of the valid
one. And define two different versions of the network map. Same PID names,
but one or two CIDRs flip from one PID to another in the two maps. The
server would also alternate between those two network maps.

- Wendy Roome