Subject: Re: [Ecrit] planned-changes: two questions

From: "Caron, Guy" <[email-masked]>
Date: Wed, 1 Sep 2021 19:28:39 +0000

Ok. You have convinced me of the necessity of validating the URI before the Server stores it and that the security considerations in RFC5222 do not cover this threat.

I observe that this vulnerability was already mentioned in the Security section of the planned-changes draft:

The server is subject to abuse by clients because it is being asked to store something and may need to send data to an uncontrolled URI.

However, the original threat vector and associated response is probably insufficient to cover the DoS you raised, hence this new proposal to test the URI.

So, I’m conceptually favorable to adding a mechanism to test the URI before being stored by the Server.

I would like this mechanism to be as simple as possible and I think the goals can be achieved by testing the URI and receiving a response with at least one of the IDs previously supplied by the Server. I would leave it to policy at the Server to determine when and how often the URIs would be tested, with a recommendation (MUST? SHOULD?) to test it upon reception before storing it.

We probably need to say something about what to do if the URI test fails. Do we inform the Client that provided the URI that it may have been compromised? Do we simply drop the URI silently? Do we block the Client that provided the URI entirely?

I’m not sure if the <command> thing is necessary since it requires the Server to keep state. I would think that if the URI has been tested and stored, it means it is active and valid until it is deleted through the reception of an empty <plannedChange> for example, or replaced through the reception of a new URI in <plannedChange>.

Lastly, I’m not sure we still need the warning ‘uriNotStored’ given that we’re going with one generic URI per Client. The only argument I can think of for keeping it would be the proliferation of Clients within the coverage aria of the Server that would be so large that the Server couldn’t store them all. A counter argument to that would be that doing so could be seen as giving privileges to certain Clients over others.

Thanks,

Guy

De : Brian Rosen <[email-masked]>
Envoyé : 31 août 2021 13:54
À : Randall Gellens <[email-masked]>
Cc : Caron, Guy <[email-masked]>; ECRIT <[email-masked]>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Of course an attacker would conceal itself and register multiple URIs under different identities.  But the bigger  problem is that if a large planned change occurred, the victim could receive a large number of notifications it didn’t expect.

With a test transition, we know that the client expects to see notifications, we only need one per URI (we could specify one per domain — after the scheme and before the first slash).  If we don’t get the ID, we know not to allow that URI to be provisioned.  So it’s one test transaction vs a large number of planned change transactions.

Brian

On Aug 31, 2021, at 1:35 PM, Randall Gellens <[email-masked]<mailto:[email-masked]>> wrote:

If a malicious client registers a URI that is designed to attack a third site, the test transaction causes the LoST server to connect to it and POST a command. Without a test transaction, the LoST server stores the URI and at a future time connects to it and sends a command. Either way, a LoST client can register potentially one URI per queried location, and a LoST server will connect to that URI and POST a message. A queried location could be tied to many other locations, so there could be many locations for which a change would trigger the LoST server to use the URI, but is that worse?
--Randall
On 31 Aug 2021, at 10:27, Brian Rosen wrote:
If the work group wants to have the LoST server keep the URI until expressly deleted, that’s okay with me.

Authenticating the client to the server doesn’t mean the URI is authenticated.  We can’t restrict the URI to be the same entity as the client running the LoST transaction.  And the clients are wide ranging.  Could be an enterprise running its own LIS for example.  We can’t assume the North American PKI is workable everywhere, and even that doesn’t extend to an enterprise LIS.  ISTM we have to run a test transaction with the notification service to make sure it’s what we think it is.

Brian

On Aug 31, 2021, at 9:01 AM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

Inline.

Guy

-----Message d'origine-----
De : Brian Rosen <[email-masked]<mailto:[email-masked]>>
Envoyé : 31 août 2021 08:11
À : Caron, Guy <[email-masked]<mailto:[email-masked]>>
Cc : Randall Gellens <[email-masked]<mailto:[email-masked]>>; ECRIT <[email-masked]<mailto:[email-masked]>>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

You delete the URI when you delete the record in the LIS.
[GC] That's fine but that was not the question Randall posed. He asked whether the URI is deleted after a notification. I agree that if the Client does not host the location anymore that the URI associated with that location in the Server should be deleted. This could be achieved with an empty <plannedChange>.

I don’t think this is covered in 5222.  The mechanism causes the LoST server to send notifications to the client, but the client is allowed to put any URI in the record, and it can add it to as many records as it wants.
[GC] I thought we agreed on using one generic URI per Client. Clients should be authenticated by the Servers.
 An evil implementation could record URIs against multiple targets that were unaware that the evil implementation did it, until they got a large number of PUSH transactions they didn’t expect or understand as a result of a large planned change.
[GC] Only authenticated Clients should be allowed to provide URIs to be stored by the Servers.

The proposed mechanism qualifies the client URI before its used in a planned change.

Brian

On Aug 31, 2021, at 7:37 AM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

Well, this is not going in the direction I thought.

What is the purpose of deleting the URIs at the Server post-validation?

Regarding opening a new DoS, I guess I'm not following. Wouldn't this case be covered by the security considerations in RFC 5222?

What you're proposing puts back significant load on the Servers (a key consideration for creating planned-changes in the first place) and complicates the mechanism.

Guy

-----Message d'origine-----
De : Ecrit <[email-masked]<mailto:[email-masked]>> De la part de Brian Rosen Envoyé :
30 août 2021 11:22 À : Randall Gellens <[email-masked]<mailto:[email-masked]>> Cc
: ECRIT <[email-masked]<mailto:[email-masked]>> Objet : [EXT]Re: [Ecrit] planned-changes: two
questions

Answer 1: yes.  Since there is going to be a revalidation, just deleting the setting seems right to me.
Answer 2: Up to server.  If I were implementing, I would hash the real ID with the URI and some kind of predictable nonce.

We probably have to say more about how the server identifies the client, so that replacement of the URI works.  Could we say we use the domain of the URI (the entire domain with all the dots) to identify the client, and anything can occur after it (meaning a slash and whatever)?  If we do that, then how would delete the notification?  Force there to be something other than the domain (ugly).  Explicit delete request?

Hmmm, we’ve opened a DoS attack: a rogue client stores a bunch of URIs for servers it wants to victimize.  In North America we have a real simple solution for that, because we have a PKI, so we know, for sure, who the client is, and could restrict who we allow to store URIs, but that wouldn’t be true in general.  Also, it would be nice for the client to have confidence the mechanism worked before it needed it.

So
Let’s add a “command” to plannedChange in the findService request.
And, have the client have a response to the notification which is the
ID (json with the 200)

The client starts by sending a command of “initialize”.  The domain is the identity of the client.  The response is an immediate notification to the with whatever LI was in the request and an ID.  The  response by the client (which is the notification web server) is a piece of json containing the ID.  We can say that the LI in this initialize command could be something simple like the Country Code that wouldn’t get a planned change.

Thereafter, the LoST server (notification client) periodically repeats this keepalive notification every day or week with the initialize LI.  The client has to respond with the ID.

The regular notification request is a command of “notify”.  The server ignores a request for notification from an uninitialized client.
The notification can be deleted with a command of “delete”.  If you delete the initialize LI, then the server won’t send any more notifications to that client and deletes all URIs it was saving for that client.  The client would have to re-initialize to reset.

Brian

On Aug 27, 2021, at 5:41 PM, Randall Gellens <[email-masked]<mailto:[email-masked]>> wrote:

I think we're moving to a model where:
- In a query, a client can request to be notified when the location
should be revalidated;
- In the response, the server provides an ID which the client
associates with the location it just validated;
- The server sends a notification to the URI, containing the ID;
- The client revalidates each location with which that ID is associated.

Question 1: Does the server delete/inactivate the URI once it has sent the notification?

Question 2: Presumably, when the client revalidates the location(s), it will again request notification.  Does the server return the same ID as before, or a different ID?  A different ID could perhaps be useful in edge cases where the server didn't send or the client didn't get the notification, but any utility seems small.  If it's the same ID, then the answer to question 1 can be that the URI remains active until the client asks to no longer be notified (by sending an empty URI?).

--Randall

_______________________________________________
Ecrit mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ecrit

_______________________________________________
Ecrit mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ecrit
----------------------------------------------------------------------
-------- External Email: Please use caution when opening links and
attachments / Courriel externe: Soyez prudent avec les liens et
documents joints

------------------------------------------------------------------------------
External Email: Please use caution when opening links and attachments / Courriel externe: Soyez prudent avec les liens et documents joints

________________________________
External Email: Please use caution when opening links and attachments / Courriel externe: Soyez prudent avec les liens et documents joints

---

From: Brian Rosen <[email-masked]>
Date: Wed, 1 Sep 2021 19:16:37 -0400

Inline

> On Sep 1, 2021, at 3:28 PM, Caron, Guy <[email-masked]> wrote:
> 
Good

> I would like this mechanism to be as simple as possible and I think the goals can be achieved by testing the URI and receiving a response with at least one of the IDs previously supplied by the Ser…
I think you are suggesting we add text that says the LoST server can send a notification to the LoST client that has no changes, and just serves as a keep alive.  I’m sure we can make that work but generally, I find this kind of implicit test to be less satisfactory than an explicit keep alive transaction.   I get that an actual change may not be seen as a change to the client, and thus it works with no real code, but it doesn’t seem as clean as an explicit keep alive.  I’d be interested in herring from others on that
> We probably need to say something about what to do if the URI test fails. Do we inform the Client that provided the URI that it may have been compromised? Do we simply drop the URI silently? Do we…
At minimum wait and try again, the client could be down for a while.  I would not allow a lot of location records to be marked with the URI that didn’t test valid.

I am somewhat leery of creating circumstances where the client has to go through and revalidate it’s entire record set because the mechanism failed in some way.  There clearly would be circumstances where that happened, but missing a keep alive shouldn’t be one of them.  The initial one however should be a hard block.  If you can’t get the first ID returned, then we have to not accept other registrations.
> I’m not sure if the <command> thing is necessary since it requires the Server to keep state. I would think that if the URI has been tested and stored, it means it is active and valid until it is de…
I don’t think there is any server state other than the URI is valid.  Each transaction stands alone.  

Without the delete, you can’t remove one location record from the set that gets notified of planned changes.  You can change the URI, but that changes it for all notifications.  

> Lastly, I’m not sure we still need the warning ‘uriNotStored’ given that we’re going with one generic URI per Client. The only argument I can think of for keeping it would be the proliferation of C…

We’re creating circumstances where the client won’t get a notification.  I think we need to tell them that.  

One limit I might put on a client is how many locations they can ask to be notified.  If they try to set up a situation where no matter what changed, they get a notification, I think that may be too much to ask, and the server can have a limit.  It would be okay if it’s one URI per valid record at the LIS, but the LoST server doesn’t have that data.
> On Aug 31, 2021, at 1:35 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>  
> On Aug 31, 2021, at 9:01 AM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Inline.
> 
> Guy
> 
> De : Brian Rosen <[email-masked] <mailto:[email-masked]>> 
> À : Caron, Guy <[email-masked] <mailto:[email-masked]>>
> Cc : Randall Gellens <[email-masked] <mailto:[email-masked]>>; ECRIT <[email-masked] <mailto:[email-masked]>>
> On Aug 31, 2021, at 7:37 AM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
> 
> De : Ecrit <[email-masked] <mailto:[email-masked]>> De la part de Brian Rosen Envoyé : 
> 30 août 2021 11:22 À : Randall Gellens <[email-masked] <mailto:[email-masked]>> Cc 
> : ECRIT <[email-masked] <mailto:[email-masked]>> Objet : [EXT]Re: [Ecrit] planned-changes: two 
> questions
> 
> On Aug 27, 2021, at 5:41 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ecrit <https://www.ietf.org/mailman/listinfo/ecrit>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ecrit <https://www.ietf.org/mailman/listinfo/ecrit>

---

From: "Randall Gellens" <[email-masked]>
Date: Wed, 01 Sep 2021 16:51:39 -0700

I'm concerned that the mechanism not get too complex.  I think we might 
consider having servers consolidate the records to associate with a URI 
to limit the number of notifications, which can mitigate the ability to 
leverage the notification mechanism as an attack against unsuspecting 
entities.  We might allow a server to consolidate identical URIs and 
send up to some number of IDs in one notification.  We can require 
clients to support at least this number of IDs in one notification.

If we do have a test function, it should be blocking: the LoST server 
tests it immediately, and if it fails for any reason, the server returns 
uriNotStored and the client can try again later.

--Randall

On 1 Sep 2021, at 16:16, Brian Rosen wrote:

>> Ok. You have convinced me of the necessity of validating the URI 
>> before the Server stores it and that the security considerations in 
>> RFC5222 do not cover this threat.
>>
>> I observe that this vulnerability was already mentioned in the 
>> Security section of the planned-changes draft:
>>
>> The server is subject to abuse by clients because it is being asked 
>> to store something and may need to send data to an uncontrolled URI.
>>
>> However, the original threat vector and associated response is 
>> probably insufficient to cover the DoS you raised, hence this new 
>> proposal to test the URI.
>>
>> I would like this mechanism to be as simple as possible and I think 
>> the goals can be achieved by testing the URI and receiving a response 
>> with at least one of the IDs previously supplied by the Server. I 
>> would leave it to policy at the Server to determine when and how 
>> often the URIs would be tested, with a recommendation (MUST? SHOULD?) 
>> to test it upon reception before storing it.
> I think you are suggesting we add text that says the LoST server can 
> send a notification to the LoST client that has no changes, and just 
> serves as a keep alive.  I’m sure we can make that work but 
> generally, I find this kind of implicit test to be less satisfactory 
> than an explicit keep alive transaction.   I get that an actual change 
> may not be seen as a change to the client, and thus it works with no 
> real code, but it doesn’t seem as clean as an explicit keep alive.  
> I’d be interested in herring from others on that
>>
>> We probably need to say something about what to do if the URI test 
>> fails. Do we inform the Client that provided the URI that it may have 
>> been compromised? Do we simply drop the URI silently? Do we block the 
>> Client that provided the URI entirely?
> At minimum wait and try again, the client could be down for a while.  
> I would not allow a lot of location records to be marked with the URI 
> that didn’t test valid.
>
> I am somewhat leery of creating circumstances where the client has to 
> go through and revalidate it’s entire record set because the 
> mechanism failed in some way.  There clearly would be circumstances 
> where that happened, but missing a keep alive shouldn’t be one of 
> them.  The initial one however should be a hard block.  If you can’t 
> get the first ID returned, then we have to not accept other 
> registrations.
>>
>> I’m not sure if the <command> thing is necessary since it requires 
>> the Server to keep state. I would think that if the URI has been 
>> tested and stored, it means it is active and valid until it is 
>> deleted through the reception of an empty <plannedChange> for 
>> example, or replaced through the reception of a new URI in 
>> <plannedChange>.
> Without the delete, you can’t remove one location record from the 
> set that gets notified of planned changes.  You can change the URI, 
> but that changes it for all notifications.
>
>>
>> Lastly, I’m not sure we still need the warning ‘uriNotStored’ 
>> given that we’re going with one generic URI per Client. The only 
>> argument I can think of for keeping it would be the proliferation of 
>> Clients within the coverage aria of the Server that would be so large 
>> that the Server couldn’t store them all. A counter argument to that 
>> would be that doing so could be seen as giving privileges to certain 
>> Clients over others.
>
> We’re creating circumstances where the client won’t get a 
> notification.  I think we need to tell them that.
>
> One limit I might put on a client is how many locations they can ask 
> to be notified.  If they try to set up a situation where no matter 
> what changed, they get a notification, I think that may be too much to 
> ask, and the server can have a limit.  It would be okay if it’s one 
> URI per valid record at the LIS, but the LoST server doesn’t have 
> that data.
>>
>> Thanks,
>>
>> Guy
>>
>>
>> Of course an attacker would conceal itself and register multiple URIs 
>> under different identities.  But the bigger  problem is that if a 
>> large planned change occurred, the victim could receive a large 
>> number of notifications it didn’t expect.
>>
>> With a test transition, we know that the client expects to see 
>> notifications, we only need one per URI (we could specify one per 
>> domain — after the scheme and before the first slash).  If we 
>> don’t get the ID, we know not to allow that URI to be provisioned.  
>> So it’s one test transaction vs a large number of planned change 
>> transactions.
>>
>> Brian
>>
>>
>>
>> On Aug 31, 2021, at 1:35 PM, Randall Gellens 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> If a malicious client registers a URI that is designed to attack a 
>> third site, the test transaction causes the LoST server to connect to 
>> it and POST a command. Without a test transaction, the LoST server 
>> stores the URI and at a future time connects to it and sends a 
>> command. Either way, a LoST client can register potentially one URI 
>> per queried location, and a LoST server will connect to that URI and 
>> POST a message. A queried location could be tied to many other 
>> locations, so there could be many locations for which a change would 
>> trigger the LoST server to use the URI, but is that worse?
>> --Randall
>> Authenticating the client to the server doesn’t mean the URI is 
>> authenticated.  We can’t restrict the URI to be the same entity as 
>> the client running the LoST transaction.  And the clients are wide 
>> ranging.  Could be an enterprise running its own LIS for example.  We 
>> can’t assume the North American PKI is workable everywhere, and 
>> even that doesn’t extend to an enterprise LIS.  ISTM we have to run 
>> a test transaction with the notification service to make sure it’s 
>> what we think it is.
>>
>> Brian
>>
>>
>>
>>
>> Cc : Randall Gellens <[email-masked] 
>> <mailto:[email-masked]>>; ECRIT <[email-masked] 
>> <mailto:[email-masked]>>
>> [GC] That's fine but that was not the question Randall posed. He 
>> asked whether the URI is deleted after a notification. I agree that 
>> if the Client does not host the location anymore that the URI 
>> associated with that location in the Server should be deleted. This 
>> could be achieved with an empty <plannedChange>.
>>
>> I don’t think this is covered in 5222.  The mechanism causes the 
>> LoST server to send notifications to the client, but the client is 
>> allowed to put any URI in the record, and it can add it to as many 
>> records as it wants.
>>  An evil implementation could record URIs against multiple targets 
>> that were unaware that the evil implementation did it, until they got 
>> a large number of PUSH transactions they didn’t expect or 
>> understand as a result of a large planned change.
>> Regarding opening a new DoS, I guess I'm not following. Wouldn't this 
>> case be covered by the security considerations in RFC 5222?
>>
>> What you're proposing puts back significant load on the Servers (a 
>> key consideration for creating planned-changes in the first place) 
>> and complicates the mechanism.
>>
>> Guy
>>
>>
>> Answer 2: Up to server.  If I were implementing, I would hash the 
>> real ID with the URI and some kind of predictable nonce.
>>
>> We probably have to say more about how the server identifies the 
>> client, so that replacement of the URI works.  Could we say we use 
>> the domain of the URI (the entire domain with all the dots) to 
>> identify the client, and anything can occur after it (meaning a slash 
>> and whatever)?  If we do that, then how would delete the 
>> notification?  Force there to be something other than the domain 
>> (ugly).  Explicit delete request?
>>
>> Hmmm, we’ve opened a DoS attack: a rogue client stores a bunch of 
>> URIs for servers it wants to victimize.  In North America we have a 
>> real simple solution for that, because we have a PKI, so we know, for 
>> sure, who the client is, and could restrict who we allow to store 
>> URIs, but that wouldn’t be true in general.  Also, it would be nice 
>> for the client to have confidence the mechanism worked before it 
>> needed it.
>>
>> So
>> The client starts by sending a command of “initialize”.  The 
>> domain is the identity of the client.  The response is an immediate 
>> notification to the with whatever LI was in the request and an ID.  
>> The  response by the client (which is the notification web server) is 
>> a piece of json containing the ID.  We can say that the LI in this 
>> initialize command could be something simple like the Country Code 
>> that wouldn’t get a planned change.
>>
>> Thereafter, the LoST server (notification client) periodically 
>> repeats this keepalive notification every day or week with the 
>> initialize LI.  The client has to respond with the ID.
>>
>> The regular notification request is a command of “notify”.  The 
>> server ignores a request for notification from an uninitialized 
>> client.
>> The notification can be deleted with a command of “delete”.  If 
>> you delete the initialize LI, then the server won’t send any more 
>> notifications to that client and deletes all URIs it was saving for 
>> that client.  The client would have to re-initialize to reset.
>>
>> Brian
>>
>>
>> On Aug 27, 2021, at 5:41 PM, Randall Gellens 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> Question 2: Presumably, when the client revalidates the location(s), 
>> it will again request notification.  Does the server return the same 
>> ID as before, or a different ID?  A different ID could perhaps be 
>> useful in edge cases where the server didn't send or the client 
>> didn't get the notification, but any utility seems small.  If it's 
>> the same ID, then the answer to question 1 can be that the URI 
>> remains active until the client asks to no longer be notified (by 
>> sending an empty URI?).
>>
>> --Randall
>>
>> <https://www.ietf.org/mailman/listinfo/ecrit>
>>
>> <https://www.ietf.org/mailman/listinfo/ecrit>
>> External Email: Please use caution when opening links and attachments 
>> / Courriel externe: Soyez prudent avec les liens et documents joints
>>
>>
>> External Email: Please use caution when opening links and attachments 
>> / Courriel externe: Soyez prudent avec les liens et documents joints

---

From: "Caron, Guy" <[email-masked]>
Date: Thu, 2 Sep 2021 19:48:20 +0000

Inline under [GC].

If you agree with my comments inline below, here is how I see the process (building on what Randall initially provided on this thread):

1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated;
3) Immediately thereafter, the Server sends a ‘test’ notification to the URI, without any ID;
4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject silently the URI and block the Client permanently/provides a ‘uriNotStored’ warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.

Guy

De : Brian Rosen <[email-masked]>
Envoyé : 1 septembre 2021 19:17
À : Caron, Guy <[email-masked]>
Cc : Randall Gellens <[email-masked]>; ECRIT <[email-masked]>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Inline

On Sep 1, 2021, at 3:28 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

I would like this mechanism to be as simple as possible and I think the goals can be achieved by testing the URI and receiving a response with at least one of the IDs previously supplied by the Server. I would leave it to policy at the Server to determine when and how often the URIs would be tested, with a recommendation (MUST? SHOULD?) to test it upon reception before storing it.
I think you are suggesting we add text that says the LoST server can send a notification to the LoST client that has no changes, and just serves as a keep alive.  I’m sure we can make that work but generally, I find this kind of implicit test to be less satisfactory than an explicit keep alive transaction.   I get that an actual change may not be seen as a change to the client, and thus it works with no real code, but it doesn’t seem as clean as an explicit keep alive.  I’d be interested in herring from others on that
[GC] Could we not, for example, declare a value/element/attribute 'test' to be passed in the notification with no IDs? Would that meet your idea of an explicit test mechanism?

I am somewhat leery of creating circumstances where the client has to go through and revalidate it’s entire record set because the mechanism failed in some way.  There clearly would be circumstances where that happened, but missing a keep alive shouldn’t be one of them.  The initial one however should be a hard block.  If you can’t get the first ID returned, then we have to not accept other registrations.
[GC] On a second thought, what's the value of testing the URI after the first test? If it passed, it's valid and the Server stores it for use when a planned change occurs. If after that and for some reason, the Client is not responsive to the notification, does the Server care?

Without the delete, you can’t remove one location record from the set that gets notified of planned changes.  You can change the URI, but that changes it for all notifications.
[GC] Why? A Client that no longer wants to be notified for a given ID sends an empty <plannedChange> for the LI it no longer has. If that ID at the Server is associated with only one location (say one specific civic number), it deletes the Client's URI. If it is associated with more than one record (say a range of civic numbers), it accepts the transaction but keeps the URI. A notification for that ID sent to the Client that only had the civic number would result in a no-op.

We’re creating circumstances where the client won’t get a notification.  I think we need to tell them that.
[GC] Only if the URI fails the validity test.

One limit I might put on a client is how many locations they can ask to be notified.  If they try to set up a situation where no matter what changed, they get a notification, I think that may be too much to ask, and the server can have a limit.  It would be okay if it’s one URI per valid record at the LIS, but the LoST server doesn’t have that data.
[GC] It's one generic URI per Client with one or more IDs stored by the Client against its LIs, right? As you suggesting we go back to one URI per Client record in its database?

Thanks,

Guy

De : Brian Rosen <[email-masked]<mailto:[email-masked]>>
Envoyé : 31 août 2021 13:54
À : Randall Gellens <[email-masked]<mailto:[email-masked]>>
Cc : Caron, Guy <[email-masked]<mailto:[email-masked]>>; ECRIT <[email-masked]<mailto:[email-masked]>>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Brian

Brian

Inline.

Guy

-----Message d'origine-----
De : Brian Rosen <[email-masked]<mailto:[email-masked]>>
Envoyé : 31 août 2021 08:11
À : Caron, Guy <[email-masked]<mailto:[email-masked]>>
Cc : Randall Gellens <[email-masked]<mailto:[email-masked]>>; ECRIT <[email-masked]<mailto:[email-masked]>>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Brian

Guy

-----Message d'origine-----

So

Brian

--Randall

---

From: Brian Rosen <[email-masked]>
Date: Thu, 2 Sep 2021 17:29:28 -0400

> On Sep 2, 2021, at 3:48 PM, Caron, Guy <[email-masked]> wrote:
> 
> Inline under [GC].
>  
> 3) Immediately thereafter, the Server sends a ‘test’ notification to the URI, without any ID;\
[br[For every new ID?  Or just once?  I wanted this to be a one time registration.
> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject…
[br]I don’t think an explicit failure is a problem.  The LoST server can limit retries if it needs to.

> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAs…
[br]I wanted periodic keep alives.  How would that work?
> On Sep 1, 2021, at 3:28 PM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>  
[br] I suppose.  Doesn’t seem much different than a parameter to the findService, but I’m okay with it
> [GC] On a second thought, what's the value of testing the URI after the first test? If it passed, it's valid and the Server stores it for use when a planned change occurs. If after that and for som…
Generally when we have things that don’t do anything for months and then we need it, I would like to see periodic keep alives.  Things change on both sides and both sides have a vested interest in making sure the mechanism works when they need it.
> [GC] Why? A Client that no longer wants to be notified for a given ID sends an empty <plannedChange> for the LI it no longer has. If that ID at the Server is associated with only one location (say…
That means that the client has to be identified by something like the TLS cert, because the URI isn’t there for the delete.  I was using the domain of the URI as the ID of the client.

But we have another problem: I think the server does not track the LI.  On any findService with the new option, it looks for a matching record and returns the ID it has for that record.  It’s the client that tracks possible multiple LIs for one ID, not the server.  So in your example, the client would notice that any address in the range had the same ID.  It would get one notification if the range record changed, and it would update all its records in the range.  But if it deleted one of a set of records that have the same ID, it wouldn’t request delete.  Only if it got down to one record with a given ID, and it needed to delete that record, would it request delete for that ID at the server.  
> [GC] Only if the URI fails the validity test.  
[br]I wanted to allow the server to limit the number of IDs it was tracking for a given client.  I wanted to limit the number of queries the server is responding to (although it might well silently discard a DOS attack.  
> [GC] It's one generic URI per Client with one or more IDs stored by the Client against its LIs, right? As you suggesting we go back to one URI per Client record in its database?
[br]  One generic URI per client.  So if the client asks for its URI to be stored against too many IDs, it can be told no.
> À : Randall Gellens <[email-masked] <mailto:[email-masked]>>
> Cc : Caron, Guy <[email-masked] <mailto:[email-masked]>>; ECRIT <[email-masked] <mailto:[email-masked]>>

---

From: "Randall Gellens" <[email-masked]>
Date: Thu, 02 Sep 2021 14:45:17 -0700

I think a simpler approach is that when a query includes a URI, before 
returning the response, the server POSTS to the URI.  If it receives a 
success response, it stores the URI and returns the response.  If it 
receives a non-success response, it informs the client that the URI was 
not stored.  We can use uriNotStored or have a new warning such as 
uriTestFailure.  This way, the server does not need to maintain a 
pending URI and retry the test, and the client knows immediately if 
there's a problem.  Presumably, if the URI is bogus, the URI host will 
not return a success result.  I think that should be good enough, but if 
we want stronger protection, we could tie in the TLS authentication when 
the server does the POST.

--Randall

On 2 Sep 2021, at 14:29, Brian Rosen wrote:

>> If you agree with my comments inline below, here is how I see the 
>> process (building on what Randall initially provided on this thread):
>>
>> 1) In a validation query, a Client can request to be notified when 
>> the proffered LI should be revalidated, and provides a URI to send 
>> the notifications to;
>> 2) In the validation response, the Server provides an ID that the 
>> Client associates with the LI it just validated;
>> 4) The recipient at the URI is expected to respond with the ID 
>> provided in step 2. If it does, the Server stores the URI for future 
>> notifications. If it does not, the Server [let’s pick one: reject 
>> silently the URI and block the Client permanently/provides a 
>> ‘uriNotStored’ warning response to the URI {not compatible with 
>> the current proposal to test outside of LoST}/reject silently the URI 
>> and block the Client temporarily/other?];
>> 5) Some time after, the Server notifies the Client of an upcoming 
>> planned change by sending a notification to the successfully tested 
>> URI with the location ID;
>> 6) The client revalidates each LI in its database that matches the ID 
>> as of the date of the planned change. If no ID matches, it is a no-op 
>> at the Client. Revalidations may also result in no-op at the Client.
>> 7) LIs at the Client that are invalidated by the planned change are 
>> modified in its database to be valid (which probably mean another 
>> revalidation cycle) with an effective date set to <revalidateAsoF> 
>> value.
>> than an explicit keep alive transaction.   I get that an actual 
>> change may not be seen as a change to the client, and thus it works 
>> with no real code, but it doesn’t seem as clean as an explicit keep 
>> alive.  I’d be interested in herring from others on that
>> [GC] Could we not, for example, declare a value/element/attribute 
>> 'test' to be passed in the notification with no IDs? Would that meet 
>> your idea of an explicit test mechanism?
>> them.  The initial one however should be a hard block.  If you 
>> can’t get the first ID returned, then we have to not accept other 
>> registrations.
>> [GC] On a second thought, what's the value of testing the URI after 
>> the first test? If it passed, it's valid and the Server stores it for 
>> use when a planned change occurs. If after that and for some reason, 
>> the Client is not responsive to the notification, does the Server 
>> care?
> Generally when we have things that don’t do anything for months and 
> then we need it, I would like to see periodic keep alives.  Things 
> change on both sides and both sides have a vested interest in making 
> sure the mechanism works when they need it.
>>
>>
>>
>> [GC] Why? A Client that no longer wants to be notified for a given ID 
>> sends an empty <plannedChange> for the LI it no longer has. If that 
>> ID at the Server is associated with only one location (say one 
>> specific civic number), it deletes the Client's URI. If it is 
>> associated with more than one record (say a range of civic numbers), 
>> it accepts the transaction but keeps the URI. A notification for that 
>> ID sent to the Client that only had the civic number would result in 
>> a no-op.
> That means that the client has to be identified by something like the 
> TLS cert, because the URI isn’t there for the delete.  I was using 
> the domain of the URI as the ID of the client.
>
> But we have another problem: I think the server does not track the LI. 
>  On any findService with the new option, it looks for a matching 
> record and returns the ID it has for that record.  It’s the client 
> that tracks possible multiple LIs for one ID, not the server.  So in 
> your example, the client would notice that any address in the range 
> had the same ID.  It would get one notification if the range record 
> changed, and it would update all its records in the range.  But if it 
> deleted one of a set of records that have the same ID, it wouldn’t 
> request delete.  Only if it got down to one record with a given ID, 
> and it needed to delete that record, would it request delete for that 
> ID at the server.
>>
>>
>>
> [br]I wanted to allow the server to limit the number of IDs it was 
> tracking for a given client.  I wanted to limit the number of queries 
> the server is responding to (although it might well silently discard a 
> DOS attack.
>>
>>
>> what changed, they get a notification, I think that may be too much 
>> to ask, and the server can have a limit.  It would be okay if it’s 
>> one URI per valid record at the LIS, but the LoST server doesn’t 
>> have that data.
>> [GC] It's one generic URI per Client with one or more IDs stored by 
>> the Client against its LIs, right? As you suggesting we go back to 
>> one URI per Client record in its database?
> [br]  One generic URI per client.  So if the client asks for its URI 
> to be stored against too many IDs, it can be told no.
>>
>>
>>
>> Thanks,
>>
>> Guy
>>
>>

---

From: Brian Rosen <[email-masked]>
Date: Thu, 2 Sep 2021 17:51:45 -0400

You are proposing that you set up a TLS connection, POST on that connection, the server sets up a reverse connection, POSTs on that, the client responds to that POST, and then the server responds to that post?  Not a great plan I think.  If second TLS connection already exists, so you are asking to do a complete transaction B->A before the response to A->B I might be less worried.  Doesn’t sound like a great plan

Also, we need only do this once per client.  Not once per validation.  

Brian

> On Sep 2, 2021, at 5:45 PM, Randall Gellens <[email-masked]> wrote:
> 
> I think a simpler approach is that when a query includes a URI, before returning the response, the server POSTS to the URI. If it receives a success response, it stores the URI and returns the response. If it receives a non-success response, it informs the client that the URI was not stored. We can use uriNotStored or have a new warning such as uriTestFailure. This way, the server does not need to maintain a pending URI and retry the test, and the client knows immediately if there's a problem. Presumably, if the URI is bogus, the URI host will not return a success result. I think that should be good enough, but if we want stronger protection, we could tie in the TLS authentication when the server does the POST.
> 
> --Randall
> 
>> On Sep 2, 2021, at 3:48 PM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Inline under [GC].
>>

---

From: "Randall Gellens" <[email-masked]>
Date: Thu, 02 Sep 2021 15:04:20 -0700

We need to do it once per new URI.  If the URI already is stored, 
there's no reason to validate it again.

The overhead of a new TCP/TLS connection and one round-trip isn't much, 
and is only once per new URI.

Also, for URI deletion, we should have an explicit way to do this, 
perhaps a "delete" parameter along with the URI to be deleted.

--Randall

On 2 Sep 2021, at 14:51, Brian Rosen wrote:

> You are proposing that you set up a TLS connection, POST on that 
> connection, the server sets up a reverse connection, POSTs on that, 
> the client responds to that POST, and then the server responds to that 
> post?  Not a great plan I think.  If second TLS connection already 
> exists, so you are asking to do a complete transaction B->A before the 
> response to A->B I might be less worried.  Doesn’t sound like a 
> great plan
>
>
>> I think a simpler approach is that when a query includes a URI, 
>> before returning the response, the server POSTS to the URI. If it 
>> receives a success response, it stores the URI and returns the 
>> response. If it receives a non-success response, it informs the 
>> client that the URI was not stored. We can use uriNotStored or have a 
>> new warning such as uriTestFailure. This way, the server does not 
>> need to maintain a pending URI and retry the test, and the client 
>> knows immediately if there's a problem. Presumably, if the URI is 
>> bogus, the URI host will not return a success result. I think that 
>> should be good enough, but if we want stronger protection, we could 
>> tie in the TLS authentication when the server does the POST.
>>
>> --Randall
>>
>>> 3) Immediately thereafter, the Server sends a ‘test’ 
>>> notification to the URI, without any ID;\
>>> the current proposal to test outside of LoST}/reject silently the 
>>> URI and block the Client temporarily/other?];
>>> 6) The client revalidates each LI in its database that matches the 
>>> ID as of the date of the planned change. If no ID matches, it is a 
>>> no-op at the Client. Revalidations may also result in no-op at the 
>>> Client.
>>> the goals can be achieved by testing the URI and receiving a 
>>> response with at least one of the IDs previously supplied by the 
>>> Server. I would leave it to policy at the Server to determine when 
>>> and how often the URIs would be tested, with a recommendation (MUST? 
>>> SHOULD?) to test it upon reception before storing it.
>>> with no real code, but it doesn’t seem as clean as an explicit 
>>> keep alive.  I’d be interested in herring from others on that
>>> fails. Do we inform the Client that provided the URI that it may 
>>> have been compromised? Do we simply drop the URI silently? Do we 
>>> block the Client that provided the URI entirely?
>>>  I would not allow a lot of location records to be marked with the 
>>> URI that didn’t test valid.
>>>
>>> I am somewhat leery of creating circumstances where the client has 
>>> to go through and revalidate it’s entire record set because the 
>>> the first test? If it passed, it's valid and the Server stores it 
>>> for use when a planned change occurs. If after that and for some 
>>> reason, the Client is not responsive to the notification, does the 
>>> Server care?
>>> [GC] Why? A Client that no longer wants to be notified for a given 
>>> ID sends an empty <plannedChange> for the LI it no longer has. If 
>>> that ID at the Server is associated with only one location (say one 
>>> it accepts the transaction but keeps the URI. A notification for 
>>> that ID sent to the Client that only had the civic number would 
>>> result in a no-op.
>> But we have another problem: I think the server does not track the 
>> LI.  On any findService with the new option, it looks for a matching 
>>> Clients within the coverage aria of the Server that would be so 
>>> large that the Server couldn’t store them all. A counter argument 
>>> to that would be that doing so could be seen as giving privileges to 
>>> certain Clients over others.
>>>
>> the server is responding to (although it might well silently discard 
>> a DOS attack.
>>>
>>>
>>> Of course an attacker would conceal itself and register multiple 
>>> URIs under different identities.  But the bigger  problem is that if 
>>> a large planned change occurred, the victim could receive a large 
>>> third site, the test transaction causes the LoST server to connect 
>>> to it and POST a command. Without a test transaction, the LoST 
>>> server stores the URI and at a future time connects to it and sends 
>>> a command. Either way, a LoST client can register potentially one 
>>> URI per queried location, and a LoST server will connect to that URI 
>>> and POST a message. A queried location could be tied to many other 
>>> ranging.  Could be an enterprise running its own LIS for example.  
>>> We can’t assume the North American PKI is workable everywhere, and 
>>> even that doesn’t extend to an enterprise LIS.  ISTM we have to 
>>> run a test transaction with the notification service to make sure 
>>> it’s what we think it is.
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>>
>>> [GC] I thought we agreed on using one generic URI per Client. 
>>> Clients should be authenticated by the Servers.
>>> that were unaware that the evil implementation did it, until they 
>>> got a large number of PUSH transactions they didn’t expect or 
>>> Regarding opening a new DoS, I guess I'm not following. Wouldn't 
>>> this case be covered by the security considerations in RFC 5222?
>>>
>>> identify the client, and anything can occur after it (meaning a 
>>> slash and whatever)?  If we do that, then how would delete the 
>>> real simple solution for that, because we have a PKI, so we know, 
>>> for sure, who the client is, and could restrict who we allow to 
>>> store URIs, but that wouldn’t be true in general.  Also, it would 
>>> be nice for the client to have confidence the mechanism worked 
>>> before it needed it.
>>>
>>> So
>>> The  response by the client (which is the notification web server) 
>>> is a piece of json containing the ID.  We can say that the LI in 
>>> this initialize command could be something simple like the Country 
>>> Code that wouldn’t get a planned change.
>>>
>>> External Email: Please use caution when opening links and 
>>> External Email: Please use caution when opening links and 
>>> External Email: Please use caution when opening links and

---

From: Brian Rosen <[email-masked]>
Date: Fri, 3 Sep 2021 12:37:28 -0400

It would be great to hear from others on what they think of these ideas.  If we haven’t explained what we’re talking about sufficiently, what is unclear?

Do you like Randall’s “do the roundtrip notify before completing the findService response, which could then have an error response?  (Once per URI) or my “notify immediately after a new URI is presented, and periodically after that (keep alive) with no explicit error on the findService, noting that the client knows the notification is coming and if it doesn’t appear there was an error.
Do you think we should have an explicit URI-not-stored error or silently discard?
Do you think periodic keep alives (per URI) should be supported?
Do you think we need an explicit delete of an ID from triggering notify at a given URI?
If your answer to the above is “yes” would you prefer an explicit delete “command” or use something in the LI that somehow indicated delete?

Any other concerns?

> On Sep 2, 2021, at 6:04 PM, Randall Gellens <[email-masked]> wrote:
> 
> We need to do it once per new URI. If the URI already is stored, there's no reason to validate it again.
> 
> The overhead of a new TCP/TLS connection and one round-trip isn't much, and is only once per new URI.
> 
> Also, for URI deletion, we should have an explicit way to do this, perhaps a "delete" parameter along with the URI to be deleted.
> 
> --Randall
> 
>> On Sep 2, 2021, at 5:45 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Brian Rosen <[email-masked]>
Date: Fri, 3 Sep 2021 12:40:29 -0400

Dunno, seems pretty ugly to me.  Suppose the notify transaction ran close to some kind of timeout for some reason.  You might push the findService transaction over its timeout.  I’m probably reaching a bit there.

Brian

---

From: "Randall Gellens" <[email-masked]>
Date: Fri, 03 Sep 2021 14:19:58 -0700

I see the concern, but I think the alternative is uglier and more 
fragile.  It also seems more complex.  If there's a timeout during the 
query, the client tries again, probably after waiting a bit, which seems 
cleaner than noticing that it didn't get the confirmation POST.  Absent 
congestion or overload, a single TCP/TLS setup and POST round-trip 
should be very fast.

--Randall

On 3 Sep 2021, at 9:40, Brian Rosen wrote:

> Dunno, seems pretty ugly to me.  Suppose the notify transaction ran 
> close to some kind of timeout for some reason.  You might push the 
> findService transaction over its timeout.  I’m probably reaching a 
> bit there.
>
> Brian
>
>
>> the client responds to that POST, and then the server responds to 
>> that post?  Not a great plan I think.  If second TLS connection 
>> already exists, so you are asking to do a complete transaction B->A 
>> before the response to A->B I might be less worried.  Doesn’t sound 
>> like a great plan
>>
>>
>>> On Sep 2, 2021, at 5:45 PM, Randall Gellens 
>>> client that the URI was not stored. We can use uriNotStored or have 
>>> a new warning such as uriTestFailure. This way, the server does not 
>>>> provided in step 2. If it does, the Server stores the URI for 
>>>> future notifications. If it does not, the Server [let’s pick one: 
>>>> reject silently the URI and block the Client permanently/provides a 
>>>> and how often the URIs would be tested, with a recommendation 
>>>> (MUST? SHOULD?) to test it upon reception before storing it.
>>>> I think you are suggesting we add text that says the LoST server 
>>>> can send a notification to the LoST client that has no changes, and 
>>>> just serves as a keep alive.  I’m sure we can make that work but 
>>>> generally, I find this kind of implicit test to be less 
>>>> satisfactory than an explicit keep alive transaction.   I get that 
>>>> an actual change may not be seen as a change to the client, and 
>>>> thus it works with no real code, but it doesn’t seem as clean as 
>>>> an explicit keep alive.  I’d be interested in herring from others 
>>>> on that
>>>> 'test' to be passed in the notification with no IDs? Would that 
>>>> meet your idea of an explicit test mechanism?
>>>> At minimum wait and try again, the client could be down for a 
>>>> while.  I would not allow a lot of location records to be marked 
>>>> with the URI that didn’t test valid.
>>>>
>>> Generally when we have things that don’t do anything for months 
>>> and then we need it, I would like to see periodic keep alives.  
>>> Things change on both sides and both sides have a vested interest in 
>>> making sure the mechanism works when they need it.
>>>>
>>>>
>>>>
>>>> I’m not sure if the <command> thing is necessary since it 
>>>> requires the Server to keep state. I would think that if the URI 
>>>> has been tested and stored, it means it is active and valid until 
>>>> it is deleted through the reception of an empty <plannedChange> for 
>>>> associated with more than one record (say a range of civic 
>>>> numbers), it accepts the transaction but keeps the URI. A 
>>>> notification for that ID sent to the Client that only had the civic 
>>>> number would result in a no-op.
>>> That means that the client has to be identified by something like 
>>> the TLS cert, because the URI isn’t there for the delete.  I was 
>>> using the domain of the URI as the ID of the client.
>>>
>>> changed, and it would update all its records in the range.  But if 
>>> it deleted one of a set of records that have the same ID, it 
>>> wouldn’t request delete.  Only if it got down to one record with a 
>>> given ID, and it needed to delete that record, would it request 
>>> delete for that ID at the server.
>>>>
>>>>
>>>>
>>>> argument I can think of for keeping it would be the proliferation 
>>>> of Clients within the coverage aria of the Server that would be so 
>>>> to that would be that doing so could be seen as giving privileges 
>>>> to certain Clients over others.
>>>>
>>> tracking for a given client.  I wanted to limit the number of 
>>> queries the server is responding to (although it might well silently 
>>> discard a DOS attack.
>>>>
>>>>
>>>> One limit I might put on a client is how many locations they can 
>>>> ask to be notified.  If they try to set up a situation where no 
>>>> matter what changed, they get a notification, I think that may be 
>>>> too much to ask, and the server can have a limit.  It would be okay 
>>>> if it’s one URI per valid record at the LIS, but the LoST server 
>>>> doesn’t have that data.
>>>> URIs under different identities.  But the bigger  problem is that 
>>>> if a large planned change occurred, the victim could receive a 
>>>> large number of notifications it didn’t expect.
>>>>
>>>> don’t get the ID, we know not to allow that URI to be 
>>>> provisioned.  So it’s one test transaction vs a large number of 
>>>> planned change transactions.
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>>
>>>> URI per queried location, and a LoST server will connect to that 
>>>> URI and POST a message. A queried location could be tied to many 
>>>> other locations, so there could be many locations for which a 
>>>> change would trigger the LoST server to use the URI, but is that 
>>>> worse?
>>>> --Randall
>>>> authenticated.  We can’t restrict the URI to be the same entity 
>>>> as the client running the LoST transaction.  And the clients are 
>>>> wide ranging.  Could be an enterprise running its own LIS for 
>>>> example.  We can’t assume the North American PKI is workable 
>>>> everywhere, and even that doesn’t extend to an enterprise LIS.  
>>>> ISTM we have to run a test transaction with the notification 
>>>> service to make sure it’s what we think it is.
>>>>
>>>> Brian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Question 2: Presumably, when the client revalidates the 
>>>> location(s), it will again request notification.  Does the server 
>>>> return the same ID as before, or a different ID?  A different ID 
>>>> could perhaps be useful in edge cases where the server didn't send 
>>>> or the client didn't get the notification, but any utility seems 
>>>> small.  If it's the same ID, then the answer to question 1 can be 
>>>> that the URI remains active until the client asks to no longer be 
>>>> notified (by sending an empty URI?).
>>>>
>>>> --Randall
>>>>

---

From: Brian Rosen <[email-masked]>
Date: Tue, 7 Sep 2021 13:10:54 -0400

I assume many work group members were on vacation for the past several days (at least), and am resending this to poke you all for comment, please.

I’d like to get a new version out in the next few days so if you could opine, I would appreciate it.

Brian

> On Sep 3, 2021, at 12:37 PM, Brian Rosen <[email-masked]> wrote:
> 
>> On Sep 2, 2021, at 6:04 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Caron, Guy" <[email-masked]>
Date: Wed, 8 Sep 2021 00:31:27 +0000

Inline.

Guy

Envoyé : 2 septembre 2021 17:29

On Sep 2, 2021, at 3:48 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

Inline under [GC].

[GC] Just once per offered URI.

4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject silently the URI and block the Client permanently/provides a ‘uriNotStored’ warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
[br]I don’t think an explicit failure is a problem.  The LoST server can limit retries if it needs to.
[GC] Ok for HTTP failures but what I’m talking about is when it fails to return the ID, like in your DoS example.

[GC] You mean at step 3? As I mentioned below, I was wondering about the necessity for periodic tests. If the group is convinced it is needed, the Server can simply redo step 3 and the Client can respond with one or many IDs it has from that Server.

Guy

Inline

But we have another problem: I think the server does not track the LI.  On any findService with the new option, it looks for a matching record and returns the ID it has for that record.  It’s the client that tracks possible multiple LIs for one ID, not the server.  So in your example, the client would notice that any address in the range had the same ID.  It would get one notification if the range record changed, and it would update all its records in the range.  But if it deleted one of a set of records that have the same ID, it wouldn’t request delete.  Only if it got down to one record with a given ID, and it needed to delete that record, would it request delete for that ID at the server.
[GC] The ‘delete’ is only to tell the Server “I don’t want to be notified for that ID anymore” but in the grand scheme of things, this will be one of many (thousands?) IDs it got from the Server. Getting a notification that results in a no-op shouldn’t be too much to ask from the Client.

We’re creating circumstances where the client won’t get a notification.  I think we need to tell them that.
[GC] Only if the URI fails the validity test.
[br]I wanted to allow the server to limit the number of IDs it was tracking for a given client.
[GC] If we allow multiple IDs in the notification, does this concern go away?

 I wanted to limit the number of queries the server is responding to (although it might well silently discard a DOS attack.

One limit I might put on a client is how many locations they can ask to be notified.  If they try to set up a situation where no matter what changed, they get a notification, I think that may be too much to ask, and the server can have a limit.  It would be okay if it’s one URI per valid record at the LIS, but the LoST server doesn’t have that data.
[GC] It's one generic URI per Client with one or more IDs stored by the Client against its LIs, right? As you suggesting we go back to one URI per Client record in its database?
[br]  One generic URI per client.  So if the client asks for its URI to be stored against too many IDs, it can be told no.
[GC] Hummm... What recourse a Client would have in this circumstance for those LIs that the URI was rejected?

Thanks,

Guy

Brian

Brian

Inline.

Guy

-----Message d'origine-----
De : Brian Rosen <[email-masked]<mailto:[email-masked]>>
Envoyé : 31 août 2021 08:11
À : Caron, Guy <[email-masked]<mailto:[email-masked]>>
Cc : Randall Gellens <[email-masked]<mailto:[email-masked]>>; ECRIT <[email-masked]<mailto:[email-masked]>>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Brian

Guy

-----Message d'origine-----

So

Brian

--Randall

---

From: "Randall Gellens" <[email-masked]>
Date: Wed, 08 Sep 2021 12:15:40 -0700

Hi ECRIT Members,

There are a few technical choices to be made in the planned-changes 
draft.  Up to now it's been just Brian, Guy, and myself discussing the 
issues.  We need other group members to speak up, please.  If the issues 
aren't clear, please say what is confusing.

--Randall

Forwarded message:

> From: Brian Rosen <[email-masked]>
> To: Randall Gellens <[email-masked]>
> Cc: Caron, Guy <[email-masked]>, ECRIT <[email-masked]>
> Subject: Re: [Ecrit] planned-changes: two questions
> Date: Fri, 3 Sep 2021 12:37:28 -0400
>
> It would be great to hear from others on what they think of these 
> ideas.  If we haven’t explained what we’re talking about 
> sufficiently, what is unclear?
>
> Do you like Randall’s “do the roundtrip notify before completing 
> the findService response, which could then have an error response?  
> (Once per URI) or my “notify immediately after a new URI is 
> presented, and periodically after that (keep alive) with no explicit 
> error on the findService, noting that the client knows the 
> notification is coming and if it doesn’t appear there was an error.
> If your answer to the above is “yes” would you prefer an explicit 
> delete “command” or use something in the LI that somehow indicated 
> delete?
>
> Any other concerns?
>

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 15:17:48 -0400

Okay, I think the three of us are converging,  Here is a restatement of your description:
1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated.   The server may silently ignore repeated requests to store a URI where the test in 4 below fails.
3) Immediately thereafter, if the URI is new to the Server, the Server sends a ‘test’ notification to the URI, with an empty ID.
4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications.  If it does not, the server ignores the request to store the URI. 
5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
8) The Server may send ’test’ notifications to the URI without any ID as a form of “keep-alive”.  Any ID provided by the Server to the client may be used as the response to the test transaction

There has been a discussion of sending more than one ID in a transaction.  I think that is a decent idea, but I worry about how big that could be.  Either we put a hard limit in the text or have something in the response to the test transaction that specifies a size limit for that client.

Brian

> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[email-masked]> wrote:
>

---

From: "Caron, Guy" <[email-masked]>
Date: Wed, 8 Sep 2021 19:43:56 +0000

I think we are.

Small nit in step 7: <revalidateAsoF> should be <revalidateAsOf>

Related questions:

1.       Should it be exactly on <revalidateAsOf>?

2.       What will be the precision of that datetimestamp, hour, minute, second, sub-second (type="xs:dateTime" supports that level of granularity)?

To your question about multiple IDs in the notification (I assume this is what you meant by “more than one ID in a transaction”), I would leave it to the Server to determine. The Client should abide to the list of IDs it gets notification for but nothing precludes a Client to perform validations outside of the planned-changes mechanism. In fact, a Client may choose to revalidate its entire database periodically even if it supports planned-changes. I don’t like it but there is no text that precludes this to happen.

Guy

Envoyé : 8 septembre 2021 15:18

Brian

On Sep 7, 2021, at 8:31 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 15:53:58 -0400

> On Sep 8, 2021, at 3:43 PM, Caron, Guy <[email-masked]> wrote:
> 
> I think we are.
>  
Yes
> 2.       What will be the precision of that datetimestamp, hour, minute, second, sub-second (type="xs:dateTime" supports that level of granularity)?
I get the issue.  I think we should be reasonable.  Let’s say it’s a minute of granularity, and that a request to validate asOf the dateTime of revalidayeAsOf should be using the new LI.  So revalidateAsOf is the time new data is present, not the last time old data was present.
> To your question about multiple IDs in the notification (I assume this is what you meant by “more than one ID in a transaction”), I would leave it to the Server to determine. The Client should abid…
What I’m worried about is a million IDs in a single transaction when a city annexes a large suburb.  The client has to be prepared to accept the largest transaction the server will issue.  I want to either put a hard limit on it, or allow the client to tell the server how big the transaction is.  This is in units of IDs.  We limited the size of URIs, but with one URI per client, we can get rid of that.  If we allowed an ID to be as big as 256 bytes, a million IDs is 256Mbytes.  I don’t think clients should be obligated to accept that in one POST. 

> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: "Caron, Guy" <[email-masked]>
Date: Wed, 8 Sep 2021 20:03:19 +0000

Ah, your concern was for the Client. But I think the Server has more at stake here. If it puts too much IDs in the notification, it will be bombarded by revalidation requests that it may not be able to handle. It your large amalgamation example, if I was the Server, I would spread the notifications over a number of days (we probably should state that the Client SHOULD perform the revalidations within x hours of the notification). The Client would have to live with a large number of IDs anyway.

Guy

Envoyé : 8 septembre 2021 15:54

On Sep 8, 2021, at 3:43 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

I think we are.

Related questions:

Guy

Brian

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 16:16:55 -0400

Yes, the server can control the rate of revalidation by spacing out notifications, but so far the client can’t control the size of one notification.  If I built the LVF on any modern self scaling infrastructure, it could easily handle a million revalidations, one at a time in a short time by a bunch of clients, but a single client may fall over with a single 256M byte transaction.

I say this watching a web server do something weird on a 4Mbyte transaction.

I’m okay with a client getting 1M or 10M IDs notified in a short period of time, but not in a single transaction.

Brian

> On Sep 8, 2021, at 4:03 PM, Caron, Guy <[email-masked]> wrote:
> 
> On Sep 8, 2021, at 3:43 PM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>  
> I think we are.
>

---

From: "Caron, Guy" <[email-masked]>
Date: Wed, 8 Sep 2021 20:47:44 +0000

Isn’t this covered by HTTP configs, both at the server and client side? Would a POST with a body too large to handle by a Client be responded with 413 status code or something alike?

Guy

Envoyé : 8 septembre 2021 16:17

Brian

On Sep 8, 2021, at 4:03 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

Guy

I think we are.

Related questions:

Guy

Brian

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 16:53:17 -0400

Yeah, probably, but I don’t want the server to have to binary search to find what the client accepts when we could just have the client tell us at the beginning.  We’re already returning the ID as a flag to tell the server we’re good to go with the URI.  Just add a size limit there and it’s done.  Note that the response to the test transaction is already different from the response to an actual notification of a planned change.  The former returns the ID, the latter doesn’t return anything.

Brian

> On Sep 8, 2021, at 4:47 PM, Caron, Guy <[email-masked]> wrote:
> 
> On Sep 8, 2021, at 4:03 PM, Caron, Guy <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: "Caron, Guy" <[email-masked]>
Date: Wed, 8 Sep 2021 20:55:32 +0000

Ok. I’ll look at what you will propose then.

As I stated earlier, I would like this mechanism to be as simple as possible.

Thanks,

Guy

Envoyé : 8 septembre 2021 16:53

Brian

On Sep 8, 2021, at 4:47 PM, Caron, Guy <[email-masked]<mailto:[email-masked]>> wrote:

Guy

Brian

Guy

I think we are.

Related questions:

Guy

Brian

---

From: "Randall Gellens" <[email-masked]>
Date: Wed, 08 Sep 2021 14:11:02 -0700

The proposed mechanism seems fragile.  A client has no way to know if 
its request to store a URI was accepted.  That makes it hard to debug.  
If there are errors of any kind when the server initially verifies a 
URI, the client has no idea.  Even if the client repeats the 
transaction, the server will silently discard the URI.  To me, that's 
asking for problems.

In my proposal, when the client requests to be notified and provides a 
URI, the server immediately does a parallel POST to that URI.  If it 
gets a 202 Accepted response to the POST (or we can choose a different 
value), it stores the URI and returns the query result.  If it gets a 
different response, it does not store the URI and returns the query 
result with a uriNotStored warning.  If a client is trying to get the 
server to launch an attack on a third party entity, that entity will 
likely not support the URI in the first place, or will likely return an 
error response.  If you want additional verification, have the server 
include the queried location in its test POST.  That way, if the client 
maliciously sent a URI of a third party LIS, that LIS will know it does 
not have a query with that location outstanding.

Doing a parallel POST from the server to the client is one small 
transaction; this mechanism provides immediate feedback to both client 
and server.  If there are DNS failures or network congestion or 
whatever, the server returns uriNotStored and the client can try again 
later.

As for multiple IDs in a single notification POST, having the client 
specify in the initial request the maximum it is prepared to accept 
seems reasonable.  We can require that clients support a minimum number. 
  If a server has more than the maximum, it sends them in multiple 
POSTs, with whatever separation in time it chooses.  A server can 
include fewer IDs in a POST than the client's maximum.

We can have a 'test' notification value that is used both for the 
initial verification test and for periodic keep-alive tests.

--Randall

On 8 Sep 2021, at 12:17, Brian Rosen wrote:

> 1) In a validation query, a Client can request to be notified when the 
> proffered LI should be revalidated, and provides a URI to send the 
> notifications to;
> Client associates with the LI it just validated.   The server may 
> silently ignore repeated requests to store a URI where the test in 4 
> below fails.
> 3) Immediately thereafter, if the URI is new to the Server, the Server 
> sends a ‘test’ notification to the URI, with an empty ID.
> notifications.  If it does not, the server ignores the request to 
> store the URI.
> 8) The Server may send ’test’ notifications to the URI without any 
> ID as a form of “keep-alive”.  Any ID provided by the Server to 
> the client may be used as the response to the test transaction
>
>
> There has been a discussion of sending more than one ID in a 
> transaction.  I think that is a decent idea, but I worry about how big 
> that could be.  Either we put a hard limit in the text or have 
> something in the response to the test transaction that specifies a 
> size limit for that client.
>
> Brian
>
>> [GC] Ok for HTTP failures but what I’m talking about is when it 
>> fails to return the ID, like in your DoS example.
>>
>>
>>
>> [GC] You mean at step 3? As I mentioned below, I was wondering about 
>> the necessity for periodic tests. If the group is convinced it is 
>> needed, the Server can simply redo step 3 and the Client can respond 
>> with one or many IDs it has from that Server.

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 17:22:48 -0400

I don’t think there is any fragility in the URI setup.  For a new URI, the client requests the server to keep it in a findService.  That transaction concludes by sending an ID.  The server immediately sends the test notification, to which the client responds with the ID.  At that point both sides know the URI was accepted and the client is valid.  That may be repeated periodically so both sides know the other is happy with the arrangements.  If the client doesn’t get the test notification, it knows the URI isn’t accepted.  If the server doesn’t see the ID, it’s knows that was a bogus request, or there is some other problem, and it shouldn’t save the URI.

I proposed a null ID as the test transaction flag.  That would be a piece of XML with the right namespace, the right element name, but no value.  I think that is adequate and meets Guy’s minimal mechanism criteria.  The same XML is always sent.  It either has a set of IDs or it’s empty.

Brian

> On Sep 8, 2021, at 5:11 PM, Randall Gellens <[email-masked]> wrote:
> 
> The proposed mechanism seems fragile. A client has no way to know if its request to store a URI was accepted. That makes it hard to debug. If there are errors of any kind when the server initially verifies a URI, the client has no idea. Even if the client repeats the transaction, the server will silently discard the URI. To me, that's asking for problems.
> 
> In my proposal, when the client requests to be notified and provides a URI, the server immediately does a parallel POST to that URI. If it gets a 202 Accepted response to the POST (or we can choose a different value), it stores the URI and returns the query result. If it gets a different response, it does not store the URI and returns the query result with a uriNotStored warning. If a client is trying to get the server to launch an attack on a third party entity, that entity will likely not support the URI in the first place, or will likely return an error response. If you want additional verification, have the server include the queried location in its test POST. That way, if the client maliciously sent a URI of a third party LIS, that LIS will know it does not have a query with that location outstanding.
> 
> Doing a parallel POST from the server to the client is one small transaction; this mechanism provides immediate feedback to both client and server. If there are DNS failures or network congestion or whatever, the server returns uriNotStored and the client can try again later.
> 
> As for multiple IDs in a single notification POST, having the client specify in the initial request the maximum it is prepared to accept seems reasonable. We can require that clients support a minimum number. If a server has more than the maximum, it sends them in multiple POSTs, with whatever separation in time it chooses. A server can include fewer IDs in a POST than the client's maximum.
> 
> We can have a 'test' notification value that is used both for the initial verification test and for periodic keep-alive tests.
> 
> --Randall
>

---

From: "Randall Gellens" <[email-masked]>
Date: Wed, 08 Sep 2021 14:45:59 -0700

Why is a subsequent transaction better than a parallel one?  The 
subsequent transaction model delays knowledge of the state.

If the client doesn't see an immediate POST, it doesn't know if there's 
been an error or not.

If the client sees a POST and sends an ID, it doesn't know if the server 
received it and stored the URI.

If the server tries a POST but gets an error, it may retry but meanwhile 
the client may retry the query, since it didn't get a POST.

Should a client retain IDs if it hasn't seen a POST?

Also, requiring that the client send an ID seems like it may constrain 
client architectures since any part of the client system that receives a 
POST must know which IDs are pending.

What is the objection to a parallel POST?

--Randall

On 8 Sep 2021, at 14:22, Brian Rosen wrote:

> I don’t think there is any fragility in the URI setup.  For a new 
> URI, the client requests the server to keep it in a findService.  That 
> transaction concludes by sending an ID.  The server immediately sends 
> the test notification, to which the client responds with the ID.  At 
> that point both sides know the URI was accepted and the client is 
> valid.  That may be repeated periodically so both sides know the other 
> is happy with the arrangements.  If the client doesn’t get the test 
> notification, it knows the URI isn’t accepted.  If the server 
> doesn’t see the ID, it’s knows that was a bogus request, or there 
> is some other problem, and it shouldn’t save the URI.
>
> I proposed a null ID as the test transaction flag.  That would be a 
> piece of XML with the right namespace, the right element name, but no 
> value.  I think that is adequate and meets Guy’s minimal mechanism 
> criteria.  The same XML is always sent.  It either has a set of IDs or 
> it’s empty.
>
> Brian
>
>> In my proposal, when the client requests to be notified and provides 
>> a URI, the server immediately does a parallel POST to that URI. If it 
>> gets a 202 Accepted response to the POST (or we can choose a 
>> different value), it stores the URI and returns the query result. If 
>> it gets a different response, it does not store the URI and returns 
>> the query result with a uriNotStored warning. If a client is trying 
>> to get the server to launch an attack on a third party entity, that 
>> entity will likely not support the URI in the first place, or will 
>> likely return an error response. If you want additional verification, 
>> have the server include the queried location in its test POST. That 
>> way, if the client maliciously sent a URI of a third party LIS, that 
>> LIS will know it does not have a query with that location 
>> outstanding.
>>
>> transaction; this mechanism provides immediate feedback to both 
>> client and server. If there are DNS failures or network congestion or 
>> whatever, the server returns uriNotStored and the client can try 
>> again later.
>>
>> seems reasonable. We can require that clients support a minimum 
>> number. If a server has more than the maximum, it sends them in 
>> multiple POSTs, with whatever separation in time it chooses. A server 
>> can include fewer IDs in a POST than the client's maximum.
>>
>> 3) Immediately thereafter, if the URI is new to the Server, the 
>> Server sends a ‘test’ notification to the URI, with an empty ID.
>> 8) The Server may send ’test’ notifications to the URI without 
>> any ID as a form of “keep-alive”.  Any ID provided by the Server 
>> to the client may be used as the response to the test transaction
>>
>>
>> transaction.  I think that is a decent idea, but I worry about how 
>> big that could be.  Either we put a hard limit in the text or have

---

From: Brian Rosen <[email-masked]>
Date: Wed, 8 Sep 2021 19:31:33 -0400

Inline.  I think we need other opinions.  Certainly, we don’t have rough consensus.

> On Sep 8, 2021, at 5:45 PM, Randall Gellens <[email-masked]> wrote:
> 
> Why is a subsequent transaction better than a parallel one? The subsequent transaction model delays knowledge of the state.
> 
> If the client doesn't see an immediate POST, it doesn't know if there's been an error or not.
> 
Yes it does.  If it doesn’t see an immediate POST, then there was a problem and it has to try again.
> If the client sees a POST and sends an ID, it doesn't know if the server received it and stored the URI.
> 
Technically true, but vanishingly small issue.  You had a complete HTTPS transaction, but somehow the client isn’t sure the server got the ID. 
> If the server tries a POST but gets an error, it may retry but meanwhile the client may retry the query, since it didn't get a POST.
> 
That's always going to be something the server has to deal with: re-enrollment of the URI.  The client gets a normal response, and that tells it the URI was accepted.
> Should a client retain IDs if it hasn't seen a POST?
No, lack of POST means URI is not enrolled
> Also, requiring that the client send an ID seems like it may constrain client architectures since any part of the client system that receives a POST must know which IDs are pending.
> 
I don’t agree.  This is a pretty normal “I send you a magic number and you have to return it to me” thing.  There are no “pending IDs”.  This is the first ID, so you need to return it, and only it, but it’s the only ID the client has at this point.
> What is the objection to a parallel POST?
I was taught that you don’t embed an HTTP transaction in another with the same partner because things go wrong with timeouts and logic.  It’s not a parallel POST, it’s an HTTPS transaction one way that encloses another HTTPS transaction the other way.  You are dependent on the inner one completing before the outer one times out or has some other issue.  I just think that’s dangerous.  I will admit that I was taught this quite a long time ago (before HTTPS for sure) and things could have changed, but I can’t recall an API that does that.  Also note that keep alive works exactly the same way as the initial POST.  In your proposal, those are different.  That’s not much, but it’s some code.  The “test” transaction tells you that the mechanism works and your URI is being retained, whether that happens the first time, or a year later.
>> On Sep 8, 2021, at 5:11 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Randall Gellens" <[email-masked]>
Date: Wed, 08 Sep 2021 16:42:25 -0700

> problem and it has to try again.

Meanwhile, the server may be retrying.  There could have been a temp DNS 
issue or other temp failure

> Technically true, but vanishingly small issue.  You had a complete 
> HTTPS transaction, but somehow the client isn’t sure the server got 
> the ID.
>> If the server tries a POST but gets an error, it may retry but 
>> meanwhile the client may retry the query, since it didn't get a POST.
>>
> That's always going to be something the server has to deal with: 
> re-enrollment of the URI.  The client gets a normal response, and that 
> tells it the URI was accepted.

How long does the client retain the pending IDs waiting for a 
verification POST?

>> Also, requiring that the client send an ID seems like it may 
>> constrain client architectures since any part of the client system 
>> that receives a POST must know which IDs are pending.
>>
> I don’t agree.  This is a pretty normal “I send you a magic number 
> and you have to return it to me” thing.  There are no “pending 
> IDs”.  This is the first ID, so you need to return it, and only it, 
> but it’s the only ID the client has at this point.

So it's not a pending ID.  It's not used for notification, just 
verification.  That wasn't clear.

> I was taught that you don’t embed an HTTP transaction in another 
> with the same partner because things go wrong with timeouts and logic. 
>  It’s not a parallel POST, it’s an HTTPS transaction one way that 
> encloses another HTTPS transaction the other way.  You are dependent 
> on the inner one completing before the outer one times out or has some 
> other issue.  I just think that’s dangerous.  I will admit that I 
> was taught this quite a long time ago (before HTTPS for sure) and 
> things could have changed, but I can’t recall an API that does that.

It seems simpler than completing a transaction that isn't actually 
complete.  It seems simpler that both sides know the state, without 
ambiguity.

> Also note that keep alive works exactly the same way as the initial 
> POST.  In your proposal, those are different.  That’s not much, but 
> it’s some code.  The “test” transaction tells you that the 
> mechanism works and your URI is being retained, whether that happens 
> the first time, or a year later.

My proposal used the same 'test' in a POST  to the URI for both initial 
verification and subsequent keep-alives.

>> URI, the client requests the server to keep it in a findService.  
>> That transaction concludes by sending an ID.  The server immediately 
>> sends the test notification, to which the client responds with the 
>> ID.  At that point both sides know the URI was accepted and the 
>> client is valid.  That may be repeated periodically so both sides 
>> know the other is happy with the arrangements.  If the client 
>> doesn’t get the test notification, it knows the URI isn’t 
>> accepted.  If the server doesn’t see the ID, it’s knows that was 
>> a bogus request, or there is some other problem, and it shouldn’t 
>> save the URI.
>>
>> criteria.  The same XML is always sent.  It either has a set of IDs 
>> or it’s empty.
>>
>> Brian
>>
>>> On Sep 8, 2021, at 5:11 PM, Randall Gellens 
>>> its request to store a URI was accepted. That makes it hard to 
>>> debug. If there are errors of any kind when the server initially 
>>> verifies a URI, the client has no idea. Even if the client repeats 
>>> the transaction, the server will silently discard the URI. To me, 
>>> that's asking for problems.
>>>
>>> a URI, the server immediately does a parallel POST to that URI. If 
>>> it gets a 202 Accepted response to the POST (or we can choose a 
>>> likely return an error response. If you want additional 
>>> verification, have the server include the queried location in its 
>>> test POST. That way, if the client maliciously sent a URI of a third 
>>> party LIS, that LIS will know it does not have a query with that 
>>> location outstanding.
>>>
>>> client and server. If there are DNS failures or network congestion 
>>> or whatever, the server returns uriNotStored and the client can try 
>>> again later.
>>>
>>> multiple POSTs, with whatever separation in time it chooses. A 
>>> server can include fewer IDs in a POST than the client's maximum.
>>>
>>>> [GC] You mean at step 3? As I mentioned below, I was wondering 
>>>> about the necessity for periodic tests. If the group is convinced 
>>>> it is needed, the Server can simply redo step 3 and the Client can 
>>>> respond with one or many IDs it has from that Server.
>>>
>>

---

From: "Caron, Guy" <[email-masked]>
Date: Thu, 9 Sep 2021 13:51:32 +0000

I’m aligned with Brian on the proposal to embed a reverse HTTP transaction within an existing one. I think this is asking for trouble.

Guy

Envoyé : 8 septembre 2021 19:32

On Sep 8, 2021, at 5:45 PM, Randall Gellens <[email-masked]<mailto:[email-masked]>> wrote:

--Randall

Brian

On Sep 8, 2021, at 5:11 PM, Randall Gellens <[email-masked]<mailto:[email-masked]>> wrote:

Brian

---

From: James Kinney <[email-masked]>
Date: Thu, 9 Sep 2021 10:29:22 -0400

Forgive me if this is a duplicate, but I think I only responded to Randall
the first time. See forwarded email below.

Randall, I think I am caught up a little and I also asked for some feedback
from our LDB Location Subsystem team. My two cents are below, possibly more
to come from others...

I see the URI that is exposed by the LIS client as a callback or webhook. I
could be off on that, but it seems like the LIS is exposing a callback for
the ECRF/LVF to contact. I think it could be seen as a dynamic webhook.

   - In my opinion, you want the ECRF to test whether the URI is reachable
   before responding to the findService; so my recommendation so far is
   "round-trip before the response." That seems more explicit of a test,
   versus the client expecting a callback, timing-out, and indicating a
   failure from the lack of a signal.
   - However, even if you test before the response, I think the keep-alive
   could still be a good idea. This could allow a LIS to cancel a pending
   request, or it could exchange the URI for example. I am not sure any of
   those are necessary, but basically, you are communicating some form of
   mutual status of a planned change request if the ECRF is testing the
   webhook periodically.
   - The only opinion I have on *silent-discard* is that I would lean
   towards the solution that creates the least of a possible vector for
   exploit, e.g. reflection type attacks. On the surface for me that is
   silent-discard.
   - Yes on periodic, I think the minimal amount of traffic it adds is well
   worth the mutual status shared by exercising the URI and poking the
   association. A LIS could still use the lack of keep-alive(s) to alarm
   itself.
   - On the last two *ID* points I need to go back and do some more
   catch-up, I am not quite getting the question totally.

Best regards,
*James Kinney*

*Chief Innovation Officer | INdigital *

*direct: 260.469.2132*

*email: [email-masked] <[email-masked]>

*main office: *877.469.2010

*website: *indigital.net

*address: *1616 Directors Row, Fort Wayne, IN 46808

*join us online:* Facebook <https://www.facebook.com/INdigital911/> |
Twitter <http://twitter.com/indigital> | Linkedin
<https://www.linkedin.com/company/indigital911>

CONFIDENTIALITY: The information contained in this transmission is
privileged and confidential. It is intended to be received by the
individual or entity named above. If the reader or this message is not the
intended recipient, the reader is hereby notified that any consideration,
dissemination, or duplication of this communication is strictly prohibited.
This email and any attachments are confidential and may also be privileged.
If received in error, please do not disclose the contents to anyone, notify
the sender by return email, and delete this email and any attachments from
your system.

On Wed, Sep 8, 2021 at 3:15 PM Randall Gellens <[email-masked]>
wrote:

> Hi ECRIT Members,
>
> There are a few technical choices to be made in the planned-changes draft.
> Up to now it's been just Brian, Guy, and myself discussing the issues. We
> need other group members to speak up, please. If the issues aren't clear,
> please say what is confusing.
>
> --Randall
>
> Forwarded message:
>
> It would be great to hear from others on what they think of these ideas.
> If we haven’t explained what we’re talking about sufficiently, what is
> unclear?
>
> Do you like Randall’s “do the roundtrip notify before completing the
> findService response, which could then have an error response?  (Once per
> URI) or my “notify immediately after a new URI is presented, and
> periodically after that (keep alive) with no explicit error on the
> findService, noting that the client knows the notification is coming and if
> it doesn’t appear there was an error.
> If your answer to the above is “yes” would you prefer an explicit delete
> “command” or use something in the LI that somehow indicated delete?
>
> Any other concerns?
>
> Also, for URI deletion, we should have an explicit way to do this, perhaps
> a "delete" parameter along with the URI to be deleted.
>
> --Randall
>
> connection, the server sets up a reverse connection, POSTs on that, the
> client responds to that POST, and then the server responds to that post?
> Not a great plan I think.  If second TLS connection already exists, so you
> are asking to do a complete transaction B->A before the response to A->B I
> might be less worried.  Doesn’t sound like a great plan
>
>
> receives a non-success response, it informs the client that the URI was not
> stored. We can use uriNotStored or have a new warning such as
> uriTestFailure. This way, the server does not need to maintain a pending
> URI and retry the test, and the client knows immediately if there's a
> problem. Presumably, if the URI is bogus, the URI host will not return a
> success result. I think that should be good enough, but if we want stronger
> protection, we could tie in the TLS authentication when the server does the
> POST.
>
> --Randall
>
> If you agree with my comments inline below, here is how I see the process
> (building on what Randall initially provided on this thread):
>
> 2) In the validation response, the Server provides an ID that the Client
> associates with the LI it just validated;
> 4) The recipient at the URI is expected to respond with the ID provided in
> step 2. If it does, the Server stores the URI for future notifications. If
> it does not, the Server [let’s pick one: reject silently the URI and block
> the Client permanently/provides a ‘uriNotStored’ warning response to the
> URI {not compatible with the current proposal to test outside of
> LoST}/reject silently the URI and block the Client temporarily/other?];
>
> 5) Some time after, the Server notifies the Client of an upcoming planned
> change by sending a notification to the successfully tested URI with the
> location ID;
> 6) The client revalidates each LI in its database that matches the ID as
> of the date of the planned change. If no ID matches, it is a no-op at the
> Client. Revalidations may also result in no-op at the Client.
> revalidation cycle) with an effective date set to <revalidateAsoF> value.
>
> *De :* Brian Rosen <[email-masked]>
> *Envoyé :* 1 septembre 2021 19:17
> *À :* Caron, Guy <[email-masked]>
> *Cc :* Randall Gellens <[email-masked]>; ECRIT <[email-masked]>
> *Objet :* [EXT]Re: [Ecrit] planned-changes: two questions
>
> Inline
>
>
> Ok. You have convinced me of the necessity of validating the URI before
> the Server stores it and that the security considerations in RFC5222 do not
> cover this threat.
>
> The server is subject to abuse by clients because it is being asked to
> store something and may need to send data to an uncontrolled URI.
>
> However, the original threat vector and associated response is probably
> insufficient to cover the DoS you raised, hence this new proposal to test
> the URI.
>
> I would like this mechanism to be as simple as possible and I think the
> goals can be achieved by testing the URI and receiving a response with at
> least one of the IDs previously supplied by the Server. I would leave it to
> policy at the Server to determine when and how often the URIs would be
> tested, with a recommendation (MUST? SHOULD?) to test it upon reception
> before storing it.
>
> I think you are suggesting we add text that says the LoST server can send
> a notification to the LoST client that has no changes, and just serves as a
> keep alive.  I’m sure we can make that work but generally, I find this kind
> of implicit test to be less satisfactory than an explicit keep alive
> transaction.   I get that an actual change may not be seen as a change to
> the client, and thus it works with no real code, but it doesn’t seem as
> clean as an explicit keep alive.  I’d be interested in herring from others
> on that
> *[GC] Could we not, for example, declare a value/element/attribute 'test'
> to be passed in the notification with no IDs? Would that meet your idea of
> an explicit test mechanism?*
>
> We probably need to say something about what to do if the URI test fails.
> Do we inform the Client that provided the URI that it may have been
> compromised? Do we simply drop the URI silently? Do we block the Client
> that provided the URI entirely?
>
> At minimum wait and try again, the client could be down for a while.  I
> would not allow a lot of location records to be marked with the URI that
> didn’t test valid.
>
> I am somewhat leery of creating circumstances where the client has to go
> through and revalidate it’s entire record set because the mechanism failed
> in some way.  There clearly would be circumstances where that happened, but
> missing a keep alive shouldn’t be one of them.  The initial one however
> should be a hard block.  If you can’t get the first ID returned, then we
> have to not accept other registrations.
> *[GC] On a second thought, what's the value of testing the URI after the
> first test? If it passed, it's valid and the Server stores it for use when
> a planned change occurs. If after that and for some reason, the Client is
> not responsive to the notification, does the Server care?*
>
> Generally when we have things that don’t do anything for months and then
> we need it, I would like to see periodic keep alives.  Things change on
> both sides and both sides have a vested interest in making sure the
> mechanism works when they need it.
>
>
>
>
> I’m not sure if the <command> thing is necessary since it requires the
> Server to keep state. I would think that if the URI has been tested and
> stored, it means it is active and valid until it is deleted through the
> reception of an empty <plannedChange> for example, or replaced through the
> reception of a new URI in <plannedChange>.
>
> Without the delete, you can’t remove one location record from the set that
> gets notified of planned changes.  You can change the URI, but that changes
> it for all notifications.
> *[GC] Why? A Client that no longer wants to be notified for a given ID
> sends an empty <plannedChange> for the LI it no longer has. If that ID at
> the Server is associated with only one location (say one specific civic
> number), it deletes the Client's URI. If it is associated with more than
> one record (say a range of civic numbers), it accepts the transaction but
> keeps the URI. A notification for that ID sent to the Client that only had
> the civic number would result in a no-op.*
>
> That means that the client has to be identified by something like the TLS
> cert, because the URI isn’t there for the delete.  I was using the domain
> of the URI as the ID of the client.
>
> But we have another problem: I think the server does not track the LI.  On
> any findService with the new option, it looks for a matching record and
> returns the ID it has for that record.  It’s the client that tracks
> possible multiple LIs for one ID, not the server.  So in your example, the
> client would notice that any address in the range had the same ID.  It
> would get one notification if the range record changed, and it would update
> all its records in the range.  But if it deleted one of a set of records
> that have the same ID, it wouldn’t request delete.  Only if it got down to
> one record with a given ID, and it needed to delete that record, would it
> request delete for that ID at the server.
>
>
>
>
> Lastly, I’m not sure we still need the warning ‘uriNotStored’ given that
> we’re going with one generic URI per Client. The only argument I can think
> of for keeping it would be the proliferation of Clients within the coverage
> aria of the Server that would be so large that the Server couldn’t store
> them all. A counter argument to that would be that doing so could be seen
> as giving privileges to certain Clients over others.
>
>
> *[GC] Only if the URI fails the validity test.  *
>
> tracking for a given client.  I wanted to limit the number of queries the
> server is responding to (although it might well silently discard a DOS
> attack.
>
>
>
> One limit I might put on a client is how many locations they can ask to be
> notified.  If they try to set up a situation where no matter what changed,
> they get a notification, I think that may be too much to ask, and the
> server can have a limit.  It would be okay if it’s one URI per valid record
> at the LIS, but the LoST server doesn’t have that data.
> *[GC] It's one generic URI per Client with one or more IDs stored by the
> Client against its LIs, right? As you suggesting we go back to one URI per
> Client record in its database?*
>
> [br]  One generic URI per client.  So if the client asks for its URI to be
> stored against too many IDs, it can be told no.
>
>
>
>
> Thanks,
>
> Guy
>
>
> *De :* Brian Rosen <[email-masked]>
> *Envoyé :* 31 août 2021 13:54
> *À :* Randall Gellens <[email-masked]>
> *Cc :* Caron, Guy <[email-masked]>; ECRIT <[email-masked]>
> *Objet :* [EXT]Re: [Ecrit] planned-changes: two questions
>
> under different identities.  But the bigger  problem is that if a large
> planned change occurred, the victim could receive a large number of
> notifications it didn’t expect.
>
> notifications, we only need one per URI (we could specify one per domain —
> after the scheme and before the first slash).  If we don’t get the ID, we
> know not to allow that URI to be provisioned.  So it’s one test transaction
> vs a large number of planned change transactions.
>
> Brian
>
>
>
>
> On Aug 31, 2021, at 1:35 PM, Randall Gellens <[email-masked]>
> wrote:
>
> If a malicious client registers a URI that is designed to attack a third
> site, the test transaction causes the LoST server to connect to it and POST
> a command. Without a test transaction, the LoST server stores the URI and
> at a future time connects to it and sends a command. Either way, a LoST
> client can register potentially one URI per queried location, and a LoST
> server will connect to that URI and POST a message. A queried location
> could be tied to many other locations, so there could be many locations for
> which a change would trigger the LoST server to use the URI, but is that
> worse?
> --Randall
> authenticated.  We can’t restrict the URI to be the same entity as the
> client running the LoST transaction.  And the clients are wide ranging.
> Could be an enterprise running its own LIS for example.  We can’t assume
> the North American PKI is workable everywhere, and even that doesn’t extend
> to an enterprise LIS.  ISTM we have to run a test transaction with the
> notification service to make sure it’s what we think it is.
>
> Brian
>
>
>
>
>
> On Aug 31, 2021, at 9:01 AM, Caron, Guy <[email-masked]> wrote:
>
> Inline.
>
> Guy
>
> [GC] That's fine but that was not the question Randall posed. He asked
> whether the URI is deleted after a notification. I agree that if the Client
> does not host the location anymore that the URI associated with that
> location in the Server should be deleted. This could be achieved with an
> empty <plannedChange>.
>
> I don’t think this is covered in 5222.  The mechanism causes the LoST
> server to send notifications to the client, but the client is allowed to
> put any URI in the record, and it can add it to as many records as it wants.
>  An evil implementation could record URIs against multiple targets that
> were unaware that the evil implementation did it, until they got a large
> number of PUSH transactions they didn’t expect or understand as a result of
> a large planned change.
> On Aug 31, 2021, at 7:37 AM, Caron, Guy <[email-masked]> wrote:
>
> Regarding opening a new DoS, I guess I'm not following. Wouldn't this case
> be covered by the security considerations in RFC 5222?
>
> What you're proposing puts back significant load on the Servers (a key
> consideration for creating planned-changes in the first place) and
> complicates the mechanism.
>
> Guy
>
>
> De : Ecrit <[email-masked]> De la part de Brian Rosen Envoyé :
> 30 août 2021 11:22 À : Randall Gellens <[email-masked]> Cc
> : ECRIT <[email-masked]> Objet : [EXT]Re: [Ecrit] planned-changes: two
> questions
>
> Answer 2: Up to server.  If I were implementing, I would hash the real ID
> with the URI and some kind of predictable nonce.
>
> We probably have to say more about how the server identifies the client,
> so that replacement of the URI works.  Could we say we use the domain of
> the URI (the entire domain with all the dots) to identify the client, and
> anything can occur after it (meaning a slash and whatever)?  If we do that,
> then how would delete the notification?  Force there to be something other
> than the domain (ugly).  Explicit delete request?
>
> Hmmm, we’ve opened a DoS attack: a rogue client stores a bunch of URIs for
> servers it wants to victimize.  In North America we have a real simple
> solution for that, because we have a PKI, so we know, for sure, who the
> client is, and could restrict who we allow to store URIs, but that wouldn’t
> be true in general.  Also, it would be nice for the client to have
> confidence the mechanism worked before it needed it.
>
> So
> The client starts by sending a command of “initialize”.  The domain is the
> identity of the client.  The response is an immediate notification to the
> with whatever LI was in the request and an ID.  The  response by the client
> (which is the notification web server) is a piece of json containing the
> ID.  We can say that the LI in this initialize command could be something
> simple like the Country Code that wouldn’t get a planned change.
>
> Thereafter, the LoST server (notification client) periodically repeats
> this keepalive notification every day or week with the initialize LI.  The
> client has to respond with the ID.
>
> The regular notification request is a command of “notify”.  The server
> ignores a request for notification from an uninitialized client.
> The notification can be deleted with a command of “delete”.  If you delete
> the initialize LI, then the server won’t send any more notifications to
> that client and deletes all URIs it was saving for that client.  The client
> would have to re-initialize to reset.
>
> Brian
>
>
>
> On Aug 27, 2021, at 5:41 PM, Randall Gellens <[email-masked]>
> wrote:
>
> Question 2: Presumably, when the client revalidates the location(s), it
> will again request notification.  Does the server return the same ID as
> before, or a different ID?  A different ID could perhaps be useful in edge
> cases where the server didn't send or the client didn't get the
> notification, but any utility seems small.  If it's the same ID, then the
> answer to question 1 can be that the URI remains active until the client
> asks to no longer be notified (by sending an empty URI?).
>
> --Randall
>
> [email-masked]
> [email-masked]
> External Email: Please use caution when opening links and attachments /
> Courriel externe: Soyez prudent avec les liens et documents joints
>
>
>
>
> ------------------------------
> *External Email:* *Please use caution when opening links and attachments
> /* *Courriel externe:* *Soyez prudent avec les liens et documents joints*
>
>
> ------------------------------
> *External Email:** Please use caution when opening links and attachments
> / **Courriel externe:** Soyez prudent avec les liens et documents joints*
>
>
>
>
> [email-masked]

---

From: Jeff Martin <[email-masked]>
Date: Thu, 9 Sep 2021 20:08:44 +0000

Asking a server to store and later user a client-provided uri opens the door for abuse by malicious actors.   Which got me to thinking, what if we removed the client-provided uri entirely?  If no uris then no risk of abuse.

Keep the new optional <plannedChange asOf="..."> but without 'uri' in the client's <findServiceRequest>.  Keep the new optional <ttl> in the server's <findServiceResponse> as a hint to clients that may want to initiate revalidation from the client side.  Keep the new optional server "unique location ID" in the server response as discussed on this list (but not yet in published draft), which the client could choose to associate with the client's internal records.

The server periodically (server policy) issues a set of "potentially impacted" ids, where "potentially impacted" has already been discussed on this list. Each set includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.  The server keeps the sets going back some interval defined by server policy.

Client sends a "discovery" query to server, where server response is: the url(s) of the most recent and past set(s); and a TTL hint on how long until next list likely to be released.  Client then queries any of these urls, and server responds with the set that includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.

For clients that want to be highly proactive for changes, the client can store the "unique location id" from <findServiceResponse> and associate to the client's internal records.  The client then periodically queries the server for the set(s) of impacted IDs, and client gets a copy of each set. The client uses these sets of impacted ids from the servers list to check the clients internal records, and client can then revalidate those.

No more client-provided uris entirely removing the possibility of abuse by malicious actors.

The data in the issued set could be further extended to convey upcoming potential impacts to ids that have not yet been impacted.

/Jeff/

From: Ecrit <[email-masked]> On Behalf Of Caron, Guy
Sent: Thursday, September 9, 2021 7:52 AM
To: Brian Rosen <[email-masked]>; Randall Gellens <[email-masked]>
Cc: ECRIT <[email-masked]>
Subject: Re: [Ecrit] planned-changes: two questions

Guy

________________________________________________________
On Sep 8, 2021, at 5:45 PM, Randall Gellens <[email-masked]<mailto:[email-masked]>> wrote:

Brian

Brian

NOTICE TO RECIPIENT: This email, including attachments, may contain information which is confidential, proprietary, attorney-client privileged and / or controlled under U.S. export laws and regulations and may be restricted from disclosure by applicable State and Federal law. Nothing in this email shall create any legal binding agreement between the parties unless expressly stated herein and provided by an authorized representative of Comtech Telecommunications Corp. or its subsidiaries. If you are not the intended recipient of this message, be advised that any dissemination, distribution, or use of the contents of this message is strictly prohibited. If you received this message in error, please notify us immediately by return email and permanently delete all copies of the original email and any attached documentation from any computer or other media.

---

From: Brian Rosen <[email-masked]>
Date: Thu, 9 Sep 2021 17:14:54 -0400

I understand it, but I don’t want to do it.

We talk about this as planned changes, but we all know that sometimes, there is not a lot of planning.  The advantage to the notification is that it works immediately.  We have a polling mechanism (just revalidate). This mechanism is push = I’m telling you that it’s going to change at this time.  I’m reluctant to get away from that.

Note that if a client is worried about misuse, it has several things it can do.  The obvious one is don’t use it.  It can also use some infrastructure that is completely separate from the LoST server that it doesn’t mind being abused if it happens as the URI server.  It should be using mutual auth and know what the cert the LoST server has.  For North America, we have a PKI and that cert is going to be very good for that purpose.  I’ll make sure the Security Considerations section says that.

> On Sep 9, 2021, at 4:08 PM, Jeff Martin <[email-masked]> wrote:
> 
> From: Ecrit <[email-masked] <mailto:[email-masked]>> On Behalf Of Caron, Guy
> To: Brian Rosen <[email-masked] <mailto:[email-masked]>>; Randall Gellens <[email-masked] <mailto:[email-masked]>>
> Cc: ECRIT <[email-masked] <mailto:[email-masked]>>
> On Sep 8, 2021, at 5:45 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>  
> NOTICE TO RECIPIENT: This email, including attachments, may contain information which is confidential, proprietary, attorney-client privileged and / or controlled under U.S. export laws and regulations and may be restricted from disclosure by applicable State and Federal law. Nothing in this email shall create any legal binding agreement between the parties unless expressly stated herein and provided by an authorized representative of Comtech Telecommunications Corp. or its subsidiaries. If you are not the intended recipient of this message, be advised that any dissemination, distribution, or use of the contents of this message is strictly prohibited. If you received this message in error, please notify us immediately by return email and permanently delete all copies of the original email and any attached documentation from any computer or other media. _______________________________________________
> Ecrit mailing list

---

From: "Caron, Guy" <[email-masked]>
Date: Thu, 9 Sep 2021 21:20:36 +0000

+1

Guy

De : Ecrit <[email-masked]> De la part de Brian Rosen
Envoyé : 9 septembre 2021 17:15
À : Jeff Martin <[email-masked]>
Cc : [email-masked]
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

On Sep 9, 2021, at 4:08 PM, Jeff Martin <[email-masked]<mailto:[email-masked]>> wrote:

/Jeff/

From: Ecrit <[email-masked]<mailto:[email-masked]>> On Behalf Of Caron, Guy
Sent: Thursday, September 9, 2021 7:52 AM
To: Brian Rosen <[email-masked]<mailto:[email-masked]>>; Randall Gellens <[email-masked]<mailto:[email-masked]>>
Cc: ECRIT <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Ecrit] planned-changes: two questions

Guy

Brian

Brian

---

From: Brandon Abley <[email-masked]>
Date: Thu, 9 Sep 2021 17:42:19 -0400

I am catching up with this discussion quite late . . . I was on holiday
when this started and I am catching up with a long backlog. As a general
matter, I have been familiar with this draft for a very long time and
support it; this is a novel and extremely useful feature for dealing with
LoST servers as we start to see end-state NG9-1-1 deployment more broadly
as aside from jurisdictional changes, PSAP consolidation is quite common. I
will give the newest version a deep read with detailed comments once it is
available.

With the most recent proposal, I agree that the PKI(s) being used in North
America provide a good deal of protection against malicious use; the LoST
server will know at the time of authentication whether the traffic is valid
9-1-1 traffic and who it is coming from, and there can be built in logic to
accept e.g. only a request from Role "LoST"/"ECRF" from within the same PKI
to register planned changes. There is also from the 2014 Forest Guide
requirements document the concept of "internal" (residing within an ESInet)
versus "external" (residing outside of an ESInet) for ECRFs/LoST servers.
The external instance may be used for validation or testing by anyone, and
is broadly vulnerable to attack; while traffic allowed to resolve on an
internal instance is restricted only to ESInet-originated traffic (such as
an authoritative LoST server). I am not sure if this concept is worth
considering here also under Security Considerations.

-Brandon

On Thu, Sep 9, 2021 at 5:20 PM Caron, Guy <g.caron=[email-masked]>
wrote:

> +1
>
>
>
> Guy
>
>
>
> *De :* Ecrit <[email-masked]> *De la part de* Brian Rosen
> *Envoyé :* 9 septembre 2021 17:15
> *À :* Jeff Martin <[email-masked]>
> *Cc :* [email-masked]
> We talk about this as planned changes, but we all know that sometimes,
> there is not a lot of planning.  The advantage to the notification is that
> it works immediately.  We have a polling mechanism (just revalidate). This
> mechanism is push = I’m telling you that it’s going to change at this
> time.  I’m reluctant to get away from that.
>
>
>
> Note that if a client is worried about misuse, it has several things it
> can do.  The obvious one is don’t use it.  It can also use some
> infrastructure that is completely separate from the LoST server that it
> doesn’t mind being abused if it happens as the URI server.  It should be
> using mutual auth and know what the cert the LoST server has.  For North
> America, we have a PKI and that cert is going to be very good for that
> purpose.  I’ll make sure the Security Considerations section says that.
>
>
>
> Asking a server to store and later user a client-provided uri opens the
> door for abuse by malicious actors.   Which got me to thinking, what if we
> removed the client-provided uri entirely?  If no uris then no risk of abuse.
>
>
>
> Keep the new optional <plannedChange asOf="..."> but without 'uri' in the
> client's <findServiceRequest>.  Keep the new optional <ttl> in the server's
> <findServiceResponse> as a hint to clients that may want to initiate
> revalidation from the client side.  Keep the new optional server "unique
> location ID" in the server response as discussed on this list (but not yet
> in published draft), which the client could choose to associate with the
> client's internal records.
>
>
>
> The server periodically (server policy) issues a set of "potentially
> impacted" ids, where "potentially impacted" has already been discussed on
> this list. Each set includes: the ids that have been "potentially impacted"
> since the previous set was issued; the time-range covered; and a TTL hint
> on how long until the server is likely to issue the next list.  The server
> keeps the sets going back some interval defined by server policy.
>
>
>
> Client sends a "discovery" query to server, where server response is: the
> url(s) of the most recent and past set(s); and a TTL hint on how long until
> next list likely to be released.  Client then queries any of these urls,
> and server responds with the set that includes: the ids that have been
> "potentially impacted" since the previous set was issued; the time-range
> covered; and a TTL hint on how long until the server is likely to issue the
> next list.
>
>
> For clients that want to be highly proactive for changes, the client can
> store the "unique location id" from <findServiceResponse> and associate to
> the client's internal records.  The client then periodically queries the
> server for the set(s) of impacted IDs, and client gets a copy of each set.
> The client uses these sets of impacted ids from the servers list to check
> the clients internal records, and client can then revalidate those.
>
>
>
> The data in the issued set could be further extended to convey upcoming
> potential impacts to ids that have not yet been impacted.
>
>
>
>
>
> /Jeff/
>
>
>
> *From:* Ecrit <[email-masked]> *On Behalf Of *Caron, Guy
> *Sent:* Thursday, September 9, 2021 7:52 AM
> *To:* Brian Rosen <[email-masked]>; Randall Gellens <
> [email-masked]>
> *Cc:* ECRIT <[email-masked]>
> *Subject:* Re: [Ecrit] planned-changes: two questions
>
>
>
> I’m aligned with Brian on the proposal to embed a reverse HTTP transaction
> within an existing one. I think this is asking for trouble.
>
>
>
> Guy
>
>
>
> *Envoyé :* 8 septembre 2021 19:32
> Why is a subsequent transaction better than a parallel one? The subsequent
> transaction model delays knowledge of the state.
>
> Technically true, but vanishingly small issue.  You had a complete HTTPS
> transaction, but somehow the client isn’t sure the server got the ID.
>
> I don’t agree.  This is a pretty normal “I send you a magic number and you
> have to return it to me” thing.  There are no “pending IDs”.  This is the
> first ID, so you need to return it, and only it, but it’s the only ID the
> client has at this point.
>
> I was taught that you don’t embed an HTTP transaction in another with the
> same partner because things go wrong with timeouts and logic.  It’s not a
> parallel POST, it’s an HTTPS transaction one way that encloses another
> HTTPS transaction the other way.  You are dependent on the inner one
> completing before the outer one times out or has some other issue.  I just
> think that’s dangerous.  I will admit that I was taught this quite a long
> time ago (before HTTPS for sure) and things could have changed, but I can’t
> recall an API that does that.  Also note that keep alive works exactly the
> same way as the initial POST.  In your proposal, those are different.
> That’s not much, but it’s some code.  The “test” transaction tells you that
> the mechanism works and your URI is being retained, whether that
> happens the first time, or a year later.
>
> --Randall
>
> I don’t think there is any fragility in the URI setup.  For a new URI, the
> client requests the server to keep it in a findService.  That transaction
> concludes by sending an ID.  The server immediately sends the test
> notification, to which the client responds with the ID.  At that point both
> sides know the URI was accepted and the client is valid.  That may be
> repeated periodically so both sides know the other is happy with the
> arrangements.  If the client doesn’t get the test notification, it knows
> the URI isn’t accepted.  If the server doesn’t see the ID, it’s knows that
> was a bogus request, or there is some other problem, and it shouldn’t save
> the URI.
>
>
>
> I proposed a null ID as the test transaction flag.  That would be a piece
> of XML with the right namespace, the right element name, but no value.  I
> think that is adequate and meets Guy’s minimal mechanism criteria.  The
> same XML is always sent.  It either has a set of IDs or it’s empty.
>
>
>
> Brian
>
>
>
> The proposed mechanism seems fragile. A client has no way to know if its
> request to store a URI was accepted. That makes it hard to debug. If there
> are errors of any kind when the server initially verifies a URI, the client
> has no idea. Even if the client repeats the transaction, the server will
> silently discard the URI. To me, that's asking for problems.
>
> URI, the server immediately does a parallel POST to that URI. If it gets a
> 202 Accepted response to the POST (or we can choose a different value), it
> stores the URI and returns the query result. If it gets a different
> response, it does not store the URI and returns the query result with a
> uriNotStored warning. If a client is trying to get the server to launch an
> attack on a third party entity, that entity will likely not support the URI
> in the first place, or will likely return an error response. If you want
> additional verification, have the server include the queried location in
> its test POST. That way, if the client maliciously sent a URI of a third
> party LIS, that LIS will know it does not have a query with that location
> outstanding.
>
> transaction; this mechanism provides immediate feedback to both client and
> server. If there are DNS failures or network congestion or whatever, the
> server returns uriNotStored and the client can try again later.
>
> specify in the initial request the maximum it is prepared to accept seems
> reasonable. We can require that clients support a minimum number. If a
> server has more than the maximum, it sends them in multiple POSTs, with
> whatever separation in time it chooses. A server can include fewer IDs in a
> POST than the client's maximum.
>
> We can have a 'test' notification value that is used both for the initial
> verification test and for periodic keep-alive tests.
>
> --Randall
>
> associates with the LI it just validated.   The server may silently ignore
> repeated requests to store a URI where the test in 4 below fails.
> it does not, the server ignores the request to store the URI.
> 8) The Server may send ’test’ notifications to the URI without any ID as a
> form of “keep-alive”.  Any ID provided by the Server to the client may be
> used as the response to the test transaction
>
>
>
>
>
> There has been a discussion of sending more than one ID in a transaction.
> I think that is a decent idea, but I worry about how big that could be.
> Either we put a hard limit in the text or have something in the response to
> the test transaction that specifies a size limit for that client.
>
>
>
> Brian
>
>
>
> *[GC] Just once per offered URI.*
>
>
>
>
> *[GC] Ok for HTTP failures but what I’m talking about is when it fails to
> return the ID, like in your DoS example.*
>
>
>
>
>
>
> *[GC] You mean at step 3? As I mentioned below, I was wondering about the
> necessity for periodic tests. If the group is convinced it is needed, the
> Server can simply redo step 3 and the Client can respond with one or many
> IDs it has from that Server. *
>
>
>
>
>
> NOTICE TO RECIPIENT: This email, including attachments, may contain
> information which is confidential, proprietary, attorney-client privileged
> and / or controlled under U.S. export laws and regulations and may be
> restricted from disclosure by applicable State and Federal law. Nothing in
> this email shall create any legal binding agreement between the parties
> unless expressly stated herein and provided by an authorized representative
> of Comtech Telecommunications Corp. or its subsidiaries. If you are not the
> intended recipient of this message, be advised that any dissemination,
> distribution, or use of the contents of this message is strictly
> prohibited. If you received this message in error, please notify us
> immediately by return email and permanently delete all copies of the
> original email and any attached documentation from any computer or other
> media. _______________________________________________
> Ecrit mailing list
> [email-masked]

---

From: "Randall Gellens" <[email-masked]>
Date: Thu, 09 Sep 2021 16:02:14 -0700

I think the open issues are:

(1) Mitigating the risk that a malicious client will request 
notifications for a location chosen to generate a large number of 
notifications to a third party URI as a means of attacking that third 
party.

Mechanism (A): when a client query requests notifications, the server 
returns a query response that contains an ID for the queried location, 
without indicating if the URI has been stored or not.  Subsequently, the 
server performs a POST to the URI with an empty (or 'test') 
notification.  If the response to the POST includes the ID that was 
returned in the query result, the server stores the URI, otherwise it 
discards it.  If the server encounters errors performing the POST (e.g., 
temp DNS issues), it MAY retry at an interval of its choosing.  The 
client assumes that the URI was not stored unless it knows it responded 
to a POST from the server with the ID.  If the client does not know that 
the URI was stored, it may retry the query.  Once the client knows that 
the URI was stored, the client associates the ID with the queried 
location.

Mechanism (B): when a client query requests notifications, the server 
immediately performs a POST to the URI containing an empty (or 'test') 
notification.  If it receives a specified result (e.g., 202 Accepted), 
it stores the URI.  The query response includes an ID for the queried 
location.  If it does not get the expected response or is not able to 
perform the POST, it includes the waning 'uriNotStored' in the query 
response.  The client knows if the URI was stored or not, and if it was, 
it associates the ID with the queried location.

With either (A) or (B), if the server stores the URI, it associates it 
with the underlying location record.  If the server is aware that the 
underlying location will change (e.g., an annexation or zip code change 
or street name change), it uses POST to send a notification containing 
the ID to the stored URI.  It may send multiple IDs in one POST.  The 
server may also send periodic empty (or 'test') notifications as a 
ping/keep-alive mechanism to verify that the URI still works.

Please remind me again why we can't use a simple mechanism of requiring 
that the domain of the URI match the domain in the client's certificate 
when TLS was negotiated for the initial request?

(2) Periodic test/keep-alive POSTs to saved URIs.

(3) Multiple IDs in one notification.  Clients may validate and request 
notifications for many locations; there may be a change that affects a 
large number of locations (e.g., an annexation).  Do we want to permit 
multiple IDs in a notification?  Is there a limit?  One proposal is that 
in the initial query where a client requests notifications, it may 
specify a maximum number of IDs in one POST.  The server stores this 
with the URI.  A server may send fewer IDs than the maximum but not 
more.  We can mandate a minimum number of IDs that clients MUST accept.

(4) A mechanism for a client to withdraw a URI.  This could be a query 
for the same location as was used to request that the URI be stored, 
with an indication that the URI should be deleted.  We can choose that 
this be an empty URI value or an explicit 'delete' attribute.

--Randall

On 9 Sep 2021, at 14:42, Brandon Abley wrote:

>> No more client-provided uris entirely removing the possibility of 
>> abuse by
>> malicious actors.
>>
>>
>>
>> Inline.  I think we need other opinions.  Certainly, we don’t have 
>> rough
>> consensus.
>>
>>
>>
>>
>>
>> On Sep 8, 2021, at 5:45 PM, Randall Gellens 
>> <[email-masked]>
>> wrote:
>>
>>
>>
>> subsequent
>> HTTPS
>> meanwhile
>> constrain
>> with the
>> URI, the
>> piece
>> <[email-masked]>
>> wrote:
>>
>>
>>
>> its
>> a
>> gets a
>> location
>> client and
>> seems
>> initial
>> Okay, I think the three of us are converging,  Here is a restatement 
>> of
>> your description:
>>
>> the
>> Client
>> Server
>> provided in
>> planned
>> as
>> any ID as a
>> transaction.
>> the
>> Client
>> 3) Immediately thereafter, the Server sends a ‘test’ notification 
>> to the
>> URI, without any ID;\
>>
>> provided in
>> [br]I don’t think an explicit failure is a problem.  The LoST 
>> server can
>> limit retries if it needs to.
>>
>> planned
>> as

---

From: Dan Banks <[email-masked]>
Date: Fri, 10 Sep 2021 16:13:17 +0000

I am opposed to asking LoST servers to store URIs as well.  I believe it significantly overcomplicates the solution and is the biggest reason why I have not supported the planned changes draft.

Several years ago we implemented a mechanism to allow a LIS to know that it needs to revalidate.  Roughly described, that mechanism consists of a web API that can be queried for changes after a particular time.  The response is a summary of changes in terms of partial civic addresses that allow the LIS to identify records that are likely to be affected, or possibly that there have been so many changes that a full revalidation is recommended.  We’ve made the API discoverable via U-NAPTR using the same app string as a given LoST server.

Yes, the LIS has to poll the various LoST servers and ask if there have been any changes.  That is a cheap operation to implement, and the practical difference between knowing the very instant a change is published versus finding out within a few minutes is not significant in any way.

Dan

From: Ecrit <[email-masked]> On Behalf Of Jeff Martin
Sent: Thursday, September 9, 2021 4:09 PM
To: [email-masked]
Subject: Re: [Ecrit] planned-changes: two questions

/Jeff/

Guy

Brian

Brian

---

From: Brian Rosen <[email-masked]>
Date: Fri, 10 Sep 2021 12:30:12 -0400

Minutes?

So the clients of this service poll for changes every few minutes?

What is a “partial civic address”?  Example?  Something like an A1/A2/A4 but no street name/number, meaning any address in this unincorporated community name?

ISTM that a planned change quite often can’t be fixed in a way that code could do it.  So, we still need to advertise the change well in advance of the change, and have the ability to do the AsOf validation.  Changing the way the client discovers that a change is coming is one thing, not knowing well in advance is, I think, unacceptable. 

Brian

> On Sep 10, 2021, at 12:13 PM, Dan Banks <dbanks=[email-masked]> wrote:
>

---

From: Jeff Martin <[email-masked]>
Date: Fri, 10 Sep 2021 20:11:12 +0000

Brian wrote:
> [the client] not knowing well in advance [of a server change] is, I think, unacceptable

In my outlined suggestion, I ended with:
The data in the [server] issued set could be further extended to convey upcoming potential impacts to ids that have not yet been impacted.

The server's issued set(s) could contain 'invalidAsOf' indicators with the locationIds, similar to the draft's server-pushed <locationInvalidated invalidAsOf="..." ...>.   So if a server knows "well in advance", say 24 hours, as long as the clients pull within 24 hours the clients would also know well in advance.   My outlined suggestion also includes server-provided TTL in server responses to give hints to clients about polling frequency.

Balancing the tradeoffs of polling vs uri+push requires answering this question:  What will be the typical and/or exceptional lead-times for servers operators to know about upcoming changes?

  *   If it's common that server operators themselves often identify changes with less than one-hour notice, then the complexity of client-registered-uri with server-push is more justified.

  *   But if it's common that server operators usually identify changes with one day or more notice, and one-hour notice is very uncommon, then the simplicity of polling is more justified.

IMO based on experience at my company, days of lead time (not hours) is typical for server operators dealing with "planned changes" that affect validity.  So the simplicity of polling with responses that include  'invalidAsOf' is better than the complexity of uri+push, whose benefits would rarely be realized in an environment dominated by days  of lead time.

/Jeff/

From: Brian Rosen <[email-masked]>
Sent: Friday, September 10, 2021 10:30 AM
To: Dan Banks <dbanks=[email-masked]>
Cc: Jeff Martin <[email-masked]>; [email-masked]
Subject: Re: [Ecrit] planned-changes: two questions

Minutes?

What is a "partial civic address"?  Example?  Something like an A1/A2/A4 but no street name/number, meaning any address in this unincorporated community name?

ISTM that a planned change quite often can't be fixed in a way that code could do it.  So, we still need to advertise the change well in advance of the change, and have the ability to do the AsOf validation.  Changing the way the client discovers that a change is coming is one thing, not knowing well in advance is, I think, unacceptable.

Brian

___________________________________________________________________
On Sep 10, 2021, at 12:13 PM, Dan Banks <dbanks=[email-masked]> wrote:

Several years ago we implemented a mechanism to allow a LIS to know that it needs to revalidate.  Roughly described, that mechanism consists of a web API that can be queried for changes after a particular time.  The response is a summary of changes in terms of partial civic addresses that allow the LIS to identify records that are likely to be affected, or possibly that there have been so many changes that a full revalidation is recommended.  We've made the API discoverable via U-NAPTR using the same app string as a given LoST server.

Dan

/Jeff/

I'm aligned with Brian on the proposal to embed a reverse HTTP transaction within an existing one. I think this is asking for trouble.

Guy

Inline.  I think we need other opinions.  Certainly, we don't have rough consensus.

Why is a subsequent transaction better than a parallel one? The subsequent transaction model delays knowledge of the state.
If the client doesn't see an immediate POST, it doesn't know if there's been an error or not.
Yes it does.  If it doesn't see an immediate POST, then there was a problem and it has to try again.
If the client sees a POST and sends an ID, it doesn't know if the server received it and stored the URI.
Technically true, but vanishingly small issue.  You had a complete HTTPS transaction, but somehow the client isn't sure the server got the ID.
If the server tries a POST but gets an error, it may retry but meanwhile the client may retry the query, since it didn't get a POST.
That's always going to be something the server has to deal with: re-enrollment of the URI.  The client gets a normal response, and that tells it the URI was accepted.
Should a client retain IDs if it hasn't seen a POST?
No, lack of POST means URI is not enrolled
Also, requiring that the client send an ID seems like it may constrain client architectures since any part of the client system that receives a POST must know which IDs are pending.
I don't agree.  This is a pretty normal "I send you a magic number and you have to return it to me" thing.  There are no "pending IDs".  This is the first ID, so you need to return it, and only it, but it's the only ID the client has at this point.
What is the objection to a parallel POST?
I was taught that you don't embed an HTTP transaction in another with the same partner because things go wrong with timeouts and logic.  It's not a parallel POST, it's an HTTPS transaction one way that encloses another HTTPS transaction the other way.  You are dependent on the inner one completing before the outer one times out or has some other issue.  I just think that's dangerous.  I will admit that I was taught this quite a long time ago (before HTTPS for sure) and things could have changed, but I can't recall an API that does that.  Also note that keep alive works exactly the same way as the initial POST.  In your proposal, those are different.  That's not much, but it's some code.  The "test" transaction tells you that the mechanism works and your URI is being retained, whether that happens the first time, or a year later.
--Randall
On 8 Sep 2021, at 14:22, Brian Rosen wrote:
I don't think there is any fragility in the URI setup.  For a new URI, the client requests the server to keep it in a findService.  That transaction concludes by sending an ID.  The server immediately sends the test notification, to which the client responds with the ID.  At that point both sides know the URI was accepted and the client is valid.  That may be repeated periodically so both sides know the other is happy with the arrangements.  If the client doesn't get the test notification, it knows the URI isn't accepted.  If the server doesn't see the ID, it's knows that was a bogus request, or there is some other problem, and it shouldn't save the URI.

I proposed a null ID as the test transaction flag.  That would be a piece of XML with the right namespace, the right element name, but no value.  I think that is adequate and meets Guy's minimal mechanism criteria.  The same XML is always sent.  It either has a set of IDs or it's empty.

Brian

3) Immediately thereafter, if the URI is new to the Server, the Server sends a 'test' notification to the URI, with an empty ID.
8) The Server may send 'test' notifications to the URI without any ID as a form of "keep-alive".  Any ID provided by the Server to the client may be used as the response to the test transaction

Brian

3) Immediately thereafter, the Server sends a 'test' notification to the URI, without any ID;\

4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let's pick one: reject silently the URI and block the Client permanently/provides a 'uriNotStored' warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
[br]I don't think an explicit failure is a problem.  The LoST server can limit retries if it needs to.
[GC] Ok for HTTP failures but what I'm talking about is when it fails to return the ID, like in your DoS example.

NOTICE TO RECIPIENT: This email, including attachments, may contain information which is confidential, proprietary, attorney-client privileged and / or controlled under U.S. export laws and regulations and may be restricted from disclosure by applicable State and Federal law. Nothing in this email shall create any legal binding agreement between the parties unless expressly stated herein and provided by an authorized representative of Comtech Telecommunications Corp. or its subsidiaries. If you are not the intended recipient of this message, be advised that any dissemination, distribution, or use of the contents of this message is strictly prohibited. If you received this message in error, please notify us immediately by return email and permanently delete all copies of the original email and any attached documentation from any computer or other media.
_______________________________________________
Ecrit mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ecrit<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>

---

From: Brian Rosen <[email-masked]>
Date: Mon, 13 Sep 2021 08:30:53 -0400

The kind of thing I’ve seen many times is an annexation.  It’s announced months, sometimes years in advance.  While there are clear boundaries, they very often don’t match postal boundaries or the existing “unincorporated community name”, A4.  It takes weeks to months of prep to figure out which entries will change and which stay the same.  Of course a good reason it takes that long is that at present, there is nothing like LoST Planned Changes, so it’s got a whole lot of manual work.  I am aware that sometimes, things really do change where you don’t find out more than 24 hours ahead.  Usually, that’s poor planning, rather than an actual short fuse change, but it happens, and has to be dealt with.

Just polling is a really poor answer IMO, because you have to revalidate the entire dataset to find the entries that changed.  I’ve built many systems with TTL where the TTL was gradually reduced around a change so that the entries that changed had short TTLs and the entries that didn’t had longer ones.  That only works if you know far enough in advance that a change is coming.  That’s generally true for the events that planned-changes is designed for, but not always.  If your norm is 90 day TTL, and something comes up where you find out 30 or 60 days in advance, then that idea won’t work.

Dan’s system fixes that, by having a poll for changes.  He publishes a list of entries that will change.  That can be polled frequently, because it’s small.  If I understand the “partial civic” idea, it can often be very small.

I still prefer push, because it actually fits the problem: the server wants to tell the client that something is changing.  But I’m willing to do what Dan suggests.  It’s not my favorite, but it will work.  Just using TTL won’t, in my opinion.  Either you have to do way too frequent full revalidations, or you miss events.

I’ve never liked the fact that LoST servers have 99.99% of transactions that are mostly worthless - revalidations of things that didn’t change.  

But as editor, I will write up whatever the work group consensus is.

Brian

> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[email-masked]> wrote:
> 
> Brian wrote:
> If it's common that server operators themselves often identify changes with less than one-hour notice, then the complexity of client-registered-uri with server-push is more justified.
>  
> But if it's common that server operators usually identify changes with one day or more notice, and one-hour notice is very uncommon, then the simplicity of polling is more justified.
>  
> https://www.ietf.org/mailman/listinfo/ecrit <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>

---

From: "Caron, Guy" <[email-masked]>
Date: Mon, 13 Sep 2021 19:43:30 +0000

[Resending to the ECRIT list as my reply posted at 13h00 EST was not distributed]

While the separate API idea has some merit, it is still is a blind pull. Also, it seems that, based on the high level description, there is no filtering based on the Clients being made; every Client gets the list of all upcoming changes. Further, it is yet another interface to support on the Server while the current mechanism leverages an existing one.

I still favour the URI approach and I think that the proposed security mechanism on the table is sufficient and workable.

Guy

De : Ecrit <[email-masked]<mailto:[email-masked]>> De la part de Brian Rosen
Envoyé : 13 septembre 2021 08:31
À : Jeff Martin <[email-masked]<mailto:[email-masked]>>
Cc : Dan Banks <dbanks=[email-masked]<mailto:dbanks=[email-masked]>>; [email-masked]<mailto:[email-masked]>
Objet : [EXT]Re: [Ecrit] planned-changes: two questions

Brian

On Sep 10, 2021, at 4:11 PM, Jeff Martin <[email-masked]<mailto:[email-masked]>> wrote:

/Jeff/

---

From: "Randall Gellens" <[email-masked]>
Date: Mon, 13 Sep 2021 14:27:01 -0700

Do LISs frequently revalidate addresses?  An address that's user 
configured is likely to be wrong (not where the user is at the time of a 
call) but still valid.

If I understand Dan's suggestion and how it would fit into the draft, 
the LoST server returns an ID in the response, as in the current draft, 
but instead of the client providing a URI which the server stores and 
later sends a POST to, we define a new LoST query (e.g., 
"plannedChangesRequest"), and in the response to that, the server 
includes sets of IDs and "asOf" dates.  Is that it?

--Randall

On 13 Sep 2021, at 5:30, Brian Rosen wrote:

> The kind of thing I’ve seen many times is an annexation.  It’s 
> announced months, sometimes years in advance.  While there are clear 
> boundaries, they very often don’t match postal boundaries or the 
> existing “unincorporated community name”, A4.  It takes weeks to 
> months of prep to figure out which entries will change and which stay 
> the same.  Of course a good reason it takes that long is that at 
> present, there is nothing like LoST Planned Changes, so it’s got a 
> whole lot of manual work.  I am aware that sometimes, things really do 
> change where you don’t find out more than 24 hours ahead.  Usually, 
> that’s poor planning, rather than an actual short fuse change, but 
> it happens, and has to be dealt with.
>
> Just polling is a really poor answer IMO, because you have to 
> revalidate the entire dataset to find the entries that changed.  
> I’ve built many systems with TTL where the TTL was gradually reduced 
> around a change so that the entries that changed had short TTLs and 
> the entries that didn’t had longer ones.  That only works if you 
> know far enough in advance that a change is coming.  That’s 
> generally true for the events that planned-changes is designed for, 
> but not always.  If your norm is 90 day TTL, and something comes up 
> where you find out 30 or 60 days in advance, then that idea won’t 
> work.
>
> Dan’s system fixes that, by having a poll for changes.  He publishes 
> a list of entries that will change.  That can be polled frequently, 
> because it’s small.  If I understand the “partial civic” idea, 
> it can often be very small.
>
> I still prefer push, because it actually fits the problem: the server 
> wants to tell the client that something is changing.  But I’m 
> willing to do what Dan suggests.  It’s not my favorite, but it will 
> work.  Just using TTL won’t, in my opinion.  Either you have to do 
> way too frequent full revalidations, or you miss events.
>
> I’ve never liked the fact that LoST servers have 99.99% of 
> transactions that are mostly worthless - revalidations of things that 
> didn’t change.
>
>> The data in the [server] issued set could be further extended to 
>> convey upcoming potential impacts to ids that have not yet been 
>> impacted.
>>
>> The server's issued set(s) could contain 'invalidAsOf' indicators 
>> with the locationIds, similar to the draft's server-pushed 
>> <locationInvalidated invalidAsOf="..." ...>.   So if a server knows 
>> "well in advance", say 24 hours, as long as the clients pull within 
>> 24 hours the clients would also know well in advance.   My outlined 
>> suggestion also includes server-provided TTL in server responses to 
>> give hints to clients about polling frequency.
>>
>> Balancing the tradeoffs of polling vs uri+push requires answering 
>> this question:  What will be the typical and/or exceptional 
>> lead-times for servers operators to know about upcoming changes?
>>
>> If it's common that server operators themselves often identify 
>> changes with less than one-hour notice, then the complexity of 
>> client-registered-uri with server-push is more justified.
>>
>> But if it's common that server operators usually identify changes 
>> with one day or more notice, and one-hour notice is very uncommon, 
>> then the simplicity of polling is more justified.
>>
>> IMO based on experience at my company, days of lead time (not hours) 
>> is typical for server operators dealing with "planned changes" that 
>> affect validity.  So the simplicity of polling with responses that 
>> include  'invalidAsOf' is better than the complexity of uri+push, 
>> whose benefits would rarely be realized in an environment dominated 
>> by days  of lead time.
>>
>> /Jeff/
>>
>>
>> What is a “partial civic address”?  Example?  Something like an 
>> A1/A2/A4 but no street name/number, meaning any address in this 
>> unincorporated community name?
>>
>> ISTM that a planned change quite often can’t be fixed in a way that 
>> code could do it.  So, we still need to advertise the change well in 
>> advance of the change, and have the ability to do the AsOf 
>> validation.  Changing the way the client discovers that a change is 
>> coming is one thing, not knowing well in advance is, I think, 
>> unacceptable.
>>
>> Brian
>>
>>
>>
>>
>>
>> On Sep 10, 2021, at 12:13 PM, Dan Banks 
>> <dbanks=[email-masked]> wrote:
>>
>> I am opposed to asking LoST servers to store URIs as well.  I believe 
>> it significantly overcomplicates the solution and is the biggest 
>> reason why I have not supported the planned changes draft.
>>
>> Several years ago we implemented a mechanism to allow a LIS to know 
>> that it needs to revalidate.  Roughly described, that mechanism 
>> consists of a web API that can be queried for changes after a 
>> particular time.  The response is a summary of changes in terms of 
>> partial civic addresses that allow the LIS to identify records that 
>> are likely to be affected, or possibly that there have been so many 
>> changes that a full revalidation is recommended.  We’ve made the 
>> API discoverable via U-NAPTR using the same app string as a given 
>> LoST server.
>>
>> Yes, the LIS has to poll the various LoST servers and ask if there 
>> have been any changes.  That is a cheap operation to implement, and 
>> the practical difference between knowing the very instant a change is 
>> published versus finding out within a few minutes is not significant 
>> in any way.
>>
>> Dan
>>
>> Asking a server to store and later user a client-provided uri opens 
>> the door for abuse by malicious actors.   Which got me to thinking, 
>> what if we removed the client-provided uri entirely?  If no uris then 
>> no risk of abuse.
>>
>> Keep the new optional <plannedChange asOf="..."> but without 'uri' in 
>> the client's <findServiceRequest>.  Keep the new optional <ttl> in 
>> the server's <findServiceResponse> as a hint to clients that may want 
>> to initiate revalidation from the client side.  Keep the new optional 
>> server "unique location ID" in the server response as discussed on 
>> this list (but not yet in published draft), which the client could 
>> choose to associate with the client's internal records.
>>
>> impacted" ids, where "potentially impacted" has already been 
>> discussed on this list. Each set includes: the ids that have been 
>> "potentially impacted" since the previous set was issued; the 
>> time-range covered; and a TTL hint on how long until the server is 
>> likely to issue the next list.  The server keeps the sets going back 
>> some interval defined by server policy.
>>
>> Client sends a "discovery" query to server, where server response is: 
>> the url(s) of the most recent and past set(s); and a TTL hint on how 
>> long until next list likely to be released.  Client then queries any 
>> of these urls, and server responds with the set that includes: the 
>> ids that have been "potentially impacted" since the previous set was 
>> issued; the time-range covered; and a TTL hint on how long until the 
>> server is likely to issue the next list.
>>
>> For clients that want to be highly proactive for changes, the client 
>> can store the "unique location id" from <findServiceResponse> and 
>> associate to the client's internal records.  The client then 
>> periodically queries the server for the set(s) of impacted IDs, and 
>> client gets a copy of each set. The client uses these sets of 
>> impacted ids from the servers list to check the clients internal 
>> records, and client can then revalidate those.
>>
>> abuse by malicious actors.
>>
>> The data in the issued set could be further extended to convey 
>> upcoming potential impacts to ids that have not yet been impacted.
>>
>>
>> /Jeff/
>>
>> I’m aligned with Brian on the proposal to embed a reverse HTTP 
>> transaction within an existing one. I think this is asking for 
>> trouble.
>>
>> Guy
>>
>> rough consensus.
>> <[email-masked]> wrote:
>>
>> re-enrollment of the URI.  The client gets a normal response, and 
>> that tells it the URI was accepted.
>>
>> I don’t agree.  This is a pretty normal “I send you a magic 
>> number and you have to return it to me” thing.  There are no 
>> “pending IDs”.  This is the first ID, so you need to return it, 
>> and only it, but it’s the only ID the client has at this point.
>>
>> with the same partner because things go wrong with timeouts and 
>> logic.  It’s not a parallel POST, it’s an HTTPS transaction one 
>> way that encloses another HTTPS transaction the other way.  You are 
>> dependent on the inner one completing before the outer one times out 
>> or has some other issue.  I just think that’s dangerous.  I will 
>> admit that I was taught this quite a long time ago (before HTTPS for 
>> sure) and things could have changed, but I can’t recall an API that 
>> does that.  Also note that keep alive works exactly the same way as 
>> the initial POST.  In your proposal, those are different.  That’s 
>> not much, but it’s some code.  The “test” transaction tells you 
>> that the mechanism works and your URI is being retained, whether that 
>> <[email-masked]> wrote:
>>
>> of your description:
>> to the URI, without any ID;\
>> server can limit retries if it needs to.
>> information which is confidential, proprietary, attorney-client 
>> privileged and / or controlled under U.S. export laws and regulations 
>> and may be restricted from disclosure by applicable State and Federal 
>> law. Nothing in this email shall create any legal binding agreement 
>> between the parties unless expressly stated herein and provided by an 
>> authorized representative of Comtech Telecommunications Corp. or its 
>> subsidiaries. If you are not the intended recipient of this message, 
>> be advised that any dissemination, distribution, or use of the 
>> contents of this message is strictly prohibited. If you received this 
>> message in error, please notify us immediately by return email and 
>> permanently delete all copies of the original email and any attached 
>> documentation from any computer or other media.
>> <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>

---

From: Brian Rosen <[email-masked]>
Date: Mon, 13 Sep 2021 17:45:17 -0400

Well, since there is no planned changes, it’s suggested that LISs revalidate periodically.  I think 30 days is the suggested frequency.
A user configured address is validated when they enter it.  That doesn’t work well now, but newer systems using LoST will make that immediate.  The systems usually don’t allow an invalid location to be configured.  I never thought that was a good idea, but that’s what I’ve seen.  They don’t initiate service, or keep the old address until it’s validated.

Yes. I think you have described Dan’s proposal correctly.  He didn’t have IDs, the system delivers addresses and AsOf dates.  He had the notion of a partial civic address, which may make the change list much smaller.

Brian

> On Sep 13, 2021, at 5:27 PM, Randall Gellens <[email-masked]> wrote:
> 
> Do LISs frequently revalidate addresses? An address that's user configured is likely to be wrong (not where the user is at the time of a call) but still valid.
> 
> If I understand Dan's suggestion and how it would fit into the draft, the LoST server returns an ID in the response, as in the current draft, but instead of the client providing a URI which the server stores and later sends a POST to, we define a new LoST query (e.g., "plannedChangesRequest"), and in the response to that, the server includes sets of IDs and "asOf" dates. Is that it?
> 
> --Randall
> 
>> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Brian wrote:
>> From: Brian Rosen <[email-masked] <mailto:[email-masked]>> 
>> To: Dan Banks <dbanks=[email-masked] <mailto:dbanks=[email-masked]>>
>> Cc: Jeff Martin <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
>> On Sep 10, 2021, at 12:13 PM, Dan Banks <dbanks=[email-masked] <mailto:dbanks=[email-masked]>> wrote:
>>  
>> From: Ecrit <[email-masked] <mailto:[email-masked]>> On Behalf Of Jeff Martin
>> To: [email-masked] <mailto:[email-masked]>

---

From: "Randall Gellens" <[email-masked]>
Date: Mon, 13 Sep 2021 15:12:41 -0700

Fitting his concept into the draft leads us to retain the concept of the 
server returning an ID with a query result; the change is that instead 
of a notification URI we have a new query.  I suppose this new query 
(e.g., "plannedChangesRequest") could either return a list of all IDs 
potentially invalidated with an "asOf" date, or the query could contain 
a set of IDs and the response can indicate the "asOf" date for each 
queried ID.

--Randall

On 13 Sep 2021, at 14:45, Brian Rosen wrote:

> Well, since there is no planned changes, it’s suggested that LISs 
> revalidate periodically.  I think 30 days is the suggested frequency.
> A user configured address is validated when they enter it.  That 
> doesn’t work well now, but newer systems using LoST will make that 
> immediate.  The systems usually don’t allow an invalid location to 
> be configured.  I never thought that was a good idea, but that’s 
> what I’ve seen.  They don’t initiate service, or keep the old 
> address until it’s validated.
>
> Yes. I think you have described Dan’s proposal correctly.  He 
> didn’t have IDs, the system delivers addresses and AsOf dates.  He 
> had the notion of a partial civic address, which may make the change 
> list much smaller.
>
> Brian
>
>> configured is likely to be wrong (not where the user is at the time 
>> of a call) but still valid.
>>
>> the LoST server returns an ID in the response, as in the current 
>> draft, but instead of the client providing a URI which the server 
>> stores and later sends a POST to, we define a new LoST query (e.g., 
>> whole lot of manual work.  I am aware that sometimes, things really 
>> do change where you don’t find out more than 24 hours ahead.  
>> Usually, that’s poor planning, rather than an actual short fuse 
>> change, but it happens, and has to be dealt with.
>>
>> I’ve built many systems with TTL where the TTL was gradually 
>> reduced around a change so that the entries that changed had short 
>> TTLs and the entries that didn’t had longer ones.  That only works 
>> if you know far enough in advance that a change is coming.  That’s 
>> Dan’s system fixes that, by having a poll for changes.  He 
>> publishes a list of entries that will change.  That can be polled 
>> frequently, because it’s small.  If I understand the “partial 
>> civic” idea, it can often be very small.
>>
>>> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>> Brian wrote:
>>> To: Dan Banks <dbanks=[email-masked] 
>>> <mailto:dbanks=[email-masked]>>
>>> Cc: Jeff Martin <[email-masked] 
>>> <mailto:[email-masked]>>; [email-masked] 
>>> <mailto:[email-masked]>
>>> ISTM that a planned change quite often can’t be fixed in a way 
>>> that code could do it.  So, we still need to advertise the change 
>>> well in advance of the change, and have the ability to do the AsOf 
>>> <dbanks=[email-masked] 
>>> <mailto:dbanks=[email-masked]>> wrote:
>>>
>>> I am opposed to asking LoST servers to store URIs as well.  I 
>>> believe it significantly overcomplicates the solution and is the 
>>> biggest reason why I have not supported the planned changes draft.
>>>
>>> the practical difference between knowing the very instant a change 
>>> is published versus finding out within a few minutes is not 
>>> significant in any way.
>>>
>>> Dan
>>>
>>> what if we removed the client-provided uri entirely?  If no uris 
>>> then no risk of abuse.
>>>
>>> Keep the new optional <plannedChange asOf="..."> but without 'uri' 
>>> in the client's <findServiceRequest>.  Keep the new optional <ttl> 
>>> in the server's <findServiceResponse> as a hint to clients that may 
>>> want to initiate revalidation from the client side.  Keep the new 
>>> optional server "unique location ID" in the server response as 
>>> discussed on this list (but not yet in published draft), which the 
>>> client could choose to associate with the client's internal records.
>>>
>>> Client sends a "discovery" query to server, where server response 
>>> is: the url(s) of the most recent and past set(s); and a TTL hint on 
>>> how long until next list likely to be released.  Client then queries 
>>> any of these urls, and server responds with the set that includes: 
>>> the ids that have been "potentially impacted" since the previous set 
>>> was issued; the time-range covered; and a TTL hint on how long until 
>>> the server is likely to issue the next list.
>>>
>>> To: Brian Rosen <[email-masked] <mailto:[email-masked]>>; 
>>> Randall Gellens <[email-masked] 
>>> <mailto:[email-masked]>>
>>> HTTPS transaction, but somehow the client isn’t sure the server 
>>> got the ID.
>>>
>>> sure) and things could have changed, but I can’t recall an API 
>>> that does that.  Also note that keep alive works exactly the same 
>>> way as the initial POST.  In your proposal, those are different.  
>>> That’s not much, but it’s some code.  The “test” transaction 
>>> tells you that the mechanism works and your URI is being retained, 
>>> whether that happens the first time, or a year later.
>>>
>>> --Randall
>>> piece of XML with the right namespace, the right element name, but 
>>> no value.  I think that is adequate and meets Guy’s minimal 
>>> mechanism criteria.  The same XML is always sent.  It either has a 
>>> set of IDs or it’s empty.
>>>
>>> Brian
>>>
>>>
>>> privileged and / or controlled under U.S. export laws and 
>>> regulations and may be restricted from disclosure by applicable 
>>> State and Federal law. Nothing in this email shall create any legal 
>>> binding agreement between the parties unless expressly stated herein 
>>> and provided by an authorized representative of Comtech 
>>> Telecommunications Corp. or its subsidiaries. If you are not the 
>>> intended recipient of this message, be advised that any 
>>> dissemination, distribution, or use of the contents of this message 
>>> is strictly prohibited. If you received this message in error, 
>>> please notify us immediately by return email and permanently delete 
>>> all copies of the original email and any attached documentation from 
>>> any computer or other media.

---

From: Brian Rosen <[email-masked]>
Date: Tue, 14 Sep 2021 08:28:55 -0400

Dan’s proposal doesn’t use IDs.  He returns partial civic addresses.

> On Sep 13, 2021, at 6:12 PM, Randall Gellens <[email-masked]> wrote:
> 
> Fitting his concept into the draft leads us to retain the concept of the server returning an ID with a query result; the change is that instead of a notification URI we have a new query. I suppose this new query (e.g., "plannedChangesRequest") could either return a list of all IDs potentially invalidated with an "asOf" date, or the query could contain a set of IDs and the response can indicate the "asOf" date for each queried ID.
> 
> --Randall
> 
>> On Sep 13, 2021, at 5:27 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Randall Gellens" <[email-masked]>
Date: Tue, 14 Sep 2021 09:57:25 -0700

I wasn't clear that Dan was making a specific proposal; I interpreted 
his description as what he implemented.  As I understand, he uses a 
separate API (not LoST) that returns partial civic addresses.  What I 
described is how I would fit what he did into what the draft does now, 
specifically, a new LoST query instead of a non-LoST API, and IDs.

--Randall

On 14 Sep 2021, at 5:28, Brian Rosen wrote:

>> Fitting his concept into the draft leads us to retain the concept of 
>> the server returning an ID with a query result; the change is that 
>> instead of a notification URI we have a new query. I suppose this new 
>> query (e.g., "plannedChangesRequest") could either return a list of 
>> all IDs potentially invalidated with an "asOf" date, or the query 
>> could contain a set of IDs and the response can indicate the "asOf" 
>> date for each queried ID.
>>
>> --Randall
>>
>>> On Sep 13, 2021, at 5:27 PM, Randall Gellens 
>>> If I understand Dan's suggestion and how it would fit into the 
>>> draft, the LoST server returns an ID in the response, as in the 
>>> current draft, but instead of the client providing a URI which the 
>>> server stores and later sends a POST to, we define a new LoST query 
>>> (e.g., "plannedChangesRequest"), and in the response to that, the 
>>> server includes sets of IDs and "asOf" dates. Is that it?
>>>
>>> --Randall
>>>
>>> months of prep to figure out which entries will change and which 
>>> stay the same.  Of course a good reason it takes that long is that 
>>> at present, there is nothing like LoST Planned Changes, so it’s 
>>> got a whole lot of manual work.  I am aware that sometimes, things 
>>> really do change where you don’t find out more than 24 hours 
>>> ahead.  Usually, that’s poor planning, rather than an actual short 
>>> fuse change, but it happens, and has to be dealt with.
>>>
>>> I still prefer push, because it actually fits the problem: the 
>>> server wants to tell the client that something is changing.  But 
>>> I’m willing to do what Dan suggests.  It’s not my favorite, but 
>>> it will work.  Just using TTL won’t, in my opinion.  Either you 
>>> have to do way too frequent full revalidations, or you miss events.
>>>
>>> transactions that are mostly worthless - revalidations of things 
>>> that didn’t change.
>>>
>>>> On Sep 10, 2021, at 4:11 PM, Jeff Martin 
>>>> <[email-masked] <mailto:[email-masked]>> 
>>>> wrote:
>>>>
>>>> Brian wrote:
>>>> IMO based on experience at my company, days of lead time (not 
>>>> hours) is typical for server operators dealing with "planned 
>>>> changes" that affect validity.  So the simplicity of polling with 
>>>> responses that include  'invalidAsOf' is better than the complexity 
>>>> of uri+push, whose benefits would rarely be realized in an 
>>>> environment dominated by days  of lead time.
>>>>
>>>> /Jeff/
>>>>
>>>>
>>>> From: Ecrit <[email-masked] 
>>>> <mailto:[email-masked]>> On Behalf Of Jeff Martin
>>>> The server periodically (server policy) issues a set of 
>>>> "potentially impacted" ids, where "potentially impacted" has 
>>>> already been discussed on this list. Each set includes: the ids 
>>>> that have been "potentially impacted" since the previous set was 
>>>> issued; the time-range covered; and a TTL hint on how long until 
>>>> the server is likely to issue the next list.  The server keeps the 
>>>> sets going back some interval defined by server policy.
>>>>
>>>> is: the url(s) of the most recent and past set(s); and a TTL hint 
>>>> on how long until next list likely to be released.  Client then 
>>>> queries any of these urls, and server responds with the set that 
>>>> includes: the ids that have been "potentially impacted" since the 
>>>> previous set was issued; the time-range covered; and a TTL hint on 
>>>> how long until the server is likely to issue the next list.
>>>>
>>>> For clients that want to be highly proactive for changes, the 
>>>> client can store the "unique location id" from 
>>>> <findServiceResponse> and associate to the client's internal 
>>>> records.  The client then periodically queries the server for the 
>>>> set(s) of impacted IDs, and client gets a copy of each set. The 
>>>> client uses these sets of impacted ids from the servers list to 
>>>> check the clients internal records, and client can then revalidate 
>>>> those.
>>>>
>>>> From: Ecrit <[email-masked] 
>>>> <mailto:[email-masked]>> On Behalf Of Caron, Guy
>>>> dependent on the inner one completing before the outer one times 
>>>> out or has some other issue.  I just think that’s dangerous.  I 
>>>> will admit that I was taught this quite a long time ago (before 
>>>> HTTPS for sure) and things could have changed, but I can’t recall 
>>>> an API that does that.  Also note that keep alive works exactly the 
>>>> same way as the initial POST.  In your proposal, those are 
>>>> different.  That’s not much, but it’s some code.  The 
>>>> “test” transaction tells you that the mechanism works and your 
>>>> URI is being retained, whether that happens the first time, or a 
>>>> year later.
>>>>
>>>> --Randall
>>>> That transaction concludes by sending an ID.  The server 
>>>> immediately sends the test notification, to which the client 
>>>> responds with the ID.  At that point both sides know the URI was 
>>>> accepted and the client is valid.  That may be repeated 
>>>> periodically so both sides know the other is happy with the 
>>>> arrangements.  If the client doesn’t get the test notification, 
>>>> it knows the URI isn’t accepted.  If the server doesn’t see the 
>>>> ID, it’s knows that was a bogus request, or there is some other 
>>>> problem, and it shouldn’t save the URI.
>>>>
>>>> The proposed mechanism seems fragile. A client has no way to know 
>>>> if its request to store a URI was accepted. That makes it hard to 
>>>> In my proposal, when the client requests to be notified and 
>>>> provides a URI, the server immediately does a parallel POST to that 
>>>> URI. If it gets a 202 Accepted response to the POST (or we can 
>>>> choose a different value), it stores the URI and returns the query 
>>>> result. If it gets a different response, it does not store the URI 
>>>> and returns the query result with a uriNotStored warning. If a 
>>>> client is trying to get the server to launch an attack on a third 
>>>> party entity, that entity will likely not support the URI in the 
>>>> first place, or will likely return an error response. If you want 
>>>> additional verification, have the server include the queried 
>>>> location in its test POST. That way, if the client maliciously sent 
>>>> a URI of a third party LIS, that LIS will know it does not have a 
>>>> query with that location outstanding.
>>>> As for multiple IDs in a single notification POST, having the 
>>>> client specify in the initial request the maximum it is prepared to 
>>>> accept seems reasonable. We can require that clients support a 
>>>> minimum number. If a server has more than the maximum, it sends 
>>>> them in multiple POSTs, with whatever separation in time it 
>>>> chooses. A server can include fewer IDs in a POST than the client's 
>>>> maximum.
>>>> silently ignore repeated requests to store a URI where the test in 
>>>> 4 below fails.
>>>> future notifications.  If it does not, the server ignores the 
>>>> request to store the URI.
>>>> any ID as a form of “keep-alive”.  Any ID provided by the 
>>>> Server to the client may be used as the response to the test 
>>>> transaction
>>>>
>>>>
>>>> binding agreement between the parties unless expressly stated 
>>>> herein and provided by an authorized representative of Comtech 
>>>> all copies of the original email and any attached documentation 
>>>> from any computer or other media.

---

From: Brian Rosen <[email-masked]>
Date: Tue, 14 Sep 2021 14:23:51 -0400

Yeah, but if we were going from a push (notification) mechanism to a pull (polling) mechanism, which I oppose, but if we were, then I think what Dan did is pretty good, and better than having IDs the client has to track and much better than just a TTL.

Brian

> On Sep 14, 2021, at 12:57 PM, Randall Gellens <[email-masked]> wrote:
> 
> I wasn't clear that Dan was making a specific proposal; I interpreted his description as what he implemented. As I understand, he uses a separate API (not LoST) that returns partial civic addresses. What I described is how I would fit what he did into what the draft does now, specifically, a new LoST query instead of a non-LoST API, and IDs.
> 
> --Randall
> 
>> On Sep 13, 2021, at 6:12 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Randall Gellens" <[email-masked]>
Date: Tue, 14 Sep 2021 11:57:31 -0700

using IDs seems simpler and unambiguous.  If using civic address 
elements, what are the matching rules?

--Randall

On 14 Sep 2021, at 11:23, Brian Rosen wrote:

> Yeah, but if we were going from a push (notification) mechanism to a 
> pull (polling) mechanism, which I oppose, but if we were, then I think 
> what Dan did is pretty good, and better than having IDs the client has 
> to track and much better than just a TTL.
>
> Brian
>
>> described is how I would fit what he did into what the draft does 
>> now, specifically, a new LoST query instead of a non-LoST API, and 
>> IDs.
>>
>> --Randall
>>
>>> On Sep 13, 2021, at 6:12 PM, Randall Gellens 
>>> instead of a notification URI we have a new query. I suppose this 
>>> new query (e.g., "plannedChangesRequest") could either return a list 
>>> of all IDs potentially invalidated with an "asOf" date, or the query 
>>>> announced months, sometimes years in advance.  While there are 
>>>> clear boundaries, they very often don’t match postal boundaries 
>>>> or the existing “unincorporated community name”, A4.  It takes 
>>>> weeks to months of prep to figure out which entries will change and 
>>>> which stay the same.  Of course a good reason it takes that long is 
>>>> that at present, there is nothing like LoST Planned Changes, so 
>>>> it’s got a whole lot of manual work.  I am aware that sometimes, 
>>>> things really do change where you don’t find out more than 24 
>>>> hours ahead.  Usually, that’s poor planning, rather than an 
>>>> actual short fuse change, but it happens, and has to be dealt with.
>>>>
>>>> TTLs and the entries that didn’t had longer ones.  That only 
>>>> works if you know far enough in advance that a change is coming.  
>>>> That’s generally true for the events that planned-changes is 
>>>> designed for, but not always.  If your norm is 90 day TTL, and 
>>>> something comes up where you find out 30 or 60 days in advance, 
>>>> then that idea won’t work.
>>>>
>>>>> <locationInvalidated invalidAsOf="..." ...>.   So if a server 
>>>>> knows "well in advance", say 24 hours, as long as the clients pull 
>>>>> within 24 hours the clients would also know well in advance.   My 
>>>>> outlined suggestion also includes server-provided TTL in server 
>>>>> responses to give hints to clients about polling frequency.
>>>>>
>>>>> responses that include  'invalidAsOf' is better than the 
>>>>> complexity of uri+push, whose benefits would rarely be realized in 
>>>>> an environment dominated by days  of lead time.
>>>>>
>>>>> /Jeff/
>>>>>
>>>>>
>>>>> What is a “partial civic address”?  Example?  Something like 
>>>>> an A1/A2/A4 but no street name/number, meaning any address in this 
>>>>> validation.  Changing the way the client discovers that a change 
>>>>> is coming is one thing, not knowing well in advance is, I think, 
>>>>> unacceptable.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Several years ago we implemented a mechanism to allow a LIS to 
>>>>> know that it needs to revalidate.  Roughly described, that 
>>>>> mechanism consists of a web API that can be queried for changes 
>>>>> after a particular time.  The response is a summary of changes in 
>>>>> terms of partial civic addresses that allow the LIS to identify 
>>>>> records that are likely to be affected, or possibly that there 
>>>>> have been so many changes that a full revalidation is recommended. 
>>>>>  We’ve made the API discoverable via U-NAPTR using the same app 
>>>>> string as a given LoST server.
>>>>>
>>>>> have been any changes.  That is a cheap operation to implement, 
>>>>> and the practical difference between knowing the very instant a 
>>>>> change is published versus finding out within a few minutes is not 
>>>>> significant in any way.
>>>>>
>>>>> Dan
>>>>>
>>>>> Asking a server to store and later user a client-provided uri 
>>>>> opens the door for abuse by malicious actors.   Which got me to 
>>>>> thinking, what if we removed the client-provided uri entirely?  If 
>>>>> no uris then no risk of abuse.
>>>>>
>>>>> in the server's <findServiceResponse> as a hint to clients that 
>>>>> may want to initiate revalidation from the client side.  Keep the 
>>>>> new optional server "unique location ID" in the server response as 
>>>>> “pending IDs”.  This is the first ID, so you need to return 
>>>>> it, and only it, but it’s the only ID the client has at this 
>>>>> point.
>>>>>
>>>>> logic.  It’s not a parallel POST, it’s an HTTPS transaction 
>>>>> one way that encloses another HTTPS transaction the other way.  
>>>>> You are dependent on the inner one completing before the outer one 
>>>>> times out or has some other issue.  I just think that’s 
>>>>> dangerous.  I will admit that I was taught this quite a long time 
>>>>> ago (before HTTPS for sure) and things could have changed, but I 
>>>>> can’t recall an API that does that.  Also note that keep alive 
>>>>> works exactly the same way as the initial POST.  In your proposal, 
>>>>> those are different.  That’s not much, but it’s some code.  
>>>>> The “test” transaction tells you that the mechanism works and 
>>>>> your URI is being retained, whether that happens the first time, 
>>>>> or a year later.
>>>>>
>>>>> --Randall
>>>>> I don’t think there is any fragility in the URI setup.  For a 
>>>>> new URI, the client requests the server to keep it in a 
>>>>> findService.  That transaction concludes by sending an ID.  The 
>>>>> server immediately sends the test notification, to which the 
>>>>> client responds with the ID.  At that point both sides know the 
>>>>> URI was accepted and the client is valid.  That may be repeated 
>>>>> it knows the URI isn’t accepted.  If the server doesn’t see 
>>>>> the ID, it’s knows that was a bogus request, or there is some 
>>>>> other problem, and it shouldn’t save the URI.
>>>>>
>>>>> I proposed a null ID as the test transaction flag.  That would be 
>>>>> a piece of XML with the right namespace, the right element name, 
>>>>> but no value.  I think that is adequate and meets Guy’s minimal 
>>>>> provides a URI, the server immediately does a parallel POST to 
>>>>> that URI. If it gets a 202 Accepted response to the POST (or we 
>>>>> can choose a different value), it stores the URI and returns the 
>>>>> query result. If it gets a different response, it does not store 
>>>>> the URI and returns the query result with a uriNotStored warning. 
>>>>> If a client is trying to get the server to launch an attack on a 
>>>>> third party entity, that entity will likely not support the URI in 
>>>>> the first place, or will likely return an error response. If you 
>>>>> want additional verification, have the server include the queried 
>>>>> location in its test POST. That way, if the client maliciously 
>>>>> sent a URI of a third party LIS, that LIS will know it does not 
>>>>> have a query with that location outstanding.
>>>>> or whatever, the server returns uriNotStored and the client can 
>>>>> try again later.
>>>>> client specify in the initial request the maximum it is prepared 
>>>>> to accept seems reasonable. We can require that clients support a 
>>>>> chooses. A server can include fewer IDs in a POST than the 
>>>>> client's maximum.
>>>>> planned change by sending a notification to the successfully 
>>>>> tested URI with the location ID;
>>>>> 7) LIs at the Client that are invalidated by the planned change 
>>>>> are modified in its database to be valid (which probably mean 
>>>>> another revalidation cycle) with an effective date set to 
>>>>> <revalidateAsoF> value.
>>>>> future notifications. If it does not, the Server [let’s pick 
>>>>> one: reject silently the URI and block the Client 
>>>>> permanently/provides a ‘uriNotStored’ warning response to the 
>>>>> planned change by sending a notification to the successfully 
>>>>> tested URI with the location ID;
>>>>> 7) LIs at the Client that are invalidated by the planned change 
>>>>> are modified in its database to be valid (which probably mean 
>>>>> another revalidation cycle) with an effective date set to 
>>>>> <revalidateAsoF> value.
>>>>> NOTICE TO RECIPIENT: This email, including attachments, may 
>>>>> contain information which is confidential, proprietary, 
>>>>> attorney-client privileged and / or controlled under U.S. export 
>>>>> laws and regulations and may be restricted from disclosure by 
>>>>> applicable State and Federal law. Nothing in this email shall 
>>>>> create any legal binding agreement between the parties unless 
>>>>> expressly stated herein and provided by an authorized 
>>>>> representative of Comtech Telecommunications Corp. or its 
>>>>> subsidiaries. If you are not the intended recipient of this 
>>>>> message, be advised that any dissemination, distribution, or use 
>>>>> of the contents of this message is strictly prohibited. If you 
>>>>> received this message in error, please notify us immediately by 
>>>>> return email and permanently delete all copies of the original 
>>>>> email and any attached documentation from any computer or other 
>>>>> media.

---

From: Brian Rosen <[email-masked]>
Date: Tue, 14 Sep 2021 16:21:04 -0400

Pretty easy I think: if present in the list, must match.  If not present, any value matches.  That pretty much works for everything I can think of (entire street, entire A5, entire A4, entire PCN, entire street address (meaning ignore BLD/FLR/UNIT/ROOM).  It’s a lot smaller than IDs for a large change, and bigger for a very small change, which is the right direction.  IDs are more precise - Dan’s idea will have more locations that won’t change, but that’s not much of a problem I think.

I will say that the differences don’t matter much to me; if consensus is gotta be push, and use IDs rather than partial civics, I’ll be okay with it.  I strongly prefer push, and somewhat prefer partial civics, but I’ll do whatever consensus is.

Brian

> On Sep 14, 2021, at 2:57 PM, Randall Gellens <[email-masked]> wrote:
> 
> using IDs seems simpler and unambiguous. If using civic address elements, what are the matching rules?
> 
> --Randall
> 
>> On Sep 14, 2021, at 12:57 PM, Randall Gellens <[email-masked] <mailto:[email-masked]>> wrote:
>>