Subject: Re: [CDNi] CDNI Requirements I-D

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 1 Feb 2011 11:11:33 +0000

Francois,

Thanks for the reply, see inline. I've snipped those parts where we are in agreement.

Ben

On 31 Jan 2011, at 12:42, Francois Le Faucheur wrote:

> Hi Ben,
>>  R7  The CDNI solution MAY support acquisition across CDNs based on
>>      other protocols than HTTP.
>> 
>> s/other protocols than/protocols other than/
> 
> OK. I assume same goes for R6.
> 

Yes :-)

>>  R8  The CDNI solution SHOULD support cascaded CDN redirection (CDN1
>>      redirects to CDN2 that redirects to CDN3) to an arbitrary number
>>      of levels.
>> 
>> Based on Haibin’s e-mail may be include an additional requirement that it should be possible to limit the number of levels.
> 
> This is covered in the "Request-Routing section" by R34:
> "   
> R34  The CDNI Request-Routing API SHOULD support optional enforcement
>        of a limit on the number of successive CDN redirections for a
>        given request.
> "
> or do you feel this is broader than Request-Routing and should be moved to the "Generic Requirements" section 

I am not sure that loop prevention is required for all APIs but it is probably more generally applicable than just Request Routing, e.g. if I issue a "content deletion" request which is subsequently cascaded we probably need to ensure it doesn't loop.

>>  R13  The CDNI Control API SHOULD allow the Downstream CDN to
>>       communicate to the Upstream CDN aggregate information on the
>>       Downstream CDN capabilities, resources and affinities (i.e.
>>       Preferences or cost).  This information can be taken into
>>       account by the Upstream CDN Request Routing system in its CDN
>>       Selection decisions.  This information could, for example,
>>       include:
>>       *   supported content types and delivery protocols
>>       *   footprint (e.g. layer-3 coverage)
>>       *   a set of metrics/attributes (e.g.  Streaming bandwidth,
>>           storage resources, distribution and delivery priority)
>>       *   a set of affinities (e.g.  Preferences, indication of
>>           distribution/delivery fees)
>>       *   information to facilitate request redirection (e.g.
>>           Reachability information of Downstream CDN Request Routing
>>           system). 
>> 
>> I am not sure what the exact split should be but I’d suggest “information to facilitate request redirection” fits under the CDNI Request Routing API and much of the other examples may fit best under the CDNI Metadata API.
> 
> The requirement was intended to bring up "information to facilitate request redirection”. In fact, we should just mention exactly that at the beginning of the requirement.  To me, pretty much all of the examples listed above fit in that category.
> I agree this should be move to the "Request-Routing API" section. This is in line with the recent thread on the list. I think this applies also to R14, R15, R20, R21 and R22.

OK. I think the Request Routing API is a better place for those requirements but for some things like "information on the capabilities, resources and affinities of CDNs to which the Downstream CDN may (in turn) redirect requests" I do wonder whether the Metadata API may be more appropriate under some circumstances.

I think what should drive whether they fit into Request Routing API or Metadata API is the scope of the information they are expressing. If it is CDN-wide policy I tend to think metadata API is the better place, if it's policy within the scope of that particular request routing transaction then the Request Routing API is probably the right place.

I suggest you move them to be requirements on the Request Routing API and when we are doing the detailed CDNI specification design we'll have a better idea of the scope of policy we are communicating and the best API to place it under.

>> My thinking is that the CDNI Metadata API is really about distributing information about an identifiable object (e.g. metadata on a piece of content, geo-blocking policy, etc.) and for example we could treat information about footprint in a very similar way to how we would treat distribution of IP block details for geo-blocking. As such information is needed by more than just request routers Metdata API seems a better fit.
> 
> Interesting. I see your point about a similar potential need to communicate "footprint" descriptions for the sake of conveying geo-blocking content distribution policies (as part of CDNI Metadata) and for the sake of conveying CDN footprint coverage (as part of CDNI Request Routing API).
> But I think we can just make a note of that and make sure that if/when we get to interface specification, a single mechanisms is defined to convey such footprint description. Then CDNI Metadata interface can convey information that lists allowed/blocked footprints, while Request-Routing interface can convey information that associates capabilities/attributes/etc to footprints.
> 
> In other words, my suggestion is that we keep the discussion on requirements to convey geo-blocking under CDNI Metadata interface and requirements to convey CDN coverage/capabilities/attributes as part of the Request Routing interface, and just consider as a potential encoding optimization the fact those two may share some common information elements aka a "footprint").
> 
> Makes sense? 
> 

Yes. I think we will need to specify a naming scheme for well-known geo-blocks (e.g. based on ISO 3166) so I don't see an issue with both APIs using the same naming convention for referencing ip blocks (My geo-location database has approximately 13k ip blocks for the UK [it would be closer to 20k if they were expressed as valid CIDR blocks) and I don't want to be passing that volume of data on request routing transactions).

>>  R23  The CDNI Control API SHOULD support virtualization of the
>>       Downstream CDN, so that the Downstream CDN appears as multiple
>>       virtual Downstream CDN.  In that case, the information discussed
>>       in the previous requirement MUST be complemented with a Virtual
>>       CDN Identifier that is unique in the context of the pair of CDNs
>>       on both sides of the CDNI Control API.
>> 
>> I am unsure about this requirement as I am not sure exactly what it is asking for, is there a use case that would illustrate it by way of example?
> 
> a Downstream CDN may offer multiple delivery services. For example, it could present a virtual CDN that covers one country only (which may be of interest to other CDNs that just need coverage of that country) or it could present a virtual CDN that covers one continent (including the previous country). Alternatively, it could present two virtual CDNs, one comprising a given set of resources, and another one with a larger set of resources.
> 

OK, but how much does the upstream CDN need to know about how (or even whether) the downstream CDN is virtualised? Can the downstream CDN not just present itself as two independent CDNs and the upstream doesn't know (or care) whether they are two "physical" CDNs or two "virtual" CDNs.

As you agreed to remove the last sentence I'm OK with the requirement staying in - if we need to specify something to support virtualisation we can, if, as I suspect, we don't need to specify anything specifically for virtualisation the requirement is harmless.

>>  R25  The CDNI Control API SHOULD allow bootstrapping of the Content
>>       Acquisition API.  This information could, for example, include:
>>       *   negotiation of the Content Acquisition protocol to be used
>>           (e.g.  HTTP, HTTPS, FTP, ATIS C2), with some granularity
>>           (e.g.  On a per content type basis).
>> 
>>  R26  The CDNI Control API SHOULD allow the Upstream CDN to distribute
>>       the information necessary to bootstrap delivery authorization
>>       mechanisms to be performed by Downstream CDN.  This information
>>       could, for example, include:
>>       *   information necessary for surrogates to perform URI
>>           signature based validation
>> 
>> For R25 & R26 I think the role of the CDNI Control API is to exchange/negotiate what protocols & authorization mechanisms a CDN supports. Communication of the specific protocol/auth mechanism to use & associated details (credentials etc.) should be on a per object (or object set) basis via the CDNI Metadata API.
> 
> I agree this is a better split.
> 
> I propose to reword as:
> "
> Rx:	The CDNI Control API SHOULD allow bootstrapping of the Content Acquisition API. This could, for example, include exchange and negotiation of the Content Acquisition protocols to be used across the CDNs (e.g. HTTP, HTTPS, FTP, ATIS C2).
> 
> Ry:	The CDNI Control API SHOULD allow exchange and negotiation of delivery authorization mechanisms to be supported across the CDNs (e.g. URI signature based validation).
> "
> 

Works for me.

>>  R33  The CDNI Request-Routing loop prevention mechanism SHOULD allow
>>       routing of the request (as opposed to the request loop being
>>       simply interrupted without routing the request).
>> 
>> It’s not clear to me what “SHOULD allow routing of the request” means, could you provide an example of how it may work?
> 
> Say, there is a redirection loop (CDN1-->CDN2-->CDN3--->CDN1), the solution could have a pure loop detection mechanism which will result in a given request/query being dropped when it starts looping. The result would be that the request is never honored and the user never gets the content. The objective of the requirement is to say that we want more than that: we want to make sure that the loop is detected and the loop is effectively prevented so that the user will get the content somehow.
> Just for example, CDN3 could realize that it is about to redirect a request to CDN1, while the request had been redirected earlier via CDN1, so CDN3 may change its "normal" decision and instead serve the request itself or redirect to another CDN that is not in the "loop".
> 

OK, I understand now.

>>  R34  The CDNI Request-Routing API SHOULD support optional enforcement
>>       of a limit on the number of successive CDN redirections for a
>>       given request.
>> 
>> “SHOULD support optional enforcement” - Does this mean the CDNI Request Routing API SHOULD support a mechanism but that honouring the limit is optional?
> 
> No. It is intended to mean that CDNI Request Routing API should support a mechanism but invoking the mechanism is optional.
> 
> I propose we reword as:
> "The CDNI Request-Routing API SHOULD support an optional mechanism enforcing a limit on the number of successive CDN redirections for a given request."
> 

Works for me.

>>  R43  The CDNI Metadata API SHOULD support a pre-positioning model
>>       whereby the Upstream CDN requests the Downstream CDN to acquire
>>       content and associated distribution metadata (and possibly
>>       position those into Downstream CDN surrogates) at an appropriate
>>       time (now or a specified future time), before the content is
>>       actually requested by endusers.  [Editor's Note: how much
>>       influence the Upstream CDN ought to have on pre-positioning of
>>       the content on surrogates inside the Downstream CDN is TBD].
>>       [Editor's note: this functionality is currently described in
>>       draft-jenkins-cdni-problem-statement-01 under the Metadata API.
>>       Should it be moved under the Control API (since it is not
>>       strictly about exchange of metadata but more about triggering
>>       multiple actions in downstream CDN, one of them being possibly
>>       to get metadata) ?].
>> 
>> The “trigger” is probably better suited to the Control API but the actual exchange should be via the metadata API, otherwise we will end up having two different ways to distribute metadata and I think we want to avoid that situation.
> 
> Agreed. I propose to:
> 
> *1) add a requirement in the Control API section:
> "
> Rx:	The CDNI Control API SHOULD support triggering and control for a pre-positioning model whereby the Upstream CDN requests the Downstream CDN to acquire content and/or associated distribution metadata (and possibly position those into Downstream CDN surrogates) at an appropriate time (now or a specified future time), before the content is actually requested by endusers. Note that the actual exchange of metadata and the actual content acquisition are outside the scope of the Control API: those are supported, respectively, via the CDNI Metadata API and the Content Acquisition API. [Editor's Note: how much influence the Upstream CDN ought to have on pre-positioning of the content on surrogates inside the Downstream CDN is TBD].
> "
> 

works for me but s/endusers/end users/

> *2)  reword R42 and R43 to:
> "
> Rx:	The CDNI Metadata API MUST allow the Upstream CDN to provide the Downstream CDN with content distribution metadata of inter-CDN scope.
> 
> Ry:	The CDNI Metadata API MUST support exchange of CDNI metadata for both the dynamic acqusition model (whereby the Downstream CDN surrogates acquire content when it is actually requested by end users) and the pre-positioning model (whereby the Upstream CDN requests the Downstream CDN to acquire content and/or associated distribution metadata before the content is actually requested by endusers). [Editor's Note: how much influence the Upstream CDN ought to have on pre-positioning of the content on surrogates inside the Downstream CDN is TBD].
> " 
> 

works for me

>> R44/R45 - I am not sure we need R44 & R45. R42 & R43 are sufficient. If we have a method to dynamically acquire metadata (R42) and a method to trigger a downstream CDN to go collect some metadata (R43) I think we have R44 and R45 covered with the granularity of how much is communicated in advance controlled by the “trigger”.
> 
> I don't think R44 and R45 really follow from R42 and R43.
> R42 and R43 just say that there needs to be a way to pass metadata and that this should work both for dynamic acquisition and for prepositioning scenarios. It does not imply much on whether this is achieved by pushing all metadata, pushing some allowing to pull the rest, issuing a trigger that allows the pull.
> 

OK provided "initially communicated" is understood to be vague enough that it does not pre-suppose a particular distribution model I'm happy to leave this until we're doing the detailed design.

>>  R47  The CDNI Metadata API MUST also provide the necessary
>>       information to allow the Downstream CDN to attempt to acquire
>>       the content from the content Origin Server, in case it can not
>>       be obtained from the Upstream CDN (e.g.  Because of a failure
>>       scenario).  Note that the content Origin Server may or may not
>>       be willing to serve the content to the Downstream CDN since the
>>       Content Provider may not have a direct agreement/relationship
>>       with the Downstream CDN for delivery of this content.
>> 
>> In one way this kind of addresses my previous comment
> 
> Right.
> 
>> but my preference would be to combine the two requirements into a single requirement as follows:
>> 
>> “Whether in the pre-positioning model or a dynamic acquisition model, the CDNI Metadata API MUST provide the necessary information to allow the Downstream CDN to acquire the content from one or more upstream sources. Note that some upstream sources (e.g. the content Origin Server) may or may not be willing to serve the content to the Downstream CDN, if this policy is known to the upstream CDN then it MAY omit those sources when exchanging CDNI metadata.
> 
> I propose to expand a bit and spread over two requirements:
> 
> "
> Rx:	Whether in the pre-positioning model or a dynamic acquisition model, the CDNI Metadata API MUST provide the necessary information to allow the Downstream CDN to acquire the content from an upstream source (e.g. Acquisition protocol and Uniform Resource Identifier in Upstream CDN- or rules to construct this URI). 
> 
> Ry:The CDNI metadata MUST allow signaling of one or more upstream sources, where each upstream source can be in the Upstream CDN, in another CDN, the CSP origin server or any arbitrary source designated by the Upstream CDN. Note that some upstream sources (e.g. the content origin server) may or may not be willing to serve the content to the Downstream CDN, if this policy is known to the upstream CDN then it may omit those sources when exchanging CDNI metadata.
> "
> 
> (it says "may omit" and mot "MAY omit" because the document defines requirement on the protocol itself, not so much how an implementation must/should/may use the protocol knobs).
> 

Works for me.

>> R55 appears to be a duplicate of R55 so I suggest it is deleted.
> 
> I think you mean R58.

:-)

> The trick is that cascaded CDNs may or may not be kept within Initial scope.
> So R55 says: if cascaded CDNs are in initial scope then, the requirement MUST be met in initial scope.
> R58 says that : no matter what, the requirement is to be met in subsequent scope (ie beyond initial scope).
> 

OK I understand.

---

From: "Lee, Yiu" <[email-masked]>
Date: Tue, 1 Feb 2011 18:15:03 +0000

Hi Ben,

On 2/1/11 6:11 AM, "Ben Niven-Jenkins" <[email-masked]> wrote:

>I am not sure that loop prevention is required for all APIs but it is
>probably more generally applicable than just Request Routing, e.g. if I
>issue a "content deletion" request which is subsequently cascaded we
>probably need to ensure it doesn't loop.

[YL] I agree with Ben that we loop prevention should only apply on the
RR-API.

>Yes. I think we will need to specify a naming scheme for well-known
>geo-blocks (e.g. based on ISO 3166) so I don't see an issue with both
>APIs using the same naming convention for referencing ip blocks (My
>geo-location database has approximately 13k ip blocks for the UK [it
>would be closer to 20k if they were expressed as valid CIDR blocks) and I
>don't want to be passing that volume of data on request routing
>transactions).

[YL] As Dan Wing pointed out in his other email, I have yet figured out
what is the impact when we consider IPv4/IPv6 transition mechanism. At
minimum, it is ok for the metadata to contain the actual IP address block
information, but all request coming from or going to the end user must be
IP address family (ie FQDN) agnostic.

>OK, but how much does the upstream CDN need to know about how (or even
>whether) the downstream CDN is virtualised? Can the downstream CDN not
>just present itself as two independent CDNs and the upstream doesn't know
>(or care) whether they are two "physical" CDNs or two "virtual" CDNs.
>
>As you agreed to remove the last sentence I'm OK with the requirement
>staying in - if we need to specify something to support virtualisation we
>can, if, as I suspect, we don't need to specify anything specifically for
>virtualisation the requirement is harmless.

[YL] Ben made me to rethink this requirement. Should we drop the
requirement about virtualized CDN altogether?

>>> It¹s not clear to me what ³SHOULD allow routing of the request² means,
>>>could you provide an example of how it may work?
>> 
>> Say, there is a redirection loop (CDN1-->CDN2-->CDN3--->CDN1), the
>>solution could have a pure loop detection mechanism which will result in
>>a given request/query being dropped when it starts looping. The result
>>would be that the request is never honored and the user never gets the
>>content. The objective of the requirement is to say that we want more
>>than that: we want to make sure that the loop is detected and the loop
>>is effectively prevented so that the user will get the content somehow.
>> Just for example, CDN3 could realize that it is about to redirect a
>>request to CDN1, while the request had been redirected earlier via CDN1,
>>so CDN3 may change its "normal" decision and instead serve the request
>>itself or redirect to another CDN that is not in the "loop".
>> 
>
>OK, I understand now.

[YL] The loop prevention mechanism should be also smart enough to prevent
the follow case:
(CDN1-->CDN2-->CDN3-->CDN4-->CDN2)

In Francois example, CDN1 can identify who is the origin and detect the
loop. In the new example, CDN2 isn't the origin, so CDN2 will need to
identify itself in the request so that CDN4 will know.

>OK provided "initially communicated" is understood to be vague enough
>that it does not pre-suppose a particular distribution model I'm happy to
>leave this until we're doing the detailed design.

[YL] This is our intention not to "bias" any model in the requirement
draft :-)

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 1 Feb 2011 18:58:14 +0000

Yiu,

See inline, I've only included the relevant parts of your message and
snipped the rest.

> [YL] I agree with Ben that we loop prevention should only apply on the

Either you misunderstood me or you wrote something different to what you
meant. I think loop prevention applies to more than the request routing
API (e.g. The Control API or whatever API triggers content deletion) but
it is probably not required on all APIs (e.g. Metadata API).

Looping requests on the request routing and control APIs are "dangerous".
Looping data on the Metadata and Logging interfaces is probably not
"dangerous" but we may still want to have loop prevention as a sanity
check (looping data is probably a good indicator something more
fundamental is broken somewhere, either in the network or in the CDN
configuration).

> identify itself in the request so that CDN4 will know.

Agreed, I think we can achieve both using a pretty simple method similar
to BGP's AS-Path but we can debate solutions when we are at the detailed
design stage.

> [YL] This is our intention not to "bias" any model in the requirement

Great :-)

Ben

---

From: Francois Le Faucheur <[email-masked]>
Date: Tue, 1 Feb 2011 20:17:09 +0100

Works for me.

Francois

---

From: Francois Le Faucheur <[email-masked]>
Date: Tue, 1 Feb 2011 20:28:15 +0100

Yiu,

> requirement about virtualized CDN altogether?

The approach suggested above seems fine for me: we keep the requirement in for now and if it turns out to be achievable without any specific mechanisms, that's just great. This is only suggested as a potential "beyond initial scope" item, so I think we may as well keep track of it.

Francois