Subject: [CDNi] New Metadata Interface Draft (draft-cjlmw-cdni-metadata-00)

From: "Matt Caulfield (mcaulfie)" <[email-masked]>
Date: Mon, 9 Jul 2012 19:44:41 +0000

Hi everyone,

As we announced in April, a group of us (Ben Niven-Jenkins, Kent Leung, Rob Murray, Grant Watson, and I) have been working on a merge of two of the CDNI metadata interface proposals:

-       draft-caulfield-cdni-metadata-core-00

-       draft-jenkins-cdni-metadata-00

The merged draft has now been submitted: http://tools.ietf.org/html/draft-cjlmw-cdni-metadata-00

We will be presenting on this draft in Vancouver. Any feedback on the list is also appreciated.

Thanks,
Matt

---

From: Kevin J Ma <[email-masked]>
Date: Mon, 16 Jul 2012 16:40:24 -0400

Hi Matt,

  A couple of initial comments/questions about the merged draft:

  - I do not understand the need for a heirarchical separation of the
    Acquisition and Delivery objects?  Why is the property "key" not
    sufficient for designating the function of the metadata?

    Though it could be argued that these are logically different
    functions, the extra layer seems unnecessary, and introduces some
    complexity to the inheritance model?

    With inheritance-based override, can a lower level PathMetadata
    object override just an Auth List, or does it have to replace the
    entire Delivery object?  Does it have to place the Auth List
    inside a lower level Delivery object, or can it just define a new
    Auth List with no Delivery object wrapper?  Is there a concept of
    leaf vs. branch objects for override purposes?

    In general, if there are an arbitrary number of levels below each
    object, and an arbitrary number of levels of PathMetadata which
    may contain objects with arbitrary levels of metadata which may
    override previous levels, is there a need to check that duplicate
    Metadata do not exist within a given subtree, and/or should that
    restriction be stated explicity?

  - I see that the PathMetadata defines some restrictions to what it
    may contain, however, this does not seem like a scalable method of
    managing object restrictions.  As we move forward and more vendor
    specific metadata are defined, managing metadata collisions using
    the such a method seems complex, in that the CDNI Metadata RFC
    could need to be continuously updated to add more restrictions.
    It would be nice if there was a general set of rules for how and
    where to define new metadata, rather than creating highly
    customized objects which have complex modification restrictions?

  - I assume that the inheritance model requires that PathMatch
    objects must match the path of all PathMatch objects above it in
    the hierarchy?  i.e., I cannot add a PathMatch for "/bar/baz"
    inside a PathMetadata whose PathMatch was for "/foo"?  It is
    probably worth stating explicitly?

    How are ambiguous patterns (e.g., "/*/foo/*") handled?  Are there
    defined restrictions on the PathMatch objects to guarantee that
    all lookups will be deterministic?

  - In the ACLRule object, if you can only have either an allow or a
    deny Property but not both, and the ACLRule list can only have
    either Location or TimeWindow properties but not both, it is not
    clear that much is gained by combining them into the same ACL and
    ACLRule objects?  In the future, if new types of ACLs are added,
    would we modify these existing objects (and rev the CDNI Metadata
    RFC), or just add separate object definitions?

    The reuse of object definitions, in this case, does not seem to
    buy us anything?  Defining separate LocationACL/LocationACLRule
    objects and TimeWindowACL/TimeWindowACLRule objects, where
    allow/deny is just an attribute/property would result in
    essentially the same implementation, but without the
    interdependendies, which would simplify future revisions?

  - In the Property definitions, I assume that the "key" is the term
    that follows the "Property:" tag, and that the "Description:" and
    "Type:" indented below describe the "value"?

    It was not clear to me what the "Mandatory:" tag implied?
    Mandatory to specify the Property in every object?  Mandatory to
    implement support for the Property?  or Mandatory to enforce the
    Property?  It might be useful to explicitly define Mandatory?

  - In the HostMatch object, the hostname Property is defined as a
    String?  Should there be Pattern support for hostnames?  Also,
    does hostnames support both IP and DNS values?  Also, does the
    hostname support a URI prefix that should be matched as part of
    the hostname, and not be included in the PathMatch?

  - For Lists of objects, I did not see a specific ordering defined?
    Is there a way to enforce ordering?

  - With respect to extensibility, can an "x-" object be added to any
    other object, or are there restrictions on where a new custom
    object may be added into the hierarchy?

    Can I add a new ACL type?  Can I add a new pattern matching option
    to the HostMatch and/or PathMatch objects (fundamentally changing
    the way hosts and paths are matched)?  Can I add new types of
    location algorithms (e.g., GPS coordinates or zip code) to the
    Location object?  Or is it intended more for just creating a new
    hierarchy under the Delivery object?

  - With respect to the "ignorable" Property, does that affect the
    entire parent object to which it is added?  If it is a Property
    itself (rather than an attribute of a Property), then does that
    require a wrapper object around the actual Property and the
    "ingnorable" Property, for all new Properties?  Should it be a
    standard attribute like Mandatory?

  - Is there a way for the client to request metadata for a given URI,
    rather than recursively checking and requesting PathMetadata?
    Doesn't this lead to alot of lookups (even local cache lookups)
    for each request, to find all of the applicable PathMetadata?
    Are there optimizations for local cache lookup?

  Note: I have also uploaded an updated version of my draft:
        http://tools.ietf.org/html/draft-ma-cdni-metadata-03

thanx.

--  Kevin J. Ma

From: [email-masked] [mailto:[email-masked]] On Behalf Of Matt Caulfield (mcaulfie)
Sent: Monday, July 09, 2012 3:45 PM
To: [email-masked]
Cc: Ben Niven-Jenkins ([email-masked])
Subject: [CDNi] New Metadata Interface Draft (draft-cjlmw-cdni-metadata-00)

Hi everyone,

Thanks,
Matt

---

From: "Matt Caulfield (mcaulfie)" <[email-masked]>
Date: Tue, 17 Jul 2012 20:01:46 +0000

Hi Kevin,

Thank you for the comments, please see my responses inline below.

Regards,

Matt

From: Kevin J Ma [mailto:[email-masked]]
Sent: Monday, July 16, 2012 4:40 PM
To: Matt Caulfield (mcaulfie); [email-masked]
Cc: Ben Niven-Jenkins ([email-masked])
Subject: RE: New Metadata Interface Draft (draft-cjlmw-cdni-metadata-00)

Hi Matt,

  - I do not understand the need for a heirarchical separation of the
    Acquisition and Delivery objects?  Why is the property "key" not
    sufficient for designating the function of the metadata?
[Matt>] The separation was created to categorize metadata properties. It's not essential. It may be useful if one server in a CDN is delivery a piece of content and another server is acquiring. The acquisition server could limit itself to looking at the acquisition metadata while the delivery server could limit itself to looking at the delivery metadata. Would you suggest removing this distinction?

    Though it could be argued that these are logically different
    functions, the extra layer seems unnecessary, and introduces some
    complexity to the inheritance model?
[Matt>] I don't think that the extra layer introduces any additional complexity. In our model, objects can contain objects which can contain objects, the delivery/acquisition objects aren't a special case.

[Matt>] Any object at any level can be overridden. So yes, just the Auth list could be overridden, there is no need to override the entire delivery object. Yes, the schema must be preserved - so to override the auth list it would be wrapped by a delivery object. No, there is no concept of leaf vs. branch. Based on your comment, I think we could spend more time on the description of inheritance.

[Matt>] I'm not sure what you mean by duplicate metadata. Each PathMetadata object defines a bunch of metadata properties, each one is unique. Each PathMetadata object either inherits or overrides the metadata properties of its parent.

[Matt>] I agree, this approach is not scalable. The general rule is that for DNS-redirection, only the HostMetadata object can be referenced by a request router. Any metadata properties which are added and need to be available at request-routing time should be restricted to the HostMetadata object. I think we could consider relaxing the restrictions on the PathMetadata object and allow the operator to choose which properties to place in each object. Will look into this further.

[Matt>] I don't think this is a requirement. The interface doesn't break if someone adds what is effectively an unreachable child node. In practice it should be avoided and we could add a note stating so.

    How are ambiguous patterns (e.g., "/*/foo/*") handled?  Are there
    defined restrictions on the PathMatch objects to guarantee that
    all lookups will be deterministic?
[Matt>] I don't think we've specified the exact behavior of the regex. This will need to be added.

[Matt>] Using a common ACL and ACLRule definitions only avoids redundancy in the draft. A better description may have kept the TimeWindow ACL completely separate from the Location ACL (separate object definitions). If more metadata properties fit the "ACL" model, then they should be added alongside TimeWindow and Location.

[Matt>] Agree.

[Matt>] Correct.

[Matt>] Will need to add some definition of Mandatory to the text before we start using it. Mandatory means that that property must be defined in the metadata for a piece of content - there is no default value.

[Matt>] We debated whether or not to allow patterns in hostnames and decided against it. Ben can provide some of the background there. Yes, the hostname should support both IP an DNS values. No, the hostname does not include any other part of the URI. Will need to add some clarification on the hostname to the draft.

  - For Lists of objects, I did not see a specific ordering defined?
    Is there a way to enforce ordering?
[Matt>] I'm not sure I understand your question. All lists are ordered. Can you clarify?

  - With respect to extensibility, can an "x-" object be added to any
    other object, or are there restrictions on where a new custom
    object may be added into the hierarchy?
[Matt>] An "x-" object can be added anywhere.

[Matt>] It is intended for adding new metadata properties, however it could be used for changing the structure of the metadata tree or changing how we do path matches as you suggest.

[Matt>] Yes, the ignorable property affects the entire object and all the objects it contains. The "ignorable" property can be thought of as an attribute. In the XML encoding, it would likely be an attribute (and not an element). In JSON, yes, it can lead to wrapper object.

[Matt>] No, there's no way to get the metadata for a specific URL without traversing the tree. This approach enforces cacheability. As for local lookups, it's up to the CDN how it store metadata internally. The local lookup speed is independent of the metadata interface speed.

  Note: I have also uploaded an updated version of my draft:
        http://tools.ietf.org/html/draft-ma-cdni-metadata-03
[Matt>] Thanks, I will check it out.

thanx.

--  Kevin J. Ma

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Matt Caulfield (mcaulfie)
Sent: Monday, July 09, 2012 3:45 PM
To: [email-masked]<mailto:[email-masked]>
Cc: Ben Niven-Jenkins ([email-masked]<mailto:[email-masked]>)
Subject: [CDNi] New Metadata Interface Draft (draft-cjlmw-cdni-metadata-00)

Hi everyone,

Thanks,
Matt

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 17 Jul 2012 21:06:28 +0100

Hi Kevin,

Thanks for reading the draft, some good questions. My opinion inline.

>     sufficient for designating the function of the metadata?

This is a good question and something we discussed at length amongst the authors but probably don't really talk about in the draft.

We discussed a lot the trade offs between embedding all CDNI metadata properties together versus splitting some out into different object that could be independently cached/retrieved.

We decided not to try to optimise too early so we took an approach of allowing any object to be embedded or independently addressable. This leads to the data model being slightly more convoluted and it can certainly be optimised a bit IMO. But we decided on an approach where we first had a stable/agreed model with the WG and then we could revisit it see what optimisations make sense.
>     entire Delivery object?

This is an area where we need to flesh out the specification and not one we've fully fleshed out yet.

My current thinking is to only have to specify those properties which you want to override compared to the parent PathMetadata object. 

>  Does it have to place the Auth List
>     Auth List with no Delivery object wrapper?

Currently yes but when we revisit the data model to look for optimisations this is a good candidate.

>  Is there a concept of

I'm not exactly sure what you mean by leaf vs branch

>     restriction be stated explicity?

I don't think there's a need to check for duplicates explicitly as they represent different content items/paths/hosts but a uCDN may want to do so as part of optimising it's representation of the metadata but that's an implementation decision.

>     managing object restrictions.

I disagree. It is an editorial device to avoid repeating definitions for the same properties with identical text in multiple places in the document.

>  As we move forward and more vendor

We're not defining complex modification restrictions we're ruling out of scope those properties which simply do not make sense to include in a PathMetadata object.

How we handle vendor extensions is a separate issue, that I will admit needs some more work and the ideas in the current draft need refinement.

However both HostMetadata & PathMetadata objects would allow vendor extensions and it is up to the specification for those extensions to decide whether the appropriate place to place their extension is within the HostMetdata, PathMetadata, or both objects.

>     probably worth stating explicitly?

Yes. In your example there would not be a URL a client could request that would ever match both path patterns and so the CDNI Metdata associated with the /foo path would never get applied. I have no problem stating that more explicitly in the next version.
 
>     all lookups will be deterministic?

IMO The PathMatches are ordered and the first match wins, so in the case of ambiguous/overlapping patterns the uCDN needs to ensure it has ordered the PathMatches appropriately to get the desired deterministic behaviour.
 
>     RFC), or just add separate object definitions?

I think this is an example of where in trying to explain the concept of ACLs earlier on in the document we carried that abstract model maybe a little too far into the encoding. We should re-look at that part to see if we can clean it up along the lines you propose.
 
>     "Type:" indented below describe the "value"?

Correct.

>     Property?  It might be useful to explicitly define Mandatory?

Mandatory to specify the property in every object.

My thinking is that for a given delivery protocol there will be a mandatory set of properties/capabilities a dCDN is expected to have. If the dCDN cannot support the required "flavour" of the delivery protocol, it cannot perform the delivery (and would advertise that as part of its capabilities etc.). 

Generally in the CDNI Metadata specification, if a property is mandatory to specify in an object it is because it is essential for the delivery of that delivery protocol.

For a given delivery protocol there would need to be a separate specification (IMO) that specified the mandatory to implement/understand CDNI Metadata properties in order for a dCDN to claim "compliance" with a given delivery protocol.

>     String?  Should there be Pattern support for hostnames?

I'd need to think about that a bit more first but I think there might be some value to that.

>  Also,
>     does hostnames support both IP and DNS values?

I'm not sure of the use case for IP values as clients connecting to CDNs aren't typically given a URL with an IP address as a hostname.

>   Also, does the

No, the hostname is essentially the HTTP Host: header. All URL prefixes are via PathMetadata.

>     Is there a way to enforce ordering?

Lists are ordered. In JSON they would be an array with N entries, with index[0] being first and index[N-1] being last.

>     object may be added into the hierarchy?

It's up to whoever specifies the "x-" property to define which objects it applies to and what restrictions apply to it.

I think this is an area of the draft that isn't fully fleshed out and needs more thought and your input is welcome.

My current thinking is to split extensions clearly into "mandatory" and "optional" categories which could for example be expressed by having a property of "mandatory" which contained a dictionary of 'mandatory' extensions and likewise for optional extensions.

The basic rule would be if there is a property in the "mandatory" category that you don't understand then you cannot perform the content delivery. You can still perform the delivery if there are "optional" extensions you don't understand.

But like I say it's an area I think we definitely need to do some more thinking, refinement etc of.

>     hierarchy under the Delivery object?

IMO you can potentially do all those things. I think if you had lots of them as mandatory extensions you might be better off minting a vendor specific delivery protocol identifier to give dCDNs a heads-up that they needed to support a lot of vendor/extension-specific capabilities, but there are certainly other ways to do that too (depending how the capabilities advertisement interface evolves).

>     Are there optimizations for local cache lookup?

This partly goes back to optimising the data model once we have it stable. But also in reality I don't expect the layers of hierarchical metadata to extend to that many and if a dCDN is being selected one would hope that it is doing a fairly decent amount of delivery for that domain/uCDN and so it would need most of that portion of the metadata tree anyway.

So the initial lookup from the "root" of the CDNI Metadata tree to the PathMetadata for the domain/site is non optimal but the results are highly cacheable and so any penalty (assuming the dCDN doesn't prefetch the top couple of tiers of objects anyway) is only on first retrieval (from that point there's nothing to stop the dCDN pre-validating cache freshness for objects to make sure it always has a fresh version in it's cache).

>         http://tools.ietf.org/html/draft-ma-cdni-metadata-03 

Thanks. I'll give it a read when I get a chance.

Ben

> _______________________________________________
> CDNi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cdni

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Tue, 17 Jul 2012 21:21:26 +0100

Kevin, Matt,

Just zoning in on one comment

> [Matt>] We debated whether or not to allow patterns in hostnames and decided against it. Ben can provide some of the background there. Yes, the hostname should support both IP an DNS values. No, th…

Off hand I can't remember the obviously good reasons I used to convince Matt it was a bad idea.

I remember we discussed whether to allow a list of hosts in a HostMetadata and what occurs to me with that is that it doesn't buy you much over having multiple HostMetadata objects linking to the same PathMetadata objects but I'm sure I had other reasons for not doing it.

Having a special pattern to say all subdomains of this host might be useful but I need to think about it more in case it's ruled out by one of the previous reasons that I can't remember right now.

Ben

---

From: Rob Murray <[email-masked]>
Date: Wed, 18 Jul 2012 08:36:53 +0000

On 17/07/2012 21:21, "Ben Niven-Jenkins" <[email-masked]> wrote:

>> [Matt>] We debated whether or not to allow patterns in hostnames and
>>decided against it. Ben can provide some of the background there. Yes,
>>the hostname should support both IP an DNS values. No, the hostname does
>>not include any other part of the URI. Will need to add some
>>clarification on the hostname to the draft.
>

I think there were a couple of reasons, neither of them killer arguments
but they leant us towards the current position...

  - the HostIndex isn't ordered between CDNs, any given downstream
request-router or surrogate may have a HostIndex assembled from several
upstreams. If patterns aren't allowed in the HostMatch, either two CDNs
are advertising the same host or not. There's no need for any more
complicated disambiguation.

  - in an implementation, if patterns aren't allowed, the requested
hostname can be hashed to do the HostIndex lookup, there's no need to do a
linear search matching patterns across all known sites for every request.
(I'm sure there are ways to make the search more efficient, but this keeps
it simple.)

Another thought was that domains tend to be split for a reason - it seems
likely that different subdomains will have different metadata properties,
though we have no data to support that.

Rob.

>
>I remember we discussed whether to allow a list of hosts in a
>HostMetadata and what occurs to me with that is that it doesn't buy you
>much over having multiple HostMetadata objects linking to the same
>PathMetadata objects but I'm sure I had other reasons for not doing it.
>
>Having a special pattern to say all subdomains of this host might be
>useful but I need to think about it more in case it's ruled out by one of
>the previous reasons that I can't remember right now.
>
>Ben
>
>

---

From: Kevin J Ma <[email-masked]>
Date: Wed, 18 Jul 2012 15:10:24 -0400

Hi Matt,

  thanx for the reponse.  comments inline:

> From: Matt Caulfield (mcaulfie) [mailto:[email-masked]]
> Sent: Tuesday, July 17, 2012 4:02 PM
> To: Kevin J Ma; [email-masked]
> [Matt>] The separation was created to categorize metadata properties. It's
> not essential. It may be useful if one server in a CDN is delivery a piece
> of content and another server is acquiring. The acquisition server could
> limit itself to looking at the acquisition metadata while the delivery
> server could limit itself to looking at the delivery metadata. Would you
> suggest removing this distinction?

It raises question in my mind about how override works (see response to Ben).

> [Matt>] I don't think that the extra layer introduces any additional
> complexity. In our model, objects can contain objects which can contain
> objects, the delivery/acquisition objects aren't a special case.
> 
> [Matt>] Any object at any level can be overridden. So yes, just the Auth
> list could be overridden, there is no need to override the entire delivery
> object. Yes, the schema must be preserved - so to override the auth list
> it would be wrapped by a delivery object. No, there is no concept of leaf
> vs. branch. Based on your comment, I think we could spend more time on the
> description of inheritance.
> 
> [Matt>] I'm not sure what you mean by duplicate metadata. Each
> PathMetadata object defines a bunch of metadata properties, each one is
> unique. Each PathMetadata object either inherits or overrides the metadata
> properties of its parent.

I guess I was thinking more about scope, not across multiple PathMetadata
objects, but within a single one, e.g.:

Delivery has an auth property.  If someone add a new x-azuki property to
the Delivery property and the x-azuki property has its own auth property.
Is there a conflict?

Presumably the delivery/x-azuki/auth property is distinct from the
delivery/auth property?  And the precedence of the two auth properties
would be defined by the x-azuki/auth property?  If there was also a
delivery/x-cisco/auth property and a delivery/x-velocix/auth it gets
a little more complex.  Should there be restrictions on naming and/or
overriding existing metadata object functions, or is it just left up
to the implementors to do the right thing?

> [Matt>] I'm not sure I understand your question. All lists are ordered.

I do not see any text in the draft which defines what a "List" is, or
how they are ordered?  JSON arrays have inherent ordering, but XML lists
do not.  Also, JSON and XML may only be an on-the-wire encoding.  The
data model should specify an order from which to generate the JSON/XML?

> [Matt>] It is intended for adding new metadata properties, however it
> could be used for changing the structure of the metadata tree or changing
> how we do path matches as you suggest.
> 
> [Matt>] Yes, the ignorable property affects the entire object and all the
> objects it contains. The "ignorable" property can be thought of as an
> attribute. In the XML encoding, it would likely be an attribute (and not
> an element). In JSON, yes, it can lead to wrapper object.

So, would "ignorable" affect all children as well?  Or only the objects
in the current level, not including other objects referenced through a
PathMatch object?

> [Matt>] No, there's no way to get the metadata for a specific URL without
> traversing the tree. This approach enforces cacheability. As for local
> lookups, it's up to the CDN how it store metadata internally. The local
> lookup speed is independent of the metadata interface speed.

I'm not sure that cacheability is really affected.  I understand the desire
to be able to individually reference and store objects.  The recursive tree
structure does not really enhance that.  Having all the PathMatch objects in
a flat single layer structure would not change cacheability (or increase the
storage requirements), but it might make it easier to natively support lookup
for given URIs?

Though compacted representations could be generated for faster local lookups,
the un-compacted representation is still needed to be able to determine how
to refresh metadata or get more specific (additional PathMatch) metadata.

thanx.

--  Kevin J. Ma

> As we announced in April, a group of us (Ben Niven-Jenkins, Kent Leung,
> Rob Murray, Grant Watson, and I) have been working on a merge of two of
> the CDNI metadata interface proposals:

---

From: Kevin J Ma <[email-masked]>
Date: Wed, 18 Jul 2012 15:11:43 -0400

Hi Ben,

  thanx for the response.  comments inline:

> -----Original Message-----
> From: Ben Niven-Jenkins [mailto:[email-masked]]
> Sent: Tuesday, July 17, 2012 4:06 PM
> To: Kevin J Ma
> Cc: Matt Caulfield (mcaulfie); [email-masked]; Ben Niven-Jenkins
> ([email-masked])
> Subject: Re: [CDNi] New Metadata Interface Draft (draft-cjlmw-cdni-
> metadata-00)
> 
> Hi Kevin,
> 
> This is a good question and something we discussed at length amongst the
> authors but probably don't really talk about in the draft.
> 
> We discussed a lot the trade offs between embedding all CDNI metadata
> properties together versus splitting some out into different object that
> could be independently cached/retrieved.
> 
> We decided not to try to optimise too early so we took an approach of
> allowing any object to be embedded or independently addressable. This
> leads to the data model being slightly more convoluted and it can
> certainly be optimised a bit IMO. But we decided on an approach where we
> first had a stable/agreed model with the WG and then we could revisit it
> see what optimisations make sense.

Understood.  My specific concerns were with:
a) how extra levels might impact local cache implementations, i.e.,
   which objects can be compacted locally vs. how much the structure
   must be maintained, in order to understand how to update individual
   objects without traversing the entire recursion tree, and
b) how error prone inheritance-based override might be if there are
   alot of levels at which errant overrides could be applied.

> My current thinking is to only have to specify those properties which you
> want to override compared to the parent PathMetadata object.
>

My confusion comes from whether or not a Delivery object with an Auth List
inside it overrides the Delivery object or just the Auth object.  It could
be that the Delivery object is just a logical/organizational object that is
never overriden in its entirety, in which case it could be considered a
"branch", but the Auth List is a metadata object that requires actions,
which could classify it as a "leaf".

If my root Delivery object has a protocol, location ACL, and auth list,
and a Delivery object further down only specifies a location ACL:
a) is it valid, since it does not have a protocol?
b) does it only override the location ACL?
   or does it imply that no auth list should be used?
   i.e., do you have to re-specify every field?
         if not are there rules for what falls through?
c) if it only overrides the location ACL, if one wanted to not have an
   auth list at the lower level, does that require specifying an empty
   auth list to override the root level auth list?

As has been noted, we probably need to expand the discussion of the
inheritance rules.

[snip]
> I'm not sure of the use case for IP values as clients connecting to CDNs
> aren't typically given a URL with an IP address as a hostname.

For the public Internet I would agree, however, there are cases where MSOs
building internal CDNs prefer (for whatever reason) IP addresses.

> No, the hostname is essentially the HTTP Host: header. All URL prefixes

So, in a case where a given CDN requires some type of URI path prefix,
would the CDN specific URI path prefix live in the PathMatch, or would
it be up to the local CDN to recognize the URI path prefix, and then
match against the remainder of the URL?  e.g.,

  client requests cdn1.com/cdn1_prefix/video/x.m3u8
  cdn1 redirects to cdn2.com/cdn2_prefix/video/x.m3u8

Do we think this a valid use case?  If so, should the PathMatch have
a URI of "video/*.m3u8" or "cdnN_prefix/video/*.m3u8"?  If the latter,
whose responsibility is it to properly construct that PathMatch and
what information would need to be passed between CDNs to do that?

> Lists are ordered. In JSON they would be an array with N entries, with
> index[0] being first and index[N-1] being last.

XML is a little more problematic with unordered lists.

For this to work, the list itself must be cached as an object, as it
contains the explicit ordering?  It is not just the objects which it
references that need to be cached and refreshed, i.e., the list order
could change even if none of the objects in the list have changed?

If using the native JSON directly, it is probably implied, but as a
generic model for other implementations it might be worth explicitly
defining the properties of a list?

[snip]
> IMO you can potentially do all those things. I think if you had lots of
> them as mandatory extensions you might be better off minting a vendor
> specific delivery protocol identifier to give dCDNs a heads-up that they
> needed to support a lot of vendor/extension-specific capabilities, but
> there are certainly other ways to do that too (depending how the
> capabilities advertisement interface evolves).

Agreed.  Being overly flexible here could open up some interesting issues
wrt being able to fundamentally change the way mandatory properties behave.
But it does come down to how well the capabilities are advertised.

thanx.

--  Kevin J. Ma

> > The merged draft has now been submitted:
> http://tools.ietf.org/html/draft-cjlmw-cdni-metadata-00
> >

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Thu, 19 Jul 2012 19:17:24 +0100

> complicated disambiguation.

Ah yes, that was the primary reason I didn't like wildcarded hosts in HostIndex/HostMetdata. It leads to non-determinism in dCDNs if they have multiple uCDNs advertising CDNI Metadata for the same host wildcard.

Ben

---

From: Kevin J Ma <[email-masked]>
Date: Thu, 19 Jul 2012 15:28:15 -0400

Could be worth a note in the draft, for future reference...

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Ben Niven-Jenkins
> Sent: Thursday, July 19, 2012 2:17 PM
> To: Rob Murray
> Cc: Ben Niven-Jenkins; [email-masked]
> Subject: Re: [CDNi] Host pattern matches in CDNI Metadata (was Re: New
> Metadata Interface Draft (draft-cjlmw-cdni-metadata-00))
> 
> 
> 
> Ah yes, that was the primary reason I didn't like wildcarded hosts in
> HostIndex/HostMetdata. It leads to non-determinism in dCDNs if they have
> multiple uCDNs advertising CDNI Metadata for the same host wildcard.
> 
> Ben
> 
> >

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Thu, 19 Jul 2012 21:18:03 +0100

Kevin,

More responses inline.

>   objects without traversing the entire recursion tree, and

If an implementation wants to maintain a local CDNI metadata cache and compact/fold the multiple levels into a single "resource" in that local cache it is free to do so and allowing arbitrary levels Vs a fixed number of levels doesn't affect that.

The main impact to the local cache implementation is that compacting/folding hurts cacheability & may result in the size of the local cache having to be larger than the total size of all the individual resources (if individual resources are referenced by multiple other resources). The local cache must use the most conservative Expiry/TTL of all the compacted/folded resources. If the individual resources being compacted/folded have significantly different TTLs (e.g. I could imagine the TTLs for Location objects could be significantly longer than for HostMetadata/SiteMetadata objects) then some resources will need to be re-validated more often than would be necessary if the local cache was not performing compacting/folding.

>   alot of levels at which errant overrides could be applied.

Error prone in what way?

I don't see that dCDN implementations become more error prone because we allow multiple levels as they will just reuse the same processing code for each level.

I can see that it might make it more likely that the "content provider user" configuring the CDN/CDNI Metadata in the first place may be more likely to make an error and forget to override a property but I don't see that as much as a problem as in reality I don't expect there to be that many levels actually used.

I have a design principle that once you go past requiring 2 of something, you may as well design for allowing an arbitrary number of those things rather than choosing an artificial fixed value as a maximum.

> inside it overrides the Delivery object or just the Auth object.

IMO it would just override the Auth object.

>  It could

Yes it is valid.

>   or does it imply that no auth list should be used?

It just overrides the Location ACL.

>         if not are there rules for what falls through?

Everything falls through unless it is overridden by the child object.

>   auth list to override the root level auth list?

It would require specifying an empty Auth list.

> As has been noted, we probably need to expand the discussion of the

Agreed.

> building internal CDNs prefer (for whatever reason) IP addresses.

Let me understand this use case a bit more.

Is it that the MSO/CDN operator prefers that redirects are performed to IP addresses?

or

That the value of the Host: header the client presents in its GET request is an IP address?

I'm aware of the former, I'm not aware of the latter.

> what information would need to be passed between CDNs to do that?

CDNI Metadata is expressed by the uCDN to the dCDN. If the dCDN needs to use additional path prefixes material so that it can understand incoming requests that have been redirected to it from another CDN then it may need to tranform the CDNI Metadata it received from the uCDN when expressing that CDNI Metadata to other dCDNs in the chain. It may also be able to do the same through a mechanical conversion that doesn't require transforming the CDNI Metadata it receives from the uCDN.

So in the case you have above, if you took a CDNI Metadata feed from both CDN1 & CDN2 the PatternMatch they each advertise for the same content /video/x.m3u8 would be different (tailored to their own requirements for path_prefixing to enable them to remap to the original content request) but the delivery properties associated with those different PatternMatches would be the same (assuming dCDNs are not allowed to change/override the original delivery properties).

> XML is a little more problematic with unordered lists.

Good job my preference is that we use a JSON encoding then :-)

(we put both in the draft as we couldn't reach consensus among the authors whether to use JSON or XML)

> defining the properties of a list?

Sure we can do that.

Ben

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Thu, 19 Jul 2012 22:01:34 +0100

Kevin, Matt,

Jumping in on one point below.

> the Delivery property and the x-azuki property has its own auth property.

IMO it should be up to the specification of the x-azuki property to define what it's behaviour is where there is an interaction between its presence and the presence of another property.

> a little more complex.

Not really unless someone is expecting to use the delivery/x-azuki/auth property alongside the delivery/x-velocix/auth property to get a different behaviour than if they'd only included one or the other of the two properties.

But that's what you get if you try to use two vendor-specific extensions from different vendors at the same time in the hope that the presence of them both will give you a behaviour that neither by themselves would give you. It'll be the same situation regardless of how we represent vendor specific extensions IMO.

Ben

>  Should there be restrictions on naming and/or

---

From: Kevin J Ma <[email-masked]>
Date: Thu, 19 Jul 2012 17:27:23 -0400

Hi Ben,

  comments inline:

> Sent: Thursday, July 19, 2012 4:18 PM
> To: Kevin J Ma

[snip]
> >   alot of levels at which errant overrides could be applied.

Error prone in the definition and resolution of object names/hierarchies
going forward, especially wrt overrides.  I personally prefer separation
of metadata definitions (e.g., how do you construct an ACL list) from the
metadata concept (i.e., we need a location ACL list).  Please see the
continued discussion below...

[snip]
> Everything falls through unless it is overridden by the child object.

So, the child is a leaf in my terms?  And only leaf objects are overridable.
But the Location ACL is actualy made up of other objects (i.e., ACL and
ACLRule objects, and Location/TimeWindow objects below that), so it's not
actually a leaf object in the data model defined in the draft, but it is a
leaf object in what I see as the CDNI metadata model abstraction, which
has the distinction between CDNI metadata properties (i.e., the concept),
and the ancillary objects that make up the value (i.e., the definition).

The Location ACL may be an object, and that object may be serializable into
the JSON just as it is currently defined, but it's not clear to me that we
would want to allow individual portions of that object to be overridden?
The current model doesn't specify which objects can and cannot be overridden.
If we want certain objects (and all their children) to be treated as atomic
units, then I think that needs to be represented in the data model.

[snip]
> I'm aware of the former, I'm not aware of the latter.

We have seen cases where there is no DNS in the network and the client is
only presented with an IP address.  I don't know how prevalent it is, but
it should not really be an issue to support, should it?

> CDNI Metadata is expressed by the uCDN to the dCDN. If the dCDN needs to
> use additional path prefixes material so that it can understand incoming
> requests that have been redirected to it from another CDN then it may need
> to tranform the CDNI Metadata it received from the uCDN when expressing
> that CDNI Metadata to other dCDNs in the chain. It may also be able to do
> the same through a mechanical conversion that doesn't require transforming
> the CDNI Metadata it receives from the uCDN.
> 
> So in the case you have above, if you took a CDNI Metadata feed from both
> CDN1 & CDN2 the PatternMatch they each advertise for the same content
> /video/x.m3u8 would be different (tailored to their own requirements for
> path_prefixing to enable them to remap to the original content request)
> but the delivery properties associated with those different PatternMatches
> would be the same (assuming dCDNs are not allowed to change/override the
> original delivery properties).

So, CDN1 would receive /video/*.m3u8 (from either a uCDN or CP), and add in
the /cdn1_prefix to its local PathMatch objects resulting in local PathMatch
objects with /cdn1_prefix/video/*.m3u8.  When CDN2 requested the metadata,
CDN1 would provide /cdn1_prefix/video/x.m3u8 to CDN2, rather than stripping
off the /cdn1_prefix and providing /video/*.m3u8?

CDN2 would then need to know to strip off the /cdn1_prefix, before adding
its own /cdn2_prefix to its PathMatch objects?  (Presumably the request
routing already knows about the required rewrite between CDN1 and CDN2.)

thanx.

--  Kevin J. Ma

---

From: Kevin J Ma <[email-masked]>
Date: Thu, 19 Jul 2012 17:37:32 -0400

Hi Ben,

  inline:

> Sent: Thursday, July 19, 2012 5:02 PM
> To: Kevin J Ma
> IMO it should be up to the specification of the x-azuki property to define
> what it's behaviour is where there is an interaction between its presence
> and the presence of another property.

Fair enough.

thanx.

--  Kevin J. Ma

> Not really unless someone is expecting to use the delivery/x-azuki/auth
> property alongside the delivery/x-velocix/auth property to get a different
> behaviour than if they'd only included one or the other of the two
> properties.
> 
> But that's what you get if you try to use two vendor-specific extensions
> from different vendors at the same time in the hope that the presence of
> them both will give you a behaviour that neither by themselves would give
> you. It'll be the same situation regardless of how we represent vendor
> specific extensions IMO.
> 
> Ben
>

---

From: "Matt Caulfield (mcaulfie)" <[email-masked]>
Date: Mon, 23 Jul 2012 15:25:41 +0000

Hi Kevin,

Your feedback is very helpful.

My responses inline.

Thanks,
Matt

-----Original Message-----
From: Kevin J Ma [mailto:[email-masked]] 
Sent: Wednesday, July 18, 2012 3:10 PM
To: Matt Caulfield (mcaulfie); [email-masked]
Cc: Ben Niven-Jenkins ([email-masked])
Subject: RE: New Metadata Interface Draft (draft-cjlmw-cdni-metadata-00)

Hi Matt,

  thanx for the reponse.  comments inline:

> [Matt>] The separation was created to categorize metadata properties. 
> It's not essential. It may be useful if one server in a CDN is 
> delivery a piece of content and another server is acquiring. The 
> acquisition server could limit itself to looking at the acquisition 
> metadata while the delivery server could limit itself to looking at 
> the delivery metadata. Would you suggest removing this distinction?

It raises question in my mind about how override works (see response to Ben).
[Matt>] I think I understand your confusion. The best way to think about the inheritance rules is not by what overrides what. Instead, imagine that the dCDN needs to know the value of parameter X. The dCDN has already found the branch of HostMetadata and PathMetadata which apply to the request. To find the value of parameter X, the dCDN first looks for that parameter in the leaf PathMetadata. If the value is not found, it looks in the parent, and if it's not found, looks in the parent's parent, all the way up. If the parameter value is not found, it takes on the default value. 

> complexity. In our model, objects can contain objects which can 
> contain objects, the delivery/acquisition objects aren't a special case.
> 
> [Matt>] Any object at any level can be overridden. So yes, just the 
> Auth list could be overridden, there is no need to override the entire 
> delivery object. Yes, the schema must be preserved - so to override 
> the auth list it would be wrapped by a delivery object. No, there is 
> no concept of leaf vs. branch. Based on your comment, I think we could 
> spend more time on the description of inheritance.
> 
> PathMetadata object defines a bunch of metadata properties, each one 
> is unique. Each PathMetadata object either inherits or overrides the 
> metadata properties of its parent.

I guess I was thinking more about scope, not across multiple PathMetadata objects, but within a single one, e.g.:

Delivery has an auth property.  If someone add a new x-azuki property to the Delivery property and the x-azuki property has its own auth property.
Is there a conflict?

Presumably the delivery/x-azuki/auth property is distinct from the delivery/auth property?  And the precedence of the two auth properties would be defined by the x-azuki/auth property?  If there was also a delivery/x-cisco/auth property and a delivery/x-velocix/auth it gets a little more complex.  Should there be restrictions on naming and/or overriding existing metadata object functions, or is it just left up to the implementors to do the right thing?
 [Matt>] I think Ben supplied a good answer to this question.

> [Matt>] I'm not sure I understand your question. All lists are ordered.

I do not see any text in the draft which defines what a "List" is, or how they are ordered?  JSON arrays have inherent ordering, but XML lists do not.  Also, JSON and XML may only be an on-the-wire encoding.  The data model should specify an order from which to generate the JSON/XML?
[Matt>] Noted, we will need to add a description of Lists to the next revision in the data model section. In XML encoding, we'll need to add an attribute which provides the list index (this was an oversight).

> could be used for changing the structure of the metadata tree or 
> changing how we do path matches as you suggest.
> 
> [Matt>] Yes, the ignorable property affects the entire object and all 
> the objects it contains. The "ignorable" property can be thought of as 
> an attribute. In the XML encoding, it would likely be an attribute 
> (and not an element). In JSON, yes, it can lead to wrapper object.

So, would "ignorable" affect all children as well?  Or only the objects in the current level, not including other objects referenced through a PathMatch object?
[Matt>] "ignorable" affects all children as well. Will need to clarify this in the next revision when we improve the extensibility section.

> [Matt>] No, there's no way to get the metadata for a specific URL 
> without traversing the tree. This approach enforces cacheability. As 
> for local lookups, it's up to the CDN how it store metadata 
> internally. The local lookup speed is independent of the metadata interface speed.

I'm not sure that cacheability is really affected.  I understand the desire to be able to individually reference and store objects.  The recursive tree structure does not really enhance that.  Having all the PathMatch objects in a flat single layer structure would not change cacheability (or increase the storage requirements), but it might make it easier to natively support lookup for given URIs?
 
[Matt>] With respect to cacheability, I was referring to the idea of requesting metadata for a given URI. Assume that we form a request for a metadata object by putting the content request URL in the query string to the metadata server. If this is the approach, then I don't think that HTTP caching can help us. 

[Matt>] With respect to flat vs tall tree, I think you're right - cacheability is not affected. However, I think that the multi-level tree provides better space efficiency for a few reasons. 1) With the flat tree there could be redundancy between siblings. In a multi-level model, any redundant metadata would be kept in a single parent node. 2) In a flat tree, the root node would be much larger than the multi-level tree. 3) In a multi-level tree, the cache TTL of each node could be proportional to its tree level (the root could have a long TTL).

Though compacted representations could be generated for faster local lookups, the un-compacted representation is still needed to be able to determine how to refresh metadata or get more specific (additional PathMatch) metadata.
[Matt>] This information could be stored in whatever format is most reasonable to the dCDN. It need not be "un-compacted". 

thanx.

--  Kevin J. Ma

> Subject: [CDNi] New Metadata Interface Draft 
> (draft-cjlmw-cdni-metadata-
> 00)
> 
> Hi everyone,
> 
> As we announced in April, a group of us (Ben Niven-Jenkins, Kent 
> Leung, Rob Murray, Grant Watson, and I) have been working on a merge 
> of two of the CDNI metadata interface proposals:

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Wed, 25 Jul 2012 19:53:43 +0100

Kevin,

> it should not really be an issue to support, should it?

Right, it's not an issue, we can change the draft to make allowing IP addresses more explicit.

I was trying to understand the use case more to check it was actually interconnectable but I think it is, so that's OK.

Ben