Subject: Related entries and feeds: links and link expansion

From: Pablo Castro <[email-masked]>
Date: Mon, 18 Feb 2008 22:23:26 -0800

While going through application scenarios for the ADO.NET Data Services Framework (Project Astoria) one of the first things we noticed is that data-centric applications usually want to bring down graphs of related resources in each interaction with the server. For example, if you are retrieving a resource that represents an "Event", you may want to also bring in the set of related Contact resources that are invited or the "Venue" resource where the event will take place. This write up briefly describes how we model associations between resources as Atom links and proposes a usage pattern of the atom:link element to support retrieving resource graphs in a single response. We're looking for feedback on the approach and also to get folks thinking about inlined content and whether it should be considered an extension to Atom.

More context on Astoria support for Atom here:
http://blogs.msdn.com/astoriateam/archive/2008/02/13/atompub-support-in-the-ado-net-data-services-framework.aspx

1. Links for modeling associations between resources

Related resources can be seen at the instance level as "links" in Atom terms. Of course, from the data application development perspective, it's interesting to make this discoverable at the service description (schema) level. In Astoria data services the underlying model is the Entity Data Model (EDM), which describes data in terms of "Entities" (instances of Entity Types) and associations between entities. In the context of the Atom interface, Entities are mapped to entries and Associations to links. So by looking at the service description a developer can discover the links that will be present in an entry of a given type.

We model related entries or feeds using a link with a "rel" attribute of "related", and with a "type" of either "application/atom+xml;type=feed" or "application/atom+xml;type=entry" depending on the cardinality of the other end of the association.

One tricky aspect is that we need to indicate which association it is. At the model level we have a "navigation property" that identifies the starting "end" of the association (e.g. "Attendees", "Venue"). We currently put that name in the "title" attribute of the link. That solution is not perfect, as we try not to overload constructs that are for human-readable content. However, the alternative is to use a custom attribute, and we've been trying not to introduce custom attributes unless absolutely needed. Another option would be to use different "rel" values to specify the relationship, which feels natural but makes it much less likely that generic processors will be able to do something interesting with it.

Do these trade-offs sound reasonable? Is any of the other options more appropriate?

Continuing with the Events sample, this is what an entry (/Events(456)) with links looks like:

<entry xml:base="http://localhost:81/EventsSample/"
       xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
       xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
       xmlns="http://www.w3.org/2005/Atom"
       m:type="EventsSample.Event">
  <id>http://localhost:81/EventsSample/Events(456)</id>
  <title type="text"></title>
  <updated>2008-02-17T02:52:38Z</updated>
  <author>
    <name />
  </author>
  <link rel="edit" title="Event" href="Events(456)" />
  <link rel="related" type="application/atom+xml;type=entry" title="Venue" href="Events(456)/Venue" />
  <link rel="related" type="application/atom+xml;type=feed" title="Attendees" href="Events(456)/Attendees" />
  <content type="application/xml">
    <d:EventID m:type="Int32">456</d:EventID>
    <d:Name>Big Party</d:Name>
    <d:NoteToAttendees>It's going to be a great party!</d:NoteToAttendees>
    <d:DateAndTime m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>
  </content>
</entry>

From the data modification perspective, links pointing to other resources in the service can be specified in the payload of POST and PUT operations, to establish links between the resource being manipulated and other existing resources.

2. Expanding links inline

As I summarized at the beginning of this note, we want to enable clients to request whole sub-graphs of data starting at some resource or set of resources. There are two aspects that need to be addressed: how does the client indicate that it wants one or more links expanded and how are the expanded links represented on the response.

How link expansion is requested is outside of the atom-syntax problem space, so I'll just briefly state what we currently do in case you have an opinion: data services support the query string option "$expand" to request link expansion. So you could say "/Events?$expand=Attendees " to retrieve all Events and all contacts that are attendees for each of them, or "/Events(456)?$expand=Attendees" to retrieve a single event (with key 456) and its attendees. Expand syntax allows for deep expands such as "Attendees/BestFriend" (expand Attendees, and on the expanded entry(es) expand BestFriend) and wide expands such as "Venue, Attendees/BestFriend" meaning expand two immediate links, and for the Attendees one further expand its BestFriend link.

For representing expanded links we put the expanded content inside the link element itself. According to section 4.2.7 of RFC 4287:

"The "atom:link" element defines a reference from an entry or feed to  a Web resource.  This specification assigns no meaning to the content (if any) of this element."

So it seems that adding content to the link element is not disallowed and at the same time it does not overlap with any existing semantics given to such construct. Based on that we thought it would be the perfect place for this information, as the link itself already contains the metadata about the link that we needed.

When a client indicates that the target of a link should be expanded, the server responds with the Atom representation of the resources pointed at by links wrapped in an <inline> element. For example, for "/Events(456)?$expand=Attendees,Venue" the response would be:

<entry xml:base="http://localhost:81/EventsSample/"
       xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
       xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
       xmlns="http://www.w3.org/2005/Atom"
       m:type="EventsSample.Event">
  <id>http://localhost:81/EventsSample/Events(456)</id>
  <title type="text"></title>
  <updated>2008-02-17T03:01:18Z</updated>
  <author>
    <name />
  </author>
  <link rel="edit" title="Event" href="Events(456)" />
  <link rel="related" type="application/atom+xml;type=entry" title="Venue" href="Events(456)/Venue">
    <m:inline>
      <entry m:type="EventsSample.Venue">
        <id>http://localhost:81/EventsSample/Venues(789)</id>
        <title type="text"></title>
        <updated>2008-02-17T03:01:18Z</updated>
        <author>
          <name />
        </author>
        <link rel="edit" title="Venue" href="Venues(789)" />
        <link rel="related" type="application/atom+xml;type=entry" title="SalesContact" href="Venues(789)/SalesContact" />
        <content type="application/xml">
          <d:VenueID m:type="Int32">789</d:VenueID>
          <d:Name>The Cool Place</d:Name>
          <d:Description>Great place for parties!</d:Description>
          <d:Capacity m:type="Int32">1500</d:Capacity>
          <d:Type>Nightclub</d:Type>
        </content>
      </entry>
    </m:inline>
  </link>
  <link rel="related" type="application/atom+xml;type=feed" title="Attendees" href="Events(456)/Attendees">
    <m:inline>
      <feed>
        <title type="text">Attendees</title>
        <id>http://localhost:81/EventsSample/Events(456)/Attendees</id>
        <updated>2008-02-17T03:01:18Z</updated>
        <link rel="self" title="Attendees" href="Events(456)/Attendees" />
        <entry m:type="EventsSample.Contact">
          <id>http://localhost:81/EventsSample/Contacts(123)</id>
          <title type="text"></title>
          <updated>2008-02-17T03:01:18Z</updated>
          <author>
            <name />
          </author>
          <link rel="edit" title="Contact" href="Contacts(123)" />
          <link rel="related" type="application/atom+xml;type=entry" title="BestFriend" href="Contacts(123)/BestFriend" />
          <content type="application/xml">
            <d:ContactID m:type="Int32">123</d:ContactID>
            <d:FirstName>John123</d:FirstName>
            <d:LastName>Doe123</d:LastName>
            <d:EmailAddress>[email-masked]</d:EmailAddress>
            <d:Phone>123-456-123</d:Phone>
            <d:BirthDate m:type="Nullable`1[System.DateTime]">1990-04-01T00:00:00</d:BirthDate>
          </content>
        </entry>
        <entry m:type="EventsSample.Contact">
          <id>http://localhost:81/EventsSample/Contacts(124)</id>
          <title type="text"></title>
          <updated>2008-02-17T03:01:18Z</updated>
          <author>
            <name />
          </author>
          <link rel="edit" title="Contact" href="Contacts(124)" />
          <link rel="related" type="application/atom+xml;type=entry" title="BestFriend" href="Contacts(124)/BestFriend" />
          <content type="application/xml">
            <d:ContactID m:type="Int32">124</d:ContactID>
            <d:FirstName>John124</d:FirstName>
            <d:LastName>Doe124</d:LastName>
            <d:EmailAddress>[email-masked]</d:EmailAddress>
            <d:Phone>123-456-124</d:Phone>
            <d:BirthDate m:type="Nullable`1[System.DateTime]">1990-05-01T00:00:00</d:BirthDate>
          </content>
        </entry>
        <!-- more entries for contacts that will be attendees in this party -->
      </feed>
    </m:inline>
  </link>
  <content type="application/xml">
    <d:EventID m:type="Int32">456</d:EventID>
    <d:Name>Big Party</d:Name>
    <d:NoteToAttendees>It's going to be a great party!</d:NoteToAttendees>
    <d:DateAndTime m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>
  </content>
</entry>

I focused on the GET operations above. We think it would be better to stay away from attempting to support full modification operations on expanded graphs. In particular, we do not handle PUT on more than one entry at a time today. We do support POSTing an expanded graph, and we simply create all the nested entries and link them to the parent entry, creating the whole graph in a single operation.

Feedback in general about this approach would be greatly appreciated.

(also posted this in the Astoria team blog)

Thanks
-pablo

---

From: James M Snell <[email-masked]>
Date: Tue, 19 Feb 2008 08:23:11 -0800

Hello Pablo,

Thanks for the runthrough... comments below..

Pablo Castro wrote:
> [snip]
> We model related entries or feeds using a link with a "rel" attribute of "related", and with 
> a "type" of either "application/atom+xml;type=feed" or "application/atom+xml;type=entry" 
> depending on the cardinality of the other end of the association.
> 
> One tricky aspect is that we need to indicate which association it is. At the model level 
> we have a "navigation property" that identifies the starting "end" of the association (e.g. 
> "Attendees", "Venue"). We currently put that name in the "title" attribute of the link. 
> That solution is not perfect, as we try not to overload constructs that are for human-readable 
> content. However, the alternative is to use a custom attribute, and we've been trying not to 
> introduce custom attributes unless absolutely needed. Another option would be to use different 
> "rel" values to specify the relationship, which feels natural but makes it much less likely 
> that generic processors will be able to do something interesting with it.
> 

Most generic processors aren't doing much with links yet anyway.  Using 
specific rel attribute values would be better.

>   <link rel="related" type="application/atom+xml;type=feed" title="Attendees" href="Events(456)/Attendees" />

I'd much rather see this:

<link
    rel="http://schemas.microsoft.com/ado/2007/08/dataservices/venue"
    type="application/atom+xml;type=entry"
    title="Venue"
    href="Events(456)/Venue" />
<link
    rel="http://schemas.microsoft.com/ado/2007/08/dataservices/attendees"
    type="application/atom+xml;type=feed"
    title="Attendees"
    href="Events(456)/Attendees" />

>     <d:DateAndTime m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>

FWIW, your content element is incorrect.  There needs to be a top level 
element, e.g.

  <content type="application/xml">
    <d:foo>
    <d:EventID m:type="Int32">...</d:EventID>
    <d:Name>...</d:Name>
    <d:NoteToAttendees>...</d:NoteToAttendees>
    <d:DateAndTime m:type="DateTime">...</d:DateAndTime>
    </d:foo>
  </content>

 > [snip]
> [snip]
> How link expansion is requested is outside of the atom-syntax problem space, so 
> I'll just briefly state what we currently do in case you have an opinion: data services 
> support the query string option "$expand" to request link expansion. So you could say 
> "/Events?$expand=Attendees " to retrieve all Events and all contacts that are attendees 
> for each of them, or "/Events(456)?$expand=Attendees" to retrieve a single event (with 
> key 456) and its attendees. Expand syntax allows for deep expands such as 
> "Attendees/BestFriend" (expand Attendees, and on the expanded entry(es) expand BestFriend) 
> and wide expands such as "Venue, Attendees/BestFriend" meaning expand two immediate links, 
> and for the Attendees one further expand its BestFriend link.
> 

Works for me :-)

> "The "atom:link" element defines a reference from an entry or feed to  a Web resource.  
> This specification assigns no meaning to the content (if any) of this element."
> 
> So it seems that adding content to the link element is not disallowed and at the same time 
> it does not overlap with any existing semantics given to such construct. Based on that we 
> thought it would be the perfect place for this information, as the link itself already 
> contains the metadata about the link that we needed.
> 

+1. This is an excellent use of the extensibility of Atom.

> When a client indicates that the target of a link should be expanded, the server responds 
> with the Atom representation of the resources pointed at by links wrapped in an <inline> 
> element. For example, for "/Events(456)?$expand=Attendees,Venue" the response would be:
> 
> [snip]
> [snip]

I like this approach very much.  Existing (properly implemented) 
processors will be able to safely ignore the inline content without loss 
of function and would still have the ability to extract useful 
information from the document.

> I focused on the GET operations above. We think it would be better to stay away from 
> attempting to support full modification operations on expanded graphs. In particular, 
> we do not handle PUT on more than one entry at a time today. We do support POSTing an 
> expanded graph, and we simply create all the nested entries and link them to the 
> parent entry, creating the whole graph in a single operation.
> 

Sounds good.  PUT would definitely be problematic.

- James

---

From: Pablo Castro <[email-masked]>
Date: Tue, 19 Feb 2008 15:49:04 -0800

Hello James,

Thanks for the quick feedback. Good point on the rel values of links, my only concern is what will generic processors do with that, but I like that it captures the data that's for other computers in the "rel" attribute and frees up Title to be used for something readable.

>> FWIW, your content element is incorrect.  There needs to be a top level element, e.g.

I'll claim ignorance on this. Why do we need a wrapping element? Not particularly passionate one way or the other, just curious to hear why is it required...

Thanks
-pablo

-----Original Message-----
From: James M Snell [mailto:[email-masked]]
Sent: Tuesday, February 19, 2008 8:23 AM
To: Pablo Castro
Cc: [email-masked]
Subject: Re: Related entries and feeds: links and link expansion

Hello Pablo,

<link

Works for me :-)

- James

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 19 Feb 2008 21:10:54 -0800

Pablo Castro wrote:
> One tricky aspect is that we need to indicate which 
> association it is. At the model level we have a "navigation 
> property" that identifies the starting "end" of the 
> association (e.g. "Attendees", "Venue"). We currently put 
> that name in the "title" attribute of the link. That solution 
> is not perfect, as we try not to overload constructs that are 
> for human-readable content. However, the alternative is to 
> use a custom attribute, and we've been trying not to 
> introduce custom attributes unless absolutely needed. Another 
> option would be to use different "rel" values to specify the 
> relationship, which feels natural but makes it much less 
> likely that generic processors will be able to do something 
> interesting with it.

There is no point in including a link to something totally unrelated to
the entry, so it is safe to assume that *any* link references related
content. rel="related" provides no useful information except to say that
the link is not an alternate link (rel="alternate" is the default).
Generic processors are unlikely to do anything with rel="related" that
they wouldn't do for some unknown link relation.

Instead of *custom* relations, we should develop common vocabularies for
different domains. But, that goes against the idea of allowing users to
shoehorn their existing ERD models into Atom documents, because it
requires them to design with Atom in mind beforehand.

> <entry>
>  <link rel="related"
> <link rel="related"
>       title="Attendees" href="Events(456)/Attendees" />
>    <d:DateAndTime
m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>
> </entry>

In this example, most of the elements of the Atom syntax (id, title,
author, link/@rel, link/@title, summary) are either unused or misused.
If the goal is to interoperate with other Atom implementations, I would
expect the entry to look something like this:

  <entry>
   <id>http://organization.org/events/Events(456)</id> 
   <!-- http://localhost... is never going to be a good atom:id
        because it is not even attempting to be globally unique.
        If this entry represents an event, then there is no need
        to have atom:id separate from the event ID. I actually
        recommend using urn:uuid for atom:id all the time, but
        that is a separate issue for another day... -->
   <title type="text">Big Party</title>
   <author>
     <name>A useful name</name>
   </author>
   <category scheme='http://organization.org/events' term='Event'/>
   <link rel="edit" title="Event" href="Events(456)" />
   <link rel="http://organization.org/events/Venue"
         type="application/atom+xml;type=entry"
         title="Venue"
         href="Events(456)/Venue" />
   <link rel="http://organization.org/events/Attendees"
         type="application/atom+xml;type=feed"
         title="Attendees" href="Events(456)/Attendees" />
   <summary>It's going to be a great party!</summary>
   <content type="application/xml">
     <d:DateAndTime
m:type="DateTime">2008-03-05T06:00:00Z</d:DateAndTime>
     <!--Use valid Atom date constructs to represent dates/times -->
   </content>
  </entry>

If you cannot nicely map the user's domain model into Atom's general
purpose data model then that is a really good indication that Atom
shouldn't be used. Otherwise, you aren't going to be able to
interoperate usefully with anybody except other Astoria users.

> As I summarized at the beginning of this note, we want to 
> enable clients to request whole sub-graphs of data starting 
> at some resource or set of resources. There are two aspects 
> that need to be addressed: how does the client indicate that 
> it wants one or more links expanded and how are the expanded 
> links represented on the response.

> For representing expanded links we put the expanded content 
> inside the link element itself. According to section 4.2.7 of 
> RFC 4287:
> 
> "The "atom:link" element defines a reference from an entry or 
> feed to  a Web resource.  This specification assigns no 
> meaning to the content (if any) of this element."
> 
> So it seems that adding content to the link element is not 
> disallowed and at the same time it does not overlap with any 
> existing semantics given to such construct. Based on that we 
> thought it would be the perfect place for this information, 
> as the link itself already contains the metadata about the 
> link that we needed.

How do you deal with other extensions that also embed information within
the content of atom:link elements? How can you distinguish these other
extensions from the expanded data?

> I focused on the GET operations above. We think it would be 
> better to stay away from attempting to support full 
> modification operations on expanded graphs. In particular, we 
> do not handle PUT on more than one entry at a time today. We 
> do support POSTing an expanded graph, and we simply create 
> all the nested entries and link them to the parent entry, 
> creating the whole graph in a single operation.
> 

Instead of expanding the links within an entry, why not return a feed,
where each entry represents one entity in the result set, and make use
of link/@rel='alternate' or link/@rel='self'. For example:

 <feed>
    ...
   <entry>
     ...
     <link rel="http://organization.org/events/Venue"
           href="Events(456)/Venue"/>
     <link rel="http://organization.org/events/Attendees"
           href="Events(456)/Attendees"/>
     ...
     <category scheme='http://organization.org/events' term='Event'/>
   </entry>
   <entry>
     ...
     <link rel='self' href='Events(456)/Venue'/>
     <category scheme='http://organization.org/events' term='Venue'/>
     ...
   </entry>
   <entry>
     ...
     <link rel='self' href='Events(456)/Attendees'/>
     <category scheme='http://organization.org/events' term='Person'/>
     ...
   </entry>
 </feed>

The consumer of the feed can see that the second and third entries are
copies of the resources that are being linked to, which avoids the need
to hit the server to retrieve them. This is leveraging the pre-defined
Atom semantics instead of creating a new mechanism. In other words,
treat "$expand" as a request to include supplementary entries in the
feed. You might need a way of distinguishing the primary and
supplementary entries, but that shouldn't be too hard to come up with.

Notice that this scheme also seems to be easy to extend to support batch
updates of multiple parts of the subgraph using whatever general batch
update mechanisms become popular.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 19 Feb 2008 21:19:36 -0800

Brian Smith wrote:
> In this example, most of the elements of the Atom syntax (id, 
> title, author, link/@rel, link/@title, summary) are either 
> unused or misused. If the goal is to interoperate with other Atom 
> implementations, I would expect the entry to look something like this:
>
> <snip>

Actually, I would go farther and say that an Atom feed for events should
look something like CalAtom (http://robubu.com/CalAtom/CalAtom.html),
which embeds iCalendar into Atom. If custom markup is used for
everything, there is no way that users would be able to transfer these
events between an Atom-enabled calendaring application and the custom
application.

- Brian

---

From: Pablo Castro <[email-masked]>
Date: Tue, 19 Feb 2008 23:30:17 -0800

Hi Brian,

Thank you for your feedback.

Relations: the more we talk about this the more I agree that rel='related' doesn't really mean much. It seems that putting the name of the association in a form that fits the rel attribute is the most natural to do here. The one thing that would be interesting is to allow processors that know about this kind of links but don't know about a particular schema to identify them. May be a common prefix in the URI that acts as the value of 'rel' is what solves that.

Once we have something like that in place, if we allow developers to customize the names of the links (rels), they can align them with common vocabularies if they exist for a given domain.

Unused elements: I understand the concern about the unused Atom elements. We are thinking of adding more support to map elements of your data model that are a good match for Atom constructs to minimize that effect. That said, it would be good to explore is whether it is interesting to push Atom to somewhat new spaces where it makes a great format because of the fact that is broadly known, even thought for a particular case no all the elements of the Atom format are relevant (this is clearly not new, there are many folks that already explored this space).

Feeds instead of nesting: note that the <inline> element that contains the expanded data inside the link is in the Astoria namespace, so multiple implementations can clearly tell apart content that's part of their format and content that is not.

One unfortunate effect of having $expand result a feed is that it changes the nature of the entity that is sent back to the client. If you say /People('abc') you get an entry, wouldn't be at least weird that if you add $expand as in /People('abc')?$expand=Addresses then the response is now a feed?

Thanks
-pablo

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Smith
Sent: Tuesday, February 19, 2008 9:11 PM
To: 'atom-syntax'
Subject: RE: Related entries and feeds: links and link expansion

  <entry>

 <feed>
    ...
   <entry>
     ...

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 20 Feb 2008 05:14:50 -0800

Pablo Castro wrote:
> Once we have something like that in place, if we allow 
> developers to customize the names of the links (rels), they 
> can align them with common vocabularies if they exist for a 
> given domain.

<snip>

> Unused elements: I understand the concern about the unused 
> Atom elements. We are thinking of adding more support to map 
> elements of your data model that are a good match for Atom 
> constructs to minimize that effect. That said, it would be 
> good to explore is whether it is interesting to push Atom to 
> somewhat new spaces where it makes a great format because of 
> the fact that is broadly known, even thought for a particular 
> case no all the elements of the Atom format are relevant 
> (this is clearly not new, there are many folks that already 
> explored this space).

If you can easily allow your users to map their custom schemas into
(pseudo-)standard formats based on Atom, such that the resulting
documents can be usefully processed by tools that don't know anything
about the database schemas (e.g. importing the events into Outlook),
then that is a useful application of Atom. But, I don't see any
technical advantage of using Atom if you aren't using its core features.
What is the advantage of using Atom over plain-old-XML or SOAP? In
particular, what is the advantage of replacing <Venue>, <Event>, and
<Attendee> with <entry m:type='EventSample.Venue'>, <entry
m:type='EventSample.Event'>, and <entry m:type='EventSample.Attendee'>?
If you need to represent multiple entities in a single document, why not
use <soap:Envelope>?

> One unfortunate effect of having $expand result a feed is 
> that it changes the nature of the entity that is sent back to 
> the client. If you say /People('abc') you get an entry, 
> wouldn't be at least weird that if you add $expand as in 
> /People('abc')?$expand=Addresses then the response is now a feed?

It wouldn't be more weird than having entries and feeds nested within
atom:link elements. Besides, I don't know how I can tell that
"/People('abc')" is going to return one entry or multiple entries just
by looking at it. Plus, RESTful software will not attempt to extract
information out of the IRIs; instead, it will base its expectation on
the format of the result based on the context in which it saw the
hyperlink.

- Brian