Subject: Google Sitemaps: Yet another "RSS" or site-metadata format and Atom "competitor"

From: "Bob Wyman" <[email-masked]>
Date: Fri, 3 Jun 2005 17:28:35 -0400

Google has released "Sitemaps" which is yet-another-format for doing "site
summary" or site-metadata publication. Unlike commonly used RSS formats or
Atom, Sitemaps are only intended to communicate information about URLs that
have been updated - not the content of those URLs. However, it seems quite
clear to me that even though the goals of Sitemaps are a distinct subset of
those we have for Atom, there isn't really much reason why they needed to go
off and define a completely different and incompatible format.

Perhaps if this group had dealt with the issue of "pinging" this divergence
wouldn't have occurred. (Yes, I know it wasn't in-charter.) The Sitemap
format can be thought of as very much like a "ping file" since it basically
contains little more than info on the update time for urls. 

I've blogged about SiteMaps at:

http://bobwyman.pubsub.com/main/2005/06/google_sitemaps.html

There are two formats defined by Sitemaps:

1.       A Sitemap format which lists URLs and provides their last-update
time, priority, and update-frequency.

2.       A Sitemap index that points to other sitemaps. (One sitemap is
required for each unique path-tree on a site.)

Something similar to Sitemaps would actually be quite useful in helping us
work out the issues related to sites that have multiple feeds (i.e. three
flavors of RSS and one or two flavors of Atom.). One might provide a Sitemap
Index that listed each feed. This would make identifying duplicate feeds
much easier than it is today. A Sitemap might also be a really interesting
place to put information on a sites signing and encryption keys.

Perhaps we can convince Google to work with the Atom Working Group on
starting an effort to define more compatible formats and also to build a
pinging standard that would be useful to both the blogosphere and the
traditional web. We've proven in the blogosphere that providing
site-metadata and summary information is very useful and we've proven the
value of pinging. Now, it will be wonderful if we can help move those
methods to the rest of the web.

            bob wyman

---

From: "Arve Bersvendsen" <[email-masked]>
Date: Sat, 04 Jun 2005 00:29:34 +0200

On Fri, 03 Jun 2005 23:28:35 +0200, Bob Wyman <[email-masked]> wrote:

> starting an effort to define more compatible formats

Actually, Google Sitemaps is already compatible with the Google sitemap  
service. From  
<URL:https://www.google.com/webmasters/sitemaps/docs/en/faq.html#s8>:

# We also support the Open Archives Initiative (OAI) protocol for metadata
# harvesting, a popular protocol in the library world. If your sitemaps are
# already available in OAI-PMH version 2.0 format, you are welcome to  
submit
# these. We also accept RSS 2.0 and Atom 0.3 syndication feeds, using the
# link/lastMod fields.

-- 
Arve

---

From: "Arve Bersvendsen" <[email-masked]>
Date: Sat, 04 Jun 2005 00:43:34 +0200

On Sat, 04 Jun 2005 00:29:34 +0200, Arve Bersvendsen <[email-masked]>  
wrote:

> service.

err. Make that "compatible with Atom"

-- 
Arve

http://virtuelvis.com - http://userjs.org

---

From: "Bob Wyman" <[email-masked]>
Date: Fri, 3 Jun 2005 18:48:31 -0400

Arve Bersvendsen wrote:
> Actually, Google Sitemaps is already compatible with [Atom].
	Yes, I've had a number of folk send me mail pointing to the FAQ that
I did not read as closely as I should. In fact, it is great that Google is
willing to accept Atom 0.3 files instead of just their Sitemap format.
Hopefully, this move will encourage many traditional web sites to start
producing Atom files to serve the role of "Rich Site Summaries" and allow us
to expand our feed services to cover non-blogs as well. 
	I do still think it unfortunate that Google felt compelled to invent
yet-another-format for Sitemaps. Because of Google's support for the Sitemap
format, it is inevitable that aggregation services are going to have to
start reading Sitemaps in addition to the billions of flavors of legacy RSS
and Atom files. Nonetheless, given the importance of any format that Google
supports, should we be considering providing support in Atom for the
priority element or some easily mapped equivelant? Also, I wonder what it
will take to get Google so say that they'll support Atom 1.0? It seems
obvious that they would, but it would be nice to know when they expect to do
it...
	The Sitemap Index files are proper and useful additions to what we
have now in the blogosphere and we should look carefully at either trying to
figure out how to incorporate this idea into what we do for Atom. Perhaps
something along the lines of Sitemap indexes could be incorporated into the
autodiscovery specification?

	bob wyman

---

From: James M Snell <[email-masked]>
Date: Fri, 03 Jun 2005 16:11:00 -0700

>and Atom files. Nonetheless, given the importance of any format that Google
Unless aggregation services simply choose to ignore the new spec in 
favor of RSS & Atom.  I agree that there appears to be no justification 
for the new format based solely on a read of the spec.  Why else would 
they ALSO support Atom 0.3 and RSS?  If Atom 0.3 and RSS can be used 
adequately, why push yet another XML syntax that duplicates the basic 
form and function? 

>autodiscovery specification?

The sitemap indexes look very similar to Atom protocol collection 
documents with the exception of the changefreq and priority elements.  I 
see absolutely no reason why the following wouldn't work... (other than 
the fact that the Atom spec isn't finished yet.....)

<?xml version="1.0" encoding='utf-8'?>
<collection xmlns="http://purl.org/atom/app#"
   next="http://localhost:8080/roller/atom/myblog/entry/77088a1" >
   <member title="The Connells: Fun and Games"
         href="http://localhost:8080/roller/atom/myblog/entry/7700a0" 
         updated="2005-04-16T23:07:08-0400" 
         gsm:changefreq="daily"
         gsm:priority="0.8"/>
   <member title="The Connells: Boylan Heights"
         href="http://localhost:8080/roller/atom/myblog/entry/7700c9" 
         updated="2005-04-15T23:06:09-0400" 
         gsm:changefreq="weekly"
         gsm:priority="0.1"/>
   <member title="The Connells: Gladiator"
         href="http://localhost:8080/roller/atom/myblog/entry/7700c8" 
         updated="2005-04-14T14:05:31-0400" 
         gsm:changefreq="weekly"
         gsm:priority="0.2"/>
</collection>

---

From: Walter Underwood <[email-masked]>
Date: Fri, 03 Jun 2005 16:26:18 -0700

--On June 3, 2005 6:48:31 PM -0400 Bob Wyman <[email-masked]> wrote:
> yet-another-format for Sitemaps.

Yep. The could have used good ol' Infoseek sitelist.txt. Here is a copy
from eight years ago:

<http://web.archive.org/web/19970529104229/http://software.infoseek.com/products/ultraseek/docs/sitelist.html>

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Graham <[email-masked]>
Date: Sat, 4 Jun 2005 01:02:29 +0100

I don't see how a highly specialized format for a particular task is  
a competitor to or even compatible with what Atom does. There's  
nothing in our charter that says we've failed if it isn't possible to  
do everything conceivably related to website updates with Atom. And I  
don't see how Atom would benefit from becoming a jack of all trades.

Graham

---

From: Eric Scheid <[email-masked]>
Date: Sat, 04 Jun 2005 14:45:10 +1000

On 4/6/05 9:26 AM, "Walter Underwood" <[email-masked]> wrote:

> Yep. The could have used good ol' Infoseek sitelist.txt.

doesn't have revisit interval, doesn't have page-in-site priority, both
which seem important to google for scheduling purposes.

e.

---

From: Eric Scheid <[email-masked]>
Date: Sat, 04 Jun 2005 15:03:32 +1000

> don't see how Atom would benefit from becoming a jack of all trades.

Agree. Googles' sitemap format, Infoseek's sitelist.txt, and other similar
things only serve to provide a list of links to updated pages. They don't
serve to carry content or author metadata or such forth.

They are just a glorified "submit your url to be indexed" functionality for
search engines. Only you don't have to submit one URL at a time.

e.

---

From: "Bob Wyman" <[email-masked]>
Date: Sat, 4 Jun 2005 01:52:38 -0400

Graham wrote:
> a competitor to or even compatible with what Atom does.
	The "highly specialized task" which is performed using the Sitemap
format is providing lists of changed web pages on sites. This is precisely
the function that is performed in many applications of Atom. The only
difference between the target of Sitemap and Atom is that Sitemap works with
web sites that are not blogs and Atom is usually used with web sites that
are blogs. However, the differences between these two kinds of site are
virtually non-existent. 
	Atom doesn't need to be a "jack of all trades" to handle the job
that Sitemaps handle. It is already quite capable of doing the job.  And, as
James Snell points out in an earlier message, collection documents would
handle well the job of providing Sitemap indexes. 
	It seems quite clear that the Sitemap and Sitemap index formats have
little to offer that isn't already provided by Atom. This obviously leads to
the question of why Google went to the trouble of defining these formats. It
would be real nice if someone from Google could provide a touch of
explanation...

	bob wyman

---

From: Robert Sayre <[email-masked]>
Date: Sat, 4 Jun 2005 03:03:18 -0400

On 6/4/05, Bob Wyman <[email-masked]> wrote:
> the function that is performed in many applications of Atom. 

Yes, but the required fields are different. Other than than that, it's
pretty much the same dictionary or a-list situation we're used to.
Like rel=nofollow, this seems geared towards speeding up the crawl,
with a bunch of rhetoric surrounding it. Maybe I'm just cynical.

Robert Sayre

---

From: "Bob Wyman" <[email-masked]>
Date: Sat, 4 Jun 2005 13:17:31 -0400

Robert Sayre wrote:
> it's pretty much the same dictionary or a-list situation we're
> used to. Like rel=nofollow, this seems geared towards speeding up
> the crawl, with a bunch of rhetoric surrounding it. Maybe I'm just
> cynical.
	Why would you have to be cynical to suggest that support for
Sitemaps would speed up the crawl for Google? Wouldn't that be a good thing
for everyone involved? 
	* Google benefits by being able to do things more efficiently. 
	* Google users get timelier and more complete results. 
	* Publishers get their data "visible" more quickly than before.
	This seems like a win-win for everyone! There's no need for cynicism
to see the benefits. Heck, these are all the same benefits that have
motivated at least some of us to invest heavily in Atom and other
syndication formats.
	I think it would be really wonderful if we were to find that
non-blog web sites start producing this data. My only regret is that Google
has confused matters by publishing this new format to muddy the waters...
It's time that the rest of the web got the same kind of excellent service
that we've been providing to blogs.
	My hope is, however, that we'll be able to convince non-blog web
sites to use Atom instead of the Google proprietary format when publishing
this data. Using Atom would mean that a large number of existing systems
would be able to process their data rather than just Google. Also, by using
Atom, people will be able to go beyond merely listing URLs and will be able
to include summary data and other bits of metadata that other applications
can exploit. This could be a good thing for Atom -- if we can convince
people to use it instead of the Google proprietary format. If we don't
convince folk to use Atom, then we're going to find that Sitemaps provides a
"good enough" solution and it will be very hard to convince people to
convert to the more expressive Atom format in the future.

	bob wyman

---

From: James Tauber <[email-masked]>
Date: Sat, 4 Jun 2005 13:15:43 -0400

I've been serving my site (not the change log but the actual site) as  
an Atom feed for as long as my blog has had an Atom feed.

One nice (and unintentional) side effect of this is that I can  
subscribe to the site map in Bloglines and have Bloglines tell me  
when my non-blog pages are being referenced as well as my blog pages.

James

On 03/06/2005, at 8:02 PM, Graham wrote:

>
> I don't see how a highly specialized format for a particular task  
> is a competitor to or even compatible with what Atom does. There's  
> nothing in our charter that says we've failed if it isn't possible  
> to do everything conceivably related to website updates with Atom.  
> And I don't see how Atom would benefit from becoming a jack of all  
> trades.
>
> Graham
>
>

---

From: Henry Story <[email-masked]>
Date: Mon, 6 Jun 2005 10:02:03 +0200

I completely agree. At the core, a feed is just a list of state  
changes to web resources.
It will be  the perfect format for notifying search engines of all  
the changes to a web
site,  thereby massively reducing the time it will take them to crawl  
the web.

Henry Story

On 3 Jun 2005, at 23:28, Bob Wyman wrote:
> Google has released “Sitemaps” which is yet-another-format for  
> doing “site summary” or site-metadata publication. Unlike commonly  
> used RSS formats or Atom, Sitemaps are only intended to communicate  
> information about URLs that have been updated – not the content of  
> those URLs.

---

From: gstein <[email-masked]>
Date: Tue, 7 Jun 2005 03:17:04 -0700

On 6/4/05, Bob Wyman <[email-masked]> wrote:
>...

It was not published "to muddy the waters". That implies a specific
intent which was *definitely* not present.

>...

"proprietary" connotes closed. We published the spec and encourage
other search engines to use it. There is no intent to close or control
it.

How about this: you have a web site with 10 *million* URLs on it. What
format are you going to use? Is Atom appropriate at that scale?

[note: I'm not saying one is or isn't... to be honest, I'm not all
that familiar with the sitemaps schema and its features; but I do know
it was designed to scale, whereas I don't think Atom was]

Cheers,
-g

---

From: "Bob Wyman" <[email-masked]>
Date: Tue, 7 Jun 2005 09:18:11 -0400

> intent which was *definitely* not present.
	Please accept my apologies for what was poor writing. I can see how
you read my sentence as implying intent to muddy. It wasn't my intent,
however, to imply that. I should have written. "publishing this new format
*and* muddying the waters." My intent was to say that publishing the format
has the effect of muddying the waters. I wasn't trying to say that Google
was intentionally doing this.

> "proprietary" connotes closed.
	I'm using the older definition of proprietary which means simply
"not standard." I see nothing wrong with saying "open and proprietary"
format... I don't think one implies the exclusion of the other.

> How about this: you have a web site with 10 *million* URLs on it. 
> What format are you going to use? Is Atom appropriate at that scale?
	No. I don't think Atom would work well with 10 million URLs. At
least not as currently defined. I do think, however, that it would have been
useful to try to at least have a conversation about defining some subset of
Atom that would address the need. I think a result could have ended up
looking much like the Sitemap format but offered a smoother migration path
from Atom as we know it to the more terse format and the reverse.

	Please understand that I think that on-the-whole, the efforts by
Google to popularize the Sitemap process and "syndication" by non-blogs is
absolutely wonderful! I'm only grumbling about the formats... 

	bob wyman

---

From: Walter Underwood <[email-masked]>
Date: Tue, 07 Jun 2005 07:58:21 -0700

--On June 7, 2005 3:17:04 AM -0700 gstein <[email-masked]> wrote:
> other search engines to use it. There is no intent to close or control it.

"Proprietary" means "owned". Google clearly owns "Google Sitemaps".
The license requires derivative works to keep the same license. That is
control.

It was designed in isolation, for Google's use. That is a closed spec.

For example, the priority element is not specified well enough for another
engine to implement it compatibly. Does it apply to ranking, crawl order
or duplicate preference?

An open process would have at least looked at the proposed extensions 
for robots.txt and earlier formats like Infoseek sitelist.txt.

wunder

---

From: Danny Ayers <[email-masked]>
Date: Thu, 16 Jun 2005 13:45:19 +0200

Ach, late to the thread again.

When I saw Google Sitemaps I also thought of RDF Site Summary, and did
a sitemap2rss.xsl. But as noted already the role of RSS has mutated
from site summary to a content delivery format, so it wasn't a very
good fit. But it was straightforward to take their data and express it
as RDF in a more natural fashion, see:
http://dannyayers.com/archives/2005/06/03/google-sitemap/

I think Atom could have been a good format for this purpose without
the need for any other. I disagree with Walter, I believe the Sitemap
terms are defined well enough to be reused.

I've pasted that of priority below - it basically says where a
crawler's resources are limited, URLs with a higher priority on a site
should be crawled first/in preference to others on the site. That
could be a nice like Atom extension.

Cheers,
Danny.

-----

priority
Definition
Optional. The priority of a particular URL relative to other pages on
the same site. The value for this tag is a number between 0.0 and 1.0,
where 0.0 identifies the lowest priority page(s) on your site and 1.0
identifies the highest priority page(s) on your site.

The default priority of a page is 0.5.

Please note that the priority you assign to a page has no influence on
the position of your URLs in a search engine's result pages. Search
engines use this information when selecting between URLs on the same
site, so you can use this tag to increase the likelihood that your
more important pages are present in a search index.

Also, please note that assigning a high priority to all of the URLs on
your site will not help you. Since the priority is relative, it is
only used to select between URLs on your site; the priority of your
pages will not be compared to the priority of pages on other sites.

-- 

http://dannyayers.com