Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group item?

From: "Mark Watson" <[email-masked]>
Date: Fri, 9 Dec 2005 12:57:05 -0500

Rod,

Actually the requirement to be able to use source data from a previously
received version of the AO is, at its heart, very simple and quite
independent of the FEC Scheme. It does not even need to be described in
terms of source block or even symbols: We need to ensure only that the
receiver has the information about which parts (byte ranges) of the
previous version appear unchanged in the new version and where (byte
ranges) in the new version they appear.

>From this, the receiver can construct a partial copy of the new version
of the AO - it will have some parts missing which can be thought of as
'erasures'.

So far I have not mentioned source blocks or source symbols!

Whatever source blocking algorithm applies to the new AO can then be
applied to this partially received copy to determine which symbols of
the object are completely known. These can then be considered as
'received source symbols'.

Of course in this description there is the possibility that the receiver
has some partial symbols which it would have to discard. There may be
things we can do with source block construction etc. which may or may
not be FEC Scheme specific which could improve on this (but which are
sub-cases of the above, not divergences).

Of course anything FEC Scheme-specific needs to go in the FEC Schemes
and not in the aggregation draft but for the moment the aggregation
draft should at least include the basic requirement above.

Regards,

Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Michael Luby
Sent: Friday, December 09, 2005 3:02 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked]
Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group
item?

A couple of comments below.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, December 09, 2005 2:23 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]
Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group
item?

Sending a 99% identical large multipart object for a 1% is indeed a very
bad efficiency. Not that you wouldn't do it anyway if you were
delivering in a file carousel mode.

*** I didn't mean to suggest sending a 99% file for updating 1%, sorry
if I
gave that impression.  However, you are right that this can happen, if
you
are continually refreshing a set of files, but you would hope that you
would
set the refresh rate in such a way that you aren't that inefficient.

The topic is sufficiently separate to merit a separate I-D so maybe we
don't need any thorough description of it under the AO I-D's physical
aggregation text. So I think Mike's suggestion is the way to go.

*** I'm sorry that I missed that it had already been decided to roll the
incremental update feature into the main AO document.  Given that this
has
already been agreed, I think that is the way to go.  I'll leave it to
the
coauthors to figure out how to roll this feature in and then comment on
it
afterwards.

Cheers, Rod.

>-----Original Message-----
>From: ext Michael Luby [mailto:[email-masked]] 
>Sent: 08 December, 2005 17:44
>To: Walsh Rod (Nokia-NRC/Tampere); [email-masked]
>Cc: [email-masked]; [email-masked]; [email-masked]; 
>[email-masked]; [email-masked]
>Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a 
>working group item?
>
>Hi Rod,
>There are a bunch of issues with what you suggest, and perhaps 
>the most important is that this is a very inefficient way to 
>do incremental file updates.  What I suggest as the way 
>forward is for somebody (you can guess
>who) submit an individual submission that makes a proposal on 
>how to do incremental updates, and then we can discuss in the 
>working group and if there is acceptance this can be rolled 
>into a file aggregate working group item once that is 
>accepted.  I'm not sure it is useful to have email discussions 
>on this as it seems more efficient to have a concrete proposal 
>to discuss.
>Mike
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of [email-masked]
>Sent: Thursday, December 08, 2005 7:10 AM
>To: [email-masked]
>Cc: [email-masked]; [email-masked]; [email-masked]; 
>[email-masked]
>Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a 
>working group item?
>
>Hi Christoph, Mark, et al.
> 
>Just on the "incremental update" issue...
> 
>The obvious method is to ensure that some of the source blocks 
>are identical between AO x and AO x+1, and with the same 
>FEC-OTI it's trivial to use source blocks of AO x with AO x+1. 
>It's important to specify a single simple way for the client 
>to recognise this feature.
>Since the client gets the AO structure anyway as byte ranges 
>(it must do for partial recovery), then it can easily 
>calculate which blocks are the same for the same AO (i.e. AO 
>using the same fileURI). For different source blocks it just 
>receives symbols for the new one (AO x aren't of much use to 
>those source blocks in this scheme). The question is whether 
>to make this FEC code specific.
> 
>For some FEC codes (e.g. if you take the whole AO as one 
>massive source block), you'll need to rely on an understanding 
>of "where the encoded symbols came from" (e.g. the PCM).
> 
>Probably the easiest way to deal with this in the I-D is to 
>explain the simple way to do this for identical blocks, and 
>then explain that mapping redundant symbols back to source 
>symbols could be used for different blocks with some common 
>(file) data, but leaving the exact details of mapping back 
>something like a PCM out of scope. Then all angles are covered 
>without any need for rocket science.
> 
>Sound good?
> 
>Cheers, Rod.
> 
>Ps Since I'm not a FEC expert, please be kind about any 
>glaringly stupid suggestions on FEC :)
> 
>
>
>________________________________
>
>	From: [email-masked]
>[mailto:[email-masked]] On Behalf Of ext 
>Christoph Neumann
>	Sent: 06 December, 2005 00:32
>	To: Walsh Rod (Nokia-NRC/Tampere)
>	Cc: [email-masked]; [email-masked]; 
>[email-masked]; [email-masked]
>	Subject: Re: [Rmt] Should FLUTE File Aggregation I-D be 
>a working group item?
>	
>	
>	Hi all,
>	 
>	sorry for my late reply to all your comments. I think 
>quite interesting issues have been raised.
>	I will only talk about physical aggregation, since the 
>logical aggregation scheme seems quite obvious to everyone.
>	 
>	First, what is clear to all of us, is that the object 
>aggregation scheme should be more than just a gzip or tar of 
>several files.
>	 
>	Therefore, we need to clearly identify the advantages 
>we want and can provide. From the previous mails I identfied 
>the following
>features:
>	I. Partial reception: Extraction of files from 
>partially recovered AO
>	 
>	II. Improved transmission performances (eliminating 
>coupon collector + better FEC performances) 
>	 
>	III. Provide FDT information (MIME-type, URI etc.) for 
>each of the files that have been aggregated
>	 
>	IV. Backward compatibility:
>	    This feature can be interpreted in two ways:
>	      1) Receivers which do not support the aggregation 
>scheme, should be able to interpret at least the source 
>symbols sent for the files being aggregated.
>	      2) Receivers which do not support the aggregation 
>scheme, should be able to interpret all source and parity 
>symbols of the aggregated object. I.e the receivers can 
>reconstruct the entire aggregated object, and then decompose 
>it into the files being aggregated. 
>	 
>	V. Incremental update: It should be possible to reuse 
>previously received symbols of an old aggregated object or of 
>old files, when these files are reused in a new aggregated object.
>	    
>	VI. Single file transmission: It should be possible to send the
>files being aggregated individually.   
>	 
>	 
>	The "gzip approach", only supports features II. and IV.2).
>	 
>	The draft presents a solution that supports feature I. II. III.
>IV.2) and nothing more. 
>	The MIME format was chosen primarily with issue IV.2) in mind.
>As mentioned in your comments it is maybe not the best 
>solution and we may have a look at some alternatives.
>	 
>	Another proposal that came out of the discussions looks 
>interesting to me: sending each individual file on a separate 
>TOI and having an aggregated object that uses another TOI. 
>	The parity symbols produced for the aggregated object 
>are generated based on all source symbols of the files being 
>aggregated.
>This means that only parity symbols are send for the AO.
>	With that approach the FDT must list the TOIs of the 
>files being aggregated and their offsets within the aggregated 
>object (maybe some additional attributes will also be needed).
>	As mentioned by Mark, the FDT would be much simpler 
>compared to the current approach proposed in the draft. 
>However, we must clearly think about the mechanisms used by 
>such an approach.
>	 
>	This new approach should be able to cover the features I. II.
>III. IV.1) V. and VI. 
>	The only concern is that receivers which do not support 
>the aggregation scheme cannot use the parity symbols
>	of the aggregated object. However I think this is 
>acceptable, since the sender may produce and send parity 
>symbols for each individual file, if he suspects having 
>receivers that do not support the aggregation scheme. 
>	 
>	I think it is worth investigating this proposal, and I 
>am a volunteer for doing that. As pointed out by Mark, the 
>resulting draft will be rather different from the one 
>presently proposed. With regards to the comments in this 
>thread, I think that the new document should then become a 
>working group item. 
>	 
>	cheers,
>	 
>	Christoph
>	 
>	 
>	On 12/2/05, [email-masked] < [email-masked] 
><mailto:[email-masked]> > wrote: 
>
>		This is an excellent point. Basically we do not 
>want to specify
>		applications in RMT and vigilance is essential. 
>		
>		(I'll skip over the logical aggregation chatter 
>as the thread is focused
>		on physical aggregation).
>		
>		FLUTE already allows content encoding for 
>transport - great idea taken
>		from HTTP. FDTs provide the necessary 
>identification for files and the 
>		CENC header extension for FDT Instances. As 
>such it is possible to
>		"code" (e.g. zlib/gzip) a single file. For 
>doing this to a group of
>		files you have but 2 options:
>		- gzip each file and transport separately 
>		- gzip together in an application and reveal 
>only the .zip file & MIME
>		type to FLUTE (presumably with no content 
>encoding in this case)
>		
>		These 2 options are fine in their respective 
>use cases; and note I'm 
>		using zlib and gzip only to make the discussion clearer
>- there are
>		several other examples which may be relevant (tarball,
>...)
>		
>		The problem is the 2 options force you to lose 
>out on transport
>		functionality. You either forget FEC 
>performance (many small files) or 
>		you forget file delivery (i.e. FLUTE becomes a 
>container shipment
>		activity with no knowledge of files).
>		
>		If you need the atomic file deliver primary 
>function of FLUTE, as well
>		as the FEC performance of larger objects then 
>you'll need a 3rd option 
>		at the transport layer = the physical 
>aggregation technique.
>		
>		A really exciting part is that although 
>transport grouping binds files
>		together (in an all-or-nothing sense); with 
>partial recovery you would
>		not only be able to serve all-or-nothing applications.
>i.e. handling
>		this as transport really can improve transport 
>performance in ways that
>		"above the FEC" application functions would not.
>		
>		Cheers, Rod.
>		
>		PS if gzip provided the right data about files 
>in an external/exportable 
>		MIME type and allowed partial recovery, then 
>I'd go for simple adoption
>		of gzip, but it doesn't - so I think we can 
>either specify this simple
>		object aggregation, or we create gzip2 which 
>does have the features we 
>		need at FLUTE transport. I don't like the idea 
>of inventing gzip either,
>		and I'm not too keen on evangelising the 
>adoption of re-invented gzip2
>		:)
>		
>		
>		
>		>-----Original Message-----
>		>From: ext Joerg Ott [mailto: [email-masked]]
>		>Sent: 28 November, 2005 10:09
>		>To: Walsh Rod (Nokia-NRC/Tampere)
>		>Cc: [email-masked];
>[email-masked];
>		> [email-masked]
><mailto:[email-masked]> ; [email-masked]
>		>Subject: Re: [Rmt] Should FLUTE File 
>Aggregation I-D be a 
>		>working group item? 
>		>
>		>Hi Rod,
>		>
>		>I think fioe aggregation is basically a useful thing to
>		>consider as the motivation clearly points out. 
> In Vancouver,
>		>I raised the point about re-inventing gzip.  
>The following 
>		>statement in the motivation section makes me a bit
>nervous:
>		>
>		>    Another motivation for having an object aggregation
>scheme compared
>		>    to a basic archive based solution (e.g. tarball),
>is that no extra 
>		>    transformation (i.e. archive creation or
>extraction) is required at
>		>    either the FLUTE sender or receiver.  Everything is
>managed
>		>    automatically by the transport mechanism according
>to transport- 
>		>    specific optimizations and can be transparent to
>upper applications
>		>    (i.e. built on top of FLUTE), or enhanced by
>application hints on
>		>    file relationships, without breaking the basic
>semantics of FLUTE 
>		>    sessions.
>		>
>		>Yes, you can always push functionality down a 
>layer (which
>		>still means that you have to perform the 
>function but you just
>		>abstract from its details for the application 
>and limit its 
>		>amount of work and the flexibility in applying 
>this function).
>		> But you should carefully ask yourself how far 
>you want to
>		>take this.  Will we see semantically more complex
>		>relationships tomorrow?  Compression of data?  ...?
>How do 
>		>you deal with partially received objects?
>		>How complex does the API to your transport get?
>		>
>		>There is a certain number of things that only 
>the application
>		>can do right by itself.  I would like to see a clear 
>		>borderline drawn of what shall go into the 
>FLUTE transport and
>		>what should stay up at the application layer 
>and then we
>		>should make sure that the extensions we define 
>now are going
>		>to be orthogonal to other features we may want 
>to add to the 
>		>transport later and that they are easily usable by the
>		>application for what it may want to do.
>		>
>		>(Remember that we are not building "systems" 
>in the Transport
>		>Area and that we do not necessarily have to 
>build application 
>		>libraries either.  So, how much is needed for 
>interoperability
>		>in FLUTE?)
>		>
>		>Cheers,
>		>Joerg
>		>
>		>> Hi Mark et al.
>		>>
>		>> Thanks for the support
>		>>
>		>> <I'll take time to consider the AO semantic 
>suggestions - an
>		>email for
>		>> another day>
>		>>
>		>>
>		>>>Some other minor comments:
>		>>
>		>>
>		>>>- I don't think we need to worry about 
>backwards compatibility with 
>		>>>FLUTE receivers compliant to the 
>experimental version which did not
>		>>>have element extensibility in the FDT XML - 
>this was a
>		>mistake and the
>		>>>'experimental' phase is exactly supposed to 
>allow us to find and 
>		>>>correct such mistakes.
>		>>
>		>>
>		>> Now the update FLUTE is practically ready 
>(anticipate one last I-D
>		>> update), I agree.
>		>>
>		>> The AO I-D previously the use of non-TOI=0 
>FDTs to help  with Exp 
>		>> backwards compatibility. Non-TOI=0 FDTs are 
>allowed anyway, but we
>		>> need not explain that this is useful in handling the
>		>experimental FDT
>		>> extensibility issue.
>		>>
>		>> 
>		>>>- the draft includes some text attaching 
>semantics to TOI ordering -
>		>>>something we removed from FLUTE in favour of 
>FDT Instance IDs. We
>		>>>should stick with the FLUTE principle that 
>later version of 
>		>a file is
>		>>>signalled by using a different TOI in an FDT 
>with a larger FDT
>		>>>Instance ID - it doesn't matter whether the 
>TOI value is greater or
>		>>>less than the previous one. 
>		>>
>		>>
>		>> Note, for FLUTE there is no such semantic. 
>The client has no
>		>gauntees
>		>> about FDT Instance numbers, no garuntees 
>about TOI numbers.
>		>>
>		>> Generally, using TOI and 'FDT Instance 
>number' values for this has 
>		>> similar issues (assuming wrap-around is 
>handled smoothly,
>		>higher/later
>		>> values represent later versions, gaps in the 
>range of number are
>		>> feasible, FDT Instances may never be 
>received). TOI keeps semantics 
>		>> with the file declaration; FDT requires 
>inheritence from FDT
>		>Instance
>		>> to files described for that Instance.
>Implemantation-wise there's
>		>> little difference between the two: probably 
>the server will be 
>		>> incrementing FDT Instance numbers and not describing
>		>out-of-date TOIs
>		>> on new Instances; probably the server will 
>use higher TOI values for
>		>> later versions; the client must store TOI 
>with fileURI. There is no 
>		>> requierment that the client stores FDT 
>Instance numbers.
>		>>
>		>> Quick FLUTE quote:
>		>>
>		>>    A receiver of the file delivery session keeps an
>FDT database for
>		>>    received file description entries.  The receiver
>maintains the 
>		>>    database, for example, upon reception of FDT
>Instances.
>		>Thus, at any
>		>>    given time the contents of the FDT database
>represent the
>		>receiver's
>		>>    current view of the FDT of the file delivery
>session.  Since each 
>		>>    receiver behaves independently of other receivers,
>it
>		>SHOULD NOT be
>		>>    assumed that the contents of the FDT database are
>the same for all
>		>>    the receivers of a given file delivery session. 
>		>>
>		>>    Since FDT database is an abstract concept, the
>structure and the
>		>>    maintaining of the FDT database are left to
>individual
>		>>    implementations and are thus out of scope of this
>specification. 
>		>> ...
>		>>    *  The receiver SHOULD NOT use a received FDT
>Instance to
>		>interpret
>		>>       packets received beyond the expiration time of
>the FDT
>		>Instance.
>		>>
>		>> i.e. the expiry time of file description 
>should be kept with other
>		>> file data, but FDT Instance ID is up to the 
>implementation.
>		>>
>		>> Forcing clients to additional store this 
>value is not much trouble - 
>		>> but it is _not_ "the FLUTE principle that 
>later version of a file is
>		>> signaled by using a different TOI in an FDT 
>with a larger
>		>FDT Instance
>		>> ID". Which is exactly why the AO draft 
>offers the simplest 
>		>solution to
>		>> this issue. If we make the assumptions of 
>"well-behaved
>		>simpple FLUTE
>		>> servers" above, there really is nothing 
>between FDT Instance ID and
>		>> TOI to be used for this purpose. The reason 
>I give all this 
>		>background
>		>> is that this doesn't hold for some obvious 
>AO usage...
>		>>
>		>> For a physically aggregated object (e.g. 
>very large) with some small
>		>> updates (e.g. 2 file versions are new from 
>10,000 files). The server 
>		>> will use TOI (i) for the initial full object files
>		>descriptions. It is
>		>> highly likely that the big aggregated object 
>will continue to be
>		>> transmitted for some time and with only 
>minimal updates it 
>		>is probably
>		>> more bandwidth efficient to send the symbols 
>for the original object
>		>> and the few (e.g. 2) new file versions 
>seperates (more efficient due
>		>> to, e.g. fountain, symbols of earlier 
>reception still being 
>		>useful to
>		>> all receivers regarless of complete, partial 
>or not-started
>		>receiption
>		>> status). So the server sends FDT (ii) with 
>the new TOIs for
>		>the files
>		>> which have been updated. <At this point 
>using either TOI or FDT 
>		>> Instance ID would work>. However, some objects of FDT
>		>Insatnce (i) are
>		>> now changed/removed so that the server need 
>to describe the slighlty
>		>> old large aggregated object on TOI (iii). 
>All recievers 
>		>throw away the
>		>> new files (even though the TOI is higher) as 
>the old files are
>		>> described in a later FDT Instance.
>		>>
>		>> FLUTE is specifically designed to allow this 
>ebhaviour and make no 
>		>> assumption the receivers FDT/file database 
>state and no
>		>assumption on
>		>> reception of individual FDT Instances.
>		>>
>		>> Using TOI for this is simple and works. FDT 
>Instance ID
>		>fails. Hence,
>		>> the mention of TOI for this purpose in the I-D.
>(Originally it was
>		>> start with TOI=1 and increment by 1, but 
>that's overkill as just
>		>> 'higher value' is needed for subseqent 
>versions of a file (fileURI) 
>		>> instance)
>		>>
>		>> (I've been trying to explain this for a 
>while - but it's not
>		>important
>		>> in 3GPP rel6, and I'm not up-to-date with 
>DVB or OMA to see if
>		>> something's fractured there - so maybe this 
>is the first time I've 
>		>> explained by mail.)
>		>>
>		>>
>		>>>- the draft mentions that one might only 
>consider file aggregation
>		>>>when the aggregate is less than a single source block
>- I
>		>think it is 
>		>>>simpler and has no disadvantages if you 
>decouple the source blocking
>		>>>(which is a feature of the FEC scheme 
>chosen) from the file
>		>>>aggregation. If files belong together in an 
>aggregate then 
>		>they should
>		>>>be put together and then if the FEC code 
>needs to split that
>		>data into
>		>>>blocks this is ok and does not result in any worse
>		>performance so long
>		>>>as it correctly interleaves the packets from 
>the different blocks. 
>		>>
>		>>
>		>> Exactly.
>		>>
>		>> The I-D does not mean to bind AO and FEC blocking.
>The
>		>discussion is for
>		>> implementation guidence: for files much 
>greater than the block size 
>		>> there's no FEC benefit in physical 
>aggregation; for the
>		>composite files
>		>> being less than the block size it always 
>makes sense to aggregate for
>		>> robustness; inbetween it's a compromise; the 
>I-D absolutely does not 
>		>> impose any FEC instance retraints and so 
>it's implementation
>		>specific to
>		>> optomise complexity and performance in this aspect.
>The guidence is
>		>> valuable. If we inadvertantly made normative 
>specification 
>		>on this, it
>		>> was an error to be corrected.
>		>>
>		>> Also, the logical grouping between objects 
>can be used to get the
>		>> "belongs together" semantics for multiple AOs - e.g.
>where partial
>		>> recovery is undesirable, or different levels 
>of robustness are
>		>> desirable).
>		>>
>		>> Cheers, Rod.
>		>>
>		>>
>		>>
>		>>
>		>>
>		>> 
>		>>
>		>>
>		>>
>		>>
>		>> _______________________________________________
>		>> Rmt mailing list
>		>> [email-masked]
>		>> https://www1.ietf.org/mailman/listinfo/rmt
>		>>
>		>
>		>
>		
>		_______________________________________________
>		Rmt mailing list
>		[email-masked]
>		https://www1.ietf.org/mailman/listinfo/rmt
>		
>		
>		
>
>
>
>_______________________________________________
>Rmt mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rmt
>
>

_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

---

From: [email-masked]
Date: Tue, 13 Dec 2005 16:22:42 +0200

Hi Mark

Thanks for the comments.

As you say, it can even work if the blocking of old and new AOs are
different. This would make a nice simulation for a journal paper too:
"variation in blocking and the effect on partial recovery of aggregated
objects using recovery codes".

I gather your proposal is to include some text like I mentioned, but
maybe make it simpler, and also mention that the same (or indeed any)
source blocking need not be assumed for partial recovery to work?

I support your proposal "anything FEC Scheme-specific needs to go in the
FEC Schemes and not in the aggregation draft but the aggregation draft
should at least include the basic requirement". (Note, "for the moment"
is deleted).

I think this requirement is already fulfilled since the structure (byte
ranges) of the AOs are already available to the client. However, it
would be appropriate to improve the implementer guidelines on this so
more implementations chose to include the feature.

Cheers, Rod.

>From: ext Mark Watson [mailto:[email-masked]] 
>Sent: 09 December, 2005 19:57
>Cc: [email-masked]
>Actually the requirement to be able to use source data from a 
>previously received version of the AO is, at its heart, very 
>simple and quite independent of the FEC Scheme. It does not 
>even need to be described in terms of source block or even 
>symbols: We need to ensure only that the receiver has the 
>information about which parts (byte ranges) of the previous 
>version appear unchanged in the new version and where (byte
>From this, the receiver can construct a partial copy of the 
>new version of the AO - it will have some parts missing which 
>can be thought of as 'erasures'.
>
>Whatever source blocking algorithm applies to the new AO can 
>then be applied to this partially received copy to determine 
>which symbols of the object are completely known. These can 
>then be considered as 'received source symbols'.
>
>Of course in this description there is the possibility that 
>the receiver has some partial symbols which it would have to 
>discard. There may be things we can do with source block 
>construction etc. which may or may not be FEC Scheme specific 
>which could improve on this (but which are sub-cases of the 
>above, not divergences).
>
>Of course anything FEC Scheme-specific needs to go in the FEC 
>Schemes and not in the aggregation draft but for the moment 
>the aggregation draft should at least include the basic 
>requirement above.
>
>Regards,
>
>Mark
>
>Behalf Of Michael Luby
>Sending a 99% identical large multipart object for a 1% is 
>indeed a very bad efficiency. Not that you wouldn't do it 
>anyway if you were delivering in a file carousel mode.
>
>*** I didn't mean to suggest sending a 99% file for updating 
>1%, sorry if I gave that impression.  However, you are right 
>that this can happen, if you are continually refreshing a set 
>of files, but you would hope that you would set the refresh 
>rate in such a way that you aren't that inefficient.
>
>The topic is sufficiently separate to merit a separate I-D so 
>maybe we don't need any thorough description of it under the 
>AO I-D's physical aggregation text. So I think Mike's 
>suggestion is the way to go.
>
>*** I'm sorry that I missed that it had already been decided 
>to roll the incremental update feature into the main AO 
>document.  Given that this has already been agreed, I think 
>that is the way to go.  I'll leave it to the coauthors to 
>figure out how to roll this feature in and then comment on it 
>afterwards.
>
>Cheers, Rod.
>
>
>working group 
>>There are a bunch of issues with what you suggest, and 
>perhaps the most 
>>important is that this is a very inefficient way to do 
>incremental file 
>>updates.  What I suggest as the way forward is for somebody (you can 
>>guess
>how to do 
>>incremental updates, and then we can discuss in the working group and 
>>if there is acceptance this can be rolled into a file 
>aggregate working 
>>group item once that is accepted.  I'm not sure it is useful to have 
>>email discussions on this as it seems more efficient to have 
>a concrete 
>>proposal to discuss.
>>Mike
>>
>>[email-masked]
>working group 
>>The obvious method is to ensure that some of the source blocks are 
>>identical between AO x and AO x+1, and with the same FEC-OTI it's 
>>trivial to use source blocks of AO x with AO x+1.
>>It's important to specify a single simple way for the client to 
>>recognise this feature.
>>Since the client gets the AO structure anyway as byte ranges (it must 
>>do for partial recovery), then it can easily calculate which 
>blocks are 
>>the same for the same AO (i.e. AO using the same fileURI). For 
>>different source blocks it just receives symbols for the new 
>one (AO x 
>>aren't of much use to those source blocks in this scheme). 
>The question 
>>is whether to make this FEC code specific.
>> 
>massive source 
>>block), you'll need to rely on an understanding of "where the encoded 
>>symbols came from" (e.g. the PCM).
>> 
>explain the 
>>simple way to do this for identical blocks, and then explain that 
>>mapping redundant symbols back to source symbols could be used for 
>>different blocks with some common
>>(file) data, but leaving the exact details of mapping back something 
>>like a PCM out of scope. Then all angles are covered without any need 
>>for rocket science.
>> 
>>Sound good?
>> 
>>Cheers, Rod.
>> 
>>Ps Since I'm not a FEC expert, please be kind about any glaringly 
>>stupid suggestions on FEC :)
>> 
>>
>>
>>[mailto:[email-masked]] On Behalf Of ext Christoph 
>>Neumann
>a working 
>>group item?
>>	sorry for my late reply to all your comments. I think quite 
>>interesting issues have been raised.
>>	I will only talk about physical aggregation, since the logical 
>>aggregation scheme seems quite obvious to everyone.
>>	 
>aggregation 
>>scheme should be more than just a gzip or tar of several files.
>>	 
>we want and can 
>>provide. From the previous mails I identfied the following
>>features:
>scheme, should 
>>be able to interpret at least the source symbols sent for the files 
>>being aggregated.
>scheme, should 
>>be able to interpret all source and parity symbols of the aggregated 
>>object. I.e the receivers can reconstruct the entire 
>aggregated object, 
>>and then decompose it into the files being aggregated.
>>	 
>previously 
>>received symbols of an old aggregated object or of old files, when 
>>these files are reused in a new aggregated object.
>>	    
>interesting to 
>>me: sending each individual file on a separate TOI and having an 
>>aggregated object that uses another TOI.
>are generated 
>>based on all source symbols of the files being aggregated.
>files being 
>>aggregated and their offsets within the aggregated object (maybe some 
>>additional attributes will also be needed).
>>However, we must clearly think about the mechanisms used by such an 
>>approach.
>>	 
>>	The only concern is that receivers which do not support the 
>>aggregation scheme cannot use the parity symbols
>acceptable, since 
>>the sender may produce and send parity symbols for each individual 
>>file, if he suspects having receivers that do not support the 
>>aggregation scheme.
>>	 
>am a volunteer 
>>for doing that. As pointed out by Mark, the resulting draft will be 
>>rather different from the one presently proposed. With regards to the 
>>comments in this thread, I think that the new document should then 
>>become a working group item.
>>	 
>>	cheers,
>>	 
>>	Christoph
>>	 
>>	 
>as the thread is 
>>focused
>only the .zip file & 
>>MIME
>you'll need a 3rd 
>>option
>performance in ways 
>>that
>>		PS if gzip provided the right data about files in an 
>>external/exportable
>does have the features 
>>we
>of inventing gzip 
>>either,
>TOI value is greater 
>>or
>use higher TOI values 
>>for
>with fileURI. There is 
>>no
>10,000 files). The 
>>server
>for the original 
>>object
>to describe the 
>>slighlty
>I-D absolutely does 
>>not