Subject: [siprec] Metadata XML document error representation for whole doc or for portion of doc

From: "Parthasarathi R (partr)" <[email-masked]>
Date: Fri, 25 Feb 2011 22:14:45 +0530

Hi all,
 
I'm interested in hearing whether it makes more appropriate to create the (syntax or semantic) error for the whole Metadata XML document or for section/portion of XML document. Each approach has its own pros and cons. 
 
Whole Metadata XML document error handling::
           Any syntax of semantic error is seen in the XML document, the whole document will be rejected by SIP response code (like 415/488).  
Pros: 
          1) Here SRC has no need to decode XML document while receiving the error response
          2) Simple Implementation as the operation is success or failure only
Cons:
          1) Even most of the XML document is acceptable, the document is rejected in SRS 
           2) It is not possible to identify the exact error 
 
Error handling for Portion of XML document:: 
Any syntax or semantic error has to be represented with the specific XML element has the error. The portion of the xml element will be identified by the id corresponding to the element. The error has to be passed from SRS to SRC in the XML format with special XML syntax. say participant AOR is not accepted then XML error sends back to indicate that partipant element is not accepted.
 
Open item:
1) What SIP response code to be sent when the portion of the XML is accepted   
 
Pros:
         1) The portion of the element will only be rejected and rest of the XML is accepted in SRS.
          2) Provide more granular control
Cons:
         1) SRC *MUST* be XML decoding functionality with response handling to identify the error
         2) Complex design in nature for metadata itself
 
Please provide your valuable input about which approach will be most suitable from SIPREC perspective.
 
Thanks
Partha

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 25 Feb 2011 12:52:41 -0500

On 2/25/2011 11:44 AM, Parthasarathi R (partr) wrote:
> Hi all,
> I'm interested in hearing whether it makes more appropriate to create
> the (syntax or semantic) error for the whole Metadata XML document or
> for section/portion of XML document. Each approach has its own pros and
> cons.

General comments on this subject:

I don't think we should expect that the transmitted metadata information 
will be processed prior to responding to the message that conveys it. 
For one thing it is a layer violation. For another, the processing of 
this data could be *harder* than processing the sip message, and so need 
to be offloaded and queued for processing.

So I don't think any error report about the metadata should be 
*expected* in the response to the message carrying it. (We may still 
*permit* an error report to be conveyed in that response, but it should 
also be possible to send it later.)

IMO its not important to give fine-grained error information. It almost 
certainly won't result in the sender "fixing" what it is doing. Its 
mostly just information that can be used for reporting/fixing errors. So 
a more general "couldn't handle this", plus perhaps some free form field 
for conveying info potentially useful to someone fixing things, should 
be sufficient.

Two possible variations on that:

If we plan for options and extensions in the mechanism, and define some 
rules for how those are handed, we may want a way to say "I didn't 
understand x and skipped it". Right now I only see that for "extension 
data", and I'm not certain we want to be chatty about that.

Current thinking (not yet presented to the group) is that the messages 
conveying metadata will consist of a sequence of individual packets of 
metadata to be added to the collected metadata for the RS. There could 
be a bunch of these in the message, and they will need to be processed 
in sequence. If there is trouble along the way, it probably compromises 
the ability to process subsequent pieces.

We might want to treat the entire message as atomic, with all the stuff 
in it being processed, or none of them.

OR, we might want to say these are processed one by one until there is a 
problem, and then no more are processed. (Until when?)

This has implications on the protocol for conveying these things, 
especially since the "responses" will probably be delayed and the SRC 
will not want for a response before sending more.

My *inclination* right now is to perhaps sequence number the individual 
metadata update packets, and have a response ack/nack up to a particular 
sequence number. If there is an error, then no more would be processed 
until some handshake occurred. That would allow the updates to be 
pipelined without regard to message boundaries.

We also need a philosophy here. In general I think we must assume that 
the SRC and SRS are implemented correctly. Once that has been proven 
wrong, by detection of an error, there is a real question of whether to 
attempt to continue the RS, or simply give it up. Just giving up and 
dropping it has the merit of simplicity. OTOH, I know that often 
recordings are essential, and so it may be better to keep going in 
crippled form with as much as you can figure out. I don't have an answer 
here yet.

	Thanks,
	Paul

> Any syntax of semantic error is seen in the XML document, the whole
> document will be rejected by SIP response code (like 415/488).
> Pros:
> Any syntax or semantic error has to be represented with the specific XML
> element has the error. The portion of the xml element will be identified
> by the id corresponding to the element. The error has to be passed from
> SRS to SRC in the XML format with special XML syntax. say participant
> AOR is not accepted then XML error sends back to indicate that partipant
> element is not accepted.
> Open item:
> _______________________________________________
> siprec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/siprec

---

From: "Parthasarathi R (partr)" <[email-masked]>
Date: Sat, 26 Feb 2011 01:05:12 +0530

Paul,
 
Thanks for your detailed mail. IIRC, you propose to represent the error for the whole XML document instead of individual specific element in XML error. I agree with you that extension data error has to be part of the metadata XML error handling. Let us see others opinion in this approach.
 
You have brought back discussion during last interim meeting whether error handling in SIPREC solution draft has to done immediately in the response of SIP transaction or later. I'll span the separate thread for discussing as I'm not fully convinced with sending error message later after sending 200 OK for the request. 
 
Thanks
Partha

________________________________

From: [email-masked] on behalf of Paul Kyzivat (pkyzivat)
Sent: Fri 2/25/2011 11:22 PM
To: [email-masked]
Subject: Re: [siprec] Metadata XML document error representation for whole doc or for portion of doc

        Thanks,
        Paul

---

From: "Parthasarathi R (partr)" <[email-masked]>
Date: Sat, 26 Feb 2011 01:05:24 +0530

Hi all, 
 
The separate question brought by Paul in this below mail thread and others during last interim meeting is whether Metadata XML body will be processed before sending the response to SRC or not. I believe that each transaction in SIP message will be completed when 200 OK is send for the request. For example: INVITE carries metadata XML document from SRC to SRS, 200 OK from SRS indicates that INVITE with message body (metadata XML) is accepted. In case of any error, it *MUST* be conveyed in the form of non-2xx response. Still I could not get the point that  200 OK will be sent but message body processing error will be conveyed through new SIP mechanism. XML usage as message body (like PIDF) is prevalent in SIP and SIPREC is not exception to this, Could you please point out any existing mechanism which sends 200 OK immediately without processing message body and sending the error later through other SIP mechanism. 
 
The general concern is whether it is possible to process complete SIPREC XML body before sending the response to SRC. INVITE with provisional response like 100 will provide ample time for SRS to process the message body and convey the appropriate response code (2xx or non-2xx). The question may arise how SRC will queue forthcoming transaction till the first transaction is complete. IMO, the real question is whether SRC or SRS has to queue the messages as SRS is not processed the first message itself.  It is preferred to queue in SRC rather than sending to SRS which could not digest the earlier messages itself. As Paul indicated in the below mail, there will be troublesome situation in queuing the message in SRS without complete processing in case the first message has error how to deal with the rest of the queued message in SRS. It will be better to queue the rest of the message in SRC queue till the processing completing in SRS. 
 
Please read inline
 
Thanks
Partha

I don't think we should expect that the transmitted metadata information
will be processed prior to responding to the message that conveys it.
For one thing it is a layer violation.

<Partha>  Could you please explain in detail about for the layer violation here </Partha>

For another, the processing of
this data could be *harder* than processing the sip message, and so need
to be offloaded and queued for processing.

        Thanks,
        Paul

---

From: "Henry Lum" <[email-masked]>
Date: Fri, 25 Feb 2011 13:35:48 -0800

Hi Paul,

Leon and I had a similar discussion about the different philosophies of
handling errors. If we opt for a chatty protocol to ack/nack each
message before sending the next one, then what should SRC do when the
SRS believes there is an error with a message? We are not sure that the
SRC can actually do anything constructive at this point to recover from
the error other than reporting the presence of an error. It is possible
that the SRS can give up on the RS, but I think SRS should just continue
to record the media despite perceived errors in metadata; this is
similar to a browser getting badly formatted HTML but will still try to
render as much as possible. We are also thinking of providing the SRS
with a mechanism to request the SRC to ask for the current snapshot so
that the SRC/SRS can reset the states in the case where SRS is in a
really bad state. By doing so, we may be able to avoid providing nack
responses and avoid using a chatty protocol.

I agree sequencing of messages make sense since SRC should be sending
only updates to metadata rather than sending the entire metadata view
for every message.

Regards,
Henry

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Paul Kyzivat
> Sent: Friday, February 25, 2011 12:53 PM
> To: [email-masked]
> > 1) Here SRC has no need to decode XML document while receiving the
> error
> > response
> > 1) Even most of the XML document is acceptable, the document is
> rejected
> > in SRS
> > 1) The portion of the element will only be rejected and rest of the
> XML
> > is accepted in SRS.
> > 1) SRC *MUST* be XML decoding functionality with response handling
to
> > identify the error

-------------------------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail and any files attached may contain confidential and proprietary information of Alcatel-Lucent and/or its affiliated entities. Access by the intended recipient only is authorized. Any liability arising from any party acting, or refraining from acting, on any information contained in this e-mail is hereby excluded. If you are not the intended recipient, please notify the sender immediately, destroy the original transmission and its attachments and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Copyright in this e-mail and any attachments belongs to Alcatel-Lucent and/or its affiliated entities.

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 25 Feb 2011 17:51:31 -0500

On 2/25/2011 2:35 PM, Parthasarathi R (partr) wrote:
> Paul,
> Thanks for your detailed mail. IIRC, you propose to represent the error
> for the whole XML document instead of individual specific element in XML
> error. I agree with you that extension data error has to be part of the
> metadata XML error handling. Let us see others opinion in this approach.

First lets get some terminology straight.

We will have some body part in a sip message, containing an xml 
"document". Over time there will be a series of these within the RS.

I think each such xml document will consist of a sequence of "additions" 
(for lack of a better word) to the metadata being accumulated at the SRS.

One option is to report an error for that whole "document", indicating 
that it is in error and that none of it was processed. However this 
probably means the SRC must await a confirmation before sending further 
metadata. That doesn't seem very good.

Another option is to sequence number the individual "additions" not only 
within a single document, but across the entire RS. (Think of these as 
analogous to CSeq values.) And then to report status based on these 
numbers. My thought is to report something like "error found with 
addition #nnn". And then, since these are generally interdependent, to 
ignore any further additions until such time as the SRC and SRS can 
resync. Or to just close out the RS entirely after sending the error.

This allows pipelining of the data, without awaiting confirmation that 
the prior data was processed.

> You have brought back discussion during last interim meeting whether
> error handling in SIPREC solution draft has to done immediately in the
> response of SIP transaction or later. I'll span the separate thread for
> discussing as I'm not fully convinced with sending error message later
> after sending 200 OK for the request.

OK. Will wait and reply to that.

	Thanks,
	Paul

> Thanks
> Partha
>
> ------------------------------------------------------------------------
> *From:* [email-masked] on behalf of Paul Kyzivat (pkyzivat)
> *Sent:* Fri 2/25/2011 11:22 PM
> *To:* [email-masked]
> *Subject:* Re: [siprec] Metadata XML document error representation for
> whole doc or for portion of doc
>
>

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 25 Feb 2011 18:04:56 -0500

Partha,

On 2/25/2011 2:35 PM, Parthasarathi R (partr) wrote:
> Hi all,
> The separate question brought by Paul in this below mail thread and
> others during last interim meeting is whether Metadata XML body will be
> processed before sending the response to SRC or not. I believe that each
> transaction in SIP message will be completed when 200 OK is send for the
> request. For example: INVITE carries metadata XML document from SRC to
> SRS, 200 OK from SRS indicates that INVITE with message body (metadata
> XML) is accepted. In case of any error, it *MUST* be conveyed in the
> form of non-2xx response. Still I could not get the point that 200 OK
> will be sent but message body processing error will be conveyed through
> new SIP mechanism. XML usage as message body (like PIDF) is prevalent in
> SIP and SIPREC is not exception to this, Could you please point out any
> existing mechanism which sends 200 OK immediately without processing
> message body and sending the error laterthrough other SIP mechanism.

This has come up in numerous contexts.

In discussion of INFO packages, it was spelled out that if an info 
package requires a "response", then that response needs to be in another 
INFO in the other direction, not in the response to INFO.

In the case of event packages, information is unidirectional.

In the case of MESSAGE, there is no guarantee that the message has been 
displayed to the user or otherwise processed before the response is sent.

In the case of location-conveyance there has been some discussion of 
whether the location must be validated before responding and the answer 
was "no".

In the siprec case, the "recorder" is likely to be an "application" 
built on top of a sip stack. The ways these things are built vary. There 
is no guarantee that the application will have the opportunity to 
process the the body before the stack replies. They may run in differing 
environments with some loose coupling. And its quite likely that the 
recorder will need to do some database accesses in the process of 
validating and processing the metadata. (E.g. to verify that the IDs of 
references in the metadata have already been defined.) It is entirely 
unreasonable to expect that such db access will go on while the sip 
stack is waiting to send its response.

*If* a particular implementation can do its processing before the sip 
response is sent, it may make sense to permit it to send an "ack" about 
it in the response. But we should not require that.

	Thanks,
	Paul

> The general concern is whether it is possible toprocess complete SIPREC
> XML body before sending the responseto SRC. INVITE with provisional
> response like 100 will provide ample time for SRS to process the message
> body and convey the appropriate response code (2xx or non-2xx). The
> question may arise how SRC will queue forthcoming transaction till the
> first transaction is complete. IMO, the real question is whether SRC or
> SRS has to queue the messages as SRS is not processed the first message
> itself. It is preferred to queue in SRC rather than sending to SRS which
> could not digest the earlier messages itself. As Paul indicated in the
> below mail, there will be troublesome situation in queuing the message
> in SRS without complete processing in case the first message has error
> how to deal with the rest of the queued message in SRS. It will be
> better to queue the rest of the message in SRC queue till the processing
> completing in SRS.
> Please read inline
> Thanks
> Partha
>

---

From: "Parthasarathi R (partr)" <[email-masked]>
Date: Sat, 26 Feb 2011 22:06:10 +0530

Paul,
 
Please read inline.
 
Thanks
Partha

From: Paul Kyzivat (pkyzivat)
Sent: Sat 2/26/2011 4:21 AM
To: Parthasarathi R (partr)
Cc: [email-masked]
Subject: Re: [siprec] Metadata XML document error representation for whole doc or for portion of doc

<Partha> The other part of message body will have SDP. </Partha>

<Partha>  Most of the SIPREC transaction will have stream (SDP) involved as  we are recording the media. SRS is UAS which processes SDP, negotiate stream parameters in the response.  In  case SDP and XML body exist in the same message body(metadata), SDP will be processed before sending the response but whether XML processing will be deferred as it will take more time to process? Also, SRC MUST wait for the response in case SDP message body is sent towards SRS as per offer/answer model. So awaiting for the response before forwarding further metdata is not so bad. Still I'm not very clear about pipeline of SDP + XML (metadata) in SRS.  </Partha>

        Thanks,
        Paul

---

From: "Elwell, John" <[email-masked]>
Date: Mon, 28 Feb 2011 15:24:38 +0100

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Henry Lum
> Sent: 25 February 2011 21:36
> To: Paul Kyzivat; [email-masked]
> Subject: Re: [siprec] Metadata XML document error 
> representation for whole doc or for portion of doc
> 
> Hi Paul,
> 
> render as much as possible.
[JRE] I can perhaps draw an analogy with RFC 3265 event packages. My understanding is that the 200 OK response to NOTIFY requests doesn't mean much more than "I received this NOTIFY request in the context of a valid subscription and I support the MIME type(s) attached". There is no provision for conveying error information beyond the basic SIP response codes for rejecting the request. A specific event package might require more than this, although I don't think that is common practice and I don't think it was the intent of RFC 3265. Moreover, RFC 3265 and event packages do not provide any other mechanism, apart from the NOTIFY response, for reporting issues in notified data.

I think metadata transmissions and updates can be treated the same way, that a 200 response does not mean metadata was correct - only that it was received and passed to the application. If we decide that the SRS application needs to be able to report errors to the SRC, that would need to be a separate mechanism.

> We are also thinking of providing the SRS
[JRE] Yes, and the request to resend full metadata could optionally include some sort of error report that gave rise to the request. That error report could, for example, include the sequence number of the metadata update that was found to be in error.

John (as individual)

> > > for section/portion of XML document. Each approach has 
> its own pros
> > and
> > > cons.
> > 
> > mostly just information that can be used for 
> reporting/fixing errors.
> > So
> > OR, we might want to say these are processed one by one 
> until there is
> > a
> > > Any syntax or semantic error has to be represented with 
> the specific
> > XML
> > > 1) What SIP response code to be sent when the portion of 
> the XML is
> > > accepted
> > > Pros:
> > > Please provide your valuable input about which approach 
> will be most
> > > suitable from SIPREC perspective.
> > > Thanks
> > > Partha
> > >
> > >
> > >
> --------------------------------------------------------------
> -----------------------------------------------------
> CONFIDENTIALITY NOTICE: This e-mail and any files attached 
> may contain confidential and proprietary information of 
> Alcatel-Lucent and/or its affiliated entities. Access by the 
> intended recipient only is authorized. Any liability arising 
> from any party acting, or refraining from acting, on any 
> information contained in this e-mail is hereby excluded. If 
> you are not the intended recipient, please notify the sender 
> immediately, destroy the original transmission and its 
> attachments and do not disclose the contents to any other 
> person, use it for any purpose, or store or copy the 
> information in any medium. Copyright in this e-mail and any 
> attachments belongs to Alcatel-Lucent and/or its affiliated entities.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 28 Feb 2011 10:22:22 -0500

> I think metadata transmissions and updates can be treated the same way, that a 200 response does not mean metadata was correct - only that it was received and passed to the application. If we decid…

Yes, I think so too.

> [JRE] Yes, and the request to resend full metadata could optionally include some sort of error report that gave rise to the request. That error report could, for example, include the sequence numbe…

That is sort of what I had in mind about getting the two sides in sync.
But we will have to think more about exactly what it means.
At best I think it will describe the *current* state of things in this 
RS, without regard to what has happened in the past.

Even that may present some technical challenges. But we should keep this 
desire in mind.

	Thanks,
	Paul

---

From: "Henry Lum" <[email-masked]>
Date: Mon, 28 Feb 2011 08:13:17 -0800

> > [JRE] Yes, and the request to resend full metadata could optionally
> include some sort of error report that gave rise to the request. That
> error report could, for example, include the sequence number of the
> metadata update that was found to be in error.
> 

[[hlum]] I agree it would be reasonable for the SRC to send the current
state as a snapshot and forget what happened previously. The subsequent
sequence numbers only refers to the current snapshot rather than the
original one, so in a way we are resetting the sequence number.

---

From: Leon Portman <[email-masked]>
Date: Mon, 28 Feb 2011 18:18:21 +0200

+1

Leon

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henry Lum
Sent: Monday, February 28, 2011 6:13 PM
To: Paul Kyzivat; Elwell, John
Cc: [email-masked]
Subject: Re: [siprec] Metadata XML document error representation for whole doc or for portion of doc

> >> We are also thinking of providing the SRS with a mechanism to 
> >> request the SRC to ask for the current snapshot
> so
> Even that may present some technical challenges. But we should keep 
> this desire in mind.

[[hlum]] I agree it would be reasonable for the SRC to send the current state as a snapshot and forget what happened previously. The subsequent sequence numbers only refers to the current snapshot rather than the original one, so in a way we are resetting the sequence number.