Subject: [Speechsc] FWD: Comments on Speech Services Control

From: David Oran <[email-masked]>
Date: Tue, 10 Dec 2002 07:26:07 -0500

------------ Forwarded Message ------------
Date: Tuesday, December 10, 2002 12:24 AM -0800
From: Karl Auerbach <[email-masked]>
To: [email-masked], [email-masked]
Subject: Comments on Speech Services Control

Below are some thoughts/comments I had on Speech Services Control from the
meeting that was held at the recent IETF in Atlanta.  Sorry for taking so
long to get this written.

I attended the meeting out of curiousity, so please bear with me if I
repeat things that others may have mentioned or simply miscomprehended
some basic notions.

Some of these comments derive from the work I did on IP/TV, an RTP/RTCP
based audio/video application.

Since I'm not on the mailing list, I have sent these comments to you
directly.  If these make sense, you may wish to forward them to the
discussion list.

REFERENCE TIMING

The matter of time is one that is important to any multimedia application
- and I include speech services under the rubric of multimedia because
what may seem to be a simple speech stream may actually be accompanied by
other streams, such as written transcripts (imagine a "follow the bouncing
ball" kind of application) or other tracks, such as a commentary, language
translation, or powerpoint-like slide show.

In all of these cases, there needs to be a time reference that can
coordinate all of these streams.

In RTP/RTCP a time reference to GMT is established by RTCP.  But many
applications have taken short-cuts and don't bother with RTCP.

It is my suggestion that any speech services control efforts emphasize the
need to actually use RTCP to create a believable reference to GMT for
media streams.  And I'd also suggest that any derived items - like words
or phrases from a speech stream - have RTP media timestamps that are
consistent with the timestamp of the stream from which they were taken.
In other words, when a stream is broken into parts, the parts should bear
timestamps showing where they came from in the original stream and not
simply always start at zero.

On an ancillary point - some RTP receiver implementations (particularly
those that have to synchronize multiple RTP/RTCP streams, e.g. audio and
video streams) do not even try to begin to render their output (whether
that be audio or video) until they have heard an initial RTCP packet.
This can mean a rendering delay of several seconds, which can be very
frustrating to a user.

TIMING - NTP REPRESENTATION IN RTP

RTP/RTCP carries timestamps in an NTP format.  It isn't easy to write fast
and correct code to convert these NTP formats to the seconds/microseconds
(or seconds/nanoseconds) formats used in operating systems and the
language libraries.

As is done in the IP checksum RFCs, it may be worthwhile to publish source
code for these conversions.

MEDIA FORMATS/CODECS

Some media encodings use forward and backwards interpolations from base of
key samples.  (This may be more prevalent in video than in audio.)  Such
encodings are dificult to pick-up and play-out from arbitrary points -
often playout must begin from a base or key sample.  (In such cases,
muting plus accellerated playout possibly can be used to hide the leading
synchronization from the user.)  When using codecs of this nature, it may
be difficult to split spoken speech into discreet words without prepending
sufficient codex synchronization context information to each word as well
as an indication of when the actual word begins.  In other words, raw
media files for each word may not be sufficient; some additional wrapper
will be needed to indicate the where (and when) in the sample the actual
word sound begins and where/when it ends.

There is an ancillary issue - there might be a temptation to transcode
from original formats to encodings that are more easily chopped into
discreet words.  However, transcoding often causes loss of quality and
comprehensibility.

HUMAN INTERFACES

Humans are used to buttons and knobs that have instantaneous effect.  In a
multimedia/speech control application, when the user says "stop", the
stream must stop, and stop right now.  When the actual source of the
material is a remote computer and there is a non-zero transit time across
the network, there can easily arise some ambiguities about where/when a
stream has been stopped and what should happen when the stream is resumed.
It may be necessary to require the receiving application to buffer a
reasonable amount of traffic-in-flight that is received after the user
stops a stream. (This can be made more difficult by the matter mentioned
previously in which certain codecs can not be started at arbitrary
points.)  [This gets far more complex in multicast cases.]

Arbitrary jumps through a stream can become quite a burden on the sender
- some encodings may require the sender to read through the sending file
to find the desired new start point.  We found that it was sometimes quite
useful to preprocess certain media files to build indices that could be
used to accellerate jumps.

		--karl--

---------- End Forwarded Message ----------

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc