Synchronizing multi-media streams

51 downloads 86464 Views 240KB Size Report
Outline of Presentation. ▻ Fundamental need for synchronization .... [2] A Guide to MPEG Fundamentals and Protocol Analysis (Including. DVB and ATSC), An ...
Synchronizing multi-media streams Kishan Shenoi CTO, Qulsar WSTS – 2013, San Jose, April 16-18, 2013

Outline of Presentation  Fundamental need for synchronization  Alignment of multiple streams  Conventional approach  Time alignment in multi-media  Using time-stamps for alignment  Concluding remarks

Fundamental need for Synchronization analog

ADC

digital

Processing Transmission (Store/Replay)

Conversion clock fADC

digital

DAC

analog

Conversion clock fDAC

Df = frequency difference

 Real time transmission of audio/video over digital networks

requires conversion from analog-to-digital (at source) and digital-to-analog (at destination)  Impact of frequency difference (Df ):  Eventually buffers will overflow/underflow (e.g. slips) (“obvious”)  Pitch Modification Effect (PME) (analogous to Doppler) makes recovered symbol clock ≠ transmit symbol clock (not so “obvious”)  Recovered waveform ≠ original waveform (more than just additive noise)

Alignment of multiple streams Transmission time = Tvideo

video

ADC

encode

Packetized Transmission

decode

DAC

video

Conversion clock fDAC-1

Conversion clock fADC-1

Df = frequency difference Df = frequency difference audio

ADC

Transmission time = Taudio encode

Packetized Transmission

Conversion clock fADC-2

decode

DAC

audio

Conversion clock fDAC-2

Df = frequency difference

Alignment required in time and frequency between the (multiple) streams

Conventional Approach Video Path Screen

Camera C

B1

Sampling frequency

Sampling frequency

m

b1

SP-V

B2

P-V

IP-V

B3

D-V

B4

Time-stamps

Video clock

Recovered Video clock

Time-stamps

Time-stamps

SP-A

microphone

b2

Recovered Audio clock

Time-stamps

Audio clock

P-A

IP-A

Audio Path

b3

D-A

S

b4

s speaker

 RTP time-stamps are based on a “count” of samples  Additional step required to translate “count” to “time”  Frequency offset between video/audio clocks can introduce

QoE impairments

Timing Alignment in Multimedia Video Path C

B1

SP-V

B2

B3 ( vi TS P d

Sampling frequency

STC, PCR, DTS PTS

P-AV

b1

SP-A

Recovered Video clock Recovered System clock

DT Sa ( au nd P dio TS )

b2

b3

S

o)

IP-AV

Sampling frequency

m

de

B4

n Sa DT STC, PCR

Time-stamps System clock

D-V

Recovered Audio clock

D-A

b4

s

Audio Path

 Frequency offset (wander) between audio and video sampling results in

loss of lip-sync – use System Clock for both  Frequency offset (wander) between send-side and receive-side system

clock results in freeze (video), breaks (audio), and possible loss of lipsync (align System Clock)

Using Time-stamps for Alignment  Emulate a constant delay:  Generate a “creation” time-stamp C when a block of digital samples are collected from the A/D  Predetermine a suitable delay X  Convert block to analog at time (C+X)  Time-stamps for audio and video are struck using

a common System Clock  System Clock at source and destination are

synchronized  Synchronization best achieved using:  Common PTP Grandmaster  Common GNSS (GPS)

Concluding Remarks  Using time-stamps linked to a common clock provides

the following benefits:  Alignment of audio in frequency  Alignment of video in frequency

 Alignment between multiple streams (audio and

video)  Jitter buffer action to absorb network PDV  Prescribed delay  Audio and video sources do not have to be in same

device (or geographic location)

Thank You!

Questions? Kishan Shenoi Title: CTO, Qulsar, LLC Email: [email protected] www.qulsar.com Back-up slides follow

MPEG-2 Timing Model

Robust Circuit Emulation Service

MPEG-2 timing (based on CES)

References  [1] MPEG2 Tutorial, http://www.bretl.com/mpeghtml/MPEGindex.htm.

 [2] A Guide to MPEG Fundamentals and Protocol Analysis (Including

DVB and ATSC), An MPEG Tutorial from Tektronix.  [3] ATIS IPTV Exploratory Group Report and Recommendation, Draft

Revision: 1.5, June 17, 2005.  [4] ATIS-0800002, IPTV Architecture Requirements.  [5] ISO/IEC 13818-1, Information Technology ― Generic coding of

moving pictures and associated audio information ― Systems  [6] ITU-T Recommendation H.222.0 {equivalent to [5]}  [7] ISO/IEC 13818-9, Information Technology ― Generic coding of

moving pictures and associated audio information ― Part 9: Extension for real-time interface for systems decoders.  [8] ETSI TS 102 034 v1.1.1 (2003-5), Digital Video Broadcast (DVB);

Transport of MPEG-2 Based DVB Services over IP Based Networks.