Tips, Tricks and Secrets of the Android Multimedia APIs - Tistory

Tips, Tricks and Secrets of the Android Multimedia APIs Doug Stevenson Wednesday, February 27, 2013 Slides gzip: http://goo.gl/pBXS5 Code Samples: http://goo.gl/XYTJu

Overview of Core Android Audio Capabilities Android prescribes native support for: ●

Audio encoding and decoding

●

Video encoding and decoding

●

Image encoding and decoding

●

Streaming audio and video

http://developer.android.com/guide/appendix/media-formats.html 2

Audio Support Decoding: ●

AAC, AMR-WB, AMR-NB, MP3, MIDI, Vorbis, WAVE

●

Since Android 3.1: FLAC

●

Since Android 4.1: AAC ELD

Encoding: ●

AAC LC, AMR-WB, AMR-NB

●

Since Android 4.1: HE-AACv1, AAC-ELD, WAVE

Containers: ●

3GPP, MPEG-4, MP3, FLAC, OGG, WAVE, MPEG-TS (3.0+)

3

Video Support Decoding ●

H.263, H.264 (baseline profile), MPEG-4 SP

●

Since Android 2.3.3: VP8

Encoding ●

H.263

●

Since Android 3.0: H.264 (baseline profile)

Containers: ●

3GPP, MPEG-4, WebM, MPEG-TS (3.0+), Matroska (4.0+) 4

Streaming Support ●

RTSP

●

HTTP progressive

●

Since Android 3.0: HTTP live (draft)

●

Since Android 3.1: HTTPS

5

Notable Exceptions Samsung devices have ●

DivX

●

WMV

●

AVI

●

FLV

Recommendation: Don’t count on these 6

AudioManager System Service Provides access to top level audio functions ●

Volume

●

Routing

●

Focus

System Service instance: AudioManager am = (AudioManager) Context.getSystemService(Context.AUDIO_SERVICE); 7

AudioManager: System Audio Streams Audio streams types are identified by AudioManager.STREAM_* integer constants: ● STREAM_ALARM ● STREAM_DTMF ● STREAM_MUSIC ● STREAM_NOTIFICATION ● STREAM_RING ● STREAM_VOICE_CALL 8

AudioManager: Get Stream Volume ●

Minimum volume for any stream is 0.

●

Maximum volume for a stream: –

●

AudioManager.getStreamMaxVolume(stream_type)

Current volume for a stream is: –

AudioManager.getStreamVolume(stream_type)

9

AudioManager: Set Stream Volume ●

●

Adjust volume of the most relevant stream –

AudioManager.adjustVolume(direction, flags)

–

Directions: ADJUST_LOWER, ADJUST_RAISE, ADJUST_SAME

Adjust volume of a specific stream –

●

Adjust volume of the most relevant stream OR the given fallback –

●

AudioManager.adjustStreamVolume(stream_type, direction, flags) AudioManager.adjustSuggestedVolume(direction, stream_type, flags)

Directly set the given stream’s volume –

AudioManager.setStreamVolume(stream_type, volume)

10

AudioManager: Stream Volume Flags FLAG_ALLOW_RINGER_MODES Allows system to switch into vibrate mode when attempting to drop ringer volume below 0 FLAG_PLAY_SOUND Allows system to play a sound when changing volume FLAG_REMOVE_SOUND_AND_VIBRATE Removes queued feedback for volume changes FLAG_SHOW_UI Shows system UI for volume adjustment (toast) FLAG_VIBRATE Allows vibrate if going into vibrate ringer mode

11

AudioManager: Audio Mode Audio Mode is the current high-level function of device audio. int AudioManager.getMode() MODE_NORMAL Not in a call MODE_RINGTONE Phone is ringing MODE_IN_CALL Currently in a call MODE_IN_COMMUNICATION Currently in a VOIP session

12

AudioManager: Audio Focus ●

Helps apps cooperate with audio output when trying to play simultaneously

●

First available with API 8, Froyo

●

Apps can indicate different kinds of audio focus –

Long term audio (e.g. music, video, games)

–

Temporary audio (e.g. notifications, feedback, driving directions)

13

AudioManager: Audio Focus API int AudioManager.requestAudioFocus(listener, stream_type, duration_hint) int AudioManager.abandonAudioFocus(listener)

Both return one of: ● AUDIOFOCUS_REQUEST_GRANTED ●

AUDIOFOCUS_REQUEST_FAILED.

14

AudioManager: Audio Focus Duration Hint duration_hint options AUDIOFOCUS_GAIN Long term focus request; e.g. music, video AUDIOFOCUS_GAIN_TRANSIENT Temporary focus request; e.g. notifications, driving directions AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK Like AUDIOFOCUS_GAIN_TRANSIENT, except current focus holder may keep playing if it “ducks” its volume 15

AudioManager: Audio Focus Change Listener Interface AudioManager.OnAudioFocusChangeListener class FocusListener implements AudioManager.OnAudioFocusChangeListener { public void onAudioFocusChange(int focusChange) { // Act on gaining/losing focus } }

The listener you provide to requestAudioFocus will get called with focus change updates as they happen. 16

AudioManager: Audio Focus Change Values AUDIOFOCUS_GAIN You are gaining focus when duration hint was AUDIOFOCUS_GAIN AUDIOFOCUS_LOSS ●

●

Indicates loss of focus for unknown duration

You should stop playing AUDIOFOCUS_LOSS_TRANSIENT ●

●

Indicates temporary loss of focus

●

Focus should be regained again soon

●

Resume playback when you get AUDIOFOCUS_GAIN again

17

AudioManager: Audio Focus Change Values AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK ●

Temporary loss of focus to another app using duration hint AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK

●

You may continue to play if you duck volume

●

You should gain focus again soon (and restore volume)

●

Consider restoring volume gently

18

AudioManager: “Becoming Noisy” What happens to audio after the headset is disconnected? Register a BroadcastReceiver for the action ACTION_AUDIO_BECOMING_NOISY context.registerReceiver( receiver, new IntentFilter(AudioManager.ACTION_AUDIO_BECOMING_NOISY) );

19

SoundPool Maintains a simple pool of sounds in memory for low latency playback. Good for: ●

Simple games that play a variety of sounds

●

Soundboard style apps

20

SoundPool Features ●

Sounds may be loaded from raw resources, assets, or local files

●

Sounds may be played, paused/resumed, stopped at any time

●

Individual sounds may be played simultaneously

●

Looping

●

Adjustment of playback rate (time and pitch)

●

Adjustment of volume

21

SoundPool Tips Good to know: ●

Prior to Android 2.2 (API level 8), there was no good way to know when a sound is done loading. –

●

Now there is OnLoadCompleteListener

Suggest using OGG/Vorbis encoding for samples and using assets to bundle them in your APK.

22

SoundPool Demo See ActivitySoundPool.java

23

MediaPlayer Plays both audio and video files of types supported by the device. Can play media from: ●

Local files (only using FileDescriptor)

●

Streaming HTTP and RTSP servers (with buffering)

●

Content URIs

24

MediaPlayer State Machine ●

Strict state machine that must be followed

●

Wrong call sequence? Throws unchecked exceptions. –

●

Don't catch them; fix your app

Diagram is in the javadoc

25

MediaPlayer Video Playback For video playback, one of the following is required: ●

●

a SurfaceHolder obtained from a SurfaceView in your view hierarchy a Surface, usually from a TextureView’s SurfaceTexture (with Android 4.0 or later)

Android VideoView source code is a good place to start to understand how MediaPlayer can show video. –

frameworks/base/core/java/android/widget/VideoView.java 26

MediaPlayer Tips ●

●

●

Calls documented as asynchronous may still have observed delays (e.g. streaming media) Consider moving all MediaPlayer methods off the main thread (HandlerThread is handy, queues and serializes work sent to it) MediaPlayer in conjunction with a Service can keep music playback alive in the background.

27

MediaPlayer features in Gingerbread (API 9) Attach audio effects ● ●

getAudioSessionId() / setAudioSessionId() Create an effect using session id (android.media.audiofx)

●

attachAuxEffect(int effect_id)

●

setAuxEffectSendLevel(float level)

28

MediaPlayer features in Jellybean (API 16) Select from multiple audio and video tracks (e.g. multiple languages and camera angles) ●

getTrackInfo() to iterate tracks

●

selectTrack()

Display timed text (subtitles) (SubRip only) ●

addTimedTextSource() to make the text tracks available

●

getTrackInfo() / selectTrack()

●

setOnTimedTrackListener()

●

deselectTrack()

29

MediaPlayer features in Jellybean (API 16) Video scaling modes ● ● ●

setVideoScalingMode(int mode) VIDEO_SCALING_MODE_SCALE_TO_FIT VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING

Seamless transition between sources ●

setNextMediaPlayer() to queue the next MediaPlayer object

●

Supposed to be seamlessly transitioned from the current

●

May not work as seamlessly as expected 30

MediaPlayer: Memory Management Call release() when done with a MediaPlayer instance to free backing native resources.

31

MediaRecorder Records audio using recording hardware (microphone) ●

Has a state machine similar to MediaPlayer

●

Does not work on the emulator

●

Requires permission android.permission.RECORD_AUDIO –

●

Implicit feature: android.hardware.microphone

Not all devices have a mic

32

MediaRecorder: Choosing Audio Source Choose audio source using setAudioSource() with a constant from MediaRecorder.AudioSource ● DEFAULT ● MIC ● CAMCORDER ● VOICE_UPLINK, VOICE_DOWNLINK, VOICE_CALL

●

–

http://code.google.com/p/android/issues/detail?id=4075

–

http://code.google.com/p/android/issues/detail?id=2117

VOICE_RECOGNITION, VOICE_COMMUNICATION (API 11) 33

MediaRecorder: Choosing Output Container Choose output container format using setOutputFormat() and a constant from MediaRecorder.OutputFormat ● DEFAULT ● THREE_GPP ● RAW_AMR ● MPEG_4 ●

AMR_WB (API level 10, GB)

●

AMR_NB (API level 10, GB)

●

AAC_ADTS (API level 16, JB)

34

MediaRecorder: Choosing Output Codec Choose output audio format using setAudioEncoder() and a constant from MediaRecorder.AudioEncoder ● DEFAULT ● AMR_NB ●

AMR_WB (API level 10, GB)

●

AAC (API level 10, GB)

●

AAC_ELD (API level 16, JB)

●

HE_AAC (API level 16, JB) 35

MediaRecorder: Memory Management Call release() when done with a MediaPlayer instance to free backing native resources.

36

MediaPlayer/MediaRecorder Demo See ●

ActivityMediaPlayerAudio.java

●

ActivityMediaPlayerVideo.java

●

ActivityMediaRecorderAudio.java

37

AudioTrack ●

Low level audio playback mechanism

●

Raw, uncompressed PCM only

●

Good if doing digital signal processing

Remember: MediaPlayer → compressed AudioTrack → raw 38

AudioTrack Modes ●

●

Static: for loading an entire sound into memory for low latency playback (similar to SoundPool) Streaming: for writing a continuous stream of audio that may be too big to fit in memory

39

AudioTrack Constructor new AudioTrack( int streamType, int sampleRateInHz, int channelConfig, int audioFormat, int bufferSizeInBytes, int mode );

40

AudioTrack Constructor Args streamType ●

selects the Android audio stream destination (AudioManager STREAM_* constants)

sampleRateInHz ●

selects the sample rate (e.g. 44100 for CD quality = 44.1KHz)

channelConfig ●

●

selects output channel configuration (AudioFormat CHANNEL_OUT_* constants) typically CHANNEL_OUT_MONO or CHANNEL_OUT_STEREO 41

AudioTrack Constructor Args audioFormat ●

Selects the output audio format (AudioFormat ENCODING_* constants)

●

ENCODING_PCM_16BIT or ENCODING_PCM_8BIT (only 16 guaranteed)

buffserSizeInBytes ●

● ●

Interpretation depends on mode: –

if MODE_STATIC: max size of entire sound

–

if MODE_STREAM: buffer for collecting audio before output

All writes to this AudioTrack must be smaller than this. Must be at least AudioTrack.getMinBufferSize(sampleRateInHz, channelConfig, audioFormat) with same values here 42

AudioTrack Constructor Args mode ●

MODE_STATIC or MODE_STREAM

43

AudioTrack: Writing Audio PCM is written to AudioTrack using the following methods: ●

public int write(byte[] audioData, int offsetInBytes, int sizeInBytes)

●

public int write(short[] audioData, int offsetInShorts, int sizeInShorts)

44

AudioTrack: MODE_STATIC 1. Instantiate AudioTrack 2. write() all the samples 3. Call play() 4. Allow the audio to play as long as you want 5. Call stop() 6. To play again, call reloadStaticData(), goto 3 7. Call release() to free native resources 45

AudioTrack: MODE_STREAM 1. Instantiate AudioTrack 2. Call play() 3. write() samples to play as they become available 4. Call stop() 5. Call release() to free native resources

46

AudioTrack: MODE_STREAM Tips ●

Writing to AudioTrack can be a blocking operation – don't use the main thread.

●

Use a dedicated thread for ALL AudioTrack calls.

●

Set a higher priority for the writing thread: ●

●

android.os.Process.setThreadPriority(android.os.Process. THREAD_PRIORITY_AUDIO);

Write continuously. No data available? Write small zeroed buffers while waiting (or hear a click when audio playback disengages). 47

AudioTrack: Markers and Periods AudioTrack markers and periods track live playback ●

●

●

Markers invoke a callback when the active playback reaches a certain point Periods invoke a callback when the period time reaches a multiple Position is measured in samples (aka frames), NOT bytes (one 16 bit sample = 2 bytes)

48

AudioTrack: Markers and Periods setNotificationMarkerPosition(int pos) getNotificationMarkerPosition() ●

Get/set the marker position, measured in samples

●

Listener will get called when the marker is crossed

setPositionNotificationPeriod(int period) ● ●

Sets the period for notifications, measured in samples Listener will be called when the playback head reaches a multiple of the period 49

AudioTrack: Markers and Periods setPlaybackPositionUpdateListener(listener) Sets the listener to be called when the marker is crossed or a period elapses during playback class MyListener implements OnPlaybackPositionUpdateListener { void onMarkerReached(AudioTrack track) { } void onPeriodicNotification(AudioTrack track) { } }

50

AudioTrack: Looping int setLoopPoints( int startInFrames, int endInFrames, int loopCount ) ●

Sets up playback looping between a start and end frame.

●

Only makes sense in static mode.

51

AudioTrack Features in Gingerbread (API 9) Attach audio effects (just like MediaPlayer) ● ●

getAudioSessionId() / setAudioSessionId() Create an effect with session id (android.media.audiofx)

●

attachAuxEffect(int effect_id)

●

setAuxEffectSendLevel(float level)

52

AudioRecord Used for capturing raw audio data from the device’s microphone or other audio inputs. ●

No local audio capture

53

AudioRecord Constructor new AudioRecord( int audioSource, int sampleRateInHz, int channelConfig, int audioFormat, int bufferSizeInBytes );

54

AudioRecord Constructor Args audioSource ●

Selects the audio source (MediaRecorder.AudioSource constant, e.g. MIC)

sampleRateInHz ● ●

Selects the sample rate (e.g. 44100 for CD quality = 44.1KHz) All devices are supposed to support 44100, but expect 16000 and 8000 as well

55

AudioRecord Constructor Args channelConfig ●

●

Selects output channel configuration (AudioFormat CHANNEL_OUT_* constants) Typically CHANNEL_OUT_MONO unless device has stereo recording mics

audioFormat ●

●

Selects the output audio format (AudioFormat ENCODING_* constants) ENCODING_PCM_16BIT or ENCODING_PCM_8BIT 56

AudioRecord Constructor Args audioSource ●

Selects the audio source (MediaRecorder.AudioSource constant, e.g. MIC)

sampleRateInHz ● ●

Selects the sample rate (e.g. 44100 for CD quality = 44.1KHz) All devices are supposed to support 44100, but expect 16000 and 8000 as well

57

AudioRecord Constructor buffserSizeInBytes ● ●

Internal AudioRecord buffer size Must be at least AudioRecord.getMinBufferSize(sampleRateInHz, channelConfig, audioFormat) with same values here

58

AudioRecord: Reading Audio PCM is read from AudioRecord using the following methods: ●

public int read(byte[] audioData, int offsetInBytes, int sizeInBytes)

●

public int read(short[] audioData, int offsetInShorts, int sizeInShorts)

●

public int read(ByteBuffer audioBuffer, int sizeInBytes)

●

Reads can block, so always read in a dedicated thread.

●

Read continuously, or lose audio samples 59

AudioRecord: Reading Audio byte[] buffer = new byte[min_buffer_size]; audioRecord.startRecording(); while (!finished) { int bytes_read = audioRecord.read(buffer, 0, min_buffer_size); if (bytes_read > 0) { // Do something with buffer } else { break; } } audioRecord.stop(); 60

AudioTrack / AudioRecord Demo See ActivityAudioTrack.java

61

New Media APIs in JellyBean MediaMetadataRetriever ●

Extract metadata from the media container

●

Extract still video frames as Bitmaps

MediaExtractor ●

Demuxes and pulls encoded A/V from some input source

MediaCodec ●

Encodes/compresses raw A/V data

●

Decodes/decompresses A/V to raw format 62

Decoding Audio with JB APIs Use MediaExtractor and MediaCodec in tandem 1. Do all work OFF the main thread 2. Initialize a new MediaExtractor with data source 3. Select desired tracks from it 4. Initialize a MediaCodec decoder using metadata from MediaExtractor 5. Loop: –

Read encoded data from MediaExtractor

–

Write encoded data to MediaCodec's input buffers

–

Read decoded data from MediaCodec's output buffers 63

MediaExtractor: Create and Init AssetFileDescriptor fd = getAssets().openFd(asset_name); MediaExtractor extractor = new MediaExtractor(); extractor.setDataSource( fd.getFileDescriptor(), fd.getStartOffset(), fd.getLength() ); fd.close();

64

MediaExtractor: Selecting a Track MediaFormat format; for (int i = 0; i < extractor.getTrackCount(); i++) { format = extractor.getTrackFormat(i); if (track_is_interesting) { extractor.selectTrack(i); } }

65

MediaCodec: Create and Init String mime_type = format.getString(MediaFormat.KEY_MIME); MediaCodec mediaCodec = MediaCodec.createDecoderByType(mime_type); mediaCodec.configure( format, // MediaFormat from selected track null, // Surface, for video only null, // MediaCrypto, for encrypted source only 0 // flag CONFIGURE_FLAG_ENCODE if encoding only ); mediaCodec.start(); ByteBuffer[] codecInputBuffers = mediaCodec.getInputBuffers(); ByteBuffer[] codecOutputBuffers = mediaCodec.getOutputBuffers();

66

Recap MediaExtractor ●

Ready to read encoded audio from source

MediaCodec ●

Ready to decode encoded audio

Input Buffers ●

Used to send encoded audio to MediaCodec

Output Buffers ●

Used to receive decoded audio 67

Reading and Decoding Encoded Data Pt. 1 int inputBufIndex = mediaCodec.dequeueInputBuffer(-1); // obtain buf ByteBuffer dstBuf = codecInputBuffers[inputBufIndex]; int sampleSize = mediaExtractor.readSampleData(dstBuf, 0); long presentationTimeUs = 0; boolean inputEnded = false; if (sampleSize < 0) { inputEnded = true; sampleSize = 0; } else { presentationTimeUs = mediaExtractor.getSampleTime(); }

68

Reading and Decoding Encoded Data Pt. 2 mediaCodec.queueInputBuffer( inputBufIndex, 0,

// offset within input buf

sampleSize, presentationTimeUs, inputEnded ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : 0 ); if (!inputEnded) { mediaExtractor.advance(); }

69

Receiving Decoded Data BufferInfo bufferInfo = new BufferInfo(); int res = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); if (res >= 0) { ByteBuffer buf = codecOutputBuffers[res]; // receive and process decoded data } else if (res == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) { codecOutputBuffers = mediaCodec.getOutputBuffers(); } else if (res == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) { MediaFormat oformat = mediaCodec.getOutputFormat(); // Make adjustments based on new format } else if (res == MediaCodec.INFO_TRY_AGAIN_LATER) { // Nothing available now, maybe later }

70

About Output Buffers ●

May be a “direct” ByteBuffer –

●

You may have to copy it to local byte array to process in java

If you call a get* method on it to copy its data, you have to clear the ByteBuffer before releasing it back to the MediaCodec

ByteBuffer buf = codecOutputBuffers[idx]; byte[] readBuffer = new byte[bufferInfo.size]; buf.get(readBuffer, 0, bufferInfo.size); buf.clear(); mediaCodec.releaseOutputBuffer(outputBufIndex, false); 71

MediaExtractor/MediaCodec Demo See ActivityMediaCodecAudioDecode.java

72

Audio with the NDK ●

OpenSL ES 1.0.1 APIs available with Gingerbread

●

Portable to other platforms with OpenSL ES

●

Some Android-specific extentions

●

Decode to PCM available in ICS (level 14)

73

Thank you!

Slides gzip: http://goo.gl/pBXS5 Code Samples: http://goo.gl/XYTJu

74