Greetings iPhone / iOS developers.

This page introduces the new AVAnimator library that makes it easy to implement non-trivial animated video content in iOS. The types of animations discussed here do not include view transition animations or layer property animations, as these are already well supported in iOS. What iOS does not support is the rapid display of a sequence of images, like this:

While iOS does support H.264 video, there are a number of situations where using H.264 is not ideal, such as:

1. Non-fullscreen videos

In iOS 3, only fullscreen H.264 video playback was possible. A developer could not display custom controls over H.264 movie content. Things have improved in iOS 4, as it is now possible to display H.264 movie content in only a portion of the screen.

2. Multiple simultaneous videos

The H.264 playback support in iOS is implemented with a hardware decoder, so playing a movie is efficient. But the result is that it is impossible to play multiple videos at the same time. An app that features two characters on screen at the same time cannot be implemented with two H.264 videos, since only one H.264 video can be decoded at a time.

3. Looping/Combining videos

Seamlessly looping or playing H.264 videos back to back is difficult because the built-in H.264 playback classes introduce small delays when restarting or switching between videos. The end result is visual glitches that make an app look unprofessional.

4. No alpha channel support in H.264

It is not possible to create a H.264 video with an alpha channel. A typical use would be for an animated character or a live video shot against a green screen. A video producer would make use of an alpha channel so that no specific background would appear behind the character. When the app is run, different background pictures or colors could be set and the character would be blended into the background. Unfortunately, creating a H.264 video with an alpha channel is not currently possible.

5. No support for lossless video content

H.264 is great for generic video content, but the video encoding approach is lossy and does not provide good results for certain types of content. Text elements in particular look surprisingly bad when encoded with H.264 because the anti-aliased font edges are blurred in a way that significantly reduces readability.

6. H.264 may not provide the smallest video file size

For general purpose video, H.264 videos will be smaller than a lossless video. But, in a number of cases encoding with H.264 can create files that are larger than a compressed lossless video. For example, a video made from a series of screen captures, a computer generated scene or animated movie with a limited color palette. Results will depend on the specific video content.

7. Cannot sync H.264 video to an external audio file

An H.264 video can be encoded to include both audio and video in one movie file, but it is not possible to sync playback of H.264 video to an external audio file. The result is that if the same video needs to be synced to multiple audio tracks, then each video needs to be encoded with each audio track. The result is an app download size that is larger than it needs to be.

The AVAnimator library addresses the problems identified above and makes a number of hard things easy. The implementation uses highly optimized ARM asm code for optimal blit performance. Memory usage at runtime is minimal and there are no memory leaks. The AVAnimator library supports loading three types of video files, Quicktime (Animation codec), APNG, and MVID. The MVID file format is similar to the Animation codec except that the in-memory layout and pixel format is optimized for iOS. All the video formats support lossless video and a full alpha channel. Video content can be rendered into a UIView or to a Core Graphics CALayer.

This AVAnimator library is the result of much research and long hours of development. The library consists of a set of Objective-C classes and low level C code to read the Quicktime, APNG, and MVID files and decode video frames. This Quicktime Animation decoder for iOS supports 16BPP, 24BPP, and 32BPP modes. The APNG decoder supports 24BPP and 32BPP modes. The MVID decoder supports 16BPP, 24BPP, and 32BPP modes and does not require a loading stage like the Quicktime and APNG formats.

The AVAnimator library is currently at version 1.1.2. The full source code is included in each of the example downloads.

• 1.1.2 (Jan 2012)

For Xcode 4.2, fixes compilation with Clang
Add ability to capture final animation frame as app goes into background
Add logic to recover from resource loading error


• 1.1 (Jun 2011)

Add APNG support
Add zero copy from memory mapped files optimization
Add regression tests to QTFileParserApp example


• 1.0 (Apr 2011)

Initial Release
For Xcode 3.X
Target iOS 3.X API

Topics:

1. Quicktime Animation codec
2. APNG decoder
3. PNG zlib compression util
4. QTFileParserApp Example
5. StreetFighter Example
6. Animated App Boot Example
7. AVSync Example
8. ApngPlayer Example
9. License Issues

Quicktime Animation codec : Taking it back to the old school...

You might be surprised to learn that the iPhone does not support Quicktime. Why would this be? Apple invented Quicktime, so why is a developer writing iOS applications not be able to access Quicktime APIs and related functionality? The only functionality available is playback of H.264 encoded videos, with very little flexibility.

On the desktop, there is a huge assortment of tools that can read and write Quicktime files. For example, one could create fancy animations using After Effects and export the result to a Quicktime mov file using the lossless Animation codec. A Quicktime file would play just fine on the desktop, but the only way to put this animation into an iOS app would be to encode with H.264.

The specific areas where H.264 is weak is accurate encoding of graphics created from a limited color palette and encoding of large regions of the same color. For example, a screen capture, or an animation created by an artist using a limited range of colors. For example:

The results of encoding this ghostly little guy as a 480x320 Quicktime Animation vs the iPhone H.264 preset in Quicktime 7 are as follows:

The Quicktime Animation encoded movie seems worse because the file is larger. But, one must consider the effect of adding data compression to the mix. After compressing with 7zip (7zip gives better results than gzip or bzip2), the resulting file sizes are:

The H.264 video did not compress much, but the Quicktime Animation encoded mov got a LOT smaller. This is just a trivial example with 6 frames, but it shows how the combination of a modern data compression library like 7zip and the older Quicktime Animation codec can result in a significant reduction in file sizes.

If an app were to include many animations, the file size difference would start to add up quickly. As an example, my own iPhone app called iPractice makes extensive use of complex animations and clocks in at just over 11 megs. Early testing showed that the download size would be about 250 megs if H.264 video were used instead.

File size is important, but there is another reason that the Quicktime Animation codec is useful. The Animation codec was designed long ago in an era when computers were significantly slower. As a result, it is very simple and even very limited hardware can do a good job of displaying video encoded with the Animation codec. The iPhone is an embedded device with a rather slow CPU, so any low-level code must keep these limitations in mind and be significantly optimized. The Quicktime Animation codec support included in the AVAnimator library is highly optimized and includes a number of specialized modes to address specific functional requirements.

For example, minimal CPU usage and minimal file size can be addressed via a runtime conversion from an attached mov file to an optimized internal format. On the other hand, if instant playback with no startup or conversion time is required, an mov file can be opened and rendered directly. A Quicktime Animation file supports frame deltas, meaning only the pixel level data that changes from one frame to the next is stored. These frame deltas can be read and used at runtime to modify only those pixels that change from one frame to the next. Frame deltas are a significant runtime optimization, available with the Quicktime Animation and MVID input formats. In addition, the Quicktime Animation codec supports 16BPP movies (rgb555 pixels), which means that at runtime, each frame buffer is one half the size of a 24BPP or 32BPP frame buffer. Also, don't forget that the Animation codec is lossless. So, you don't need to worry about H.264 encoding ruining your artist's expertly crafted colors.

(Importing a Quicktime Movie)

• http://omino.com/sw/qt_tools/
• qtexportani (A qt_export Shell Script)

Download and install the qt_tools package and the qtexportani script to make it easy to process any existing Quicktime movie so that it can be imported by the AVAnimator library. Note that you may need to run "sudo mkdir /usr/local/bin" before the qt_tools installer will work. The qtexportani script invokes the command line converter with the proper arguments so that a movie file with a single Animation codec track is generated. A typical usage might be to convert a Quicktime PNG movie to the Animation codec, and then convert the .mov to the iOS optimized .mvid format, like so:

$ qtexportani Superwalk_PNG.mov
wrote export_Superwalk_PNG.mov

$ qtaniframes -mvid export_Superwalk_PNG.mov
wrote export_Superwalk_PNG.mvid

See QTFileParserApp Example for more info about the .mvid format.

APNG Decoder : Advanced Lossless Compression

The AVAnimator library includes support for the new APNG animation format. An APNG file can be downloaded or attached to the project resources and displayed in the same way as a Quicktime Animation. In many cases, APNG files will compress to smaller sizes than Quicktime files. Using the png8 palette format in particular can lead to amazing reductions in file size. The downside is that decoding an APNG file can take 2 or 3 times longer than a Quicktime Animation, but the decoding need only happen once and the results can be cached. The APNG format supports full alpha channel and partial transparency in palette mode. File sizes depend on the specific video content being encoded.

Working with APNG files can sometimes prove tricky, because the format is rather new and poorly supported in many tools. Some of the most useful utilities are:

• japng
• apngasm
• apngopt
• apngdis
• apng2gif
• gif2apng

If you simply want to extract the frames of an APNG file as PNG images, the following command line utility is useful.

PNG zlib compression util : Maximum Compression for APNG files

The Quicktime Ghost Animation example above shows that using a 7zip compressed .mov file saves a lot of space as compared to encoding as H.264. In a similar way, a large amount of space can be saved by using 7zip to compress an APNG file. For comparison, the file sizes for a 7zip compressed .mov vs an .apng that was created with apngasm and apngopt are as follows.

The APNG is smaller than the 7zip compressed Quicktime file. One could attach this Ghost_opt.apng as an iOS project resource and use it as is. But a developer looking to generate the smallest possible download has another option. An APNG/PNG file implicitly applies zlib compression to IDAT data chunks stored inside the file, and each frame of an animation is stored in a different chunk. The result is that data segments appearing in different frames are not compressed together.

One can use 7zip to compress data common to different frames, but in order to do that the zlib compression must be removed before 7zip compression is applied. The following shows the resulting file size of the Ghost animation when the zlib compression is removed and then the file is 7zip compressed.

Amazing! The combination of palette APNG and 7zip compression has reduced the already small file size by half. To implement this optimization, use the following command line utility. This utility is a customized version of apngopt that reads an APNG file and writes an optimized version that uses no zlib compression.

QTFileParserApp Example : Everything and the kitchen sink

This example iPhone app contains the full source code for the AVAnimator library as well as utility classes that display a GUI which works like the built in movie controls. A number of example animations are included. The examples show how to display video and how to sync video to an audio stream. There is a demonstration of the 32BPP (Millions+) support, which can be used to render video with a full alpha channel. The examples show both the best case (a 16 BPP animation that is able to display at 30 FPS) as well as the worst case (a very complex alpha composite where almost every pixel changes on every frame). There is also an example of rendering directly into a CALayer, for you CoreAnimation users out there. This example also includes an iOS port of the 7zip LZMA SDK which provides a powerful decompression framework that is far more effective that zlib or bzip2 when compressing video content.

In addition, this download includes a command line MacOSX utility that reads a Quicktime Animation movie file and extracts the individual frames as a series of PNG images. This example is useful for testing an existing mov file on the desktop to make sure it can be decoded by the library. Only a Quicktime mov file that contains video encoded with the Animation codec (and no audio) can be decoded by the AVAnimator library. This command line utility can also be used to preprocess a .mov file into a .mvid file, an iOS optimized format that loads faster and compresses better than a .mov file. The .mvid format supports optimized frame delta blits and each pixel of a movie with an alpha channel would already be premultiplied, so that no load time preprocessing is required.

StreetFighter Example : StreetFighter II for iPhone

This iPhone project is a quick little hack based on this page that shows a punch, kick, and special move button. Note that unlike the UIImageView.animationImages approach used in the linked example, this implementation will not waste tons of memory holding animation frames. When the user presses a button, Ryu throws a punch, kick, or a fireball. This example is not a playable game, it is just a quick little demo of playing audio and video as a result of user actions. This demo was whipped up in a couple of hours with some files off the net. What this example demonstrates is how quickly one can create an iPhone app that makes use of existing audio/video resources.

Animated App Boot Example : Display animation at app boot time

This example iPhone app shows how to create an animation that is displayed when the app starts. The animation shows a spinning laser that becomes a company logo. This example is implemented in two ways for comparison purposes. The first implementation uses a compressed animation file attached as a resource. The second implementation demonstrates APNG animation file format support. The APNG implementation uses an 8 bit palette, which saves a lot of space (1/2 meg) but takes a bit longer to decode during the very first boot. After the first boot, the decoded video is saved and any future boot will begin to animate in less than a second. In comparison, a boot animation implemented with the built-in H.264 support takes about 4 seconds to load each time the app is started (results on 3G iPhone).

AVSync Example : Synchronize video to an audio track

This example project demonstrates how a video track can be displayed in sync with an audio track using the AVAnimator library. The video content is a C major scale lesson from iPractice, a digital practice assistant for guitar players. One requirement for a musical app is that the visuals need to be tightly synchronized to the audio track. If the video sync is off by as little as 1/20th of a second, the user experience will suffer. This example actually contains 2 audio tracks where one is a fast tempo of 70 BPM and the other is a slow tempo of 45 BPM. This example contains only one video track, so the playback rate of the video track is dynamically adjusted to match the tempo of the audio track. This type of playback rate adjustment and dynamic selection of the audio track used with a specific video track is not possible using the built-in movie player classes shipped with iOS. In addition, this example shows how a developer can use custom movie controls that look like the built-in movie controls.

ApngPlayer Example : APNG file player app for iOS

This example contains the full source code for the APNG file player utility program available in the app store. This program handles the details of downloading an APNG file via the network and then rendering each frame in the animation. This example app is designed for both iPhone and iPad.

License Issues : details matter

The AVAnimator library was constructed with the specific goal of providing a library that can be easily (and legally) incorporated into an iPhone app and sold via the app store. Libraries licensed under the LGPL (including libquicktime, quicktime4linux, and ffmpeg) are in murky legal territory due to static linking restrictions in the LGPL license. Read more about this issue here:

iOS/LGPL incompatibility
LGPL and App Store

The AVAnimator library does not include code covered by the LGPL, so this specific issue is not a problem. The AVAnimator library is dual licensed. Downloads included on this page are GPL licensed, you are free to download, evaluate, and use the AVAnimator software under the terms of the GPL.

A separate license is available to enable integration without being bound by the terms of the GPL. Companies or individuals that would like to integrate the AVAnimator software into a product can obtain a license for a one-time cost of 475 USD. Please contact me directly to finalize a license for a specific usage.

Individuals or companies producing educational software or other software deemed to be in the public interest may be eligible to license the AVAnimator library at no cost. A no cost license will be evaluated on a case by case basis.