AVAnimator supports two kinds of video content, lossless and lossy. Lossless mode is ideal for animations that make use of limited color spaces and have lots of pixels that do not change from frame to frame, like this:
Lossy video content is supported using H.264 encoded video stored in a .m4v file. Unlike lossless mode, H.264 uses advanced video compression to encode an approximation of the original video. The H.264 format supports highly complex video data and works well for recorded video like this:
AVAnimator supports the following video file formats:
- Quicktime (Animation Codec) (on the desktop)
Download size of the final application is a critical concern when it comes to delivering your iOS application in the app store. Apps that are too large and take too long to download will result in lost sales. AVAnimator can help to minimize final app size. Support for either lossless or lossy video content means that you can choose the most effective option for your specific situation. For most video, H.264 encoded with ffmpeg and x264 will generate the smallest possible file. But in some cases, a H.264 encoded file will actually be larger and of lower quality than lossless encoded video. While H.264 is great for general video content, it is really terrible at encoding computer generated scenes like screen captures or animated movies with limited color usage.
H.264 can be played directly or decoded to Maxvid format. Files in the APNG or GIF89a format can be decoded to Maxvid. While Maxvid files are typically too large to be directly attached as a project resource, a developer can compress a plain Maxvid file using 7zip on the desktop and then decompress on the iOS device. A Quicktime file can be converted to Maxvid format on the desktop. H.264 format supports 24BPP video only. The APNG decoder supports 24BPP and 32BPP modes. The GIF89a decoder supports a 24BPP mode with a single transparent pixel value. The Maxvid format supports 16BPP, 24BPP, and 32BPP modes.
This AVAnimator library is the result of much research and long hours of development. The library consists of a set of Objective-C classes and low level C code to read the supported video formats and write an optimal Maxvid file. Once a Maxvid file is decompressed to disk, each frame image can be sent to the screen with highly optimized code.
- Quicktime Animation codec
- Maxvid format
- APNG decoder
- GIF89a decoder
- H.264 encoder and decoder
- H.264 + alpha channel decoder
- The sRGB colorspace
Quicktime Animation codec : Taking it back to the old school...
You might be surprised to learn that the iPhone does not support Quicktime. Why would this be? Apple invented Quicktime, so why is a developer writing iOS applications not able to access Quicktime related functionality? It is possible to play and decode frames from H.264 videos in recent versions of iOS, but a developer will quickly run into the limitations inherent in this one size fits all approach.
On the desktop, there are a huge assortment of tools that can read and write quicktime files. For example, one could create fancy animations using After Effects and export the result to a Quicktime mov file using the lossless Animation codec. A Quicktime file would play just fine on the desktop, but the only way to put this animation into an iOS app would be to encode with H.264.
The specific areas where H.264 is weak is accurate encoding of graphics created from a limited color palette and encoding of large regions of the same color. For example, a screen capture, or an animation created by an artist using a limited range of colors. Like so:
The results of encoding this ghostly little guy as a 480x320 Quicktime Animation vs the iPhone H.264 preset in Quicktime 7 are as follows:
The Quicktime Animation encoded movie seems worse, because the file is larger. But, one must consider the effect of adding data compression to the mix. After compressing with 7zip (7zip gives better results than gzip or bzip2), the resulting file sizes are:
The H.264 video did not compress much, but the Quicktime Animation encoded mov got a LOT smaller. This is just a trivial example with 6 frames, but it shows how the combination of a modern data compression library like 7zip and the older Quicktime Animation codec can result in a significant reduction in file sizes.
If an app were to include many animations, the file size difference would start to add up quickly. Just to give you an example, my own iPhone app called iPractice makes extensive use of complex 16BPP animations and clocks in at just over 11 megs. Early testing showed that the download size would be about 250 megs if H.264 video were used instead.
File size is important, but there is another reason that the Quicktime Animation codec is useful. The Animation codec was designed long ago in an era when computers were not fast. As a result, it is very simple and even very limited hardware can do a good job of displaying video encoded with the Animation codec. The iPhone is an embedded device with a rather slow CPU, so any low level code must keep these limitations in mind and be significantly optimized. More recent iPhone and iPad models feature faster CPUs, but an iOS device is still significantly more limited than a desktop machine. Developers must consider older devices when designing new software because many people still use older iPhone hardware.
Both Quicktime Animation and Maxvid support frame deltas, meaning only the pixel level data that changes from one frame to the next is stored in the file. These frame deltas can be read and used at runtime to modify only those pixels that change from one frame to the next. Frame deltas are a significant runtime optimization, only available with the Maxvid format.
The Quicktime Animation codec is no longer directly supported as a readable format in AVAnimator. In all cases, it is better to convert a Quicktime Animation movie into the Maxvid format on the desktop. The Maxvid format is an improved version of the approach used in the Quicktime Animation codec, but with specific changes to improve memory usage, runtime performance, and compression on iOS.
Maxvid format :
The Maxvid format is basically the same as a Quicktime file that contains only Animation formatted video data. Maxvid is a lossless video container that supports 16BPP (rgb555), 24BPP, and 32BPP (RGB+A) pixel data. The Maxvid format is specifically optimized to use minimal memory at runtime. In addition, a Maxvid file tends to compress more effectively using 7zip than a Quicktime formatted file. Data encoded to Maxvid is lossless, so you don't need to worry about lossy encoding (like H.264) messing up your artist's expertly crafted colors.
Once written to disk, a Maxvid file can be mapped into memory such that a specific video frame is contained in one mapping. The mapping is created when a video frame is needed and then the mapping is removed when the frame is no longer needed. This logic makes it possible to load very large video files without using up all the system memory. Mapping files into memory also means that looping a video is implemented in the most optimal way, since a specific video frame might still be available in the page cache the next time that frame in reached in a looping video.
Pixel data stored in a Maxvid file is optimized for iOS hardware. A pixel in the Quicktime Animation codec is formatted as big endian ARGB while a pixel stored in a Maxvid file is stored as little endian BGRA. For 32BPP pixels, color components are pre-multiplied with the alpha value. This is basically the same optimization that Xcode adds to each PNG image file attached to a project. The result is that image data can be passed directly into the video card without having to rearrange the pixel layout or do a costly multiply operation for every pixel. This is an important optimization for individual images, but it is even more important when applied to every frame (image) in a movie.
For more detailed info about this pixel layout optimization, take a look at this excellent article by Jeff LaMarche.
APNG Decoder : Lossless Compression
The AVAnimator library includes support for the new APNG animation format. An APNG file can be downloaded or attached to the project resources and displayed in the same way as a Quicktime Animation. In many cases, APNG files will compress to smaller sizes than Quicktime files. Using the png8 palette format in particular can lead to amazing reductions in file size. The downside is that decoding an APNG file can take 2 or 3 times longer than a Quicktime Animation, but the decoding need only happen once and the results can be cached. The APNG format supports full alpha channel and partial transparency in palette mode. File sizes depend on the specific video content being encoded.
GIF89a Decoder : Palette Compression
AVAnimator now includes support for converting existing animated gifs to the more runtime performant maxvid format. If an app needs to make use of existing GIFs online or from an artist that produces files in the GIF89a format then this support is useful. File sizes depend on the specific video content being encoded.
H.264 encoder and decoder :
AVAnimator includes support for decoding H.264 video to Maxvid format. In addition, a Maxvid file can be encoded to H.264. The encoding and decoding logic is implemented using native iOS APIs that make use of H.264 encoding and decoding hardware. H.264 supports only 24BPP video, no alpha channel is available by default in H.264.
One might find H.264 encoding at runtime useful in an application that records videos and then submits the video to a website. Say, for example, a video effect was to be applied to the video stream and then the result needed to be encoded as H.264 before it could be uploaded. A video recorded from the camera would need to be decoded from H.264 and saved as a Maxvid file. A video effect could then be applied by iterating over each frame in the video and processing the image, perhaps color frames could be converted to grayscale and then the result would be saved into a second Maxvid file. Finally, the grayscale video file could be encoded as H.264. The encoded H.264 would be significantly smaller than the Maxvid file and would upload to a website much more quickly.
H.264 + alpha channel decoder :
An exciting feature of AVAnimator is the ability to decode a H.264 + alpha channel video. Normally, H.264 does not support an alpha channel, only 24BPP video is supported. AVAnimator makes it possible to split a RGB+A video into RGB and A components stored in two different H.264 videos. The RGB video contains the Red, Green, and Blue color components encoded as a H.264 video. The A (Alpha component) video contains a grayscale representation of the Alpha channel. These two video components are combined back together at runtime into a single RGB+A video in Maxvid format.
Here are two frames from a walk cycle video with an alpha channel. Normally, this type of video would encode better with a lossless codec as opposed to H.264, but just ignore that for the purposes of this example.
The pixels around the walking character are fully transparent, so that the white background can be seen through the image. If the RGB+A components are split, then the RGB and A components would look like this:
This very simple example contains only fully transparent and fully opaque pixels. A real video shot in front of a green screen would contain pixels that are partially transparent. Partial transparency pixels would be represented with gray values in between black and white. Splitting the video into components and then encoding each as a H.264 can save significant amounts of space for video data that encodes well using H.264. Videos that would be several megs encoded as a Maxvid file can be compressed down to a couple of hundred kilobytes.
The sRGB colorspace : one gamut to rule them all
The sRGB colorspace is the only colorspace supported on iOS hardware. All RGB pixel values sent to iOS video hardware are assumed to be in the sRGB colorspace. So, how does one deal with colors defined in another colorspace like Adobe RGB? The trick is just to avoid the problem on the iOS device. All colorspace conversion is dealt with on the desktop, both for performance reasons and because MacOSX includes ColorSync which does the conversion automatically.
Images tagged with a specific colorspace and Quicktime files tagged with a specific colorspace are automatically converted to sRGB when encoding to Maxvid format. One can import images with weird custom colorspaces as long as the images are tagged with an embedded color profile. But, what about content that is not tagged? Basically, there is no perfect answer for how to deal with untagged content. The approach AVAnimator takes is to simply assume that all untagged content is already in the sRGB colorspace. Defaulting to sRGB makes it possible to import untagged Quicktime files exported from a tool like After Effects and assume that the color values are in the sRGB colorspace. Certain tools like After Effects are unable to embed a color profile in a Quicktime file, so import logic must make a default assumption as to the colorspace. An animator will need to double check that external tools like After Effects are configured to emit Quicktime files in the sRGB colorspace, then everything will convert automatically.
More info about colorspaces:
- sRGB colorspace
- sRGB vs. Adobe RGB
- ColorSync in MacOSX
- Untagged content
- Color Settings in Photoshop