Images: Reading JPEG Thumbnails

The problem: I want to create thumbnails for a folder of JPEGs. Many of these JPEGs came from a digital camera, so they're several thousand pixels wide. Loading the entire image and then scaling down to a 64x64 thumbnail sounds pretty wasteful: what else can I do? I'm 99% sure most digital cameras these days are embedding thumbnails inside the JPEG files for just this kind of problem...

Discussion

As far as I can tell: ImageIO won't read JPEG thumbnails unless you have JAI installed?

I googled "java example: ImageIO reading thumbnail" to see what I could find on the subject:

The first link was from Sun, and it said this:

3.3.4 Reading "Thumbnail" Images

Some image formats allow a small preview image (or multiple previews) to be stored alongside the main image. These "thumbnail" images are useful for identifying image files quickly, without the need to decode the entire image.

Applications can determine how many thumbnail images associated with a particular image are available by calling:

reader.getNumThumbnails(imageIndex);

If a thumbnail image is present, it can be retrieved by calling:

int thumbailIndex = 0;
BufferedImage bi;
bi = reader.readThumbnail(imageIndex, thumbnailIndex);

Right. Good. This makes sense. The problem is: it doesn't seem to really work.

The second link was much more insightful. A developer wrote:

I should have remarked that the JAI Image I/O Tools JPEG reader supports via the thumbnail method calls all thumbnails embedded in the JFIF APP0, JFXX APP0, and EXIF APP1 marker segments. Please see this javadoc for more information:

http://download.java.net/media/jai-imageio/javadoc/1.1/overview-summary.html#JPEG

I think that the only thumbnails supported by the Java SE Image I/O JPEG reader via the thumbnail method calls are those in the JFIF and JFXX marker segments. If you are unable to use JAI Image I/O Tools for some reason you could however derive the EXIF thumbnail by parsing the contents of the "unknown" node in the image metadata corresponding to the EXIF APP1 marker segment.

I think he just summed up the problem I was seeing. If I run this code (without JAI installed), I get an exception:

Iterator iterator = ImageIO.getImageReadersBySuffix("jpeg");
while(iterator.hasNext()) {
 ImageReader reader = (ImageReader)iterator.next();
 try {
  reader.setInput( ImageIO.createImageInputStream(jpeg) );
  BufferedImage thumbnail = reader.readThumbnail(0, 0);
 } catch(Exception e) {
  e.printStackTrace();
 }
}

(The exception I get is: java.lang.IndexOutOfBoundsException: No such thumbnail at com.sun.imageio.plugins.jpeg.JPEGImageReader.readThumbnail(JPEGImageReader.java:1354). Which makes sense based on what the second link said.)

Now I have nothing against JAI itself. But when I went to grab a copy just now: the jar was over 5 MB!

This is ridiculous. For widgets I present in this blog I'm not going to try to attach a 5 MB jar. (Is that even legal? I didn't look up the licensing.) Or worse: I'm not going to insist that the user separately download other packages to get my software to run well. It should run well straight out of the box.

I decided to try to extract the thumbnails myself. I did not want to write a fully-functional JPEG decoder (what are the odds I could compete with OS-specific optimized code that already exists?), but I might benefit from my own metadata retrieval.

Research

The first thing I needed was test cases. Lots of and lots of test cases. In addition to all the JPEGs lying around on my computer, I wanted some public domain images I could write unit tests against. I explored Flickr, and found a collection of images with no known copyright. I saved a few of them here.

Next I'd need to roll up my sleeves and study file specifications. My first search result pointed to this specifications PDF, but it is dated to 1992. I did more research and found that this PDF is a better fit.

A JPEG is (until the image data begins) clearly separated into discrete chunks of data. So modeled after the ZipInputStream, I wrote the JPEGMarkerInputStream to read each chunk separately.

According to the specifications: there are two types of markers that are supposed to provide all our thumbnails: APP0 and APP1.

The JAI documentation confirms this:

The [JAI] JPEG reader supports thumbnails. These may be derived from JFIF APP0, JFXX APP0, or EXIF APP1 marker segments.

About the APP0 Block

This marker was introduced in the original 1992 specification. It includes lots of basic metadata (resolution, version info), and can optionally include a thumbnail.

However none of my sample images use this marker to embed a thumbnail. I even branched out and scanned every JPEG on my computer: not one in over 11,000 JPEGs embedded a thumbnail this way. Technically the package I introduce later should support these thumbnails, but since I have zero test cases to work with: I can't be certain I parse them correctly.

At this point I believe it is safe to say: these are (at best) extremely rare in the real world.

About the APP1 Block

This block is the trove of information. This includes designated (but optional) fields for everything you can imagine: width, resolution, aperture value, the encoding software name, copyright info, etc. Also it includes a pointer to a mini-JPEG file (still in the APP1 block) to serve as a thumbnail.

A Surprise Block

... but as I was exploring my test files, I found several other markers. These were often application-specific. I don't have specifications for most of these, and I didn't invest too much time in trying to figure them out. However one started to stand out: the APP13 marker.

This marker is supposedly written by Adobe. One page refers to this as the "Adobe IRB" block, where I assume (from googling) that "IRB" stands for "Image Resource Block". What's interesting about this block is that it also appears to contain a mini-JPEG. In fact: if a test image contained only 1 thumbnail, it was several times more likely to contain a thumbnail in an APP13 block than an APP1 block.



So I invested time in parsing this block, too. (If I ignored this block I'd be missing nearly 1/3 of my possible thumbnails!)

To be fair: no one collection of test cases can represent "the majority" of JPEGs in the world. It might be the case that my collection is extremely skewed, and on your computer not a single JPEG has an APP13 marker. But still: this seemed like a lead worth following.

To be safe, I wanted to check to see if other markers also existed that embedded mini-JPEG thumbnails. I added a unit test that searched all my test files for the number of occurrences of the marker 0xFFD8 (the start-of-image marker): the results indicate that I'm recording all the mini-JPEGs embedded in my test cases. (But if a thumbnail is encoded in another format -- such as a PNG -- then I may still be missing it.)

Results

Finally I put all this together in the JPEGMetaData class. You can download this class (source included) here

I casually added methods to retrieve properties and comments, too, because they were trivial to parse. Unfortunately what isn't easy to parse is: the dimensions of the actual image! (Or maybe someone reading this can cue me in to where to find this information? For now I'll continue to use my ImageSize
class to fetch this information without actually loading the image.)