Thumbnails: Formatting Thumbnails

This article discusses how to tastefully format/decorate a thumbnail. Previous articles discuss how to scale images efficiently, including bmps, jpgs and pngs, and how to extract existing thumbnails from large jpg images, but this article focuses on the question: and then what?

Context

Why do we need to decorate thumbnails? We don't need to, I guess. This is a matter of taste. For comparison, here are a few examples of what other people are doing:

	Mac OS Finder This is a set of thumbnails when I browse a folder of images in the Finder on Mac 10.7. This is one of the most complicated formatting I've found: it includes a shadow, a white border, and then a thin dark border.
	iTunes In iTunes: the thumbnails only have a light shadow. Here you might be able to make more assumptions about the content of the thumbnail: it is probably a very brightly colored image with lots of eye-catching patterns and texture. (That is: an artist specifically composed this thumbnail to fill every pixel.)
	Woot.com Woot.com doesn't bother with shadows, but they still add some very distinct borders around certain thumbnails. In this case: the borders help identify what is the "clickable area" for these thumbnails.
	Amazon These non-rectangular thumbnails from Amazon.com have no shadows or borders at all. These are also the largest thumbnails in this set of examples: they occupy so much space that they are visually distinct without any help from formatting. Microsoft.com also appears to follow this model (using large, undecorated thumbnails).

The decision regarding when and how to decorate a thumbnail might be better left to graphic designers than to software engineers, but for now my point is simply: this is an established convention that you may sometimes want to follow.

Overview

My most immediate goal is to create a file browsing UI, so I want to emulate the first example shown above: photo thumbnails in the Mac Finder. There are three basic elements in this image: the drop shadow, the border, and the thumbnail itself.

This applet demonstrates these 3 layers:

This jar (including source code) is available here.

This applet demonstrates the BasicThumbnail class and its ability to format a thumbnail image. The default options displayed are the preset options that resemble the Mac OS thumbnail (simply referred to by the static field BasicThumbnail.Aqua).

Each layer of the thumbnail (shadow, border, image) is a separate object with its own concept of padding (insets). They are painted in the order shown in the list (from left to right), so that each layer overlaps all the previous layers. To modify a layer: click that layer in the list and use the controls in the inspector.

All layers support varying amounts of curvature for the corners. A curvature of zero is a sharp (rectangular) corner, and a high curvature is a very rounded corner.

Shadow

In this model a shadow is just a series of strokes of increasing thickness. (That is: a shadow of 1 pixel, then 3 pixels, then 5 pixels, etc.) Each stroke is solid black with a very light opacity (usually less than 20 out of 255).

Since this can be several pixels thick: you have control of both the inner and outer curvature.

Border

A border is a solid-color frame. Technically this can use any color, but the UI here restricts it to a shade of gray for simplicity's sake. Like the shadow: this supports an inner and outer curvature.

Image

This layer contains the actual thumbnail. It also contains an optional background color, and one curvature control to apply rounded edges.

In the applet above: selecting the image layer lets you control the rotation. The rotation is technically a property of the entire BasicThumbnail (it applies to all layers), however it was placed in the image's inspector for convenience when modeling the GUI.

Layer Controls

The top row of the applet contains a set of buttons to add, remove, and reorder layers. Also the "<>" button displays the source code used to represent the thumbnail in the editor.

Usage & Implementation

Once a BasicThumbnail is constructed, you can simply call:

BufferedImage img = thumbnail.create( sourceImage, maxSize);

Where "sourceImage" is an Image, URL, or File, and "maxSize" is a java.awt.Dimension object. The returned thumbnail will be either the width or height of the argument dimensions, with the other dimension scaled proportionately as needed.

The actual scaling is handled abstract by the parent class Thumbnail: this is where the logic lives that accepts a file, URL or image and scales it. The subclasses (such as the BasicThumbnail) are responsible for calculating exactly what dimensions to scale the source to, and then how to render that scaled image.

If a rotation has been supplied: then a binary search is used to calculate the exact scaling factor required to fit the source image inside the thumbnail with all the specified decorations.

Conclusion

My final thumbnail formatting is not a perfect replica of Apple's formatting, but it is sufficiently close for my needs. (Also Apple can modify this formatting at any time, so maintaining an exact copy is a very ambitious ideal.) What I have now is a strong improvement over a plain (undecorated) thumbnail, so I am happy with the result.

I encountered a few thorny issues along the way; the main two both had to do with adding rotation, and subtle rendering bugs I had to hunt down. In hindsight the rotation feature may have been more trouble than it was worth, but I'm happy with the finished result.

This applet has a more carefully crafted UI than usual. This has the unfortunate side-effect of bloating the jar's file size (because it includes the UI for the angle dial, the grayscale slider, and the pixel-magnification-panel, among other things)... but I think this bloat is worth it: this applet generates copy-and-paste-able code! On a personal/selfish level: practicing UI design here is very useful, even if it comes at a cost of inconveniencing other developers with a larger jar than necessary.

Most aesthetic layouts probably only contain only 1 or 2 layers. Maybe 3. So I have mixed feelings about this applet starting out with a 5-layer demo. This complicated demo is probably the exception rather than the rule, but I wanted to demonstrate what this architecture is capable of. In most cases, though: you probably only need about 2 layers to make a subtle-but-significant improvement in appearance.

Text: Swivel Animations

I just wrapped another great year at SXSW. As expected: after 5 days of back-to-back sessions I'm exhausted, my brain is full, my feet are sore, and all the wonderful innovative ideas I have bubbling up inside won't really see the light of day at work because, well, pesky reality keeps getting in the way.

They always encourage you to tweet during SXSW sessions. Sometimes moderators actually refer to the tweets during each session to redirect the conversation or answer questions. During one session instead of presentating slides or a close up of the speaker: the projectors simply showed live tweets. But they were all at right angles to each other, and the camera swiveled around to focus on each one. I wish I knew exactly what program did that (can anyone tell me?), but I decided I wanted to achieve the same effect.

The Plan

What I want to focus on right now is the core of the animation: manipulating AffineTransforms to zoom in, out, and swivel around a large static canvas. To achieve this I started with a minimal interface:

public interface TransformAnimation {
  /**
  *
  * @param progress a float within [0, 1].
  * @return the AffineTransform for the argument.
  */
  public AffineTransform getTransform(float progress,int viewWidth,int viewHeight);
}

The piece that followed was also pretty straight-forward: the SwivelPathAnimation is the animation I was looking for. The class itself is very small, but it took a few days to develop because I was trying to develop the UI at the same time. (What good is the animation without a way to test it?)

You construct the animation with a series of Highlight objects. These highlights have a center, a width and height, and an angle of rotation. So if you think of this animation as moving a camera around a page: this controls the placement, zoom and rotation of the camera. Each highlight also keeps track of its duration. When you construct a SwivelPathAnimation you also define the default swivel/transition duration: so each highlight can have its own unique duration, but it's assumed all your transitions should be the same duration.

The path the camera takes is automatic. It is guaranteed to cross the center of each highlight, and it my sincerest hope that it will do so gracefully (without wild curves or excessive detours). This draws on several other articles relating to working with bezier shapes, including this one, possibly this one, and most definitely this one.

Deep inside the animation are controls that ration out the timing as needed. The animation doesn't stop moving when it's displaying a highlight: it just slows down to a very slow speed. Then when it's time for a transition the speed picks back up again.

Here is the applet demo of the finished product:

You can download the applet (source included) here. If the applet doesn't load (or isn't responsive enough), you can also download a sample movie file here. On some browsers/platforms the flicker/animation might be so bad you're better off running the jar as an executable instead of an applet.

The UI

This might (?) be the most complicated UI I've put together for a proof-of-concept applet on my blog so far, so I want to explain it some.

First the easy part: the animation is on the right. Once everything is loaded, just press the play button or drag the slider to see your animation in action.

The left side is the interesting part. The left panel is the overview of the entire canvas. And you can interact with it in a few different ways.

It is populated with the RSS feed from one of my favorite sites: fmylife.com. (As a result: some of the content is for mature audiences. I assume most of my readers are not especially young, though...). So when you see a spinny widget when it starts up: that is what you're waiting on. If you try to view this applet/app without internet connectivity a pre-saved copy of a feed from earlier this week is used instead.

At the bottom of the left panel are a set of controls. I recommend turning down the number of blurbs to about 5 if you want to dabble with the highlights: it makes the UI much more manageable. Now in the overview you can:

Click once to create a new highlight.

Click and drag an existing highlight to a new location.

Click and drag the corner of a highlight to resize it.

Shift+click and drag the corner of a highlight to rotate it.

Also in the controls below: you can use the "Clear" button at any time to start over.

The Canvas

Converting an RSS feed into the layout in the overview took a lot of experimenting. I ended up with a tidy class (only about 100 lines of actual code): the TextClusterPaintable.

There are a few essential building blocks I had to piece together first, though: the TextBoxPaintable breaks a java.lang.String up into a graphic representation, and the CompositePaintable is the abstract parent of the TextClusterPaintable that lets you compose different Paintables together as part of a bigger canvas. But the TextClusterPaintable is the entity that actually decides where each block of text goes, and at what angle. There's probably room for improvement in this class (it uses a kind of brute-force approach to resolve visual collisions), but it gets the job done for now.

Extra Effects

After I had that much working: I went back and looked for ways to improve the animation. The first oddity I noticed was that as you pan around the edges of the canvas: you see a lot of blank space. That is: you can clearly see where there is content and where there is not. This animation is supposed to evoke the feeling of lightly grazing over an unending stream of ideas, so seeing large blank areas off to the side is not the right model.

So to fix this I decided to tile the canvas. In the CompositePaintable I added an enum titled Tiling:

The text cluster uses the Tiling.OFFSET model to help reduce the obvious visual effect of a repeating pattern. There is still some white space (depending on how many blurbs we grabbed and how we arranged them), but the overall effect is much better.

This accentuated a growing performance problem: the text boxes are stored as giant complex java.awt.Shapes. (Because treating them as java.lang.Strings would eventually result in a non-continuous animation: the whole point of rendering Strings vs Shapes is that they are abstract and the toolkit can (and will) optimize how they are rendered. We have to use Shapes to guarantee a continuous animation as we rotate and zoom.) These are not trivial to render, and when you invoke 7 calls to paint(..) instead of 1: a minor annoyance becomes a major obstacle. To address this the CompositePaintable now checks the rotated bounds of each child Paintable against the current clipping.

Also at this point the animation was black-text-on-white-background. There's a time and place to embrace the "less is more" philosophy, but this was a little extreme. I decided to add a splash of color to the background. Also I decided the background should be at a different depth than the text itself. (That is: imagine the text is 6 inches below the camera, then the background is 4 inches below that.) This contrast of motion -- which hopefully will be so subtle most users won't notice it -- will add to the overall effect of depth and motion.

I gravitated towards my own collection of tiled images, and eventually settled on one of the more modest (and non-animated) options. The colors were a little too bold (they competed with the text), so I softened them a little. Also at first I applied it as-is -- in a vertical orientation -- but the contrast between vertical and horizontal as the animation swiveled around was too weird. Using 45 degrees looked much better. And lastly I realized: there's no reason the background can't also move. When this particular pattern slides up or down the effect is similar to the Barberpole illusion, so that was a nice touch too.

The original animation I saw at SXSW had its own special look: only the focused blurb was fully opaque -- all others were translucent. Also the background color would change every time the camera panned around. These are also nice effects, but in this iteration of the project I wanted to work with a static foreground layer. (I'd love to branch out into lots of text effects some day, and then that can be combined with this project for an even nicer effect).

Conclusion

There is a slight visual aliasing issue when the text (which is stored as a giant Shape) very slowly moves at an angled direction, but I don't know if that's something we can ever fix.

Overall this worked out as I'd hoped, and I'd like to keep dabbling with similar effects in the future. Also I'd like to point out that everything in this project (except a few generics and enums) is compatible with Java 1.4 which was written just over a decade ago (in February of 2002). My point is: why didn't we do this kind of thing 10 years ago? The technology is there: we just need to take advantage of it. (Can you tell I just came from SXSW?)

And lastly: I had to make up a lot of names to complete this project, including "SwivelPathAnimation" and "JFancyBox" and "TextClusterPaintable". If there exist more standardized names for these ideas please let me know and I'll happily refactor my code accordingly.

Text: Effects and Block Shadows

This weekend I revisited an old project: the calligraphy stroke.

I rewrote it using a handy unassuming class called the CalligraphyPathWriter. This is fewer lines of code than the previous model, pipes the data more efficiently, and includes a couple of extra functions.

It no longer only makes strokes. The same basic notion of tracing a shape to apply a calligraphy-like stroke can be applied to creating drop shadows. The only difference is a stroke straddles the path, and a drop shadow lies entirely to the side of the original shape.

The CalligraphyPathWriter constructor includes two floats to define these offsets: if you use 0 and 30: then you're effectively adding a drop shadow to your shape. If you use -5 and 5: then you're effectively creating a stroked outline of your shape.

You can also create a drop-shadow-like effect with code like this:

Graphics2D g;
Shape myShape;
...
double dx = Math.cos(angle);
double dy = Math.sin(angle);
for(int height = 0; height < 30; height++) {
   g.translate(height*dx, height*dy);
   g.fill(myShape);
   g.translate(-height*dx, -height*dy);
}

... but if your shape or java.awt.Paint are complex: this might be expensive. (Also two of the four effects shown below can't be reproduced this way.)

The CalligraphyPathWriter can reduce this down to:

GeneralPath blockShadow = new GeneralPath();
CalligraphyPathWriter writer = 
     new CalligraphyPathWriter(angle, 0, height, blockShadow);
writer.write(myShape);
g.fill(blockShadow);

The call write(myShape) iterates over the input shape and populates the blockShadow path with the silhouette of the block shadow.

Applying To Text

Here is a simple applet that creates animated text effects using these new block shadows:

You can download the applet (source included) here.

Type any text in the text field provided. (Any text is acceptable, but it should probably be 20 characters or less?) Then select any one of the buttons:

• The "Outline" button uses the Scribbler class to randomize the texture of the text, fill and shadow. A little random vertical offset is added, and lastly a light drop shadow is traced (but not filled) around each letter.
  
• The "Punch" button separately punches each word forward with an obvious drop shadow. (Also try writing a string with no spaces: then each character is punched.)
  
• The "Wave" button fills and traces the block shadow to reinforce the 3D-look.
  
• The "Explode" uses another obvious drop shadow, but each character is assigned an angle based on how far away it is from the center. (Then the height has to be divided by the sine of that angle so the baseline is constant.)

My immediate goal this weekend was just to use block shadows for something fun. In the long term I want to define an architecture similar to what I did for simple 2D-geometric transitions. This demo applet uses a very loosely defined TextEffect interface, but it needs a lot more work to support a variety of effects for different export formats.

Also in the future it would be nice to apply gradients to these block shadows. A height-based gradient effect will be very easy. An angle-based effect (which I think would be more popular) will be a little trickier, but is still manageable.

Text: Searching JTextComponents

This entry has to do with searching for a phrase in JTextComponents. There's a lot of ground to cover, so I'll start you off with the demo and then go into details. (Give this applet a few seconds to load... it has a lot of text to read.)

This jar (source included) is available here.

Text Search Bars

What is the point of a search bar? This is an interface decision more than a technical one. A search bar has several advantages over a search dialog:

1. A dialog will physically be in the wrong location. It will start out centered, which probably covers part of the text the user wants to search.
2. Providing a search bar in a fixed location will help users visually scan the interface more easily. They'll know where to look. It's a matter of identifying a fixed point vs searching for it.
3. Dialogs are usually (and perhaps originally) designed for the convenience of programmers. You can cram a program full of thousands of dialogs... but the screen real estate is unlimited. It is much harder to create a well-balanced window with just the right amount of essential interface controls without becoming too cluttered. It takes more work, and we usually don't bother.
4. Of course a modal search dialog is worse. Modal dialogs are discouraged here, among other places.

I modeled these search bars after the ones I saw in Firefox and Safari. They include:

• a search field
  
• next/previous buttons
  
• a checkbox for matching case
  
• a dismiss button
  
• a label including the occurrences of the search phrase
  
• a toggle button for "Highlight All"

The order (and presence) varies, but those are the basics. Safari automatically highlights all, and Firefox uses a toggle button. The anchor varies in both toolbars, but the components are still listed in about that order.

Text Search Dialog

Of course sometimes a dialog might be appropriate: that's up to you to decide. If searching is a feature so rarely used that it doesn't make sense to keep it in the main interface: that's justifiable. Or if your software is so versatile that searching is nothing but an auxiliary function? That's OK too.

So I included a minimal dialog. It automatically performs case insensitive, wrap-around searches. It's not modal, but it does dismiss itself when the user hits the return key.

No matter which model you use (a search dialog or a search bar), you probably want to also include typical menu shortcuts for find, find again, and find previous.

Searching Text

Most of the rest of this project was GUI work: so I was in my prime. The actual methods that do the searching/counting, though... I admit I'm pretty clueless about.

My first instinct was to convert the document into a String, and then use .indexOf() and .lastIndexOf() to search for everything. Now I have absolutely no proof for what I'm about to say, but that sure feels like a horrible approach. Especially when we have to convert the entire string to a specific case to avoid case sensitivity.

So instead I went spelunking around, and tried using the Segment class. Now the method is much hairier, but it walks through chunks of characters at a time to find the search phrase: at no time do I have an extra copy of the whole document floating around.

(Remember when we count the occurrences of the search phrase: we perform a bajillion searches, so this method needs to be relatively light.)

If anyone has any experience/suggestions on this subject, here is the code in question.

The Visuals

When you use the Safari-style search bar you're introducing two extra layers of visual effects into your JFrame:

The middle layer is known as the TextHighlightSheet. The topmost layer is a SearchHighlight. These layers have a few of things in common:

1. They are both added to the JLayeredPane.
2. They both are tied to their "parent" JTextComponent. (It's not technically their parent component as far as the Swing hierarchy is concerned, but it's useful to think of them as bound to that text component.) When AncestorListeners or ComponentListeners are notified that the TextComponent has moved around: these other layers have to immediately respond.
3. They're both animated. Or at least they can be. The middle sheet may fade in/out, and the highlight can have any number of customized animations.

Now I'll discuss each in a little more detail.

The TextHighlightSheet

This is the more clever component: it has to highlight all the occurrences of the search phrase, without looking at irrelevant hits. So it will first calculate the index that is mapped to the point (0, 0) of the viewport, and then the index that is mapped to (width, height). Then search only within those indices, and calculate the shapes of all the hits.

The hits are outlined in rectangles, and Area objects are used to join overlapping rectangles. Usually I avoid the Area class like the plague, but combined rectangles shouldn't be a hassle: they're small, and they contain only lines. Paint those rectangles white, and then paint everything around them in a translucent black (to add the overall shadow).

Finally a combination of the right clipping, translation, and a TextOnlyGraphics2D let us call parentTextComponent.paint(g) to render the text.

The Firefox search bar uses the same component as the Safari search bar. (Although in the Firefox bar you have to explicitly turn it on with the "Highlight All" button.) The Firefox sheet, however, doesn't have a translucent shadow. Also it uses different padding and colors. All of this is covered in the javadocs if you really want to customize it.

The sheet has isActive() and setActive() methods to control whether it is visible. The TextSearchBar automatically follows one of two behaviors:

1. If the "Highlight All" button is visible: then that button is the only factor controlling whether the sheet is active. This is how Firefox behaves.
   
2. If the "Highlight All" button is invisible: then the sheet is made active only when the search field has the keyboard focus. This is how Safari behaves.

SearchHighlight

This is a very different component, and is subclassed directly into the look you're aiming for. The AbstractSearchHighlight subclasses into the AquaSearchHighlight, the FirefoxSearchHighlight, and the NudgeSearchHighlight.

When constructed this component creates a BufferedImage of one particular hit of the search phrase. Then a javax.swing.Timer is activated, and this component is ready for action.

The subclasses can animate the transform and opacity of the image over the duration of the animation.

It is important that the sheet and the effect be designed to visually complement each other. For example: the highlight sheet for the Safari search bar is designed to work with the Aqua or the Nudge effects, but the Firefox effect uses smaller padding around the hit phrases.

Also the Firefox effect cheats and uses an infinite duration, so the effect never really goes away. I'm not sure exactly what should happen in this scenario if the JTextComponent is editable? (Any thoughts?) I guess the effect should go away as soon you modify the document such that that particular hit phrase is changed? This is not an issue in the Safari/Nudge models, simply because the animation will quickly dispose of itself. So this is a bug in the Firefox effect, but I'm not sure what the ideal behavior is.

Each effect is created by calling TextSearch.highlight(). This looks up the value for UIManager.get("textSearchHighlightEffect"), and instantiates the class name provided. I won't go into detail here, but the point is: you can customize this if you want. (You don't even have to extend the com.bric.plaf.AbstractSearchHighlight if you don't want to.)

Conclusion

In v1.0 of this project I just included the search bars and the effects. In v2.0 I added the middle sheet. If I get around to a v3.0: what next? (Aside from existing bugs.)

Speaking of bugs: I suppose you also need to be able to perform a revised search with every key stroke in the search field. This is how Firefox behaves. This is slightly different from hitting the enter key in the text field, because that will search for the next occurrence starting at the end of the current selection... but continuous typing really needs a search that starts at the beginning of the current selection. Another day, maybe...