Images: Reading JPEG Thumbnails

The problem: I want to create thumbnails for a folder of JPEGs. Many of these JPEGs came from a digital camera, so they're several thousand pixels wide. Loading the entire image and then scaling down to a 64x64 thumbnail sounds pretty wasteful: what else can I do? I'm 99% sure most digital cameras these days are embedding thumbnails inside the JPEG files for just this kind of problem...

Discussion

As far as I can tell: ImageIO won't read JPEG thumbnails unless you have JAI installed?

I googled "java example: ImageIO reading thumbnail" to see what I could find on the subject:

The first link was from Sun, and it said this:

3.3.4 Reading "Thumbnail" Images

Some image formats allow a small preview image (or multiple previews) to be stored alongside the main image. These "thumbnail" images are useful for identifying image files quickly, without the need to decode the entire image.

Applications can determine how many thumbnail images associated with a particular image are available by calling:

reader.getNumThumbnails(imageIndex);

If a thumbnail image is present, it can be retrieved by calling:

int thumbailIndex = 0;
BufferedImage bi;
bi = reader.readThumbnail(imageIndex, thumbnailIndex);

Right. Good. This makes sense. The problem is: it doesn't seem to really work.

The second link was much more insightful. A developer wrote:

I should have remarked that the JAI Image I/O Tools JPEG reader supports via the thumbnail method calls all thumbnails embedded in the JFIF APP0, JFXX APP0, and EXIF APP1 marker segments. Please see this javadoc for more information:

http://download.java.net/media/jai-imageio/javadoc/1.1/overview-summary.html#JPEG

I think that the only thumbnails supported by the Java SE Image I/O JPEG reader via the thumbnail method calls are those in the JFIF and JFXX marker segments. If you are unable to use JAI Image I/O Tools for some reason you could however derive the EXIF thumbnail by parsing the contents of the "unknown" node in the image metadata corresponding to the EXIF APP1 marker segment.

I think he just summed up the problem I was seeing. If I run this code (without JAI installed), I get an exception:

Iterator iterator = ImageIO.getImageReadersBySuffix("jpeg");
while(iterator.hasNext()) {
 ImageReader reader = (ImageReader)iterator.next();
 try {
  reader.setInput( ImageIO.createImageInputStream(jpeg) );
  BufferedImage thumbnail = reader.readThumbnail(0, 0);
 } catch(Exception e) {
  e.printStackTrace();
 }
}

(The exception I get is: java.lang.IndexOutOfBoundsException: No such thumbnail at com.sun.imageio.plugins.jpeg.JPEGImageReader.readThumbnail(JPEGImageReader.java:1354). Which makes sense based on what the second link said.)

Now I have nothing against JAI itself. But when I went to grab a copy just now: the jar was over 5 MB!

This is ridiculous. For widgets I present in this blog I'm not going to try to attach a 5 MB jar. (Is that even legal? I didn't look up the licensing.) Or worse: I'm not going to insist that the user separately download other packages to get my software to run well. It should run well straight out of the box.

I decided to try to extract the thumbnails myself. I did not want to write a fully-functional JPEG decoder (what are the odds I could compete with OS-specific optimized code that already exists?), but I might benefit from my own metadata retrieval.

Research

The first thing I needed was test cases. Lots of and lots of test cases. In addition to all the JPEGs lying around on my computer, I wanted some public domain images I could write unit tests against. I explored Flickr, and found a collection of images with no known copyright. I saved a few of them here.

Next I'd need to roll up my sleeves and study file specifications. My first search result pointed to this specifications PDF, but it is dated to 1992. I did more research and found that this PDF is a better fit.

A JPEG is (until the image data begins) clearly separated into discrete chunks of data. So modeled after the ZipInputStream, I wrote the JPEGMarkerInputStream to read each chunk separately.

According to the specifications: there are two types of markers that are supposed to provide all our thumbnails: APP0 and APP1.

The JAI documentation confirms this:

The [JAI] JPEG reader supports thumbnails. These may be derived from JFIF APP0, JFXX APP0, or EXIF APP1 marker segments.

About the APP0 Block

This marker was introduced in the original 1992 specification. It includes lots of basic metadata (resolution, version info), and can optionally include a thumbnail.

However none of my sample images use this marker to embed a thumbnail. I even branched out and scanned every JPEG on my computer: not one in over 11,000 JPEGs embedded a thumbnail this way. Technically the package I introduce later should support these thumbnails, but since I have zero test cases to work with: I can't be certain I parse them correctly.

At this point I believe it is safe to say: these are (at best) extremely rare in the real world.

About the APP1 Block

This block is the trove of information. This includes designated (but optional) fields for everything you can imagine: width, resolution, aperture value, the encoding software name, copyright info, etc. Also it includes a pointer to a mini-JPEG file (still in the APP1 block) to serve as a thumbnail.

A Surprise Block

... but as I was exploring my test files, I found several other markers. These were often application-specific. I don't have specifications for most of these, and I didn't invest too much time in trying to figure them out. However one started to stand out: the APP13 marker.

This marker is supposedly written by Adobe. One page refers to this as the "Adobe IRB" block, where I assume (from googling) that "IRB" stands for "Image Resource Block". What's interesting about this block is that it also appears to contain a mini-JPEG. In fact: if a test image contained only 1 thumbnail, it was several times more likely to contain a thumbnail in an APP13 block than an APP1 block.



So I invested time in parsing this block, too. (If I ignored this block I'd be missing nearly 1/3 of my possible thumbnails!)

To be fair: no one collection of test cases can represent "the majority" of JPEGs in the world. It might be the case that my collection is extremely skewed, and on your computer not a single JPEG has an APP13 marker. But still: this seemed like a lead worth following.

To be safe, I wanted to check to see if other markers also existed that embedded mini-JPEG thumbnails. I added a unit test that searched all my test files for the number of occurrences of the marker 0xFFD8 (the start-of-image marker): the results indicate that I'm recording all the mini-JPEGs embedded in my test cases. (But if a thumbnail is encoded in another format -- such as a PNG -- then I may still be missing it.)

Results

Finally I put all this together in the JPEGMetaData class. You can download this class (source included) here

I casually added methods to retrieve properties and comments, too, because they were trivial to parse. Unfortunately what isn't easy to parse is: the dimensions of the actual image! (Or maybe someone reading this can cue me in to where to find this information? For now I'll continue to use my ImageSize
class to fetch this information without actually loading the image.)

Shapes: Parametric Equations (and Spirals!)

This article is split up in 2 parts:

1. Converting a parametric graph into a PathIterator
2. An applet demoing a Spiral2D class

Introduction

A coworker recently wanted to paint a spiral. He found a programming book that included some sample code to generate spirals, but there were two catches:

For a small company like ours: use of that code required a $500 license fee.

Even then: the spiral he rendered was basically a polyline. (That is: it was rendered as a series of lines, and had no curvature.)

Both of these restrictions made that implementation a bad idea, so I decided to flesh out a better model. What I ended up with is a model that helps convert parametric equations to java.awt.Shapes, and a subclass that caters specifically to spirals.

Parametric Equations and PathIterators

Suppose you have a parametric graph expressed as x(t) and y(t): how do you draw that in Java?

At its most basic level: drawing a shape in Java requires a java.awt.geom.PathIterator.

Preferably we'll then package this iterator in a java.awt.Shape, but with just an iterator you can use the following code to create a fillable/drawable shape:

GeneralPath path = new GeneralPath();
path.append(myPathIterator, false);

A PathIterator breaks a path up into a series of bezier segments: so our first task is to bridge the gap between a parametric expression and bezier curve.

Mathematically this is trivial to compute, but it may require a couple of pages of scrap paper to derive/understand everything.

Converting Parametric Equations into Bezier Curves

The most flexible bezier segment type that Java supports is the cubic segment, so that's the segment type we want to focus on. A cubic bezier segment has 4 control points: two for each end of the segment, and two to define the tangent slopes at each endpoint. Our goal is break our parametric function into a series of cubic bezier segments. That is: we're starting out with the parametric functions x(t) and y(t), and we have to extract from that a series of segments, where each segment is defined by 4 points.

Before we dig in to the math here, we need to make an important distinction: we're starting with a parametric equation that uses a variable t. We're going to convert that into a series of adjacent bezier curves, but a bezier curve is also a parametric equation. So I'm going to refer to those segments as using t_bezier for their t-values. So we have "t" (for the master equation) and "t_bezier" (for the individual segments used to replicate the master equation).

If we want our bezier curve to fit a range of t-values from [ta, tb], then we know that:

x_bezier(0) = x(ta);
dx_bezier(0)/dt = dx(ta)/dt;
x_bezier(1) = x(tb);
dx_bezier(1)/dt = dx(tb)/dt;

This only matches our bezier curve to the endpoints. The closer those endpoints are: the better a match this piecewise solution is going to give us. (If they are infinitesimally close: they will be a perfect match.) This is discussed more below. For now: assume that "ta" and "tb" are reasonably spaced.

Since a cubic bezier curve can be expressed in terms of a cubic polynomial, we also know that:

x_bezier(t_bezier) = ax*(t_bezier^3) + bx*(t_bezier^2) + cx*(t_bezier) + dx;
x_bezier(0) = ax*(0^3) + bx*(0^2) + cx*(0) + dx = dx;
dx_bezier(0) = 3*ax*(0^2) + 2*bx*(0) + cx = cx;
x_bezier(1) = ax*(1^3) + bx*(1^2) + cx*(1) + dx = ax + bx + cx + dx;
dx_bezier(1) = 3*ax*(1^2) + 2*bx*(1) + cx = 3*ax + 2*bx + cx;

So it follows that:

x(ta) = dx;
dx(ta)/dt = cx;
x(tb) = ax + bx + cx + dx;
dx(tb)/dt = 3*ax + 2*bx + cx;

So now we've mapped our source equation to the polynomial coefficients we need. But a cubic bezier curve is traditionally expressed as a series of 4 control points -- not as 4 coefficients. The article "Shapes: an Introduction" explains most of the math needed to make this conversion. Eventually if we solve for the control points in terms of the original parametric equation, then we'll end up with:

end1 = x(ta);
end2 = x(tb);
ctrl1 = (dx(ta)/dt+3*x(ta))/3;
ctrl2 = (3*x(tb)-dx(tb)/dt)/3;

That is all the information we need to define a cubic bezier segment that replicates our original graph from [ta, tb]. (We also have to apply exactly the same logic to the y-equations/coefficients, but the end result is the same system using "x" instead of "y".)

This logic has been built in to the abstract ParametricPathIterator, so there is a PathIterator that breaks a parametric equation up into thousands of smaller bezier segments as needed. The most crucial piece of information we're lacking now is: what are the values for "ta" and "tb"? If they are too widely spaced apart: then the piecewise segments will not match the graph well. If they are too close together: then the shape becomes unnecessarily complex. This spacing is the responsibility of the subclass, because the varying complexity of the parametric graph we're plotting will affect the interval between t values.

Spirals

Now that we have a model to convert parametric graphs to PathIterators: creating a Spiral2D should be easy. First we have to identify the parametric equations involved. Traditionally a spiral is expressed as:

x(t) = centerX+coilGap*t*cos(2*pi*t)
y(t) = centerY+coilGap*t*sin(2*pi*t)

After working with this format, though, I quickly realized I wanted at least two other variables:

angularOffset: to affect the initial angle.

coilOffset: to affect the coil offset.

My revised definition of a spiral looks like this:

x(t) = centerX+coilGap*(t + coilOffset)*cos(2*pi*t + angularOffset)
y(t) = centerY+coilGap*(t + coilOffset)*sin(2*pi*t + angularOffset)

We'll also need to know the derivative. (Of course we could also just numerically approximate the derivative, but let's calculate it correctly since this article is supposed to be instructive...) After refreshing my memory regarding the product rule, I ended up with these equations:

dx/dt = coilGap*cos(2*pi*t+angularOffset)-2*pi*coilGap*(t + coilOffset)*sin(2*pi*t+angularOffset)
dy/dt = coilGap*sin(2*pi*t+angularOffset)+2*pi*coilGap*(t + coilOffset)*cos(2*pi*t+angularOffset)

The Spiral2DPathIterator is the finished product. It extends the ParametricPathIterator and implements the spiral equations above. Each coil (from 0 to 360 degrees) is broken up into 8 bezier segments; this is not a magic number -- it is simply what empirically looked "good". Also (in keeping with Java conventions) there is a separate Spiral2D object that implements the java.awt.Shape interface for your convenience.

Here is an applet demonstrating the final implementation:

This applet (source included) is available here.

p.s. Try clicking in the applet to define the endpoint for the spiral. (Also try using the shift key.)

Shapes: An Introduction

I try to sprinkle in some fun open-source code snippets in most of my articles -- but this article is different. This article is simply an overview of the java.awt.Shape object. Several other articles build on (or assume knowledge of) what I cover here.

What is a java.awt.Shape?

When I refer to a "shape" in Java, I'm probably referring to the java.awt.Shape interface. The most interesting method in this class is getPathIterator(AffineTransform) -- the results of all the other methods can be derived from this method.

OK, so what is a java.awt.geom.PathIterator?

This is the heart of shape information in Java. This iterator is similar to the human gesture of dragging a pen from one point in space to another. A path is a series of segments, and there are 5 basic types of segments:

1. SEG_MOVETO: this is the initial position of a path. Each path must begin with this segment. If a MOVETO segment is not followed by any other segments, though: that path is considered empty. You might get a dot if you called Graphics2D.draw(shape), but you won't get anything if you call Graphics2D.fill(shape).
   
2. SEG_LINETO: this segment requires an x-coordinate and a y-coordinate. This connects a line from the last path location to the point indicated.
   
3. SEG_QUADTO: this segment requires two points: the end point and a control point (more on control points later.)
   
4. SEG_CUBICTO: this segment requires three points: two control points and an end point.
   
5. SEG_CLOSE: this is an optional segment that connects the previous segment to the SEG_MOVETO that began this shape. Even if you draw 4 sides of a square, you need to add a SEG_CLOSE for some java.awt.Strokes to correctly render the tips of all corners.

A single PathIterator can contain any number of MOVETO's. Each MOVETO is equivalent to lifting your pen off a piece of paper and repositioning it to start a new segment.

Bezier Control Points

Quadratic and cubic segments use control points. These are ingenious points that are used to tug an existing path in certain directions. There's a great wikipedia page that describes this concept in detail (with diagrams and everything). What I want to do is focus on the math. I'll break this up into the quadratic and the cubic cases:

Quadratic Parametric Equations

The javadocs are the place to start when you want to study the math here. They describe a quadratic segment as:

P(t) = B(2,0)*CP + B(2,1)*P1 + B(2,2)*P2
       0 <= t <= 1

B(n,m) = mth coefficient of nth degree Bernstein polynomial
       = C(n,m) * t^(m) * (1 - t)^(n-m)
C(n,m) = Combinations of n things, taken m at a time
       = n! / (m! * (n-m)!)

This sounds a lot more complex than it actually is if you're willing to break it down. We can get rid of C(n,m):

B(n,m) = mth coefficient of nth degree Bernstein polynomial
       = [ ( n! / (m! * (n-m)!) ) * t^(m) * (1 - t)^(n-m) ]

And then expand the first equation:

P(t) = [ ( 2! / (0! * (2-0)!) ) * t^(0) * (1 - t)^(2-0) ]* CP +
       [ ( 2! / (1! * (2-1)!) ) * t^(1) * (1 - t)^(2-1) ] * P1 +
       [ ( 2! / (2! * (2-2)!) ) * t^(2) * (1 - t)^(2-2) ] * P2

Simplify everything:

P(t) = (1 - 2*t + t^2) * CP +
       [ (2*t - 2*t^2) ] * P1 +
       [ t^(2) ] * P2

This is still not really useful to me, though. What I want is an expression in terms of t:

P(t) = (CP-2*P1+P2)*(t^2) +
       (-2*CP+2*P1)*t +
       CP

In my code this usually manifests itself something like this:

double ax = lastX-2*x1+x2;
double bx = -2*lastX+2*x1;
double cx = lastX;

Likewise you can replace "x" with "y" to get the y coefficients. From here you can treat your bezier segment as a good ole parametric equation. You can always spot-check your math (in both the quadratic and cubic case below) by checking the values at t=0 and t=1. At t=0 you get lastX -- which is the right starting point -- and at t=1 you simply add all those terms together -- and the terms cancel to x2.

I hardly ever use quadratic segments, though. If the user needs more than lines, the cubic segment gives you a lot more flexibility...

Cubic Parametric Equations

Again, we'll start by looking at the javadocs for a cubic curve:

P(t) = B(3,0)*CP + B(3,1)*P1 + B(3,2)*P2 + B(3,3)*P3
0 <= t <= 1

B(n,m) = mth coefficient of nth degree Bernstein polynomial
       = C(n,m) * t^(m) * (1 - t)^(n-m)
C(n,m) = Combinations of n things, taken m at a time
       = n! / (m! * (n-m)!)

Expand that:

P(t) = [ 3! / (0! * (3-0)!) * t^(0) * (1 - t)^(3-0) ] * CP +
       [ 3! / (1! * (3-1)!) * t^(1) * (1 - t)^(3-1) ] * P1 +
       [ 3! / (2! * (3-2)!) * t^(2) * (1 - t)^(3-2) ] * P2 +
       [ 3! / (3! * (3-3)!) * t^(3) * (1 - t)^(3-3) ] * P3 

Now simplify:

P(t) = [ (1 - t)^3 ] * CP +
       [ 3 * t^(1) * (1 - t)^2 ] * P1 +
       [ 3 * t^(2) * (1 - t)^1 ] * P2 +
       [ t^(3) ] * P3 

Expand the exponents:

P(t) = [ (1 - 3*t + 3*t^2 - t^3) ] * CP +
       [ (3*t - 6*t^2 + 3*t^3) ] * P1 +
       [ (3*t^2 - 3*t^3)^1 ] * P2 +
       [ t^(3) ] * P3 

But what we really want is the equation in terms of t:
Lastly:

P(t) = [-CP+3*P1-3*P2+P3]*(t^3) + 
       [3*CP-6*P1+3*P2]*(t^2) +
       [-3*CP+3P1]*t +
       CP

So now we have the cubic equation expressing this curve in terms of T. Here is how I'd usually integrate this into my code:

double ax = -lastX+3*x1-3*x2+x3;
double bx = 3*lastX-6*x1+3*x2;
double cx = -3*lastX+3*x1;
double dx = lastX;

... and you can likewise define ay, by, cy and dy.

This concludes the introduction; look up other articles that are prefaced with "Shape" to see possible uses for this.

Colors: A Good GUI for Selecting Colors

This article explores the question: what is a good GUI for picking a color?

Dialogs

The first instinct of many developers will be to use a dialog. And there is a dialog that comes standard with Java that does the trick. It takes one line to invoke:

Color newColor = JColorChooser.showDialog(component, title, initialColor)



I have several complaints about this dialog:

• The default tab is poorly designed. Look at all that wasted vertical space! And I don't like the strong shadows that surrounds each cell.
  
• The colorful components are not keyboard accessible. You can navigate the dialog using the keyboard if you eventually tab to a text field or spinner: but then you have to choose your color by numbers. Do you like choosing a color by numbers? It seems more natural to choose a color from a set of colors.
  
• The dialog uses tabs. This is perhaps acceptable on the grounds that this is intended to be a universal component. But a better design would only present one suitable model. The user's goal here is to select a color. In this dialog they can get distracted with all the different components. Depending on how you count them, I see at least 23 different components the user can click here: this is overdoing it.

This prompted me to create a new dialog. It's default view features a color wheel. You can navigate it with the keyboard. There are no tabs, but it does offer subtle controls to change the view. This is modeled after what Photoshop uses, and they must know a thing or two about how professional artists want to choose a color, right?

However most users are not professional artists, so this is still overkill. Also when possible it is better to avoid dialogs. The book About Face makes these comments about dialogs:

"A dialog box is another room; have a good reason to go there.

... Build functions into the window where they are used.

... Putting functions in a dialog box emphasizes their separateness from the main task."

Palettes

Instead I would like to focus on palettes. A grid of colors. This could be presented in a small JWindow, or it could be presented in a JPopupMenu.

Here size is everything. The key difference in this concept and the dialogs mentioned before is that this is small. In nuanced jargon: this is a "transient", and the previous two dialogs are more "sovereign". This is one component - not several.

Here is an applet that provides several consecutive drafts of different palettes:

This applet (source included) is available here.

Here is a brief history of each draft:

1. This was my first reaction to the words "pick a color". Every pixel counts. You have thousands of colors here to choose from. Not every color is represented, but you get light and dark shades that really stand out. (Compared to less saturated, muddy shades.)
   
2. But quickly someone pointed out: this doesn't have black and white! Those are very important colors. As are shades of gray. So the second draft included a special column for white/black/gray.
   
3. We had this model deployed in a desktop application for a while. Eventually the feedback came back to us: it's too many options. It was nearly impossible to ever pick the same color.
   

   The previous models were based on my instincts as a developer: if I give you more choices, that gives you more control. More control is better, right? But in fact people don't usually like too many choices. Here is a quote from The Power of Persuasion:
   

   "The greater the number of choices, the happier the consumer - so it's assumed.

   In a recent series of experiments ... a group of Columbia University chocolate lovers were given a choice between six different flavors of Godiva chocolates. A second group was asked to select between thirty different flavors. Subjects given the extensive flavor choices rated their selection as less tasty, less satisfying and less enjoyable than did the limited-choice group. The thirty-flavor subjects expressed more regrets about their selection and were less likely to choose chocolate as payment for participating in the experiment."

   In other words: suppose a user goes off in search of the color green. But then they quickly see that there are 300 different shades to choose from! This is a source of stress to the user. Give them a few tasteful options and they'll actually be happier.
4. Then we designed a product for a younger audience. We wanted to thin the color pool down even more. Also a common complaint was that users needed brown. So we added a second special column for brown.
5. But for graphic designers and other artists: they really might want more than 60 colors. This model uses keyboard modifiers to let the user get thousands of more choices. There are modifiers for "flesh tones" and "earth tones", as well as to toggle to a continuous spectrum of colors.
6. Another way to give more options is to add scrollbars. A color is generally considered to have three dimensions (RGB, HSB, RYB, HSL, etc.). A rectangle will only present two of these dimensions. If you add a scrollbar, though: you can represent another dimension. Also you can add a scrollbar for opacity. And because I still like the modifier idea: you can press a modifier key to toggle between a continuous spectrum and a grid. In theory this model offers every color conceivable to the user. (In practice it is subject to rounding, though.)
7. But the HSB model is not very aesthetic. Rob Camick has a nice article about using the hue-saturation-luminance model instead. Isn't it just all-around better looking?
8. Lastly: yellow. In HSB or HSL: the hue is still abstractly based on an RGB model (that is: the primary colors of light). The problem here is if you split the RGB model into 6ths, you'll get: red, yellow, green, cyan, blue, purple. What about orange? You'll have to split the model into 12ths to see orange: it's a tiny sliver in an RGB-based spectrum. Instead I added a special class called HueDistribution, and I use it to give each of these colors equal weight: red, orange, yellow, green, cyan, blue, purple.

You'll notice in the applet these drafts all have funny names: the list presents them as ColorPaletteUIs. The component shown is a com.bric.swing.ColorPalette, and it can be configured with several different look-and-feels.

Also the controls at the top of the applet are client properties that affect the look of the UI. They are purely aesthetic - they make no functional difference. Personally I hate selecting a color from uniform cells due to the contrast effect of placing the cells side-by-side. These options provide a stylized way to break up that effect.

I'd like to add this sidenote: these were intended for (and tested against) use in desktop apps: not applets. I'd recommend downloading and running these jars as applications to really observe their behavior. So far my limited testing shows these bugs:

• On Vista in IE the alt modifier doesn't work. (Well, it toggles IE's menu bar... so it is technically working...)
  
• On OpenSUSE in Firefox the popup color well (shown below) doesn't work.

... there are probably many more. But if these bugs only reproduce in applet form: I'm not too worried about them.

Integrating Into A Window

Almost done. But not quite. Where should the ColorPalette component live? It is possible to put in in your main window, but you would only want to do that if your user constantly needed to select a color. In a painting program, for example.

Or I'm developing a graphics application where I put two ColorPalettes side-by-side: one contains the usual spectrum of colors, and the other contains translucent shades of white/black. You might also want to provide a separate set of themed colors (if you were working on, say, a PowerPoint-like program). Or a set of recently-used colors.

However in several applications you can't devote that much window space to the act of choosing a color. Consider iWeb, for example: for your text, shadows, and other effects you need to choose a color, but the inspector is already crammed full of useful odds and ends

In this case what I'd want is a ColorWell. This is about the height of a JLabel (and the width is arbitrary). I gave it an Aqua-ish border to blend in on my Mac.

In order to keep it abstract I defined 4 client properties that are associated with an ActionListener. By default these are what these actions do:

1. downKeyAction: displays a ColorPalette below the ColorWell.
   
2. spaceKeyAction: provides a ColorPicker dialog for your advanced user.
   
3. singleClickAction: same as downKeyAction
   
4. doubleClickAction: same as spaceKeyAction

So in conclusion: here is applet that contains everything: the ColorWell, the ColorPalette, and the ColorPicker:

This applet (source included) is available here.

Survey time! Please take a minute to comment.

This time last year I merged all my blog projects into one codebase, and I moved that codebase to a dev.java.net project. I couldn't be happier: this was a great move. It's backed up, it's easy to check out the workspace on other machines, etc.

Also I automated the process of building/updating the jars: one java app called BlogUpdater combs through the workspace and creates a jar for every java file with a main() method. As more and more projects share small common files this really helped me keep things up-to-date.

Help Me Help You

Now it's that time of year again where I want to step back and look at how I do what I do. What can be improved? Both for me and for you. Any feedback is welcome. Feel free to email me or post it here as a comment.

Specific conversation starters could -- but don't have to -- include:

Currently I list every blog article in a list on the left. Is this a good format? There are over 40 things in this list. By this time next year: there may be 50 or 60? Here is a similar repository that follows a very different format. Also Ken's blog is much less cluttered. Thoughts?

What subjects would you like to hear about? Writing these things up takes time... if I can cater to what you're interested in it will be a win-win for both of us.

In a given project: what do you want to hear about? I don't have a fixed template: so far I've just rambled about whatever I think is interesting. This might be performance, how to render something, architecture, etc.

On a bigger, more abstract scale: does anyone have any suggestions for other ways to focus this time & energy? Maybe I could join forces with other bloggers? (Who did you have in mind?) Or I could work on specific open-source java projects my skill set might lend itself to? etc. If a great suggestion comes up: I might just freeze this blog the way it is and pick up that other project.

Does it bother anyone that I don't have a solid versioning system? (That is: if you use the ColorPicker project, there is no version number associated with it.)

In the articles would you like more code samples? More screenshots? More diagrams?

Does anything about my programming style prevent you from integrating my jars into your own projects? Anything in the java code that you'd like to see changed? Don't be shy!

I recently sketched out a list of big changes I wanted to make to my blog and its presentation; but then I decided it might be much more useful to just ask you, my readers, what you think first...

p.s. if you like things just the way they are, and would like to see more of "the same" articles: please say that too. The only way I'll gauge the majority opinion is if you speak up. :)

Thanks for reading, and thanks for the 9,000+ hits I've received this year. (Give or take a few thousand hits that might just be search engine bots?)

Text: Prompts and Search Fields

Here's what I want to replicate:

The concept of a text field is nothing new, but this undertaking requires two other components:

1. A TextFieldUI with rounded corners, and a search icon.
2. Prompting: gray text prompting the user with instructions or other useful information.

Apple's Solution

It's elegant. It's simple:

textField.putClientProperty( "JTextField.variant", "search");
textField.putClientProperty( "JTextField.Search.Prompt", prompt);

The first line is well documented here. The second line is only casually mentioned here.

It's a great solution -- if you're only trying to support Macs. What I want is a cross-platform solution. This is the same problem that triggered my new ButtonUI's, and an indeterminate progress widget. If I may get on my soap box for a minute: some of the client properties Apple's Java team implements are really not a good use of their time. To be fair: some properties they implement are great (window opacity, types of title bars, window decorations, etc). Meanwhile there are JVM-crashing bugs, accessibility bugs, etc, that Apple's team can address but I cannot... so why implement properties like this? (end of rant.)

Creating a RoundTextFieldUI

So a pure-Java solution requires making a new TextFieldUI. This wasn't so hard, if you extend the BasicTextFieldUI. The key was figuring out I had to override the getVisibleEditorRect() method. This controls the rectangle the text appears in, so I can offset everything to draw the rounded rectangle.

Also by overriding modelToView() I was able to tweak the caret height (the x-position and width I didn't change).

Here is the class I came up with. It includes a client property "useSearchIcon" to render a magnifying glass icon.

That part was surprisingly simple. Next I wanted to deal with prompting.

Using XSwingX

This is a little project hosted at code.google.com. It has prompting built-in, and a very simple programming interface. But it was surprisingly tricky to put a demo together. It requires the SwingX package. But not the latest version: there are some incompatibilities in the latest version of SwingX integrated with XSwingX. We need version 0.9.2.

Here is a jar that includes all the necessary classes to get the job done. But if you start counting: that comes to 32 classes. Also if you start looking at the source code: the word "hack" keeps coming up. It doesn't seem like a great solution.

My Own Approach

I wanted to try something simple. Something done in one, maybe two classes. My first thought was to extend JTextField, and shuffle the color/text based on when the text field has the focus:

public class PromptTextField extends JTextField {
       static FocusListener focusListener = new FocusListener() {
             public void focusGained(FocusEvent e) {
                   PromptTextField src = (PromptTextField)e.getSource();
                   if(src.state==STATE_PROMPT) {
                         src.setState(STATE_NORMAL);
                   }
             }
  
             public void focusLost(FocusEvent e) {
                   PromptTextField src = (PromptTextField)e.getSource();
                   if(src.getText().length()==0)
                         src.setState(STATE_PROMPT);
             }
       };
       private static final int STATE_UNDEFINED = 0;
       private static final int STATE_NORMAL = 1;
       private static final int STATE_PROMPT = 2;
 
       private Color normalColor = SystemColor.textText;
       protected Color promptColor = Color.gray;
       protected String promptText;
       private int state = STATE_UNDEFINED;

       public PromptTextField(String text, String prompt, int columns) {
             super(text, columns);
             promptText = prompt;
             normalColor = getForeground();
             addFocusListener(focusListener);
             if(text.length()==0) {
                   setState(STATE_PROMPT);
             } else {
                   setState(STATE_NORMAL);
             }
       }
 
       public String getText() {
             if(state==STATE_PROMPT) {
                   return "";
             }
             return super.getText();
       }
 
       private void setState(int s) {
             if(s==state)
                   return;
             state = s;
             if(s==STATE_PROMPT) {
                   if(state==STATE_NORMAL && getText().length()>0)
                         throw new IllegalArgumentException("the state should not be set to STATE_PROMPT if there is text already in the text field (\""+getText()+"\")");
                   super.setForeground(promptColor);
                   super.setText(promptText);
             } else if(s==STATE_NORMAL) {
                   super.setForeground(normalColor);
                   super.setText("");
             }
       }
 
       public void setText(String s) {
             if(state==STATE_UNDEFINED) {
                   super.setText(s);
                   return;
             }
             if(s==null) s = "";
             if(hasFocus() || s.length()>0) {
                   state = STATE_NORMAL;
                   super.setForeground(normalColor);
                   super.setText(s);
             } else if(s.length()==0) {
                   state = STATE_PROMPT;
                   super.setForeground(promptColor);
                   super.setText(promptText);
             }
       }
}

I was impressed that, well, that this worked. It's very minimalist. However it's also not very good: it's an awkward hack that overrides the getText() and setText() methods. This means the getText() method might reflect something that has nothing in common with the data stored in getDocument().

Rob Camick's Approach

Who is Rob Camick? I don't know. But now that I found this article I'm going to pay closer attention. He talks about another model to achieve prompting: add a component over the text field.

It's simple. It's not a "hack," it's just plain old clever.

There's a problem, though: text placement. In my case -- because I'm using a TextFieldUI of my own creation -- it'll be hard to line up the prompted text exactly over the existing text.

... unless I use a TextOnlyGraphics2D and the same type of TextFieldUI.

public class TextFieldPrompt extends JTextField {
     public TextFieldPrompt(JTextField parent,
         Color promptColor,
         String promptText) {
         super(promptText);
       
         if(promptColor==null)
             promptColor = Color.gray;
       
         parent.add(this);
       
         setFocusable(false);
         setEditable(false);
         setForeground(promptColor);
         setOpaque(false);
       
         ... install listeners & UI ...
     }
   
   
     public void paint(Graphics g) {
         Graphics2D g2 = (Graphics2D)g;
         g2 = new TextOnlyGraphics2D( g2, null );
         super.paint( g2 );
     }
   
     public boolean contains(int x, int y) {
         return false;
     }
}

Conclusion

Here is an applet demonstrating the final model:


This jar (source included) is available here.

Text: Searching JTextComponents

This entry has to do with searching for a phrase in JTextComponents. There's a lot of ground to cover, so I'll start you off with the demo and then go into details. (Give this applet a few seconds to load... it has a lot of text to read.)

This jar (source included) is available here.

Text Search Bars

What is the point of a search bar? This is an interface decision more than a technical one. A search bar has several advantages over a search dialog:

1. A dialog will physically be in the wrong location. It will start out centered, which probably covers part of the text the user wants to search.
2. Providing a search bar in a fixed location will help users visually scan the interface more easily. They'll know where to look. It's a matter of identifying a fixed point vs searching for it.
3. Dialogs are usually (and perhaps originally) designed for the convenience of programmers. You can cram a program full of thousands of dialogs... but the screen real estate is unlimited. It is much harder to create a well-balanced window with just the right amount of essential interface controls without becoming too cluttered. It takes more work, and we usually don't bother.
4. Of course a modal search dialog is worse. Modal dialogs are discouraged here, among other places.

I modeled these search bars after the ones I saw in Firefox and Safari. They include:

• a search field
  
• next/previous buttons
  
• a checkbox for matching case
  
• a dismiss button
  
• a label including the occurrences of the search phrase
  
• a toggle button for "Highlight All"

The order (and presence) varies, but those are the basics. Safari automatically highlights all, and Firefox uses a toggle button. The anchor varies in both toolbars, but the components are still listed in about that order.

Text Search Dialog

Of course sometimes a dialog might be appropriate: that's up to you to decide. If searching is a feature so rarely used that it doesn't make sense to keep it in the main interface: that's justifiable. Or if your software is so versatile that searching is nothing but an auxiliary function? That's OK too.

So I included a minimal dialog. It automatically performs case insensitive, wrap-around searches. It's not modal, but it does dismiss itself when the user hits the return key.

No matter which model you use (a search dialog or a search bar), you probably want to also include typical menu shortcuts for find, find again, and find previous.

Searching Text

Most of the rest of this project was GUI work: so I was in my prime. The actual methods that do the searching/counting, though... I admit I'm pretty clueless about.

My first instinct was to convert the document into a String, and then use .indexOf() and .lastIndexOf() to search for everything. Now I have absolutely no proof for what I'm about to say, but that sure feels like a horrible approach. Especially when we have to convert the entire string to a specific case to avoid case sensitivity.

So instead I went spelunking around, and tried using the Segment class. Now the method is much hairier, but it walks through chunks of characters at a time to find the search phrase: at no time do I have an extra copy of the whole document floating around.

(Remember when we count the occurrences of the search phrase: we perform a bajillion searches, so this method needs to be relatively light.)

If anyone has any experience/suggestions on this subject, here is the code in question.

The Visuals

When you use the Safari-style search bar you're introducing two extra layers of visual effects into your JFrame:

The middle layer is known as the TextHighlightSheet. The topmost layer is a SearchHighlight. These layers have a few of things in common:

1. They are both added to the JLayeredPane.
2. They both are tied to their "parent" JTextComponent. (It's not technically their parent component as far as the Swing hierarchy is concerned, but it's useful to think of them as bound to that text component.) When AncestorListeners or ComponentListeners are notified that the TextComponent has moved around: these other layers have to immediately respond.
3. They're both animated. Or at least they can be. The middle sheet may fade in/out, and the highlight can have any number of customized animations.

Now I'll discuss each in a little more detail.

The TextHighlightSheet

This is the more clever component: it has to highlight all the occurrences of the search phrase, without looking at irrelevant hits. So it will first calculate the index that is mapped to the point (0, 0) of the viewport, and then the index that is mapped to (width, height). Then search only within those indices, and calculate the shapes of all the hits.

The hits are outlined in rectangles, and Area objects are used to join overlapping rectangles. Usually I avoid the Area class like the plague, but combined rectangles shouldn't be a hassle: they're small, and they contain only lines. Paint those rectangles white, and then paint everything around them in a translucent black (to add the overall shadow).

Finally a combination of the right clipping, translation, and a TextOnlyGraphics2D let us call parentTextComponent.paint(g) to render the text.

The Firefox search bar uses the same component as the Safari search bar. (Although in the Firefox bar you have to explicitly turn it on with the "Highlight All" button.) The Firefox sheet, however, doesn't have a translucent shadow. Also it uses different padding and colors. All of this is covered in the javadocs if you really want to customize it.

The sheet has isActive() and setActive() methods to control whether it is visible. The TextSearchBar automatically follows one of two behaviors:

1. If the "Highlight All" button is visible: then that button is the only factor controlling whether the sheet is active. This is how Firefox behaves.
   
2. If the "Highlight All" button is invisible: then the sheet is made active only when the search field has the keyboard focus. This is how Safari behaves.

SearchHighlight

This is a very different component, and is subclassed directly into the look you're aiming for. The AbstractSearchHighlight subclasses into the AquaSearchHighlight, the FirefoxSearchHighlight, and the NudgeSearchHighlight.

When constructed this component creates a BufferedImage of one particular hit of the search phrase. Then a javax.swing.Timer is activated, and this component is ready for action.

The subclasses can animate the transform and opacity of the image over the duration of the animation.

It is important that the sheet and the effect be designed to visually complement each other. For example: the highlight sheet for the Safari search bar is designed to work with the Aqua or the Nudge effects, but the Firefox effect uses smaller padding around the hit phrases.

Also the Firefox effect cheats and uses an infinite duration, so the effect never really goes away. I'm not sure exactly what should happen in this scenario if the JTextComponent is editable? (Any thoughts?) I guess the effect should go away as soon you modify the document such that that particular hit phrase is changed? This is not an issue in the Safari/Nudge models, simply because the animation will quickly dispose of itself. So this is a bug in the Firefox effect, but I'm not sure what the ideal behavior is.

Each effect is created by calling TextSearch.highlight(). This looks up the value for UIManager.get("textSearchHighlightEffect"), and instantiates the class name provided. I won't go into detail here, but the point is: you can customize this if you want. (You don't even have to extend the com.bric.plaf.AbstractSearchHighlight if you don't want to.)

Conclusion

In v1.0 of this project I just included the search bars and the effects. In v2.0 I added the middle sheet. If I get around to a v3.0: what next? (Aside from existing bugs.)

Speaking of bugs: I suppose you also need to be able to perform a revised search with every key stroke in the search field. This is how Firefox behaves. This is slightly different from hitting the enter key in the text field, because that will search for the next occurrence starting at the end of the current selection... but continuous typing really needs a search that starts at the beginning of the current selection. Another day, maybe...