Survey time! Please take a minute to comment.

This time last year I merged all my blog projects into one codebase, and I moved that codebase to a dev.java.net project. I couldn't be happier: this was a great move. It's backed up, it's easy to check out the workspace on other machines, etc.

Also I automated the process of building/updating the jars: one java app called BlogUpdater combs through the workspace and creates a jar for every java file with a main() method. As more and more projects share small common files this really helped me keep things up-to-date.

Help Me Help You

Now it's that time of year again where I want to step back and look at how I do what I do. What can be improved? Both for me and for you. Any feedback is welcome. Feel free to email me or post it here as a comment.

Specific conversation starters could -- but don't have to -- include:

Currently I list every blog article in a list on the left. Is this a good format? There are over 40 things in this list. By this time next year: there may be 50 or 60? Here is a similar repository that follows a very different format. Also Ken's blog is much less cluttered. Thoughts?

What subjects would you like to hear about? Writing these things up takes time... if I can cater to what you're interested in it will be a win-win for both of us.

In a given project: what do you want to hear about? I don't have a fixed template: so far I've just rambled about whatever I think is interesting. This might be performance, how to render something, architecture, etc.

On a bigger, more abstract scale: does anyone have any suggestions for other ways to focus this time & energy? Maybe I could join forces with other bloggers? (Who did you have in mind?) Or I could work on specific open-source java projects my skill set might lend itself to? etc. If a great suggestion comes up: I might just freeze this blog the way it is and pick up that other project.

Does it bother anyone that I don't have a solid versioning system? (That is: if you use the ColorPicker project, there is no version number associated with it.)

In the articles would you like more code samples? More screenshots? More diagrams?

Does anything about my programming style prevent you from integrating my jars into your own projects? Anything in the java code that you'd like to see changed? Don't be shy!

I recently sketched out a list of big changes I wanted to make to my blog and its presentation; but then I decided it might be much more useful to just ask you, my readers, what you think first...

p.s. if you like things just the way they are, and would like to see more of "the same" articles: please say that too. The only way I'll gauge the majority opinion is if you speak up. :)

Thanks for reading, and thanks for the 9,000+ hits I've received this year. (Give or take a few thousand hits that might just be search engine bots?)

Text: Prompts and Search Fields

Here's what I want to replicate:

The concept of a text field is nothing new, but this undertaking requires two other components:

1. A TextFieldUI with rounded corners, and a search icon.
2. Prompting: gray text prompting the user with instructions or other useful information.

Apple's Solution

It's elegant. It's simple:

textField.putClientProperty( "JTextField.variant", "search");
textField.putClientProperty( "JTextField.Search.Prompt", prompt);

The first line is well documented here. The second line is only casually mentioned here.

It's a great solution -- if you're only trying to support Macs. What I want is a cross-platform solution. This is the same problem that triggered my new ButtonUI's, and an indeterminate progress widget. If I may get on my soap box for a minute: some of the client properties Apple's Java team implements are really not a good use of their time. To be fair: some properties they implement are great (window opacity, types of title bars, window decorations, etc). Meanwhile there are JVM-crashing bugs, accessibility bugs, etc, that Apple's team can address but I cannot... so why implement properties like this? (end of rant.)

Creating a RoundTextFieldUI

So a pure-Java solution requires making a new TextFieldUI. This wasn't so hard, if you extend the BasicTextFieldUI. The key was figuring out I had to override the getVisibleEditorRect() method. This controls the rectangle the text appears in, so I can offset everything to draw the rounded rectangle.

Also by overriding modelToView() I was able to tweak the caret height (the x-position and width I didn't change).

Here is the class I came up with. It includes a client property "useSearchIcon" to render a magnifying glass icon.

That part was surprisingly simple. Next I wanted to deal with prompting.

Using XSwingX

This is a little project hosted at code.google.com. It has prompting built-in, and a very simple programming interface. But it was surprisingly tricky to put a demo together. It requires the SwingX package. But not the latest version: there are some incompatibilities in the latest version of SwingX integrated with XSwingX. We need version 0.9.2.

Here is a jar that includes all the necessary classes to get the job done. But if you start counting: that comes to 32 classes. Also if you start looking at the source code: the word "hack" keeps coming up. It doesn't seem like a great solution.

My Own Approach

I wanted to try something simple. Something done in one, maybe two classes. My first thought was to extend JTextField, and shuffle the color/text based on when the text field has the focus:

public class PromptTextField extends JTextField {
       static FocusListener focusListener = new FocusListener() {
             public void focusGained(FocusEvent e) {
                   PromptTextField src = (PromptTextField)e.getSource();
                   if(src.state==STATE_PROMPT) {
                         src.setState(STATE_NORMAL);
                   }
             }
  
             public void focusLost(FocusEvent e) {
                   PromptTextField src = (PromptTextField)e.getSource();
                   if(src.getText().length()==0)
                         src.setState(STATE_PROMPT);
             }
       };
       private static final int STATE_UNDEFINED = 0;
       private static final int STATE_NORMAL = 1;
       private static final int STATE_PROMPT = 2;
 
       private Color normalColor = SystemColor.textText;
       protected Color promptColor = Color.gray;
       protected String promptText;
       private int state = STATE_UNDEFINED;

       public PromptTextField(String text, String prompt, int columns) {
             super(text, columns);
             promptText = prompt;
             normalColor = getForeground();
             addFocusListener(focusListener);
             if(text.length()==0) {
                   setState(STATE_PROMPT);
             } else {
                   setState(STATE_NORMAL);
             }
       }
 
       public String getText() {
             if(state==STATE_PROMPT) {
                   return "";
             }
             return super.getText();
       }
 
       private void setState(int s) {
             if(s==state)
                   return;
             state = s;
             if(s==STATE_PROMPT) {
                   if(state==STATE_NORMAL && getText().length()>0)
                         throw new IllegalArgumentException("the state should not be set to STATE_PROMPT if there is text already in the text field (\""+getText()+"\")");
                   super.setForeground(promptColor);
                   super.setText(promptText);
             } else if(s==STATE_NORMAL) {
                   super.setForeground(normalColor);
                   super.setText("");
             }
       }
 
       public void setText(String s) {
             if(state==STATE_UNDEFINED) {
                   super.setText(s);
                   return;
             }
             if(s==null) s = "";
             if(hasFocus() || s.length()>0) {
                   state = STATE_NORMAL;
                   super.setForeground(normalColor);
                   super.setText(s);
             } else if(s.length()==0) {
                   state = STATE_PROMPT;
                   super.setForeground(promptColor);
                   super.setText(promptText);
             }
       }
}

I was impressed that, well, that this worked. It's very minimalist. However it's also not very good: it's an awkward hack that overrides the getText() and setText() methods. This means the getText() method might reflect something that has nothing in common with the data stored in getDocument().

Rob Camick's Approach

Who is Rob Camick? I don't know. But now that I found this article I'm going to pay closer attention. He talks about another model to achieve prompting: add a component over the text field.

It's simple. It's not a "hack," it's just plain old clever.

There's a problem, though: text placement. In my case -- because I'm using a TextFieldUI of my own creation -- it'll be hard to line up the prompted text exactly over the existing text.

... unless I use a TextOnlyGraphics2D and the same type of TextFieldUI.

public class TextFieldPrompt extends JTextField {
     public TextFieldPrompt(JTextField parent,
         Color promptColor,
         String promptText) {
         super(promptText);
       
         if(promptColor==null)
             promptColor = Color.gray;
       
         parent.add(this);
       
         setFocusable(false);
         setEditable(false);
         setForeground(promptColor);
         setOpaque(false);
       
         ... install listeners & UI ...
     }
   
   
     public void paint(Graphics g) {
         Graphics2D g2 = (Graphics2D)g;
         g2 = new TextOnlyGraphics2D( g2, null );
         super.paint( g2 );
     }
   
     public boolean contains(int x, int y) {
         return false;
     }
}

Conclusion

Here is an applet demonstrating the final model:


This jar (source included) is available here.

Text: Searching JTextComponents

This entry has to do with searching for a phrase in JTextComponents. There's a lot of ground to cover, so I'll start you off with the demo and then go into details. (Give this applet a few seconds to load... it has a lot of text to read.)

This jar (source included) is available here.

Text Search Bars

What is the point of a search bar? This is an interface decision more than a technical one. A search bar has several advantages over a search dialog:

1. A dialog will physically be in the wrong location. It will start out centered, which probably covers part of the text the user wants to search.
2. Providing a search bar in a fixed location will help users visually scan the interface more easily. They'll know where to look. It's a matter of identifying a fixed point vs searching for it.
3. Dialogs are usually (and perhaps originally) designed for the convenience of programmers. You can cram a program full of thousands of dialogs... but the screen real estate is unlimited. It is much harder to create a well-balanced window with just the right amount of essential interface controls without becoming too cluttered. It takes more work, and we usually don't bother.
4. Of course a modal search dialog is worse. Modal dialogs are discouraged here, among other places.

I modeled these search bars after the ones I saw in Firefox and Safari. They include:

• a search field
  
• next/previous buttons
  
• a checkbox for matching case
  
• a dismiss button
  
• a label including the occurrences of the search phrase
  
• a toggle button for "Highlight All"

The order (and presence) varies, but those are the basics. Safari automatically highlights all, and Firefox uses a toggle button. The anchor varies in both toolbars, but the components are still listed in about that order.

Text Search Dialog

Of course sometimes a dialog might be appropriate: that's up to you to decide. If searching is a feature so rarely used that it doesn't make sense to keep it in the main interface: that's justifiable. Or if your software is so versatile that searching is nothing but an auxiliary function? That's OK too.

So I included a minimal dialog. It automatically performs case insensitive, wrap-around searches. It's not modal, but it does dismiss itself when the user hits the return key.

No matter which model you use (a search dialog or a search bar), you probably want to also include typical menu shortcuts for find, find again, and find previous.

Searching Text

Most of the rest of this project was GUI work: so I was in my prime. The actual methods that do the searching/counting, though... I admit I'm pretty clueless about.

My first instinct was to convert the document into a String, and then use .indexOf() and .lastIndexOf() to search for everything. Now I have absolutely no proof for what I'm about to say, but that sure feels like a horrible approach. Especially when we have to convert the entire string to a specific case to avoid case sensitivity.

So instead I went spelunking around, and tried using the Segment class. Now the method is much hairier, but it walks through chunks of characters at a time to find the search phrase: at no time do I have an extra copy of the whole document floating around.

(Remember when we count the occurrences of the search phrase: we perform a bajillion searches, so this method needs to be relatively light.)

If anyone has any experience/suggestions on this subject, here is the code in question.

The Visuals

When you use the Safari-style search bar you're introducing two extra layers of visual effects into your JFrame:

The middle layer is known as the TextHighlightSheet. The topmost layer is a SearchHighlight. These layers have a few of things in common:

1. They are both added to the JLayeredPane.
2. They both are tied to their "parent" JTextComponent. (It's not technically their parent component as far as the Swing hierarchy is concerned, but it's useful to think of them as bound to that text component.) When AncestorListeners or ComponentListeners are notified that the TextComponent has moved around: these other layers have to immediately respond.
3. They're both animated. Or at least they can be. The middle sheet may fade in/out, and the highlight can have any number of customized animations.

Now I'll discuss each in a little more detail.

The TextHighlightSheet

This is the more clever component: it has to highlight all the occurrences of the search phrase, without looking at irrelevant hits. So it will first calculate the index that is mapped to the point (0, 0) of the viewport, and then the index that is mapped to (width, height). Then search only within those indices, and calculate the shapes of all the hits.

The hits are outlined in rectangles, and Area objects are used to join overlapping rectangles. Usually I avoid the Area class like the plague, but combined rectangles shouldn't be a hassle: they're small, and they contain only lines. Paint those rectangles white, and then paint everything around them in a translucent black (to add the overall shadow).

Finally a combination of the right clipping, translation, and a TextOnlyGraphics2D let us call parentTextComponent.paint(g) to render the text.

The Firefox search bar uses the same component as the Safari search bar. (Although in the Firefox bar you have to explicitly turn it on with the "Highlight All" button.) The Firefox sheet, however, doesn't have a translucent shadow. Also it uses different padding and colors. All of this is covered in the javadocs if you really want to customize it.

The sheet has isActive() and setActive() methods to control whether it is visible. The TextSearchBar automatically follows one of two behaviors:

1. If the "Highlight All" button is visible: then that button is the only factor controlling whether the sheet is active. This is how Firefox behaves.
   
2. If the "Highlight All" button is invisible: then the sheet is made active only when the search field has the keyboard focus. This is how Safari behaves.

SearchHighlight

This is a very different component, and is subclassed directly into the look you're aiming for. The AbstractSearchHighlight subclasses into the AquaSearchHighlight, the FirefoxSearchHighlight, and the NudgeSearchHighlight.

When constructed this component creates a BufferedImage of one particular hit of the search phrase. Then a javax.swing.Timer is activated, and this component is ready for action.

The subclasses can animate the transform and opacity of the image over the duration of the animation.

It is important that the sheet and the effect be designed to visually complement each other. For example: the highlight sheet for the Safari search bar is designed to work with the Aqua or the Nudge effects, but the Firefox effect uses smaller padding around the hit phrases.

Also the Firefox effect cheats and uses an infinite duration, so the effect never really goes away. I'm not sure exactly what should happen in this scenario if the JTextComponent is editable? (Any thoughts?) I guess the effect should go away as soon you modify the document such that that particular hit phrase is changed? This is not an issue in the Safari/Nudge models, simply because the animation will quickly dispose of itself. So this is a bug in the Firefox effect, but I'm not sure what the ideal behavior is.

Each effect is created by calling TextSearch.highlight(). This looks up the value for UIManager.get("textSearchHighlightEffect"), and instantiates the class name provided. I won't go into detail here, but the point is: you can customize this if you want. (You don't even have to extend the com.bric.plaf.AbstractSearchHighlight if you don't want to.)

Conclusion

In v1.0 of this project I just included the search bars and the effects. In v2.0 I added the middle sheet. If I get around to a v3.0: what next? (Aside from existing bugs.)

Speaking of bugs: I suppose you also need to be able to perform a revised search with every key stroke in the search field. This is how Firefox behaves. This is slightly different from hitting the enter key in the text field, because that will search for the next occurrence starting at the end of the current selection... but continuous typing really needs a search that starts at the beginning of the current selection. Another day, maybe...

Gradients: a Halftone Gradient

So much to do: so little time! I have 3 other fun subjects to blog about... all of them in varying states of completion. But this is the most finished, so it's the first to write up.

Halftone Gradients

I don't know why, but I've always wanted a gradient that looked like halftoning.

But for some reason whenever I thought of tackling it: I always envisioned myself writing a PaintContext and coming up with some ugly code that involved lots of varying radii, and antialiasing things by hand, and doing clever things with AffineTransforms and dot products that I'm not really well-versed in. And maybe that is the better/more efficient way to approach this subject... but there's a much easier way.

My last blog entry used transformed tiles to achieve a multiple-color gradient. If you look at a halftone gradient as a tiling pattern, then suddenly it becomes a piece of cake. I just need to create a tall, skinny tile that transitions from one color to another.

If I were a better blog author, I might try inserting a diagram here. I'd demonstrate exactly what the tile looks like, and maybe discuss the concept of halftoning a little bit more.

However I'm not a better author. I'm much too lazy. Plus (and I'm not *entirely* bs-ing here): I really do believe a hands-on interactive model will explain this project more thoroughly than diagrams ever will. So... here's a demo applet showing off the HalftoneGradient class:



The jar (source included) is available here.

Your Basic Options

Although I started with circles, it's even easier to make a tiled pattern using diamonds or isosceles triangles. The size of each shape scales in proportion to the width of the tile, and where in the gradient that shape is drawn.

I'm unsure, though, if the circles are done correctly. I didn't find an exact algorithm for describing how halftoning should work -- but I admit I didn't look too closely. If a gradient is expressed in terms of t=[0,1]: does the gradient at t = .5 really look correct? Is that a half-way point between the two colors? For the circles I made the radius related to t^.8 to try to improve the balance. There is surely a more scientific way to approach this subject, but I didn't have the interest or patience. Does anyone have any experience with this subject? Basically the proportion of color1 to color2 should increase linearly with t.

Oh, and as an afterthought I added the "offset" property, and the "animate" checkbox. How could I resist? (Have you seen my art? I love animating things.)

Making a Tiled Pattern Not Tile

This paint delegates all the hard work to a transformed TexturePaint, so it should be tiling like mad when the "Cycle" checkbox is unselected, right? This was probably the hardest part of the project: if a point lies outside of the strip created between P1 and P2: how do I make it a solid color?

Originally I went to Sun's website and downloaded the source for the TexturePaintContext. I was determined to roll up my sleeves and make an adaptation that solved this. It didn't take me long to realize I was waaaaaaay over my head, though, as I looked through their source code. Aside from subtle problems (like referencing non-public classes to manipulate rasters), they somehow eliminated floats/doubles. Everything was int-based, on a scale of [0, Integer.MAX_VALUE]. While probably more efficient: this hurt my head. If you locked me in a dark prison cell with just bagels and a laptop: maybe in a few days I could figure this out. But I wanted to figure this out in a few hours, so I moved on to plan B:

Instead I just rewrote the raster the TexturePaintContext returns. Is it the most efficient response? No. Does it work? Yes. The class is called CoveredContext. (It's non-public, although it's included in the jar mentioned above.) In its original draft the hard work looked like this:

public Raster getRaster(int x, int y, int w, int h) {
     WritableRaster raster = (WritableRaster)context.getRaster(x, y, w, h);
     ...
     double[] matrix = new double[6];
     transform.getMatrix(matrix);
     for(int y2 = 0; y2 < h; y2++) {
         raster.getDataElements(0, y2, w, 1, data);
    
         for(int x2 = 0; x2 < data.length; x2++) {
             double x3 = x2+x;
             double y3 = y2+y;

             // apply the transform manually:
             y3 = matrix[1]*x3+matrix[3]*y3+matrix[5];

             if(y3 < 1) {
                 data[x2] = color1;
             } else if(y3 >= distance-1) {
                 data[x2] = color2;
             }
         }
    
         raster.setDataElements(0, y2, w, 1, data);
      }
      return raster;
}

The final draft, though, is much more mathematically clever. (It calculates the two x-points in a row of pixels that mark the beginning and end of the gradient, and flood fills everything beyond those points.) But it's basically the same approach in spirit. It involves many more special cases, though.

Conclusion

So there you have it. Very painless. The code, including comments and javadoc, probably comes to less than 500 lines of code. Eventually I plan to integrate this into some other projects I'm working on, but that will have to wait.

I haven't tried to break this down and scrutinize performance yet. This class first needs to demonstrate that it, in fact, useful...

Gradients: a Boring Discussion

This really is a boring entry. In most of my articles I tried to include some sexy snippets of code, but this article is just an exploration of gradients and paints. There are no exciting conclusions. But I learned a lot, and in case you find yourself researching the same topic: here's what I found.

The catalyst that started all this was when I started studying some old code at work. A few years ago we devised a series of complicated mechanisms to help optimize rendering gradients. (Most of them were somehow based on Vincent Hardy's Java 2D API Graphics book.) Did our tricks work? And if they did: do they still?

In this entry when I say "gradient" I'm referring to what is now known as a MultipleGradientPaint. This is now standard in Java 1.6, but in earlier versions the only official Paint that came standard with Java is the GradientPaint (a 2-color gradient).

One problem here is that there are, to my knowledge, only a couple of gradients to work with. There are the linear and radial gradients in Hardy's book. Then later on Batik came along with linear and radial gradients designed for SVG. Funny thing is: Vincent Hardy helped write these, too. (Along with Nicholas Talian, Jim Graham, and Jerry Evans.) Which makes sense, but for the sake of this article I'd like to have some really distinct classes to compare.

A Simple Java 1.4-friendly Linear Gradient

So my first priority was to make a really simple alternative gradient to test against.

A few years ago Romain Guy pointed out you could improve performance with horizontal and vertical gradients if you used a TexturePaint instead of an actual gradient. And last year I combined TexturePaints with AffineTransforms in this article. So if you consider a linear gradient as basically a repeating tiled image... and you can transform that tile... then voila! You can make a GradientTexturePaint.

This paint creates a 1-pixel tall tiled image, and then finds the appropriate AffineTransform to make that tile stretch the two control points you provide. Here is a demo. Use the slider at the top to control the colors, and the two white circles adjust the endpoints of the gradient:

(You can download this demo, source included, here.)

You can right-click the preview area to change the rendering hints. Using Sun's pipeline there is a visible difference when the interpolation hints are set to BILINEAR or BICUBIC. For convenience I also made the ANTIALIASING key affect the interpolation, too.

Under Quartz, however, you have to set the INTERPOLATION, and define the COLOR_RENDERING. It doesn't matter what it is (SPEED or QUALITY), but it has to be defined. But the code I added to intercept the ANTIALIAS hint doesn't work... ugh. It's a wreck. Quartz is forever a mystery.

Meanwhile, here is a screenshot contrasting the difference RenderingHints make:

The biggest downside to this gradient is that it has to tile. (Unlike the GradientPaint class, which shows only one color beyond the two control points.) Also, as we see later: it's performance isn't great.

But the advantages are:

• It's Java 1.4 compatible. (Java 1.3, probably?)


• It requires no other classes. It's very lightweight to add to your project.

So. Problem #1 solved: I have at least one more gradient to compare.

Comparing Gradients

Let's focus on linear gradients. Here are the classes we can test against:

1. com.sun.glf.goodies.GradientPaintExt from Hardy's book.


2. com.bric.awt.GradientTexturePaint mentioned above.


3. org.apache.batik.ext.awt.LinearGradientPaint from Batik.


4. java.awt.LinearGradientPaint in Java 1.6+.


5. DelegatePaint: this is a small shell wrapped around the java.awt.LinearGradientPaint. I wanted to see if the LinearGradientPaint was receiving special optimized treatment under-the-hood in the graphics pipeline.

I put together several tests -- each test measures the median time and memory allocation required for an operation. The tests include:

• Creating PaintContexts.


• Calling Graphics2D.fillRect().


• Calling Graphics2D.fill(Ellipse2D).


• Performing these operations through a small AffineTransform.


• Also compare what it costs to recycle a gradient vs constructing it from scratch.

The test I used is here (source included). Here are the results after running it in a variety of environments.

There's a lot of data. Oiyee. What does it mean?

Creating Rasters vs Blitting

For starters: look at the first two lines. This involves getting a PaintContext and calling PaintContext.getRaster(). Notice that these are considerably faster than the next couple of lines that involve calling Graphics2D.fillRect() for the same number of pixels. This brought me to my first sad realization about Paint objects: the bottleneck in performance is deep in the graphics pipeline. This makes sense, but it is frustrating because it's harder to modify that behavior. Werner has taken on the task of creating a new Graphics2D pipeline, but it's a big undertaking.

Where does the rest of the time go? In the Sun pipeline (that is, when Quartz is not active), the time may look like this:

The vast majority of the time is spent blitting. This seems especially unfortunate for simple calls to fillRect() (because the "hard" work is already done, right?). But it is what it is.

Using Small AffineTransforms

What about those lines that usually take 1-3 milliseconds? The tests with "reduced" in the name? These paint both rectangles and ellipses through a transform that scales the x and y coordinates by 0.2.

The catch here is when Quartz is active: it doesn't do too well. This is where the "optimized" keyword comes in. Based on this research -- and some of the old code we had lying around work -- I added another method to the OptimizedGraphics2D class. Basically when using Quartz, and painting with anything other than a java.awt.Color, we get the raster from a PaintContext and then paint that as a BufferedImage. It looks something like this:

PaintContext context = currentPaint.createContext(ColorModel.getRGBdefault(), 
    rect, rect, getTransform(), oldHints);
BufferedImage scratchImage = getScratchImage(rect.width, rect.height);
WritableRaster paintRaster = (WritableRaster)context.getRaster(rect.x, rect.y, rect.width, rect.height);
BufferedImage newImage = new BufferedImage(context.getColorModel(), paintRaster, false, null);
Graphics2D g = scratchImage.createGraphics();
g.setComposite(AlphaComposite.Clear);
g.fillRect(0, 0, rect.width, rect.height);
g.translate(-rect.x, -rect.y);
g.transform( myTransform );
g.setColor(Color.black);
g.setComposite(AlphaComposite.SrcOver);
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, 
    RenderingHints.VALUE_ANTIALIAS_ON);
g.fill(s);
g.setTransform(identityTransform);
g.setComposite(AlphaComposite.SrcIn);
g.drawImage(newImage, 0, 0, null);
g.dispose();
drawImage(scratchImage, 
    rect.x, rect.y, rect.x+rect.width, rect.y+rect.height, 
    0, 0, rect.width, rect.height, 
    null, null);
newImage.flush();

So we're painting the mask in black, and then using a SrcIn composite to render the fill of our paint.

I'd recommend avoiding Quartz if you can. This is one of a handful of surprises we've found over the years, and Apple has been steadily pushing the Sun pipeline for some time now.

Note that in these tests (in the GradientTest app), there is a crucial line involved in testing:

OptimizedGraphics2D.testingOptimizations = true;

This means the code I mentioned above is always being called. But in regular usage that method will only be called when Quartz is active. (The OptimizedGraphics2D is very careful about when it employs its special tricks.)

What About the Delegates?

Good news! The delegates are worthless. In Java 1.6 I wrapped the awt.LinearGradientPaint in a custom Paint object (and also wrapped their PaintContext, too). I know java.awt.Colors and java.awt.BasicStroke used to get special optimized treatment in Quartz (and maybe in Sun's pipeline?)... so I wanted to see if that was the case here. It isn't. This is great: it means what we see is what we get. And it means we can ignore those columns of data.

What About Sharing?

Here "sharing" refers to using a static gradient, instead of constructing a new gradient object in each test.

In the "Java 1.5" and "Java 1.6" tabs, there is practically no difference in the time it takes between sharing and non-sharing tests. This is also welcome news. It means the Paint objects aren't doing any clever caching of their own, and it makes developers' lives easier.

However. When Quartz is active, there's a speed improvement gained by sharing the gradient object. And the memory allocation? Wow. 0 bytes in some cases? Something special is going on here. And I have no idea what it is. Moving on. (Seriously, how am I supposed to cover all this material?)

The Gradient Classes

One of the most obvious comparisons is to look at the columns side-by-side. What stands out?

The first thing that pops out at me: the GradientTexturePaint loses. It's not horrible, but it's always behind the other classes in terms of speed.

Surprisingly in Java 1.6 the Batik class slightly outperform the awt class. I assumed the awt class was, well, based-on the Batik model. The code should be the same, right? Apparently not. Calling fillRect() is 25% faster when you use Batik.

I find the memory allocation results are looking pretty shady here. The GradientTexturePaint takes 0 bytes to run some of these tests? I wouldn't be surprised if the tests/measured are somehow flawed. This doesn't make sense.

Managing Memory

Usually when I measure memory allocation, I add lines that call garbage collection before a sensitive block of code. But it turns out this can dramatically skew some results in this case: because of how some gradients are cached. (WeakReferences can get involved! Calling garbage collection unnecessarily destroys the cached raster they refer to.)

The more I look at this, the more is looks like the Java2D architecture for Paints was poorly thought out. A Paint has one real method of interest: createContext(). (Already I'm scratching my head: why not make the method called getContext()? The distinction is that create() requires making a new object, and get() might simply return an existing object. This could have huge implications if you paint thousands of lines at varying transforms. But this is trivial compared to the next part...)

A PaintContext has a magic method called getRaster(). It returns a Raster object. It looks simple enough, until you consider how it is used: you might be painting millions of shapes! Is each call to this method creating a new Raster? Rasters are huge; this can really affect how fast memory is burned through. Instead the method should accept a destination WritableRaster as an argument. This avoids the cost of creating new objects constantly. Also it clearly defines the role of the method. With the existing signature: I don't know what happens to a raster object when I hand it to the caller. Am I free to reuse it? Am I responsible for disposing it? The graphics pipeline may want to hang on to it until a certain number of rasters queue up, and then process it. (This would be especially useful if opaque shapes were rendering on top of each other: it may be the case that the first shape is completely obscured by latter shapes, so its raster is irrelevant. I'm not saying this is done: just that it's possible.)

However "good" paints do not create rasters constantly. Batik paints, for example, have a static method: getCachedRaster(ColorModel cm,int w,int h). As long as you're asking for a width and height greater than 32 pixels: one static raster will be continuously recycled. They've been doing this for years and the world doesn't fall apart, so apparently it's a safe thing to do. So if you're writing your own PaintContext: recycle your rasters. Look at Batik's example. (They also dispose/release rasters in a clever way.) Their example is important because it's efficient, and because it obviously works with the Sun pipeline.

Conclusions

I don't really have any conclusions; I just felt like this article should end with a "Conclusions" section. :)

The only useful things to come out of this are:

• I added a new trick to the OptimizedGraphics2D class. However as I phase out Quartz more and more, it doesn't seem that important.


• I made a light multiple-color gradient for Java 1.4. Since I strive to maintain backwards compatibility in most of my projects: this may quickly end up in my custom UI components.

If you aren't worried about backwards compatibility: Java 1.6's gradient paints look perfectly sound. The Batik classes outperformed them in some cases, but overall I think they're a perfectly good choice if you're shopping around.

I apologize for the haphazard way material was presented here, but there really is so much ground to cover. I could spent hours (and pages) studying the subtleties of the numbers I collected; it's hard to figure out exactly what is important and how to present it.

If you'd like to see me add more tests, or have insight into the results already mentioned here: please be in touch. If you have specific questions that might be hard to answer through comments please email me directly.

AlphaComposites: Which Does What?

It seems like every 4 months or so I come across a particular problem that I want to solve using AlphaComposites. The problem is I'm a very visual person; the equations listed in the javadocs don't help me visualize what each composite actually looks like.

Here is an applet that demonstrates most of the possible variables at play:

This applet (source included) is available here.

Also there are several AlphaComposite types where it makes a huge difference whether you're rendering images vs shapes. In this applet if "Use Images" is selected then the shapes you see are first rendered in a BufferedImage, and then those images are rendered together.

That's all. This is a clean, short article -- as much for my benefit as anyone else's.

Buttons: New UI's

This article features some new ButtonUIs I've been working on.

Coming from a strong background in Macs, the first UIs I wanted to emulate are the ones found here. But I abstracted some common patterns and made a generic model that can easily be adapted to other looks, too.

Here's a demo applet showing off what I have so far. The UI's are displayed in a JInternalFrame so you can resize the window to see how they scale. (They're originally presented at their preferred size.)

This app (source included) is available here.

Why Bother?

Why bother to make these UI's when Apple provided some convenient client properties to get the same buttons in Java? I have several possible reasons:

These are cross-platform. You can use them anywhere.

These are pure Java. This means you can override methods of interest. Also if you find a bug you can go exploring...

Resizing. It turns out -- last I checked -- that Apple's buttons are vector-based. (Try rendering them through an AffineTransform!) However: as JComponents they don't always resize well. (Especially the segmented variety.) If you have an icon that's just a couple of pixels larger than allowed: then those couple of pixels may just dangle off the edge.

Vertical segments. Although horizontal segments are much more common/important: why not offer vertical segments?

Java 1.4 compatible. (Probably even Java 1.3 compatible -- I haven't checked?) At work we strive to maintain backward compatibility for several years, so this is an advantage for us.

As a disclaimer I should add: it is not my intention to make pixel-perfect replicas of any UI's. Instead my goal is to be "passable". There may be some improvements, and there be some regressions. If you have questions or comments, please let me know. I may be able to defend certain design decisions, or you may convince me to change something. :) But speaking of pixel-precision: I want to thank Werner for giving me some great advice for certain rendering problems I was having!

Architecture

After some discussion with Thasso, I decided to simplify the original architecture. Before there was too much emphasis on how to let subclasses define their own shape, and not enough flexibility with the fill.

In this version the FilledButtonUI is the parent of all my other ButtonUIs. This object is responsible for the shape of the button area; the subclasses are only interested in the fill and in rendering.

All the UI's can automatically receive Mac-like focus-rings either inside or outside the shape of the button (or both). In the demo, most of the rings are usually outside except in the 3x3 block of buttons: here the focus needs to be painted internally, because external focus wouldn't show in the the centermost cell. If you want to render a customized type of focus, you can override the getFocusPainting() method to return PAINT_NO_FOCUS. (This means that the FilledButtonUI isn't automatically painting focus for you, and you need to implement your own painting code somewhere else.)

Client Properties

In the demo above all the buttons share the same instance of one ButtonUI. The UI looks at client properties in the button to distinguish the shape. To set up a button on the bottom-left corner of a grid of buttons, you can call:

myButton.putClientProperty( HORIZONTAL_POSITION, LEFT );
myButton.putClientProperty( VERTICAL_POSITION, BOTTOM );

(It is assumed if one of these properties is missing that the intended value is ONLY.)

Also these UI's support an arbitrary shape. To create a circular button you can call:

myButton.putClientProperty(SHAPE, new Ellipse2D.Float(0,0,100,100));

(The shape will be scaled until the icon and text of a button fit inside it.)

In the comments below Andre requested something similar to this feature. Personally I only really see myself using this feature for circles, but it is up to your discretion to decide if you think other shapes are a good fit for your UI.

Thasso pointed out the current implementation requires that buttons be mutually exclusive in their layout: if two buttons are supposed to be nestled together in, say, a diamond-like pattern, then the FilledButtonUI is probably not a good choice. While that sounds like a fun design challenge to address (as far as programming is concerned), this project is already very large in scope and I don't want to take on more than I can handle.

ButtonClusters

You probably don't want to comb through your UI and set up client properties for each toolbar to get the correct segment positions. And even if you did: supposed you have a toolbar with buttons A, B, C and D, but then in some situations you made D invisible. At this point button C needs to become the right-most button in the set, instead of being a middle button. Updating this constantly seems like a real chore.

The ButtonCluster object automates this for you. It works under a couple of assumptions, though. It does not directly affect the layout of your buttons: it assumes when you pass it an array of buttons that they will always be adjacent in your UI, and that they will be presented in that order. A ButtonCluster has no way of knowing if that is or is not the case; it assumes that part of your UI is unchanging.

When you create a cluster you can ask it to be standardized or not. (This is largely in response to Michael's comments below.) If this boolean is true, then the FilledButtonUI will arrange each button in a cluster to have approximately the same width and height. This will result in larger GUI components, but with a more balanced look.

Also be sure to check out the static install() methods in the javadoc: that may be the only method you need to call from your code to incorporate this project into your app.

Embellishments

Extras. Eye candy. I added a few to the demo applet just to demonstrate how easy they are.

The blinking focus is built into the FilledButtonUI, and probably required about 20 lines of code. This is a feature a user once requested for accessibility reasons: users who can't easily use a mouse largely rely on keyboard focus to navigate software, but sometimes the focus is so subtle in a crowded interface that it's hard to spot. The blinking really helps the human eye to find the area of interest.

The other two extras (shimmer and zoom) are UIEffects. This is a very simple but fun little class that the FilledButtonUI supports when rendering a button. In a way this model is my attempt to compensate for not having multiple inheritance: I can make easy little effects that I can apply to any and all FilledButtonUI subclasses. (And remember Java2D's most basic drawing tools can go a long way if you're creative.)

Keyboard Focus

Speaking of keyboard focus, that reminds me (can you tell that writing this article is very much a stream-of-consciousness process?)... I'm also very pleased with the FocusArrowListener. This listens for arrow keys, and shifts the focus accordingly when a key is pressed. It's a very simple concept, but I think it should be applied to most GUI elements that do not already have defined behaviors for the arrow keys. JToolBars already do something very similar: pressing the the left and right arrow keys is generally equivalent to pressing tab and shift-tab. (Except when you get to one extreme of a JToolBar the focus will wrap around to the other side.)

Again: thanks goes to Werner for some helpful tips on this element.

Conclusion

The incentive for this project was to get some consistent aesthetic UI's to use in our desktop applications. With that goal in mind I put together what I hope is a relatively adaptable model for all sorts of other button UI's to work from.

I've spent a lot of time on this in the last few weeks and I need to move on for now. If you have improvements/fixes you want to make: the code is in the jar (and available on an SVN server), so you're welcome to do whatever is necessary. And if you email me afterwards I might be able to incorporate the changes into the official release.

As of this writing the features I would most like to add are HTML-support (in text) and a FadingUIEffect. Neither of these should require overhauling anything major: but they are outside my current needs.

Feel free in the comments to talk about your favorite ButtonUI's from other sources/L&F's, and what makes them your favorite.

Approaches to "UPSERT"

This week in the Database Programmer we look at something
called an "UPSERT", the strange trick where an insert
command may magically convert itself into an update if
a row already exists with the provided key. This trick
is very useful in a variety of cases. This week we will
see its basic use, and next week we will see how the same
idea can be used to materialize summary tables efficiently.

An UPSERT or ON DUPLICATE KEY...

The idea behind an UPSERT is simple. The client issues
an INSERT command. If a row already exists with the
given primary key, then instead of throwing a key
violation error, it takes the non-key values and updates
the row.

This is one of those strange (and very unusual) cases
where MySQL actually supports something you will not
find in all of the other more mature databases. So if you
are using MySQL, you do not need to do anything special
to make an UPSERT. You just add the term "ON DUPLICATE
KEY UPDATE" to the INSERT statement:

insert into table (a,c,b) values (1,2,3)
    on duplicate key update
     b = 2,
     c = 3

The MySQL command gives you the flexibility to specify
different operation on UPDATE versus INSERT, but with
that flexibility comes the requirement that the UPDATE
clause completely restates the operation.

With the MySQL command there are also various considerations
for AUTO_INCREMENT columns and multiple unique keys.
You can read more at the MySQL page for the
"http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html"
>INSERT ... ON DUPLICATE KEY UPDATE feature.

A Note About MS SQL Server 2008

MS SQL Server introduced something like UPSERT in
SQL Server 2008. It uses the MERGE command, which is
a bit hairy, check it out in this
"http://www.databasejournal.com/features/mssql/article.php/3739131/UPSERT-Functionality-in-SQL-Server-2008.htm"
>nice tutorial.

Coding a Simpler UPSERT

Let us say that we want a simpler UPSERT, where you do not
have to mess with SQL Server's MERGE or rewrite the entire
command as in MySQL. This can be done with triggers.

To illustrate, consider a shopping cart with a natural key
of ORDER_ID and SKU. I want simple application code that
does not have to figure out if it needs to do an INSERT or
UPDATE, and can always happily do INSERTs, knowing they will
be converted to updates if the line is already there.
In other words, I want simple application code that just keeps
issuing commands like this:

INSERT INTO ORDERLINES
       (order_id,sku,qty)
VALUES 
       (1234,'ABC',5)

We can accomplish this by a trigger. The trigger must occur
before the action, and it must redirect the action to an
UPDATE if necessary. Let us look at examples for MySQL,
Postgres, and SQL Server.

A MySQL Trigger

Alas, MySQL giveth, and MySQL taketh away. You cannot code
your own UPSERT in MySQL because of an extremely severe
limitation in MySQL trigger rules. A MySQL trigger may not
affect a row in a table different from the row originally
affected by the command that fired the trigger. A MySQL
trigger attempting to create a new row may not affect
a different row.

Note: I may be wrong about this. This limitation has bitten
me on several features that I would like to provide for MySQL.
I am actually hoping this limitation will not
apply for UPSERTs because the new row does not yet exist, but
I have not had a chance yet to try.

A Postgres Trigger

The Postgres trigger example is pretty simple, hopefully the
logic is self-explanatory. As with all code samples, I did
this off the top of my head, you may need to fix a syntax
error or two.

CREATE OR REPLACE FUNCTION orderlines_insert_before_F()
RETURNS TRIGGER
 AS $BODY$
DECLARE
    result INTEGER; 
BEGIN
    SET SEARCH_PATH TO PUBLIC;
    
    -- Find out if there is a row
    result = (select count(*) from orderlines
                where order_id = new.order_id
                  and sku      = new.sku
               )

    -- On the update branch, perform the update
    -- and then return NULL to prevent the 
    -- original insert from occurring
    IF result = 1 THEN
        UPDATE orderlines 
           SET qty = new.qty
         WHERE order_id = new.order_id
           AND sku      = new.sku;
           
        RETURN null;
    END IF;
    
    -- The default branch is to return "NEW" which
    -- causes the original INSERT to go forward
    RETURN new;

END; $BODY$
LANGUAGE 'plpgsql' SECURITY DEFINER;

-- That extremely annoying second command you always
-- need for Postgres triggers.
CREATE TRIGGER orderlines_insert_before_T
   before insert
   ON ORDERLINES
   FOR EACH ROW
   EXECUTE PROCEDURE orderlines_insert_before_F();

A SQL Server Trigger

SQL Server BEFORE INSERT triggers are significantly different
from Postgres triggers. First of all, they operate at the
statement level, so that you have a set of new rows instead
of just one. Secondly, the trigger must itself contain an
explicit INSERT command, or the INSERT never happens. All of this
means our SQL Server example is quite a bit more verbose.

The basic logic of the SQL Server example is the same as the
Postgres, with two additional complications. First, we must use
a CURSOR to loop through the incoming rows. Second, we must
explicitly code the INSERT operation for the case where it
occurs. But if you can see past the cruft we get for all of that,
the SQL Server exmple is doing the same thing:

CREATE TRIGGER upsource_insert_before
ON orderlines
INSTEAD OF insert
AS
BEGIN
    SET NOCOUNT ON;
    DECLARE @new_order_id int;
    DECLARE @new_sku      varchar(15);
    DECLARE @new_qty      int;
    DECLARE @result       int;

    DECLARE trig_ins_orderlines CURSOR FOR 
            SELECT * FROM inserted;
    OPEN trig_ins_orderlines;

    FETCH NEXT FROM trig_ins_orderlines
     INTO @new_order_id
         ,@new_sku
         ,@new_qty;

    WHILE @@Fetch_status = 0 
    BEGIN
        -- Find out if there is a row now
        SET @result = (SELECT count(*) from orderlines
                        WHERE order_id = @new_order_id
                          AND sku      = @new_sku
                      )
    
        IF @result = 1 
        BEGIN
            -- Since there is already a row, do an
            -- update
            UPDATE orderlines
               SET qty = @new_qty
             WHERE order_id = @new_order_id
               AND sku      = @new_sku;
        END
        ELSE
        BEGIN
            -- When there is no row, we insert it
            INSERT INTO orderlines 
                  (order_id,sku,qty)
            VALUES
                  (@new_order_id,@new_sku,@new_qty)
            UPDATE orderlines

        -- Pull the next row
        FETCH NEXT FROM trig_ins_orderlines
         INTO @new_order_id
             ,@new_sku
             ,@new_qty;

    END  -- Cursor iteration

    CLOSE trig_ins_orderlines;
    DEALLOCATE trig_ins_orderlines;

END

A Vague Uneasy Feeling

While the examples above are definitely cool and nifty,
they ought to leave a certain nagging doubt in many
programmers' minds. This doubt comes from the fact that
an insert is not necessarily an insert anymore,
which can lead to confusion. Just imagine the new programmer
who has joined the team an is banging his head on his desk
because he cannot figure out why his INSERTS are not
working!

We can add a refinement to the process by making the
function optional. Here is how we do it.

First, add a column to the ORDERLINES table called
_UPSERT that is a char(1). Then modify the trigger so that
the UPSERT behavior only occurs if the this column holds
'Y'. It is also extremely import to always set this value
back to 'N' or NULL in the trigger, otherwise it will appear
as 'Y' on subsequent INSERTS and it won't work properly.

So our new modified explicit upsert requires a SQL statement
like this:

INSERT INTO ORDERLINES
       (_upsert,order_id,sku,qty)
VALUES
       ('Y',1234,'ABC',5)

Our trigger code needs only a very slight modification.
Here is the Postgres example, the SQL Server example should
be very easy to update as well:

   ...trigger declration and definition above
   IF new._upsert = 'Y'
      result = (SELECT.....);
      _upsert = 'N';
   ELSE
      result = 0;
   END IF;
   
   ...rest of trigger is the same

Conclusion

The UPSERT feature gives us simplified code and fewer
round trips to the server. Without the UPSERT there are
times when the application may have to query the server to
find out if a row exists, and then issue either an UPDATE
or an INSERT. With the UPSERT, one round trip is eliminated,
and the check occurs much more efficiently inside of the
server itself.

The downside to UPSERTs is that they can be confusing if
some type of explicit control is not put onto them such as
the _UPSERT column.

Next week we will see a concept similar to UPSERT used
to efficiently create summary tables.

[Publi-Info] D�couvrez et prenez part au projet Moblin

Comme le savent ce qui suivent ce blog depuis un moment, il n'est pas dans mes habitudes de sponsoriser mon contenu et de vous parler de machines � caf�, de plantes vertes ou de gastronomie sur ce blog dans le but de me mettre un peu d'argent en poche.

J'ai choisi de vous parler de Moblin parce que j'ai moi-m�me r�cement install� Ubuntu sur mon ordinateur, et fait en m�me temps mes premiers pas dans le monde de libre et de Linux, qui m'enthousiasme par la richesse de sa communaut� et par les perspectives nouvelles qu'il ouvre sur l'informatique, enfin en tout cas pour moi qui m'�tait jusque l� cantonn� � Windows.

Moblin est un OS open source destin� aux netbook, t�l�phones portables et MIDs.

Bas� sur GNU/Linux, le projet a �t� lanc� en 2007 par Intel dans le but de devenir la plateforme Linux pour mobile la plus aboutie.

Je pourrais m'�tendre un peu plus sur les objectifs et les caract�ristiques de projet, mais je ne r�siste pas � l'envie de vous d�voiler � quoi ressemble un ordinateur tournant sous Moblin :

On peut remarquer que l'interface prend la forme d'onglets regroupants les principaux logiciels et fonctions de syst�me, ce qui n'est pas sans rappeller l'interface de Xandros sur les eeePC.

Voila cette m�me interface en fonctionnement :

Sympa non ?

Personellement je trouve cette interface tr�s ergonomique, dans le sens ou on peut acc�der rapidement aux fonctions essentielles pour l'utilisateur lambda, telles que le navigateur, le lecteur audio ...

Alors bien s�r cela parait tr�s limit� pour l'utilisateur averti qui ne peut se contenter des fonctions de base de son odinateur, mais cette interface convient on ne peut mieux � des personnes qui ont besoin d'un OS simple et rapide pour effectuer des t�ches basiques dans un environnement graphique chaleureux.

De plus, l'optimisation de Moblin pour les appareils mobiles rend sa consommation en �nergie moindre, vous assurant une automie cons�quente.

Moblin est aujourd'hui en plein d�veloppement par une communaut� grandissante, que vous pouvez rejoindre en vous rendant sur Moblin Zone ou Moblin.org.

Vous pourrez ainsi, si vous �tes d�veloppeur, participer � l'�laboration de Moblin. Et si vos comp�tences en code se limitent comme les miennes � pas grand-chose, vous pouvez t�l�charger Moblin, le tester sur votre propre machine et participer � son d�veloppement en reportant vos impression et en aidant les d�veloppeurs � faire de Moblin "la plateforme Linux pour mobile la plus aboutie".

Article sponsoris�

The Relational Model

If you look at any system that was born on and for the
internet, like Ruby on Rails, or the PHP language, you find
an immense wealth of resources on the internet itself, in
endless product web sites, blogs, and forums. But when
you look for the same comprehensive information on products
or ideas that matured before the web you find it is not there.
Relational databases stand out as a product family that matured
before the internet, and so their representation in cyberspace
is very different from the newer stuff.

The Math Stuff

You may have heard relational theorists argue that the
strength of relational databases comes from their solid
mathematical foundations. Perhaps you have wondered,
what does that mean? And why is that good?

To understand this, we have to begin with
>Edsger W. Dijkstra, a pioneer in the area of computer
science with many accomplishments to his name. Dijkstra
believed that the best way to develop a system or program
was to begin with a mathematical description of the system,
and then refine that system into a working program. When
the program completely implemented the math, you were
finished.

There is a really huge advantage to this approach. If you
start out with a mathematical theory of some sort, which
presumably has well known behaviors, then the working program
will have all of those behaviors and, put simply, everybody
will know what to expect of it.

This approach also reduces time wasted on creative efforts
to work out how the program should behave. All those
decisions collapse intot he simple drive to make the program
mimic the math.

A Particular Bit of Math

It so happens that there is a particular body of math
known as Relational Theory, which it seemd to
>E. F. Codd would be a very nice fit for storing
business information. In his landmark 1970 paper
>A Relational Model of Data for Large Shared Data Banks
(pdf) he sets out to show how these mathematical
things called "relations" have behaviors that would be
ideal for storing business models.

If we take the Dijkstra philosophy seriously, which is to
build systems based on well-known mathematical theories,
and we take Codd's claim that "Relations" match well to
business record-keeping needs, the obvious conclusion is
that we should build some kind of "Relational" datastore,
and so we get the Relational Database systems of today.

So there in a nutshell is why relational theorists are
so certain of the virtues of the relational model, it's
behaviors are well-known, and if you can build something
that matches them, you will have a very predictable
system.

They are Still Talking About It

If you want to know more about the actual mathematics,
check out the "http://groups.google.com/group/comp.databases.theory/topics"
>comp.databases.theory Usenet group, or check out
Wikipedia's articles on "http://en.wikipedia.org/wiki/Relational_algebra"
>Relational Algebra and "http://en.wikipedia.org/wiki/Relational_calculus"
>Relational Calculus.

A Practical Downside

The downside to all of this comes whenever the mathematical
model describes behaviors that are contrary to human goals
or simply irrelevant to them. Examples are not hard to
find.

When the web exploded in popularity, many programmers found
that their greatest data storage needs centered on documents
like web pages rather than collections of atomic values
like a customer's discount code or credit terms. They found
that relational databases were just not that good at storing
documents, which only stands to reason because they were never
intended to. In theory the model could be stretched,
(if the programmer stretched as well), but the programmers
could feel in their bones that the fit was not right, and they
began searching for something new.

Another example is that of calculated values. If you have
shopping cart, you probably have some field "TOTAL" somewhere
that stores the final amount due for the customer. It so
happens that such a thing violates relational theory, and there
are some very bright theorists who will refuse all requests
for assistance in getting that value to work, because you
have violated their theory. This is probably the most shameful
behavior that relational theorists exhibit - a complete
refusal to consider extending the model to better reflect
real world needs.

The Irony: There are No Relational Databases

The irony of it all is that when programmers set out to build
relational systems, they ran into quite a few practical
downsides and a sort of consensus was reached to break the
model and create the SQL-based databases we have today.
In a truly relational system a table would have quite
a few more rules on it than we have in our SQL/TABLE based
systems of today. But these rules must have seemed
impractical or too difficult to implement, and they were
scratched.

There is at least one product out there that claims to
be truly relational, that is "http://en.wikipedia.org/wiki/Dataphor">Dataphor.

The Weird Optional Part

Probably the grandest irony in the so-called relational
database management systems is that any programmer can
completely break the relational model by making bad
table designs. If your tables are not normalized, you
lose much of the benefits of the relational model,
and you are completely free to make lots of
non-normalized and de-normalized tables.

Conclusion

I have to admit I have always found the strength of
relational databases to be their simplicy and power,
and not so much their foundations (even if shaky) in
mathematical theory. A modern database is very good
at storing data in tabular form, and if you know how
to design the tables, you've got a great foundation for
a solid application. Going further, I've always found
relational theorists to be unhelpful in the extreme in
the edge cases where overall application needs are not
fully met by the underlying mathematical model. The
good news is that the products themselves have all of
the power we need, so I left the relational theorists
to their debates years ago.

D�sactivez les fonctions inutiles de Vista

Sous Vista comme sous XP, de nombreux petits logiciels et fonctions inutiles alourdissent le fonctionnement de votre PC et prennent inutilement de l�espace sur votre disque.

Parmi ces fonctions inutiles on trouve par exemple les jeux, l�espace de collaboration Windows, ou les composants pour Tablet PC, ou encore la Compression diff�rentielle � distance �

Toutes ces logiciels sont heureusement d�sactivables facilement via le panneau de configuration.

Pour commencer votre grand nettoyage, rendez vous dans le panneau de configuration et double cliquez sur Programmes et fonctionnalit�s :

Dans la colonne de gauche de la fen�tre qui s�ouvre, cliquez sur Activer ou d�sactiver des fonctionnalit�s Windows :

C�est l� que nous entrons dans le vif du sujet. Dans la fen�tre qui s�ouvre, vous pouvez d�sactiver les fonctions qui vous sont inutiles :

Personnellement, voila les les fonctionnalit�s que j�ai conserv�es. Vous remarquerez qu�il n�en reste plus beaucoup, mais si vous doutez de l�inutilit� de ce que j�ai d�sactiv�, un petite recherche sur internet vous permettra de vous forger un avis sur la question.

Une fois votre m�nage achev�, cliquez sur OK.

Une fen�tre s�ouvre alors, vous faisant patienter pendant la d�sactivation des composants d�s�lectionn�s. Lorsqu�il se fermera, vous serez invit� � red�marrer votre ordinateur.

Notez que vous pouvez � tout moment r�activer une fonctionnalit� que vous avez d�sactiv�e en revenant au m�me endroit et en re-cochant la case.

Acc�l�rez votre ordinateur et r�duisez la sollicitation de votre disque dur en d�sactivant l�indexation

Windows, que ce soit XP ou Vista, dispose par d�faut du service d�indexation activ�. Ce service permet de r�f�rencer tous les fichiers pr�sents sur les diff�rentes partitions de l�ordinateur dans un index, ce qui a pour effet d�acc�l�rer les recherches.

Toutefois, ce services sollicite constamment votre processeur et votre disque pour maintenir � jour son index, et son utilit� est tr�s discutable quand on n�effectue des recherches que de temps en temps, car  elles restent possibles mais se trouvent juste l�g�rement ralenties par l�absence d�index.

Il est donc tr�s int�ressant, en vue d�une augmentation de vos performance et de la pr�servation de votre disque -qui a une dur�e de vie limit�e- d�emp�cher ce service d�indexation d�officier sur votre PC.

Pour commencer, allez dans le Poste de travail et double-cliquez sur votre disque dur tout en laissant la touche Alt enfonc�e pour ouvrir ses propri�t�s :

Dans cette fen�tre, d�cochez la ligne Indexer ce lecteur pour une recherche rapide et validez par OK :

Un avertissement va s�afficher, vous demandant si vous voulez affecter les modifications au lecteur C:\ ou aux lecteur et � ses sous-dossier et fichiers.

Choisissez Appliquer les modifications au lecteur C:\, aux sous-dossiers et aux fichiers.

Validez par OK et attendez la fin du processus d�application des param�tres.

Il ne vous reste plus qu�� appliquer le m�me proc�d� aux autres disques et partitions pr�sents dans le Poste de travail pour d�sactiver compl�tement l�indexation.

I Am But a Humble Filing Clerk

This week we are returning to the series on Philosophy,
and we will nail down the role of data and the
database in any application that requires such
things.

This is the Database Programmer blog, for anybody who wants
practical advice on database use.

There are links to other essays at the bottom of this post.

This blog has two tables of contents, the
Topical Table of Contents and the list
of
Database Skills.

Review of The Absolute

In the first essay in this series, "http://database-programmer.blogspot.com/2008/09/quest-for-absolute.html"
>The Quest For the Absolute, I offered the opinion
that all programmers by nature seek absolutes to
simplify and inform the development effort. Taking
a page from the ancient Greek philosopher Aristotle,
I suggested that the best absolute was the quest for
the virtuous program, which is to say a program
that served its purpose well.

A program that serves its purpose well is one that
meets the needs of the check signer, the end-user,
and the programmer. The check signer needs some
strategic goal to be met, the end-user must be
productive, and the programmer must make a living.
If a program achieves all of these, it is an ideal
virtuous program, and has satisfied the absolute
requirements that are true of all programs.

Considering the Decision Maker

Normally we think of a decision maker as some important
person who has the power to choose your product or
services, or to give her money to your competitor.
She makes her decision based on how well she can judge
who will meet her strategic needs.

Although the decision maker will have vastly different
needs in different situations, and is usually thinking
at a high level, she has at least one
need that is universal: the simple need to keep and
use records. She needs a filing system. All of her
stated goals will assume that you both know
this unstated goal is down there at the foundation of
the entire proposed system.

We programmers often forget this simple fact because
computers have been around long enough that we
do not remember that in their original forms it was
impossible to mistake that computers were just
electronic filing systems. Way back when
I was a kid the day came when phone bills started
arriving with an "IBM Card" slipped into them. You
returned the card with your check -- they were moving
their files into the electronic age. Then came
electronic tickets on airlines -- nothing more than
a big filing system. The modern web sites we visit
to buy tickets are nothing but an interface to what
remains a filing system at its heart.

The Virtuous Programmer

So if we go back to the idea of "virtue" as the Greeks
thought of it, which means serving your function well,
a virtuous programmer will remember always that he
is but humble filing clerk. This is not his entire
purpose, but it is the beginning of all other purposes
and the foundation that the higher purposes are
built upon.

Not Just Relational

This principle is general to all programming. An
email server is a program that must receive and
store email for later retrieval. What good is an
email server that cannot store anything? What
good is a camera without its memory card? What
good is a mobile phone without its contacts list?
What good is a image editing program if it cannot
read and write files?

So all programs exist to process data, and the
business application programmer knows that in his
context this means we are really making giant sexy
record-keeping systems. We are the guys that
color-code the filing cabinets.

Does Not Mean Relational Is Required

This idea, that we are filing clerks, does not
automatically mean we must pick relational databases
for the persistence layer -- the question of what
filing system to use is a completely different
question.

Conclusion

If we begin with the idea that the ideal program
meets the needs of decision maker, end-user, and
programmer, and if we consider first the needs of
the decision maker, then we begin with the universal
strategic need to keep good records. The ideal
programmer knows this need is at the bottom of all
other needs, and remembers always that we are but
humble filing clerks.

Related Essays

This blog has two tables of contents, the
Topical Table of Contents and the list
of
Database Skills.

Other philosophy essays are:

• >Prepare Now For Possible Future Head Transplant
  
• >The Quest for The Absolute
  
• >I Am But A Humble Filing Clerk (this essay)
  
• >Why I Do Not Use ORM
  
• >Minimize Code, Maximize Data
  

A Comprehensive Database Security Model

This week I am taking a bit of a departure. Normally I write
about things I have already done, but this week I want to
speculate a bit on a security model I am thinking of coding
up. Basically I have been asking myself how to create a
security model for database apps that never requires elevated
privileges for code, but still allows for hosts sharing multiple
applications, full table security including row level and
column level security, and structural immunity to SQL injection.

The Functional Requirements

Let's consider a developer who will be hosting multiple
database applications on a server, sometimes instances of the
same application for different customers. The applications
themselves will have different needs, but they all boil down
to this:

• Some applications will allow surfers to join the site
  and create accounts for themselves, while others will be
  private sites where an administrator must make user accounts.
  
• Some applications will not contain sensitive data, and
  so the site owner wants to send forgotten passwords in email
  -- which means the passwords must be stored in plaintext. Other
  site owners will need heightened security that disallows
  storing of passwords in plaintext.
  
• In both cases, administrators must of course be able to
  manage accounts themselves.
  
• The system should be structurally immune
  to SQL injection.
  
• It must be possible to have users with the same user id
  ("Sheilia", "John", etc.) on multiple applications who are
  actually totally different people.
  
• The application code must never need to run at an
  elevated privelege level for any reason -- not
  even to create accounts on public sites where
  users can join up and conduct transactions.
  
• It must be possible for the site owners or their
  agents to directly
  connect to the database at very least for querying and
  possibly to do database writes without going through our
  application.
  
• Users with accounts on one app must never be able to
  sign on to another app on the same server.
  

These requirements represent the most flexible possible
combination of demands that I have so far seen in real life.
The question is, can they be met while still providing
security? The model I'd like to speculate on today says
yes.

Informed Paranoia Versus Frightened Ignorance

Even the most naive programmer knows that the internet
is not a safe place, but all too often a lot of security
advice you find is based on frightened ignorance
and takes the form, "never do x, you don't know what might
happen." If we are to create a strong security model,
we have to do better than this.

Much better is to strive to be like a strong system architect,
whose approach is based on informed paranoia.
This hypothetical architect knows everybody is out
to compromise his system, but he seeks a thorough knowledge
of the inner workings of his tools so that he can
engineer the vulnerabilities out as much as possible.
He is not looking to write rules for the programmer
that say "never do this", he is rather looking to make it
impossible for the user or programmer to compromise
the system.

Two Examples

Let us consider a server hosting two applications, which
are called "social" and "finance".

The "social" application is a social networking site with
minimal security needs. Most important is that the site
owners want members of the general public to sign up, and
they want to be able to email forgotten passwords
(and we can't talk them out of it) -- so we
have to store passwords in plaintext.

The "finance" application is a private site used by employees
of a corporation around the world. The general public is
absolutely not welcome. To make matters worse however, the
corporation's IT department demands to be able to directly
connect to the database and write to the database without
going through the web app. This means the server will have
an open port to the database. Sure it will be protected with
SSL and passwords, but we must make sure that only users
of "finance" can connect, and only to their own application.

Dispensing With Single Sign-On

There are two ways to handle connections to a database. One
model is to give users real database accounts, the other is
to use a single account to sign on to the database. Prior to
the web coming along, there were proponents of both models in
the client/server world, but amongst web developers the single
sign-on method is so prevalent that I often wonder if they
know there is any other way to do it.

Nevertheless, we must dispense with the single sign-on method
at the start, regardless of how many people think that Moses
carved it on the third tablet, because it just has too many
problems:

• Single Sign-on is the primary architectural flaw that makes
  SQL injection possible. As we will see later, using real
  database accounts makes your site (almost) completely immune
  to SQL injection.
  
• Single Sign-on requires a connection at the maximum privilege
  level that any system user might have, where the code then decides
  what it will let a particular user do. This is a complete
  violation of the requirement that code always run at the lowest
  possible privilege level.
  
• Single Sign-on totally prevents the requirement that
  authorized agents be allowed to connect to the database and
  directly read and write values.
  

So single sign-on just won't work with the requirements listed.
This leads us to creating real accounts on the database server.

Real Accounts and Basic Security

When you use a real database account, your code connects
to the database using the username and password provided
by the user. Anything he is allowed to do your code will
be allowed to do, and anything he is not allowed to do will
throw and error if your code tries to do it.

This approach meets quite a few of our requirements nicely.
A site owner's IT department can connect with the same
accounts they use on the web interface -- they have
the same privileges in both cases. Also, there is no
need to ever have application code elevate its privilege
level during normal operations, since no regular users should ever be
doing that. This still leaves the issue of how to create
accounts, but we will see that below.

A programmer who thinks of security in terms of what code
can run will have a very hard time wrapping his head around
using real database accounts for public users. The trick to
understanding this approach
is to forget about code for a minute and to
think about tables. The basic fact of database application
security is that all security
resolves to table permissions. In other words, our security
model is all about who can read or write to what tables, it is
not about who can run which program.

If we grant public users real database accounts, and they
connect with those accounts, the security must be handled
within the database itself, and it comes down to:

• Defining "groups" as collections of users who share
  permissions at the table level.
  
• Deciding which groups are allowed select, insert, update,
  and delete privileges on which tables.
  
• Granting and revoking those privileges on the server itself
  when the database is built.
  
• At very least row-level security will be required, wherein
  a user can only see and manipulate certain rows in a table.
  This is how you keep users from using SQL Injection to mess
  with each other's order history or member profiles.
  
• Column security is also very nice to finish off the
  picture, but we will not be talking about that today as it
  does not play into the requirements.
  

Now we can spend a moment and see why this approach eliminates
most SQL Injection vulnerabilities. We will imagine a table of
important information called SUPERSECRETS. If somebody could
slip in a SQL injection exploit and wipe out this table we'd all
go to jail, so we absolutely cannot allow this.
Naturally, most users would have no privileges on
this table -- even though they are directly connected to the
database they cannot even see the table exists, let alone
delete from it. So if our hypothetical black hat
somehow slips in ";delete from supersecrets"
and our code fails to trap for it, nothing happens. They have
no privlege on that table. On the other side of things, consider
the user who is privileged to delete from that table. If this
user slips in a ";delete from supersecrets" he is only going to
the trouble with SQL Injection to do something he is perfectly
welcome to do anyway through the user interface. So much
for SQL injection.

To repeat a point made above: row-level security is a must.
If you grant members of a social site global UPDATE privileges
on the PROFILES table, and you fail to prevent a SQL Injection,
all hell could break loose. Much better is the ability to
limit the user to seeing only his own row in the PROFILE table,
so that once again you have created a structural immunity
to SQL injection.

Anonymous Access

Many public sites allow users to see all kinds of information
when they are not logged on. The most obvious example would
be an eCommerce site that needs read access to the ITEMS table,
among others. Some type of anonymous access must be allowed
by our hypothetical framework.

For our two examples, the "social" site might allow limited
viewing of member profiles, while the "finance" application
must show absolutely nothing to the general public.

If we want a general solution that fits both cases, we opt
for a deny-by-default model and allow each application
to optionally have an anonymous account.

First we consider deny-by-default. This means simply that
our databases are always built so that no group has any
permissions on any tables. The programmer of the "social"
site now has to grant certain permissions to the anonymous
account, while the programmer of the "finance" application
does nothing - he already has a secure system.

But still the "finance" site is not quite so simple. An anonymous
user account with no privileges can still log in, and
that should make any informed paranoid architect nervous.
We should extend
the deny-by-default philosophy so the framework will
not create an anonymous
account unless requested. This way the programmer of the
"finance" application still basically does nothing, while
the programmer of the "social" must flip a flag to create
the anonymous account.

Virtualizing Users

If we are having real database accounts, there is one small
detail that has to be addressed. If the "social" site has
a user "johnsmith" and the finance application has a user
of the same name, but they are totally different people,
we have to let both accounts exist but be totally separate.

The answer here is to alias the accounts. The database
server would actually have accounts "finance_johnsmith" and
"social_johnsmith". Our login process would simply take
the username provided and append the code in front of it
when authenticating on the server. 'nuf said on that.

Allowing Public Users To Join

The "social" site allows anybody to join up and create
an account. This means that somehow the web application
must be able to create accounts on the database server.
Yet it must do this without allowing the web code to
elevate its privileges, and while preventing the disaster
that would ensue if a user on the "social" site somehow
got himself an account on the "finance" site.

Believe it or not, this is the easy part! Here is how it
works for the "social" site:

• Create a table of users. The primary key is the user_id
  which prevents duplication.
  
• For the social site, there is a column called
  PASSWORD that stores the password in plaintext.
  
• Allow the anonymous account to INSERT into this table!
  (Remember though that deny-by-default means that so far
  this account has no other privileges).
  
• Put an INSERT trigger on the table that automatically creates
  an aliased user account, so that "johnsmith" becomes
  "social_johnsmith". The trigger also sets the password.
  
• A DELETE trigger on the table would delete users if
  the row is deleted.
  
• An UPDATE trigger on the table would update the password
  if the user UPDATES the table.
  
• Row level security is an absolute must.
  Users must be able to
  SELECT and UPDATE table, but only their own row. If your
  database server or framework cannot support row-level
  security, it's all out the window.
  

This gives us a system that almost gets us where we need
to be: the general public can create acounts,
the web application does not need to elevate its privileges,
users can set and change their passwords, and no user can
see or set anything for any other user. However, this leaves
the issue of password recovery.

In order to recover passwords and email them to members of
the "social" site, it is tempting to think that
the anonymous account must be able to
somehow read the users table, but that is no good because
then we have a structural flaw where a successful
SQL injection would expose user accounts. However, this
also turns out to be easy. There are two options:

• Write a stored procedure that the anonymous user is
  free to execute, which does not return a password but
  actually emails it directly from within the database
  server. This requires your database server be able to
  send emails. (Postgres can, and I assume SQL Server
  can, and I don't really know about mySql).
  
• Create a table for password requests, allow inserts
  to it but nothing else. A trigger sends the email.
  In this approach you can track email recovery requests.
  

For the "finance" application we cannot allow any of this
to happen, so again we go to the deny-by-default idea. All
of the behaviors above will not happen unless the programmer
sets a flag to turn them on when the database is built.

This does leave the detail of how users of the "finance"
application will reset their passwords.
For details on how a secure app can still allow password
resets, see my posting of Sept 7 2008 "http://database-programmer.blogspot.com/2008/09/advanced-table-design-secure-password.html"
>Secure Password Resets.

One More Detail on Public Users

We still have one more detail to handle for public users.
Presumably a user, having joined up, has more privileges than
the anonymous account. So the web application must be able
to join them into a group without elevating its privileges.
The solution here is the same as for creating the account:
there will be a table that the anonymous user can make
inserts into (but nothing else), and a trigger will join
the user to whatever group is named.

Except for one more detail. We cannot let the user join
whatever group they want, only the special group for members.
This requirement can be met by defining the idea of a "freejoin"
group and also a "solo" group. If the anonymous user inserts
into a user-group table, and the requested group is flagged
as allowing anybody to join, the trigger will allow it, but
for any other group the trigger will reject the insert.
The "solo" idea is similar, it means that if a user is in
the "members" group, and that group is a "solo" group, they
may not join any other groups. This further jails in
members of the general public.

Almost Done: User Administration

In the last two sections we saw the idea of a table of users
and a cross-reference of users to groups. This turns out to
solve another issue we will have: letting administrators
manage groups. If we define a group called "user_administrators"
and give them total
power on these tables, and also give them CRUD screens
for them, then we have a user administrator system.
This works for both the "social" and the "finance" application.

The triggers on the table have to be slightly different
for the two cases, but that is a small exercise to code
them up accordingly.

Cross-Database Access

Believe it or not, the system outlined above has met all of
our requirements except one. So far we have a system that never
requires the web server to have any elevated priveleges within
the database, allows members of the public to join some sites
while barring them from others, is structurally immune from
SQL injection, allows different people on different sites to
have the same user id, and allows administrators
of both sites to directly manage accounts. Moreover, we
can handle both plaintext passwords and more serious
reset-only situations.

This leaves only one very thorny issue: cross-database
access. The specific database server I use most is PostgreSQL,
and this server has a problem (for this scenario) anyway,
which is that out-of-the-box, a database account can connect
to any database. This does not mean the account has any
priveleges on the database, but we very seriously do not want
this to happen at all. If a member of the "social" site can
connect to the "finance" app, we have a potential vulnerability
even if he has zero privileges in that database. We would be
much happier if he could not connect at all.

In Postgres there is a solution to this, but I've grown to
not like it. In Postgres you can specify that a user can only
connect to a database if they are in a group that has the
same name as the database. This is easy to set up, but it
requires changing the default configuration of Postgres.
However, for the sheer challenge of it I'd like to work out
how to do it without requiring that change. So far I'm
still puzzling this out. I'd also like to know that the
approach would work at very least on MS SQL Server and
mySql.

Conclusion

Most of what is in this week's essay is not that radical to
any informed database veteran. But to web programmers
who were unfortunate enough to grow up in the world
of relational-databases-must-die nonsense, it is probably
hard or impossible to imagine a system where users are
connecting with real database accounts. The ironic thing
is that the approached described here is far more secure
than any single sign-on system, but it requires the programmer
to shift thinking away from action-based code-centric models
to what is really going on: table-based privileges. Once
that hurdle is past, the rest of it comes easy.