Images: Scaling JPEGs and PNGs

The Problem

Images -- especially JPEG images -- can be huge.

Last weekend for example: I went to a Demetri Martin booksigning. I got my photo taken with the author with my phone. I tried to upload it, and I was surprised to see it was 800+ KB, at about 2500 x 2000 pixels.

On the one hand: this is great, right? I had no idea my phone had that level of resolution.

On the other hand: can your Java app effortlessly open up and work with this image? Ours can't. We have a painting application where the canvas size is around 1,000 pixels. The problem is: we have to open the JPEG at its original size and then scale it down:

(2500 pixels wide) x (2000 pixels tall) x (4 bytes*) = 19 MB

... but what we ultimately want is a 1000 x 800 pixel image: about 3 MB. We have to churn through 22 MB of memory to get what we want. And while this image is bigger than I expected, it's not "that big". Our testing team found an image to test with that's 20,000 x 15,000 pixels. That's 1.1 GB of memory uncompressed. Our app can't spare 1.1 GB of RAM to open (or create a thumbnail of) an image.

(This article is a little longer than I'd like, but I promise it ends well. Feel free to skip to the charts below if you're in a hurry.)

*I used 4 bytes because although a JPEG is opaque and only requires 3 bytes, in our apps we nearly always convert it to a TYPE_INT_RGB. This is backed by an int-based raster, so whether we use the 4th byte or not: it's allocated.

The Context

I was still in college when Java 1.4 was released. But even then I remembered discovering the ImageIO classes: why would anyone use anything else? In fact when I did start work a year or two later: I had to support some legacy Java 1.3 code. Loading java.awt.Image objects instead of java.awt.image.BufferedImage objects seemed ridiculous: why would you want to use an Image when you could use a BufferedImage?

But now I've come to the conclusion that the Image class is hugely underrated. (And I'm surprised to see it still does its job so well, several years later when I really needed it.)

The Existing Architecture

The Image architecture is intentionally abstract -- and perhaps intentionally cryptic. There are 3 important classes related to images:

ImageObserver: this simply receives updates about an image. Things like the width and height, and whether the image is fully loaded.

ImageProducer: this is the big hairy abstract entity that actually creates pixel data. Every Image has an ImageProducer behind it.

ImageConsumer: this listens to an ImageProducer and makes sense of the incoming data. We could use this to construct a BufferedImage, for example. This is the object we're interested in for this article.

The Solution

What I want to do now is devise an ImageConsumer that does not buffer an image in memory: instead it pipes the pixel data through efficiently and scales the image in one pass.

There is a very important red flag here: according to the very loose relationship between an ImageProducer and an ImageConsumer: the pixels may be delivered in any order. They could be tiles. They could come in non-continuous rows. You don't know. It so happens in all my tests that the default JPEG decoder does deliver the pixel data in a single pass -- one row at a time. But this is not guaranteed, and it may change at any time. (This is a behind-the-scenes detail that you don't have any control over unless you write your own JPEG reader.) I'll come back to this point later.

To address the vague contract of the ImageConsumer: I wrote the PixelIterator for my own use a while ago: it promises pixel data will be delivered one consecutive row at a time. (Either top-to-bottom or bottom-to-top.) Then I used this class in this article to dynamically scale images: the ScalingIterator filters a large image and scales it on-the-fly to a smaller set of dimensions. The resulting image is antialiased, although the rounding is vague and not strictly adhering to any set of interpolation rules.

So what is missing in this system is the class that converts a java.awt.Image to a PixelIterator so the ScalingIterator can process it. To fill this niche I wrote the GenericImageSinglePassIterator. The ImageProducer passes data to this object in one thread, and as calls to myIterator.next(...) are made this data is processed and passed on to the caller. The ImageProducer thread is blocked as pixel data is released until that data is requested: this guarantees data is not kept in memory unnecessarily. (Either thread will timeout after 5 seconds of no feedback, though, and report than an error has occurred.)

Results

I wrote a class called ScalingJPEGDemo to demonstrate this class. It uses this image from a recent vacation. This image is 2750 x 2063 pixels, or 21 MB uncompressed.

I wrote a handful of other scaling methods to compare against. In the ScalingJPEGDemo class I defined this interface:

static interface ScalingReader {
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable;
   public String getName();
 }
}

(The new code I developed for this project can handle incoming images much larger than 2750 x 2063, but some of the other contenders can't.)

These are the different readers I compared:

SinglePassReader

This is the new class I wrote (described above). It is fast and antialiased, but most importantly it leaves a low memory footprint.

ScalingReader singlePassReader = new ScalingReader() {
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   URL url = getImageURL();
   return GenericImageSinglePassIterator.createScaledImage( url, maxSize);
 }

 public String getName() {
   return "singlePassReader";
 }
};

scalingGraphicsReader

When I searched online, this is one of the two most commonly recommended approaches to scaling graphics. This relies on the Graphics2D interpolation rules to scale the incoming image. The problem is: if the scaling factor is less than 50%, then the resulting image looks aliased.

ScalingReader scalingGraphicsReader = new ScalingReader() {
 Component component = new JPanel();
 MediaTracker mediaTracker = new MediaTracker(component);
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);

   URL url = getImageURL();
   Image image = Toolkit.getDefaultToolkit().createImage(url);
   mediaTracker.addImage(image, 0);
   mediaTracker.waitForAll();

   BufferedImage thumbnail = new BufferedImage(newSize.width, newSize.height, BufferedImage.TYPE_INT_ARGB);
   Graphics2D g = thumbnail.createGraphics();
   g.setRenderingHint(RenderingHints.KEY_INTERPOLATION,  RenderingHints.VALUE_INTERPOLATION_BILINEAR);
   g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
   g.drawImage(image, 0, 0, newSize.width, newSize.height, null);
   g.dispose();

   mediaTracker.removeImage(image);
   image.flush();

   return thumbnail;
 }

 public String getName() {
   return "scalingGraphicsReader";
 }
};

transformReader

This is practically the same as the previous model. The only difference is this uses an AffineTransform. You never know how graphics pipelines process these requests, and over the years I've discovered some bizarre performance issues when you paint something two different ways. However in this case: these two image-scaling methods performed identically.

ScalingReader transformReader = new ScalingReader() {
 Component component = new JPanel();
 MediaTracker mediaTracker = new MediaTracker(component);
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);

   URL url = getImageURL();
   Image image = Toolkit.getDefaultToolkit().createImage(url);
   mediaTracker.addImage(image, 0);
   mediaTracker.waitForAll();

   BufferedImage thumbnail = new BufferedImage(newSize.width, newSize.height, BufferedImage.TYPE_INT_ARGB);
   Graphics2D g = thumbnail.createGraphics();
   double scale = ((double)newSize.width)/((double)originalSize.width);
   g.scale(scale, scale);
   g.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR);
   g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
   g.drawImage(image, 0, 0, null);
   g.dispose();

   mediaTracker.removeImage(image);
   image.flush();

   return thumbnail;
 }

 public String getName() {
   return "transformReader";
 }
};

scaledInstanceReader

This is the approach most commonly insulted as "the old-fashioned way" to scale images. But it's still a legitimate approach. In my opinion it also looks better (because it is antialiased) than the two approaches above.

ScalingReader scaledInstanceReader = new ScalingReader() {
 Component component = new JPanel();
 MediaTracker mediaTracker = new MediaTracker(component);
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);

   URL url = getImageURL();
   Image image = Toolkit.getDefaultToolkit().createImage(url);
   mediaTracker.addImage(image, 0);
   mediaTracker.waitForID(0);

   Image image2 = image.getScaledInstance(newSize.width, newSize.height, Image.SCALE_SMOOTH);
   mediaTracker.addImage(image2, 1);
   mediaTracker.waitForID(1);

   mediaTracker.removeImage(image);
   mediaTracker.removeImage(image2);
   image.flush();

   return image2;
 }

 public String getName() {
   return "scaledInstanceReader";
 }
};

replicateScaleFilter

This is an interesting class I had never used before. Although the API promises it will be an aliased image, I was interested to see how quickly it performed.

ScalingReader replicateScaleFilter = new ScalingReader() {
 Component component = new JPanel();
 MediaTracker mediaTracker = new MediaTracker(component);
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);

   URL url = getImageURL();
   Image image = Toolkit.getDefaultToolkit().createImage(url);
   mediaTracker.addImage(image, 0);
   mediaTracker.waitForAll();

   ImageProducer producer = image.getSource();
   ImageFilter scalingFilter = new ReplicateScaleFilter(newSize.width, newSize.height);
   ImageProducer scaledProducer = new FilteredImageSource(producer, scalingFilter);
   Image image2 = Toolkit.getDefaultToolkit().createImage(scaledProducer);

   mediaTracker.addImage(image2, 1);
   mediaTracker.waitForID(1);

   mediaTracker.removeImage(image);
   mediaTracker.removeImage(image2);

   return image2;
 }

 public String getName() {
   return "replicateScaleFilter";
 }
};

areaAveragingScaleFilter

This is similar to the replicateScaleFilter, except this filter antialiases pixels. It will be more expensive, but the results will be better.

ScalingReader areaAveragingScaleFilter = new ScalingReader() {
 Component component = new JPanel();
 MediaTracker mediaTracker = new MediaTracker(component);
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);

   URL url = getImageURL();
   Image image = Toolkit.getDefaultToolkit().createImage(url);
   mediaTracker.addImage(image, 0);
   mediaTracker.waitForAll();

   ImageProducer producer = image.getSource();
   ImageFilter scalingFilter = new AreaAveragingScaleFilter(newSize.width, newSize.height);
   ImageProducer scaledProducer = new FilteredImageSource(producer, scalingFilter);
   Image image2 = Toolkit.getDefaultToolkit().createImage(scaledProducer);

   mediaTracker.addImage(image2, 1);
   mediaTracker.waitForID(1);

   mediaTracker.removeImage(image);
   mediaTracker.removeImage(image2);

   return image2;
 }

 public String getName() {
   return "areaAveragingScaleFilter";
 }
};

imageIOReader

As far as I can tell: the ImageIO package doesn't support scaling, but it does support subsampling. In addition to antialiasing, this presents a new problem none of the other readers face:

Subsampling only works in discrete intervals. Suppose you want to subsample a 400 x 400 image with a subsample factor of 2: you'll get a 200 x 200 pixel image. Now suppose you use a subsample factor of 4: you'll get a 100 x 100 pixel image. Now suppose you wanted an image that was 150 x 150 pixels? You're out of luck. There is no subsample factor that gives you this result. The reader below calculates a subsample factor, but still provides a BufferedImage of the requested size: the remaining pixels are filled in with black.

The thumbnail to the right is "lucky", because it only introduces 1 row and column of black. But in other cases, or with other pixel dimensions, this extra padding can be several dozen pixels.

final ImageReader reader = (ImageReader)ioReaderList.get(a);
ScalingReader imageIOReader = new ScalingReader() {
 public Image create(Dimension originalSize,Dimension maxSize) throws Throwable {
   URL url = getImageURL();
   Dimension newSize = Scaling.scaleDimensionsProportionally(originalSize, maxSize);
   InputStream in = url.openStream();
   try {
     ImageInputStream imageIn = ImageIO.createImageInputStream( in );
     reader.setInput(imageIn);
     int w = reader.getWidth(0);
     int h = reader.getHeight(0);

     int sub = 1;
     if (w>h)
       sub = MathG.ceilInt(w/maxSize.getWidth());
     else
       sub = MathG.ceilInt(h/maxSize.getHeight());

     BufferedImage bi = new BufferedImage(newSize.width, newSize.height,BufferedImage.TYPE_INT_RGB);

     ImageReadParam param = reader.getDefaultReadParam();
     param.setSourceSubsampling(sub,sub,0,0);
     param.setDestination(bi);
     return reader.read(0,param);
   } finally {
     in.close();
   }
 }

 public String getName() {
   if(ioReaderList.size()==1) {
     return "imageIOReader";
   } else {
     String s = reader.getClass().getName();
     if(s.indexOf('.')!=-1) {
     s = s.substring(s.lastIndexOf('.')+1);
     }
     return "imageIOReader<"+s+">";
   }
 }
};

I tested these 7 methods with my test image, measuring the time and memory allocation each used. The memory allocation measurement is shaky because I'm simply referring to Runtime.getRuntime().freeMemory(), but it should give us a general idea. (However an unplanned garbage collection can skew the results.) In the tests I slowly incremented the destination image width by 100 pixels (from a 100 pixel wide copy to a 2000 pixel wide copy).

I collected results from 6 different environments (including XP, Mac, Windows 7, and Linux). All the results here, but for brevity's sake I'll just show off the results from Windows 7. (Not because it's remarkably better or worse than other OS's, but because I assume Windows 7 is incredibly common, and to show all the results would bloat this article.)

Here is a graph of the time each method required:

The slowest methods here are areaAveragingScaleFilter and scaledInstanceReader. Based on this chart I'd suggest they're the same function (that is: I believe Image.getScaledInstance() uses an AreaAveragingFilter). But other than my new class: these are the only two models that can antialias if you scale more than 50%.

Next in line is the replicateScaleFilter. This is the slowest of the aliased methods.

Next is the new class I wrote: it's around .8 seconds. It's by no means the fastest method, but it is the fastest antialiased method.

The other results are a little garbled. I'd suggest the transformReader and the scalingGraphicsReader are also identical under the hood, based on these numbers. The imageIOReader appears to jump around at specific intervals: I assume this relates to when the subsample factor increases.

But none of that is terribly important to me: I'm OK with asking the user to tolerate a 2-second delay when they give me a megapixel JPEG. What I'm interested in is the memory allocation:

Just like before: the transformReader and scalingGraphicsReader overlap so much its hard to notice there are 2 results. And the areaAveragingScaleFilter and scaledInstanceReader results are also so close they also seem identical.

There are two clear winners in this chart: imageIOReader and the new singlePassReader.

The imageIOReader, however, doesn't let you scale to arbitrary dimensions. For example: when we attempt to scale the 2750 pixel wide image to 2000 pixels: we use a subsample factor of 2. So the resulting image is 1375 pixels wide. The remaining 625 pixels (approximately 30% of the width) are black.

That leaves the singlePassReader as my winner. Was this contest fair? Probably not. I created the tests, and I created a custom-tailored solution to pass the tests. But I don't think my goals (low memory allocation and antialiasing) are asking too much; and I'm sharing this solution because I believe other developers might want the same results.

I could discuss the results of each OS in detail, but the broad conclusions are the same in each case. Some other charts show strong dips in memory at certain levels: I assume this is unsolicited garbage collection at work. This raises the interesting point that although other methods may churn through several MB of memory: they don't necessarily keep it all in use. So that's promising. I haven't tried to study exactly how much they would require if they had constant garbage collection. (For example: if you were in an environment with a fixed amount of memory and frequent garbage collection, which methods could still scale the incoming image?)

Conclusion

These results are great. This lets me efficiently create thumbnails and open images at a maximum resolution without burning through half my app's memory.

You can download the app that created the output above here. Note it includes a large embedded JPEG to test with. You can download a smaller package that includes these classes here.

This article focused on talking about JPEGs, but the images all came from the abstract Toolkit.createImage(...), which supports JPEGs as well as PNGs. And PNGs are stored in an orderly row-based model, so they should automatically work with this architecture, too.

However. It is important to note that this code is not guaranteed to work because it assumes the image data can be collected in a single pass. Which is to say: I still definitely plan on using this class, but I'll always wrap it in try/catch clauses in case it fails.

Someday I'd like to also develop the GenericImageRandomAccessIterator. For this I'm envisioning a RandomAccessFile-based model. This will work with arbitrary passes, and it may require even less memory: but it will be noticeably slower because it relies on the file system.

Text: Effects and Block Shadows

This weekend I revisited an old project: the calligraphy stroke.

I rewrote it using a handy unassuming class called the CalligraphyPathWriter. This is fewer lines of code than the previous model, pipes the data more efficiently, and includes a couple of extra functions.

It no longer only makes strokes. The same basic notion of tracing a shape to apply a calligraphy-like stroke can be applied to creating drop shadows. The only difference is a stroke straddles the path, and a drop shadow lies entirely to the side of the original shape.

The CalligraphyPathWriter constructor includes two floats to define these offsets: if you use 0 and 30: then you're effectively adding a drop shadow to your shape. If you use -5 and 5: then you're effectively creating a stroked outline of your shape.

You can also create a drop-shadow-like effect with code like this:

Graphics2D g;
Shape myShape;
...
double dx = Math.cos(angle);
double dy = Math.sin(angle);
for(int height = 0; height < 30; height++) {
   g.translate(height*dx, height*dy);
   g.fill(myShape);
   g.translate(-height*dx, -height*dy);
}

... but if your shape or java.awt.Paint are complex: this might be expensive. (Also two of the four effects shown below can't be reproduced this way.)

The CalligraphyPathWriter can reduce this down to:

GeneralPath blockShadow = new GeneralPath();
CalligraphyPathWriter writer = 
     new CalligraphyPathWriter(angle, 0, height, blockShadow);
writer.write(myShape);
g.fill(blockShadow);

The call write(myShape) iterates over the input shape and populates the blockShadow path with the silhouette of the block shadow.

Applying To Text

Here is a simple applet that creates animated text effects using these new block shadows:

You can download the applet (source included) here.

Type any text in the text field provided. (Any text is acceptable, but it should probably be 20 characters or less?) Then select any one of the buttons:

• The "Outline" button uses the Scribbler class to randomize the texture of the text, fill and shadow. A little random vertical offset is added, and lastly a light drop shadow is traced (but not filled) around each letter.
  
• The "Punch" button separately punches each word forward with an obvious drop shadow. (Also try writing a string with no spaces: then each character is punched.)
  
• The "Wave" button fills and traces the block shadow to reinforce the 3D-look.
  
• The "Explode" uses another obvious drop shadow, but each character is assigned an angle based on how far away it is from the center. (Then the height has to be divided by the sine of that angle so the baseline is constant.)

My immediate goal this weekend was just to use block shadows for something fun. In the long term I want to define an architecture similar to what I did for simple 2D-geometric transitions. This demo applet uses a very loosely defined TextEffect interface, but it needs a lot more work to support a variety of effects for different export formats.

Also in the future it would be nice to apply gradients to these block shadows. A height-based gradient effect will be very easy. An angle-based effect (which I think would be more popular) will be a little trickier, but is still manageable.

When does a practice become BEST?

How many good references does an idea need before it becomes a best practice?

For starters, Ola Hallengren's script for Backup, Index Maintenance, and DBCC checks.

I've done the research and spoke to Ola in person at the 2010 PASS Summit.

Ola's good references come from no less than Microsoft, Red Gate Software, Quest Software, and Microsoft MVPs.

I think Ola Hallengren's maintenance solution is a BEST PRACTICE.  

I have a few questions for you.  

Do you agree?  

Do you use Ola's maintenance solution?  

Do you use Ola's solution as is or have you modified it?  

Please respond in the comments.

Mouse Click Events: Adding Wiggle Room

MouseListeners have a mouseClicked() method, but it is only triggered if the mouse does not move between the mouse being pressed and later being released.

This is working as designed, but as touchpads and tablets become more available: I think this design is outdated. It is very easy on certain devices to press the mouse, accidentally move the mouse, and then release the mouse. (Even the word "mouse" here is outdated: we're dealing with fingers and tablet pens.)

The good news is: this does not affect buttons. JButtons use a more complicated model to trigger any associated ActionListeners (based on ButtonModels, armed and pressed states). The issue I'm describing only relates to when MouseListener.mouseClicked() is called.

Here is an applet that demonstrates the problem, and a potential solution:

When mouseClicked() is called the color changes to green. If it is called again then the color changes to yellow.

If you press the mouse down and move it a couple pixels before releasing: one label will highlight green and the other will not. (Based on one mouse click neither label should turn yellow: if it does then two calls to mouseClicked() are being made, and that's a serious bug.)

The pixel tolerance is a global static field. I picked 10 as an arbitrary number; you can fine tune this as needed.

Implementation

The ClickEventEnabler has only one method: install().

The current implementation just adds one AWTEventListener to the Toolkit. When mousePressed() and later mouseReleased() is called: we check to see if the distance between the mouse pressed and released location is too large.

If it is within an acceptable range, we create a Runnable to invoke later. If in the meantime a call mouseClicked() comes in for the same component: the runnable is cancelled. So if the user doesn't move the mouse, or if future Java implementations address this concern: this event listener won't send a redundant mouseClicked() event.

If everything passes inspection: then we create our own MouseEvent.MOUSE_CLICKED event and dispatch it ourselves.

So this requires top-level access to the event queue, but aside from possibly dispatching click events: it is very benign.

In the future: it might be useful to implement a similar change to recognize double-clicks. For now I'm pilot testing this single-click solution.

The javadoc link above references the source code, but there's also a jar to download here.

SQL Rally abstracts submitted

I've submitted two abstracts for the SQLRally.

If you're interested in learning PowerShell by example or want to hear how I automated our login provisioning and compliance reporting. Vote for my sessions.

It's a win-win situation in the PowerShell track for attendees no matter who you vote for.

Another great bunch of abstracts in the DBA track.

Get out and vote.

Maintaining One Code Base with Possibly Conflicting Custom Features

Today's essay deals with the tricky issue of custom features
for individual customers who are running instances of your
software.

The question comes by way of a regular reader who prefers to
remain anonymous, but asks this:

... I work on a large (to me, anyway) application that serves as a client database, ticket system, time-tracking, billing, asset-tracking system. We have some customers using their own instances of the software. Often, those customers want additional fields put in different places (e.g., a priority column on tickets). This results in having multiple branches to account for versions with slight changes in code and in the database. This makes things painful and time-consuming in the long run: applying commits from master to the other branches requires testing on every branch; same with database migrate scripts, which frequently have to be modified.

Is there an easier way? I have thought about the possibility of making things "optional" in the database, such as a column on a table, and hiding its existence in the code when it's not "enabled." This would have the benefit of a single code set and a single database schema, but I think it might lead to more dependence on the code and less on the database -- for example, it might mean constraints and keys couldn't be used in certain cases.

Restating the Question

Our reader asks, is it better to have different code branches
or to try to keep a lot of potentially conflicting and optional
items mixed in together?

Well, the wisdom of the ages is to maintain a single code branch,
including the database schema. I tried exactly once, very early
in my career, to fork my own code, and gave up almost within days.
When I went to work in larger shops I always arrived in a situation
where the decision had already been made to maintain a single
branch. Funny thing, since most programmers cannot agree on the
color of the sky when they're staring out the window, this is
the only decision I have ever seen maintained with absolute
unanimity no matter how many difficulties came out of it.

There is some simple arithmetic as to why this is so. If you have
single feature for a customer that is giving you a headache, and
you fork the code, you now have to update both code branches for
every change plus regression test them both, including the feature
that caused the headache. But if you keep them combined you only
have the one headache feature to deal with. That's why people
keep them together.

Two Steps

Making custom features work smoothly is a two-step process.
The first step is arguably more difficult than the second,
but the second step is absolutely crucial if you have
business logic tied to the feature.

Most programmers when confronted with this situation
will attempt to make various features optional. I
consider this to be a mistake because it complicates
code, especially when we get to step 2. By far the
better solution is to make features ignorable
by anybody who does not want them.

The wonderful thing about ingorable features is
they tend to eliminate the problems with apparently
conflicting features. If you can rig the features
so anybody can use either or both, you've eliminated
the conflict.

Step 1: The Schema

As mentioned above, the first step is arguably more
difficult than the second, because it may involve
casting requirements differently than they are
presented.

For example,
our reader asks about a priority column on tickets,
asked for by only one customer. This may seem like
a conflict because nobody else wants it, but we
can dissolve the conflict when we make the feature
ignorable. The first step involves doing this at
the database or schema level.

But first we should mention that the UI is easy,
we might have a control panel
where we can make fields invisible. Or maybe our
users just ignore the fields they are not interested
in. Either way works.

The problem is in the database.
If the values for priority come
from a lookup table, which they should,
then we have a foreign key, and
we have a problem if we try to ignore it:

• We can allow nulls in the foreign key, which is
  fine for the people ignoring it, but
  
• This means the people who require it can end
  up with tickets that have no priority because it does
  not prevent a user from leaving it blank.
  

A simple answer here is to pre-populate your priority
lookup table with a value of "Not applicable", perhaps
with a hardcoded id of zero. Then we set the default
value for the TICKET.priority to zero. This means people
can safely ignore it because it will always be valid.

Then, for the customer who paid for it, we just go in
after the install and delete the default entry. It's
a one-time operation, not even worth writing a script
for, and it forces them to create a set of priorities
before using the system. Further, by leaving the
default of zero in there, it forces valid answers
because users will be dinged with an FK violation if
they do not provide a real priority.

For this particular example, there is no step 2, because
the problem is completely solved at the schema level.
To see how to work with step 2, I will make up an
example of my own.

Step 2: Unconditional Business Logic

To illustrate step 2, I'm going to make up an
example that is not really appropriate to our
reader's question, frankly because I cannot think
of one for that situation.

Let's say we have an eCommerce system, and one
of our sites wants customer-level discounts based
on customer groups, while another wants discounts
based on volume of order -- the more you buy, the
deeper the discount. At this point most programmers
start shouting in the meeting, "We'll make them
optional!" Big mistake, because it makes for lots
of work. Instead we will make them ignorable.

Step 1 is to make ignorable features in the schema.
Our common code base contains a table of customer
groups with a discount percent, and in the customers
table we make a nullable foreign key to the customer
groups table. If anybody wants to use it, great, and
if they want to ignore it, that's also fine. We do
the same thing with a table of discount amounts,
we make an empty table that lists threshhold amounts
and discount percents. If anybody wants to use it
they fill it in, everybody else leaves it blank.

Now for the business logic, the calculations of
these two discounts. The crucial idea here is
not to make up conditional logic that tries to
figure out whether or not to apply the discounts.
It is vastly easier to always apply both
discounts, with the discounts coming out zero for
those users who have ignored the features.

So for the customer discount, if the customer's
entry for customer group is null, it will not match
to any discount, and you treat this as zero.
Same for the sale amount discount, the lookup to
see which sale amount they qualify doesn't find
anything because the table is empty, so it treats
it as zero.

So the real trick at the business logic level is
not to figure out which feature to use, which leads
to complicatec conditionals that always end up
conflicting with each other, but to always use
all features and code them so they have no effect
when they are being ignored.

Conclusion

Once upon a time almost everybody coding for a living
dealt with these situations -- we all wrote code that
was going to ship off to live at our customer's site.
Nowadays this is less common, but for those of us
dealing with it it is a big deal.

The wisdom of the ages is to maintain a common code
base. The method suggested here takes that idea
to its most complete implementation, a totally common
code base in which all features are active all of
the time, with no conditionals or optional features
(except perhaps in the UI and on printed reports),
and with schema and business logic set up so that
features that are being ignored simply have no
effect on the user.

Opinion: SQL Rally Abstract Process

I'm really enjoying the process that the SQL Rally is using to select speakers.

I've submitted abstracts to several SQL Saturdays and the PASS Summit but this has been the most interesting process so far. What I really liked about the SQL Rally submission process:

• Blind submission


• Two abstract limit


• Community votes for the final slate of speakers.

The blind submission process eliminates being concerned about what others will do. Submit what you have, if it's worthy, it will be selected. If not, move on.      

The two abstract limit levels the playing field for less experienced speakers who haven't compiled as much content but have gained enough speaking experience at the local level to move up to the regional level.  I think this limit also has produced better abstracts.  I think the candidates spent more time sharpening their abstracts because they only had two chances to get a slot.

Finally, letting the PASS community vote was a great idea but stretching the voting out over a month is brilliant from a marketing perspective.  The organizers have developed a very community focused method of creating maximum interest in this event.  Well done!

I'd like to see all or parts of this process used for future PASS events.