How accurate is this conversion function?

Because only the top 16 bits are used, the actual function we are dithering to match is a staircase, with each step a point where the top 16 bits of the floating point number change. (note that the plotting program tilted the vertical sides of the stairsteps slightly).

The following snapshots compare the proper to_sRGB() function (yellow) with the function actually delivered by the lookup table (red). I zoomed in on the two points shown as gray squares in the left image. The worst error is at .5 (right picture), where at several points the error approaches .5 of an output pixel. The error is reduced as you approach 1 due to the curve's derivative becoming less. The accuracy is much better in normal image colors, which are around .18 and well below the .25 that is shown in the middle graph.

The staircase is irregular because the table was filled in so that bytes read in convert back to the same bytes with no dithering. This can be seen as the stairsteps that land exactly on a horizontal integer line. If this is not done the error never exeeds 1/3 of a pixel. However solid 8-bit grays read in and then written out would come out dithered and shifted in intensity, this is unacceptable if the image is to be processed repeatedly.


Back to main page