Phone Vision 04 – Premultiplied Alpha

13 01 2011

Previously we explored encoding pixels from color components.  This works perfectly if our alpha value is 255, and in most cases it will be.  But, what if we need transparency?

 

Pixel Format (again)

From MSDN:

WriteableBitmap Pixel Format in Silverlight

When assigning colors to pixels in your bitmap, use pre-multiplied colors. The format used by the Silverlight WriteableBitmap is ARGB32 (premultiplied RGB). The format becomes relevant if you are populating the integer values in the Pixels array.

The initial pixel values in the dimensioned array are 0, which will render as black if left unaltered.

I want to clarify one thing.  The documentation says the pixel values are set to 0 by default (which is correct) and that it will render as black if left unaltered (which is incorrect).  It will render as transparent because the alpha values are also 0.  Don’t believe me?  Try it.

 

Transparency

Why would we ever want transparency when we are processing images?  Perhaps we want to highlight a point of interest.  You could leave the original image intact and just place a semi-transparent image over the top of it and let Silverlight blend them for you.  Regardless of the need, here’s how you do it.

If we were to create an image with a blue background covered with a semi-transparent (say, alpha = 128) white square, this is the output we would expect:

image_thumb_thumb

If you remember the MSDN documentation referenced in the last lesson it noted that the pixel format was premultiplied ARGB32.  This means that the RGB value has already been multiplied before storing the pixel.

The Code

Now we are going to try to recreate the blue color above using two different techniques.  The one will be straight RGB (i.e. not premultiplied) and the other will be premultiplied alpha.

1.) First, let’s set up a couple of blue rectangles to test.

<Grid x:Name="LayoutRoot"

          Background="White">

        <Grid.RowDefinitions>

            <RowDefinition />

            <RowDefinition />

        </Grid.RowDefinitions>

        <Rectangle

            Width="300"

            Height="300"

            Fill="Blue" />

               

        <Rectangle

            Grid.Row="1"

            Width="300"

            Height="300"

            Fill="Blue" />       

    </Grid>

 

Running this simple program should produce:

 

image

 

2.) Now we want to create a swatch for comparison.  Add a New Item, select Windows Phone User Control and call it “Swatch” (for comparing the techniques):

image

 

3.) Replace Swatch’s default <Grid> with this:

    <Canvas Width="100" Height="100">

        <Rectangle

            Fill="Blue"

            Width="100"

            Height="100" 

            Stroke="Black"

            StrokeThickness="2" />

        <Rectangle

            Fill="White"

            Opacity=".5"

            Width="100"

            Height="100"

            Stroke="Black"

            StrokeThickness="2" />

    </Canvas>

All we are doing is overlaying a blue rectangle with a semi-transparent (Opacity=.5) white rectangle.

4.) Add Swatch to the MainPage.xaml for comparison:

<Grid x:Name="LayoutRoot"

        Background="White">

    <Grid.RowDefinitions>

        <RowDefinition />

        <RowDefinition />

    </Grid.RowDefinitions>

    <Rectangle

        Width="300"

        Height="300"

        Fill="Blue" />       

      

    <Rectangle

        Grid.Row="1"

        Width="300"

        Height="300"

        Fill="Blue" />

       

    <local:Swatch x:Name="SwatchControl"

        Margin="190,0,190,-50"

        VerticalAlignment="Bottom" />

</Grid>

This output should look something like this:

image

 

4.) Add the two overlay images:

    <Grid x:Name="LayoutRoot"

          Background="White">

        <Grid.RowDefinitions>

            <RowDefinition />

            <RowDefinition />

        </Grid.RowDefinitions>

        <Rectangle

            Width="300"

            Height="300"

            Fill="Blue" />

       

        <Image x:Name="StraightRGB"

            Grid.Row="0"

            Width="300"

            Height="300" />

       

        <Rectangle

            Grid.Row="1"

            Width="300"

            Height="300"

            Fill="Blue" />

       

        <Image x:Name="PremultipliedAlpha"

            Grid.Row="1"

            Width="300"

            Height="300"/>

       <local:Swatch x:Name="SwatchControl"

            Margin="190,0,190,-50"

            VerticalAlignment="Bottom" />

    </Grid>

The output will be the same as above because we haven’t actually set the source for these images.

4.) Finally, the code behind:

private void AcquireImageButton_Click(object sender, EventArgs e)

{

    WriteableBitmap straightRGBImage =

        new WriteableBitmap(300, 300);

 

    WriteableBitmap premultipliedAlphaImage =

        new WriteableBitmap(300, 300);

 

    for (int pixelIndex = 0; pixelIndex < 90000; pixelIndex++ )

    {

        byte alpha = 128;

        byte red = 255;

        byte green = 255;

        byte blue = 255;

 

        double scaleAlpha = alpha / 255.0;

 

        // we are not using scaleAlpha here

        straightRGBImage.Pixels[pixelIndex] =

                (alpha << 24)

                | (red << 16)

                | (green << 8 )

                | blue;

 

        // notice the alpha value is NOT scaled

        // it’s also very important to scale BEFORE

        // shifting the values

        premultipliedAlphaImage.Pixels[pixelIndex] =

                (alpha << 24)

                | ((byte)(red * scaleAlpha) << 16)

                | ((byte)(green * scaleAlpha) << 8 )

                | (byte)(blue * scaleAlpha);

 

    }

 

    StraightRGB.Source = straightRGBImage;

    PremultipliedAlpha.Source = premultipliedAlphaImage;

}

 

Output

Launch the program and tap the acquire button.  I’ll let you guess which one is right.

image

 

Summary

Premultiplied alpha is pretty straightforward, but it will cost performance in tight loops.  In most cases we won’t have to worry about this because the images we typically deal with don’t have transparencies.

Download Code

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2004%20-%20PremultipliedAlpha.zip

Up next: A Brief History of Color





Phone Vision – 03 Encoding Color

10 01 2011

Last time we learned how to extract the individual color values from a WriteableBitmap.  While this will allow us to analyze the image, we haven’t modified it in any way.  If we want to modify the color of the pixel in a meaningful fashion, we need to revisit the pixel format for a WriteableBitmap.

 

Encoding Color Components

Recall the format for the pixel is

image_thumb4

The assumption here is that we have the separated alpha, red, green, and blue components, but we want to combine them into the ARGB32 pixel format.  So, how do we encode those colors properly?  We can simply left shift each component value the correct number of bits then add them together.  Huh?

What we really want to do is something like this:

image

+

image

+

image

+

image

=

image_thumb4

Another Bitwise Operator

Left Shift

How can we shift the bits into their correct positions?  I bet you know where I’m headed with this – the << (LEFT SHIFT) operator:

operator input shift output
<< 1 1 10
<< 1 2 100
<< 1 3 1000
<< 1 4 10000

It should be relatively easy to see that the appropriate shifts are:

component shift
blue 0
green 8
red 16
alpha 24
 

Using our new found knowledge we can rewrite this as:

int pixel = (alpha << 24) + (red << 16) + (green << 8 ) + blue;

Cake.

 

A Note About Or

Above we used simple addition to combine the shifted color values, however, the traditional technique for combining these is the | (OR) operator.  Here is the table of outputs for |:

operator bit one bit two output
| 0 0 0
| 0 1 1
| 1 0 1
| 1 1 1

How can we use this?  Well, if we | a bit with 0 we get the value of the original bit.  (i.e. 1|0=1 and 0|0=0 – original bit in bold).  This means that we can get the exact same results above with this line of code:

int pixel = (alpha << 24) | (red << 16) | (green << 8 ) | blue;

Why would we want to do this?  My gut says: performance.  In my experimentation, however, it turns out to be pretty minor.  For 100,000,000,000 iterations (that’s 100 billion) it took 17 minutes 48 seconds for the + operation and 17 minutes 33 seconds for the | operation (on my dev machine).  So yes, the | operator is slightly more efficient, but barely.

I use the | version in my code, because image processing is intense and even small performance gains add up when you are looping through hundreds of thousands of pixels.

 

Transparency

It’s important to note that the above code only works correctly for alpha = 255. Since we are working with images this is not typically an issue.  In the next episode we will discuss how to handle transparency.

Summary

Encoding a color from its components is about as easy as we’d expect it to be assuming you understand some fundamentals of computer science.

Download Code

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2003%20-%20EncodingColors.zip

Up next: Premultiplied Alpha

 





Phone Vision – 02 Extracting Color

6 01 2011

Last time we set up a very simple project that will allow us to display and manipulate image data using a WriteableBitmap.  Before we can do anything cool though, we need to understand a few things about the WriteableBitmap class.

Pixel Collection

The WriteableBitmap has an integer array called Pixels that represents the 2D texture of the bitmap.  In most cases we will be looping through Pixels (using PixelWidth and PixelHeight) to perform our operations.  You can find the data for a pixel located at (x,y) in the WritableBitmap wb  with:

int color = wb.Pixels[x + y * wb.PixelWidth];

Without knowing how the color is stored we really can’t do much with this data.  Let’s take a peak under the hood.

Pixel Format

The format used by the Silverlight WriteableBitmap is ARGB32 (premultiplied RGB – we’ll cover this later).  This means that the color is represented by a 32-bit integer with the following format:

image

As you can see, each channel (alpha, red, green, and blue) uses 8 bits, giving a possible 28 or 256 intensities for each (0-255). 

 

Extracting Color Components

Extracting each component intensity can be done by masking the color value with 0xFF then shifting the number right by 8 bits.  The heck you say?

Bitwise Operators

And

As their name implies, bitwise operators operate on one bit at a time.  The operator we are concerned with for now is & (AND).

operator bit one bit two output
& 0 0 0
& 0 1 0
& 1 0 0
& 1 1 1

Notice in the & operator when a bit is 0 the output bit will always be 0.  Wherever there is a 1 in a bit, the result is always the value of the other bit.  This trait will allow us to create what is called a mask (or bitmask).  We cover the bits we don’t care about with 0.  So the mask 0xFF above in binary is…

image

& this with the pixel and we will get the value for the last 8  bits of the color.

Let’s work with an example.  I randomly picked #FFDB91D6.  #FFDB91D6 in binary:

image

&

image

=

image

 

Voila.

  

 

Right Shift

 

If we were to use the & alone we could only recover the values for the blue channel.  If only we could shift the color bits to match our mask…   Turns out there is another handy bitwise operator for doing just that: >> (RIGHT SHIFT)

 

operator input shift output
>> 10001000 1 01000100
>> 10001000 2 00100010
>> 10001000 3 00010001
>> 10001000 4 00001000

 

Notice how the bits shift right?  This is equivalent to dividing by powers of 2.  Don’t let the leading zeros confuse you.  If you divide 1120 by 10 (in base 10) you get 0112.  If you divide again you get 0011 (drop the remainder).  See?

 

image

>> 8

=

image

 

Now the green bits match our mask.  Hooray!

 

Full Example

 

Let’s run through our example from above (#FFDB91D6) in its entirety:

 

1.) First, let’s get the blue bits:

 

image

&

image

=image

 

 

2.) Shift the bits right 8 places:

 

image

>> 8

=

image

 

 

3.) Recover the green bits:

 

image

&

image

=

image 

 

 

4.) Shift the bits right another 8 places:

 

image

>> 8

=

image

 

 

5.) Extract the red bits:

 

image

&

image

=image

 

 

6.) Last shift!!!

 

image

>> 8

=

image

 

 

7.) Extract the alpha bits:

 

 image

=image

 

 

We don’t have to mask the last value because it’s already exactly what we need.  Let’s see how this plays out in code.

 

int pixel = (int)0xFFDB91D6;

byte blue = (byte)(pixel & 0xFF);

pixel >>= 8;

byte green = (byte)(pixel & 0xFF);

pixel >>= 8;

byte red = (byte)(pixel & 0xFF);

pixel >>= 8;

byte alpha = (byte)(pixel);

Quite elegant and very efficient.

 

Summary

Recovering colors from the WriteableBitmap is pretty straightforward once you understand the pixel structure and how to mask them.

Download Code

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2002%20-%20ExtractingColors.zip

Note:  This code doesn’t actually modify the image in any way.  We will do that in the next installment.

 

Up next: Encoding Color