Phone Vision 08 – Intro to Histograms

27 01 2011

Now that we’ve finally manipulated a few images we’re starting to get somewhere.   Today we are going to take some strides in classifying images.  It would be nice if we had an objective way to say “that image is really bright”.  It shouldn’t come as a surprise that this is actually pretty easy.

What is a Histogram?

Histogram is really just a fancy word for counting.  In our case, we are counting how many pixels are at each intensity level between black and white.  Let’s revisit our friendly orangutan.

The chart below the picture is his histogram.  The horizontal-axis represents the intensity while the vertical-axis represents the number of pixels with that intensity.

Creating the Histogram

When you are converting an image to grayscale you can trivially create the histogram data with it.  The process is as simple as looping through every pixel (which we have to do to convert to grayscale anyway) and adding up the intensities.

//we are going to use the

//average of the r, g, and b

//values for our grayscale conversion

WriteableBitmap ToAverageGrayscale(WriteableBitmap bmp)

{

WriteableBitmap grayscale =

new WriteableBitmap(bmp.PixelWidth, bmp.PixelHeight);

//there are 256 intensities

//so a simple array of the ints

//should suffice for holding the counts

int[] histogramData = new int[256];

//we will use this to determine

//what the number of the highest

//occurring intensity is

//we are simply going to use this value

//to scale our plot

int maxCount = 0;

for (int pixelIndex = 0;

pixelIndex < bmp.Pixels.Length;

pixelIndex++)

{

int pixelColor = bmp.Pixels[pixelIndex];

byte red = (byte)((pixelColor & 0xFF0000) >> 16);

byte green = (byte)((pixelColor & 0xFF00) >> 8);

byte blue = (byte)(pixelColor & 0xFF);

//this is not rocket science

byte intensity = (byte)((red + green + blue) / 3);

//once again we repeat the intensity

grayscale.Pixels[pixelIndex] = 255 << 24

| intensity << 16

| intensity << 8

| intensity;

//simply add another to the count

//for that intensity

histogramData[intensity]++;

if (histogramData[intensity] > maxCount)

{

maxCount = histogramData[intensity];

}

}

//this would typically be a completely

//separate function, but it’s easier

//in this case to perform the operation

//with the grayscale conversion

PlotHistogram(histogramData, maxCount);

return grayscale;

}

public void PlotHistogram(int[] counts, int maxCount)

{

//in MainPage we have a canvas that will

//hold the histogram rectangles

HistogramPlot.Children.Clear();

//our tallest rectangle will be the height

//of the canvas, so we want to scale the

//them down from there

double scale = HistogramPlot.Height / maxCount;

//in our case we will always have 256 counts

//but better safe than sorry

double rectWidth = HistogramPlot.Width / counts.Length;

for (int intensity = 0; intensity < counts.Length; intensity++)

{

//there is no magic going on here

//for each intensity we are going to build

//a rectangle with a height proportional to

//its count

Rectangle histRect = new Rectangle();

histRect.Width = rectWidth;

histRect.Height = counts[intensity] * scale;

histRect.SetValue

(Canvas.LeftProperty,

intensity * rectWidth);

histRect.SetValue

(Canvas.TopProperty,

HistogramPlot.Height – histRect.Height);

//it just gives the fill a little

//softer look

Color =

Color.FromArgb(

255,

(byte)intensity,

(byte)intensity,

(byte)intensity),

Offset = 0 });

Color = Color.FromArgb(

255,

(byte)(.25 * intensity),

(byte)(.25 * intensity),

(byte)(.25 * intensity)),

Offset = 1 });

histRect.Fill = rectFill;

//finally, let’s add it to the canvas

}

}

It really is pretty simple.

What now?

The histogram offers us an objective method for classifying images.  For instance, based on the histogram below we might say the crane is not too dark and not too light. Because it doesn’t use the entire range, however,  we might classify this as having low contrast.  Trust me, this will come in handy in future lessons.

Summary

Creating a histogram is really easy, and it gives us some basic statistics.  It doesn’t take a wild imagination to come up with some ways to use this information. Over the next few lessons we will do just that.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2008%20-%20Intro%20to%20Histograms.zip

Up next: Contrast Stretching

Phone Vision – 07 Grayscale

24 01 2011

Many useful image processing concepts are simpler and faster without color information, so now that you have a solid grasp of color, let’s remove it and take a look at the world in grayscale.

Conversion

Grayscale is a measure of the intensity of the total light at a given point.  Because the ranges for red, green, and blue in the WriteableBitmap are from 0 to 255 it is convenient for us to set the grayscale ranges similarly where the absence of light (black) is 0 and maximum intensity (white) is 255.  We just need a method to extract this intensity from the color values.

I always go with my gut and it says we should be able to average the color intensities.  Time to put on the coding gloves.

WriteableBitmap ToAverageGrayscale(WriteableBitmap bmp)

{

WriteableBitmap grayscale =

new WriteableBitmap(bmp.PixelWidth, bmp.PixelHeight);

for (int pixelIndex = 0;

pixelIndex < bmp.Pixels.Length;

pixelIndex++)

{

int pixelColor = bmp.Pixels[pixelIndex];

byte red = (byte)((pixelColor & 0xFF0000) >> 16);

byte green = (byte)((pixelColor & 0xFF00) >> 8);

byte blue = (byte)(pixelColor & 0xFF);

byte intensity = (byte)((red + green + blue) / 3);

//notice how we repeat the intensity

//across each color channel

//this makes the output "color neutral"

//i.e. gray

grayscale.Pixels[pixelIndex] = 255 << 24

| intensity << 16

| intensity << 8

| intensity;

}

return grayscale;

}

Easy peasy lemon squeezy.

This looks like mission accomplished to me.

However, we are going from over 16 million colors to 256 intensities.  Obviously we are going to lose information.  How do we know that we are keeping the best information?

Any three numbers can represent six separate colors, but only one intensity.  the following example uses the numbers 255, 128, and 64:

Simply averaging the numbers will work fine in many situations, but in this situation it breaks down.

If we are trying to approximate the brightness a human perceives, mimicking human physiology is a better approach.  In the last lesson, the Bayer filter had twice as many green filters as red or blue because humans are more sensitive to green frequencies.   Green colors should then appear brighter than other colors right?

We can modify our code to be slightly more intelligent by weighting the average toward the green colors.

WriteableBitmap ToWeightedGrayscale(

WriteableBitmap bmp,

double redWeight,

double greenWeight,

double blueWeight)

{

WriteableBitmap grayscale =

new WriteableBitmap(bmp.PixelWidth, bmp.PixelHeight);

for (int pixelIndex = 0;

pixelIndex < bmp.Pixels.Length;

pixelIndex++)

{

int pixelColor = bmp.Pixels[pixelIndex];

byte red = (byte)((pixelColor & 0xFF0000) >> 16);

byte green = (byte)((pixelColor & 0xFF00) >> 8);

byte blue = (byte)(pixelColor & 0xFF);

//notice that we are dividing by the sum of the weights

//this can be avoided by ensuring the weights add to 1

byte intensity = (byte)((red * redWeight

+ green * greenWeight

+ blue * blueWeight)

/ (redWeight + greenWeight + blueWeight));

grayscale.Pixels[pixelIndex] =

255 << 24

| intensity << 16

| intensity << 8

| intensity;

}

return grayscale;

}

Weights that work well for human vision are red = .2, green = .7, and blue =.1 (rounded from Frequently Asked Questions about Color by Charles Poynton).

As you should expect, the green colors are brighter.  If you want a grayscale image to match the perceived brightness of the color image, this technique does the job.  Is it better than the simple average?  In our contrived example, yes.  In many cases, the simple average will suffice.  Here is the grayscale orangutan using both techniques.  Can you guess which is simple and which is weighted?

(hint: the bottom one is weighted)

Summary

Bottom line: there is no “right” answer.  If you’re trying to make your grayscale image “look” right, the weights used above may be a good path.  If speed is your game, you might try just using the  green channel alone.  No technique is right for every situation.  Remember, our end goal is for the phone, not a person, to understand the image.  Have fun and play around with the numbers.  Think about the advantages and disadvantages of different techniques.

I highly recommend the Frequently Asked Questions about Color by Charles Poynton.  Warning: some of it is pretty math intensive.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2007%20-%20Grayscale.zip

Up next: Intro to Histograms

Phone Vision 06 – RGB Color Intensities

21 01 2011

I enjoyed our last blog’s bit of history, but how can we make something more useful than a spinning top?  It turns out that the fundamental principle Maxwell described for color photography is still in use today.  Though the medium is different, for all intents and purposes we are still taking three pictures – one with a red filter, one with a green filter, and one with a blue filter.  Those images are then combined to form a color photograph.

Color Intensity

When we take a picture, we are just measuring the intensity of light at a given point.  With a sensor alone we would only be capable of creating a map of intensities.  This is definitely useful in some situations, but it ignores oodles of information humans can decipher naturally – color.

Using a filter that only lets through a specific color, we  can measure the intensity of that color.  Remember, we need at least three distinct colors to create the gamut of colors our eyes can distinguish.  To capture three different colors we need three different filters.

In the tender, early years of color photography, three separate pictures had to be taken.  I can’t imagine it was easy to capture a moving target.  Some enterprising people came up with a way to split the light into the three different wavelengths and then aim those beams at three separate light sensitive plates.  Clever as it may be, it can also be expensive and bulky.  There has to be a better way.

Bayer Filter

Conceptually, this next idea is very simple.  From a manufacturing standpoint, I’m not so certain.   Regardless, most modern digital cameras use a single image sensor covered by a grid of color filters arranged in what’s called a Bayer pattern (named after its inventor Bryce Bayer).

The astute observer might notice that there are twice as many green filters as there are red or blue.  Wikipedia says this is because humans have a heck of a lot of rod cells (light sensing) and they are most sensitive to green light (~498 nm if you care).

In its raw form, the output of the camera above could be thought of as 3 separate images with with missing information:

Converting this raw data into a human friendly image is a process called demosaicing.  There are lots and lots and lots of ways to accomplish this.  They range from the very simple – like treating a 4×4 group as one pixel by averaging the green values – to a little more complex – like filling in the gaps by averaging neighboring pixels – to way beyond the scope of this blog.

Artificial Bayer Filter

The images we deal with are already demosaiced.  For this lesson we want to mess around with a raw image, but we don’t have one.  With that in mind, we are just going to have to “de-demosaic” an image.  For the color [255,128,64], the “de-demosaic” transform looks something like this:

Notice that our width will be twice the original width and our height will be twice the original height.  This will quadruple the size of the image.  Another important piece is the original coordinate (x,y) becomes 4 separate coordinates in the new image:

Let’s see what we can do.

//This assumes you already have the image in

//WriteableBitmap form. Refer to earlier lessons

//if you’re not sure how to do this.

WriteableBitmap ToBayerRaw(WriteableBitmap bmp)

{

//we need to double the size of the target bitmap

WriteableBitmap bayerRaw =

new WriteableBitmap(bmp.PixelWidth * 2, bmp.PixelHeight * 2);

//we loop through every column, row by row.

//the two loops are not necessary, but

//they demonstrate the concept easier

//than a single loop

for (int y = 0; y < bmp.PixelHeight; y++)

{

for(int x = 0; x < bmp.PixelWidth; x++)

{

//first recover the RGB pixel data

int pixel = bmp.Pixels[x + y * bmp.PixelWidth];

// remember that (x,y) translates to 4 coordinates

// g1 = (2x, 2y)

// b = (2x+1, 2y)

// r = (2x, 2y+1)

// g2 = (2x+1, 2y+1)

// also note that we are using the new pixel width

// as our offset

// notice the masks in use as well

//upper left green

bayerRaw.Pixels

[2*x + 2 * y * (bayerRaw.PixelWidth)]

= (int)(pixel & 0xFF00FF00);

//upper right blue

bayerRaw.Pixels

[2 * x + 1 + 2 * y * (bayerRaw.PixelWidth)]

= (int)(pixel & 0xFF0000FF);

//lower left red

bayerRaw.Pixels

[2*x + (2 * y + 1) * (bayerRaw.PixelWidth)]

= (int)(pixel & 0xFFFF0000);

//lower right green

bayerRaw.Pixels

[2*x + 1 + (2 * y + 1) * (bayerRaw.PixelWidth)]

= (int)(pixel & 0xFF00FF00);

}

}

return bayerRaw;

}

Since half of the pixels are green and only a quarter are red and a quarter are blue, we would expect a green tint.  Matrixesque, am I right?

Interpolation

So now we have a raw file to work with, let’s see if we can fill in some of the “gaps” we’ve created.  There are quite a few techniques to accomplish this, but we are going to ignore them and try to figure them out on our own.   Let’s start with the red pixels:

The first thing I notice is that some of ? pixels have immediate neighbors with values so let’s tackle those first.  My gut says that averaging them would yield a decent approximation:

This gives us value for 8 of the missing pixels leaving 4 behind:

If we ran through the image one more time, we could fill in the remaining gaps.  On the second run the new top, bottom, left, and right values would determine the new value of the missing pixels:

Wait a minute… Top, bottom, left and right were also the average of the original pixels.  Can we use that?  Let’s label a set of original pixels w, x, y, and z.

That means that top, bottom, left, and right are defined as:

This simplifies rather nicely (I left out a couple steps):

What I like to call the average of the original four pixels.  This is intuitive for a lot of us, but it’s nice to have some math to back it up.

Putting it all Together

WriteableBitmap Demosaic(WriteableBitmap bmp)

{

WriteableBitmap interpolatedBmp =

new WriteableBitmap(bmp.PixelWidth, bmp.PixelHeight);

// we are going to cheat and ignore the boundaries

// dealing with the boundaries is not difficult,

// but the code is long enough as it is

for (int y = 1; y < bmp.PixelHeight – 1; y++)

{

for (int x = 1; x < bmp.PixelWidth – 1; x++)

{

//first we are going to recover the pixel neighborhood.

//the mask at the end is simply to get rid of the alpha

//channel

//middlecenter is the pixel we are working with (x,y)

int topleft =

bmp.Pixels[x – 1 + (y – 1) * bmp.PixelWidth] & 0xFFFFFF;

int topcenter =

bmp.Pixels[x + (y – 1) * bmp.PixelWidth] & 0xFFFFFF;

int topright =

bmp.Pixels[x + 1 + (y – 1) * bmp.PixelWidth] & 0xFFFFFF;

int middleleft =

bmp.Pixels[x – 1 + y * bmp.PixelWidth] & 0xFFFFFF;

int middlecenter =

bmp.Pixels[x + y * bmp.PixelWidth] & 0xFFFFFF;

int middleright =

bmp.Pixels[x + 1 + y * bmp.PixelWidth] & 0xFFFFFF;

int bottomleft =

bmp.Pixels[x – 1 + (y + 1) * bmp.PixelWidth] & 0xFFFFFF;

int bottomcenter =

bmp.Pixels[x + (y + 1) * bmp.PixelWidth] & 0xFFFFFF;

int bottomright =

bmp.Pixels[x + 1 + (y + 1) * bmp.PixelWidth] & 0xFFFFFF;

int blue = 0;

int red = 0;

int green = 0;

// if we are on an even row and an even column

// like (2, 2) then we are on a green pixel

// we need to average top center and bottom

// center for red we need to average left

// middle and right middle for blue

if (y % 2 == 0 && x % 2 == 0)

{

red = (((topcenter >> 16) +

(bottomcenter >> 16)) / 2) << 16;

blue = (middleleft + middleright) / 2;

green = middlecenter;

}

// if we are on an even row and an odd column

// like (2, 1) then we are on a blue pixel

// red is an average of top left, top right,

// bottom left, and bottom right

// green is top center, bottom center,

// left middle, and right middle

else if (y % 2 == 0 && x % 2 == 1)

{

red = (((topleft >> 16) +

(topright >> 16) +

(bottomleft >> 16) +

(bottomright >> 16)) / 4) << 16;

blue = middlecenter;

green = (((topcenter >> 8 ) +

(bottomcenter >> 8 ) +

(middleleft >> 8 ) +

(middleright >> 8 )) / 4) << 8;

}

// if we are on an odd row and an even column

// like (1, 2) then we are on a red pixel

// blue is an average of top left, top right,

// bottom left, and bottom right

// green is top center, bottom center,

// left middle, and right middle

else if (y % 2 == 1 && x % 2 == 0)

{

red = middlecenter;

blue = (topleft +

topright +

bottomleft +

bottomright) / 4;

green = (((topcenter >> 8 ) +

(bottomcenter >> 8 ) +

(middleleft >> 8 ) +

(middleright >> 8 )) / 4) << 8;

}

// otherwise we are on an odd row and odd column

// like (1,1). this is a green pixel

// red left middle + right middle

// blue is top center + bottom center

else

{

red = (((middleleft >> 16) +

(middleright >> 16)) / 2) << 16;

blue = (topcenter + bottomcenter) / 2;

green = middlecenter;

}

interpolatedBmp.Pixels[x + y * interpolatedBmp.PixelWidth]

= (255 << 24) | red | green | blue;

}

}

return interpolatedBmp;

}

With a little luck, our image should look pretty close to the original.

On the left you’ll find the original followed by the Bayer filter in the middle and, finally the demosaiced is on the right.  I think we did alright, eh?  This image is large, much larger than the viewing space.  If, however, you were to perform the same procedure on an image that is smaller than the viewing space, you would most likely see artifacts from the resizing.

Summary

Hopefully you learned a little about how cameras actually work and maybe a little math.  It wasn’t too hard, but trying to keep all those colors straight can make the code messy.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2006%20-%20RGBColorIntensities.zip

Up next: Grayscale

Phone Vision 05 – A Brief History of Color

18 01 2011

You should have a pretty solid grasp of how to effectively manipulate pixels with images.  I guess now we should talk a little bit about color.  Why, after all, do we pick red, green, and blue as our primary colors from which to build things?

History

• 1666 – Isaac Newton demonstrates white light is a combination of colors.
• 1802 – Thomas Young postulates the existence of three types of photoreceptors in human eyes.
• 1849 – James David Forbes experiments with rapidly spinning tops creating the illusion of a single color out of a mixture.
• 1850 – Herman von Helmholtz classifies human color receptors as short, medium, and long corresponding to the wavelengths they are sensitive to.
• 1861 – James Clerk Maxwell demonstrates a color photographic process using red, green, and blue light.

It turns out that the long, medium, and short photoreceptors (cones) in our eyes roughly correspond to red, green, and blue, respectively.  When emitting light, these three colors can be combined in various ways to produce all of the colors most humans are capable of seeing.  There are several color spaces we can use to generate human-friendly colors, but today we will focus on RGB.

The Forbes Top

Clearly, techniques for capturing and displaying color have been refined over the last few centuries.   We now enjoy color televisions, LCD screens, web cams, and so-called ‘retina displays’ to name a few.  Today, we are going to take a step back and try to simulate the rapidly spinning tops that James Forbes first experimented with.  The results will be somewhat crude, but they should demonstrate the concepts.

The most important component of this application is actually the design of the colorized circle that will represent our top.  I tried several different approaches before settling on 36 different slices (12 per color) with the colors alternating.  Coupling this with an adjustable rotation, an illusion of color can be established.

Here is the final product with red = 0, blue = 255, and green = 128:

My attempts at a screen capture video failed.  You’ll just have to download the code and try it yourself.  The only tricky component is actually drawing the wheel so I will walk through that piece.  As always, the rest of the code will be downloadable below.

Draw a Pie

private void DrawPie()

{

//drawing board is just a canvas

DrawingBoard.Children.Clear();

//red, green, and blue are member

double colorsum = red + green + blue;

// a circle has 360 degrees or 2*Pi radians

// we need to divide that into the different color

// components.

double redAngle = 2* Math.PI * (red / colorsum);

double greenAngle = 2* Math.PI * (green / colorsum);

double blueAngle = 2* Math.PI * (blue / colorsum);

// this part is tricky

// I am essentially doing a gamma function

// to try to keep the intensities close

// I eyeballed it so it’s not 100% accurate

byte intensity =

(byte)Math.Min(

Math.Pow((red * .33

+ green * .34

+ blue * .33), 1.2), 255);

Point start = new Point(0, 100);

Random random = new Random();

// we are going to divide our top

// into 36 slices — 12 per color

const int numSlices = 36;

for (int i = 0; i < numSlices; i++)

{

SolidColorBrush brush = new SolidColorBrush(Colors.Black);

double angle = 0;

switch(i % 3)

{

case 0: //red

angle = redAngle / (numSlices / 3);

brush = new SolidColorBrush(

Color.FromArgb(255, intensity, 0, 0));

break;

case 1: //green

angle = greenAngle / (numSlices / 3);

brush = new SolidColorBrush(

Color.FromArgb(255, 0, intensity, 0));

break;

case 2: //blue

angle = blueAngle / (numSlices / 3);

brush = new SolidColorBrush(

Color.FromArgb(255, 0, 0, intensity));

break;

}

// this code was essentially lifted from the Silverlight forums

// http://forums.silverlight.net/forums/t/9013.aspx

// Yi-Lun Luo posted it as a method for drawing pie graphs

// I modified it for the color top

// I will warn you that if your angle is > 90 degrees

// you will most likely run into trouble

Path path = new Path();

PathGeometry geometry = new PathGeometry();

PathFigure figure = new PathFigure();

// The center point is 0,0.

figure.StartPoint = new Point(0, 0);

LineSegment line1 = new LineSegment();

// Draw a line from the center of the circle to the start

// point of the sector’s arc.

line1.Point = start;

figure.Segments = new PathSegmentCollection();

ArcSegment arc = new ArcSegment();

arc.Size = new Size(100, 100);

arc.SweepDirection = SweepDirection.Clockwise;

Point p = start;

//Perform a rotate on the start point of the arc

// to compute the end point.

p.X = start.X * Math.Cos(angle) – start.Y * Math.Sin(angle);

p.Y = start.X * Math.Sin(angle) + start.Y * Math.Cos(angle);

//The next sector’s arc will begin from this one’s end point.

start = p;

arc.Point = p;

LineSegment line2 = new LineSegment();

// Draw another line from the sector’s end point

// to the center of the circle.

line2.Point = new Point(0, 0);

geometry.Figures = new PathFigureCollection();

path.Data = geometry;

// Assign a random color to this sector.

path.Fill = brush;

// Since our circle ranges from -100,-100 to 100,100,

// we’ll set the Path’s Left and Top to show all the graph.

path.SetValue(Canvas.LeftProperty, 100.0);

path.SetValue(Canvas.TopProperty, 100.0);

}

}

Summary

The most useful piece from this lesson might be the code for drawing a pie graph, but a bit of history is always fun.  The bottom line is that RGB comes from biology.  Our eyes are sensitive to those colors, so if we use them to mix we should be able to create images that mimic real life.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2005%20-%20HistoryOfColor.zip

Up Next: RGB Color Intensities

Phone Vision 04 – Premultiplied Alpha

13 01 2011

Previously we explored encoding pixels from color components.  This works perfectly if our alpha value is 255, and in most cases it will be.  But, what if we need transparency?

Pixel Format (again)

WriteableBitmap Pixel Format in Silverlight

When assigning colors to pixels in your bitmap, use pre-multiplied colors. The format used by the Silverlight WriteableBitmap is ARGB32 (premultiplied RGB). The format becomes relevant if you are populating the integer values in the Pixels array.

The initial pixel values in the dimensioned array are 0, which will render as black if left unaltered.

I want to clarify one thing.  The documentation says the pixel values are set to 0 by default (which is correct) and that it will render as black if left unaltered (which is incorrect).  It will render as transparent because the alpha values are also 0.  Don’t believe me?  Try it.

Transparency

Why would we ever want transparency when we are processing images?  Perhaps we want to highlight a point of interest.  You could leave the original image intact and just place a semi-transparent image over the top of it and let Silverlight blend them for you.  Regardless of the need, here’s how you do it.

If we were to create an image with a blue background covered with a semi-transparent (say, alpha = 128) white square, this is the output we would expect:

If you remember the MSDN documentation referenced in the last lesson it noted that the pixel format was premultiplied ARGB32.  This means that the RGB value has already been multiplied before storing the pixel.

The Code

Now we are going to try to recreate the blue color above using two different techniques.  The one will be straight RGB (i.e. not premultiplied) and the other will be premultiplied alpha.

1.) First, let’s set up a couple of blue rectangles to test.

<Grid x:Name="LayoutRoot"

Background="White">

<Grid.RowDefinitions>

<RowDefinition />

<RowDefinition />

</Grid.RowDefinitions>

<Rectangle

Width="300"

Height="300"

Fill="Blue" />

<Rectangle

Grid.Row="1"

Width="300"

Height="300"

Fill="Blue" />

</Grid>

Running this simple program should produce:

2.) Now we want to create a swatch for comparison.  Add a New Item, select Windows Phone User Control and call it “Swatch” (for comparing the techniques):

3.) Replace Swatch’s default <Grid> with this:

<Canvas Width="100" Height="100">

<Rectangle

Fill="Blue"

Width="100"

Height="100"

Stroke="Black"

StrokeThickness="2" />

<Rectangle

Fill="White"

Opacity=".5"

Width="100"

Height="100"

Stroke="Black"

StrokeThickness="2" />

</Canvas>

All we are doing is overlaying a blue rectangle with a semi-transparent (Opacity=.5) white rectangle.

4.) Add Swatch to the MainPage.xaml for comparison:

<Grid x:Name="LayoutRoot"

Background="White">

<Grid.RowDefinitions>

<RowDefinition />

<RowDefinition />

</Grid.RowDefinitions>

<Rectangle

Width="300"

Height="300"

Fill="Blue" />

<Rectangle

Grid.Row="1"

Width="300"

Height="300"

Fill="Blue" />

<local:Swatch x:Name="SwatchControl"

Margin="190,0,190,-50"

VerticalAlignment="Bottom" />

</Grid>

This output should look something like this:

4.) Add the two overlay images:

<Grid x:Name="LayoutRoot"

Background="White">

<Grid.RowDefinitions>

<RowDefinition />

<RowDefinition />

</Grid.RowDefinitions>

<Rectangle

Width="300"

Height="300"

Fill="Blue" />

<Image x:Name="StraightRGB"

Grid.Row="0"

Width="300"

Height="300" />

<Rectangle

Grid.Row="1"

Width="300"

Height="300"

Fill="Blue" />

<Image x:Name="PremultipliedAlpha"

Grid.Row="1"

Width="300"

Height="300"/>

<local:Swatch x:Name="SwatchControl"

Margin="190,0,190,-50"

VerticalAlignment="Bottom" />

</Grid>

The output will be the same as above because we haven’t actually set the source for these images.

4.) Finally, the code behind:

private void AcquireImageButton_Click(object sender, EventArgs e)

{

WriteableBitmap straightRGBImage =

new WriteableBitmap(300, 300);

WriteableBitmap premultipliedAlphaImage =

new WriteableBitmap(300, 300);

for (int pixelIndex = 0; pixelIndex < 90000; pixelIndex++ )

{

byte alpha = 128;

byte red = 255;

byte green = 255;

byte blue = 255;

double scaleAlpha = alpha / 255.0;

// we are not using scaleAlpha here

straightRGBImage.Pixels[pixelIndex] =

(alpha << 24)

| (red << 16)

| (green << 8 )

| blue;

// notice the alpha value is NOT scaled

// it’s also very important to scale BEFORE

// shifting the values

premultipliedAlphaImage.Pixels[pixelIndex] =

(alpha << 24)

| ((byte)(red * scaleAlpha) << 16)

| ((byte)(green * scaleAlpha) << 8 )

| (byte)(blue * scaleAlpha);

}

StraightRGB.Source = straightRGBImage;

PremultipliedAlpha.Source = premultipliedAlphaImage;

}

Output

Launch the program and tap the acquire button.  I’ll let you guess which one is right.

Summary

Premultiplied alpha is pretty straightforward, but it will cost performance in tight loops.  In most cases we won’t have to worry about this because the images we typically deal with don’t have transparencies.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2004%20-%20PremultipliedAlpha.zip

Up next: A Brief History of Color

Phone Vision – 03 Encoding Color

10 01 2011

Last time we learned how to extract the individual color values from a WriteableBitmap.  While this will allow us to analyze the image, we haven’t modified it in any way.  If we want to modify the color of the pixel in a meaningful fashion, we need to revisit the pixel format for a WriteableBitmap.

Encoding Color Components

Recall the format for the pixel is

The assumption here is that we have the separated alpha, red, green, and blue components, but we want to combine them into the ARGB32 pixel format.  So, how do we encode those colors properly?  We can simply left shift each component value the correct number of bits then add them together.  Huh?

What we really want to do is something like this:

+

+

+

=

Another Bitwise Operator

Left Shift

How can we shift the bits into their correct positions?  I bet you know where I’m headed with this – the << (LEFT SHIFT) operator:

 operator input shift output << 1 1 10 << 1 2 100 << 1 3 1000 << 1 4 10000

It should be relatively easy to see that the appropriate shifts are:

 component shift blue 0 green 8 red 16 alpha 24

Using our new found knowledge we can rewrite this as:

int pixel = (alpha << 24) + (red << 16) + (green << 8 ) + blue;

Cake.

Above we used simple addition to combine the shifted color values, however, the traditional technique for combining these is the | (OR) operator.  Here is the table of outputs for |:

 operator bit one bit two output | 0 0 0 | 0 1 1 | 1 0 1 | 1 1 1

How can we use this?  Well, if we | a bit with 0 we get the value of the original bit.  (i.e. 1|0=1 and 0|0=0 – original bit in bold).  This means that we can get the exact same results above with this line of code:

int pixel = (alpha << 24) | (red << 16) | (green << 8 ) | blue;

Why would we want to do this?  My gut says: performance.  In my experimentation, however, it turns out to be pretty minor.  For 100,000,000,000 iterations (that’s 100 billion) it took 17 minutes 48 seconds for the + operation and 17 minutes 33 seconds for the | operation (on my dev machine).  So yes, the | operator is slightly more efficient, but barely.

I use the | version in my code, because image processing is intense and even small performance gains add up when you are looping through hundreds of thousands of pixels.

Transparency

It’s important to note that the above code only works correctly for alpha = 255. Since we are working with images this is not typically an issue.  In the next episode we will discuss how to handle transparency.

Summary

Encoding a color from its components is about as easy as we’d expect it to be assuming you understand some fundamentals of computer science.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2003%20-%20EncodingColors.zip

Up next: Premultiplied Alpha

Phone Vision – 02 Extracting Color

6 01 2011

Last time we set up a very simple project that will allow us to display and manipulate image data using a WriteableBitmap.  Before we can do anything cool though, we need to understand a few things about the WriteableBitmap class.

Pixel Collection

The WriteableBitmap has an integer array called Pixels that represents the 2D texture of the bitmap.  In most cases we will be looping through Pixels (using PixelWidth and PixelHeight) to perform our operations.  You can find the data for a pixel located at (x,y) in the WritableBitmap wb  with:

int color = wb.Pixels[x + y * wb.PixelWidth];

Without knowing how the color is stored we really can’t do much with this data.  Let’s take a peak under the hood.

Pixel Format

The format used by the Silverlight WriteableBitmap is ARGB32 (premultiplied RGB – we’ll cover this later).  This means that the color is represented by a 32-bit integer with the following format:

As you can see, each channel (alpha, red, green, and blue) uses 8 bits, giving a possible 28 or 256 intensities for each (0-255).

Extracting Color Components

Extracting each component intensity can be done by masking the color value with 0xFF then shifting the number right by 8 bits.  The heck you say?

Bitwise Operators

And

As their name implies, bitwise operators operate on one bit at a time.  The operator we are concerned with for now is & (AND).

 operator bit one bit two output & 0 0 0 & 0 1 0 & 1 0 0 & 1 1 1

Notice in the & operator when a bit is 0 the output bit will always be 0.  Wherever there is a 1 in a bit, the result is always the value of the other bit.  This trait will allow us to create what is called a mask (or bitmask).  We cover the bits we don’t care about with 0.  So the mask 0xFF above in binary is…

& this with the pixel and we will get the value for the last 8  bits of the color.

Let’s work with an example.  I randomly picked #FFDB91D6.  #FFDB91D6 in binary:

&

=

Voila.

Right Shift

If we were to use the & alone we could only recover the values for the blue channel.  If only we could shift the color bits to match our mask…   Turns out there is another handy bitwise operator for doing just that: >> (RIGHT SHIFT)

 operator input shift output >> 10001000 1 01000100 >> 10001000 2 00100010 >> 10001000 3 00010001 >> 10001000 4 00001000

Notice how the bits shift right?  This is equivalent to dividing by powers of 2.  Don’t let the leading zeros confuse you.  If you divide 1120 by 10 (in base 10) you get 0112.  If you divide again you get 0011 (drop the remainder).  See?

>> 8

=

Now the green bits match our mask.  Hooray!

Full Example

Let’s run through our example from above (#FFDB91D6) in its entirety:

1.) First, let’s get the blue bits:

&

=

2.) Shift the bits right 8 places:

>> 8

=

3.) Recover the green bits:

&

=

4.) Shift the bits right another 8 places:

>> 8

=

5.) Extract the red bits:

&

=

6.) Last shift!!!

>> 8

=

7.) Extract the alpha bits:

=

We don’t have to mask the last value because it’s already exactly what we need.  Let’s see how this plays out in code.

int pixel = (int)0xFFDB91D6;

byte blue = (byte)(pixel & 0xFF);

pixel >>= 8;

byte green = (byte)(pixel & 0xFF);

pixel >>= 8;

byte red = (byte)(pixel & 0xFF);

pixel >>= 8;

byte alpha = (byte)(pixel);

Quite elegant and very efficient.

Summary

Recovering colors from the WriteableBitmap is pretty straightforward once you understand the pixel structure and how to mask them.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2002%20-%20ExtractingColors.zip

Note:  This code doesn’t actually modify the image in any way.  We will do that in the next installment.

Up next: Encoding Color

Phone Vision – 01 Acquiring the Image

3 01 2011

All image processing projects have one thing in common: an image (or multiple images).  We need to first acquire an image, then we need to get it into a format that is conducive to editing.  Let’s get cracking.

Setting up the Project

We are going to set up a very simple project to show the original image, the processed image, and an application bar button to start the image capture.  This will be the basis for future projects.

1.) Create a new Silverlight for Windows Phone Project

2.) Modify the LayoutRoot to have two images:

<Grid x:Name="LayoutRoot"

Background="Transparent">

<Grid.RowDefinitions>

<RowDefinition />

<RowDefinition />

</Grid.RowDefinitions>

<Image x:Name="OriginalImage"

Grid.Row="0" />

<Image x:Name="ProcessedImage"

Grid.Row="1" />

</Grid>

3.) Create the application bar:

<phone:PhoneApplicationPage.ApplicationBar>

<shell:ApplicationBar IsVisible="True" >

<shell:ApplicationBarIconButton

x:Name="AcquireImageButton"

IconUri="/Images/appbar.feature.camera.rest.png"

Text="acquire"

Click="AcquireImageButton_Click" />

</shell:ApplicationBar>

</phone:PhoneApplicationPage.ApplicationBar>

Note: Windows Phone 7 icons can be found here: C:\Program Files (x86)\Microsoft SDKs\Windows Phone\v7.0\Icons

Acquiring the Image

Now for the meat of the post.  First, we need to acquire an image then convert it to a WriteableBitmap.  In the wild, we are going to want to actually take a picture, but in the lab we simply want to load an image from the project.

1.) When we press the Acquire button we want to launch the Camera Capture Task.  Once it’s complete, we can start processing.

private void AcquireImageButton_Click(object sender, EventArgs e)

{

cct.Completed += new EventHandler<PhotoResult>(cct_Completed);

cct.Show();

}

2.) Create a BitmapImage and a WriteableBitmap from the results.

void cct_Completed(object sender, PhotoResult e)

{

// if we are working with the actual device

// we actually want to take a picture

// otherwise we are going to use a jpg

// I have included in the project

if (Microsoft.Devices.Environment.DeviceType == DeviceType.Device)

{

// TaskResults can be OK, Cancel, or None

// Working with the device here is rather

// painful because it cancels every time

// for me when I’m connected so I have to

// run disconnected from the PC

{

BitmapImage bmp = new BitmapImage();

bmp.SetSource(e.ChosenPhoto);

OriginalImage.Source = bmp;

WriteableBitmap processedImage = new WriteableBitmap(bmp);

// you could put some automated processing here

ProcessedImage.Source = processedImage;

}

}

//else we are running from the emulator

//in this case the CameraCaptureTask "works"

//but it doesn’t really return anything of value

//so we are going to use an image added to the

//project

else

{

//Creating a WriteableBitmap from a

//BitmapImage that was created using a

//Uri can be a little tricky.

//If you don’t follow these steps you

//might get a cryptic exception

BitmapImage bmp = new BitmapImage();

//Options here are None, DelayCreation (default), and

//IgnoreImageCache (in Silverlight).  Mainly, we don’t want

//DelayCreation because it waits to initialize it until

//necessary (i.e. when it’s shown).  Since we are setting

//the source of OriginalBitmap to bmp, we don’t actually

//need to perform this step in this case.  In general,

//however, for my projects, I often want to process the

//image before I display it.

bmp.CreateOptions = BitmapCreateOptions.None;

//note how we are waiting for the image to

//finish opening before we try to create the

//WriteableBitmap

bmp.ImageOpened +=

new EventHandler<RoutedEventArgs>(bmp_ImageOpened);

//I added this test image as Content to the Images folder.

//I recommend taking a photo with the phone then adding it

//to your project.  This ensures the image will be the correct

//format and consistent dimensions (2592×1944) for processing

bmp.UriSource =

new Uri("Images/test_image.jpg", UriKind.Relative);

OriginalImage.Source = bmp;

}

}

void bmp_ImageOpened(object sender, RoutedEventArgs e)

{

WriteableBitmap processedImage =

new WriteableBitmap(sender as BitmapImage);

// you could put some automated processing here

ProcessedImage.Source = processedImage;

}

3.) Run it

You can really see the HTC HD7 pink color issue in this photo.

Summary

This is a fairly trivial task, but getting the WriteableBitmap functioning properly is the basis for everything from here on out.

http://cid-88e82fb27d609ced.office.live.com/embedicon.aspx/Blog%20Files/PhoneVision/PhoneVision%2001%20-%20AcquiringImages.zip

Up next: Extracting Colors

Roshambo ya for it

25 11 2010

Now live on the marketplace is a fun little rock-paper-scissors app.  Using the accelerometer, Rock Paper Scissors imitates real life.  Stop bickering with your friends and let the RPS randomizer decide who gets to ride shotgun this time.  What are you waiting for?  Download it!

Soon to be added: You vs. Windows Phone & Choose Your Fate

Gestures with the WP7 Accelerometer

22 11 2010

First and foremost, if you want in depth information about the accelerometer you should check out Dave Edson’s excellent blog post, Using the Accelerometer on Windows Phone 7.  While his post is extremely informative, it was a bit of overkill for what I needed.  I assume this may be true for a lot of you as well.

I just needed to recognize a simple gesture I wanted to see what it “looked” like to the accelerometer.  I didn’t smooth out the curve or use the AccelerometerHelper class.  Here are the steps I followed to determine that the raw data appeared to be sufficient for my needs:

1. Start the accelerometer and record the data
2. Perform the gesture
3. Stop the accelerometer
4. Use Excel to analyze the data
5. Write the gesture detection code
6. Test, Test, Test

I wrote a quick and dirty program (included below) to capture the data and send me an email.  The email is plain text that I save into notepad then import into Excel as a delimited text file.

These are the x-axis readings for one trial for me flicking the phone with my right hand:

There are two key points on this chart: When the x value hits about 1.3G, and when it hits about –2G.  My gesture detection code looks for these values, in this order, in a small time frame (about 1 second).

Tips

• Perform several trials before you start coding your gesture recognition.  If possible, use different people.  What worked 10 out of 10 times for me worked 3 out of 10 times for someone else.
• Left and Right handed gestures can be drastically different.
• Be cognizant of how long the gesture should take and code your recognition accordingly.
• Relax your gesture parameters as much as possible, but be wary of false positives.  In order to prevent stacking gestures, I put in a pause of about 250 ms between gestures.
• Give some sort of feedback when they successfully perform a gesture.  A vibration lasting about 200 ms seemed to work well for my scenario.