Image Collection Browsing Kiosk

So recently at work, I've been lucky enough to be working on quite a unique product. Its primarily a kiosk application for museums and enables users to easily browse large image collections (tens of thousands).

While the user is browsing their way through, our algorithms look at the meta-data in the selected images and start to adapt the new mix of images to the users interests. This means that each different user of the kiosk will have a completely different experience.

The application also support video and audio assets, providing the user with a very immersive experience.

I personally have worked on all parts of this application, from the UI and web-service to the tool-chain and adaptive algorithms. My favourite part is undoubtedly the UI work!

The video below shows a development build which includes an image tray, allowing users to print off their favourite images to take home.



Hopefully I'll be able to post about some useful tit-bits of knowledge I've gained while developing the system!

Reflecting on Silverlight's RichTextBox and it's XAML Export Problems

During developement of our Image Relationship software, we came across the need to use the Silverlight RichTextBox with images, and most critical of all, for the RichTextBox to persist images when saving the output to the database.

Unfortunately Silverlight has let us down again. For one reason or another, the control doesn't export any images in the RichTextBox, or any other UIElement for that matter. It just returns an empty Run element instead.

Very frustrating!

Rather than shell out hundreds of pounds for third-party rich text box or use a unsupported free one such as the Liquid tools, we decided to write our out support manually.

To do this, we needed to manually export all of the RichTextBox contents, along with any InlineUIElements. Thankfully, there is a property on the RichTextBox which allows you to gain access to all the blocks within the control. From this you can iterate through them and with the help of reflection, write them all to a string builder. Fantastic!

The export function would like like this:

public string Export(RichTextBox rtb)
        {
                StringBuilder sb = new StringBuilder();
                sb.Append("
");
                
                foreach (Block b in rtb.Blocks)
                {
                    // Look at each block. They should either be a section or paragraph.
                    if (b is Paragraph)
                    {
                        Paragraph p = b as Paragraph;
                        writeElement(sb, p, false);

                        // Go through each inline within the paragraph
                        foreach (Inline i in p.Inlines)
                        {
                            if (i is InlineUIContainer)
                            {
                                // We need to look at these seperately as they have different requriements
                                InlineUIContainer inlineContainer = (InlineUIContainer)i;
                                parseInlineContainer(sb, inlineContainer);
                            }
                            else
                            {
                                // If its not an InlineUIContainer, our write element method will help!
                                writeElement(sb, i);
                            }
                        }

                        sb.Append("");
                    }
                }

                sb.Append("
");
                return sb.ToString();
        }

Parsing the InlineUIContainer may look like this:

private void parseInlineContainer(StringBuilder sb, InlineUIContainer inlineContainer)
        {
            if (inlineContainer.Child is Image)
            {
                sb.Append("");

                // If you want to be able to export other UIElements, this is where you handle them!
                Image image = inlineContainer.Child as Image;
                if (image.Source is BitmapImage)
                {
                    BitmapImage bm = image.Source as BitmapImage;

                    string uri;
                    if (bm.UriSource.IsAbsoluteUri)
                    {
                        uri = bm.UriSource.AbsoluteUri;
                    }
                    else
                    {
                        uri = bm.UriSource.ToString();
                    }

                    sb.Append("");        
                }

                sb.Append("");
            }
        }

And fleshing out the writeElement method may look something like this:

private void writeElement(StringBuilder sb, TextElement i, bool withClosingTag)
        {
            // Incase the control has any inlines (i.e. hyperlink)
            InlineCollection inlines = null;

            Type type = i.GetType();
            sb.Append("<" + type.Name + " ");

            foreach (PropertyInfo pi in type.GetProperties())
            {
                // Check if the property name is one of the ones we want
                if (m_acceptableAttributes.Contains(pi.Name))
                {
                    // Check if its readable and isn't null
                    if (pi.CanRead && pi.GetValue(i, null) != null)
                    {
                        // Get the value object and asset its something we're expecting
                        object valueObj = pi.GetValue(i, null);

                        if (valueObj is InlineCollection)
                        {
                            // Make sure we're not grabing inlines from a paragraph
                            if (i is Paragraph == false)
                            {
                                // If we have an inline collection, it means we have children, congrats!
                                inlines = valueObj as InlineCollection;
                            }
                        }
                        else
                        {
                            // Check if this element has inlines (hyperlink), if so we need to add them 
                            string value = parsePropertyValue(i, valueObj);

                            // Append the property to the string builder
                            sb.Append(pi.Name + "=\"" + value + "\" ");
                        }
                    }
                }
            }

            // Check if we have child inlines
            if (inlines != null)
            {
                // Close of the control ready for children
                sb.Append(" >");

                foreach (Inline inline in inlines)
                {
                    writeElement(sb, inline);
                }

                // Close the control
                sb.Append("");
            }
            else
            {
                if (withClosingTag)
                {
                    // If we need a closing tag, add one
                    sb.Append(" />");
                }
                else
                {
                    // Otherwise just close the element
                    sb.Append(" >");
                }
            }
        }

And finally the parsePropertyValue method could look like this:

private string parsePropertyValue(TextElement i, object valueObj)
        {
            string value;

            if (valueObj is SolidColorBrush)
            {
                // If the value is a colour brush, use color value
                SolidColorBrush brush = valueObj as SolidColorBrush;
                value = brush.Color.ToString();
            }
            else if (valueObj is TextDecorationCollection)
            {
                // There is only every one text decoration for silverlight, underline.
                // See remarks here: http://msdn.microsoft.com/en-us/library/ms603219(v=VS.95).aspx
                value = "Underline";
            }
            else
            {
                value = valueObj.ToString();
            }

            return value;
        }

So now we have the valid XAML, surely the RichTextBox should allow us to import the InlineUIElements within the XAML, using the XAML property... well, no.

It seems the control doesn't like importing InlineUIContainers as much as it likes exporting them. So we have to manually spoon feed the control again.

To do this, we used the XamlReader object, provided by the silverlight libraries, to load the XAML we manually exported into their respective instances. From this, we use the Blocks property on the control again to add each block one by one. Not so tricky

public void Import(RichTextBox rtb, string xaml)
        {
            // Load up the XAML using the XamlReader
            Object o = XamlReader.Load(xaml);

            if (o is Section)
            {
                // Make sure its a section and clear out the old stuff in the rtb
                Section s = o as Section;
                rtb.Blocks.Clear();

                // Remove the blocks from the section first as adding them straight away
                // to the rtb will throw an exception because they are a child of two controls.
                List<Block> tempBlocks = new List<Block>();
                foreach (Block block in s.Blocks)
                {
                    tempBlocks.Add(block);
                }
                s.Blocks.Clear();

                // Add them block by block to the RTB
                foreach (Block block in tempBlocks)
                {
                    rtb.Blocks.Add(block);
                }
            }
        }

Eh voila! We have persisted a UIElement from the RichTextBox and loaded it back in again.

Of course, a solution like this would seem a bit nasty if we didn't have a nice neat export/import class which we could use, instead of the lack-luster XAML property, so thats what we've done!

You can download it below:

Source (49KB)
Binaries (8KB)

Feel free to use it any way you like!

Just to note, currently this library only exports InlineUIContainers which contain Images, but this could be easily extendible.

Also, the library isn't extensively tested, so please modify and use as you feel fit!

Silverlight Animation Memory Leak in FillBehaviour

My first post of significance! Yay!

After months, *litrally* months of debugging, heap walking and cursing, with the help of the kind people at Microsoft and the Silverlight team, I’ve found the root cause of my memory leak.

The situation was, that I was using an ItemsControl with DataTemplate, whose contents animated independently.

To create the animation, I didn’t declare the fill behaviour of the storyboard, and consequently it defaulted to HoldEnd, which of course kept the last value of the storyboard. All well and good, and this was the desired behaviour.

Turns out though, that the animation I was using, is only properly cleared up from using two of the following:

• Using FillBehaviour=”Stop” in xaml
• Using Storyboard.Stop() in code

Great! So hold end doesn’t clean up resources.

The solution is to drop the hold end usage and somehow work in Stop somewhere. After that your memory leak should be significantly less!

Again, not sure about the mechanics as its a bug in the Silverlight libraries, but from my testing, switching fill behaviours certainly works! :)

I’ll try and embellish this post soon with more information, but I’m currently still trying to understand the full implications myself!

Hope this helps someone!