/home/todd

How do you debug a program that you can’t see? This is the problem faced by users interacting with the increasing number of applications utilizing machine-learning components to train themselves to a user’s inclinations. These programs consist of an instance of a machine-learning classifier that has been trained on data from a particular user and either resides on this user’s machine, or is designed solely for interaction with this user over a network. Common examples include email filtering software and recommendation systems, such as the one utilized by Amazon.com. The machine-learned program itself is accessed through a more traditional program, such as an e-mail application, which uses the learned program to decide how incoming messages should be categorized.

Our goal is to create a visualization of the learned program that a user may interact with, allowing the user to both understand why the program makes each of its decisions, and how the program can be corrected when it makes a faulty decision. Our audience consists of end users who neither have knowledge of formal software debugging techniques nor understand how machine-learning systems operate. These are people who use their computers for work or leisure, and are not interested in spending anything more than cursory time and effort to learn, e.g., how to improve the accuracy of their SPAM filter.

One of the most powerful machine-learning systems used today are Conditional Random Fields (CRFs). These systems excel at complex, sequential tasks such as natural language processing, and thus find themselves at the heart of many machine-learned programs. We used the logic for a CRF as the data set for our visualization. This logic includes a set of features, such as words, phrases, and other identifiable aspects of data which is being run through the learned program, as well as the set of numerical values each feature uses to determine its importance to each available category.

The data used to create the visualization is a transcript of a user study. This transcript consists of the words and actions of an end user debugging a spreadsheet in Microsoft Excel. Each sentence of the transcript is assigned to one of four categories (Seeking information, information gained, information lost, or none), which makes analysis of the transcript by researchers easier. Our visualization explains the logic the learned program might use to categorize each sentence of the transcript; future work would include allowing the user to adjust this logic when it results in poor classifications. This release is not connected to an actual classifier.  The displayed explanations are randomly generated, but provide a useful idea of what a functional implementation would look like.

Transcript Viewer source code [XCode 3.1 project, requires Mac OS X 10.5 or higher]

A screenshot of our prototype Transcript Viewer

So for the past week I’ve been meaning to write up a description of the general awesomeness that was IUI ’09, but with end-of-term craziness in full force, it’s going to have to wait.  Suffice to say it was a fabulous week full of wonderful people and some fascinating research.  Photos are slowly making their way from my notebook to Flickr, and once the VLHCC ’09 submission deadline passes, the process should speed up.

Speaking of, the past week has flown by as we prepare our paper, and I’m certain the next two will be similar.  Today and tomorrow I’ve eight transcripts to code, plus an abstract to write and a clever title to generate.  Feeling pretty good about it all at this particular moment, though; met up with Andrea at a funky new coffee shop (Vibe) downtown this morning, and ran into her friend Lisa as well.  Now I’m exporting our final transcript video from the lab, then heading home to throw open some windows, put on a kettle, and code until I can code no more (and maybe a bit past that, we’ll see).  The past few evenings have ended with mini-marathons of Series Three Doctor Who, and I’m looking forward to finishing that off late tonight.

Friday I leave for IUI 2009, where I’ll be presenting a formative study about end users debugging machine-learned programs.  This will be my first conference presentation, and I’m about equal parts excited and nervous.  At the very least, I should a chance to head out with my camera in sunny Florida.  I’ve hardly even taken it out of its bag this past month, and that needs to change.

I finally finished running participants for our current research study… now to start analyzing the data.  The paper deadline is March 8th, which feels like it’s approaching rudely fast.  But hey, at least we’re still on track.  I’d been worried that bringing in participants would become a 2 or 3-week nightmare like our last study, which would have pushed things way too close to the deadline.

This weekend I bought Henri T. Cat a new scratching board, which he promptly fell in love with.  Maybe now he’ll leave my couch alone.

Also, yesterday I learned that I’ll be going to CHI 2009 in Boston!  First big conference, eep!

• Finish the design of our next experiment involving debugging machine-learned programs.  This is actually coming along pretty well, and with a little luck I’ll be running sandbox pilots next week.  Assuming we get IRB approval, we should then be able to get real participants in for the last two weeks of the month.
• Prepare a presentation of our last research paper for IUI’09.  Still haven’t even started this.
• Figure out what my CS 519 term project will be.  Hopefully I can tie it in with my research and turn it into another paper.
• Talk to my adviser about internship possibilities for this summer.  Teaching was a lot of fun last year, but I’m ready for something new.
• Get a couple of cavities filled.  I may not have a paralyzing fear of dentists, but no one looks forward to fillings.
• Make it to the rec center at least 3 times a week.  I’m actually totally on track for this!
• Cook at least one new dish each week.  Again, I’ve been doing really well at this–I’ve actually been looking forward to leftovers!
• Record something, even if just a cover tune.  Phil loaned me some excellent books on digital audio and audio engineering, but I still haven’t found the time (see above) to put any of it to practice.

Screenshot

Background

The purpose of this project is to visually explain a machine learning classifier’s logic to an end user.  Additionally, the user must be able to explain back to the classifier when its logic is faulty and how it should be fixed.

Implementation

This project was designed in C++ using OpenGL.  I used GLUT primitives such as glutSolidCube() to create the basic components, then reshaped and arranged them using OpenGL transformations.  In order to create the translucent regions, I needed to enable lighting and shading.  Blending was then used to create the illusion that each word’s bar is inside one or more of the translucent boxes.  The text was created using glutStrokeCharater() calls; regular bitmap fonts were not an option because I wanted to rotate the text beneath the word boxes.  The resulting words suffered from severe aliasing problems, so I used the OpenGL accumulation buffer to enable 8x anti-aliasing.

Usage

While this project is not actually hooked up to a classifier, the word importance can still be manipulated by dragging the bars up or down into the desired regions.  There are also controls in a secondary window for rotating the view or panning right and left.

Room for Improvement

Ideally, this explanation would be interacting with a classifier; changes made to word importance would then be instantly viewable in the application when it re-classifies based upon the user’s adjustments.  I would also have liked to make it possible to zoom in and out so that the user can fit exactly as many words on-screen as she would like.  Finally, the blending of the word boxes with the translucent boxes isn’t entirely right–the word boxes should slightly change color depending on which region they are.

Downloads

Source code (for Visual Studio 2008)

Windows Binary (Requires OpenGL, GLUT and GLUI libraries)

Yay!  Thanks to some help from Will, I’ve finally finished my 550 term project, a whole 18 hours before the due date!  Since this was a graphics course, it isn’t actually hooked up to a naive bayes classifier backend (unlike the 2D version we did over the summer).  It is, however, infinitely more enjoyable to use than the previous prototype, mostly because you can rotate the view around and zoom down the list of words.

3D naive bayes classifier visualization

After another day of coaxing OpenGL to play nice, I’ve managed to make some progress on my final project.  The goal is to create a 3D version of the 2D visualization my research group designed for the logic underlying a Bayesian classifier.  So far I’ve got it displaying blocks representing the importance of each word to the classifier (only using three words at the moment, but I’d like to scale it up so that you can just zoom down a list of hundreds), and translucent regions denoting whether a word is required, forbidden, or unimportant to the classifier’s decision.  The blending isn’t quite right for that last part, but good enough to move on.  Now I need to make it interactive, so that the user can adjust the importance of each word.

Also, Ted and Judy dropped off an electric oil heater when they passed through town today.  My apartment’s furnace does a great job of heating the family room, but that’s about it.  Now maybe my bedroom won’t be freezing each morning!  Sadly they were just passing through while I was on campus; I want to take them out to dinner sometime before heading back to Detroit for the holidays.

Yesterday I was notified that my research group’s submission to IUI’09 was accepted!  This was the first research study and resulting publication that I led, and I’d honestly not been expecting it to be accepted on the first try.  So yay!  That’s two publications in a year and half–not a bad start, I suppose   The paper detailed a formative study of barriers end users encounter when attempting to “fix” [i.e. debug] machine-learned programs, like junk mail filters or hand-writing recognizers.  These sorts of machine-learned programs are becoming increasingly common in software, and since they need to tailor themselves to a particular user’s needs, that end user is the only person available to debug them if they misbehave.  It’s really a whole new research field that can build upon existing data about end-user programmers debugging in more traditional environments, like Microsoft Excel, but the machine-learned component is a significant change from such traditional models of source code.

Now we have three weeks to address the reviewers’ comments and get a camera-ready version to the conference organizers.  The conference itself is in Florida this February, so with a little luck, I might be able to afford to go and present the results in person