•  explicitly shared objects and events: the information that make up the focus of the collaboration
• implicitly shared objects: all of the stuff that’s happening in the environments around the collaborators while the collaboration is occurring
• goals for analyzing objects and events
• interpretations and thoughts on the data: what each collaborator is thinking about the data
• level of attention on the objects: what each collaborator is paying attention to and when
• history of objects: who created it, who has touched it, etc.
• emotional reactions and personal characteristics of the collaborators
  

The authors present two systems that incorporate some of these features: CoMotion, a collaborative environment for information analysis, and Reality Instant Messaging, an online system that augments online environments by linking them to real-time happenings.

CoMotion allows users to share “frames”, which are completely interactive views on the data. Users can drag data in and out of a shared frame, share goals, questions, and interpretations of the data, in addition to automatically sharing the history of interaction with an object. In short, CoMotion incorporates all of the features of common ground except implicitly shared objects and the emotional reactions of the users.

The Command Post of the Future is a case study in the use of CoMotion. The Command Post of the Future is a research project with the goal of providing collaborative visualization tools for disaster response and combat situations. The example shown below gives an idea of the power of the system. The left side of the visualization is the frame a commander is working on and sharing with his team. It contains a map annotated with objects: small triangles represent intelligence sightings, rectangles are units, diamonds indicate the commander’s interpretations of the intelligence sightings, and circles indicate tasks for the team to accomplish. Objects are additionally encoded to indicate whether or not they have been reviewed. The right side of the visualization shows the set of tabbed frames shared from other user’s workspaces. By looking through those, the commander can quickly see what each team member is doing or considering.

Screenshot of the CoMotion Desktop in the Command Center of the Future Project. (Click on the image to enlarge it to full size and soak in the awesomeness.) Image Chuah 2003

Reality Instant Messaging, the other system that the authors present, provides implicitly shared objects and events, the history of objects, the level of attention on the objects, and well as the emotional reactions of the participants. Users can “buddy surf” to find users who are doing something of interest to them, such as watching a streaming sports program, and then subscribe to that Reality IM “channel” to interact with their buddies during the program.

In the example shown here, the users are watching a golf program. The IM stream updates with play-by-play information, which the users may interact with by requesting more information about a player or participating in guessing games about the activity at hand. Users are free to participate as much or as little as they like.

While the examples presented in this paper are compelling (CoMotion is incredibly cool and I want to play with it), I have reservations about a few points in the features of common ground. One is related to information overload; the other is related to social barriers.

First, I am unconvinced that implicitly shared information is intrinsically important; if it were, it would be explicitly shared. If the song playing in the background of a user’s workspace is important to the collaboration, the user is likely to share it. Automating that kind of sharing (which is not actually implemented in the examples in this paper) requires a clear understanding of what implicit information is important, which is highly context dependent. In general, I feel that users accustomed to internet collaboration already explicitly share the relevant implicit information, removing the necessity to automatically share it in some way.

The second reservation I have is sharing a user’s emotional/physical state. In many professional collaborative situations, particularly in disaster relief and intelligence, there significant social barriers preventing people from informing their colleagues that they are too exhausted/miserable to continue working. The authors suggest that incorporating audio and video into collaborative applications is sufficient for sharing such information, but while that provides a channel to allow other users to intuit a collaborators mental/physical state, it does not necessarily make the information explicit or encourage users to share it.

What is your experience with online or remote collaboration? What aspects of common ground do you think are most important to the collaboration process?

Chuah, M.C.; Roth, S.F.; , “Visualizing common ground,” Information Visualization, 2003. IV 2003. Proceedings. Seventh International Conference on , vol., no., pp. 365- 372, 16-18 July 2003

Notational model of sensemaking loop for intelligence analysis (Pirolli and Card, 2005)

In this paper, Pirolli and Card present a model of the “sensemaking loop”, the process by which analysts gather, filter, understand, and hypothesize about information. To build this model, they used cognitive task analysis and think aloud protocol analysis to identify the techniques and processes the analysts were using. However, the authors provide no further description of the methodology used in this study. This is particularly unfortunate as Cognitive Task Analysis does not refer to a single specific technique, but rather to a suite of approaches aimed at understanding what mental processes go into a task.

Methodological questions aside, the model of the sensemaking loop that the authors derived is an interesting description of the process. It is composed of a series of interlocking loops, through which the data becomes increasingly more structured. At one end of the structure spectrum, there are the external data sources: the things that an analyst collects to analyze. At the other end of the structure spectrum is the presentation of hypotheses: the narratives that the analyst has constructed about the data. Perhaps the most interesting point between these two extremes is the schema, which is a structure that the analysts used to organize data. That structure might be internal to the analyst’s mind, or it may be external, a  sketch the analyst has made, a map of data connections, or even a more formal tool like a computer visualization of the data.

The entire process, from external data sources to presentation, falls within two loops. The first, the foraging loop, “involves processes aimed at seeking information, searching and filtering it, and reading and extracting information”. The second, the sense making loop, involves iterative development of a mental model that best fits the evidence. These loops can be traversed from bottom to top, as we typically think of the analysis process operating, or from top to bottom, as when new analysis calls a previous hypothesis into question.

The authors identify several areas in which new tools or processes could improve the sensemaking process. In the foraging loop, they identify a tradeoff between exploration, enrichment, and exploitation. Exploration is the process of finding new information to analyze. Enriching is the process of narrowing the set of items to be analyzed to those that are most relevant and useful. Exploitation is the process of reading through the collected and enriched information and extracting useful patterns from it. The authors write, “It will generally be desirable to explore as much of the information space as possible (because there may be a cost to missing something novel in the data), but this comes at the cost of having to actually work through the material and eventually exploit it.” This is an important trade-off to evaluate when doing research; if you are designing a marketing campaign for a breakfast cereal company and you miss something in the data, the cost of having missed something will likely be relatively low. But if you are trying to figure out where a terrorist organization will strike next, missing something novel could be disastrous. The cost of missing something is therefore an important consideration when scoping a project, and it must be weighed against the cost of providing incomplete or inaccurate analysis.

Another area in the foraging loop that the authors identify as an area that could be leveraged is the amount of time spent scanning, assessing, and selecting items for further attention. They suggest developing techniques for reducing the cost of this task by highlighting important information (names, numbers, locations, etc) with “pre-attentive codings” or “re-representing documents”, such as by summary. These are both processes that ethnographers have developed to deal with the large amount of qualitative information they deal with. When writing field notes, we apply “tags” or “codes” to the documents so that we can easily find all of our field notes in which, for instance, sloth bathing was discussed by 25 year old Spanish speakers. (In that case we would be looking for a collection of tags “sloth bathing” “25 year old” “Spanish”). Summarizing documents is actually a technique that my friend Dhruv taught me: at the top of every set of field notes he makes, there is a short summary of the document, without he relevant details about the person or people he was interviewing or observing. Beneath that is generally a bulleted list of the highlights from the interaction. This kind of multi-pass processing makes the exploitation phase easier later.

I also see this leverage point as an opportunity for data mining. Much of the kind of assessment the authors discuss could be easily automated; one could simply ask the algorithm or tool to suggest information that is relevant to the information you are currently looking at, or to a particular set of concepts passed as parameters.

The authors also identify starting a new task in the loop or performing follow-up searches as areas that could be leveraged.

In the sense making loop, many of the leverage points identified are associated with well-known cognitive biases, such as confirmation bias. The first point they discuss is the limited attention span for evidence and hypotheses. Human working memory is limited, and that places limits on how much evidence someone can consider at once. This can, however, be alleviated by pushing information patterns onto “external memory” like visual displays.

Another point the authors identify is in the generation of hypotheses. Humans are fabulous at fitting data into patterns they are familiar with; it’s one of the reasons we are very good at recognizing faces. However, that also means that we are biased towards interpreting information using existing frameworks. In their analysis, the authors found that people failed to generate hypotheses. “Time pressures and data overload work against the individual analyst’s ability to rigorously follow effective methods for generating, managing, and evaluating hypotheses.”

This is a less obvious place to apply computer-aided techniques, but one where they may be especially needed. Even something as simple as taking the evidence an analyst has compiled and randomly reordering it might help the analyst look at things in new ways. More sophisticated approaches utilizing data mining and machine learning could even generate hypotheses for the analyst to validate or invalidate.

I enjoyed this paper because I could identify so much of my own analysis process in it; however, I am not convinced that the hypothesizing appears only so late in the process. I suspect that I start out with several hypotheses built from experience, before I even begin foraging. As I filter information, the hypotheses take shape, building or being discarded as I go. This all happens before the formal hypothesis building segment of the sensemaking loop (although that generally happens as well). How well does this model fit with your experiences with analysis?

Pirolli, P., and Card, S. The sensemaking process and leverage points for analyst as identified through cognitive task analysis. In International Conference on Intelligence Analysis (McLean, VA, May 2005).

A lot of Pirolli and Card’s other work looks really interesting. I have a feeling we will be re-visiting papers by these gentlemen in the near future.

However, to tide you over until the point that I can tell you all the gritty details about life as a data visualization researcher, I am embarking on a new project. I have been doing a lot of reading lately–both for my thesis work, and for the Visual Analytics course I am taking this semester. While some of the papers I’ve covered are highly specific and probably not of much interest to anyone not doing my research, others are canonical visualization papers that are both enjoyable reads and a good introduction to the discipline. So, I’ll be posting summaries and commentary on some of the papers as I come across ones that you might enjoy.

Seven Reasons OU Computer Science is Better Than Hogwarts from Rachel Shadoan on Vimeo.

However, it also presents a significant challenge. We have to develop new methods and tools to perform good remote ethnographies. In previous projects we have defaulted to Skype video calls–the next best thing, we thought, to in-person interviews. But when you live eight to ten time zones away from a participant, negotiating school, work, and leisure schedules to set up a Skype call can be nearly impossible. And, as much as we love our participants, at 3 am our time, we ethnographers much prefer to be sleeping.

The obvious solution to this was to develop an email interview. In our mind it had several advantages. It allowed the participant to respond to the questions in their own time, in a low-stress way. It removed the need for scheduling ninjitsu. And, it produced a ready-made transcript of the interaction, easing the field-note writing and analysis processes.

Our first iteration of the email interview was something like an open-ended survey. It explained the project and supplied the appropriate participant information and consent forms. Then, it listed the questions. It was in many ways a participant friendly version of the interview guide.

While the turnaround on the email interview surveys was really good from a time perspective, we felt that the answers we were getting were very short, to the point, and formal. This is in contrast to our in-person interviews, where answers to one question would often meander through several equally interesting subjects in the process of their completion.

So I thought a lot about how the in-person interviews were different from the email interviews, and I realized it was that with in-person interviews, the participant doesn’t know all of the questions you will be asking up front. Usually we tell them what kind of questions we will be asking, or what kind of information we are looking for, but the specific questions are unknown. As a result, the participant will often include a lot more information in the answer to each question. There was something about seeing all of the questions all at once that was cutting off this meandering; something about having all of the questions in front of the participant at once made the answers short and to the point.

So our solution was to send email interview questions one at a time.

This was a tremendous success. When we sent the questions one at a time, the answers were long, rich, and varied. Compare the two images below–the red circles surround answers to the same question.

We have tried out the second iteration of email interviewing on several participants, and have been blown away with their responses.

There are probably restrictions that come with this method. It is probably not appropriate for people who do not normally communicate via text-based mediums. (Our participants are very comfortable with the written communication of the internet, so in our case this has not been an issue.) It might also be less appealing to very busy executives–our coursemate Cora is doing a project with such folks, and she feels that her participants would become irritated with the process after three questions.

All things considered, however, we feel that the potential benefits of this method are numerous. Beyond the increased time that the participant has to respond to a question, there is also the increased time that the ethnographer has to consider the response, research, and contemplate a next question. While we thought that text would remove some of the richness present in non-verbal communication, we find that the opposite is true–for people who have grown up communicating via text, email, and chat, they are accustomed to explicizing the implicit, and often expand more on their feelings via text than they would likely do in person.

Finally, we feel it holds great promise for participants for whom an in-person interview would create a great deal of anxiety. This allows them time to answer the questions, and alleviates much of the pressure they might feel when faced with a note-scribbling, recording-toting ethnographer.

Stat points gained by Roger Shant

I want you to take a look at these visualizations. These are all charts depicting what skill a Plant Wars player is training, as well as how many stat points they receive as a result of their fertilizing of their plant. (There is a direct correlation between the amount of fertilizer used and the stat points gained. More fertilizer = more statpoints. So, the size of the bubbles indicates both the number of stat points gained, as well as the amount of fertilizer used.) All of the charts are on the same time scale, and so you can compare them to each other vertically.

Now, what I want you to notice in these charts are the large white spaces, which are times with the player is not fertilizing (and indeed, likely not playing the game at all, possibly even sleeping). Notice that around June-July 2009, the location of the white spaces begins to migrate. (This is most obvious in Roger Shant’s graph).

Now, when I first noticed this down-time (I call it that since I don’t know if they are sleeping are not) migration, I assumed that someone had had a shift change at work, or started a new semester. However, after seeing the same pattern on the same timescale across four different players, I began to think something was rotten in the state of Denmark. I assumed that something had gone horribly wrong with my data. Somewhere in the processing, something was terribly, terribly wrong. Maybe the log files themselves were corrupted. However, after hashing this out with Jon and ruling out all of the potential causes we could think of, we decided that all of these players might actually have changed their schedules at the same time. Indeed, because they are all really active players, it’s possible that they were changing their down-time schedules because of each other.

However, it’s impossible to know without asking them.

I can speculate all day long on why this pattern appeared. I can talk myself blue in the face spouting ideas and hypotheses on what exactly caused this shift in down-time.

But I won’t know. In fact, I can’t know, not until I ask the players whose data I’m looking at. Only once I have the story to connect with the data will the patterns in the numbers make sense. Only when I have the story and the data can we paint a complete picture.

For instance, notice that large bubble that appears on all four charts? That was another weird thing in my data that I assumed was a mistake. I asked Jon about it, and apparently there had been a server issue which had potentially eaten the stat points that players had gained during the day. So, on that day he gave players a huge stat point reward for fertilizing, to compensate for the stat points that the server issue had deleted.

For the full story on the Facebook Plant Wars rip-off, read Jon’s blog post about it.

In Plant Wars, players can increase their plant’s abilities by fertilizing, or “training”. Players can choose what ability they want to increase when they fertilize. Fertilizing Attack (referred to as Potency in the game) makes their plant’s attack stronger. Fertilizing defense (referred to as Girth in the game) makes their plant less suspectible to attack, while fertilizing speed (called Responsiveness in the game) makes their plant more agile and able to avoid attacks. When a player fertilizes, they occasionally have to successfully complete a CAPTCHA, a device to tell humans from computers. This weeds out cheaters. You can read more about the Plant Wars CAPTCHA here on Jon’s Plant Wars development blog.

Here, we have the training data of two Plant Wars players. One of the players is fairly new, having only been a member of the Plant Wars community for 100 or so days. The other player, a longer member, is now an admin.

Download the visualization PDF for a closer look!

Notice that recently, the only thing this new player has been training is Defense. This is possibly because of a Girth competition currently being held in the game. Also, notice the large blank spot from 12:00ish to 19:00ish. Nothing in happening there, so we think this is probably when the player is sleeping. (Otherwise, this player never to sleep and that blank spot is indicitative of work or school. However, given the way the blank spot travels later over time, we think it’s probably sleeping that the player is adjusting to the level of light as spring turned to summer.) Note that all times are Central time (ie, the same time as it is in Chicago, Dallas, or in this case, Tulsa, Oklahoma).

Download the visualization PDF for a closer look!

Our seasoned player takes a more methodical approach to training; he trains particular all day for several days in a row, and then moves to another skill. This appears to be a technique that he developed over time–in the early days of his tenure with Plant Wars, the kinds of skills he trained overlapped (take a look in the upper left hand corner to see that). Again, we think the blank spots are when this player sleeps, which seems to have changed over time–possibly because of a shift change.

A note on building the visualizations: these were once again processed from text log files using a Java program that I wrote, and then visualized using Excel. I’ve been experiementing with Prefuse Flare, but Flash doesn’t like large data sets one bit, so we’re kicking it old-school for the time being.

Moving Data Into the Cloud

Nothing makes me more nervous than only having one copy of a document. I am a big fan of frequent backups, or, even better, letting someone else worry about the persistence of my documents. Enter Google Documents. If you’re not familiar with Google Documents, it’s definitely worth a look. For simple (read: not flashy) office processing tasks, Google Documents provides a solid set of free tools, and allows you to store your documents on Google’s servers. This has numerous advantages.

Safety: Protect your files from hard drive failure

Hard drives are vulnerable, and laptop harddrives are particularly so. Drop them one too many times, get them a little too close to a refrigerator magnet, spill some soda on them, and in a blink, your precious data is gone. I much prefer to store my data on someone else’s servers. Google’s servers are the lifeblood of their business–if those servers fail catastrophically, Google loses catastrophic amounts of money. So they hire loads of people to look after the health and wellbeing of Google’s servers to ensure that no such thing happens. Because Google and my interests as far as server reliability are in line, I feel very comfortable storing my documents there.

Security: No need to transfer via insecure physical media

Storing documents there also means that you can access them from any computer with an internet connection, removing the need for potentially insecure physical storage devices. Flash drives (aka pen or thumb or jump drives) are very handy, it’s true, but not only are they easy to misplace–a data security nightmare–but they are also notorious for transmitting viruses from one computer to another.

Sharing: Easily share documents with other researchers across the world

In an increasingly global society, researchers often end up collaborating remotely. Instead of emailing back and forth different versions of a document, Google Documents allows you to share a single document and edit it simultaneously and collaboratively.

For this project, we are storing all of our field notes on password-protected Google Documents. However, Google Documents is not the only cloud service we are using; we are also using the WordPress servers to facilitate our data management. Our audio, video, and image files are stored privately on our WordPress project blog, which offers the same advantages I described above. For this project we are not using Flickr, but it is also a fantastic cloud service that will help you protect your pictures for posterity.

File Naming for Fun and Profit

When you have many, many files that have to be organized, it’s important to be methodical in their naming. The convention we are using for this project is as follows:

Description/ParticipantPsuedonym_DDMonYYYY_Location_City_Researcher’sIntials

(Researcher’s intials are only required if only one researcher is involved in the creation of the file, as in the second example below in which I recorded audio).

Since that’s a little opaque, here’s a couple of examples:

Field Note File: LucasBowser_15Jun2010_RTW_Dundee
Audio File: LucasBowserAudio_15Jun2010_RTW_Dundee_RS

Note that we never use a participant’s real name in the naming of a file–it is always the participant’s psuedonym.

Speaking of psuedonyms, let’s talk about how we keep track of all of the folks we talk to.

Participant Psuedonyms and Tracking Database

Alicia and I are using a particularly fun mechanism for arriving at participant psuedonyms for this project. Each participant gets one name from a video game character. Their other name is something evocative for us. We spend a little bit of time discussing the names we choose so that we both understand the thought process behind the name selection, and then the name becomes a memory tool for us, encapsulating more information.

For instance, if I were to develop a psuedonym for Alicia, I might come up with something like, “Lara Chopin”.

Lara comes from Lara Croft, an image of whom is currently Alicia’s computer’s desktop wallpaper. I’ve also seen Alicia dress up as Lara Croft from time to time (for Halloween or other fancy dress occasions). Further, Lara is similar to Laurey, the name of the heroine from the musical Oklahoma, which is a nod to Alicia’s birthplace.

Chopin was an intensely creative Polish pianist and composer, which pays homage to Alicia’s creativity but also to her heritage.

So, as the researcher, I would read “Lara Chopin” on a file name, and immediately remember that this is the Polish-American participant from Oklahoma who is very creative, and kind of a badass. It’s a helpful memory device.

However, this all needs to be recorded somewhere. So we have created a participant tracking database (as a spreadsheet on Google Docs, naturally) that maps the participant’s name to their psuedonym, and indicates whether they are all squared away on ethical and field note fronts.