Seven Reasons OU Computer Science is Better Than Hogwarts from Rachel Shadoan on Vimeo.

However, it also presents a significant challenge. We have to develop new methods and tools to perform good remote ethnographies. In previous projects we have defaulted to Skype video calls–the next best thing, we thought, to in-person interviews. But when you live eight to ten time zones away from a participant, negotiating school, work, and leisure schedules to set up a Skype call can be nearly impossible. And, as much as we love our participants, at 3 am our time, we ethnographers much prefer to be sleeping.

The obvious solution to this was to develop an email interview. In our mind it had several advantages. It allowed the participant to respond to the questions in their own time, in a low-stress way. It removed the need for scheduling ninjitsu. And, it produced a ready-made transcript of the interaction, easing the field-note writing and analysis processes.

Our first iteration of the email interview was something like an open-ended survey. It explained the project and supplied the appropriate participant information and consent forms. Then, it listed the questions. It was in many ways a participant friendly version of the interview guide.

While the turnaround on the email interview surveys was really good from a time perspective, we felt that the answers we were getting were very short, to the point, and formal. This is in contrast to our in-person interviews, where answers to one question would often meander through several equally interesting subjects in the process of their completion.

So I thought a lot about how the in-person interviews were different from the email interviews, and I realized it was that with in-person interviews, the participant doesn’t know all of the questions you will be asking up front. Usually we tell them what kind of questions we will be asking, or what kind of information we are looking for, but the specific questions are unknown. As a result, the participant will often include a lot more information in the answer to each question. There was something about seeing all of the questions all at once that was cutting off this meandering; something about having all of the questions in front of the participant at once made the answers short and to the point.

So our solution was to send email interview questions one at a time.

This was a tremendous success. When we sent the questions one at a time, the answers were long, rich, and varied. Compare the two images below–the red circles surround answers to the same question.

We have tried out the second iteration of email interviewing on several participants, and have been blown away with their responses.

There are probably restrictions that come with this method. It is probably not appropriate for people who do not normally communicate via text-based mediums. (Our participants are very comfortable with the written communication of the internet, so in our case this has not been an issue.) It might also be less appealing to very busy executives–our coursemate Cora is doing a project with such folks, and she feels that her participants would become irritated with the process after three questions.

All things considered, however, we feel that the potential benefits of this method are numerous. Beyond the increased time that the participant has to respond to a question, there is also the increased time that the ethnographer has to consider the response, research, and contemplate a next question. While we thought that text would remove some of the richness present in non-verbal communication, we find that the opposite is true–for people who have grown up communicating via text, email, and chat, they are accustomed to explicizing the implicit, and often expand more on their feelings via text than they would likely do in person.

Finally, we feel it holds great promise for participants for whom an in-person interview would create a great deal of anxiety. This allows them time to answer the questions, and alleviates much of the pressure they might feel when faced with a note-scribbling, recording-toting ethnographer.

Now, what I want you to notice in these charts are the large white spaces, which are times with the player is not fertilizing (and indeed, likely not playing the game at all, possibly even sleeping). Notice that around June-July 2009, the location of the white spaces begins to migrate. (This is most obvious in Roger Shant’s graph).

Now, when I first noticed this down-time (I call it that since I don’t know if they are sleeping are not) migration, I assumed that someone had had a shift change at work, or started a new semester. However, after seeing the same pattern on the same timescale across four different players, I began to think something was rotten in the state of Denmark. I assumed that something had gone horribly wrong with my data. Somewhere in the processing, something was terribly, terribly wrong. Maybe the log files themselves were corrupted. However, after hashing this out with Jon and ruling out all of the potential causes we could think of, we decided that all of these players might actually have changed their schedules at the same time. Indeed, because they are all really active players, it’s possible that they were changing their down-time schedules because of each other.

However, it’s impossible to know without asking them.

I can speculate all day long on why this pattern appeared. I can talk myself blue in the face spouting ideas and hypotheses on what exactly caused this shift in down-time.

But I won’t know. In fact, I can’t know, not until I ask the players whose data I’m looking at. Only once I have the story to connect with the data will the patterns in the numbers make sense. Only when I have the story and the data can we paint a complete picture.

For instance, notice that large bubble that appears on all four charts? That was another weird thing in my data that I assumed was a mistake. I asked Jon about it, and apparently there had been a server issue which had potentially eaten the stat points that players had gained during the day. So, on that day he gave players a huge stat point reward for fertilizing, to compensate for the stat points that the server issue had deleted.

For the full story on the Facebook Plant Wars rip-off, read Jon’s blog post about it.

In Plant Wars, players can increase their plant’s abilities by fertilizing, or “training”. Players can choose what ability they want to increase when they fertilize. Fertilizing Attack (referred to as Potency in the game) makes their plant’s attack stronger. Fertilizing defense (referred to as Girth in the game) makes their plant less suspectible to attack, while fertilizing speed (called Responsiveness in the game) makes their plant more agile and able to avoid attacks. When a player fertilizes, they occasionally have to successfully complete a CAPTCHA, a device to tell humans from computers. This weeds out cheaters. You can read more about the Plant Wars CAPTCHA here on Jon’s Plant Wars development blog.

Here, we have the training data of two Plant Wars players. One of the players is fairly new, having only been a member of the Plant Wars community for 100 or so days. The other player, a longer member, is now an admin.

Download the visualization PDF for a closer look!

Notice that recently, the only thing this new player has been training is Defense. This is possibly because of a Girth competition currently being held in the game. Also, notice the large blank spot from 12:00ish to 19:00ish. Nothing in happening there, so we think this is probably when the player is sleeping. (Otherwise, this player never to sleep and that blank spot is indicitative of work or school. However, given the way the blank spot travels later over time, we think it’s probably sleeping that the player is adjusting to the level of light as spring turned to summer.) Note that all times are Central time (ie, the same time as it is in Chicago, Dallas, or in this case, Tulsa, Oklahoma).

Download the visualization PDF for a closer look!

Our seasoned player takes a more methodical approach to training; he trains particular all day for several days in a row, and then moves to another skill. This appears to be a technique that he developed over time–in the early days of his tenure with Plant Wars, the kinds of skills he trained overlapped (take a look in the upper left hand corner to see that). Again, we think the blank spots are when this player sleeps, which seems to have changed over time–possibly because of a shift change.

A note on building the visualizations: these were once again processed from text log files using a Java program that I wrote, and then visualized using Excel. I’ve been experiementing with Prefuse Flare, but Flash doesn’t like large data sets one bit, so we’re kicking it old-school for the time being.

Moving Data Into the Cloud

Nothing makes me more nervous than only having one copy of a document. I am a big fan of frequent backups, or, even better, letting someone else worry about the persistence of my documents. Enter Google Documents. If you’re not familiar with Google Documents, it’s definitely worth a look. For simple (read: not flashy) office processing tasks, Google Documents provides a solid set of free tools, and allows you to store your documents on Google’s servers. This has numerous advantages.

Safety: Protect your files from hard drive failure

Hard drives are vulnerable, and laptop harddrives are particularly so. Drop them one too many times, get them a little too close to a refrigerator magnet, spill some soda on them, and in a blink, your precious data is gone. I much prefer to store my data on someone else’s servers. Google’s servers are the lifeblood of their business–if those servers fail catastrophically, Google loses catastrophic amounts of money. So they hire loads of people to look after the health and wellbeing of Google’s servers to ensure that no such thing happens. Because Google and my interests as far as server reliability are in line, I feel very comfortable storing my documents there.

Security: No need to transfer via insecure physical media

Storing documents there also means that you can access them from any computer with an internet connection, removing the need for potentially insecure physical storage devices. Flash drives (aka pen or thumb or jump drives) are very handy, it’s true, but not only are they easy to misplace–a data security nightmare–but they are also notorious for transmitting viruses from one computer to another.

Sharing: Easily share documents with other researchers across the world

In an increasingly global society, researchers often end up collaborating remotely. Instead of emailing back and forth different versions of a document, Google Documents allows you to share a single document and edit it simultaneously and collaboratively.

For this project, we are storing all of our field notes on password-protected Google Documents. However, Google Documents is not the only cloud service we are using; we are also using the WordPress servers to facilitate our data management. Our audio, video, and image files are stored privately on our WordPress project blog, which offers the same advantages I described above. For this project we are not using Flickr, but it is also a fantastic cloud service that will help you protect your pictures for posterity.

File Naming for Fun and Profit

When you have many, many files that have to be organized, it’s important to be methodical in their naming. The convention we are using for this project is as follows:

Description/ParticipantPsuedonym_DDMonYYYY_Location_City_Researcher’sIntials

(Researcher’s intials are only required if only one researcher is involved in the creation of the file, as in the second example below in which I recorded audio).

Since that’s a little opaque, here’s a couple of examples:

Field Note File: LucasBowser_15Jun2010_RTW_Dundee
Audio File: LucasBowserAudio_15Jun2010_RTW_Dundee_RS

Note that we never use a participant’s real name in the naming of a file–it is always the participant’s psuedonym.

Speaking of psuedonyms, let’s talk about how we keep track of all of the folks we talk to.

Participant Psuedonyms and Tracking Database

Alicia and I are using a particularly fun mechanism for arriving at participant psuedonyms for this project. Each participant gets one name from a video game character. Their other name is something evocative for us. We spend a little bit of time discussing the names we choose so that we both understand the thought process behind the name selection, and then the name becomes a memory tool for us, encapsulating more information.

For instance, if I were to develop a psuedonym for Alicia, I might come up with something like, “Lara Chopin”.

Lara comes from Lara Croft, an image of whom is currently Alicia’s computer’s desktop wallpaper. I’ve also seen Alicia dress up as Lara Croft from time to time (for Halloween or other fancy dress occasions). Further, Lara is similar to Laurey, the name of the heroine from the musical Oklahoma, which is a nod to Alicia’s birthplace.

Chopin was an intensely creative Polish pianist and composer, which pays homage to Alicia’s creativity but also to her heritage.

So, as the researcher, I would read “Lara Chopin” on a file name, and immediately remember that this is the Polish-American participant from Oklahoma who is very creative, and kind of a badass. It’s a helpful memory device.

However, this all needs to be recorded somewhere. So we have created a participant tracking database (as a spreadsheet on Google Docs, naturally) that maps the participant’s name to their psuedonym, and indicates whether they are all squared away on ethical and field note fronts.

Contrary to what the name suggests, field notes are not actually taken in the field while we are interviewing and observing people. Rather, while we are in the field we take scratch notes. In our scratch notes we write down things people say, make sketches of rooms, write down the time things happen, keep tallies of how many times that person has kissed a hamster in the last fifteen minutes, and a host of other information that might be useful to our research.

Once we leave the field and are back in our own homes or offices, we turn the scratch notes into field notes. This is a somewhat arduous and painful process, that involves turning things like, “6pham danc NW g 4:16-5 O” into a meaningful description like “At 4:16 pm, there were six people dancing in the northwest corner of the green, each holding small golden hamsters. They danced in a circle for five minutes.” It is best done as soon after the field work as possible, so that the appropriate details can be added in. Memory decays quickly, and then gets fuzzy and confused.

In traditional anthropology, field notes are not things that you share. They are, in many ways, considered the secret formulaes of anthropological research. Most researchers are loathe to share them. As a result, there is little standardization in how field notes are taken or taught, which creates a great deal of difficulty for an up-and-coming ethnographer. This year we have largely been left to our own devices in developing a methodology for writing field notes.

Old Methods

I have experimented with several different methods.

The first, and most natural to me as a writer, is to turn my scratch notes into a narrative. I write it up much like an interview in a women’s magazine. Things like,

“Eleanor plays with her coffee as I sit down across from her at the diner. She appears to be stirring it with a French fry, and looks up, eyes wide, when I accidentally drop my cell phone with a clatter on the table. She hates cell phones, she says.

‘Every time my cell phone rings, it’s someone wanting something. I would love to smash it with a hammer.’ She laughs, throwing back her head, her blonde ringlets shaking.”

I like this method for a couple of reasons. It provides a more cohesive, enjoyable narrative, for one. They’re much nicer to read this way. Further, they’re much nicer to write.

However, they are also somewhat less accurate depictions of an interaction, and I have to be very careful not to put in judgements. For instance, I wanted to write “and looks up, eyes wide startled, when I accidentally drop my cell phone” and “blonde ringlets shaking with mirth“. However, those additional words, which I would use so casually in my own writing, are not descriptions. They are things I have intuitively decided about the situation, based on my long period of living in the world. When someone’s eyes are very wide and their face is pale, that generally denotes some kind of shock or surprise. She is laughing, therefore she is mirthful. However, these are not necessarily accurate interpretations of a situation, and are certainly not uniform across cultures. Maybe her eyes are wide because she replaced her mascara with superglue this morning. Maybe she is laughing not with mirth but with barely surpressed rage which will shortly cause her to Hulk out and kill everyone in the cafe. Ethnographers are supposed to record what is, not record interpretations of what is. (This goal is clearly not entirely possible, as everything we observe is interpreted by our brains to some degree; however, it’s good to try.)

The second method that I have tried for field notes of interviews is direct transcription, in a question and answer format. I would record everything that I asked, and everything the participant said, word for word.

The primary disadvantage here is that is is excruciatingly painful and time consuming to transcribe these things. (I was once so desperate not to do it by hand that I recorded an interview, then called my number on Google Voice, which transcribes voicemails, and played the interview for the Google Voice voicemail in hopes that it would transcribe it for me. This was a tremendous failure.) I would often get so bogged down in getting every word recorded that I would miss paying attention to the ideas behind what the participant was saying.

The New Method

Alicia and I are trying a new and different method, utilizing the power of spreadsheets.

This is our new template:

And here is our template in use (participant’s name has been changed to protect his privacy)

We fill these in using a three-pass process.

• Pass 1: Scratch notes taken in the field
• Pass 2: Fill in the first three columns on the spreadsheet
• Pass 3: Tag each row with primary, secondary, and tertiary tags.

If both Alicia and I were present for an interaction, we would have two versions of these notes–one that she had done, and one that I had done. They will both be in the same file, on different tabs (or sheets, as they’re called in MS Excel) labeled by our initials.

There are several advantages to this approach.

Uniformity:

Writing field notes the same way every time makes it easier to write, read, and process them.

Sorting:

The tag columns allow us to read the notes chronologically (as they are written) or sort by tag and read them by subject area.

Merging:

Because they are uniform and in column format, it’s much easier to merge the notes of two researchers (or maybe even more!

Quantitative/Qualitative Fusion

Spreadsheets can easily be exported to CSV files for text mining.

Occurences of words or tags can be computed easily, so that each field note file can be tagged for relevancy. For instance, let’s say a file had 20 tags, 10 of which were about hamsters and three were about llamas. That could be tagged with a 50% hamster relevancy and a 15% llama relevancy. Then, if you were later doing a project in which you needed information about hamsters, you could search through all of the field notes of all of the projects you’ve ever done, and say, “Return all field notes with greater than a 30% hamster relevancy.”

Over time, a machine learning algorithm can learn what lines of field notes you are tagging what way, and you can develop an auto-tagger.

Easy Analysis

Most importantly, it allows for easier analysis. When you are ready to analyze the notes, you simply output everything below row eight to a single file for all of the field notes, then sort by the tags. In short, it removes the affinitization step that takes so long in traditional ethnographic analysis.

The data used in this visualization tracked Plant Wars in-game banking transactions, of which there are several varieties. A player can deposit money into a bank, transfer it to another player, or transfer it to a “colony”. (A colony is an organized group of Plant Wars players, kind of like a club.) Players can also transfer donor days or seeds to other players or colonies. Finally, a member of a colony with appropriate administrative rights can transfer things to a user. This particular visualization doesn’t show the transactions by type, but the next version of it will do so.

The data started out looking like this:

[2009/06/17 09:51:55] 2842 deposited $1,782

[2009/06/17 09:57:49] 1641 deposited $14,894

[2009/06/17 10:00:10] 3361 deposited $2,807

[2009/06/17 10:01:16] 3330 deposited $2,500

[2009/06/17 10:03:04] 69 deposited $1

[2009/06/17 10:05:40] 1641 deposited $16

[2009/06/17 10:07:04] 2387 deposited $168

[2009/06/17 10:07:22] 2387 deposited $201

[2009/06/17 10:16:16] 418 deposited $3,734

[2009/06/17 10:21:21] 644 -> 2720 : $1,000

I processed the data into comma-separated-value files (csv) using a simple Java program, and then built this visualization using Microsoft Excel.

Now the data looks like this (click on the image to go to Flickr, where you can get a GINORMOUS version of it. Or click here, to get a PDF):

The bottom and top of the chart are midnight, and the middle is noon. As you can see, banking transactions happen most often immediately after noon and immediately after midnight, thinning out significantly between 5 am and 9 am. While there is a slowdown in banking transactions in the afternoon and evening as well (after the noon rush), it is not as pronounced as the early-morning slowdown. We suspect that the transaction “rushes” happen because of the way time and money are handled in Plant Wars.

Future versions of this chart will show the transactions by type and by user.

I love Machinarium; I think it’s a great game, and I think you should play it. If you’re reading this blog, it’s likely that you can play it–it’s a downloadable Flash game, so you can probably even play it if you (*gasp*, the HORROR) only have access to a Mac. It’s point and click, and nothing jumps out to kill you, so it’s good for even high-strung folks. Arguably, it’s perfect for high-strung folks, because as I mention in the review, the game is so laid back that the calm seeps into your soul. It’s cheaper than a trip to a spa trip or a vacation on a remote tropical island! (Mom, Shanna, if you’re reading, you would love it and should download it immediately. Why procrastinate on Facebook when you can procrastinate with darling robots?)

Coming soon: a list of the things that I didn’t get to put in my article about Machinarium.