LIS 7610/CSC 7481 Project Part 3: User Evaluation Design

Last updated: August 16 2012.

The goal for this part of the project is to design the user study that will represent one of the three major parts of your project (the other two are the system you will build and the batch evaluation you will conduct).

You should think of your batch evaluation and your user study as ways of answering different aspects of the same question: "How well does our system help the user achieve their goals?" Because user studies are generally more expensive than batch evaluations (i.e., they take more time to plan, execute and analyze if done well), it is prudent to focus your user study on aspects of the question that are not amenable to batch evaluation. Some examples of these kinds of questions are:

Do novice users find the system easy to learn?
Can users easily learn to formulate effective queries using our system?
Are there common mistakes or misunderstandings that could be addressed by a better design?

Those are just examples -- you can surely think of many more. Which brings us to the second important point: user studies must be focused if they are to be useful. In other words, you need to decide on a few questions you most want to answer.

Once you have decided what you want to know, you are ready to choose a study design. There are two basic kinds of study designs:

Quantitative. In these, you do most of the work before the study by selecting an independent variable (what you change), one or more dependent variables (what you measure), and a very specific study protocol (e.g., what people search for, what order they search for them in, and how much time they have to do that). This advance work makes the analysis after the study very simple -- you just plot the relationship between the dependent variable and the independent variables on a graph and draw your conclusions (often with the help of a statistical analysis to determine the degree to which those conclusions would generalize to other users).
Observational. In these, the only thing you do before the study is design a fairly general study protocol (e.g., "take half an hour and search for two different things that interest you"). Then during the study you collect data about what happens in several ways (e.g., through over-the-shoulder observation, with a system-generated log file, and with a post-session interview). The real work then begins, as you draw insights from your observations by defining a consistent set of things that are of interest to you (e.g., recovering from mistakes) and identifying where they occurred in your data.

Generally, observational studies are necessary if you want to see things that can only be seen when people are doing an internally-motivated task (because the protocol in a quantatative study must standardize what people search for in order to make it possible to compare performance under different circumstances). Quantitative studies are normally used to compare two variants of your system under controlled conditions.

Once you have chosen goals for your study and the kind of study design you want to use, you should consult one or more examples of studies using that kind of design. Here's one of each that you could start with:

Of course, your study shouldn't be as ambitious as any of these. You'll probably want to recruit study participants from the members of the class, so to avoid asking anyone to participate in more than one study you should probably limit the size of your study to four users and the length of your sessions to two hours (including training time, which you shouldn't scrimp on if you hope to learn anything form the study).

Once you have a study design, you should test it on someone who did not contribute to your study design and who will not be one of your actual study participants. This is called a "pilot study", and you are sure to learn of the need for some improvements (e.g., in how you do training, or in how you collect data). You can go do this with almost anyone since you are not studying them, you are simply trying out your study design. But it is best if you don't use a member of your team (because they know too much!).

Normally you would want to recruit study participants who are representative of the people who would really use a system like the one you are building, and if you can find such people and motivate them to participate that would be great (but see below on IRB requirements if you want to make a publishable study out of this). But in practice, most of you will actually recruit your classmates. The only restriction on this is to avoid over-using same members of a team because they can get a learning effect.

With that as background, here is what should be in your plan:

Identify your team and what you are building
The questions you seek to answer
Which kind of study design you plan to do
Your detailed study design
A description of the kinds of results you expect to be able to report

As general guidelines, your study design should fit comfortably on four pages, and (unless all you are doing is a user study!) your study should require around 15% of the total time available for your project, including planning, execution, and analysis (9 weeks * 6 hours per week * the number of people on your team * 15% ... for a 6 person team this number is around 50 hours).

If you plan to publish the results of your study, advance approval from LSU Institutional Review Board (IRB) is required. If your results will not be published, that step is not required.

Go back to Syllabus Page.

Acknowledgement to Doug Oard (LBSC 796/INFM 718R Spring 2011).