LIS 7610/CSC 7481 Project Part 3: User Evaluation Design

Last updated: August 19, 2015.


The goal for this part of the project is to design the user study that will represent one of the three major parts of your project (the other two are the system you will build and the batch evaluation you will conduct).

You should think of your batch evaluation and your user study as ways of answering different aspects of the same question: "How well does our system help the user achieve their goals?" Because user studies are generally more expensive than batch evaluations (i.e., they take more time to plan, execute and analyze if done well), it is prudent to focus your user study on aspects of the question that are not amenable to batch evaluation. Some examples of these kinds of questions are:

Those are just examples -- you can surely think of many more. Which brings us to the second important point: user studies must be focused if they are to be useful. In other words, you need to decide on a few questions you most want to answer.

Once you have decided what you want to know, you are ready to choose a study design. There are two basic kinds of study designs:

Generally, observational studies are necessary if you want to see things that can only be seen when people are doing an internally-motivated task (because the protocol in a quantatative study must standardize what people search for in order to make it possible to compare performance under different circumstances). Quantitative studies are normally used to compare two variants of your system under controlled conditions.

Once you have chosen goals for your study and the kind of study design you want to use, you should consult one or more examples of studies using that kind of design. Here's one of each that you could start with:

Of course, your study shouldn't be as ambitious as any of these. You'll probably want to recruit study participants from the members of the class, so to avoid asking anyone to participate in more than one study you should probably limit the size of your study to four users and the length of your sessions to two hours (including training time, which you shouldn't scrimp on if you hope to learn anything form the study).

Once you have a study design, you should test it on someone who did not contribute to your study design and who will not be one of your actual study participants. This is called a "pilot study", and you are sure to learn of the need for some improvements (e.g., in how you do training, or in how you collect data). You can go do this with almost anyone since you are not studying them, you are simply trying out your study design. But it is best if you don't use a member of your team (because they know too much!).

Normally you would want to recruit study participants who are representative of the people who would really use a system like the one you are building, and if you can find such people and motivate them to participate that would be great (but see below on IRB requirements if you want to make a publishable study out of this). But in practice, most of you will actually recruit your classmates. The only restriction on this is to avoid over-using same members of a team because they can get a learning effect.

With that as background, here is what should be in your plan:

As general guidelines, your study design should fit comfortably on four pages, and (unless all you are doing is a user study!) your study should require around 15% of the total time available for your project, including planning, execution, and analysis (9 weeks * 6 hours per week * the number of people on your team * 15% ... for a 6 person team this number is around 50 hours).

If you plan to publish the results of your study, advance approval from LSU Institutional Review Board (IRB) is required. If your results will not be published, that step is not required.


Go back to
Syllabus Page.

Acknowledgement to Doug Oard (LBSC 796/INFM 718R Spring 2011).