Once you have identified the changes in knowing, being, and doing your program is hoping to achieve, you can then develop the outcome indicators to measure those changes.
An indicator is a measure that something has happened—it indicates, or points to, a measure that represents something that happened in one place that had an effect on something else. There are many types of indicators: those for inputs, activities, outputs, and outcomes. When you’re talking about the transformations or changes in knowing, doing, and being for your programs, those are considered outcome indicators.
A lot of people who are involved in providing digital and web literacy programs struggle to define the outcome indicators to use in measuring the results of their programs. Sometimes it’s hard even to know what outcomes are important. For example, does a patron being able to sign up for an email account count as a good outcome? The library’s funders might be more impressed if they were able to show that a patron found a job after learning how to setup email account at the library—but that’s much harder to track!
It’s easy to get discouraged when you’re figuring out what outcome indicators you’re going to use, or identifying for which outcomes to develop indicators. But, if you keep a few criteria in mind as you figure out your outcomes, it can help you come up with some really useful and compelling data about your program.
Criteria in developing indicators are that they are:
Valid and reliable. Valid means that it measures what it claims it measures. If you chose an indicator like “patrons create an email account” and use it to say “this means people are digitally literate,” that would be a big leap! Reliable means that the data is collected in the same way, every time. If the way you collect data doesn’t have any criteria for what that means, or is inconsistent across collection points, that data will not be valuable in measuring outcomes.
Specific and unique. Indicators only measures one thing. The most common issue here is compound indicators that measure more than one concept within the same indicator.
Observable and measurable. Indicators measure something defined, that can be seen, counted, or asked and, in the case of a survey question, people can answer accurately.
Cost-effective to collect. Data collection methods shouldn’t break the bank or take up too much staff time.
Understandable and relevant. What is measured and reported connects to the objectives of the program, and is helpful to program managers.
Time bound. Indicators measure something within a specified timeframe.
Please post three (or more) indicators of the transformations you expect library staff or patrons will achieve after participating in the Web Literacy lessons you are creating and offering. Remember, outcome indicators are often not easy to develop! Don’t worry about whether they can be measured or how, for now...
I would love to hear from you, or answer any questions you may have!
University of Washington, TASCHA