Immeasurable no more

In March 2007, NSF held a two-day workshop on evaluating the impact of projects supported by its Informal Science Education (ISE) program. The workshop attendees included highly experienced evaluators from all of the areas of ISE support, including museum exhibitions, community group projects, films and television, but it also attracted many staff from different areas of NSF as well as those from many other federal agencies, including NOAA, IMLS, NASA, NIH, and the Department of Education. ASTC and the Smithsonian also signed up.

It was apparent that learning about the impacts of investments in STEM education, and being able to document those impacts convincingly, had become a matter of great interest. The two days were fascinating, and not without some sparks flying. Many of the people at the workshop had worked quietly and thoughtfully for years to “measure the immeasurable” (Minda Borun’s memorable early description of exhibit evaluation). Now they were given a spotlight, in a room full of influential and important colleagues. Hard-won lessons, different perspectives and language, and varying visions for the roles evaluation could play in setting policy and making grants, all had to be expressed and debated.

Amazingly (at least to me), by the afternoon of the second day a consensus had emerged about what it was important to say about the theory and practice of impact evaluation, and how the NSF’s new reporting system could help capture what was being learned from each of the projects supported within the ISE portfolio. NSF had invited eight of the participants—Sue Allen, Patricia B. Campbell, Lynn D. Dierking, Barbara N. Flagg, Cecilia Garibay, Randi Korn, Gary Silverstein, and me—to write a book providing a framework for evaluating the impact of ISE projects. We left the meeting with nearly complete agreement on what we would say, and who would write what.

The next nine months, of course, were filled with e-mails, phone calls, and draft after draft after draft, trying to turn this oral consensus and a three-page table of contents into a coherent and readable book. Dave Ucko and Al DeSena were tireless in encouraging the authors and providing advice and guidance on the task. Dave eventually wrote one of the chapters himself, on the context and origins of the project within NSF. We went through three major revisions, and dozens of incremental ones. Eighteen people, inside and outside NSF, were invited to review and critique the document anonymously, and their analyses included line-by-line critiques. The last draft, version 3.12, went to Al, who cleaned up everything one last time. With a click, you’ll have our work.

It isn’t the first word on evaluation, and will not be the last. But on behalf of all of us who sought to provide a useful guide to documenting the accomplishments of the informal science education, we hope you find it helpful and even enjoyable. Cheers, Alan J. Friedman

1 comment February 20th, 2008