The Strategic Value of Program Evaluation
Program evaluation is a valuable tool to help economic development programs and organizations improve their impact and effectiveness. Faced with increased competition for limited funds and higher expectations from stakeholders, organizations can use program evaluation to advance multiple goals:
Documenting program impacts to build support from funding sources and stakeholders
Improving understanding of what works and what doesn’t
Determining critical success factors to better target resources
Identifying ways to improve operations
Creating tools to regularly document impact and assess effectiveness
Our Approach to Program Evaluation
Karl F. Seidman Consulting Services uses a three-dimensional approach to program evaluation to maximize the evaluation’s strategic value to your organization. The three dimensions of the evaluation include:
Defining the “Program Model” is a key first step to inform a sound evaluation. This “Program Model” is a concise, logical flowchart of your program’s strategy that relates your program “outputs” to the desired economic development “outcomes”. It identifies the chain of actions, or pathway, needed for your program to be effective and clarifies the appropriate impact measures for both “outputs” and “outcomes”. By elaborating the critical path to achieve your goals, the program model also helps identify key intermediate results, operating issues and relationships to evaluate.
1. Evaluating Impacts: Addressing the Attribution Problem
A critical issue for any program evaluation is determining whether the observed outcomes were caused by your program or would have occurred in the absence of your assistance. This issue is akin to the problem that Clarence the angel faced in movie, It’s a Wonderful Life: how to show what Bedford Falls would be like if George Bailey had not lived. There are three approaches to addressing the attribution problem:
- Experimental Design involves prior random assignment to experimental and control groups to compare the outcomes for those who participate in the program (the experimental group) with those who do not participate in program (control group).
- Quasi-experimental Design where a comparable control group is constructed after the fact, i.e., after the program assistance occurs to compare outcomes between those receiving assistance and those who did not receive assistance.
- Client surveys where program participants project what would have happened without the program intervention.
The most effective approach to addressing attribution is chosen based on the nature of the program, the timing of the evaluation, and the resources available for the evaluation.
2. Evaluating Program Operations
Beyond documenting impacts, the evaluation process is extremely useful in identifying ways to improve program strategy and operations to better achieve the desired results, use resources more effectively, improve client satisfaction and increase impacts. Our evaluation approach employs five tools to assess program effectiveness:
- Comparing actual intermediate outcomes to the expected pathway of change to assess whether the expected pathways and mechanisms to reach the intended impacts are occurring and what accounts for unintended results.
- Mapping the program’s transactions and processes to generate a detailed understanding of program operations and procedures. This mapping can uncover inefficiencies and potential areas for improvements.
- Benchmarking against comparable programs establishes performance standards for judging program impacts, benefits and costs. It can also identify where a program’s greatest strengths and weaknesses lie.
- Surveying clients is an invaluable way to get input from your most critical stakeholders on what is working and how to improve your program.
- Interviewing key stakeholders and partners to learn from their experience working with a program and their perceptions and recommendations about best practices and areas for improvement.
For all these steps, a knowledgeable independent evaluator is an essential and critical asset. The independent evaluator provides confidentiality to clients and stakeholders, yielding more honest and comprehensive feedback on a program while also bringing an objective and new vantage to the analysis.
3. Assessing Relationships
Successful economic and community development programs are built on strong relationships with multiple partners. With the devolution of federal programs, a greatly expanded non-profit sector, and the growth in philanthropic and corporate partnerships, today’s environment demands attention to building a strong network of relationships to support and enhance your economic development program. The third dimension of our evaluation approach assesses the strength and strategic content of current relationships while identifying new relationship-building opportunities. Three tools guide our evaluation of program relationships:
- Identify critical relationships based on your program model and operations. This analysis encompasses relationships needed to: (a) market the program and secure referrals; (b) secure complimentary actions, investment and services, (c) generate desired goals and outcomes from aligned and complimentary actions related to a program’s direct activities.
- Assess the scope and effectiveness of relationships from program data, staff and stakeholder interviews, and client surveys.
- Scan the environment to uncover new relationship opportunities.
Gain the Advantage of Direct Program Management Experience
Karl F. Seidman Consulting Services brings direct program management experience to the task of program evaluation. Karl Seidman uses his knowledge and insight from managing state and local government economic development programs to create sound strategies and recommendations to improve the effectiveness and impacts of economic development programs.
About our Program Design and Evaluation Experience
ABOUT
Karl F. Seidman Consulting Services advises clients on the planning, implementation, and evaluation of economic development strategies, policies and programs and on the analysis, planning and financing of real estate development projects.