Selected Evaluation & Strategy Projects Policy Change Infrastructure Evaluations Evaluation of Rural Policy, Rural People, W. K. Kellogg Foundation OMG is conducting a multi-year evaluation of this national initiative that aims to nurture a rural policy infrastructure both nationally and in four regions. OMG is evaluating progress in network building, strategic communications, and policy capacities as well as the actual impacts on rural policies. Cluster Evaluation of Networks for Rural Policy Development, W. K. Kellogg Foundation OMG conducted a three-year cluster evaluation of W. K. Kellogg Foundation grants to innovative local rural economic development organizations in Central Appalachia, national associations with rural policy interests, and media and communications organizations. The W. K. Kellogg Foundation piloted this Central Appalachian-focused work to increase its grantees' strategic communications capacities. It also allowed their regional and national networks to learn how to pursue their objectives within a policy framework. The evaluation incorporated surveys, qualitative interviewing, and a network analysis. In addition, OMG facilitated cluster-wide meetings to build the networks and share resources. The Listening Project for the Ford Foundation's Electronic Media Policy Portfolio of the Knowledge, Creativity & Freedom Program To help guide a new Ford Foundation program officer's new portfolio, OMG assessed the current state of the emerging electronic media policy field. For what became known as The Listening Project, the OMG team conducted interviews and focus groups across the country with grassroots organizers, policy advocates, academics, and funders to listen to those working on electronic media policy issues to determine needs, assets, theories of change, and critical opportunities. Findings were shared through a final report, a series of convenings, and conference presentations. These catalyzed critical cross-sector conversations about the potential for movement building, and assisted the Program Officer with the formation of her grantmaking strategy. Evaluation of the Ford Foundation's Electronic Media and Technology Policy Portfolio of the Knowledge, Creativity & Freedom Program Using the knowledge gained from the Listening Project, OMG was engaged by the Ford Foundation's Electronic Media Policy program officer to evaluate the early outcomes of the grantmaking. Incorporating an online survey, in-depth interviews, and case studies, the product describes how leaders and organizations work on their own and together to make electronic media policy change, and the early outcomes and challenges of the grant work. Education Evaluations Evaluation of the Program for College Access and Success, Lumina Foundation for Education OMG is conducting a multi-year evaluation of this national college access and success program in nine cities across the US. Using multiple methods including a national survey, the evaluation began with cross-site and local theory of change work, and is evaluating grantee progress and their outcomes in systems change. To read more about the program, see What We Know about Access and Success in Postsecondary Education: Informing Lumina Foundation's Strategic Direction on the Lumina Foundation website. Evaluation of the McCabe Fund, Lumina Foundation for Education OMG is conducting a two-year evaluation of the McCabe Fund, a program supporting organizations that help students--particularly first-generation college students, low-income students, and students of color--gain access to postsecondary education. In addition to evaluating the impact of the imitative, OMG is offering evaluation coaching to the McCabe grantees. Program Evaluation, Philadelphia High Schools Academies, Inc. OMG conducted a multi-year evaluation of the Philadelphia High Schools Academies, Inc., a 25-year-old educational organization that works in partnership with the business community and designated schools in the School District of Philadelphia to strengthen students' academic and occupational skills, increase job preparedness, enhance their in-school and post-graduation options, and supplement the existing curriculum. Along with qualitative data analysis, the evaluation included analysis of quantitative achievement data of its students with similar cohorts, and assistance with defining the program's key issues, progress, and near-term programming. Evaluation of the Arts Education Partnership Meetings OMG evaluated the effectiveness of the Arts Education Partnership's (AEP's) annual three meetings. Using interviews and an online survey, the work explored attendees' perceptions about meeting content and format and how well the meetings shared effective practice and created network opportunities. Findings and practical recommendations made to AEP's staff and the national steering committee were quickly incorporated to improve future meetings. Evaluation of the Chicago Arts Initiative, Chicago Community Trust OMG is currently evaluating a five-year initiative sponsored by the Chicago Community Trust, which demonstrates successful approaches to deliver quality, sequential arts education in the four arts disciplines -- visual arts, music, dance, and theater -- in eighteen K-8 classrooms. Beginning with theory of change work, the evaluation is a formative and summative evaluation in which the work focuses on the change process and impacts of the demonstrations on the Chicago Public School system, demonstration clusters, schools, administrators, teaching practice, and students. The evaluation also explores the impact and delivery of arts curriculum by arts and culture organizations. Evaluation of the Arts Education Initiative, Ford Foundation OMG is currently evaluating the Ford Foundation's Arts Education Initiative in eight US cities. The Initiative intends to build sustainable local partnerships and public will for policy in support of integrated arts education. The grantees are strengthening multi-stakeholder partnerships, and strategic communications capacities, as well as to bring to scale already strong and robust arts integration programs. Program-wide and site-specific theory of change work, and site evaluation plans, will be products of the study. In addition, the OMG team will develop a field-oriented theory of change paper to advance arts education evaluation practice. Evaluation of the New Jersey ACES, a partnership between the New Jersey Department of Education, New Jersey Council for the Arts, and the New Jersey Alliance for Arts Education The objective of ACES (Arts Create Excellent Schools) is to demonstrate how the arts can be a part of the core curriculum through the support of the development of four arts-focused schools, and to disseminate knowledge for replication across the state of New Jersey. The evaluation was formative and summative and relied on case studies and quantitative student-performance data. Evaluation of ArtsEdge, The John F. Kennedy Center for the Performing Arts OMG conducted a multi-year assessment of the prototype phase of ArtsEdge, and subsequently evaluated the pilot phase of the program. ArtsEdge was conceived as an online resource to be used by teachers, students, and arts practitioners for purposes of planning and implementing effective arts education programs while linking them with current education reform efforts at state and local levels. OMG assessed progress toward a more refined format, the expansion of its content and user base, and long range organizational planning. Documentation of the Alliance Schools Project in Texas, Annie E. Casey Foundation OMG conducted this study of the Interfaith Education Fund's Alliance Schools with the aim of examining: organizing strategies and processes for addressing school reform issues at the individual school and district levels; the relationship between organizing for school reform and broader community transformation efforts; and the role of organizing in the development and strengthening of social networks among low-income parents. This work also examined and contrasted educational outcomes for all original Alliance Schools in Texas with a comparable cohort of schools. Read more on The Annie E. Casey Foundation website: Success in School: Education Ideas that Count. Children, Youth, & Families Evaluations Evaluation of Hurricane Katrina Communities, Junior Achievement Worldwide OMG is evaluating the impact of Hurricane Katrina on the Junior Achievement (JA) programs in Louisiana, Alabama, and Mississippi. Junior Achievement offers school-based programs to expose youth to careers in the business world. Because of the hurricane disruptions, JA has turned to non-traditional alternatives to continue to serve youths in these communities. JA feels there are lessons about these adaptations for other types of crisis situations. Evaluation of the Childhood Obesity Prevention Sites, Robert Wood Johnson Foundation OMG is conducting an evaluation of the early progress made by eight Childhood Obesity Prevention sites, which fall under the Foundation's Injury Free Kids Coalition umbrella. In addition, OMG will provide grantees with training on aspects of program evaluation to enhance their skills with self-evaluation and with supporting external evaluations. The Discovery Initiative, William Casper Graustein Memorial Fund OMG is engaged in a multi-year evaluation of the Discovery Initiative, which aims to improve educational outcomes for children ages 0 - 8 in fifty Connecticut communities. In each community, a collaborative is implementing strategies to improve the quantity and quality of pre-school opportunities available; strengthen the transition from pre-school to kindergarten; and improve social, emotional, and educational outcomes for children. OMG and its evaluation partner, The Center for Assessment and Policy Development, will conduct a cross-site analysis of data from all sites, intensive case studies of six sites, and a policy study examining whether and how the initiative contributes to state or local policy changes. OMG will also be working with the Discovery communities to increase their capacity for self-evaluation. New Haven Grantmaking Initiative, Annie E. Casey Foundation OMG is currently evaluating the New Haven Grantmaking Initiative, a citywide project that aims to strengthen neighborhoods and families. OMG has worked with the funder to clarify the theory of change and identify the program's pathways to change. Program interventions include a direct services small grant program, strategic investments to promote neighborhood development and family strengthening, and convening of key stakeholders to develop partnerships and strategic alliances. Evaluation of the Data Systems Development and Management Cluster, William Penn Foundation OMG conducted a cluster evaluation of database development and management grantmaking for the Communities and Environment, and the Children, Youth and Family Programs. As part of a two-prong evaluation, OMG evaluated the demand side, or regional grantees' access to and use of the databases. Robert Hollister of Swarthmore College, led the supply side of the evaluation, which focused on the quality of the databases' design and architecture, data quality, and maintenance. Findings of both studies were used to inform the Foundation's future strategy, and were shared with the larger regional grantee population. For more, read the Cluster Evaluation Summary on the William Penn Foundation website. Evaluation of the Achieving Independence Center in Philadelphia, Department of Human Services, City of Philadelphia This process evaluation focused on documenting and extracting lessons learned from the planning and first year of implementation of a program created to support youth transitioning out of the foster care system so that they could develop necessary life skills, access jobs and career opportunities, and secure housing in order to become successful independent adults. Evaluation of the Differential Response System in Hartford, CT, Department of Children and Families, State of Connecticut OMG conducted a process evaluation of a community-based intervention designed to prevent children from entering the foster care system by working with at-risk families in their own neighborhoods. Learning Clusters, Marguerite Casey Foundation OMG facilitated the development of a learning community of eight Marguerite Casey Foundation grantees for the purpose of helping them learn from each other, connecting their constituents, and discovering new ways of strengthening the voices of the families they work with. Evaluation of Equipo - An Approach for Supporting Families and Building Social Capital, Annie E. Casey Foundation EQUIPO, or Team, was conceived as an innovative way of delivering services and supports to families in East Little Havana, a largely Hispanic neighborhood in Miami. EQUIPO relies on a network of “natural helpers” -- called Madrinas and Padrinos -- to bridge the difficulties that these residents experience in accessing services for their families. OMG's evaluation focused on three main areas of impact outcomes for families receiving services through EQUIPO: community building through the expansion of networks; development of new linkages in and outside the community; and the impact of EQUIPO participation on team members. Cluster Evaluation of Non-School Hour Literacy Programming for the Children Youth and Family Program, Heinz Endowments Moving from opportunistic grantmaking toward a more strategic strategy, the Heinz Endowments engaged OMG to evaluate the formative issues and impacts of early grantmaking that supported non-school hour literacy programming in the Pittsburgh region. The work presented recommendations for strategically focusing the work and strengthening outcomes, and developed a framework for an RFP and field strengthening process. The JA Academy Program, Junior Achievement Foundation OMG evaluated the pilot JA Academy Program, operating at seven sites in the metropolitan Boston area. High school students participating in the course create and run a company. The pilot is located in a university setting and includes a tour of the campus and a visit(s) to a business. The research uses pre- and post-tests to measure changes in students' knowledge of business and economics; planning, organization, and teamwork skills; and attitudes toward school and the future. Surveys to gauge students' and instructors' satisfaction with the course were also included in the assessment. Evaluation of Community Mobilization Project, National Association of Child Advocates OMG completed an evaluation of NACA's Community Mobilization Project, which funded by the W. K. Kellogg Foundation to strengthen the participation among low-income and minority communities in framing state policy discussions around improving services for children. For this evaluation, OMG assessed community mobilization activities in Wisconsin and Washington, states where the efforts of the NACA affiliates appear to have led to the significant engagement of “new voices” in their advocacy work around the health and human services policies being developed by states under the federal devolution process. Arts & Culture Evaluations Evaluation of the Arts and Culture Program's Operating Support Initiative, Heinz Endowment Through grantee interviews, strategic plan reviews, and financial data analysis, OMG evaluated the progress of the second phase of the Endowment's core operating and support program for arts and culture. Recommendations from the evaluation were incorporated into the program's redesign. Evaluation of the Multi-Cultural Arts Initiative, Pittsburgh Foundation and the Heinz Endowments The evaluation focused upon assessing how joint foundation support has strengthened multi-cultural arts in Pittsburgh, and how the foundations might better work together to support this effort. A set of more realistic outcomes was identified, as were a set of more focused grantmaking and technical assistance guidelines. Logic Model and Plan Development for the Heinz Endowment's Arts and Culture Grantmaking Program OMG worked with the Arts and Culture Program staff to develop a strategic logic model of their full program, and to create a plan for integrating evaluation into the ongoing work of their grantmaking program. Along with the final logic model product, the work helped sharpen the program strategy, ground expectations for outcomes, and it also created conversations about outcomes-thinking with the arts and culture community when the logic model was tested with them. Evaluation for Arena Stage, Washington, DC OMG performed an evaluation of audience development and community outreach programs. Although the evaluation investigated the full program, it also focused upon a strategic review of the fit between its community outreach theater, Living Stage, and its primary mission as a regional American theater. The work included a detailed cost/benefit analysis. Leadership Development Evaluations Eisenhower Fellowships OMG conducted an assessment of the Eisenhower Fellowships, a leadership development program for mid-career professionals from around the world. The program was started in 1955 to honor President Eisenhower, and over the years has brought over 1,000 emerging leaders to the US for travel, meetings, and site visits with American leaders and institutions. OMG's evaluation included: a review of program activities and expected outcomes (via pathway mapping); fifteen in-depth case studies of program alumni; and an online alumni survey. The research yielded evidence that the fellowship leads to strong individual-level outcomes such as increases in knowledge, skills, worldview, and moderate organizational-level and societal-level outcomes. For more, read Executive Summary of the Final Report on the Eisenhower Fellowships website. Leadership for a Changing World, Ford Foundation OMG is engaged in a multi-year formative and outcomes evaluation of the Ford Foundation's Leadership for a Changing World (LCW) program, which recognizes the achievements of outstanding social justice leaders who are not yet well-known outside their immediate communities. LCW provides financial support for their work, conducts research that explores their leadership, and encourages increased public conversation about community leadership. OMG initially worked with the program partners to articulate the program's theory of change, and assessed its implementation. OMG's work led to several program refinements and improvements. OMG has fielded several online and paper surveys, conducted focus groups and individual interviews with program staff and participants, observed meetings, maintained a database of media coverage of the program, and assessed program outcomes. Next Generation Leadership, Rockefeller Foundation OMG conducted an evaluation of the Next Generation Leadership (NGL) program. Launched in 1997, NGL focuses on identifying and nurturing leaders, providing opportunities for examining challenges to democracy and social justice, and for developing strategic collaboration among program participants. OMG assessed the effectiveness of the program's design and implementation, gathered evidence of its short-term impacts on the organizations and communities of the NGL fellows, and took stock of the contributions that NGL has made within the field of leadership development programs in the US. OMG used a theory of change approach to surface program assumptions and identify anticipated outcomes, which provided a framework for the evaluation. OMG also designed and conducted an online survey of current awardees and alumni of the program. For the full report, download the Next Generation Leadership Program: Final Assessment Report (PDF) from OMG's website. Community Building Evaluations Evaluation of the Sustained Excellence Awards Program, Fannie Mae Foundation OMG is concluding a three-year evaluation of the Sustained Excellence Awards Program, which has provided grant and loan support to a group of ten well-established and successful CDCs from different parts of the country. The program support has enabled each CDC (community development corporation) to move forward with specific housing and community development agendas while strengthening the capacity of their organizations and communities to sustain their visions and programs. The program has also supported a series of semi-annual meetings in which the CDCs shape an agenda that promotes cross-site information sharing and learning. OMG developed the overall evaluation framework for this program, conducted semi-annual site visits and interviews with the CDCs, and tracked grantee progress both individually and together. Evaluation of Neighborhood Progress, Inc. (NPI), The Cleveland Foundation and The George Gund Foundation OMG recently concluded an assessment of this local community development intermediary in Cleveland. This assessment was commissioned to review the intermediary's ten-year history; document its progress in meeting goals around neighborhood impact, CDC capacity-building, system-building, and sustainability; and to make recommendations on programs, operations, and governance. This work reviewed a wide set of housing and economic development projects and NPI's CDC operating support program. OMG also conducted interviews with CDCs, housing and community development professionals, business, government and civic leaders, and representatives of academic institutions. This evaluation built upon earlier OMG engagements focusing on NPI. Evaluation of the Detroit Community Development Funders' Collaborative OMG served for five years as part of a joint evaluation team with the Boston-based firm of Stockard, Engler & Brigham in conducting this assessment. The multi-funder collaborative was conceived as a way to strengthen the capacity of Detroit-based community development organizations engaged in neighborhood revitalization. This program has two primary areas: strengthening non-profit community development organizations; and supporting a community development system through the expansion and strategic use of financial and technical resources for neighborhood-based housing production and community revitalization efforts. Urban Vacant Land Study, Philadelphia Green, Pennsylvania Horticultural Society OMG conducted an examination of innovative methods for managing and reusing urban vacant land as part of a larger study that framed the problems associated with urban vacant land, and provided recommendations for new approaches. The study included a national scan of private, non-profit, and public responses to the growing dilemma with an extensive examination of government policies. The study promoted a new way of thinking about vacant land and attempted to redirect the conversation about the issue from “vacant land as a problem” to “vacant land as a resource.” For more, read an abstract from the study published in 2002 entitled: Old Cities/Green Cities, Communities Transform Unmanaged Land by the American Planning Association. Evaluation of the Rebuilding Communities Initiative, Annie E. Casey Foundation OMG evaluated the Rebuilding Communities Initiative, a national community development program in five cities funded by the Annie E. Casey Foundation. The assessment examined the initiative on five levels: changes in the character and capacity of community institutional infrastructure; perceived changes in quality of life of the neighborhood; impacts on the target organizations involved in the initiative; changes in larger systems providing resources and services; and the role, performance, and impact of the Foundation. Evaluation of CCRP, Comprehensive Community Revitalization Program OMG staff conducted a multi-year evaluation of CCRP (Community Revitalization Program) in New York. The assessment was designed to document the effort and assess which strategies prove effective in moving toward comprehensive neighborhood revitalization. The final report summarized the achievements and challenges of this program, and offered recommendations for future comprehensive community initiatives. For more, download the CCRP Final Assessment Report (PDF) on OMG's website. Sustainable Development Evaluations Evaluation of the Countryside Institute's International Stewardship Exchange Program, Glynwood Center, Cold Spring, NY OMG did an assessment of the Countryside Institute's International Stewardship Exchange Program. Having the education of professionals and communities in sustainable development as its primary objective, the assessment explored the impact of the Exchange on participating professionals and on six host communities. It addressed issues of local and regional public/private systems change, and made recommendations that included shifting the model to improve the program's ability to facilitate community capacity for change. Evaluation of Philadelphia Green's Community Greening and Parks Programming, William Penn Foundation and The Pew Charitable Trusts OMG conducted a multi-year formative and summative assessment of the community-based activities of the Pennsylvania Horticultural Society. The work included extensive interviewing, case studies, quantitative and qualitative data collection and review, and analysis incorporating GIS formats. The findings are being used by Philadelphia Green's staff to strengthen its overall programming.