An Evaluability Assessment of Responsible Fatherhood Programs:

Final Report

By: THE LEWIN GROUP, INC. Project Directors: Burt S. Barnow, Johns Hopkins University and David C. Stapleton, The Lewin Group

Project Staff: Gina Livermore, The Lewin Group

Consultants:

Project Officers: Linda Mellgren for the Office of the Assistant Secretary for Planning and Evaluation, HHS and Ronald Mincy for The Ford Foundation.

August 6, 1997

CONTENTS

ACKNOWLEDGMENTS

CHAPTER ONE: INTRODUCTION AND BACKGROUND

I. Introduction

II. Fatherhood Programs and Evaluation Objectives

  1. The Aim of Fatherhood Interventions
  2. Why Evaluation is Important for Fatherhood Interventions

III. Components of Program Evaluation

  1. Management Information Systems
  2. Process Evaluation
  3. Impact Evaluation
  4. Cost-Benefit and Cost-Effectiveness Evaluations

IV. Program Readiness for Evaluation

  1. Measurable Outcomes
  2. Defined Service Components and a Hypothesized Relationship to Outcomes
  3. Established Recruiting, Enrollment, and Participation Process
  4. Understanding of the Characteristics of the Target Population, Program Participants, and Program Environment
  5. Ability to Collect and Maintain Information
  6. Adequate Program Size

V. Overview of the Remaining Chapters

CHAPTER TWO: PROCESS EVALUATION

I. Introduction

II. Purpose of a Process Evaluation

III. Questions Addressed by a Process Evaluation

IV. Methods for Collecting Information

  1. Primary Data
  2. Secondary Data

V. Analysis and Reporting

  1. Analysis
  2. Reporting

VI. Conclusion

CHAPTER THREE: MAJOR DESIGN ALTERNATIVES FOR AN IMPACT EVALUATION

I. Introduction

II. Experimental vs. Non-Experimental Designs

  1. Experimental Design
  2. Non-Experimental Design
  3. Randomized Outreach Design

III. Single Site vs. Multiple Site Evaluations

  1. Program Services
  2. Program Objectives for Clients Served
  3. Target Populations

IV. Summary

CHAPTER FOUR: OUTCOME MEASUREMENT

I. Introduction

II. Potential Outcomes and Methods of Measurement

  1. Responsible Behavior
  2. Father's Relationship With Child
  3. Father's Support Capabilities
  4. Child Well-Being
  5. Co-parental or Team Relationship

III. Summary

CHAPTER FIVE: EXPLANATORY VARIABLES

I. Introduction

II. Purpose of Explanatory Variables

  1. Estimator Precision
  2. Controlling for Confounding Factors
  3. Program Interactions with Individual Characteristics
  4. Understanding the Determinants of Responsible Fatherhood Outcomes

III. Important Explanatory Variables and Their Measurement

  1. Demographic Variables
  2. Educational Attainment
  3. Work History
  4. Pre-Treatment Values of the Outcome Variables
  5. Site-Specific Factors
  6. Measures of Program Inputs

CHAPTER SIX: SAMPLING AND DATA COLLECTION

I. Introduction

II. Sampling Methodology

  1. Defining the Target Population(s)
  2. Recruiting Study Volunteers
  3. Random Assignment to Treatment and Control Groups, and Program Enrollment
  4. Sample Size

III. Methods of Data Collection

  1. Surveys
  2. Administrative Data

IV. Baseline and Follow-Up Data Collection

  1. Types of Data to Collect
  2. Timing of Data Collection

CHAPTER SEVEN: PARTICIPATION ANALYSIS

I. Introduction

II. Purposes of Participation Analysis

  1. Increase Understanding of the Determinants of Participation
  2. Control for Selection Effects
  3. Assess the Effectiveness of Outreach/Recruiting Activities

III. Conducting Participation Analysis

  1. Measuring Participation
  2. Sample Descriptive Statistics
  3. Multivariate Analysis of Participation

CHAPTER EIGHT: IMPACT ANALYSIS

I. Introduction

II. Methods for Analyzing Program Impacts at a Single Site

  1. Analysis for a Continuous Outcome Variable under an Experimental Design
  2. Application to a Non-Experimental Design
  3. Modifications for a Randomized Outreach Design
  4. Extension to Categorical and Limited Dependent Variables

III. Extension to a Multi-Site Impact Analysis

  1. Experimental Design
  2. Non-Experimental Design
  3. Randomized Outreach Design

CHAPTER NINE: SUMMARY AND CONCLUSIONS

I. Introduction

II. Purpose of this Report

III. Features of Current Fatherhood Programs

  1. Characteristics of Participants
  2. Program Objectives
  3. Service Delivery
  4. Funding Streams

IV. Findings From An Evaluability Assessment of Selected Programs

  1. Measurable Outcomes
  2. Defined Service Components and a Hypothesized Relationship to Outcomes
  3. Established Recruiting, Enrollment, and Participation Process
  4. Understanding of the Characteristics of the Target Population, Program Participants, and Program Environment
  5. Ability to Collect and Maintain Information
  6. Adequate Program Size

V. Next Steps

APPENDIX A: LIST OF EXPERTS INTERVIEWED

I. Telephone Interviews

II. Participants at Practitioners Meeting

APPENDIX B: SITE VISIT SUMMARIES

APPENDIX C: PROCESS EVALUATION INTERVIEW GUIDES

APPENDIX D: EVALUATION OF THE RACINE GOODWILL INDUSTRIES FATHERHOOD PROGRAM

* Because of the mathematicals symbols used in Appendix E, it is only available in .RTF format. Configure your browser to open an appropriate word processor if necessary.

APPENDIX E: TECHNICAL DESCRIPTION OF PARTICIPATION AND IMPACT ANALYSES

I. Participation Analysis

  1. Participation Analysis under an Experimental Design
  2. Participation Analysis in a Non-Experimental Design
  3. Participation Analysis under a Randomized Outreach Design
  4. Participation Analysis in a Multi-site Evaluation

II. Impact Analysis

  1. A Model for a Continuous Outcome Variable under an Experimental Design
  2. Application to a Non-Experimental Design
  3. Modifications for a Randomized Outreach Design
  4. Extension to Categorical and Limited Dependent Variables
  5. An Econometric Model for Jointly Analyzing the Impacts of Multiple Programs
  6. Non-Experimental Design
  7. Randomized Outreach Design

ACKNOWLEDGMENTS

This project was sponsored by The Office of the Assistant Secretary for Planning and Evaluation (ASPE), U.S. Department of Health and Human Services (DHHS) and The Ford Foundation. Linda Mellgren served as the Project Officer for ASPE and Ronald Mincy served as the Project Officer for Ford. They, and others at DHHS including Mark Fucello, Jeanine Smart, Barbara Cleveland, Anne Benson, and Barbara Cohen, provided valuable suggestions and guidance to the project.

This report benefitted greatly from the suggestions, based on a preliminary draft, received from the members of the project's Technical Review Group: Fred Doolittle (Manpower Demonstration and Research Corporation), Ronald Ferguson (Kennedy School, Harvard University), and Jeffrey Smith (Department of Economics, University of Western Ontario).

We are also grateful for the generous assistance of the fatherhood program staff we interviewed during our site visits. We would especially like to thank: Charles Ballard, President of the National Institute for Responsible Fathers and Family Revitalization; Jerry Hamilton, Manager of Disadvantaged Programs, Goodwill Industries of Southeastern Wisconsin; Joseph Jones, Director of the Baltimore City Healthy Start Men's Services Program; Wallace McLaughlin, Director of the Wishard Memorial Hospital Father Resource Program in Indianapolis; and James Worthy, Director of the St. Bernadine's Head Start Male Involvement Project in Baltimore.

Finally, we would like to thank the many academic, government, and fatherhood program experts we interviewed who provided us with much useful information on parenting, child welfare, and fatherhood issues.

Top of Page

Where to?

Table of Contents

Home Pages:
HHS Fatherhood Initiative
Assistant Secretary for Planning and Evaluation
U.S. Department of Health and Human Services

Last updated: May 29, 2002