Below, we summarize the findings of our evaluability assessment, focusing on the findings from the site visits to the five fatherhood programs. We first restate the purpose of this report in Section II. In Section III, we describe features of the fatherhood programs we visited. In Section IV, we summarize where these programs are in terms of their readiness for a formal impact evaluation. We conclude in Section V with a discussion of critical next steps for fatherhood programs to improve their viability and evaluability.
The increased interest in programs that promote responsible fatherhood and the limited information currently available on the services provided and effectiveness of these programs has generated interest in the systematic evaluation of responsible fatherhood programs. For this reason, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services and the Ford Foundation funded The Lewin Group and Johns Hopkins University to conduct an evaluability assessment of responsible fatherhood programs.
Fatherhood programs and emphasis on male parenting are relatively recent phenomena in the social service sector. Many of the programs currently in place are either very new or, if established, have been experimenting with new interventions or changing the program focus over time to meet the interests and objectives of funders. It is generally the case that fatherhood programs have not adequately documented their performance. This may be because of limited resources, a lack of experience with methods of measuring performance, or simply because the focus of program staff has been on serving fathers rather than proving that methods are effective. While program staff may believe that their activities are helping fathers and resulting in positive impacts on society, others, particularly funders, may be skeptical of evidence of program effectiveness that is limited to anecdotes.
Evaluations of responsible fatherhood programs can serve two important functions:
Systematic evaluation of fatherhood program outcomes is crucial to both program design and funding. Conducting rigorous evaluations using standard scientific methods can assist program operators in effectively planning their programs to meet funding requirements, in improving their work with fathers, and in furthering the development of the field of fatherhood research and policy. The goal of this report is to provide the Department of Health and Human Services and other policymakers with an evaluation design that can be used to evaluate a variety of responsible fatherhood programs. In addition, this report is intended to provide direction to organizations that would support or conduct evaluations by illustrating what is involved in the evaluation process and what mechanisms must be in place before a formal impact evaluation may be undertaken.
In developing this report, we conducted site visits of five fatherhood programs nationwide. The site visits allowed us to assess the readiness of fatherhood programs for formal evaluation, identify obstacles to their evaluation, and develop evaluation design alternatives that could be employed if such programs were to be formally evaluated in the future. The five programs selected for site visits were among those that had applied for funding from the U.S. Department of Health and Human Services, and are believed by DHHS staff to be representative of the more developed fatherhood programs in country.
Individual fatherhood programs vary substantially in both the specific outcomes they attempt to achieve and the activities they undertake to achieve them. Among the five programs we visited, we observed substantial variation in the numbers of fathers served, the recruiting methods used, the services fathers received, and program goals. A common theme, however, was the underlying philosophy that in order to be an effective and responsible father, men needed first to develop the capacity to take care of themselves. Below, we describe some of the features of the five fatherhood programs we visited: the characteristics of participants, program objectives, service models, and sources of funding.
The majority of fathers served by the programs we visited shared a set of common characteristics. They were most often young (age 18 to 25), low income, African American fathers with a high school education or less. Fathers were generally unmarried and unemployed and had one or more children, most often under the age of five. Most of the programs served fathers who resided in the immediate geographic vicinity (neighborhood) of the program. Two programs served fathers on a county-wide basis.
The programs we visited varied in terms of the specific outcomes each program was designed to affect. For example, one program has a particular focus on reducing infant mortality and improving child health by increasing the involvement of the father in pre-natal and child health care. This is a very specific objective not shared by the other fatherhood programs we visited. Another program, through its arrangement with the county court system, has increasing the level and consistency of child support payments as one of its primary objectives. This is only a secondary objective of the other programs we visited. There were, however, a number of objectives the programs did have in common. These include:
The above objectives represent those that fatherhood program managers believed to be the most important objectives of their programs. Through our conversations with government agencies and private funders we gained a sense of the objectives that they, as funders, believed to be most important for fatherhood programs to try to achieve. From the funder's perspective, the most important objectives include:
The programs we visited used a variety of means to recruit and enroll fathers. Four of the programs relied heavily on outreach activities conducted by program staff, advertisements, and word of mouth to attract fathers. Two of the programs recruited fathers through contacts with either mothers or children participating in the primary programs offered by their sponsoring agencies. One program relied heavily on referrals from the county social service system, and another received nearly all of its participants via mandatory referrals from the county court system.
Three of the programs we visited have open-entry/open-exit participation policies. Fathers can participate on either a regular or irregular basis. These programs are experiencing difficulty defining exactly who is an active participant in their program. This is because a number of men in their programs do not participate on a regular basis, periodically returning to the program after long intervals of non-participation.
Two of the programs we visited have more defined intervals of participation. One program has a very strictly defined six-week curriculum. Full attendance in all program activities during the six-week period is mandatory in order to continue to be a participant in the program. The other program required court-ordered fathers working less than 32 hours per week to participate in program activities until they were able to pay child support for three consecutive months. After that, participation was optional. In this program, fathers often returned periodically both on a voluntary and involuntary basis.
3. Services Offered
The programs we visited offer a range of specific services with some of the services being very similar across the programs. What differs greatly, however, is the emphasis each program has on particular services and the manner in which they are delivered. For example, all programs offer some form of a men's support group and instruction in parenting. The primary focus of one program, however, is intensive in-home counseling. None of the other programs offer this service. The main focus of another program is classroom-style instruction using a uniform curriculum that covers a range of topics including black history, parenting, and job search and employment skills. The other three programs conduct more traditional case management activities, offering men's support groups and parenting instruction as core services and providing internal or external referrals for services such as GED preparation, employment training, substance abuse treatment, and help with child support enforcement or other legal matters on an as-needed basis.
None of the programs we visited had a single, established, long-term source of funding. For most of the programs, funding comes from a variety of local community sources which often change over time. Such sources include: the county court or social welfare system, local foundations, the state, and private donations. All programs receive some portion of their funding through the Department of Health and Human Services. One of the programs receives nearly all of its funding through the federal Healthy Start program, and one program receives a share of its funding from a large national foundation.
There are several important traits that programs must develop before a rigorous impact evaluation may be conducted. These include:
Below, we describe where the fatherhood programs we visited are in their development of each trait. In general, the programs we visited appear not to be ready for a formal impact evaluation. This is due primarily to three factors: the programs are very new and still at the stage of refining recruiting methods and program services; the programs lack automated systems for tracking and reporting on clients; and the number of fathers served by most of the programs is very small.
Most of the fatherhood programs we visited were able to articulate a set of measurable outcomes believed to be influenced by the program. Among the most common were increased education and employment, reduced alcohol and drug use, improved parenting skills, and increased father involvement with his child(ren). Programs also cited some more difficult-to-measure outcomes, for example, improved attitudes or feelings toward children and improved social and family interactions.
One program had some difficulty defining a set of measurable outcomes influenced by program participation, mostly because the focus of the program was on general attitude change rather than on achieving more easily measured objectives. The primary goal of this program is to reconnect fathers with their children, or, in their words, "to turn the hearts of fathers to their children, and the hearts of children to their fathers." The underlying philosophy and secondary goal of the program is attitude change. Staff at this program believe that reconnecting fathers to their children will lead to changes in attitude and behavior leading to paternity establishment, job placement, and improved relationships with their child and the child's mother. Staff were, however, hesitant to identify specific consequences that could be used in an evaluation of their program.
Of the programs we visited, all were able to define the services they offered and, with the exception of the one program described above, link those services to hypothesized impacts on a set of measurable outcomes. The specific services offered tend to change over time, however. All programs seemed to be in the process of adding new services or refining those already in place. This is probably because most of the programs we visited are only a few years old.
Of the programs we visited, most have established recruiting and enrollment practices. Only one program is in the process of experimenting with new recruiting techniques, as it is having difficulty attracting participants. This program also has a rather lengthy pre-screening process that would be difficult to replicate in recruiting control group members if an evaluation were to be conducted. With respect to program participation, two of the programs we visited are having difficulty defining exactly who is an active participant in their program. This is because a number of men in their programs do not participate on a regular basis, periodically returning to the program after long intervals of non-participation.
All of the programs we visited seemed to have a good understanding of the population they serve and the environment in which the program operates. Many of the program managers live in or near the neighborhoods in which they operate their programs. While all but one of the programs lack an MIS, most of the programs still produce descriptive statistics on important characteristics of their participants, such as age, race, education, marital status, employment, number of children, and paternity status. In addition, most of the program managers we met seemed to be very knowledgeable about and well-linked to other agencies in the community such as state and local health and welfare agencies, child support enforcement, the criminal justice system, and agencies providing specific services to persons with low income such as housing, employment services, legal services, medical care, and substance abuse treatment.
Only one of the programs we visited has any kind of computerized tracking system, and its system was still being developed and modified at the time of our visit. Another program has an MIS, but it is being used only to track female clients enrolled in its primary program. No computerized tracking of male clients is currently conducted.
Most of the programs we visited serve a very small number of individuals, so it would be difficult for an evaluator to obtain statistically significant results. Only one program serves a relatively large number of fathers. The caseload of this program at the time of our visit was about 500 fathers. The program receives from 50 to 60 new referrals each month. This program is by far the exception. Three of the programs we visited serve only about 50 new fathers each year.
There are a number of steps that fatherhood programs can take to improve their viability and evaluability. Given the findings of this study, we suggest the following steps as being the most critical:
Return to ToC