In this chapter, we discuss important reasons why a participation analysis should be conducted in conjunction with an impact evaluation of fatherhood interventions, and present methods that may be used to perform such analyses. Participation analysis is usually an important component of formal evaluations of social interventions.
Participation analysis in its broadest sense concerns who participates in the program from among those who are in the program's target population. In an evaluation, however, participation analysis often focuses on who participates conditional on having participated in the study at some level. Under the three alternative designs we are considering, the participation analysis would focus on participation in the program conditional on volunteering to participate in the study. Under a randomized referral design, the analysis is conditioned further -- analysis of participation among volunteers who have been referred.
Analyzing who volunteers from among the program's entire target population is problematic because data on non-volunteers are not obtained in the baseline survey. Those who volunteer are not likely to be representative of all fathers in the program's target population; in particular, fathers who have the least desire to be responsible for their children are unlikely to volunteer.
Unless otherwise indicated, participation analysis, as used in the discussion below, refers to participation conditional on volunteering for the study.
There are several reasons for conducting participation analysis in conjunction with an impact evaluation of a particular program. Below, we describe the three reasons we believe to be most relevant to the evaluation of fatherhood interventions. These include: increasing knowledge of the determinants of program participation; controlling for selection bias; and assessing the effectiveness of outreach and recruiting activities.
The information obtained from conducting a participation analysis can help program staff, funders, and policymakers develop a better understanding of the factors that determine the likelihood that fathers will participate in the program. This can be useful for a variety of reasons. Improved knowledge of the characteristics of those who participate may allow program staff to better tailor their services to those who are demanding them. Participation analyses may also help identify factors that inhibit fathers from participating, allowing program staff to address such potential obstacles to participation. Finally, participation analysis provides the information needed to estimate program participation for populations not previously served by the intervention.
The three evaluation designs rely on the comparison of outcomes for participants to those of non-participants. As discussed in Chapter Five, explanatory variables may be used to control for observable differences between the two groups when estimating the impact of an intervention. If, however, there are unobserved differences between the two groups that are systematically related both to participation in the program and the outcome of interest, the estimate of the treatment effect will be biased. This type of bias is referred to as selection bias. Here, we discuss three potential sources of selection bias: self selection, program selection, and attrition.
Self Selection: Bias can arise if fathers who are more likely to succeed or have positive outcomes are also those most likely to participate in the program. For example, unobserved characteristics such as self-discipline and motivation may affect an father's likelihood of participating in a fatherhood intervention. These same characteristics may also positively affect many of the outcomes of fatherhood interventions, such as contact with the child, employment, and child support. If fathers with these unobserved characteristics are more likely to participate in a fatherhood program (i.e. are self selecting into the program) then estimates of the impact of the program may be biased. In this example, the estimated program impact would be greater than the true impact. Those who participate would have more contact with their children, be more likely to be employed, and pay more child support relative to those in the comparison group, even in the absence of the program.
Program Selection: Bias resulting from program selection effects may occur if program referral, recruiting, or acceptance policies systematically screen-out particular types of fathers from the program. If screening criteria used by program staff are related to the outcomes of interest, there is the potential for selection bias. For example, the Indianapolis FRP uses a rather intensive pre-screening application process. The pre-screen involves several interviews with FRP staff to inform the applicant about what the program involves, to determine how serious the applicant is about participating, and to assess the applicant's ability to participate and potential for successful completion of the program curriculum. The purpose of the pre-screening is to identify and enroll those most likely to succeed in the program. If this manner of participant screening is not accounted for in the evaluation design, the estimated program impact will be biased upward.
Attrition: If participants who drop-out of the program before completion have unobserved characteristics that are systematically related to program outcomes, attrition bias may result. Such a situation is analogous to the self selection bias example described above. In this case, participants are self selecting out of the program. If, for example, participants who drop out of the program are less motivated or less willing to work than those who remain, estimates of the program effect on outcomes such as employment and earnings may be greater than the true impact.
Attrition bias may also arise if follow-up data on some comparison group members cannot be obtained, and these fathers are systematically different from those for whom follow-up data is available. For example, comparison group members who cannot be reached for follow-up may be persons without a stable residence, no telephone, or who become incarcerated or institutionalized. These characteristics are also likely to affect employment and earnings outcomes. In this example, estimated program impacts will be smaller than the true impacts.
Participation analysis may also be used to assess the effectiveness of outreach and recruiting activities. Participation analyses can estimate the effect of specific outreach or recruiting activities on the likelihood that fathers will become program participants. The opportunity to perform a rigorous assessment is offered by the randomized outreach design, where the individuals who receive the outreach (and the type of outreach they receive) are randomly selected, and therefore, selection bias associated with outreach methods may be minimized. Participation analysis may also be used to determine whether the outreach and recruiting activities undertaken by the program are attracting participants from the intended target population.
In general, the assessment of outreach and recruiting activities are restricted to outreach methods which are targeted to specific individuals or groups, rather than aimed at all individuals in a particular area (e.g., ad campaigns at specific locations like schools or churches, versus radio ads that reach an entire area). This is because it will not be easy to determine who has and has not received the outreach when broad-based methods are used. Analysis of the effectiveness of outreach efforts will also be limited by the fact that only the effect on study volunteers can be determined.
In this section, we present an approach to conducting participation analyses. We begin with a discussion of problems associated with defining and measuring program participation. We then present the steps to conducting participation analyses: computing sample descriptive statistics, and conducting multivariate analyses. A more technical description of multivariate participation analysis appears in Appendix E.
One of the more complex aspects of evaluating responsible fatherhood programs (and most initiatives targeting at-risk youth and adults) is determining who is actually being served. Most of the programs we visited (as well as the literature on responsible fatherhood programs) emphasize the importance of providing services that are client-driven and flexible. As a result, potential participants may have several contacts with the intervention before formally enrolling in the program, and some participants may never be formally enrolled. Even after the commitment is made, the participant may come, disappear for a while, and then return for services. While this flexibility may be essential to a successful program and working with an at-risk population, it can complicate the evaluation process because it makes it difficult to determine when someone has become a participant and, in some cases, stopped being a participant.
Evaluation researchers generally identify two basic approaches for defining participation in programs: (1) whether an individual has completed the formalized intake process (e.g., completed an intake form); or (2) exit or completion status.(1) As discussed below, both approaches have potential drawbacks when applied to responsible fatherhood programs.
There are two main problems associated with using a formal intake process to determine program participation: (a) some fathers who complete the formal intake process may subsequently receive few services, or (b) some individuals (e.g., related family members) who do not complete the formal intake process, may receive program services. The completion of a formal intake form may or may not reflect actual and full participation in the program. For example, in the IRFFR program, it is possible for fathers to attend some and even many group sessions without completing the formal intake process. Those attending these group sessions are only asked to record their names as attending the sessions. Formal intake into the IRFFR program in Cleveland occurs when an individual is assigned to an outreach specialist and completes an intake interview. During the initial home visit, an outreach specialist interviews the individual (and perhaps other family members) and completes the intake form. At this point, the individual becomes a protégé and is expected to be available for home visits by the outreach specialist and to attend group counseling sessions. Typically, the outreach specialist would continue to meet several times a month with the individual (and perhaps other family members) to discuss and monitor goal achievement over a three- to six-month (or longer) period. Because of the tailoring of the intervention to each protégé's needs and desires, the duration of participation and types of assistance received varies considerably across protégés.
Completion of the formal intake process does not necessarily mean that the individual completes the program or even moves much beyond completion of an annual plan. For example, in one of the case records that we reviewed, a father was visited several times by an outreach specialist, completed the intake form, and disappeared from the program shortly thereafter (indicating that he expected to return in a month or so). Hence, the inclusion of fathers based on completion of the formal intake process may result in the inclusion of individuals who subsequently receive few or no services -- and hence, may result in the inclusion of individuals into the participant group who have received few, if any, substantive services.
Another potential problem with using formalized intake to determine participation is that there may be fathers and family members who do not complete the intake process, but nonetheless receive services either directly or indirectly through the program. For example, in one of the programs we visited, family members of protégés or individuals for whom funding for the outreach specialist's services could not be obtained often do not go through the formal intake process but may participate in counseling and group sessions and may be greatly affected by services delivered through the program. An alternative may be to define participation as "receipt of at least one service" as opposed to "completed formal intake".
The second approach discussed in the literature -- defining participation in terms of exit from the program or completion status -- has its own set of problems. First, it is sometimes difficult to determine when a participant has completed or is "exiting" a program. As discussed earlier, programs for at-risk youth and adults typically are flexible in terms of service provision (e.g., one site uses an outreach specialist to tailor service delivery to the client's specific needs) and may not impose penalties for irregular program participation. Of the five programs we visited, only two had rather strict participation requirements. Thus, while there may be a core of services that participants generally receive (e.g., in-home counseling and group sessions), it can be difficult to define a core set of program activities that must be received before the individual is considered to have completed participation in the program.
Second, even if a core set of program activities can be defined, individuals who do not receive this core set of activities but receive a substantial level of activities will not be included as participants -- and the evaluation will miss important potential impacts of the program. Programs serving high risk fathers often encounter high rates of attrition, though the administrators at one site indicated that attrition was not a problem for their program. If "participation" is based upon completion of a core set of activities, the evaluation could miss a significant number of individuals who received some (or perhaps a considerable) level of services.(2)
For many programs it may be very apparent what constitutes a "participant". One program we visited has a very structured program with uniform services provided to all participants over a defined, and relatively short, period of time. In this case, it is very easy to determine who is and is not a participant --the father is either attending the daily classes or he is not. For most of the other programs we visited, this was not the case. Staff at two of the programs we visited indicated that they would have difficulty determining exactly how many active participants they currently serve due to the irregular participation of many of the fathers.
In evaluating fatherhood programs, careful attention should be given to the definition of what constitutes program participation. It is possible (and probably likely) that definitions will vary across responsible fatherhood programs according to the targeted population, and the structure and types of services provided.
A first step in conducting participation analysis is to tabulate sample descriptive statistics on:(1) answers to survey questions concerning fathers' knowledge of the program and why they did or did not participate; and (2) characteristics of fathers that are thought to have an influence on participation. A comparison of descriptive statistics for these factors across participants and non-participants can identify factors that are important in determining participation.(3) Descriptive statistics also provide an overview or profile of fathers participating in the study.
In Exhibit 7.1, we illustrate how sample descriptive statistics may be compared for purposes of the participation analysis.(4) For this example, assume that a non-experimental evaluation design is used. In comparing the means across the treatment and comparison groups, we subdivided the treatment group into those who actually participated in the program and those who chose not to participate. This latter group, the treatment group non-participants, can be further subdivided to differentiate between those who chose not to begin the program (no shows) from those who began but subsequently dropped out of the program (drop outs). If attrition in the control group is a problem (i.e., follow-up data for many individuals in the control group could not be obtained), then subdividing the control group into those with and without follow-up data may be necessary.
Variable |
Comparison Group Volunteers | Treatment Group Volunteers | All Treatment | Volunteers | ||
|---|---|---|---|---|---|---|
| Participants | Non-Participants | |||||
| No Shows | Drop Outs | |||||
| Awareness of program and information source | ||||||
| Why participating or not | ||||||
| Age | ||||||
| Race | ||||||
| Employment | ||||||
| Education | ||||||
| Baseline Earnings | ||||||
| Baseline Child Support | ||||||
| Baseline Paternity Status | ||||||
| Baseline Contact with Child | ||||||
The structure of the table will depend upon the evaluation design selected. We have assumed that comparison group fathers are unable to enroll in the program and would have no reasons to be aware of its existence. In an experimental design, control group fathers may be aware of the program, but not be allowed to participated. The evaluator may find it useful to ask control group fathers what they knew about the program, whether they would have liked to participate, and whether they sought assistance from other sources because they could not participate in the program. In the randomized outreach design, the control group would also be subdivided into participants and non-participants.
A preliminary comparison of subgroup means may identify potentially problematic differences between participants and non-participants. A simple comparison of means, however, will not illustrate whether the differences are important enough (i.e. statistically significant controlling for all factors) to warrant the use of statistical methods to correct for potential selection bias in the estimation of the treatment effect. In order to determine the significance of the differences between participants and non-participants, and to control for these differences using statistical techniques, a multivariate analysis is necessary.
The details of the participation analysis will depend on which type of evaluation design is used (experimental, non-experimental, or randomized outreach) and on whether a single-site or multi-site evaluation is performed. We begin by discussing an approach to participation analysis for an experimental, single-site evaluation, then consider modifications necessary for the alternative designs and for a multi-site evaluation.
1. Participation Analysis under an Experimental Design
Under an experimental design, randomly selected volunteers are referred to the program (the treatment group) while others are not (the control group). We assume that control group members do not have the option of participating -- an assumption that is relaxed in the randomized outreach design. Hence, only the volunteers who are assigned to the treatment group can choose whether or not to participate.
The evaluator will need to estimate a particular type of multivariate econometric model for the participation decision -- a "binomial choice" model. The "logit" and "probit" models are the two most commonly used models in this general class. Such models specify that the probability of participation for an individual is a function of a set of explanatory variables. These variables should include baseline variables thought to have an impact on a father's participation decision. It should also include variables that might be used by program staff to decide whether to include a father in the program. These are the same variables that would be used to descriptively compare participants to non-participants within the treatment group, as well as to compare control group fathers to treatment group fathers.
The estimated model can be used to calculate the change in the probability of participation associated with a change in each explanatory variable. For instance, the evaluator may be able to calculate how the probability of participation increases or decreases with the age of the father, the age of the child, the current employment status of the father, etc.
The success of the participation analysis in identifying factors that are significantly associated with the likelihood of participation will depend on both the total sample size for the treatment group and the split of the treatment group into participant and non-participant subgroups. If the sample is small, it may be that all sample fathers with some specific characteristic (e.g., fathers under the age of 18) will all be in either the participant or non-participant subgroup, in which case it will not be possible to investigate the effect of that characteristic on participation other than to acknowledge that, in the sample, that characteristic alone is a perfect predictor of participation.(5)
The estimated model can also be used to compute a "conditional participation probability" for each father in the treatment group -- the probability that the father participates given his observed characteristics alone. The probability is conditional in the sense that it doesn't take into account unobserved factors that affect the father's actual participation decision. It answers the question: "What proportion of fathers with the same characteristics would participate if faced with the same decision?" The estimates reflect the fact that the father was referred to the program and also reflect any screening criteria that are applied by the program in accepting fathers.
Conditional participation probabilities have two specific uses. They can be helpful to a start-up program that is trying predict "demand" for its services, presuming the program has some information about the characteristics of fathers in the target population. A less obvious, but perhaps more important, use is in the impact analysis. As will be discussed in Chapter Eight, conditional participation probabilities play a critical role in separating the impact of a program from "selection effects" -- the effects of self-selection by fathers and screening by programs on differences in outcomes for participants and non-participants.
2. Participation Analysis in a Non-Experimental Design
The appropriate methodology for participation analysis in a non-experimental design is the same as for the experimental design. As we have described that design (Chapter Three), volunteers in the treatment group are in that group rather than the comparison group for reasons that are beyond their immediate control (e.g., their area of residence, or the hospital in which their child was born). Hence, the only choice they have is whether or not to participate in the program when offered the opportunity. This is no different than the choice offered to treatment group fathers under the experimental design.
While the methodology is the same under the two designs, the results have a different interpretation. In the experimental design, results are for volunteer fathers who have been referred to the program, while in the non-experimental design they are for volunteer fathers who happen to be in the program's target treatment population. Thus, the results are conditional on different recruitment and, perhaps, screening mechanisms. These mechanisms must be kept in mind when interpreting the findings.
3. Participation Analysis under a Randomized Outreach Design
In the randomized outreach design (Chapter Three), study volunteers are randomly assigned to receive strong (treatment) or weak (control) outreach. Fathers in either group may decide to participate in the program, but the differences in outreach are expected to result in higher participation rates among fathers who receive the treatment outreach.
Under this design, data for both the treatment and control groups would be used in the participation analysis because fathers in both groups choose whether or not to participate. One of the variables to be included in explanatory variables for the multivariate analysis would be an indicator for the treatment outreach. If the evaluators use multiple types of randomized outreach, the explanatory variables would include indicators for all types. They might also include variables measures of outreach intensity (e.g., the size of any monetary incentives). The coefficients of the treatment outreach variables would measure the impact of the variables on the propensity to participate, and could be easily converted to estimates of the effect of outreach on the probability of participation.
It is likely that participation analysis will be more fruitful under the randomized outreach design than under the experimental or non-experimental designs, for two reasons. First, holding the number of volunteers for the entire study constant, the number used in the participation analysis will be much higher under the former design than under either of the latter -- twice as large if subjects split equally between treatment and control groups. Second, the effects of the randomized outreach itself can be rigorously studied under this design, and may yield results that are important to both program operators and policy makers.
The evaluator may also find it useful to examine whether the effectiveness of the demonstration outreach varies with characteristics of fathers. For instance, the outreach may have been more effective for fathers in some age groups than in others. This will be feasible if the sample size for the evaluation is sufficiently large.
4. Participation Analysis in a Multi-site Evaluation
Opportunities for conducting informative participation analyses are improved in a multi-site evaluation beyond the opportunities available from independent evaluations of each site because of the possibility of pooling data from two or more sites. This will be especially important if sample sizes at individual sites are too small to support meaningful participation analysis.
The same multivariate methodology would be applied in a pooled analysis, but the explanatory variables need to be modified appropriately. Most importantly, variables to indicate the site should be included because participation is likely to be higher in some sites than in others even after controlling for observed baseline characteristics of individual fathers. Cross-site differences may be due to unmeasured environmental differences across sites, unmeasured differences in the target population, and/or unmeasured differences in program administration and the appeal of the program to potential participants. Another possibility is to allow for different effects of various factors across sites. In the extreme, this could mean estimating separate models for each site, but this would result in the loss of any advantage that might be gained from pooling the data. Because sample sizes for each site are likely to be modest, it would be prudent to pool the data unless there are strong prior reasons to believe that the effects of the explanatory variables on participation vary across sites.
The participation analysis for a multi-site evaluation under a randomized outreach design should also include dummy variables to indicate the site. These would capture the effects of all site-specific factors that have an impact on participation at the each site -- unique features of the environment, the program itself, and the target population. In addition, the evaluators may want to interact site dummies with the outreach treatment dummy or, if applicable, the multiple outreach variables. This would allow the evaluator to test the null hypothesis that the effect of the randomized outreach on participation is the same at all sites, and to estimate differences in effects across sites. Such an analysis might be helpful in providing information about subtleties of outreach, or about the environment in which outreach is conducted, that increase or reduce its effectiveness, especially when conducted in conjunction with a process evaluation.
1. See Martha Burt and Gary Resnick (1992). "Youth At Risk: Evaluation Issues," prepared for the U.S. Department of Health and Human Services.
2. If an experimental design is used in the evaluation, all fathers assigned to the treatment group would be analyzed and a correction for "no shows" would be employed.
3. Comparison of means includes comparison of percents for categorical variables.
4. The variables indicated in the exhibit are for illustrative purposes only.
5. Including an indicator for the characteristic in Z would result in a computational failure in maximizing the likelihood function because increasing the magnitude of the coefficient in one direction would always increase the value of the likelihood function.