In this chapter we discuss the use of explanatory variables in an impact analysis. We begin with a discussion of the reasons for using explanatory variables in a multivariate analytical framework. We then describe explanatory variables likely to be included in an analysis of responsible fatherhood program outcomes.
While some program evaluations simply compare outcomes for treatment and control group subjects (e.g., difference in means and difference in percent analyses), more frequently multivariate techniques (e.g., multiple regression and logit), are used to compare outcomes after adjusting for a set of explanatory, or control, variables. There are several reasons for using explanatory variables in multivariate models, and an understanding of these reasons is helpful in determining the value of collecting data for explanatory variables in a specific evaluation, and the types of data to be collected.
The reasons for using explanatory variables in multivariate models are, in brief:
We elaborate on these reasons below.
Estimates of program effects based on a sample of outcomes for participants and non-participants are subject to random estimation error due to "idiosyncratic factors" -- factors other than program participation that affect outcomes for those individuals. The "standard error of the estimate" is the commonly used measure of how large estimation errors are likely to be. As a rule of thumb, the chance that the absolute value of the estimation error is greater than twice the standard error of the estimate is about five percent.
The size of standard errors depends on, among other things, how much variation there is in outcomes across fathers because of idiosyncratic factors. The more such variation, the more difficult it is to determine whether differences in treatment and control group outcomes are due to such factors rather than to program participation. Idiosyncratic variation can be reduced by including explanatory variables that explain some of that variation. For instance, if some of the idiosyncratic variation is due to variation in the age of the father, then using father's age as a control variable would explain part of the idiosyncratic variation. If the reduction in idiosyncratic variation is large enough, standard errors for estimates of program effects will fall.(1)
If subjects are randomly assigned to treatment and control groups, then differences in outcomes between the two groups that are not due to the effect of the program are random. In the absence of random assignment, however, the differences may be due to systematic differences in the characteristics of subjects that are related to how the subjects were assigned to the two groups. For instance, if older fathers are more likely to be participants than younger fathers, and if age is positively related to a desirable outcome, then a positive difference between the mean outcomes for the treatment and comparison groups will be due, at least in part, to the fact that treatment fathers are, on average, older than comparison group fathers. Attributing this difference to the impact of the program would be misleading -- estimated program effects would likely overstate the true effects (positive bias). Using father age as an explanatory variable will remove this source of bias, as would controlling for other characteristics that may vary substantially across the two groups.
Many potentially systematic differences between treatment and control fathers are difficult to measure, for either conceptual or practical reasons. Those factors which remain constant over time can be controlled for by using a baseline (pre-program) value of the outcome variable as a control variable. For instance, the evaluators might compare the mean change in hours per week spent with the child over the evaluation period for the two groups, rather than mean hours at the end of the period.
Comparing changes in outcomes can be misleading, however, if the baseline value of the outcome variable is related to the individual's participation decision. For instance, suppose that fathers with low hours of child contact are more motivated to both increase hours and to participate in the program than those with more contact hours -- precisely because their current contact hours are low. Such fathers are likely to achieve greater increases in contact hours than those with higher initial hours even if they do not participate in the program, so attributing the full difference in the mean change in outcomes to participation will overstate the impact of participation on the outcome.
Not all fathers will respond to a program in the same way, and for policy purposes it may be useful to know that the program has more favorable effects on some classes of fathers than on others. Given limited resources, it may make sense to target benefits toward those fathers on whom the program is likely to have the most favorable impact.
The simplest approach to determining whether there are interactions between the impact of a program and father characteristics is to divide treatment and control or comparison group fathers into subgroups, based on the characteristics and to compare outcomes across treatment and control or comparison subgroups with the same characteristic(s). This approach will be unsatisfactory with small samples, however, as is likely to be the case for a responsible fatherhood program evaluation.
Given samples that are too small to make statistically meaningful treatment/control comparisons within subgroups, some success in measuring interactions may be achieved by specifying multivariate outcome models for all treatment and control or comparison group members in which dummy variables for program participation interact with explanatory variables for key individual characteristics -- characteristics that may be related to the size of the program's impact.(2)
While the main objective of an evaluation will be to determine the impacts of responsible fatherhood programs on the behavior of fathers and the well-being of their children, an evaluation can also enhance our general knowledge about the proximate causes of desirable, or undesirable, fatherhood behaviors. That is, the evaluation can help answer the question: What are the characteristics of fathers that are associated with the most desirable, or least desirable, outcomes? Such information could be useful in designing policies to promote responsible fatherhood, regardless of the program impacts.
The variables chosen for inclusion as explanatory variables in a multivariate model should be factors that vary across fathers in the sample and that are believed to influence or "explain" differences in the outcome being estimated. While the choice of explanatory variables will depend on the specific outcome being analyzed, the variables discussed below are likely to be important explanatory variables in an evaluation of responsible fatherhood program outcomes.
Demographic variables such as age and race/ethnicity allow the evaluator to describe the characteristics of fathers who participate in both the treatment and control groups, ensure that the two groups are comparable, and, if not, control for the differences by including demographic characteristics as explanatory variables in the multiple regression model. Demographic variables may also be important in explaining differences in the program outcomes of interest. Age may be measured as a single continuous variable representing years or as one or more categorical variables (e.g. age less than 18, 18 to 24, 24 and over). Race categories typically include black (African American), white (Caucasian), and "other." The choice of racial categories and whether or not to use race as an explanatory variable will depend on the race composition of program participants. In addition, ethnicity may be used as a control variable if there is reason to believe that there will be differences in outcomes between, for example, "Hispanics" and "non-Hispanics" or among subgroups of Hispanics.(3) Race and ethnicity may also be combined into a single set of variables.
Educational attainment, as measured at baseline, may be an important predictor of program outcomes. Educational attainment is most commonly measured as the highest grade or year of school completed. The variable may enter the analysis as a continuous variable (years of education), but more often is used as a categorical variable. An example of a categorical scheme for an educational attainment variable might be: less than high school education, high school graduate, and education beyond high school. A separate category for high school graduates with general educational degrees (GED) is often added.
Because fatherhood programs often serve very young fathers, it is important to devise educational categories that reflect "age appropriate" levels of education. For example, a sixteen year old who has not completed a high school education should not be grouped with a twenty-one year old without a high school education. As with all explanatory variables, it is important to choose categories that are meaningful in relation to the outcome being estimated and that contain more than just a few observations within each category.
Explanatory variables reflecting work history will be important to include in models that estimate program effects on work related outcomes such as employment and earnings. Factors such as years of experience and levels of prior wages or earnings are likely to be important predictors of post-program employment and earnings. Prior work experience may be measured as an indicator variable (has/has no prior experience in a formal job), as a continuous variable (number of years working in a formal job(s)), or as a categorical variable (no previous job experience, less than one year experience, 1 to 3 years, etc.). Prior earnings may be measured in terms of hourly wages and/or weekly/monthly/annual earnings. Depending on the outcome variable of interest, it may be important to know both components of earnings (hourly wage and hours of work), and therefore collect information on both wages and hours of work for use by themselves and to validate information collected on earnings.
The reason for including pre-treatment values of the outcome variables is that the evaluator will inevitably not be able to measure many of the factors that directly affect the post-treatment values of the same variables, and these unobserved factors are likely to have similar effects on the pre-treatment values. Including pre-treatment values of the outcome variables helps control for these unobserved factors. Very commonly, there will be multiple outcomes of interest, and a regression model should be estimated for each outcome.
Typically, only the pre-treatment value of each model's outcome variable is included among the explanatory variables for that model. Alternatively, one may use the change in the outcome variable as the dependent variable in the regression (as opposed to the level), omitting the pre-treatment value as an explanatory variable. This changes the interpretation of the regression estimates somewhat. For example, if affecting the level of child support payments is a program outcome of interest, the evaluator may estimate a model of child support payments using the post-program level as the dependent variable and the pre-program level as an explanatory variable. Alternatively, the effect of the program on child support payments may be estimated by using the change in the level of child support payments (the difference between pre- and post-program levels) as the dependent variable.
Inclusion of pre-treatment values of the outcome variable may substantially reduce the usefulness of other explanatory variables since the pre-treatment values of the outcome variable may capture most of the important effects of other variables on post-treatment outcomes. This, however, cannot be determined a priori and therefore it is important to obtain information on other explanatory variables. Further, these variables will be of interest for other reasons, such as analysis of impacts on specific subgroups and for use in participation analysis (discussed in Chapters Seven and Eight).
If conducting a multi-site evaluation, or if choosing a comparison group located in a different geographic area, it may be important to include variables reflecting environmental factors that affect the outcomes of interest and that vary by site. For example, if employment is one outcome of interest, it may be necessary to control for differences in labor markets across sites by including the unemployment rate as an explanatory variable. Another important environmental variable is the policy environment surrounding fatherhood related issues in a particular area. For example, child support enforcement methods and personnel in one area may be antagonistic toward fathers; in another area, they may operate in a manner that encourages cooperation with fathers. Other examples of environmental factors that may be related to outcomes of fatherhood programs include: the poverty rate, the rate of welfare recipiency, per-capita income, and crime rates.
Alternatively, a site-specific dummy variable may be used to capture all environmental differences across sites. This may be used if treatment and control groups within each site are believed to be comparable with respect to the important environmental factors, and therefore may be assigned the same site-specific dummy variable. For example, suppose a multi-site evaluation of a program operating in Cleveland and San Diego is conducted using a non-experimental comparison group design where the control groups for both sites are chosen from a geographic area adjacent to the area in which each program operates. A single site-specific dummy variable differentiating Cleveland from San Diego may be assigned, with the same value being assigned to both the treatment and control group within each site. This is possible only if it is believed that the environmental factors affecting the treatment group and control groups within each site are the same.
If, however, environmental factors differ between the treatment and control groups within each site (i.e., the adjacent geographic area from which the control groups were chosen differs substantially), then the site-specific dummy variable approach is not feasible. This situation would require assigning a different variable for each of the control and treatment groups at each site, but these variables would capture the treatment effect as well as the effects of differences in environmental factors.
If program participants receive varied types and/or levels of services, or if identification of the impact of a specific service component on program outcomes is desired, then explanatory variables representing measures of program inputs should be included in the regression model. The measure may be expressed as an indicator variable, or as a continuous variable representing the number of "units" of a particular service component received (e.g., hours of case management, length of time spent in the program, number of parenting skills seminars attended, etc.).
Return to ToC
1. Adding explanatory variables that produce only small reductions in idiosyncratic variation may, however, result in larger standard errors. Each variable added uses up some of the scarce information in the sample (i.e., reduces the degrees of freedom), and collinearity among explanatory variables can increase standard errors.
2. A specification with interactions can be sufficiently general to be equivalent to separate analyses of subgroups. Hence, this strategy can only ameliorate the small sample problem if the specification is restrictive relative to separate subgroup analyses. For instance, impacts of other explanatory variables on outcomes may be assumed to be the same regardless of the value of the interacted explanatory variable(s).
3. Puerto Rican, Cuban, Mexican, Mexican-American, Chicano, and "Other Spanish" are widely cited Hispanic subgroups.