Page 1

ASSESSING THE EVIDENCE: FINANCIAL AID Judith Scott-Clayton CCRC

I. Introduction

Of the various tools at policymakers’ disposal for increasing college access and success, financial aid policy is unquestionably the most researched. This brief review cannot serve as an exhaustive summary of this extensive literature. Instead, we aim to highlight influential articles and offer our own informed perspective regarding the key lessons and directions for future research and experimentation. For recent critical reviews of the literature, we recommend “Into College, Out of Poverty? Policies to Increase the Postsecondary Attainment of the Poor,” by Deming and Dynarski (2009), and “What Is Known About the Impact of Financial Aid? Implications for Policy,” by Bridget Terry Long (2008). For a catalog of research especially in areas where the rigorous evidence is limited, we suggest The Effectiveness of Student Aid Policies: What the Research Tells Us, a volume edited by Baum, McPherson, and Steele (2008). These three broader reviews are themselves summarized in the table accompanying this topic brief. Before discussing the evidence, it is worth explaining some of the challenges that the literature has faced in trying to establish the causal effects of financial aid. Why is it not good enough to simply compare students who receive aid with students who do not? The answer is that these students may be different for many reasons besides just their aid status. Even when studies control for observable differences between aid recipients and non-recipients, there may be reason to think that two groups differ along unobservable dimensions that may influence future outcomes. In the case of need-based aid, students who are receiving assistance tend to be more disadvantaged, so even if they benefit from financial aid, a naïve comparison might suggest negative effects. In the case of merit-based aid, students who are receiving assistance are obviously academically selected, so recipients often perform well compared to non-recipients, but this is not necessarily indicative of the scholarship’s impact. In the case of work-study, the bias is unclear: Students who are eligible for work-study are more disadvantaged in general, but those who actually receive and accept a work-study offer are likely to be positively selected from among those who are eligible. The challenge, then, is to identify some “plausibly exogenous” source of variation in who receives assistance. Following the standards of the Institute of Education Sciences (IES) and the What Works Clearinghouse, we consider findings from explicitly randomized field experiments as the highest standard of evidence, but such designs are relatively rare and in some cases simply impossible to carry out. Among quasi-experimental approaches, regression-discontinuity designs—comparing students just above and below program eligibility cutoffs—can be particularly compelling, though difference-in-difference designs (often based upon changes in

15


state, federal, or institutional policy) may provide more externally generalizable estimates. Because of the depth of literature on financial aid, we generally focus here on the studies that are at least quasi-experimental in nature; however, in some cases where there are no such studies we take a broader view of the available evidence.

II. Lessons from the Literature

1. Rigorous evidence suggests that money matters. The first lesson, grounded in over thirty years of research, is that money matters for college outcomes. The cost of college is an important factor in students’ enrollment decisions, and when students know that the cost is lower, their enrollment rates increase. Taken together, the quasi-experimental evidence suggests that an additional $1000 of financial aid (or $1000 less in college costs) may increase college enrollment by 4 percentage points (Deming & Dynarski, 2009, p. 11). Key studies showing positive effects include Dynarski’s (2003) study of the Social Security Student Benefit (SSSB) program, studies of the GI Bills (Stanley 2003; Bound & Turner, 2003), Kane’s (2007) study of the Washington, DC, Tuition Assistance Program, and several studies of state merit aid programs (Dynarski 2004; Cornwell, Mustard, & Sridhar 2006; Kane 2003).8 Until recently much of the focus in the financial aid literature was on college entry, rather than post-enrollment outcomes. Several recent studies suggest that financial aid can also improve persistence and completion (Dynarski 2008; Brock & Richburg-Hayes 2008; Scott-Clayton 2009); however, most of the programs studied involve academic achievement incentives and thus the effect of money alone is unclear. Two studies that do examine post-college effects of pure grants are Dynarski’s (2003) study of the SSSB program and Bettinger’s (2004) study of Pell Grants. Dynarski finds positive, but noisy and statistically insignificant effects on completed years of schooling. Bettinger similarly finds suggestive evidence of positive effects on persistence, but the estimates are not always robust to alternative specifications.

2. Rigorous evidence suggests that simplicity matters. An important and puzzling anomaly to the lesson above that “money matters” is the relatively weak evidence regarding the nation’s single largest grant program, federal Pell Grants. The broadest studies of the Pell Grant program—an early study by Hansen (1983) and a subsequent study by Kane (1996)—find no detectable effect of the introduction of Pell Grants on college enrollments for eligible (low-income) populations. One hypothesis for the lack of overall impacts is that the complexity of the Pell eligibility and application process obscures its benefits                                                             8

Note that state merit-aid programs may improve high school achievement, which may increase college enrollment separate from the effects of the money per se. However, the studies mentioned generally look at college enrollments just after implementation of a program, so early cohorts of recipients may have had relatively little opportunity to change their high school behaviors.

16


and prevents the program from reaching the individuals who need it most—those who are on the fence about college for financial reasons.9 The programs that have demonstrated positive impacts tend to have simple, easy-tounderstand eligibility rules and application procedures. This includes the SSSB program, GI Bills, and state merit aid programs. In contrast, the application process for federal Pell Grants is burdensome, and students do not learn their eligibility until after they are accepted to college. As described in Dynarski and Scott-Clayton (2006), it is difficult for students to respond to a subsidy they do not know about, and even those that know about it might be discouraged by the application. Recent experimental evidence by Bettinger, Long, Oreopoulos, and Sanbonmatsu (2009) suggests that simplifying the Pell Grant application process could have significant effects on college access: providing assistance with completing and submitting the federal aid application increased immediate college entry rates by 7 percentage points for the treated group. The importance of program design and delivery is likely one reason why education tax credits appear to have had little impact (Long 2004). Like Pell Grants, the value of tax credits can be difficult for families to determine in advance of the college enrollment decision. Eligibility depends on family income and tax liability, as well as on year in school and degree intention, and accessing the aid requires filling out a tax form. Importantly, tax relief may not arrive until 16 months after educational expenses are incurred. It is thus perhaps not surprising that take-up rates for the Hope Tax Credit and Lifetime Learning Tax Credit are low. Long (2004) and Dynarski (2004) also find that the benefits of both tax credits and educational savings benefits accrue disproportionately to upper-income families.

3. Rigorous evidence suggests that achievement incentives matter. A third emerging lesson from the literature is that achievement incentives appear to increase effectiveness. This evidence seems especially relevant when the focus is on improving college success (as opposed to simply access). Scott-Clayton (2009) examines a broad-based merit scholarship program in West Virginia that provided free tuition and fees, for up to four years, to eligible students as long as they maintained a minimum GPA and course load in college. She finds that the scholarship increased GPAs and credits completed in the first three years of college, but in the last year of the scholarship—while students are still receiving the money but no longer facing the minimum requirements—the program’s effect disappears. This suggests that the achievement incentives were an important mechanism driving the impacts. Brock and Richburg-Hayes (2006) also find evidence that performance-based scholarships increase GPAs and persistence in community colleges, while Angrist, Lang, and Oreopoulos (2009) find weaker evidence at a large college in Canada.10                                                             9

Note that Seftor and Turner (2002) find positive effects of Pell Grants for older “nontraditional” students and that the Bettinger (2004) study mentioned above finds weak suggestive evidence of positive effects on persistence, conditional on enrollment. Both findings are consistent with a story in which information and experience with bureaucracy is important: Older individuals may have learned about the Pell program over time, and continuing students may learn about the program once they enroll in school. Those who have recently graduated from high school but not yet enrolled may be the least informed and least equipped to figure out the process. 10 They find significant effects of a performance-based scholarship, but only for females who received additional services in addition to the financial incentive. There were no significant effects for the full sample. See accompanying table.

17


Other relevant evidence comes from Pallais (2009) who finds that a large merit-based scholarship program in Tennessee significantly improved high school achievement as measured by test scores (she finds that the increases in test scores are too large to be explained simply by increases in re-testing). Jackson (2010) finds that a program providing financial incentives to high school students for scoring well on AP exams not only improved AP exam scores, but increased college going rates and increased college performance even for those students who would have gone to college anyway.

4. Evidence is limited on the effectiveness of work-study. A fourth lesson is that while low levels of on-campus student employment (such as would be supported by the Federal Work-Study [FWS] program) may not significantly harm student outcomes, the rigorous evidence for positive effects of such assistance is very weak. The nonexperimental evidence on student employment, including non-work-study jobs, suggests negative effects (see Pascarella & Terenzini, 2005, pp. 414-415, and Hossler, Zisken, Kim, Cekic, & Gross, 2009, pp. 103-104, for recent reviews of this literature). But some studies such as Kalenkoski and Pabilonia (2010) and DeSimone (2008) suggest that the magnitude may be quite small, and some non-experimental studies suggest potentially positive effects of low-level, on-campus work. Importantly, the most credible causal examination of college student employment finds significant negative effects of on-campus work on academic outcomes. Stinebrickner and Stinebrickner (2003) analyze data on students at a small private college in Kentucky at which all students are required to work at a campus job for 10 hours per week, but some jobs offer students the possibility to work more (although the study does not separate out work-study employment, it is likely that many of these jobs are at least partially funded by FWS). Students at the college are randomly assigned by administrators to on-campus jobs, and those who are assigned to a job with additional hours available end up working more than those for whom this is not an option. The authors find that students who worked more because they were assigned to a highavailability job earned significantly lower GPAs, a decline of about 0.162 points per additional hour of work. However, because all students were working at least 10 hours per week, the study cannot address the question of whether lower hours of work might be less harmful.

5. Evidence on the effect of loans is limited, but emerging evidence suggests that design is important. A fifth lesson is that while loans are unpopular, they are a critical element in college financing, and their design might be significantly improved to minimize students’ repayment risks. There is very little rigorous research on the consequences of student loans. Dynarski (2005) finds suggestive evidence of positive effects of student loan expansions in the U.S. in the early 1990s on college attendance, but the estimates are not highly robust to specification checks. Findings from the non-experimental literature “can at best be described as mixed” (Heller 2009, p. 46), perhaps because studies are inconsistent in their choice of counterfactual: Are we comparing $1 of loans to $1 of grants, $1 of work-study, or to no aid at all? Based on the nonexperimental evidence, Heller (2009) concludes that college enrollments are not as sensitive to

18


loans as to grants. This is unsurprising given that loans are not worth as much to students, but since they also cost less, it is unclear whether loans may still be cost-effective compared to grants. If loans are less effective than grants, one important reason might be debt aversion: Some students simply dislike being in debt, even when that debt enables an investment with high average returns. A recent experiment by Erica Field (2009) finds strong evidence that students (in this case, law school admits) are debt averse. Admitted students at one school were randomly assigned to receive either (1) a public service scholarship which would convert to a loan if students did not pursue public service after graduation, or (2) a loan which would be forgiven if students decided to pursue public service after graduation. The two treatments were financially equivalent, yet framing the program as a “loan which can be forgiven if you pursue public service” was much less effective in inducing students to public service than a “grant which will convert to a loan if you do not pursue public service.” Like the FAFSA simplification study, Field’s findings provide further evidence that the details of program design and marketing can be critical.

III. Open Questions

1. Should aid amounts vary by class year? Are there creative ways to target limited financial aid dollars to maximize the impact on outcomes that matter beyond enrollment? For example, should freshmen and seniors be paid the same or different amounts? Note that some countries (or some universities in some countries) have recently enacted policies in which students are charged more if they are still enrolled beyond 150% of the typical program length. There is some evidence from Italy that such a program improved time-to-degree at one university (Garibaldi, Giavazzi, Ichino, & Rettore, 2007). Note that Pell eligibility currently extends to 18 full-time equivalent terms, or more than 200% of the scheduled length of the typical BA program.

2. Should aid awards by prorated according to students’ course loads? Should awards be prorated by course load? Note that currently Pell Grant policy is asymmetrically generous in that it allows students enrolled less-than-full-time to still claim a prorated grant, but students enrolling for more than the full-time load are not paid more than the full-time amount. Students are considered “full-time” by the federal aid system if they enroll for 12 credits per semester, but typically will need to complete at least 15 credits per semester in order to graduate in four years. Since many state and institutional programs follow the federal “full-time” designation, there are few additional sources of aid for students who want to attend at

19


a higher intensity and graduate in four years.11 Moreover, the policy may give students the impression that a five-year timeline is typical or even recommended.

3. What are the distributional consequences of achievement incentives? What is the best way to balance achievement incentives with second chances? Is there evidence that some groups of students are more or less responsive to conditional versus unconditional financial assistance? More work could be done to test how financial incentives affect not just overall outcomes, but the distribution of outcomes across different groups.

4. Are some types of student loans more effective than others? Given the widespread reliance on student loans, a more interesting question than whether they work at all is whether they could be made to work better. So, how sensitive are students to the particular design of student loans? Are there ways to make student loans more attractive and less risky for students, without drastically increasing costs? Policy alternatives include incomecontingent loan repayment plans (such as the ones in place in the U.K., Australia, and New Zealand), or “insurance” plans which would help students make their loan payments if they cannot find a job after graduation.12 Alternatively, given that students are unlikely to perceive the difference between subsidized and unsubsidized loans until after they graduate, it would be interesting to test whether students are more responsive to an unsubsidized loan packaged with an upfront grant than to a subsidized loan with the same present discounted value.

IV. Selected Bibliography, by Financial Aid Subtopic

[*] indicates article will be summarized in attached table. Recent reviews of the literature *Baum, S, McPherson, M., & Steele, P. (Eds.). (2008). The effectiveness of student aid policies: What the research tells us. New York, NY: The College Board.

                                                            11

West Virginia’s state merit scholarship, PROMISE, is one notable exception. Students may claim this scholarship in the freshman year even if they enroll at the minimum full-time level, but need to complete 30 credits per year in order to renew for the following year. 12 Note that recent changes in federal loan policy give students additional protections if they experience low income after graduation, but it is unclear whether these changes are transparent enough for students to understand in advance of the decision to take a loan or not.

20


*Deming, D., & Dynarski, S. (2009). Into college, out of poverty? Policies to increase the postsecondary attainment of the poor (NBER Working Paper No. 15387). Cambridge, MA: National Bureau of Economic Research. *Long, B. T. (2008). What is known about the impact of financial aid? Implications for policy (NCPR Working Paper). New York, NY: National Center for Postsecondary Research. Pell Grants *Bettinger, E. (2004). How financial aid affects persistence. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 207-238). Chicago, IL: University of Chicago Press. Hansen, W. L. (1983). Impact of student financial aid on access. In J. Froomkin (Ed.), The crisis in higher education (pp. 84-96). New York, NY: The Academy of Political Science. *Kane, T. J. (1995). Rising public college tuition and college entry: How well do public subsidies promote access to college (NBER Working Paper No. 5164)? Cambridge, MA: National Bureau of Economic Research. Kane, T. J. (1996). Lessons from the largest school voucher program ever: Two decades of experience with Pell grants. In B. Fuller, R. Elmore, & G. Orfield (Eds.), Who chooses? Who loses? Culture, institutions and the unequal effects of school choice. New York, NY: Teachers College Press. *Mundel, D. (2008). Do increases in Pell and other grant awards increase college-going among lower income high school graduates? Evidence from a ‘natural experiment.’ Unpublished paper, Brookings Institution, Washington, DC. Rice, L. D., & Mundel, D. (2008). The impact of increases in Pell grant awards on college-going among lower-income youth (CCF Brief No. 40). Washington, DC: Brookings Center on Children and Families. Retrieved at http://www.brookings.edu/papers/2008/12_pell_grants_rice.aspx *Seftor, N., & Turner, S. (2002). Back to school: Federal student aid policy and adult college enrollment. Journal of Human Resources, 37(2), 336-352. Stedman, J. B. (2003). Federal Pell grant program of the Higher Education Act: Background and reauthorization. Washington, DC: U.S. Congressional Research Service. Tuition reductions and other non-merit based assistance Abraham, K., & Clark, M. (2006). Financial aid and students’ college decisions: Evidence from the District of Columbia Tuition Assistance Grant program. Journal of Human Resources, 41(3), 578-610.

21


Angrist, J. D. (1993). The effect of veterans benefits on education and earnings. Industrial and Labor Relations Review, 46(4), 637-652. *Avery, C., Hoxby, C., Jackson, C., Burek, K., Poppe, G., & Raman, M. (2006). Cost should be no barrier: An evaluation of the first year of Harvard’s Financial Aid Initiative (NBER Working Paper No. 12029). Cambridge, MA: National Bureau of Economic Research. Bound, J., & Turner, S. (2002). Going to war and going to college: Did World War II and the G.I. Bill increase educational attainment for returning veterans? Journal of Labor Economics, 20(4), 784-815. Dynarski, S. (2002). The behavioral and distributional implications of aid for college. American Economic Review, 92(2), 279-285. *Dynarski, S. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279-288. Garibaldi, P., Giavazzi, F., Ichino, A., & Rettore, E. (2007). College cost and time to complete a degree: Evidence from tuition discontinuities (NBER Working Paper No. 12863). Cambridge, MA: National Bureau of Economic Research. Hansen, W. L. (1983). Impact of student financial aid on access. In J. Froomkin (Ed.), The crisis in higher education (pp. 84-96). New York, NY: The Academy of Political Science. Heller, D. E. (1997). Student price response in higher education: An update of Leslie and Brinkman. Journal of Higher Education, 68(6), 624-659. *Kane, T. J. (2007). Evaluating the impact of the DC Tuition Assistance Grant program. Journal of Human Resources, 42(3), 555-582. *Stanley, M. (2003). College education and the mid-century G.I. bills. Quarterly Journal of Economics, 118(2), 671-708. van der Klaauw, W. (2002, November).Estimating the effect of financial aid offers on college enrollment: A regression-discontinuity approach. International Economic Review, 43(4), 1249-1287. Merit-based grants and financial incentives *Angrist, J. D., Lang, D., & Oreopoulos, P. (2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1(1), 136-163. *Brock, T., & Richburg-Hayes, L. (2006). Paying for persistence: Early results of a Louisiana scholarship program for low-income parents attending community college. New York, NY: MDRC.

Â

22


Cornwell, C., Mustard, D., & Sridhar, D. (2006). The enrollment effects of merit-based financial aid: Evidence from Georgia’s HOPE scholarship. Journal of Labor Economics, 24, 761786. DesJardins, S. L., & McCall, B. P. (2007). The impact of the Gates Millennium Scholars program on selected outcomes of low-income minority students: A regression discontinuity analysis. Unpublished manuscript, University of Michigan. Retrieved from http://www-personal.umich.edu/~bpmccall/Desjardins_McCall_GMS_June_2008.pdf Dynarski, S. (2000, September). Hope for whom? Financial aid for the middle class and its impact on college attendance. National Tax Journal, 53(3), 629-661. Dynarski, S. (2004a). The new merit aid. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 63-100). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research. *Dynarski, S. (2008). Building the stock of college-educated labor. Journal of Human Resources, 43(3), 576-610. Goodman, J. (2008). Who merits financial aid?: Massachusetts’ Adams Scholarship. Journal of Public Economics, 92(10-11), 2121-2131. Heller, D., & Marin, P. (Eds.). (2003). Who should we help? The negative social consequences of merit aid scholarships. Cambridge, MA: Harvard Civil Rights Project. *Jackson, C. K. (2010). A stitch in time: The effects of a novel incentive-based high-school intervention on college outcomes (NBER Working Paper No. 15722). Cambridge, MA: National Bureau of Economic Research. Kane, T. J. (2003). A quasi-experimental estimate of the impact of financial aid on college-going (NBER Working Paper No. 9703). Cambridge, MA: National Bureau of Economic Research. *Pallais, A. (2009). Taking a chance on college: Is the Tennessee Education Lottery Scholarship a winner? Journal of Human Resources, 44(1), 199-222. *Scott-Clayton, J. (2009). On money and motivation: A quasi-experimental analysis of financial incentives for college achievement. Unpublished manuscript, Teachers College, Columbia University. Retrieved from http://faculty.tc.columbia.edu/upload/js3676/JSC_WVCollIncentives_FullDraft_Oct2009 .pdf Loans Baum, S. (2003b). The role of student loans in college access (National Dialogue on Student Financial Aid Research Report No. 5). New York, NY: College Board.

23


Burdman, P. (2005). The student debt dilemma: Debt aversion as a barrier to college access. Berkeley, CA: University of California, Center for Studies in Higher Education. Campaigne, D.A., & Hossler, D. (1998). How do loans affect the educational decisions of students? In R. Fossey & M. Bateman (Eds.), Condemning students to debt (pp. 85-104). New York, NY: Teachers College Press. Chapman, C. (1994). Income-contingent college loans: Correspondence. Journal of Economic Perspectives, 8(4), 205-206. Cofer, J., & Somers, P. (2000). A comparison of the influence of debtload on the persistence of students at public and private colleges. Journal of Student Financial Aid, 30(2), 39-58. Callender, C., & Jackson, J. (2005). Does the fear of debt deter students from higher education? Journal of Social Policy, 34(2), 39-58. *Dynarski, S. (2005). Loans, liquidity and schooling decisions. Unpublished manuscript, Harvard University, Cambridge, MA. *Field, E. (2009). Educational debt burden and career choice: Evidence from a financial aid experiment at NYU law school. American Economic Journal: Applied Economics, 1(1), 1-21. Krueger, A., & Bowen, W. G. (1993). Policy watch: Income-contingent college loans. Journal of Economic Perspectives, 7(3), 193-201. Perna, L. (2007). Understanding high school students' willingness to borrow to pay college prices. Research in Higher Education, 49(7), 589-606. Reyes, S. L. (1995). Educational opportunities and outcomes: The role of the guaranteed student loan. Unpublished manuscript, Harvard University, Cambridge, MA. Rothstein, J., & Rouse, C. E. (2007). Constrained after college: Student loans and early career occupational choices (NBER Working Paper No. 13117). Cambridge, MA: National Bureau of Economic Research. Tax benefits Cronin, J. (1997). The economic effects and beneficiaries of the administration’s proposed higher education tax subsidies. National Tax Journal, 50(3), 519-540. *Dynarski, S. (2004b). Who benefits from the education saving incentives? Income, educational expectations and the value of the 529 and Coverdell. National Tax Journal, 57(2), 359383. *Hoxby, C. M. (1998). Tax incentives for higher education. In J. Poterba (Ed.), Tax policy and the economy. Cambridge, MA: National Bureau of Economic Research.

Â

24


Kane, T. J. (1997). Beyond tax relief: Long-term challenges in financing higher education. National Tax Journal, 50(2), 335-349. *Long, B. T. (2004b). The impact of federal tax credits for higher education expenses. In C. Hoxby (Ed.), College choices: The economics of which college, when college, and how to pay for it (pp. 101-168). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research. On-campus student employment DeSimone, J. (2008). The impact of employment during school on college student academic performance (NBER Working Paper No. 14006). Cambridge, MA: National Bureau of Economic Research. DesJardins, S. L., Ahlburg, D. A., & McCall, B. P. (2002). Simulating the longitudinal effects of changes in financial aid on student departure from college. Journal of Human Resources, 37(3), 653-679. Kalenkoski, C., & Pabilonia, S. (2010). Parental transfers, student achievement, and the labor supply of college students. Journal of Population Economics, 23(2), 469-496. Pascarella, E., & Terenzini, P. (2005). How college affects students, volume 2: A third decade of research. San Francisco, CA: Jossey-Bass. *Stinebrickner, T., & Stinebrickner, R. (2003). Working during school and academic performance. Journal of Labor Economics, 21(2), 473-491. Complexity *Bettinger, E., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2009). The role of information and simplification in college decisions: Results from the H&R Block FAFSA experiment (NBER Working Paper No. 15361). Cambridge, MA: National Bureau of Economic Research. *Dynarski, S., & Scott-Clayton, J. (2006b, June). The cost of complexity in federal student aid: Lessons from optimal tax theory and behavioral economics. National Tax Journal, 59(2), 319-356.

Â

25


Topic: Effectiveness of Financial Aid  Study Design 

Sample size, Characteristics,  and Power 

External Validity 

Baum, S, McPherson, M., &  Steele, P. (Eds.). (2008). The  effectiveness of student aid  policies: What the research tells  us. New York, NY: The College  Board. 

n/a

n/a

n/a

Simplicity and transparency in aid  systems is critical; early  awareness/understanding of aid may  improve college preparation; aid does  too little to encourage completion  rather than just access; too little  support for adult students; inherent  tension between simplicity and  tailoring programs to diverse student  needs. 

Suggestive

Deming, D., & Dynarski, S.  (2009). Into college, out of  poverty? Policies to increase the  postsecondary attainment of  the poor (NBER Working Paper  No. 15387). Cambridge, MA:  National Bureau of Economic  Research. 

n/a

n/a

n/a

Reducing college costs can increase  both college entry and persistence;  simplicity and transparency are  critical program design features;  money linked to incentives and/or  services appears to strengthen  effectiveness of financial aid. 

Strong

Long, B. T. (2008). What is  known about the impact of  financial aid? Implications for  policy (NCPR Working Paper).  New York, NY: National Center  for Postsecondary Research. 

n/a

n/a

n/a

Information and simplicity are  important; grants are more effective  than loans or tax credits; need‐based  aid is most effective for low‐income  students; financial aid is not a  panacea.  Future research should  examine importance of policy design  features, whether certain types of aid  work better/worse for certain  subpopulations (e.g. older students,  Hispanic students),  and how  institutions respond to aid.       

Strong

Citation

Summary of Findings 

Assessment of Evidence 

Notes/Caveats

Subtopic: Recent reviews 

26

Chapters take a  broader/deeper survey of  the literature than the two  recent reviews by  economists, but not all  studies are critically  reviewed, and some  important studies are left  out (e.g. work by  Stinebrickner and  Stinebrickner [2003] on  student employment).  Short piece; explains why  correlational evidence is  insufficient to establish  causal effects of aid on  outcomes; briefly reviews  the quasi‐experimental and  experimental literature;  includes summary table of  experimental and quasi‐ experimental studies.  Broader than  Deming/Dynarski but  shorter than Baum et al.;  five sections covering  justifications for aid, role of  information/complexity,  effectiveness of grants,  effectiveness of other aid  (loans, tax credits, savings  incentives), and  institutional responses to  aid. 


Topic: Effectiveness of Financial Aid  Citation 

Sample size, Characteristics,  and Power 

External Validity 

Randomized experiment 

Initially random sample of  H&R Block tax clients  meeting demographic  requirements yielded 800  dependent and 14,000  independent individuals 

Relatively strong (see  sample  description) 

Strong positive effects of full  treatment on financial aid application  for all groups. Increased college  enrollments by 7 percentage points  for dependent participants. Increased  Pell receipt by all groups. 

Strong

An interesting finding is  that the treatment  involving information‐only,  but no assistance in  completing the FAFSA,  showed no impacts. 

Simulations comparing actual  aid to simulated  amounts  calculated under  alternative  formulae 

Nationally representative  sample of financial aid  applicants from National  Postsecondary Student Aid  Survey (NPSAS 2004) 

Strong

Parents’ adjusted gross income,  marital status, family size, and  number of family members in college  explain over 75% of variation in Pell  Grant awards, yet the FAFSA collects  over 70 financial items most of which  contribute little to targeting. FAFSA is  longer and more complicated than an  IRS 1040. 

Strong

This paper focuses on  dependent, full‐time  students; later work finds  similar results for  independent students. 

Quasi‐ experimental (difference‐in‐ difference  comparing trends  over time in states  with high and low  home values) 

18‐19 year olds in CPS 1984‐ 2000; secondary analyses  use SIPP data 

Strong

Mixed/weak findings. CPS suggests a  $1000 increase in loan eligibility  increases attendance by 1.7pp, with  shift towards four‐year private  institutions.  SIPP provides much  better data on assets, and finds only  weak supporting evidence, with very  imprecise estimates, and some  implications could not be confirmed  in these data (regarding effects by  income subgroup; using IV for home  equity; linking timing to 1992). 

Inconclusive

CPS analysis is based on  state‐year median home  values, not individual  measures. Endogeneity of  home equity is possible  bias in individual data. 

Study Design 

Summary of Findings 

Assessment of Evidence 

Notes/Caveats

Subtopic: Complexity  Bettinger, E., Long, B. T.,  Oreopoulos, P., &  Sanbonmatsu, L. (2009). The  role of information and  simplification in college  decisions: Results from the H&R  Block FAFSA experiment (NBER  Working Paper No. 15361).  Cambridge, MA: National  Bureau of Economic Research.  Dynarski, S., & Scott‐Clayton, J.  (2006b, June). The cost of  complexity in federal student  aid: Lessons from optimal tax  theory and behavioral  economics. National Tax  Journal, 59(2), 319‐356. 

Subtopic: Loans  Dynarski, S. (2005). Loans,  liquidity and schooling  decisions. Unpublished  manuscript, Harvard University,  Cambridge, MA. 

27


Topic: Effectiveness of Financial Aid  Citation 

Study Design 

Field, E. (2009). Educational  debt burden and career choice:  Evidence from a financial aid  experiment at NYU law school.  American Economic Journal:  Applied Economics, 1(1), 1‐21. 

Randomized experiment 

Sample size, Characteristics,  and Power  NYU law school students  who enlisted in the study,  from the classes of 1998,  1999, 2000, and 2001 

External Validity  Somewhat  limited (see  sample  description) 

~1600 college freshmen at  large public university in  Canada; uses administrative  data and student surveys 

Relatively strong  (based on  large  sample, but  from single  university in  Canadian  context) 

Summary of Findings  Treatment groups are 14.1  percentage points more likely to take  a public‐sector job and 12.2  percentage points more likely to take  a clerkship after leaving law school.  A  $10,000 increase in school debt  reduces the likelihood of taking a  public interest job two years after law  school by approximately 6%.  

Assessment of Evidence  Strong 

Notes/Caveats This study is frequently  cited as providing evidence  that students are "debt  averse." The two  treatments were financially  equivalent, yet framing the  program as a "loan which  can be forgiven if you  pursue public service" was  much less effective in  inducing students to public  service than a "grant which  will convert to a loan if you  do not pursue public  service." 

Subtopic: Merit‐based grants, financial incentives  Angrist, J. D., Lang, D., &  Oreopoulos, P. (2009).  Incentives and services for  college achievement: Evidence  from a randomized trial.  American Economic Journal:  Applied Economics, 1(1), 136‐ 163. 

Randomized experiment 

28

No impact of financial incentives  alone for any group, but some  positive effects for women who  received the combined incentive‐ plus‐services treatment. 

Strong

Very strong study finding  relatively little effect of  large incentives. Because  students did not receive  any benefit from the  program "up front," this  may have limited their  engagement with the  program's incentives. Also,  incentives were  individually‐established  and included multiple  levels, perhaps making the  program confusing? 


Topic: Effectiveness of Financial Aid  Sample size, Characteristics,  and Power  ~500 low‐income single  parents (mothers) at two  community colleges in  Louisiana; uses  administrative data and  student surveys 

External Validity  Somewhat  limited (see  sample  description) 

Quasi‐ experimental (difference‐in‐ difference  comparing trends  in selected and  comparison states  over time) 

Representative sample of  18‐19 year olds in GA, AK,  FL, KY, LA, MS, SC using  Current Population Survey  data 

Strong

Increases the proportion of 18‐19  year olds enrolled by about 5  percentage points. 

Strong

Quasi‐ experimental (difference‐in‐ difference  comparing trends  in selected and  comparison states  over time) 

Cross‐sectional data  covering different age  groups in Georgia and  Arkansas, using Census 2000  data 

Strong

Increases the total share of an age  cohort with some college by about  1.6pp; share that has a BA by 3  percentage points (from base on 27  percent). Dropout conditional on  entry estimated to decrease about  4.3 percentage points. 

Strong

Citation

Study Design 

Brock, T., & Richburg‐Hayes, L.  (2006). Paying for persistence:  Early results of a Louisiana  scholarship program for low‐ income parents attending  community college. New York,  NY: MDRC. 

Randomized experiment 

Dynarski, S. (2004a). The new  merit aid. In C. Hoxby (Ed.),  College choices: The economics  of where to go, when to go, and  how to pay for it (pp. 63‐100).  Chicago, IL: University of  Chicago Press and the National  Bureau of Economic Research.   Dynarski, S. (2008). Building the  stock of college‐educated labor.  Journal of Human Resources,  43(3), 576‐610.  

29

Summary of Findings  Significant, large impacts on  persistence into second and third  semesters (18pp and 11pp  respectively), with all of impact on  full‐time enrollment (20pp and 11pp  respectively). Increase in GPA in  second sem (0.4/2.1) but not  significant in first sem. Total impact  on credits earned after 3 sems:  3.3/7.7 baseline. 

Assessment of Evidence  Strong 

Notes/Caveats Significant increases in full‐ time enrollment are  surprising given the  incentive was linked only  to half‐time enrollment.   Report casts impacts as  somewhat small, but could  be a story of very large  effects for a subset of  recipients.  In percentage  terms effects are quite  substantial.  This paper is an extension  of Dynarski (2000) "Hope  for Whom?" on GA Hope;  here she broadens to  seven states with similar  programs. Examines  enrollment flows, not  "stock" of college  graduates.  One of the only studies  looking at the effect of aid  on college completion, not  just enrollment. 


Topic: Effectiveness of Financial Aid  Citation 

Study Design 

Jackson, C. K. (2010). A stitch in  time: The effects of a novel  incentive‐based high‐school  intervention on college  outcomes (NBER Working Paper  No. 15722). Cambridge, MA:  National Bureau of Economic  Research. 

Quasi‐ experimental (difference‐in‐ difference  comparing  before/after  changes at  schools that had  the program to  changes at  schools that did  not)  Quasi‐ experimental  (difference‐in‐ difference  comparing trends  in selected and  comparison states  over time) 

Pallais, A. (2009). Taking a  chance on college: Is the  Tennessee Education Lottery  Scholarship a winner? Journal  of Human Resources, 44(1),  199‐222. 

Scott‐Clayton, J. (2009). On  money and motivation: A quasi‐ experimental analysis of  financial incentives for college  achievement. Unpublished  manuscript, Teachers College,  Columbia University. Retrieved  from  http://faculty.tc.columbia.edu/ upload/js3676/JSC_WVCollInce ntives_FullDraft_Oct2009.pdf. 

Quasi‐ experimental (regression  discontinuity  based upon ACT  test score cutoff  for eligibility;  before/after  comparison) 

Sample size, Characteristics,  and Power  Population of 10th graders  attending Texas high schools  from 1999‐2006; uses Texas  administrative data 

External Validity  Strong 

Random sample of  Tennessee ACT test‐takers  using ACT microdata 

Strong

6‐8 percentage point (11‐14%)  increase in test‐takers scoring 19 or  above; cannot be explained by  retesting. No significant effects on  score‐sending or stated college  preferences (if anything, tended to  increase favor for out‐of‐state  schools). 

Strong

Universe of West Virginia  public college first‐time  enrollees using WV  administrative data 

Relatively strong  (West  Virginia is  significantly  more  disadvantag ed and less  racially  diverse  than nation  as a whole)             

6.7 percentage point (25%) increase  in on‐time completion; 3.5  percentage point (7%) increase in  completion after 5 years; 6 additional  credits after four years; no effects on  GPA after 4 years; ‐$10/wk decline in  student employment; effects  disappear in 4th year when students  still receive scholarship but no longer  face academic incentives. 

Strong

30

Summary of Findings  The program increased college  enrollment, as well as GPAs and  persistence conditional on  enrollment. Program increased  college completions for minorities,  but not white students. 

Assessment of Evidence  Strong 

Notes/Caveats As the author mentioned,  there is some concern that  the enrollment effects may  partly reflect an increased  likelihood of enrolling in  Texas rather than some  other state. 

Other work has found  merit aid increases high  school GPAs, but ACT  scores are more  standardized, may be more  objective, and may be  more difficult to  manipulate via changes in  coursetaking, etc.  Provides strong evidence  that the academic  incentives embodied in  merit aid scholarships are  key to their impact; a grant  with no strings attached  would not have had the  same effect. 


Topic: Effectiveness of Financial Aid  Citation 

Study Design 

Sample size, Characteristics,  and Power 

External Validity 

Summary of Findings 

Assessment of Evidence 

Notes/Caveats

Subtopic: Pell Grants  Bettinger, E. (2004). How  financial aid affects persistence.  In C. Hoxby (Ed.), College  choices: The economics of  where to go, when to go, and  how to pay for it (pp. 207‐238).  Chicago, IL: University of  Chicago Press. 

Matched/ controlled  (looking at same  individuals over  time, whose  financial aid  varied); quasi‐ experimental  (regression  discontinuity  based on  discontinuous  changes in Pell  Grant amounts by  family  size/number in  college) 

Incoming freshman class in  1999‐2000 school year in  Ohio 2 and 4‐year public  schools; uses Ohio  administrative data 

Strong

Results suggest that larger Pell Grants  reduce the likelihood of dropout;  however the results are not robust to  alternative specifications. 

Suggestive

Kane, T. J. (1995). Rising public  college tuition and college  entry: How well do public  subsidies promote access to  college (NBER Working Paper  No. 5164)? Cambridge, MA:  National Bureau of Economic  Research. 

Quasi‐ experimental (difference‐in‐ difference  comparing states  with  bigger/smaller  tuition increases  over time;  comparing eligible  and ineligible  populations  before/after  changes in Pell  funding) 

18‐19 year olds from 1977 to  1993 using October CPS;  other analyses using  National Longitudinal Survey  of Youth (NLSY) 1979 and  High School and Beyond  (HSB) 1980 

Relatively strong (data  may be  getting  somewhat  outdated) 

College enrollments appear highly  responsive to college tuition charges,  but less so to need‐based aid.  After  Pell Grants were introduced, there  was no disproportionate increase in  enrollments by low‐income youth. 

Suggestive

31

One of the few studies to  examine persistence rather  than just initial enrollment.  The evidence is suggestive,  but the estimates are not  particularly robust.  To the  extent they can be taken at  face value, these results  could indicate that  problems of complexity  and poor information  (which have been cited as  explanations for the non‐ impact of Pell Grants on  initial entry) may be  ameliorated for students  after they enroll and learn  their aid amount.  The results at the time  were surprising: that  students are sensitive to  college costs but not to  need‐based aid? Dynarski  and Scott‐Clayton (2006)  suggest that complexity in  the Pell eligibility and  application rules may  diminish its impact. 


Topic: Effectiveness of Financial Aid  Sample size, Characteristics,  and Power  High school graduates from  middle, moderate and low‐ income families (1995‐ 2004); uses three waves of  the National Postsecondary  Student Aid Survey (NPSAS)  and CPS data 

External Validity  Strong  (though  internal  validity is  somewhat  questionabl e) 

Quasi‐ experimental (difference‐in‐ difference  comparing eligible  and ineligible  populations  before and after  increases in Pell  funding) 

Independent high school  graduates ages 22‐35; uses  October CPS for sub‐periods  between 1969 and 1990 

Relatively strong (data  may be  getting  somewhat  outdated) 

The difference‐in‐differences  estimates of the effect of Pell  eligibility on the probability of  attending college is 1.5 percentage  points for men and 1.3 percentage  points for women.  When compared  to the enrollment rates before the  introduction of the program, this  reflects a relative growth of 16  percent for men and 40 percent for  women.  

Strong

No comparison 

Households who invest in  these savings accounts;  “typical “ households with  school‐going children; using  2001 Survey of Consumer  Finances (SCF) and 2000  NPSAS 

Strong

Dynarski finds that the advantages of  the 529 and Coverdell rise sharply  with income for three reasons. Those  in the top tax brackets benefit more  from non‐educational use of a  Coverdell than those in the bottom  bracket gain from its educational use.  

Strong

Citation

Study Design 

Mundel, D. (2008). Do increases  in Pell and other grant awards  increase college‐going among  lower income high school  graduates? Evidence from a  ‘natural experiment.’  Unpublished paper, Brookings  Institution, Washington, DC. 

Quasi‐ experimental (before‐after  comparison) 

Seftor, N., & Turner, S. (2002).  Back to school: Federal student  aid policy and adult college  enrollment. Journal of Human  Resources, 37(2), 336‐352.  

Summary of Findings  During the 1999‐2004 years, the  adjusted immediate college going  rate for low income youth increased  by roughly 6‐7 percentage points,  while the adjusted rate for moderate  income youth remained  constant,  declining by roughly 0‐1 percentage  point.  The immediate college going  rate for middle income youth  increased by 4 percentage points. 

Assessment of Evidence  Inconclusive 

Notes/Caveats Paper is a policy brief  which does not provide the  detailed information  needed to evaluate the  methodology. Though  overall trends are  consistent with the  findings, other trends  could explain the findings  as well, such as changes in  state and institutional  assistance (rather than  changes in the Pell grant).  (1) The index on which the  eligibility to be awarded a  Pell grant was calculated  using the midpoint of the  category as the income and  basis for taxes.  (2) The  authors do not present a  comparison of the means  to see if treatment and  control groups were  comparable. 

Subtopic: Tax benefits  Dynarski, S. (2004b). Who  benefits from the education  saving incentives? Income,  educational expectations and  the value of the 529 and  Coverdell. National Tax Journal,  57(2), 359‐383.  

32

This is not an impact  evaluation, but an  examination of how  benefits are distributed  across the population.  However, the suggestion is  that the program is unlikely  to have much impact on  college outcomes if it is  primarily utilized by high‐ income families. 


Topic: Effectiveness of Financial Aid  Citation 

Study Design 

Long, B. T. (2004b). The impact  of federal tax credits for higher  education expenses. In C. Hoxby  (Ed.), College choices: The  economics of which college,  when college, and how to pay  for it (pp. 101‐168). Chicago, IL:  University of Chicago Press and  the National Bureau of  Economic Research.  

Quasi‐ experimental (difference‐in‐ difference  between eligible  and ineligible  individuals before  and after the  introduction of  the credits) 

Sample size, Characteristics,  and Power  College‐going individuals in  IPEDS (1993‐1994 to 1990‐ 2000) and October CPS data  (1990‐2000) 

External Validity  Strong 

Summary of Findings  Tax credits did not increase  postsecondary enrollment among  credit‐eligible students, nor were  students more likely to invest in a 4‐ year (vs. 2‐year) program. The low  take‐up rate suggests that not enough  families know about the benefit for it  to have a discernible impact.  Disconnect between the timing of the  benefit and college enrollment may  limit the effects.     

Assessment of Evidence  Suggestive 

Notes/Caveats The data are somewhat  limited for the analysis  (e.g. income data is  recorded only in brackets,  not exact amounts).   Measurement error may  attenuate results. Not clear  that trends for eligible and  ineligible students are  comparable. 

Subtopic: Tuition reductions and other non‐merit based aid  Avery, C., Hoxby, C., Jackson, C.,  Burek, K., Poppe, G., & Raman,  M. (2006). Cost should be no  barrier: An evaluation of the  first year of Harvard’s Financial  Aid Initiative (NBER Working  Paper No. 12029). Cambridge,  MA: National Bureau of  Economic Research.  Dynarski, S. (2003). Does aid  matter? Measuring the effect of  student aid on college  attendance and completion.  American Economic Review,  93(1), 279‐288. 

Quasi‐ experimental (before‐after  comparison) 

Data from 16,821 Harvard  applications in 2008 and  from 19,321 applications in  2009 

Somewhat limited (see  sample  description) 

The percentage of applicants from  families with incomes of $40,000 or  below jumped by more than 20%.  Enrollment of students qualifying for  the Initiative increased by 11% in one  year; while enrollment of students  with family incomes below $40,000  increased by nearly 20%.  

Suggestive

Quasi‐ experimental (difference‐in‐ difference  between children  with/without  deceased father,  before/after the  program's  elimination in  1982) 

12,686 youth from the NLSY  1979; "treated" group  include only those whose  father was deceased at time  of college decision 

Somewhat limited (see  sample  description;  data may  be  becoming  outdated) 

Dynarski finds that “the elimination of  the Social Security student benefit  program reduced college attendance  probabilities by more than a third.  These estimates suggest that an offer  of $1,000 in grant aid increases the  probability of attending college by  about 3.6 percentage points. Aid  eligibility also appears to increase  completed schooling.” 

Strong

33

The before‐after  comparison is somewhat  limited in that it relies on  only two years of data, and  some data are not  available for all applicants  (e.g., complete financial  data is only available for  students who enrolled).  A seminal study  documenting the effect of  student assistance on  college enrollments. 


Topic: Effectiveness of Financial Aid  Sample size, Characteristics,  and Power  Data on first‐time freshmen  (DC residents) starting in  1998 (1572 students) until  2002 using Integrated  Postsecondary Education  Data System (IPEDS)  aggregate statistics; financial  aid application data; TAG  administrative data 

External Validity  Strong 

Quasi‐ experimental (Korean War GI  bill analysis);  matched/controll ed (WWII GI bill  analysis) 

Sample of 532 individuals  from the Survey of  Occupational Change in a  Generation (OCG), a  supplement to the CPS in  1962 and 1973; sample of  240 veterans from the 1978  Survey of Veterans data 

Somewhat limited (see  sample  description) 

The combination of the Korean War  and WWII GI bills probably increased  total postsecondary attainment  among all men born between 1921  and 1933 by about 15 to 20 percent,  with smaller effects for surrounding  cohorts. The impacts were apparently  concentrated among veterans from  families in the upper half of the  socioeconomic distribution. 

Strong

Quasi‐ experimental (instrumental  variables analysis;  some students  worked more  because they  were randomly  assigned to jobs  with more hours  available) 

Sample of 2,372 first year  students at Berea College,  small private college in  Kentucky where all students  receive free tuition, room &  board; have to work at least  10 hrs per week 

Limited (see  sample  description) 

An additional hour of student  employment (above 10 hours per  week) reduces first‐year GPAs by  0.162 points on a four‐point scale. 

Strong

Citation

Study Design 

Kane, T. J. (2007). Evaluating  the impact of the DC Tuition  Assistance Grant program.  Journal of Human Resources,  42(3), 555‐582. 

Quasi‐ experimental (before‐after, also  relies upon  staggered timing  of effective tuition  changes in  Maryland,  Virginia, and other  states) 

Stanley, M. (2003). College  education and the mid‐century  G.I. bills. Quarterly Journal of  Economics, 118(2), 671‐708. 

Summary of Findings  Between 1998 and 2000 the number  of D.C. residents attending public  institutions in Virginia and Maryland  more than doubled. The impact was  largest at nonselective public four‐ year colleges, particularly  predominantly black institutions. The  total number of financial aid  applicants, Pell Grant recipients and  college entrants from D.C. also  increased by 15 percent or more. 

Assessment of Evidence  Strong 

Notes/Caveats It may not be appropriate  to attribute all of the  increase to the D.C. TAG  program. The D.C. College  Access Program (D.C.  CAP)—was a smaller, but  still significant private  program that also began  operations in six public  high schools in D.C. for  those graduating in the  spring of 2000.  It may  account for some share of  the increase in enrollment  observed over that time  period.    

Subtopic: Work‐study  Stinebrickner, T., &  Stinebrickner, R. (2003).  Working during school and  academic performance. Journal  of Labor Economics, 21(2), 473‐ 491. 

34

This is virtually the only  credible study of student  employment at the college  level; unfortunately  external validity may be  limited.  

Assessing the evidence: financial aid  

Of the various tools at policymakers’ disposal for increasing college access and success, financial aid policy is unquestionably the most re...

Read more
Read more
Similar to
Popular now
Just for you