Job Market Paper

An Exact Hypothesis Test For Samples With Few Effective Clusters


Most recent version


Abstract: I propose a hypothesis test for clustered samples. This test is exact in samples with few clusters, few ever-treated clusters, cluster size outliers, or treatment intensity outliers; these features cause previous tests to over- or under-reject true hypotheses. I derive my test by inverting the distribution of the test statistic under a standard assumption about the errors, so that critical values can be selected from a distribution that matches the test statistic. I use Monte Carlo simulations to demonstrate where this adjustment is most impactful in achieving exact tests compared to previous hypothesis tests, and I apply my test to an empirical setting from the literature.


If you would like to implement my test in R, you can use my package "clubsoda", available on github.


install.packages("devtools")

devtools::install_github("akiva-yonah-meiselman/clubsoda")

?p.value.meis

?conf.interval.meis

Publications

The Impact of Corequisite Math on Community College Student Outcomes: Evidence from Texas (forthcoming, Education Finance and Policy)

With Lauren Schudde (UT-Austin)

Just Accepted Version

Abstract: Developmental education (dev-ed) aims to help students acquire knowledge and skills necessary to succeed in college-level coursework. The traditional prerequisite approach to postsecondary dev-ed—where students take remedial courses that do not count toward a credential—appears to stymie progress toward a degree. At community colleges across the country, most students require remediation in math, creating a barrier to college-level credits under the traditional approach. Corequisite coursework is a structural reform that places students directly into a college-level course in the same term they receive dev-ed support. Using administrative data from Texas community colleges and a regression discontinuity design, we examine whether corequisite math improves student success compared with traditional prerequisite dev-ed. We find that corequisite math quickly improves student completion of math requirements without any obvious drawbacks, but students in corequisite math were not substantially closer to degree completion than their peers in traditional dev-ed after 3 years.

The Importance of Institutional Data Reporting Quality for Understanding Dev-Ed Math Enrollment and Outcomes (Community College Journal of Research and Practice, 2020)

With Lauren Schudde (UT-Austin)

Research Brief

Abstract: Student placement test records, course enrollments, and other student-level data collected by community colleges are vital for evaluating the outcomes of students in developmental education (dev-ed) courses. Researchers and policymakers rely on this information to examine the impact of existing programs and assess ongoing reforms to dev-ed – the accuracy of state administrative data is critical to those tasks. In this study, we examine math placement records in a statewide administrative data set to understand how test records provided by colleges in the state aligned with student course enrollment patterns. We highlight systematic data reporting problems, where many students lacked test scores and test exemption records necessary for policymakers and researchers to determine if they enrolled in the appropriate coursework for their needs. We also found that a non-negligible proportion of students enrolled in dev-ed math – 10% – did not require remediation due to exemption status or passing placement test scores. We conclude with a discussion of the pressing need for accuracy in data reporting, as up-to-date, high-quality student-level data are essential to evaluate ongoing reforms to developmental education.

Early Outcomes of Texas Community College Students Enrolled in Dana Center Mathematics Pathways Prerequisite Developmental Courses (Center for the Analysis of Postsecondary Readiness, 2019)

With Lauren Schudde (UT-Austin)

Research Brief

To improve outcomes in math, many Texas colleges are adopting mathematics pathways, which accelerate developmental math and tailor math courses to different majors instead of requiring all students to take algebra. This study examines whether students participating in Dana Center Mathematics Pathways (DCMP) developmental courses enroll in and pass college-level math courses at higher rates than students who take traditional developmental math courses. It employs regression analysis controlling for student characteristics using student-level data compiled by the state from the more than 20 Texas community colleges that implemented the DCMP model in 2015 and 2016.

Results from this study are encouraging. They suggest that DCMP compressed prerequisite developmental courses are effective at accelerating community college students through their math requirements. Yet this study also found systematic sorting of students into DCMP by race/ethnicity, which could exacerbate educational inequalities.

Work in Progress

Patterns, Determinants, and Consequences of Ability Tracking: Evidence From Texas Public Schools

With Sandra Black (Columbia University), Julie Cullen (UCSD), and Kate Antonovics (UCSD)

Abstract: Schools often track students into classes based on academic ability. Proponents of tracking argue that it is a low-cost tool to improve learning since education is more effective when students are segregated by ability, while opponents argue that tracking exacerbates initial differences in opportunities without strong evidence of efficacy. In fact, very little is known about either the pervasiveness or the determinants of ability tracking in the United States. To fill this gap, we use detailed administrative data from Texas to estimate the extent of ability tracking within schools for grades 4 through 8 for each year from 2011 to 2019. We then explore the nature and evolution of tracking, including how tracking changes in response to educational policies such as school accountability. Finally, we explore how exposure to tracking correlates with student mobility in the achievement distribution.

Disruptive Interactions: Long-run Peer Effects of Disciplinary Schools

With Anjali P. Verma (UT-Austin)

Working Paper

Evidence suggests that exclusionary discipline -- suspensions and temporary placements at disciplinary alternative schools -- pushes marginal students out of the school system and increases the likelihood of incarceration. This paper examines the impact of disruptive peer effects at disciplinary alternative schools on the future removal, educational attainment, and labor market outcomes of students placed at these schools. To study this, we use the linked administrative records of all high-school students in Texas with a disciplinary placement between 2004 and 2018. Noting that a large number of regular schools feed disruptive students into a single disciplinary alternative school, we exploit the over-time variation in peer composition within a disciplinary school to estimate the causal effects of peers' disruptiveness on students’ outcomes. Our results show that having a peer group with higher average disruptiveness at the disciplinary school leads to 1) an increase in students' subsequent disciplinary removals; 2) a decline in their educational attainment (lower high-school graduation, college enrollment and graduation); and 3) a decline in their adult employment and earnings (8% or 1272 USD decline in annual earnings at age 27). These results highlight the need to examine exclusionary discipline policies and adopt approaches that can mitigate the adverse effects of peers at disciplinary schools.

Exclusionary Discipline: Impact of Student Removal to Disciplinary Alternative Programs in Texas

With Anjali P. Verma (UT-Austin)

In this paper, we analyze the impact of disciplinary removals of high school students from their regular instructional environments on rates of graduation, drop-out, and other exits. We use the administrative records of all students entering 9th grade in Texas public schools between 1999 and 2014, including their referrals to the Disciplinary Alternative Education Programs (DAEPs) and Juvenile Justice Alternative Education Programs (JJAEPs) which provided educational programming during periods of disciplinary removal. Noting that physical distance to DAEP campuses is an important determinant of a school’s propensity to refer students to DAEPs, we exploit variation in the timing of the entry and exit of local DAEPs across Texas, employing an event-study design to analyze the impact of DAEPs on graduation, drop-out, and other exits. We utilize the fact that only counties of a certain size were required to establish JJAEPs to examine their impact as well.