The 2016 California policy to eliminate nonmedical vaccine exemptions and changes in vaccine coverage: An empirical policy analysis

Summary

Background

Vaccine hesitancy, the reluctance or refusal to receive vaccination, is a growing public health problem in the United States and globally. State policies that eliminate nonmedical (“personal belief”) exemptions to childhood vaccination requirements are controversial, and their effectiveness to improve vaccination coverage remains unclear given limited rigorous policy analysis. In 2016, a California policy (Senate Bill 277) eliminated nonmedical exemptions from school entry requirements. The objective of this study was to estimate the association between California’s 2016 policy and changes in vaccine coverage.

Methods and findings

We used a quasi-experimental state-level synthetic control analysis and a county-level difference-in-differences analysis to estimate the impact of the 2016 California policy on vaccination coverage and prevalence of exemptions to vaccine requirements (nonmedical and medical). We used publicly available state-level data from the US Centers for Disease Control and Prevention on coverage of measles, mumps, and rubella (MMR) vaccination, nonmedical exemption, and medical exemption in children entering kindergarten. We used county-level data individually requested from state departments of public health on overall vaccine coverage and exemptions. Based on data availability, we included state-level data for 45 states, including California, from 2011 to 2017 and county-level data for 17 states from 2010 to 2017. The prespecified primary study outcome was MMR vaccination in the state analysis and overall vaccine coverage in the county analysis.

In the state-level synthetic control analysis, MMR coverage in California increased by 3.3% relative to its synthetic control in the postpolicy period (top 2 of 43 states evaluated in the placebo tests, top 5%), nonmedical exemptions decreased by 2.4% (top 2 of 43 states evaluated in the placebo tests, top 5%), and medical exemptions increased by 0.4% (top 1 of 44 states evaluated in the placebo tests, top 2%). In the county-level analysis, overall vaccination coverage increased by 4.3% (95% confidence interval [CI] 2.9%–5.8%, p < 0.001), nonmedical exemptions decreased by 3.9% (95% CI 2.4%–5.4%, p < 0.001), and medical exemptions increased by 2.4% (95% CI 2.0%–2.9%, p < 0.001). Changes in vaccination coverage across counties after the policy implementation from 2015 to 2017 ranged from −6% to 26%, with larger increases in coverage in counties with lower prepolicy vaccine coverage. Results were robust to alternative model specifications. The limitations of the study were the exclusion of a subset of US states from the analysis and the use of only 2 years of postpolicy data based on data availability.

Conclusions

In this study, implementation of the California policy that eliminated nonmedical childhood vaccine exemptions was associated with an estimated increase in vaccination coverage and a reduction in nonmedical exemptions at state and county levels. The observed increase in medical exemptions was offset by the larger reduction in nonmedical exemptions. The largest increases in vaccine coverage were observed in the most “high-risk” counties, meaning those with the lowest prepolicy vaccine coverage. Our findings suggest that government policies removing nonmedical exemptions can be effective at increasing vaccination coverage.

Keywords:

Pediatrics – Vaccination and immunization – Public and occupational health – Vaccines – Schools – children – California – MMR vaccine

Introduction

Vaccine hesitancy, defined as the reluctance or refusal to vaccinate despite the availability of vaccinations, is a growing public health challenge in the US and globally [1–3]. The decline in vaccination rates, driven by vaccine hesitancy and lack of confidence in vaccines, has led to recent outbreaks of vaccine-preventable diseases and threatens the public health gains made against these infectious diseases over past decades [4–6]. The factors driving vaccine hesitancy are complex and include misconceptions and misinformation about vaccine safety, low perceived risk of infectious diseases, and lack of trust in healthcare providers [7, 8].

The policy debate surrounding vaccine hesitancy in the US has focused on vaccine exemptions, which provide an option for parents to waive current vaccination requirements for entry into school or daycare [9–11]. Currently, 18 states allow nonmedical exemptions to school vaccine requirements based on philosophical, personal, or other beliefs [4, 12]. Notably, all states permit medical exemptions to school immunization requirements for clinical conditions in which vaccination would be contraindicated (e.g., immunosuppression and receipt of live attenuated vaccines). Recent work indicates that in states allowing these nonmedical exemptions, the numbers of nonmedical exemptions are as high as 15%–25% in some counties. In 2019, measles emerged in seven of the 14 urban counties previously identified as high risk due to nonmedical exemptions [7]. However, there continues to be debate surrounding the effectiveness of policies that restrict nonmedical exemptions, with ongoing legislative discourse across the US [12].

Following a series of high-profile exposures and outbreaks of vaccine-preventable diseases—including a measles outbreak in Orange County, CA—California passed Senate Bill 277 (SB277), which eliminated a previous policy that allowed nonmedical exemptions to school immunization requirements [13–17]. Although previous observational studies have suggested that coverage increased in California after the policy’s implementation, the policy’s effectiveness remains unclear given natural variation in vaccination rates and lack of controlled policy evaluation [18–21]. Furthermore, the rate of medical exemptions in California increased after the policy, causing concerns that children who had received nonmedical exemptions were instead receiving medical exemptions, thus limiting the policy’s ability to increase overall vaccine coverage [7, 22].

The aim of our study was to conduct a rigorous, controlled analysis to estimate the effectiveness of California’s policy restricting nonmedical exemptions to increase vaccination coverage. To address ongoing debate surrounding these governmental policies, we provide a quasi-experimental, controlled analysis to evaluate the association between California’s policy and vaccination and exemption outcomes.

Methods

Data sources and study outcomes

We performed two empirical analyses to estimate the association of the 2016 California policy with changes in vaccination coverage and exemptions: (1) a synthetic control analysis, in which we used state-level data to create a hypothetical counterfactual “synthetic control California” to compare with the real California; and (2) a difference-in-differences analysis, in which we compared county-level data from California and control states. The California policy (SB277) went into effect on July 1, 2016, corresponding with the 2016–2017 school year. Our study objectives, methods, and planned analyses were prespecified in a preanalysis plan (S3 Appendix) [23].

For the state-level synthetic control analysis, we used publicly available state-level data from the US Centers for Disease Control and Prevention (CDC) SchoolVaxView website [24]. These data are collected by state immunization programs and aggregated by the CDC [25, 26]. Data were available from 2009 to 2017 (S1 Appendix). Data for the 2010–2011 school year were unavailable because they were not verified, so only data from 2011 onward were used in this analysis. The primary outcome for the state-level analysis was prespecified as measles, mumps, and rubella (MMR) vaccine coverage, defined as the proportion of children entering kindergarten who have received the two-dose MMR vaccine by the start of the school year. The secondary outcomes were the proportion of children entering kindergarten at the start of the school year with medical exemptions or the proportion entering kindergarten with nonmedical exemptions. We included medical and nonmedical exemptions as outcomes to evaluate whether any compensatory increases in medical exemptions were observed to be associated with the policy. We used a range of publicly available state covariate data (S1 Table). We obtained demographic data (e.g., education, income) from the US Census Bureau [27]. Data on health-related characteristics were retrieved from the Data Resource Center for Child and Adolescent Health and the Centers for Medicare and Medicaid Services [28, 29]. All results are presented as absolute percentage changes in MMR coverage and medical and nonmedical exemptions.

For the county-level difference-in-differences analysis, we used county-level data individually requested from state departments of public health. These data are compiled by county health departments using data submitted by public and private schools and then submitted to state departments of public health. We contacted health departments in all 50 states and the District of Columbia to request data. We received full county-level data from California and 16 other states for overall vaccination coverage and 17 states for vaccination exemptions (S4 Fig). When counties did not report overall vaccine coverage, we used county-level MMR coverage as a proxy for overall coverage. Missing county-level outcomes data were either imputed or excluded from the analysis, depending on the proportion of missing data (S2 Appendix). The primary outcome for the county-level analysis was prespecified as overall vaccination coverage, defined as the proportion of children entering kindergarten with all required vaccines at the start of the school year. The immunizations required to enroll in kindergarten vary by state but generally include MMR, polio, chicken pox (varicella), and diphtheria, tetanus, and pertussis (DTaP) [30]. The secondary outcomes were the proportion of children entering kindergarten at the start of the school year with medical exemptions or the proportion entering kindergarten with nonmedical exemptions. We used covariate data from the American Community Survey (ACS) [31]. In our final dataset, we only included counties with a population of at least 65,000 based on availability of covariate data. Additional information on covariate selection is described in S2 Appendix. All results are presented as absolute percentage changes in overall vaccination coverage and medical and nonmedical exemptions.

The primary outcome was MMR vaccine coverage in the state-level analysis and overall vaccine coverage in the county-level analysis. This difference in primary outcomes between analyses was prespecified and based on data availability because states do not report overall vaccine coverage to the CDC. We assumed state-level MMR coverage was a good proxy for overall coverage and stated this in our preanalysis plan (S3 Appendix). Furthermore, because the definitions of medical and nonmedical exemptions include children with exemptions for any vaccine and not just the MMR vaccine, for any given state the three outcomes did not always sum to one. In both the state- and county-level analyses, we used data for children entering kindergarten. Kindergarten is the school grade for children ages 5–6 years in the US and commonly the entry level into school. This age group is most likely to be affected by the policy change given the age of the children in relation to immunization requirements. Although the 2016 California policy also requires younger-aged children in preschool or daycare to have completed all the relevant vaccinations, kindergarten is the beginning of formal education and, as such, local and state health departments routinely collect immunization data for children in kindergarten.

Statistical analysis

We used a synthetic control study design to estimate the relationship between the California policy and vaccination coverage and exemptions at the state level. The synthetic control method is a statistical tool designed for comparative case studies, such as policy evaluations, in which only a single treated unit is available (e.g., state-level policy) [32–34]. The approach constructs a hypothetical control state (i.e., a “synthetic control California”) that matches the treated state (i.e., actual California) on the prepolicy outcome. The resulting synthetic control California provides a counterfactual estimation of the study outcome during the postpolicy period in the treated state in the absence of the treatment—i.e., it projects the outcome for California in the absence of the vaccine policy. The synthetic California is constructed with a weighted combination of control states. The synthetic control optimization algorithm estimates a weight for all nonexperimental states that minimizes the difference between the actual California and the synthetic California in the pretreatment period. Most states receive a zero weight, and only states with nonzero weights provide information for the construction of the synthetic control.

We created a synthetic control for California using untreated states as potential controls. The synthetic California was created by matching on average prepolicy outcomes (i.e., vaccine coverage and prevalence of exemptions) and demographic and health-related characteristics. These characteristic covariates were chosen using a stepwise variable selection process and cross-validation procedure to avoid overfitting (S1 Appendix). States were excluded from the construction of the synthetic control if they were missing relevant data. The key effect size for each study outcome was the difference in pre- to postpolicy change between California (i.e., the treated state) and the synthetic control California (i.e., the hypothetical untreated state). We constructed a unique synthetic control for each of the three outcomes, per convention given the data-driven nature of the process. This resulted in three synthetic control California states—one corresponding to each of the three outcomes (one primary, two secondary).

To assess whether the effect size in the synthetic control analysis was meaningful, we used conventional placebo testing. Whereas regression models provide confidence intervals (CIs) and p-values based on frequentist assumptions, inference in synthetic control methods is grounded in placebo tests (also known as permutation tests). In placebo tests, we reevaluate the effect size under the null condition (i.e., repeating the synthetic control method for each of the untreated states). The resulting placebo effect sizes for each of the control states quantify the variation in the outcome under the null hypothesis. By comparing them to the actual (California) effect size, we can determine whether the observed effect size in the treated unit is meaningful or whether it is similar in magnitude to the variation in the outcome in the absence of a treatment. The latter case would imply that our intervention or policy is not statistically meaningful [35, 36]. To conduct the placebo tests, we individually reassigned treatment status to each of the other states in the control pool. We then created a synthetic control for the new state and assessed the resulting effect size for every state in the dataset. We compared effect sizes for each state relative to California for all outcomes (S1 Appendix). We prespecified an effect size as meaningful if California was in the top fifth percentile of states.

We used a difference-in-differences study design to evaluate the association between the California policy and vaccination coverage and exemptions at the county level. The difference-in-differences design estimates the relative change in vaccination coverage and exemptions over time associated with the California policy as the difference between the treated group of counties (California) before and after the policy implementation and the control counties (counties not in California) before and after the policy implementation. To assess the parallel trends assumption required for the difference-in-differences methodology, we plotted the data for California and the control counties before the policy’s implementation (S2 Appendix). We used an ordinary least squares regression model for all outcomes and calculated robust standard errors clustered by county. We included an adjusted analysis with prespecified county-level characteristics (S2 Appendix).

Sensitivity analysis

We conducted sensitivity analyses to evaluate the robustness of our models (S1 Appendix and S2 Appendix) [37]. In the synthetic control and difference-in-differences analyses, we reran the models, iteratively excluding a single state from the control pool. These “leave-one-out” tests allowed us to evaluate the influence of individual states in the control pool on the effect size. For the county-level analysis, we performed a subanalysis in which we analyzed the data from states reporting complete overall vaccination coverage and excluding states that reported only MMR coverage. Finally, for the state-level analysis, we reran the model with varying combinations of characteristic covariates to evaluate the influence of the covariate combination on the outcome.

This study is reported per the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines (S4 Appendix). All statistical analyses were conducted in R (version 3.5.1). Public datasets and analytical materials are available online (S1 Table) [38, 39]. This was not human subjects research as the study relied on publicly available, secondary and aggregated data sources and was exempt from Stanford University’s Institutional Review Board approval.

Results

We included state-level vaccination and exemption data for 45 states, including California, from 2011 to 2017 (states excluded due to missing data are shown in S2 Table). The average prepolicy MMR vaccination coverage at the state level in the 2015 school year was 94.2% and ranged across states from 87.1% to 99.4% (94.5% in California). The average prepolicy nonmedical exemption prevalence in 2015 was 2.2% and ranged across states from 0.4% to 6.2% (2.4% in California); the average prepolicy medical exemption prevalence was 0.3% and ranged across states from 0.1% to 1.2% (0.2% in California).

We included county-level vaccination and exemption data for 17 states from 2010 to 2017 based on availability of data from state departments of public health (S6 Table). In California, the mean prepolicy county-level vaccination coverage in 2015 was 92.6% (95% CI 91.8–93.4) and ranged across counties from 68.9% to 100%, the prepolicy county average for nonmedical exemption prevalence was 3.1% (95% CI 2.5–3.6) and ranged across counties from 0% to 21.9%, and the prepolicy county average for medical exemption prevalence was 0.21% (95% CI 0.18–0.23) and ranged across counties from 0% to 1.7%.

State-level analysis

For each of the three state-level outcomes, the synthetic control California matched the observed California in the prepolicy period (Fig 1). The synthetic control California for each of the three outcomes was predominantly composed of three to five control states, with all other states having minimal weights (Fig 2). No single state had a weight greater than 50% except for Texas in the synthetic California for the nonmedical exemptions outcome. In this state-level analysis, we estimated that the 2016 California policy was associated with a 3.3% increase in MMR coverage, from 94.5% in 2015, relative to its synthetic control in the postpolicy period (top 2 of 43 states evaluated in the placebo tests, top 5%). The largest improvement in MMR coverage occurred in North Dakota (3.6%). The 2016 California policy was associated with a 2.4% decrease in nonmedical exemptions in the postpolicy period (top 2 of 43 states evaluated in the placebo tests, top 5%), from 2.4% in 2015, whereas medical exemptions increased by 0.4% (top 1 of 44 states evaluated in the placebo tests, top 2%), from 0.2% in 2015 (Fig 3). The largest decrease in nonmedical exemptions occurred in Vermont (2.8%). California had the largest increase in medical exemptions, followed by Maryland (0.2% increase).

County-level analysis

In the county-level analysis, we found that the trends for outcome variables were relatively similar between California counties and counties in control states in the prepolicy period (S5 Fig). Overall vaccination was generally similar in the prepolicy period, although California counties had slightly lower vaccination levels and greater variation than control counties. Prepolicy variation in nonmedical exemptions for California was greater than for control states (S7 Table). In the county-level analysis, we estimated that the California policy was associated with a 4.3% (95% CI 2.9–5.8, p < 0.001) absolute increase in vaccine coverage for children entering kindergarten in California compared with those in control states (Table 1). The policy was also associated with a 3.9% (95% CI 2.4–5.4, p < 0.001) absolute decrease in nonmedical exemptions and a 2.4% (95% CI 2.0–2.9, p < 0.001) absolute increase in medical exemptions compared with counties in control states (Table 1). We also estimated coefficients for the characteristic demographic covariates included in the model. Income and education status of less than a high school degree were associated with increases in vaccination coverage and decreases in medical and nonmedical exemptions (Table 1). The percentage of uninsured children was significantly associated with a small increase in medical exemptions. Education status of some college as well as bachelor’s degree or beyond were significantly associated with small decreases in nonmedical exemptions, whereas race (defined as percent white, based on prior literature [40]) and percent of uninsured children were significantly associated with small increases in nonmedical exemptions (Table 1).

There was variation in vaccination coverage and exemptions across counties before and after the policy implementation. Counties with a higher proportion of prepolicy nonmedical exemptions in 2015 (i.e., “hot spots” most at risk of outbreaks) had larger decreases in nonmedical exemptions following the policy’s implementation (Fig 4). Likewise, counties with lower prepolicy overall coverage had the largest increases in overall coverage following the policy’s implementation. The absolute change in vaccination coverage between 2015 and 2017 across counties in California ranged from −6% to 26%, with 12/57 (21.1%) of counties decreasing coverage, 28/57 (49.1%) experiencing an increase in coverage between 0.1 and 4 percentage points, and 17/57 (29.8%) experiencing an increase in coverage >4 percentage points (S8 Table). For nonmedical exemptions, 10/57 (17.5%) of counties in California either had no change or an increase, and 47/57 (82.5%) of counties had a decrease in nonmedical exemptions ranging from −0.1% to −21.5% between 2015 and 2017. All counties had an absolute increase in medical exemptions, ranging from 0.5% to 9.3%. Plumas County, population 18,742, experienced the 9.3% rise in medical exemptions, from 1.0% in 2015 to 10.3% in 2017 (S8 Table).

Sensitivity analysis

The study findings were overall robust to various sensitivity analyses. In both analyses, changing the control data included did not change the overall findings (S5 Table and S9 Table). In the state-level analysis, the effect size was robust to changes in the characteristic covariates used in the model (S4 Table). In the county-level analysis, the subanalysis excluding states reporting MMR coverage did not alter the effect size (S10 Table). Details on the sensitivity analysis are included in S1 Appendix and S2 Appendix.

Discussion

In this empirical policy analysis, we evaluated California’s 2016 policy removing nonmedical vaccine exemptions and found evidence that the policy was associated with an increase in vaccination coverage in children entering kindergarten. At the state level, the policy was associated with a 3.3% increase in MMR coverage and a 2.4% reduction in nonmedical exemptions. The policy was also associated with a 0.4% increase in medical exemptions. At the county level, often the counties with high baseline exemption prevalence and most “at risk” of an outbreak had the largest increases in vaccine coverage from the policy. Despite concerns around the observed increase in medical exemptions, our study found that the policy was associated with an overall increase in vaccine coverage. The use of two independent state- and county-level analyses with consistent results provides evidence to support considering the adoption of similar governmental policies eliminating nonmedical exemptions to help address the growing public health challenge of vaccine hesitancy across the United States and globally.

Previous descriptive analyses examined the impact of the California policy shortly after its implementation. These analyses found an increase in overall vaccination coverage among students from 92.8% in 2015 to 95.1% in 2017 along with a decrease in nonmedical exemptions [7, 22]. However, the policy’s effectiveness remained unclear because these analyses did not account for variation and trends in vaccine coverage across the US. There was also an increase in medical exemptions (from 0.2% in 2015–2016 to 0.7% in 2017–2018) that might have offset the described decreases in nonmedical exemptions [18, 22]. Two recent studies provided additional descriptive analyses of the impact of the California vaccine policies on vaccination coverage [20, 21]. In one study, the authors used county-level data for California to report the changes in the percentage of children entering kindergarten with up-to-date vaccination status, as well as geographic clustering of unvaccinated children associated with three California vaccination policies. In a subsequent study, the authors extended this work to focus specifically on SB277, the 2016 California vaccine policy. Both studies similarly found an increased percentage of children entering kindergarten with all vaccinations completed postimplementation of the California policy, though attribution to SB277 is limited by availability of comparison units to form a control group. This leaves open the possibility that observed changes may be over- or underestimated, depending on the secular trends. Our study further extended this work by providing a controlled, quasi-experimental methodological approach to estimate the impact of the 2016 California vaccine policy. Taken together, these studies contribute robust evidence to support the conclusion that vaccination coverage increased after the California policy’s implementation.

In this study, in both the county- and state-level analyses, nonmedical exemptions decreased and medical exemptions increased, a trend that supports our conclusion that the observed increase in vaccination coverage in California is associated with the implementation of the 2016 policy rather than confounding factors. Additionally, although the rise in medical exemptions could indicate that some children who may have received nonmedical exemptions in the past are now receiving medical exemptions, the net effect following the California policy was still an increase in vaccination coverage. Notably, new legislation was recently approved in California to require centralization of medical exemptions, similar to policies in place in Mississippi and West Virginia—states that also do not allow nonmedical exemptions for school entry. A centralized review may further increase vaccination coverage in California [12].

In our state-level synthetic control analysis, we used placebo testing to evaluate whether the estimated changes in study outcomes (e.g., vaccine coverage) associated with the California policy were statistically meaningful. The observed increase in MMR coverage in California associated with the 2016 policy was much greater than the changes in the majority of the placebo states and met our prespecified threshold for a significant finding (Fig 3, S3 Fig). However, we also observed notable changes in a select number of placebo states. North Dakota had an increase in coverage of 3.6% relative to its synthetic control during the year of the policy—0.3% larger than the increase in California attributable to the policy. We did not find evidence that North Dakota implemented any statewide vaccination policies at this time. The observed increase could be due to the state’s continuing efforts to address vaccine hesitancy through other mechanisms. Vermont had the largest decrease in nonmedical exemptions, 2.8% relative to its synthetic control during the year of the policy—0.4% larger than the decrease seen in California. This is likely due to Vermont’s 2015 policy, which removed philosophical exemptions [41]. Notably, the number of children enrolled in kindergarten in North Dakota and Vermont in 2016 was approximately 10,000 and 6,500, respectively, whereas the kindergarten enrollment for California in the same year was approximately 521,000 [31]. Therefore, despite the modestly larger percentage changes in smaller states like North Dakota and Vermont, the absolute number of children affected is much larger in California. Nonmedical exemptions in California showed a slight decrease in 2014. This may have been a result of another California policy passed in 2014 that required parents seeking exemptions to provide signed documentation from a healthcare provider [42].

Although the state-level increase in vaccination coverage was modest (approximately 3%), this estimate should be interpreted in the context of county-level variation in vaccine coverage and risk of outbreaks. Mainly, California counties with higher prevalence of prepolicy nonmedical exemptions and most at risk for outbreaks had larger improvements in overall coverage. Existing evidence suggests that the potential for infectious disease outbreaks is driven by hot spots of low vaccine coverage, meaning some counties may have particularly low coverage and be prone to outbreaks [4, 7, 12]. States with laws permitting personal belief exemptions or easily obtainable exemptions have greater hot spots, with high levels of nonmedical exemptions and outbreak risk, compared to states with stricter exemption laws [12]. When examining county-level data, we found a range of effect sizes for the change in vaccination coverage between 2015 and 2017, from a 6% decline in Sierra County to a 26% increase in Trinity County (S8 Table). The county-level variation in the absolute change in overall vaccination coverage and exemptions suggests that the effectiveness of the policy may be higher in most at-risk (i.e., low coverage) hot spots. The variation in the change in vaccination coverage across counties before and after the policy implementation may help explain why the increase in overall coverage does not equal the difference between the reduction in nonmedical exemptions and the increase in medical exemptions. This discrepancy may also be because our definitions of the outcome variables as the state- or county-reported vaccination and exemption percentages do not reflect missing certificate or voucher data from schools.

The importance of the observed increases in vaccine coverage in California after the implementation of the policy is further illustrated by considering how small changes in coverage have disproportionate effects on the number and size of outbreaks, as well as on public health costs [6]. A recent modeling study predicted that a 5% decline in MMR vaccine coverage driven by vaccine hesitancy would result in a 3-fold increase in measles cases in children per year, corresponding with a cost of US $2.1 million [4]. Additionally, the benefit of small increases in vaccine coverage is particularly evident when approaching approximated herd immunity levels of 90%–95% vaccine coverage. In 2015, 24 California counties had coverage levels below the range needed for herd immunity; in 2017, this number was 12 counties. This finding has implications for other states, as current data suggest that many states have counties and communities below the conventional threshold for herd immunity [4]. As multiple states continue to experience outbreaks of vaccine-preventable diseases, interventions like the California policy that increase vaccine coverage levels above the required thresholds will continue to be vital. A recent analysis of the California policy suggested that legislation may need to consider local differences because areas of northern California continued to demonstrate high rates of students without up-to-date vaccination status despite the state’s intervention [20]. Our county-level analysis found a relationship between income and education, and vaccine coverage, suggesting that socio-demographic factors may play an important role in vaccine coverage. Additional research into other correlates of vaccine hesitancy in particular regions may be necessary to obtain ideal coverage in areas that remain vaccine hesitant. Finally, settings that remain at low vaccine coverage despite removal of nonmedical exemptions may require additional interventions that include local community involvement and educational programs [43–45].

The study results should be interpreted in the context of the limitations of the data and study design. Vaccine coverage and exemption data are reported by schools, and data collection procedures vary by state and year. We included a subset of US states in the analysis (included sample size of N = 45 for state data, N = 16–17 for county data) based on data availability, which could potentially introduce bias. To address these limitations, we used two different datasets and methodologies that both aim to control for differences between California and control states and counties in order to assess the effectiveness of the California policy [45]. We also ran a set of sensitivity analyses that showed that our findings were robust to the states included in the control pool. For both the state- and county-level analyses, the statistical design assumed that no concurrent change occurred in 2016. To address this, we conducted a brief literature review to determine whether any concurrent interventions took place during this time and did not find any potentially confounding programs [46, 47]. The synthetic control is ideally constructed without any acute changes during the preintervention period, although California passed two policies aiming to increase vaccine coverage. In 2014, California passed AB2109, which required proof of having consulted a healthcare provider before receiving an exemption. In 2015, California introduced a second initiative meant to educate administrators on the conditional admissions requirements. The 2015 conditional entrants initiative, although effective, was limited in scope compared with the statewide policy and would only affect MMR coverage or overall exemptions and not nonmedical exemptions. In addition, these policies were unlikely to meaningfully affect our estimate because the calibration process of the synthetic control minimized these perturbations [42]. We plotted prepolicy trends in vaccination coverage and exemptions to assess the parallel trends assumption in the difference-in-differences analysis. Although the trends were similar between California and control counties before the intervention, there was greater variability in California over the years, especially for nonmedical exemptions. Finally, our analyses included only 2 years of postpolicy data, so we are unable to observe the persistence of increased coverage over time.

As vaccine hesitancy becomes a larger public health challenge in the US and globally, with vaccine-preventable disease outbreaks growing, the debate concerning state policies to remove nonmedical exemptions is ongoing. Although such policies do not address the larger problem of vaccine hesitancy, including a lack of confidence in vaccination, they effectively increase vaccination coverage. Our study finds that vaccine coverage increased and nonmedical exemptions decreased in the time period following the implementation of the California policy eliminating nonmedical exemptions. Our conclusions are strengthened by the use of two independent analyses, which both found similar results. These study results support the idea that state-level governmental policies to remove nonmedical exemptions can be effective strategies to increase vaccination coverage across the US.

Supporting information

S1 Appendix [docx]
The synthetic control method and state-level data.

S2 Appendix [docx]
The difference-in-differences method and county-level data.

S3 Appendix [docx]
Preanalysis plan.

S4 Appendix [docx]
STROBE checklist.

S1 Fig [docx]
Characteristic state covariate selection cutoffs.

S2 Fig [docx]
Cross validation of synthetic controls using training and testing data for variable selection.

S3 Fig [docx]
Placebo plot trajectories for control states (gray) and treated state (blue).

S4 Fig [docx]
Flow chart of data collection for county-level outcome data from state health departments for control states.

S5 Fig [docx]
Prepolicy trends in outcome variables in California and control counties.

S1 Table [docx]
Data sources for state-level synthetic control analysis.

S2 Table [docx]
States excluded from control pool due to missing data.

S3 Table [docx]
Characteristic covariate weights used to create the synthetic California for each outcome in base case analysis.

S4 Table [docx]
Characteristic covariate sensitivity analysis for state-level synthetic control analysis.

S5 Table [docx]
Control state sensitivity analysis with leave-one-out tests.

S6 Table [docx]
List of control states for the county-level difference-in-differences analysis for each outcome.

S7 Table [docx]
Prepolicy trends for outcome variables in California and control states.

S8 Table [docx]
Change in county-level outcome variables between 2015 and 2017 for California counties.

S9 Table [docx]
County-level sensitivity analysis with leave-one-out tests.

S10 Table [docx]
County-level sensitivity analysis with a subset of data reporting overall vaccine coverage.