Why did Medicare ACOs reset their benchmarks early? A pre-registered study

Overview

The visual abstract (select it for the full-size image) is drawn from the same results files as the dashboard (report/visual_abstract.py; SVG).

In the Medicare Shared Savings Program¹ (MSSP), an accountable care organization² (ACO) earns a share of the savings when its patients' spending comes in below a benchmark. The benchmark is reset at the start of each agreement period³ from the ACO's own recent spending, pulled toward its region's. An ACO can end its agreement early and start a new one⁴, which resets the benchmark early. Of the 334 ACOs whose agreements began July 1, 2019 or January 1, 2020, 29 ended them early and started new ones on January 1, 2022; 277 stayed in their agreements and 28 left the program (from the enrollment files⁵).

This post asks why they did, and what renewing changed:

• Who chose to reset, and why (H1-H3). Did early renewers stand to gain from a reset because their regions spent more than their benchmarks (the primary test), because their own spending had run high, or because renewing was the only way into the track that shares 75% of savings⁶?
• Spending after renewal (H4). Did early renewers' spending change afterwards, relative to ACOs that had not yet renewed? A difference-in-differences estimate.

Terms used throughout:

• PY: performance year, the calendar year over which CMS measures an ACO's spending against its benchmark.
• Benchmark years (BY1-BY3): the three years before an agreement starts, whose spending sets the new benchmark.
• BASIC and ENHANCED tracks: BASIC has five levels, A to E. Levels A and B are one-sided (the ACO shares savings only), and C to E are two-sided (it also repays part of any losses). ENHANCED is two-sided with the highest sharing rate.
• Glide path: BASIC's schedule for moving an ACO up a level each year, toward two-sided risk.

The analysis plan was registered on OSF Registries (DOI 10.17605/OSF.IO/C53W9)⁷ before any outcome value was read, together with the analysis code, which was written and tested on synthetic data⁸ first. Every result below is reported, whatever it shows; changes made after registration are listed separately under Deviations.

In short. The primary test found what the plan calls "evidence for the reset motive beyond past savings": early renewers' regions spent 3.4 percentage points of benchmark more than stayers' regions, relative to their current benchmarks (95% credible interval 1.7 to 5.4, p < 0.001). That is not a larger expected gain. Early renewers were beating their benchmarks by more than stayers, with savings rates about 2 points higher, and by the approximation a reset would on average have lowered both groups' benchmarks (by about 1.6% for early renewers and 1% for stayers): early renewers stood to gain more from the regional adjustment and to lose more on their own spending. In the decision model, renewing early was more likely with a higher regional pull and a higher BASIC level. Early renewers' spending growth in PY2022-PY2024 did not differ detectably from that of ACOs not yet renewed (-0.4%, -4.0% to +2.5%); a continuing pre-period difference in trends cannot be ruled out, so this is not read as an effect of renewing.

In plain English.

• Early renewers were in regions that spend more. The ACOs that reset early tended to be in places where Medicare spends more per patient, relative to the ACO's own benchmark, than the places stayers were in: about 3.4% of the benchmark more, or roughly $385 per patient per year at the median benchmark of $11,312 in PY2021 (the PY2021 results file⁹). A reset pulls the benchmark toward the region's spending, so these ACOs had more to gain from that part of it.
• But they were already beating their benchmarks. They spent further below their targets than other ACOs did. A reset also resets the target to the ACO's own recent spending, so for most of them that part would have cut the benchmark. By the study's estimate the two parts roughly cancel, and a reset would not have raised the average early renewer's benchmark.
• ACOs at higher BASIC levels renewed early more often. ACOs already at BASIC's higher, two-sided levels (C to E) were more likely to renew early; 16 of the 27 BASIC-track early renewers were at the top level, E. Renewing was their way into ENHANCED, which keeps 75% of savings, so the pattern fits an interest in the higher sharing rate, but it cannot be told apart from the regional motive, because both came with the same choice.
• Spending afterward looked about the same. Early renewers' spending growth in 2022-2024 was close to that of ACOs that had not renewed. The data rule out a drop of more than about 4% or a rise of more than about 2.5%, but a difference in trends that began before renewal could still hide a smaller effect, so this is not read as "renewing had no effect".
• What this means. ACOs that reset early were positioned to gain from the regional adjustment and from a higher sharing rate. A reset was not a simple way to get a higher benchmark.
• These are the 2022 rules. Benchmarks for agreements starting in 2024 or later give ACOs credit for past savings and soften the cut for an ACO that spends more than its region, so the same trade-off may not hold for an ACO renewing today (see Limitations).

Background for newcomers

Four short primers for readers new to the program or the methods; each opens on a click, and experienced readers can skip them.

Why pre-register

The CMS public use files¹¹ used here are public. Anyone analyzing them can try many specifications and report the one that works, often without meaning to (Simmons, Nelson and Simonsohn 2011¹²). A pre-registration fixes the question, the variables, the tests and the decision rules before the outcomes are seen, so a reader can tell confirmatory results from exploration (Nosek et al. 2018¹³).

Two details make that claim checkable here:

• An independent timestamp. Git commit dates can be set by hand, so the plan was registered on OSF Registries, which records the time itself. The plan file's sha256 and the registered code commit are recorded in REGISTRATION.md¹⁴, so the repository copy can be matched to the registered one.
• An outcome lock in the code. Every data file is hashed before it is read⁸. A file whose hash matches a performance-year results file or a county spending file could not be opened until REGISTRATION.md existed. There was no bypass flag. Before registration only the enrollment files⁵, the CMS data dictionaries (for example the PY2021 results dictionary⁹) and the regulation text¹⁵ were used, plus the column names (not the values) of the outcome files, to write the code.

How the question was chosen

The question in this post was the fourth candidate. The first three were dropped at the design stage, using only the regulation and the enrollment files:

1. Downside risk. The program's "glide path" moves ACOs into two-sided risk on a schedule¹⁶, which looked like a natural experiment. The regulation showed that the timing is partly chosen: ACOs choose the level they start at, can move up faster¹⁷, and could pause the advance for 2021 and 2022; from 2023, an ACO still in one-sided Level A or B could stay there for the rest of its agreement. The enrollment files⁵ showed that only about 10-28% of ACOs were in two-sided risk in the years the schedule called for it. Not a usable treatment.
2. The benchmark ratchet. Do ACOs spend less once their spending starts counting toward the next benchmark? In the enrollment data, benchmark-year status is collinear with how long the ACO has been in its agreement, so the effect cannot be separated from tenure. Kept only as a descriptive table.
3. A correction. An early tabulation found 139 ACOs leaving their benchmark-year schedule at the 2022 transition, and an unverified inference treated them all as early renewers. Accounting for the optional one-year extension¹⁸ of agreements that began in 2018 (from the 2020 interim final rule¹⁹) showed that only 29 renewed early into a January 1, 2022 agreement.

Those 29 became the question. A first version asked only whether they had lower savings rates. That was too narrow to explain anything, so the plan became explanatory, with two motives that predict different things.

Related work

Four strands of earlier research bear on this study. None of them, as far as a focused literature search found, studies early renewal itself.

• Rebasing creates a ratchet. The rules for setting an ACO's benchmark at renewal weaken its incentive to save, and the old weights on the benchmark years even rewarded higher spending just before a new agreement (Douven, McGuire and McWilliams 2015²⁰). That is one reason the H4 design guards against anticipation and mean reversion.
• ACOs differ widely from their regions. Even after case-mix adjustment, ACOs' spending varies widely relative to local fee-for-service spending, so moving benchmarks toward regional averages shifts them unevenly (Rose, Zaslavsky and McWilliams 2016²¹). That variation is what the regional pull measures, and why its case-mix adjustment matters (see Limitations).
• Benchmarks drive who participates. After regional spending entered benchmarks in 2017, participation shifted toward providers whose spending was low relative to their region (Lyu, Chernew and McWilliams 2023²²), and ACOs whose effective benchmark fell were 7 percentage points more likely to leave (Chen and McWilliams 2025²³). The primary result here is the same kind of selection, in when to reset rather than whether to join or leave.
• The 2019 rules changed participation decisions. Under Pathways to Success, for the first time more ACOs left the program than joined it: 73 exits against 65 entrants in 2019 (Ying et al. 2023²⁴). ACOs that had achieved savings were more likely to stay, a borderline result (odds ratio 1.76, 95% interval 0.99 to 3.09), and so were ACOs with higher spending, which the authors read as a sign that administrators "needed to see enough room" to make their savings target, the same consideration as the reset motive here. ACO leaders described reassessing potential savings against losses in light of the benchmark changes (Ying et al. 2024²⁵). Of the Track 1 ACOs facing renewal in 2019, 40% left, at a projected cost of up to $396.8 million in lost savings by 2022 if that rate continued (Moloci et al. 2021²⁶).

How these results fit, in plain English.

• Agrees with: earlier work found that ACOs pick their moment using the benchmark. They joined when the regional formula favored them (Lyu and colleagues), and they left when their benchmark fell (Chen and McWilliams). This study finds the same behavior in a new choice, when to reset: early renewers were the ACOs whose regions spent more relative to their benchmarks.
• Fits with, though that evidence is weak: after the 2019 rules, ACOs that were already saving money were more likely to stay in the program, a borderline result whose interval includes no effect (Ying and colleagues). Early renewers here were also savers, beating their benchmarks by more than other ACOs. But that same success meant a reset would have cut the own-spending part of their benchmark, so the simple story "ACOs reset to get a higher benchmark" does not fit them on average.
• Consistent with: earlier work showed that comparing an ACO with its region depends heavily on how patients' health is adjusted for (Rose and colleagues). Here too, the size of the regional difference doubles without that adjustment.
• Cannot confirm or rule out: the worry that rebasing rewards spending more just before a new agreement (Douven and colleagues). Early renewers' spending growth afterward showed no detectable difference, but the design could not separate an effect from a trend that started before renewal.
• Does not address: what departures or early renewals cost Medicare (Moloci and colleagues estimated the cost of departures); this study does not estimate costs.

This study adds the early-renewal choice: which ACOs used the option to reset their benchmark early, measured with public data under a pre-registered plan. The search used Google Scholar Labs (three queries on 2026-10-27, one rejected as too broad), and its questions, raw results and per-paper summaries are saved in the study's research/ folder.

Design

Groups

The base population is the 334 ACOs in the enrollment files⁵ whose agreements began July 1, 2019 or January 1, 2020 (plan section 3¹⁰). Both cohorts' agreements run to December 31, 2024 (42 CFR 425.200(b)(4)(ii)¹⁸ and (b)(5)), so a January 1, 2022 agreement start means an early renewal: an ACO that terminated its agreement²⁷ and immediately entered a new one (425.20, "renewing ACO"⁴).

Two motives, and why the savings rate cannot separate them

• The reset motive. A new benchmark would be higher than the current one.
• The sharing-rate motive. An ACO in the BASIC track²⁸ cannot move to the ENHANCED track during an agreement period (2018 rule²⁹), so renewing early was the only way in before its agreement ended. ENHANCED shares 75% of savings⁶ and up to 75% of losses. In the enrollment files⁵, 26 of the 27 early renewers that were in BASIC moved to ENHANCED, and no stayer did.

The expected benchmark gain from a reset has two parts: an own-spending part (how the ACO's recent spending compares with its current benchmark) and a regional part (the adjustment toward the region³). The own-spending part is close to a restatement of the savings rate. Writing out the regional adjustment shows that it contains the own-spending part too, with a negative sign. With phase-in weight \(w\), before capping,

where "pull" is the regional pull defined below and "own" the own-spending component, so the regional part is also tied to past savings. The primary test therefore uses the regional pull: how far the region's spending, adjusted to the ACO's case mix the way CMS adjusts it³, sits above the ACO's current benchmark,

where \(R_e\) is the risk-adjusted regional spending for enrollment type³ \(e\) (ESRD, disabled, aged/dual, aged/non-dual), \(s_e\) the ACO's share of person-years of that type, and \(B\) its current per-capita benchmark. It has no term for the ACO's own benchmark-year spending, though it is not free of the ACO's own data: \(B\) was set from the ACO's earlier spending, and \(R_e\) is adjusted with the ACO's own risk scores (see Limitations). An ACO can be beating its own benchmark and still stand to gain from a reset because its region spends more; that is the part of the reset motive a savings rate cannot show.

Hypotheses

As registered (plan section 1.3¹⁰):

H1-H3 describe who chose to reset and why; they are associations, not effects. H4 estimates the effect of a bundled package (a new benchmark, for almost all a move to ENHANCED, and a change in which years count toward the next benchmark) and cannot separate its parts.

Building the expected gain from public data

CMS does not publish what an ACO's benchmark would have been had it reset. The plan rebuilds it from 42 CFR 425.601³ in six steps (section 4.3¹⁰), which come down to four parts:

• the equally weighted mean³ of the ACO's per-capita spending in the three benchmark years (PY2019-PY2021), risk-adjusted and restated to the latest year's risk scores (CMS's methodology specifications, version 10³⁰, section 4.1.2, step 3), and trended to 2021 dollars with the national-regional blend;
• risk-standardized regional spending from the county files³¹, weighted by where the ACO's beneficiaries live³ (from the assigned-beneficiaries-by-county files³²) and adjusted to the ACO's case mix;
• the regional adjustment with the phase-in weights³ and caps;
• and the shares that combine the four enrollment types³.

Because several steps can only be approximated from public files, the approximation was validated before it was used: rebuilt for the regular renewers whose 2022 benchmarks CMS does publish (in the results files⁹; 101 of the 111 eligible; 10 lack either the rebuilt or the published benchmark), with a pass rule fixed in the plan (section 4.4¹⁰: Spearman correlation at least 0.80, median absolute percentage error at most 5%). Early renewers were kept out of the validation sample, and the 2022 results file was filtered as it was loaded, so validation could not reveal their actual reset.

Methods

Estimates and significance

Every confirmatory estimate is Bayesian, and significance is measured by resampling: the null distribution is simulated rather than looked up (an accessible introduction is VanderPlas's "Statistics for Hackers"; video³³, slides).

• Group comparisons (H1, H2a, H2b, H4). The estimate is the difference in means, early renewers minus stayers, within each cohort and averaged with weights proportional to the early renewers in each cohort. Its posterior is a Bayesian bootstrap (Rubin 1981³⁴): 10,000 draws of Dirichlet(1, ..., 1) weights over the ACOs in each cohort-by-group cell.

• Significance by stratified permutation. Early-renewer labels are shuffled within cohort 10,000 times, and the estimate is recomputed on each shuffle. The two-sided p-value is

  \[p = \frac{1 + \#\{\,|\hat\theta_\text{shuffled}| \ge |\hat\theta_\text{observed}|\,\}}{1 + 10{,}000}\]

Adding 1 to both counts treats the observed labels as one of the shuffles, so the p-value is never zero (Phipson and Smyth 2010³⁵). The resampling p-value decides, by rules written before the results (below). When it and the credible interval disagree, both are reported.

H3: a Bayesian decision model with measurement error

A logistic regression of early renewal on the three core predictors (regional pull, own-spending component, higher BASIC level) and five adjustment factors (cohort, size, revenue category, assignment method, first or later agreement), fit with PyMC³⁶ (NUTS³⁷, 4 chains of 2,000 draws; plan section 5.4¹⁰). Continuous predictors are scaled by two standard deviations (Gelman 2008³⁸), so each coefficient is a log-odds difference per 2 SD. With 24 events (27 eligible early renewers, 3 missing a predictor) and 9 parameters, the Normal(0, 1) priors keep the estimates stable, so a second fit with Student-t priors (Gelman et al. 2008³⁹) is reported for sensitivity. A Firth-penalized logistic regression (Firth 1993⁴⁰, fit with firthmodels⁴¹) with the core predictors and cohort is always run beside it, and decides H3 only if the Bayesian model fails its convergence checks⁴².

The own-spending component is an approximation, so it enters as a latent true value observed with error,

with \(\sigma_v\) set by the validation step before the model is fit (4.38 percentage points, from CMS's own benchmark-year spending for the validation sample; validation.json¹⁴).

Shuffling labels in a model with several predictors would test "nothing matters", not "this factor matters given the others". Each core predictor is instead tested by direct simulation under the reduced model (a parametric bootstrap): fit the model without that predictor, simulate 10,000 sets of outcomes from its posterior, refit the full model to each, and compare the observed coefficient with that null distribution. That is about 40,000 MCMC refits, which took 17 hours on 5 cores.

H4: difference-in-differences with an equivalence rule for pre-trends

The outcome is log per-capita spending. For each ACO,

and the H4 estimate is the same cohort-weighted difference in means as above, applied to \(\Delta\). Comparison ACOs are those not yet renewed: a stayer's years after it renews into a 2023 or 2024 agreement are dropped, following the not-yet-treated idea of Callaway and Sant'Anna (2021)⁴³ (plan section 5.8¹⁰).

The main threat is selection on the outcome's own path: if early renewers renewed after unusually high spending, their spending could fall afterwards by mean reversion alone. There is only one pre-period change (PY2020 to PY2021, both affected by COVID-19), and with about 29 treated ACOs a non-significant pre-trend test proves little (Roth 2022⁴⁴). So H4 counts as an effect only if an equivalence condition holds (the logic of an equivalence test⁴⁵): the pre-trend interval, scaled to the distance between period midpoints, must rule out a trend difference large enough to produce the estimate. A trend-adjusted estimate (the pre-period slope carried forward, in the spirit of Rambachan and Roth 2023⁴⁶) is reported beside it, along with propensity-weighted (Rosenbaum and Rubin 1983⁴⁷), risk-adjusted, balanced-panel and single-cohort versions.

Conventional methods, side by side

For comparison, and not used for any decision (plan section 5.7¹⁰): Welch t-tests (Welch 1947⁴⁸) and OLS with HC2 robust errors (MacKinnon and White 1985⁴⁹) for H1, H2a and H2b; maximum-likelihood logistic regression for H3 (statsmodels⁵⁰); and two-way fixed effects with an event study, errors clustered by ACO, for H4. An agreement table puts each registered verdict next to the conventional one; for H2a and H2b the registered counterpart is OLS with HC2 errors, and the Welch t-test is shown beside it.

Decision rules

Fixed before the results (plan section 7¹⁰). For H2b: p < 0.05 and positive means "early renewers' regions spent more relative to their current benchmarks, evidence for the reset motive beyond past savings"; p < 0.05 and negative is evidence against it; otherwise "no detectable difference", reported with the credible interval and the minimum detectable effect. No test here can rule the sharing-rate motive in or out, because in these data it is bundled with the reset in a single choice.

What was simulated, and when

Simulated data played no part in forming the hypotheses. Three kinds of data were used, at three stages:

The synthetic values are invented, not fitted to CMS data: no CMS outcome value was viewed to write the generator. It builds a small stand-in (270 ACOs in the two cohorts and 80 regular renewers, in 3 states of 15 counties) with lognormal spending, and it copies the real files' quirks: column names that change between years, * suppression markers, date formats, and early-renewer rows the loader must drop. It also plants known answers. Early renewal is made more likely with a higher regional pull and own-spending component (logistic coefficients 1.0 and 0.6 per SD), and spending is 4% lower after renewal. The tests⁸ check that the SQL matches an independent NumPy computation of every constructed variable, that the decision model recovers a planted direction on made-up data, and that H4 recovers the -4% to within 0.015 log points.

The power analysis uses real spreads, read after registration as the plan allows: the PY2016-PY2018 savings-rate SD for H1, the stayers' SDs of the regional pull and own-spending component for H2, and the SD of the pre-period spending change for H4 (power.json¹⁴).

Pipeline

The pipeline is in analysis/mssp-benchmark-ratchet/⁸, linked here as registered (commit a4064f5):

• Data snapshot. Every CMS file used is listed with its size, sha256 and URL in data-snapshot.tsv⁸, and each hash is checked before the file is read. The files come from four CMS datasets: ACO enrollment⁵, performance-year results⁹, county expenditures and risk scores³¹ and assigned beneficiaries by county³².
• dbt for the SQL. The constructed variables are a dbt⁵¹ project on DuckDB⁵² (dbt-duckdb, in dbt/⁸): 28 models, one per derived table, in three tagged stages, with schema and data tests (unique keys, accepted values, components adding to the gain, shares summing to one, no early renewer in the validation load). Python loads every raw table after its hash check, so dbt never reads a file directly.
• Two implementations. A synthetic-data generator⁸ writes made-up files with the real file and column names and computes the constructed variables independently in NumPy. The tests require the SQL to match it, and every stage runs end to end on synthetic data, including the recovery of a planted -4% effect in H4.
• Staged runner⁸. Each stage refuses to start until the outputs it depends on exist: validate, then power, then confirm and robustness (both need only validation and power, and ran at the same time), then did, conventional and exploratory. So the power analysis was committed before any confirmatory estimate. The robustness results were committed first, about 17 hours before the confirmatory ones, and were read only together with them, primary test first.
• Reproducible environment. uv⁵³ with a lockfile; PyMC, firthmodels and statsmodels⁵⁰ are pinned in uv.lock, and every refit has its own seed from a fixed seed sequence⁵⁴ (parallel streams), so results do not depend on the number of processes. They do vary slightly between runs, within simulation error, because the analysis table is read without a fixed row order (see Reproduce).

Writing the code before registering found gaps in the plan while it could still change: column names that drift between years (ACO_Num and ACO_ID, Sav_Rate and Sav_rate), how calibration splits between the gain's two parts, how "below its region" is compared, and the per-refit seeds.

Registration

The zip can be rebuilt from the repository and checked against the registered copy:

git archive --format=zip a4064f5 analysis/mssp-benchmark-ratchet pyproject.toml uv.lock | shasum -a 256

The attached Markdown file is the authoritative copy of the plan; OSF's summary field holds a plain-text version (checked through the OSF API before submission).

Power

Computed after registration and committed before any confirmatory estimate, as the plan requires (section 6¹⁰; power.json¹⁴). Minimum detectable effects at 80% power, two-sided alpha 0.05, for 29 vs 277 ACOs:

H3 is underpowered. At a true log-odds difference of 0.5 per 2 SD, the 95% interval excludes zero in only 17% (regional pull), 0% (own-spending component) and 14% (BASIC level) of simulated datasets. If the true effects were about that size, an "undetermined" H3 verdict was the expected outcome. Power was simulated only at 0.5.

Results

In the order the plan fixes: the validation that decides how the own-spending component is used, the primary test, then H2a, H1, H3 and H4. Each verdict quotes the decision rule written before the results (plan section 7¹⁰). Every number comes from the results files in results/¹⁴, each of which records the commit it ran at.

Validation of the benchmark approximation

Passed. For the 101 regular renewers whose 2022 benchmarks CMS publishes (of 111 eligible; 10 lack either the rebuilt or the published benchmark), the rebuilt benchmark had a Spearman correlation of 0.94 with the actual one and a median absolute percentage error of 1.6% (the rule: at least 0.80 and at most 5%). The regression of actual on rebuilt had slope 0.944 and intercept 664, inside the plan's 0.9-1.1 range for the slope, so no calibration was applied. By phase-in step of the regional adjustment: 62 ACOs at the first step (Spearman 0.93, error 2.0%) and 39 at the second (0.96, 0.9%). The step-1 check of the own-spending part against CMS's own benchmark-year spending gave Spearman 0.95 and error 1.4% (101 ACOs), and its error SD sets \(\sigma_v\) = 4.38 percentage points (validation.json¹⁴). The own-spending component is therefore used as registered, not as exploratory.

These checks are on benchmark levels, which vary widely across ACOs. The own-spending component is a difference, and its estimated error (\(\sigma_v\) = 4.38 points) is larger than its spread among stayers (SD 3.3), so at the level of one ACO it is mostly noise. H2a, the total expected gain and the offset argument below rest on it; the H1 savings rates, which CMS reports directly, are the firmer evidence that early renewers were beating their benchmarks.

Who renewed early (H2b, H2a, H1)

Early renewers minus stayers, within cohort and weighted by early renewers per cohort (confirm.json¹⁴). Expected-gain variables are in percentage points of the current benchmark (1 point is 1% of the benchmark, about $113 per patient per year at the median PY2021 benchmark of $11,312); savings rates are in percentage points.

26 early renewers and 241 stayers for the expected-gain variables, 29 and 277 for H1 (see Limitations for the exclusions).

• H2b (rule 1): the regional pull, how far the region's case-mix-adjusted spending sits above the ACO's current benchmark. p < 0.05 and positive: "early renewers' regions spent more relative to their current benchmarks, evidence for the reset motive beyond past savings". The difference is larger than the minimum detectable effect, which was computed for 29 vs 277 ACOs and so is slightly understated for the 26 vs 241 compared. In unweighted means, the early renewers' regions spent 5.0 points above their benchmarks, the stayers' 1.8 points.
• H2a (rule 2, which applies rule 1's wording to the own-spending component): how the ACO's own recent spending compares with its current benchmark (negative when it spends below it). p < 0.05 and negative; in rule 1's terms adapted to this component, their own spending was lower relative to their benchmarks, contrary to the reset motive. Early renewers were beating their benchmarks by more than stayers. H2a is close to H1 restated, so it is not counted as separate evidence.
• H1, descriptive (rule 3). Early renewers had higher savings rates in both years. No motive is attributed to the sign. (The PY2020 results file⁹ stores savings rates as fractions and the PY2021 file as percents; both are shown here in percentage points.)

The tables and the dashboard label these outcomes "supported" (H2b) and "contradicted" (H2a), shorthand for rule 1's two directions.

The two parts of the gain point in opposite directions and offset: the regional component is larger for early renewers, the own-spending component smaller, and the total expected gain does not differ. On average a reset would have lowered both groups' benchmarks, by the approximation: in unweighted means the own-spending and regional parts sum to -5.3 + 3.7 = -1.6 points for early renewers and -3.2 + 2.2 = -1.0 for stayers (exploratory.json¹⁴). That follows from the savings: rebasing to an ACO's own spending lowers the benchmark of any ACO that has been beating it. So the evidence is about where early renewers stood relative to their regions, not about a larger expected gain from resetting. That is the configuration the plan described before any outcome was read (section 1.3¹⁰): an ACO "can be beating its own benchmark and still stand to gain from a reset because its region spends more". As the plan also says (rule 8), no test here can rule the sharing-rate motive in or out.

H3: the decision model

206 BASIC-track ACOs, 24 of them early renewers (3 early renewers and 27 stayers excluded for a missing predictor; confirm.json¹⁴). Coefficients are log-odds differences per 2 SD, or for the BASIC level between levels C-E and A-B. A log-odds difference of 1.34 multiplies the odds of renewing early by about 3.8.

• Verdicts (rule 4). The regional pull and a higher BASIC level are supported; the own-spending component is undetermined. The overall test (the largest absolute core coefficient against its simulated null) gives p = 0.004. Adjustment factors are estimates only.
• What the BASIC level means. The plan's rationale for it is a smaller step to two-sided risk under ENHANCED (section 1.3¹⁰): 16 of the 27 BASIC-track early renewers were already at Level E in 2021, BASIC's highest level, for whom ENHANCED was a short further step (exploratory.json¹⁴). That bears on the sharing-rate motive at least as much as on the reset motive, and the two cannot be separated here.
• Checks. The model met its convergence checks on the first fit (largest R-hat⁵⁵ 1.001, smallest bulk effective sample size 6,661, no divergences). The Student-t fit needed its one registered rerun, then converged, and agrees: regional pull 1.65 (0.53 to 2.87), BASIC level 1.63 (0.53 to 2.85), own-spending component -0.56 (-9.98 to 7.36). In the direct-simulation tests, 0.34-0.43% of the refits did not converge. Before seeing the outcomes, the priors implied a median 17% of ACOs renewing early (95% range 1% to 70%). In posterior predictive checks the observed 24 early renewers fall inside the replicated 95% range (14 to 37), and in the top predicted decile 57% renewed against 45% predicted.
• The own-spending coefficient is the prior. Its interval is the prior's 95% range, because the measurement error (\(\sigma_v\) = 4.38) is larger than the variable's spread among stayers (SD 3.3): the model can learn almost nothing about it. Without the measurement-error model (a robustness check) it is -0.64 (-1.51 to 0.23).
• Read the sizes with care. The power analysis gave H3 a 17% (pull) and 14% (BASIC level) chance of detecting a log-odds difference of 0.5. An estimate that passes a test with power that low is likely to overstate the true effect (a "type M" error, Gelman and Carlin 2014⁵⁶), so the directions are the result, and coefficients near 1.3 (odds ratios near 4) are probably too large.

H4: spending after renewal

29 early renewers and 276 comparison ACOs not yet renewed (one stayer has no spending value in any post-period year; did.json¹⁴). The outcome is the change in log per-capita spending from PY2020-PY2021 to PY2022-PY2024, shown as a percent difference in spending growth.

• Verdict (rule 5). "No detectable difference"; the minimum detectable effect is 3.8 log points (about 3.9%).
• Not read as an effect. The pre-trend test is not evidence against parallel trends (p = 0.58), but the equivalence condition does not hold: the scaled pre-trend bound, 2.41 years times 0.048, is 0.115 log points (a log difference; about 12%), far above the H4 estimate of 0.004. The condition needed an estimate above 0.115 log points, about 12% and three times the minimum detectable effect. With one pre-period, that threshold was a property of the design, fixed at registration: for any estimate near zero, the rule could only end in "not read as an effect". A trend difference of the size the pre-period allows could hide an effect in either direction. So H4 is reported as a difference in spending changes that cannot be told apart from a continuation of a pre-period difference in trends.
• What the data do bound. Under parallel trends, the H4 interval (-4.0% to +2.5%) rules out a drop in spending growth of more than about 4% or a rise of more than about 2.5%. The trend-adjusted interval (-12.0% to +2.7%) is too wide to be informative.

• Attrition. The plan requires it by group and year (section 4.5¹⁰), and it is uneven:

  Group and status	PY2022	PY2023	PY2024
  Early renewers (29): included	29	28	25
  Early renewers: left the program	0	1	1
  Early renewers: renewed again	0	0	3
  Stayers (277): included	276	222	162
  Stayers: left the program	0	46	72
  Stayers: renewed (no longer "not yet renewed")	0	8	42
  Stayers: no spending value	1	1	1

  By PY2024, 115 of 277 stayers (42%) had dropped out of the comparison, against 4 of 29 early renewers (14%). Exits are unlikely to be random: the ACOs that left after PY2021 had the lowest savings rates (see Exploratory). The balanced panel, ACOs with all five years, is the check on this: -1.6% (-5.6% to +1.6%), p = 0.30.

Registered and conventional methods

Every verdict agrees with its conventional counterpart (reported only; it would not have changed a decision; conventional.json¹⁴):

The conventional p-values are mostly larger than the resampling ones, but no verdict changes. The Welch t-tests agree too: H2b +3.2 (1.1 to 5.3), p = 0.004; H2a -2.1 (-3.9 to -0.2), p = 0.028.

Interactive results

Results notebook

The main results tables and figures are also in a marimo⁵⁷ notebook (report/results_notebook.py¹⁴). It reads the results files through the dashboard's data builder and recomputes nothing: provenance (each results file's commit and sha256), validation, the group comparisons, H3, H4, the agreement table, robustness and exploratory results. This post is the authoritative report; some details added after the critical review (such as the H4 attrition table) are only here. The notebook page loads marimo's front end from jsDelivr, so the OSF project also holds a PDF and a Jupyter notebook with its outputs saved, which need no CDN. The files below were exported on 2026-10-27, after the review changed the H4 wording, and replaced the first upload on OSF, which keeps the earlier versions in each file's history:

Robustness

The robustness checks of plan sections 5.5¹⁰ and 5.8 (robustness.json¹⁴ and did.json). The robustness stage ran alongside the confirmatory one, and its results were read only together with the confirmatory results. They do not change the verdicts; the dashboard above shows each one next to its confirmatory estimate.

• H2b keeps its direction in every check. Difference in medians +2.8 (p < 0.001); 2020-01-01 cohort only +3.7 (p = 0.001); unadjusted county spending +6.9 (p = 0.002); Bayesian regression with cohort only +2.1 (0.5 to 3.8) and with track, level and size added +2.2 (0.7 to 3.8). The size depends on the estimator: the regression estimates are below the minimum detectable effect and well below the +3.4 difference in means, which suggests a few large values among early renewers carry part of it, and the unadjusted version is twice as large.
• H2a is weaker by medians. -1.4, p = 0.085; the other checks keep it below zero with p < 0.05 (cohort only p = 0.043, benchmark years 2-3 only p = 0.012, complete cases p = 0.005), though the adjusted regression's interval reaches zero (-2.6 to 0.02).
• H1 is weaker with adjustment. By medians +1.4 points (p = 0.061), and in the adjusted regression +1.2 (-0.1 to 2.6).
• H3's directions hold, with two intervals at zero. The Firth regression gives the same verdicts (pull p = 0.004, BASIC level p = 0.006, own-spending p = 0.18). The pull coefficient's interval reaches zero with the 2020-01-01 cohort only (1.15, -0.02 to 2.39) and with unadjusted county spending (0.75, -0.02 to 1.56); the BASIC level's stays above zero in every check.
• H4 shows no detectable difference in any check. Risk-adjusted outcome +0.3% (p = 0.85), balanced panel -1.6% (p = 0.30), propensity-weighted -0.5% (p = 0.67, largest weight 0.83), 2020-01-01 cohort only -2.9% (p = 0.13), trend-adjusted -4.0% (p = 0.14).

Exploratory

Labeled exploratory in the plan (section 5.6¹⁰; exploratory.json¹⁴): no predictions and no verdicts, and nothing here was used to choose what is reported above.

• Other measures of the regional pull. Weighted by the phase-in weight \(w\), early renewers minus stayers is +1.6 (0.7 to 2.6); the capped regional component, +1.6 (0.9 to 2.2). Both are positive in H3 too.
• Census region⁵⁸ adds nothing to the decision model: Midwest, South and West, each against the Northeast, all have intervals that include zero, and the pull and BASIC-level coefficients barely move (1.43 and 1.32).
• Leavers. The 28 ACOs that left after PY2021 had the lowest savings rates (mean 2.7 points in PY2020, against 6.4 for early renewers and 4.6 for stayers), and a regional pull (for 20 of the 28) similar to early renewers' on average (4.8 against 5.0) but much lower in the median (2.0 against 3.5).
• The two early renewers already in ENHANCED, for whom renewing could not raise the sharing rate, fall between the 26th and 68th percentile of stayers on every variable (savings rate at the 38th and 65th, regional pull at the 26th and 61st).
• Spending by agreement year rises for every group; early renewers from 2020 agreements started lower and stayed lower. As the design noted, benchmark-year status cannot be separated from tenure here, so this is description only.

Implications and open questions

For program design, the results suggest that the early-renewal option let ACOs choose when to reset. The ones that used it were those whose regions spent more relative to their benchmarks and those at higher BASIC levels, for whom ENHANCED was a short step. That is selection into an option the program created; it is not evidence that renewing changed spending. Three questions remain open:

• What did early renewal cost Medicare, through higher benchmarks or larger shared-savings payments? This study does not estimate it.
• Did the 2024 rules change who renews early? The prior savings adjustment and the offset to negative regional adjustments weaken the own-spending penalty that early renewers faced in 2022.
• How much of the choice was the sharing rate rather than the benchmark? In these data the two came with the same decision; a period in which ACOs could move to ENHANCED without resetting would separate them.

Deviations

None affecting any result. All seven stages (validate, power, confirm, robustness, did, conventional, exploratory) ran with code identical to the registered commit a4064f5⁸, and each results file was committed before its estimates were read. Each results file records the commit it ran at, and DEVIATIONS.md (the current log, on the main branch) gives the command that checks it against the registered code.

Two things changed after the last stage ran, and no result was produced with either: the dashboard's reporting script (fixed in dd8c279), which had shown PY2021 savings rates 100 times too large (they were already percents), and the lockfile uv.lock, which took security updates to filelock and virtualenv from the site's main branch. The lockfile is a registered file, so that change is the one entry in DEVIATIONS.md; rerunning the analysis exactly as registered uses the lockfile at a4064f5.

One sentence in the registered plan's background is imprecise, though nothing in the analysis depends on it. Section 1.1¹⁰ says renewing is "the only way for a BASIC-track ACO to enter the ENHANCED track, which shares 75% of savings and losses". ENHANCED shares 75% of savings⁶ but up to 75% of losses, and the bar on moving from BASIC to ENHANCED within an agreement period is in the 2018 rule's preamble²⁹, not the regulation's text (Design).

Limitations

• Bundled motives. In these data, renewing early and moving to ENHANCED are one choice, so no test here can separate the sharing-rate motive from the reset motive. The regional pull has no term for benchmark-year spending, but it is not free of the ACO's own data, and it is not a test that rules the other motive out.
• What the regional pull also measures. Its benchmark \(B\) was set from the ACO's earlier spending, and its regional spending is adjusted with the ACO's own PY2021 risk scores, so it carries a coding-intensity term. A high pull therefore partly measures how low-cost the ACO has been relative to its region, which is also a plausible reason to want ENHANCED's 75% sharing rate. The case-mix adjustment matters and has no published counterpart to validate it: unadjusted, the H2b difference is +6.9 rather than +3.4, and H3's pull interval reaches zero (0.75, -0.02 to 1.56).
• An approximate benchmark. The expected gain is rebuilt from public files; the validation bounds the error but does not remove it, and the regional part has no published counterpart to check against. The own-spending component's estimated error is larger than its spread among stayers (see Validation), so H2a and the total expected gain are weak evidence at the level of one ACO.
• Few early renewers. 29 ACOs, 24 in the decision model: the power analysis shows what can and cannot be detected, and the supported H3 coefficients are probably overestimates (see H3).
• Exclusions above the plan's 10% line. The expected-gain variables (H2a, H2b, H3 and the gain itself) could not be built for 3 of 29 early renewers (10.3%) and 36 of 277 stayers (13.0%); the plan requires reporting this as a limitation when either group passes 10% (section 3¹⁰). In every one of the 39, CMS hides the ACO's count of assigned ESRD (kidney failure) beneficiaries in every county of the 2021 county assignment file³² (it suppresses counts of 1 to 10⁵⁹), so the ESRD part of regional spending has no county weights; for 15 the aged/dual counts are hidden too. These details and the size medians below come from a diagnostic query run after the results, outside the registered pipeline; they are not in the results files. The plan's fill rules cover hidden cells in the results files, not fully hidden county weights, so these ACOs are excluded and counted, as registered. They are smaller ACOs (median 6,702 and 8,578 assigned beneficiaries for excluded early renewers and stayers, against 10,757 and 15,649 for those included), so H2a, H2b and H3 describe the larger ACOs. Separately, 1 early renewer and 3 stayers had hidden values filled by the registered rules.
• One COVID-era pre-period for H4, which is why the equivalence condition, not a pre-trend test, decides whether H4 is read as an effect.
• Do these results apply today? They describe the rules for agreements that began in 2022 (42 CFR 425.601). For agreement periods starting January 1, 2024 or later, benchmarks are set under 42 CFR 425.652⁶⁰, which changes the trade-off studied here. An ACO that saved in the years before its new agreement can receive a prior savings adjustment of up to half its average per-capita savings (capped at 5% of national per-capita spending), and its benchmark gets the higher of that and the regional adjustment (425.652(a)(8)). Negative regional adjustments are partly offset, and the annual update blends in a fixed projected trend, the Accountable Care Prospective Trend (425.652(b)). Under these rules a reset costs an ACO that has been beating its benchmark less, so the incentive to renew early, and which ACOs do so, may well differ.
• Uneven attrition in H4. By PY2024 the comparison group had lost 42% of its ACOs, to exits and later renewals, against 14% of early renewers (see H4), so its later years are increasingly selected.
• Other design threats.
  • Survivorship: the base population needs a PY2021 enrollment record, so ACOs in these cohorts that left before PY2021 are not in it.
  • Anticipation: the decision to renew was made during 2021, which is both the last pre-period year for H4 and the third benchmark year of the new benchmark.
  • Composition: a renewing ACO can change its participant list, so H4's spending can move through who is assigned rather than through behavior; the risk-adjusted version addresses this only in part.
  • Leavers: leaving the program is a third choice, and comparing early renewers only with stayers sets it aside.

Reproduce

Everything needed to rerun the study, check it, or adapt it is public: the code, the CMS data (with an archived copy), and the committed results. The steps below were run, exactly as written, from a fresh clone on macOS on 2026-10-27. The study's README has the same steps with more detail on adapting the analysis.

What you need. git, uv⁵³ (on macOS or Linux: curl -LsSf https://astral.sh/uv/install.sh | sh; for Windows see its installation page), about 1 GB of disk, and an internet connection. uv installs Python 3.12 and the exact package versions pinned in uv.lock the first time it runs. Linux should behave the same as macOS; Windows is untested. Run every command from the repository folder.

1. Get the code (seconds). The repository also holds this website, so clone only the study:

git clone --filter=blob:none --sparse https://github.com/stharrold/stharrold.github.io.git
cd stharrold.github.io
git sparse-checkout set analysis/mssp-benchmark-ratchet

2. Get the data (under a minute, about 40 MB). This downloads the 40 CMS files listed in data-snapshot.tsv⁸, keeps each one only if its sha256 matches, and puts them where the pipeline looks (~/.cache/stharrold-github-io/mssp-snapshot-2026-10-22/). It should end with All 40 files match data-snapshot.tsv.

uv run python analysis/mssp-benchmark-ratchet/fetch_snapshot.py

3. Check the installation (about 8 minutes). The first run also installs the analysis packages. It should end with 81 passed.

uv run --group analysis pytest analysis/mssp-benchmark-ratchet/tests

4. Rerun the analysis. Each stage writes one results file into --out-dir and refuses to start until the files it depends on are there, so run them in this order. Times are for 5 worker processes on an Apple M-series Mac.

OUT=~/mssp-rerun
for stage in validate power confirm robustness did conventional exploratory; do
  uv run --group analysis python analysis/mssp-benchmark-ratchet/ratchet/run.py $stage --out-dir $OUT || break
done

--workers N sets the number of processes (default: half the cores). To rerun a later stage without waiting 17 hours for confirm, first copy the committed files it needs into $OUT, for example cp analysis/mssp-benchmark-ratchet/results/{validation,power,confirm}.json $OUT/ before did.

5. Compare with the published results (seconds). This checks every number in each rerun file against the committed one and prints same, or the values that differ.

uv run python analysis/mssp-benchmark-ratchet/compare_results.py $OUT

What the test run showed. Steps 1-3 worked as written, and the validate stage reproduced the committed file exactly. The power stage reproduced every deterministic number (the standard deviations and minimum detectable effects) but not the simulated ones: H1's simulated power came out 0.776 against the committed 0.804, and H3's detection shares 15% against 17% (pull) and 15% against 14% (BASIC level). The cause is that the analysis table is read without a fixed row order, so the same seeded random draws fall on the ACOs in a different order in each run. Every stage that resamples or simulates (the bootstrap intervals, permutation p-values and MCMC fits) therefore varies slightly between runs. The differences are of the size of the simulation error (about 1.6 points for a detection share from 500 datasets, about 0.002 for a p-value of 0.03 from 10,000 shuffles), far from changing any verdict. compare_results.py lists each differing value so you can judge.

If something goes wrong.

• The fetch script prints changed for a file. CMS has replaced that file; the script takes it from the archived snapshot on OSF instead (osf.io/53aqd). --archive-only takes every file from there.
• A stage stops with a message about a missing file. An earlier stage has not written its results into $OUT yet: run the stages in order, or copy the committed files as in step 4.
• --quick is refused. It uses small test settings and works only on synthetic data, by design.

Adapt it. The README's "Adapt it" table points to the file behind each part of the analysis (who is in each group, each constructed variable, each statistical method, the settings and seeds). It also shows a dry run on synthetic data that takes minutes, and explains how to close the outcome lock in a copy, so that a variant can be pre-registered in turn rather than reported as exploratory.

Cite it. The repository's CITATION.cff gives the registration (DOI 10.17605/OSF.IO/C53W9) as the preferred citation; the results exports and data snapshot have their own DOI, through the OSF project (10.17605/OSF.IO/Z4AN5).

The results notebook runs outside the locked environment, with its tools pinned on the command line. For the other two exports, replace html with ipynb --include-outputs or pdf:

uv run --with marimo==0.25.2 --with matplotlib==3.11.2 --with nbformat==5.11.1 \
    --with 'nbconvert[webpdf]==7.17.2' --with playwright==1.61.0 \
    marimo export html analysis/mssp-benchmark-ratchet/report/results_notebook.py -o results.html

What review caught before registration

Independent reviews of the plan and code, each given only the files, found problems that would have been permanent once registered:

• The first primary test was mechanically tied to past savings. The "regional component" contains the own-spending part with a negative sign; the primary test became the regional pull.
• Raw regional spending is not what CMS compares. The regulation risk-adjusts regional spending to the ACO's case mix³, and CMS restates benchmark years to the latest year's risk (specifications, version 10³⁰, section 4.1.2); both were implemented, with the unadjusted version kept as a robustness check.
• Suppressed values. The results files hide small cells⁵⁹, and the first rule dropped any ACO with one hidden value. A fallback chain (exact recovery from totals, then shown years, then a regional ratio) replaced it, with a complete-case check.
• A permutation test that tests the wrong null in a multi-predictor model, replaced by direct simulation (above).
• Pre-trend testing with one pre-period, replaced by the equivalence condition (above).
• A status line in the plan that still said "DRAFT ... Not yet registered", and would have been frozen into the public record.

Use of AI tools

The author chose the question, hypotheses and decision rules, approved and registered the plan, ran the analysis, and is responsible for the results. Claude Code (Claude Opus 5.5; Anthropic) drafted the analysis code and parts of this post under the author's direction, and separate instances reviewed the plan and code before registration.

References

Every source cited above, numbered in order of first citation; each ↩︎ returns to a place it is cited. General terms link to Wikipedia in the text instead.