A information to avoiding the frequent pitfalls of occasion research
Occasion research are helpful instruments within the context of causal inference. They’re utilized in quasi-experimental conditions. In these conditions, the therapy is just not randomly assigned. Thus, in distinction to randomized experiments (i.e., A/B assessments), one can’t depend on a easy comparability of the means between teams to make causal inferences. In a majority of these conditions, occasion research are very helpful.
Occasion research are additionally continuously used to see if there are any pre-treatment variations between the handled and nontreated teams as a method to pretest parallel traits, a crucial assumption of a well-liked causal inference technique referred to as difference-in-differences (DiD).
Nonetheless, latest literature illustrates quite a lot of pitfalls in occasion research. If ignored, these pitfalls can have important penalties when utilizing occasion research for causal inference or as a pretest for parallel traits.
On this article, I’ll focus on these pitfalls and suggestions on methods to keep away from them. I’ll deal with the functions within the context of panel information the place I observe items over time. I’ll use a toy instance as an instance the pitfalls and suggestions. Yow will discover the total code used to simulate and analyze the info right here. On this article, I restrict using code to probably the most essential elements to keep away from cluttering.
An Illustrative Instance
Occasion research are generally used to analyze the impression of an occasion reminiscent of a brand new regulation in a rustic. A latest instance of such an occasion is the implementation of lockdowns because of the pandemic. Within the case of the lockdowns, many companies received affected as a result of individuals began spending extra time at residence. For instance, a music streaming platform might need to know whether or not individuals’s music consumption patterns have modified as a consequence of lockdowns in order that they will handle these adjustments and serve their clients higher.
A researcher working for this platform can examine whether or not the quantity of music consumed has modified after the lockdown. The researcher might use the international locations that by no means imposed a lockdown or imposed a lockdown later as management teams. An occasion examine could be applicable on this scenario. Assume for this text that the international locations that impose a lockdown keep so till the top of our commentary interval and the implementation of the lockdown is binary (i.e., ignore that the strictness of the lockdown can differ).
Occasion Research Specification
I’ll deal with occasion research within the form of:
Yᵢₜ is the result of curiosity. αᵢ is the unit-fixed results and it controls for time-constant unit traits. γₜ is the time-fixed results and it controls for time traits or seasonality. l is the time relative to the therapy and it signifies what number of intervals it has been for the reason that therapy at a given time t. For instance, l = -1 signifies that it’s one interval earlier than the therapy, l = 2 signifies that it’s two intervals after the therapy. Dˡᵢₜ is the therapy dummy for the relative time interval l at time t for unit i. Mainly, we embody each the leads and lags of the therapy. ϵᵢₜ is the random error.
The coefficient of curiosity βₗ signifies the common therapy impact in a given relative time interval l. Within the commentary interval, there are T intervals, thus, the intervals vary from 0 to T-1. The items get handled at completely different intervals. Every group of items which might be handled on the identical time composes a therapy cohort. Any such occasion examine is a difference-in-differences (DiD) design through which items obtain the therapy at completely different time limits (Borusyak et al. 2021)
Illustrative instance continued:
Consistent with our illustrative instance, I simulate a panel dataset. On this dataset, there are 10,000 clients (or items) and 5 intervals (from interval 0 to 4). I pattern unit- and time-fixed results at random for these items and intervals, respectively. General, we’ve got 50,000 (10,000 items x 5 intervals) observations on the customer-period stage. The result of curiosity is the music consumption measured in hours.
I randomly assign the shoppers to three completely different international locations. One among these international locations imposed a lockdown in interval 2, the opposite in interval 3, and one by no means imposed a lockdown. Thus, clients from these completely different international locations are handled at completely different occasions. To make it straightforward to observe, I’ll consult with the shoppers by their therapy cohorts relying on after they have been handled: cohort interval 2 and cohort interval 3 for patrons handled in intervals 2 and three, respectively. One of many cohorts is rarely handled and, thus, I consult with them as cohort interval 99 for the convenience of coding.
Within the simulation, after these clients are randomly assigned to certainly one of these cohorts, I create the therapy dummy variable deal with which equals 1 if cohort_period >= interval, 0 in any other case. deal with signifies whether or not a unit is handled in a given interval. Subsequent, I create a dynamic therapy impact that grows in every handled interval (e.g., 1 hour within the interval the place therapy occurs and a couple of hours within the interval after that). Remedy results are zero for pre-treatment intervals.
I calculate the result of curiosity hrs_listenedbecause the sum of a relentless that I randomly selected (80), unit- and time-fixed results, the therapy impact, and error (random noise) for every unit and interval. By development, the therapy (lockdowns) has a rising constructive impression on music consumption.
I skip a few of the setup and simulation elements of the code to keep away from cluttering however you’ll find the total code right here.
Within the following picture, I present a snapshot of the info. unit refers to clients, cohort_period refers to when a unit was handled. hrs_listened is the dependent variable and it measures the music consumption in a given interval in hours for a given buyer.
rm(listing = ls())
library(information.desk)
library(fastDummies)
library(tidyverse)
library(ggthemes)
library(fixest)
library(kableExtra)information <- make_data(...)
kable(head(information[, ..select_cols]), 'easy')
Within the following picture, I illustrate the traits within the common music listening by cohort and interval. I additionally mark when the international locations have imposed lockdowns for the primary time. You possibly can see that there appears to be a constructive impression of the lockdowns for each the earlier- and later-treated international locations in comparison with the shoppers from the untreated cohort.
# Graph common music listening by cohort and interval
avg_dv_period <- information[, .(mean_hrs_listened = mean(hrs_listened)), by = c('cohort_period','period')]
ggplot(avg_dv_period, aes(fill=issue(cohort_period), y=mean_hrs_listened, x=interval)) +
geom_bar(place="dodge", stat="identification") + coord_cartesian(ylim=c(79,85))+
labs(x = "Interval", y = "Hours", title = 'Common music listening (hours)',
caption = 'Cohort 2 is the early handled, cohort 3 is the late handled and cohort 99 is the by no means handled group.') +
theme(legend.place = 'backside',
axis.title = element_text(measurement = 14),
axis.textual content = element_text(measurement = 12)) + scale_fill_manual(values=cbPalette) +
geom_vline(xintercept = 1.5, coloration = '#999999', lty = 5)+
geom_vline(xintercept = 2.5, coloration = '#E69F00', lty = 5) +
geom_text(label = 'Cohort interval 2 is handled',aes(1.4,83), coloration = '#999999', angle = 90)+
geom_text(label = 'Cohort interval 3 is handled',aes(2.4,83), coloration = '#E69F00', angle = 90) +
guides(fill=guide_legend(title="Remedy cohort interval"))
Since this dataset is simulated, I do know the true therapy impact of lockdowns for every cohort and every interval. Within the following graph, I current the true therapy impact of the lockdowns.
Within the first interval after the therapy (relative interval 1), each cohorts enhance their listening by 1 hour. Within the second interval relative to the therapy, the therapy impact is 2 hours for each cohorts. For the relative interval 3, we see that the therapy impact is 3 hours.
One factor to note right here is that the therapy impact is homogenous throughout cohorts over relative intervals (e.g., 1 hrs in relative interval 1; 2 hrs in relative interval 2). Later, we’ll see what occurs if this isn’t the case.
# Graph the true therapy results
avg_treat_period <- information[treat == 1, .(mean_treat_effect = mean(tau_cum)), by = c('cohort_period','period')]
ggplot(avg_treat_period, aes(fill=issue(cohort_period), y=mean_treat_effect, x=interval)) +
geom_bar(place="dodge", stat="identification") +
labs(x = "Interval", y = "Hours", title = 'True therapy impact (hrs)',
caption = 'Cohort 2 is the early handled, cohort 3 is the late handled and cohort 99 is the by no means handled group.') +
theme(legend.place = 'backside',
axis.title = element_text(measurement = 14),
axis.textual content = element_text(measurement = 12)) + scale_fill_manual(values=cbPalette) +
guides(fill=guide_legend(title="Remedy cohort interval"))
Now, we do an occasion examine by regressing the hrs_listened on relative interval dummies. The relative interval is the distinction between interval and cohort_period. The destructive relative intervals point out the intervals earlier than the therapy and the constructive ones point out the intervals after the therapy. We use unit fixed-effects (αᵢ) and interval fixed-effects (γₜ) for all of the occasion examine regressions.
Within the following desk, I report the outcomes of this occasion examine. Unsurprisingly, there aren’t any results detected pre-treatment. Publish-treatment results are exactly and appropriately estimated as 1, 2, and three hours. So every part works up to now! Let’s see conditions the place issues don’t work as effectively…
# Create relative time dummies to make use of within the regression
information <- information %>%
# make relative 12 months indicator
mutate(rel_period = ifelse(cohort_period == 99,99,interval - cohort_period))
abstract(information$rel_period)information <- information %>%
dummy_cols(select_columns = "rel_period")
rel_per_dummies <- colnames(information)[grepl('rel_period_', colnames(data))]
# Change title w/ minuses to deal with them extra simply
rel_per_dummies_new<-gsub('-','min', rel_per_dummies)
setnames(information, rel_per_dummies, rel_per_dummies_new)
# Occasion examine
covs <- setdiff(rel_per_dummies_new, c('rel_period_99','rel_period_min1'))
covs_collapse <- paste0(covs, collapse='+')
system <- as.system(paste0('hrs_listened ~ ',covs_collapse))
mannequin <- feols(system,
information = information, panel.id = "unit",
fixef = c("unit", "interval"))
abstract(mannequin)
All the pieces labored effectively up to now however listed below are the highest 4 issues to watch out of to keep away from the potential pitfalls when utilizing the occasion examine strategy:
1. No anticipation assumption
Many functions of occasion research within the literature impose a no-anticipation assumption. No anticipation assumption implies that handled items don’t change their conduct in expectation of the therapy earlier than the therapy. When the no-anticipation assumption holds, one can use the interval earlier than the occasion as (certainly one of) the reference interval(s) and evaluate different intervals to this era.
Nonetheless, no anticipation assumption won’t maintain in some circumstances, e.g., when the therapy is introduced to the panel earlier than the therapy is imposed and the items can reply to the announcement by adjusting their conduct. On this case, one wants to decide on the reference intervals rigorously to keep away from bias. You probably have an thought of when the topics begin to anticipate the therapy and alter their conduct you need to use that interval because the de facto starting of the therapy and use the interval(s) earlier than that because the reference interval (Borusyak et al. 2021).
For instance, in the event you suspect that the topics change their conduct in l = -1 (one interval earlier than the therapy) as a result of they anticipate the therapy you need to use l = -2 (two intervals earlier than the therapy) as your reference interval. You are able to do this by dropping the Dˡᵢₜ the place l = -2 from the equation as a substitute of dropping the dummy for l = -2. This fashion you utilize the l = -2 interval because the reference interval. To test whether or not your hunch on items altering their conduct in l = -1 is true, you may test if the estimated therapy impact in l = -1 is statistically important.
Illustrative instance continued:
Going again to our illustrative instance, lockdowns are normally introduced a bit earlier than the imposition of the lockdown, which could have an effect on the items’ pre-treatment conduct. For instance, individuals would possibly already begin working from residence as soon as the lockdown is introduced however not but imposed.
In consequence, individuals can change their music-listening conduct even earlier than the precise implementation of the lockdown. If the lockdown is introduced 1 interval earlier than the precise implementation one can use the relative interval = -2 because the reference interval by dropping the dummy for the relative interval -1 from the specification.
Consistent with this instance, I copy and modify the unique information to introduce some anticipation results. I introduce a 0.5 hrs enhance within the hours listened to all items in relative interval -1. I name this new dataset with anticipation data_anticip.
The following graph exhibits the common music listening time over relative intervals. It’s straightforward to note that the listening time already begins to choose up within the relative interval -1 in comparison with the relative intervals -2 and -3. Ignoring this important change within the listening time can create deceptive outcomes.
# Summarize the hours listened over relative interval (excluding the untreated cohort)
avg_dep_anticip <- data_anticip[rel_period != 99, .(mean_hrs_listened = mean(hrs_listened)), (rel_period)]
setorder(avg_dep_anticip, 'rel_period')rel_periods <- type(distinctive(avg_dep_anticip$rel_period))
ggplot(avg_dep_anticip, aes(y=mean_hrs_listened, x=rel_period)) +
geom_bar(place="dodge", stat="identification", fill = 'deepskyblue') + coord_cartesian(ylim=c(79,85))+
labs(x = "Relative interval", y = "Hours", title = 'Common music listening over relative time interval',
caption = 'Just for the handled items') +
theme(legend.place = 'backside',
legend.title = element_blank(),
axis.title = element_text(measurement = 14),
axis.textual content = element_text(measurement = 12)) + scale_x_continuous(breaks = min(rel_periods):max(rel_periods))
Now, let’s do an occasion examine as we did earlier than by regressing the hours listened on the relative time interval dummies. Remember the fact that the one factor I modified is the impact within the relative interval -1 and the remainder of the info is precisely the identical as earlier than.
You possibly can see within the following desk that the pre-treatment results are destructive and important regardless that there aren’t any actual therapy results in these intervals. The reason being that we use the relative interval -1 because the reference interval and this messes up all of the impact estimations. What we have to do is to make use of a interval the place there isn’t any anticipation because the reference interval.
system <- as.system(paste0('hrs_listened ~ ',covs_collapse))
mannequin <- feols(system,
information = data_anticip, panel.id = "unit",
fixef = c("unit", "interval"))
abstract(mannequin)
Within the following desk, I report the occasion examine outcomes from the brand new regression the place I exploit relative interval -2 because the reference interval. Now, we’ve got the correct estimates! There isn’t a impact detected within the relative interval -3, although an impact is appropriately detected for the relative interval -1. Moreover, the impact sizes for the post-treatment intervals are actually appropriately estimated.
# Use launch interval -2 because the reference interval as a substitute
covs_anticip <- setdiff(c(covs,'rel_period_min1'),'rel_period_min2')
covs_anticip_collapse <- paste0(covs_anticip,collapse = '+')system <- as.system(paste0('hrs_listened ~ ',covs_anticip_collapse))
mannequin <- feols(system,
information = data_anticip, panel.id = "unit",
fixef = c("unit", "interval"))
abstract(mannequin)
2. Assumption of homogenous therapy results throughout cohorts
Within the equation proven earlier than, the therapy impact can solely differ by the relative time interval. The implicit assumption right here is that these therapy results are homogenous throughout therapy cohorts. Nonetheless, if this implicit assumption is fallacious the estimated therapy results will be considerably completely different than the precise therapy impact inflicting bias (Borusyak et al. 2021). An instance scenario may very well be the place earlier cohorts profit extra from the therapy in comparison with the later handled teams. Which means that the therapy results throughout cohorts differ.
The only answer to handle this difficulty is to permit for heterogeneity. To permit for the therapy impact heterogeneity between cohorts, one can estimate relative time and cohort-specific therapy results, as seen within the following specification. Within the following specification, c stands for the therapy cohort. Right here, every part is identical because the earlier specification besides that the therapy results are going to be estimated for every relative time & treatment-cohort mixture with the estimator for βₗ,c. Dᵢᶜ stands for the therapy cohort dummy for a given unit i.
Illustrative instance continued:
Within the lockdown instance, it may be that the impact of lockdowns is completely different throughout handled international locations for various causes (e.g., possibly in one of many international locations, individuals are extra prone to adjust to the brand new regulation). Thus, one ought to estimate the nation and relative time-specific therapy results as a substitute of merely estimating the relative time-specific therapy impact.
Within the unique simulated dataset, I introduce cohort heterogeneity in therapy results throughout intervals and name this new dataset data_hetero. The therapy impact for cohort interval 2 is 1.5 occasions greater than the cohort interval 3 throughout all handled intervals as illustrated within the subsequent graph.
Now, as we did earlier than, let’s run an occasion examine for the data_hetero. The outcomes of this occasion examine are reported within the following desk. Regardless that there aren’t any therapy or anticipation results within the pre-treatment intervals, the occasion examine detects statistically important results! It is because we don’t account for the heterogeneity throughout cohorts.
# Occasion examine
system <- as.system(paste0('hrs_listened ~ ',covs_collapse))
mannequin <- feols(system,
information = data_hetero, panel.id = "unit",
fixef = c("unit", "interval"))
abstract(mannequin)
Let’s account for the heterogeneity in therapy results throughout cohorts by operating the hours listened on cohort-specific relative interval dummies. Within the following desk, I report the outcomes of this occasion examine. On this desk, the therapy impact estimates for every cohort and relative interval are reported. By permitting the therapy results to differ per cohort, we account for the heterogeneity and because of this, we’ve got the correct estimates! No results are detected for the pre-treatment as they need to be.
# Create dummies for the cohort-period
information <- data_hetero %>%
dummy_cols(select_columns = "cohort_period")
cohort_dummies <- c('cohort_period_2','cohort_period_3')
# Create interactions between relative interval and cohort dummies
work together <- as.information.desk(expand_grid(cohort_dummies, covs))
work together[, interaction := paste0(cohort_dummies,':',covs)]
interact_covs <- work together$interplay
interact_covs_collapse <- paste0(interact_covs,collapse = '+')# Run the occasion examine
system <- as.system(paste0('hrs_listened ~ ',interact_covs_collapse))
mannequin <- feols(system,
information = data_hetero, panel.id = "unit",
fixef = c("unit", "interval"))
abstract(mannequin)
3. Underneath-identification within the absolutely dynamic specification within the absence of a never-treated group
In a completely dynamic occasion examine specification the place one consists of all leads and lags (normally solely relative time -1 is dropped to keep away from good multicollinearity) of the therapy, the therapy impact coefficients are usually not recognized within the absence of a non-treated group. The explanation for that is that the dynamic causal results can’t be distinguished from the mix of unit and time results (Borusyak et al. 2021). The sensible answer for that is to drop one other pre-treatment dummy (i.e., one other one of many lead therapy dummies) to keep away from the under-identification drawback.
Illustrative instance continued:
Think about that we would not have information on any untreated international locations. Thus, we solely have the handled international locations in our pattern. We will nonetheless do an occasion examine using the variation within the therapy timing. On this case, nevertheless, we’ve got to make use of not just one however a minimum of two reference intervals to keep away from under-identification. One can do that by dropping the interval proper earlier than the therapy and probably the most destructive relative interval dummies from the specification.
Within the simulated dataset, I drop the observations from the untreated cohort and name this new dataset data_under_id. Now, we’ve got solely handled cohorts in our pattern. The remainder is identical as the unique simulated dataset. Thus, we’ve got to make use of a minimum of two reference intervals by dropping the dummies for any of the pre-treatment relative interval dummies. I select to exclude the dummies for the relative intervals -1 and -3. I report the outcomes from this occasion examine beneath. As you may see now, I’ve just one relative interval estimated within the mannequin. The estimates are appropriate, nice!
4. Utilizing occasion research as a pretest for parallel traits assumption
It’s a frequent technique to make use of occasion research as a pretest for the parallel traits assumption (PTA), a vital assumption of the difference-in-differences (DiD) strategy. PTA states that within the absence of the therapy, the handled and untreated items would observe parallel traits by way of the result of curiosity. Occasion research are used to see whether or not the handled group behaves otherwise than the non-treated group earlier than the therapy happens. It’s thought that if a statistically important distinction is just not detected between the handled and untreated teams the PTA is prone to maintain.
Nonetheless, Roth (2022) exhibits that this strategy will be problematic. One difficulty is that a majority of these pretests have decrease statistical energy. This makes it more durable to detect differing traits. One other difficulty is that when you’ve got excessive statistical energy you would possibly detect differing pre-treatment results (pre-trends) regardless that they don’t seem to be so crucial.
Roth (2022) recommends a few approaches to handle this drawback:
- Don’t rely solely on the statistical significance of the pretest coefficients and take the statistical energy of the pretest under consideration. If the facility is low the occasion examine received’t be very informative with regard to the existence of a pre-tend. You probably have excessive statistical energy the outcomes of the pretest would possibly nonetheless be deceptive as you would possibly discover a statistically important pre-trend that’s not so vital.
- Take into account approaches that keep away from pretesting altogether, e.g., use financial data in a given context to decide on the correct PTA reminiscent of a conditional PTA. One other means is to make use of the later handled group because the management group in the event you suppose the handled and untreated teams observe completely different traits and are usually not as comparable. Please, see Callaway & Sant’Anna’s 2021 paper for potential methods to loosen up the PTA.
Illustrative instance continued:
Going again to the unique instance the place we’ve got three international locations, let’s say that we need to carry out a DiD evaluation and we need to discover assist indicating that the PTA holds on this context. This could imply that if the handled international locations have been to not be handled the music consumption would transfer in parallel to the music consumption within the untreated nation.
We consider using an excellent examine as a method to pretest the PTA as a result of there isn’t any method to check the PTA instantly. First, we have to take the statistical energy of the check under consideration. Roth (2021) gives some instruments to do that. Though that is out of the scope of this text, I can say that on this simulated dataset we’ve got a comparatively excessive statistical energy. As a result of the random noise is low and we’ve got a comparatively large pattern measurement with not that many coefficients to estimate. Nonetheless, it may be good to run state of affairs analyses to see how large of a pre-treatment impact one can appropriately detect.
Secondly, whatever the statistical significance standing of the pre-treatment estimates take the precise context under consideration. Do I count on the handled international locations to observe the identical traits because the untreated nation? In my simulated information, I do know this for positive as I decide what the info seems like. Nonetheless, in the actual world, it’s unlikely that this might maintain unconditionally. Thus, I’d think about using a conditional PTA by conditioning the PTA on numerous covariates that make international locations extra comparable to one another.
Conclusion
Occasion research are highly effective instruments. Nonetheless, one ought to concentrate on their potential pitfalls. On this article, I explored probably the most generally encountered pitfalls and supplied suggestions on methods to handle these utilizing a simulated dataset. I mentioned the problems referring to no anticipation assumption, heterogeneity of the therapy results throughout cohorts, under-identification within the absence of an untreated cohort, and utilizing occasion research as a pretest for PTA.
