DeclareDesign Community

Power calculations in non-standard design

Hey team,

I’ve been racking my brain for the last couple of hours trying to understand how to apply DeclareDesign to my specific situation. I was hoping somebody could at least point me in the right direction.

I’m running an experiment where I randomly pair participants up with one another to play a game. Participants can be Republican or Democrat. By definition, you can be paired either with a member of your in-group (so Republican-Republican, Democrat-Democrat), or a member of your out-group (Republican-Democrat, Democrat-Republican).

Some of the DVs are before and after measures that tracks, for example, openness to the political outgroup, whereas other DVs are behavioral games (so post-measures only).

My base model looks like the following:

model_base <- lm_robust(outcome_of-interest ~ politics*outgroup_pairing, 
                 cluster = teamID,
                 se = "stata",
                 data = data)
summary(model_base)

Where politics represents the politics of a given respondent, and outgroup_pairing is a dummy variable that represents 1 if the participant was paired with a member of the outgroup. In other specifications, I add a vector of controls to increase precision. I’m clustering standard errors at the “team” level to resolve inter-team correlations (because the team plays a game together, they will have identical scores on the game, and this may influence outcomes).

We’ve already run a pilot consisting of 70 DD pairings, 80 “mixed pairings”, and 80 RR pairings, and find that the coefficient on “outgroup_pairing” is large and significant, and the interaction is as well – so Republicans are more affected by an outgroup_pairing than Democrats. We now want to run a pre-registered version, and want to do power calculations to have a more accurate estimate of the sample size needed to detect an effect at least as large as the one detected in the pilot.

I couldn’t find anything in the documentation that sheds light on what to do here as this isn’t exactly a pre-post design with a clear-cut treatment condition. Another wrinkle here is that we will be running multiple games with the same pairing philosophy (in-group vs out_group), and will be comparing the effects in one game with the effects of another game. So we want to ask, for example, does a Democrat paired with a Republican in Game 1 show a larger change on the DV than a Democrat paired with a Republican in Game 2?

I’m not expecting a comprehensive answer, but I was hoping I could at least get some direction as to how to do power analysis in this context.

This sounds like an interesting design - here’s how I might set up a very basic system for simulating:

Start by describing the game (I’ll use rock paper scissors):

game <- function(a, b){
  ifelse(a==b, 0, ifelse( (a == 'R' & b == 'S') 
                          | (a == 'P' & b == 'R' )
                          | (a == 'S' & b == 'P'), -1, 1))
  
}

The population is three parties, D, R, and I, with some people in each party. We model the assumption that people of party I always play R. (You can elide the nesting part for performance if need be)

pop <- declare_population(parties=add_level(N=3, p=c("D", "R", "I"), n=rpois(N,20)),
                          people=add_level(N=n, rps=ifelse(p == 'I', 'R', sample(c("R", "P", "S"),N,TRUE)))

Sample 50% of each party in the population:

samp <- declare_sampling(strata=p)

This is the tricky part - generate 200 potential pairing via the under-appreciated link_level function in fabricate. Because it prevents you from linking a level to itself,
we instead make a clone of people (people2), drop the original ID, and then
proceed with the linking.

pair_off <- declare_step(handler = function(data){
  fabricate(list(people=data),
    people2=add_level(N=1), people2=modify_level(people=NULL),
    rounds=link_levels(N=200, by=join(people,people2))) 
})

You might also want to filter the above to elimate games where a person plays themself. All the variables for player 2 will be given a .1 suffix.

The potential outcome is the result of a game between two people.

p_outcome <- declare_potential_outcomes(Y=game(rps, rps.1))

Form a design:

design <- pop + samp + pair_off + p_outcome

You’d then proceed with estimands / estimators like in more typical designs. For example, if you were interested in how often there’s a tie when different combinations of parties play each other, you would see a difference in between I and everyone else, because I’s always play R in this design.

You can also build in repeated measurements by shimming in time variables along the DGP.

Hope that gets you unblocked.

1 Like