Power Analysis Guide
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 5
PowerAnalysis.m Guide
Roger Strong
Harvard University
General Notes
• PowerAnalysis_tTests.m and PowerAnalysis_ANOVA do most the work, and are called in the example scripts
– Note that PowerAnalysis_ANOVA currently works for within and mixed-factor designs, but not purely between-subjects designs
• Key Components:
– prefs.data:
• Name of a CSV le containing your data
• Top row of CSV contains column headings (used for graphing)
• Each additional row in CSV le is a trial
• For t-tests (1 factor designs), you should have 3 columns: Col 1 = sub ID, Col 2 = trial score, Col 3 = condition
• For ANOVAs (2 factor design), you should have 4 columns: Col 1 = sub ID, Col 2 = trial score, Col 3 = factor 1 names, Col 4 = factor 2
names
– prefs.N_range
• Range of number of participants to simulate. E.g., 10:10:50 will simulate with 10, 20, 30, 40, and 50 participants. is is TOTAL number
of participants (not number of subjects for condition, although these are equivalent for within-subjects designs)
– prefs.trial_range
• Range of number of trials per condition to simulate. E.g., 8:4:24 will simulate with 8, 12, 16, 20, and 24 trials per condition
– prefs.alpha
• Signicance level to use in simulations (often .05)
– prefs.nSims
• How many simulations to use for every participant/trial number combination. 10,000 is a decent estimate and runs pretty quickly, 100,000
is slower but a more stable estimate.
– prefs.comps
• Which comparisons to test for signicance. Each row is a comparison, with the condition expected to be higher magnitude listed in the rst
column, and the condition expected to have lower magnitude in the second column. A study will be classied as “successful” only if all listed
comparisons are signicant (see examples).
– prefs.condition_allocation
• Used only for between-subjects designs (ignored otherwise). Ratio of how total number of subjects should be divided between conditions
during simulations. Should be a value for each condition in data, and values should sum to 1 (100%). For example, [.5, .5] would divide
subjects evenly between two conditions. [.25, .5, .25] would use a 1:2:1 ratio for dividing subjects between 3 conditions.
– prefs.sig_ME1, prefs.sig_ME2, prefs.sig_int
• For ANOVAs (2-factor designs), whether signicant main effects for either factor or a signicant interaction is necessary for a successful
study design. Note that for mixed-factor design, the between-subjects factor is always considered the rst factor.
Plot of data and pilot simulation
parameters. Use this plot to make
sure you have specied
prefs.comps as intended.
Simulated power for each N x
number or trials per condition
combo we specied in settings. From
this, I know I could achieve about
95% power by running 90 subjects
with 16 trials per condition, for
example
Power by N and # of Trials
0.24
0.21
0.2
0.17
0.13
0.45
0.42
0.38
0.33
0.26
0.62
0.58
0.54
0.46
0.37
0.75
0.71
0.66
0.59
0.48
0.83
0.81
0.75
0.68
0.56
0.9
0.87
0.83
0.75
0.64
0.94
0.92
0.88
0.82
0.7
0.96
0.94
0.92
0.87
0.77
0.98
0.97
0.95
0.91
0.82
0.99
0.98
0.96
0.93
0.85
10 20 30 40 50 60 70 80 90 100
Total # of Subjects
24
20
16
12
8
# of Trials Per Condition
Example 1: within-subjects t-test
Pilot Data Power Analysis Settings
Power Analysis Output
- 3 columns
- 1 header row, then a row for each trial
Data_tTest_Within.csv
CSV le name as string
I decided to simulate N from 10-100 by 10
I decided to simulate trial number per condition from 8-24 by 4
Critical p-value of .05 used in simulation
10,000 sims per N x num_trials combo (sims per cell in output graph)
Only comparison I was interested in was condition
1 being larger than condition 2
Run power analysis using these settings
Used only for between-subjects designs (not used here)
Across Within
0.8
0.85
0.9
Accuracy
Pilot Data
12
Power Simulation Parameters
Successful Study Requies:
1: 2 > 1 (within subjects)
Percentages in heatmap to right
indicate percentage of simulated
studies where are all parameters
are true
BB BW WB WW
0.65
0.7
0.75
0.8
0.85
Accuracy
Pilot Data
1234
Power Simulation Parameters
Successful Study Requies:
1: 4 > 1 (within subjects)
2: 4 > 2 (within subjects)
3: 4 > 3 (within subjects)
4: 2 > 1 (within subjects)
5: 3 > 1 (within subjects)
Plot of data and pilot simulation
parameters. Use this plot to make
sure you have specied
prefs.comps as intended.
Simulated power for each N x
number or trials per condition
combo we specied in settings. From
this, I know I could achieve about
90% power by running 250 subjects
with 20 trials per condition, for
example
Pilot Data Power Analysis Settings
Power Analysis Output
- 3 columns
- 1 header row, then a row for each trial
Data_tTest_Within_Multi.csv
CSV le name as string
I decided to simulate N from 100-300 by 50
I decided to simulate trial number per condition from 8-24 by 4
Critical p-value of .05 used in simulation
10,000 sims per N x num_trials combo (sims per cell in output graph)
Studies are only considered a success if all 5 of these
comparisons are signicant
Run power analysis using these settings
Used only for between-subjects designs (not used here)
Percentages in heatmap to right
indicate percentage of simulated
studies where are all 5 of these
comparisons are true
Example 2: within-subjects t-test with multiple comparisons
Power by N and # of Trials
0.46
0.42
0.37
0.29
0.19
0.7
0.67
0.62
0.55
0.41
0.84
0.82
0.79
0.72
0.6
0.92
0.9
0.88
0.83
0.73
0.95
0.94
0.93
0.89
0.82
100 150 200 250 300
Total # of Subjects
24
20
16
12
8
# of Trials Per Condition
Power by N and # of Trials
0.51
0.44
0.34
0.2
0.72
0.65
0.52
0.31
0.85
0.78
0.65
0.41
0.92
0.87
0.75
0.51
0.96
0.92
0.84
0.59
0.98
0.96
0.89
0.67
0.99
0.97
0.92
0.73
50 75 100 125 150 175 200
Total # of Subjects
32
24
16
8
# of Trials Per Condition
Cross Return
0.8
0.85
0.9
0.95
Accuracy
Pilot Data
12 34
Horizontal
Vertical
Power Simulation Parameters
Successful Study Requies:
1: Interaction of Movement Type x Motion Direction
2: 2 > 1 (within subjects)
Power Analysis Settings
Plot of data and pilot simulation
parameters. Use this plot to make
sure you have specied
prefs.comps as intended.
Simulated power for each N x
number or trials per condition
combo we specied in settings. From
this, I know I could achieve about
96% power by running 150 subjects
with 32 trials per condition, for
example
Pilot Data
Power Analysis Output
- 4 columns
- 1 header row, then a row for each trial
Data_ANOVA_Mixed.csv
CSV le name as string
I decided to simulate N from 50:-200 by 25
I decided to simulate trial number per condition from 8-24 by 4
Critical p-value of .05 used in simulation
10,000 sims per N x num_trials combo (sims per cell in output graph)
Need condition 2 > condition 1 for study to be a success
Run power analysis using these settings
Evenly allocate subjects to the two between-subjects factor
levels
Percentages in heatmap to right
indicate percentage of simulated
studies where both of these are
true
Example 3: mixed-factors ANOVA
(1 within-subjects factor & 1 between-subjects factor)
I do NOT need main effect of factor one (between-subjects factor
for mixed designs) to be signicant for successful study
I do NOT need main effect of factor two (within-subjects factor
for mixed designs) to be signicant for successful study
I DO need signicant interaction of two factors for successful study
