Power Analysis Guide
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 5
| Download | |
| Open PDF In Browser | View PDF |
PowerAnalysis.m Guide Roger Strong Harvard University General Notes • PowerAnalysis_tTests.m and PowerAnalysis_ANOVA do most the work, and are called in the example scripts – • Note that PowerAnalysis_ANOVA currently works for within and mixed-factor designs, but not purely between-subjects designs Key Components: – prefs.data: • • • • • – prefs.N_range • – Which comparisons to test for significance. Each row is a comparison, with the condition expected to be higher magnitude listed in the first column, and the condition expected to have lower magnitude in the second column. A study will be classified as “successful” only if all listed comparisons are significant (see examples). prefs.condition_allocation • – How many simulations to use for every participant/trial number combination. 10,000 is a decent estimate and runs pretty quickly, 100,000 is slower but a more stable estimate. prefs.comps • – Significance level to use in simulations (often .05) prefs.nSims • – Range of number of trials per condition to simulate. E.g., 8:4:24 will simulate with 8, 12, 16, 20, and 24 trials per condition prefs.alpha • – Range of number of participants to simulate. E.g., 10:10:50 will simulate with 10, 20, 30, 40, and 50 participants. This is TOTAL number of participants (not number of subjects for condition, although these are equivalent for within-subjects designs) prefs.trial_range • – Name of a CSV file containing your data Top row of CSV contains column headings (used for graphing) Each additional row in CSV file is a trial For t-tests (1 factor designs), you should have 3 columns: Col 1 = sub ID, Col 2 = trial score, Col 3 = condition For ANOVAs (2 factor design), you should have 4 columns: Col 1 = sub ID, Col 2 = trial score, Col 3 = factor 1 names, Col 4 = factor 2 names Used only for between-subjects designs (ignored otherwise). Ratio of how total number of subjects should be divided between conditions during simulations. Should be a value for each condition in data, and values should sum to 1 (100%). For example, [.5, .5] would divide subjects evenly between two conditions. [.25, .5, .25] would use a 1:2:1 ratio for dividing subjects between 3 conditions. prefs.sig_ME1, prefs.sig_ME2, prefs.sig_int • For ANOVAs (2-factor designs), whether significant main effects for either factor or a significant interaction is necessary for a successful study design. Note that for mixed-factor design, the between-subjects factor is always considered the first factor. Example 1: within-subjects t-test Power Analysis Settings Pilot Data - 3 columns 1 header row, then a row for each trial CSV file name as string I decided to simulate N from 10-100 by 10 I decided to simulate trial number per condition from 8-24 by 4 Critical p-value of .05 used in simulation 10,000 sims per N x num_trials combo (sims per cell in output graph) Only comparison I was interested in was condition 1 being larger than condition 2 Data_tTest_Within.csv Used only for between-subjects designs (not used here) Run power analysis using these settings Power Analysis Output Power by N and # of Trials Accuracy Plot of data and pilot simulation parameters. Use this plot to make sure you have specified prefs.comps as intended. 0.85 0.8 1 Across Percentages in heatmap to right indicate percentage of simulated studies where are all parameters are true 2 Within # of Trials Per Condition Pilot Data 0.9 24 0.24 0.45 0.62 0.75 0.83 20 0.21 0.42 0.58 0.71 0.81 0.87 0.92 0.94 0.97 0.98 16 12 0.2 0.9 0.94 0.96 0.98 0.99 0.38 0.54 0.66 0.75 0.83 0.88 0.92 0.95 0.96 0.17 0.33 0.46 0.59 0.68 0.75 0.82 0.87 0.91 0.93 Power Simulation Parameters 8 Successful Study Requies: 1: 2 > 1 (within subjects) 0.13 0.26 0.37 0.48 0.56 0.64 10 20 30 40 50 60 0.7 70 Total # of Subjects 0.77 0.82 0.85 80 90 100 Simulated power for each N x number or trials per condition combo we specified in settings. From this, I know I could achieve about 95% power by running 90 subjects with 16 trials per condition, for example Example 2: within-subjects t-test with multiple comparisons Pilot Data - Power Analysis Settings 3 columns 1 header row, then a row for each trial CSV file name as string I decided to simulate N from 100-300 by 50 I decided to simulate trial number per condition from 8-24 by 4 Critical p-value of .05 used in simulation 10,000 sims per N x num_trials combo (sims per cell in output graph) Studies are only considered a success if all 5 of these comparisons are significant Used only for between-subjects designs (not used here) Data_tTest_Within_Multi.csv Run power analysis using these settings Power Analysis Output Power by N and # of Trials Pilot Data 0.8 0.75 0.7 0.65 1 BB 2 3 4 BW WB WW Power Simulation Parameters Percentages in heatmap to right indicate percentage of simulated studies where are all 5 of these comparisons are true Successful Study Requies: 1: 4 > 1 (within subjects) 2: 4 > 2 (within subjects) 3: 4 > 3 (within subjects) 4: 2 > 1 (within subjects) 5: 3 > 1 (within subjects) # of Trials Per Condition Plot of data and pilot simulation parameters. Use this plot to make sure you have specified prefs.comps as intended. Accuracy 0.85 24 0.46 0.7 0.84 0.92 0.95 20 0.42 0.67 0.82 0.9 0.94 16 0.37 0.62 0.79 0.88 0.93 12 0.29 0.55 0.72 0.83 0.89 8 0.19 0.41 0.6 0.73 0.82 100 150 200 250 300 Total # of Subjects Simulated power for each N x number or trials per condition combo we specified in settings. From this, I know I could achieve about 90% power by running 250 subjects with 20 trials per condition, for example Example 3: mixed-factors ANOVA (1 within-subjects factor & 1 between-subjects factor) Pilot Data - Power Analysis Settings CSV file name as string 4 columns 1 header row, then a row for each trial I decided to simulate N from 50:-200 by 25 I decided to simulate trial number per condition from 8-24 by 4 Critical p-value of .05 used in simulation 10,000 sims per N x num_trials combo (sims per cell in output graph) Need condition 2 > condition 1 for study to be a success I do NOT need main effect of factor one (between-subjects factor for mixed designs) to be significant for successful study I do NOT need main effect of factor two (within-subjects factor for mixed designs) to be significant for successful study Data_ANOVA_Mixed.csv I DO need significant interaction of two factors for successful study Evenly allocate subjects to the two between-subjects factor levels Run power analysis using these settings Power Analysis Output Power by N and # of Trials Pilot Data Horizontal Vertical 0.9 0.85 0.8 1 2 Cross Percentages in heatmap to right indicate percentage of simulated studies where both of these are true 3 4 Return Power Simulation Parameters 32 # of Trials Per Condition Plot of data and pilot simulation parameters. Use this plot to make sure you have specified prefs.comps as intended. Accuracy 0.95 0.72 0.85 0.92 0.96 0.98 0.99 24 0.44 0.65 0.78 0.87 0.92 0.96 0.97 16 0.34 0.52 0.65 0.75 0.84 0.89 0.92 0.2 0.31 0.41 0.51 0.59 0.67 0.73 50 75 100 125 150 175 200 8 Successful Study Requies: 1: Interaction of Movement Type x Motion Direction 2: 2 > 1 (within subjects) 0.51 Total # of Subjects Simulated power for each N x number or trials per condition combo we specified in settings. From this, I know I could achieve about 96% power by running 150 subjects with 32 trials per condition, for example
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf Linearized : No Page Count : 5 PDF Version : 1.4 Producer : Mac OS X 10.12.2 Quartz PDFContext Creator : PowerPoint Create Date : 2019:04:29 20:26:03Z Modify Date : 2019:04:29 20:26:03ZEXIF Metadata provided by EXIF.tools