How to create a suitable statistical test for my repeated measures problem with unequal sample sizes?

dear community,

i have the following problem: I want to test differences of specific repeated measures during a pre and a post test:

my datasets has 10 subjects, each subjects did 2 tasks (called pre and post). during each task, I did repeated measurements of 5 different parameters (each set of 5 parameters were taken at the same time) over time. In addition, each pre and post measurement can be divided into two segments A and B. Now, i want to analyze the differences between pre and post on four different levels: on each of the parameters

taking parameter, I want to compare

1. subject wise and segement wise, e.g. Subj1, pre, A to Subj1, post, A
2. subjects wise only, both segments together, e.g. Subj1, pre, A+B to Subj1, post, A+B
3. segment-wise, but all subjects, e.g. Subj1to10, pre, A to Subj1to10, post, B
4. all subjects, both segements: Subj1to10, pre, A+B to Subj1to10, post, A+B

note that segments A/B and pre and post measurements do not have equal sample size, the samples are not independent and they may not be normally distributed.

How to tell, if there is a statistical significant difference between pre and post test at each of the four levels, and a suitable post-hoc analysis to tell, how this difference might be directed (probably looking at the median?)

And of course i need to compensate for mutiple comparison due to the 4 levels and the 5 parameters is measured at the same time?

best regards

Jonas

Answer 1

[Jaynik]

Hi Jonas,

To analyze the difference between pre and post tests in MATLAB on the four different levels described, you can use the ranova function to calculate repeated measures analysis of variance along with post-hoc tests to account for multiple comparisons. Following steps can be taken:

• Use the fitrm function to create repeated measures model from the dataset. Specify the WithinDesign parameter in fitrm. Apply ranova.
• After finding the significant effects with ranova, use multcompare for the post-hoc analysis to determine where the difference lies. Pairwise comparisons can be performed between different levels of the within-subject factor like pre and post for each subject or segment. Use the parameters like bonferroniin, tukey-kramer, dunn-sidak for protection in pairwise comparisons and adjust the significance level parameter alpha to account for the risk of Type 1 errors due to multiple comparisons.
• As the data might be not be normally distributed, you can apply mathematical transformations like log, sqrt, inverse transform (1 ./ data) etc. You can also consider using non-parametric tests like signrank (Wilcoxon Signed Rank Test), ranksum (Wilcoxon Rank Sum Test), kruskalwallis (Kruskal-Wallis Test), etc. Basically, it needs to be ensured that the model accounts for the repeated measures design properly.

Here’s a simplified example of how you might set up your analysis in MATLAB:

% Assuming 'data' is a table with your measurements and factors
% 'Param1', 'Param2', ..., 'Param5' are your repeated measures

% Fit the repeated measures model
rm = fitrm(data, 'Param1-Param5 ~ Subject*Segment', 'WithinDesign', withinDesign);
ranovatbl = ranova(rm);
postHocResults = multcompare(rm, 'Timepoint', 'By', 'Subject');

% Apply multiple comparison correction if needed
c = multcompare(rm, 'CriticalValueType ', 'bonferroni'); % example using the Bonferroni method

Remember to tailor the model specification to your dataset’s structure and the specific hypotheses you are testing.

You can refer the following documentation to read more about each functions:

• fitrm: https://www.mathworks.com/help/releases/R2022a/stats/fitrm.html
• ranova: https://www.mathworks.com/help/releases/R2022a/stats/repeatedmeasuresmodel.ranova.html
• multcompare: https://www.mathworks.com/help/releases/R2022a/stats/multcompare.html
• signrank: https://www.mathworks.com/help/releases/R2022a/stats/signrank.html
• ranksum: https://www.mathworks.com/help/releases/R2022a/stats/ranksum.html
• kruskalwallis: https://www.mathworks.com/help/releases/R2022a/stats/kruskalwallis.html

I hope this helps!

[Jonas]

thanks for your response.

at the meantime, i had a statistical journey through Matlab's functions: at first, i found that since 2023b there is also a manova function, which is an n-way, multi repsonses function. then i noticed, that that function is unfortunately not able use random effects to compensate for repeated measures.

now i stick with the fitlme, which uses fixed effects and random effects to fit a statistical model. there, i can use random effects to mode the correlation introduced by measurements from the same subjects and for the correlation introduced by measurments that are close in time to each other. i fit one model for each parameter (response variable) and compensate for the multiple comparisons using bonferroni.

since i could not meet the requirements of anovan, it am using fitlme now

signrank, ranksum and kruskalwallis are all not suited since their either need paired sample values, assume independence of samples or cannot support the n-way design

best regards

[Jonas]

another addition: fitlme itself works fine, but the post hoc analysis is a pain, no multcompare or similar to simply throw it on the output. there are only contrast (and one emm package on file exchange) to perform post hoc, but it still is a pain if you have mutiple factors and want to test for significant interactions on multiple levels, e.g. up to three way.

going back to anovan is of course a possibility, but it also has problems with the combionation of nested and/or continuous factors. not necessarily the call of anovan itself, but the post hoc test will not work e.g. for continuous factors