|Explanations & examples:
Compare two estimates with each other by calculating either their difference or their ratio (quotient) and get the 95 % confidence interval of the difference or the quotient. Some estimates require logistic transformation before they can be compared and tested, for ex. odds ratios (OR), risk ratios (RR), incidence rate ratios (IRR), ... etc. Use section 1 if the estimates require log transformation, otherwise section 2. In section 1 we look at the difference between the log transformed values of the estimates and its confidence interval instead of the difference between the values themselves. Alternatively to this difference of the log transformed estimates we can also look at the quotient (ratio) between the estimates and the confidence interval of the quotient. If the 95 % confidence interval of the difference between the two log transformed estimates includes the number 0, then there is no statistically significant difference between the two estimates on a five percent significance level. This will be confirmed by the z-test which will have (in this case) a p-value above 0.05. If, on the other hand, 0 is not included in the 95 % confidence interval of this difference, then the two estimates are different from each other on a five percent significance level and the p-value will be below 0.05, meaning that the null hypothesis H0 will be rejected in this scenario (the null hypothesis claims that the two estimates are the same). If the quotient is used instead of the difference between the log transformed estimates, then the number 1 has to be included in the 95 % confidence interval of the quotient in order to accept H0, namely that the two estimates are the same; if 1 is included in the interval, the p-value will be greater than 0.05. If not, p < 0.05 and H0 is rejected and the estimates are significantly different.
Notice that we are dealing with estimates of the true values in these cases, since we don't have data from the entire population to deal with. For more info about the formulas used in the tests; please see the page medical statistics formulas.
Example:We want to compare and test the two OR (odds ratio) values 0.98 and 1.06 after a logistic regression where the effect of age on the outcome (knee osteoarthritis) was investigated. The OR value for men is 0.98 with a 95 % confidence interval of [0.90 : 1.05], meaning that for every year a man gets older he has 0.98 times lower odds of the outcome. Lower, since the OR value is under 0. The OR value for women is 1.06 with a 95 % confidence interval of [1.02 : 1.11], meaning that for every year a woman gets older she has a 1.06 times higher odds of the outcome. It is first noticed, that only the OR value of 1.06 is significant, since the number 1.00 is not included in its confidence interval. Neither of the two estimates is included in the 95 % confidence interval of the other estimate. If this had been the case the OR values would not have been significantly different and the p-value in the test would have been above 0.05. Also; there is a little overlap between the confidence intervals (one ends at 1.05 and the other starts at 1.02). If there had been no overlap between the intervals, the OR values would have been significantly different and the p-value in the test would have been under 0.05. So in this case with some overlap we can't either reject or not reject the null hypothesis just by looking at the confidence intervals alone.
In general: By comparing the estimates using the confidence intervals alone (without testing) there are three possible scenarios:
1) There is no overlap between the two 95 % confidence intervals of the estimates.
2) One of the estimates is included in the confidence interval of the other estimate (or they are both included in each other's intervals).
3) None of the estimates is included in the other estimate's interval, but there is an overlap between the intervals.
In case 1) with no overlap H0 can be rejected, because the p-value in the test will be below 0.05. In case 2) H0 cannot be rejected on a five percent significance level and the p-value in the z-test will be above 0.05. In case 3) it isn't possible to conclude anything from the confidence intervals alone. Here either the test must be made and the p-value will be above or below 0.05 or the confidence interval of the quotient can be used to see if in includes 1. If not, p < 0.05 and H0 is rejected.
The 95 % confidence interval of the difference between the log transformed estimates is [-0.1664 : 0.0094] and thus includes the number 0. Conversely, the 95 % confidence interval of the ratio between the two estimates is [0.8467 : 1.0095] and this interval includes 1. So the two OR values are not significantly different from each other on a five percent significance level. This is confirmed by the p-value in the test, which is 0.0802 and thus more than 0.05. So we do not reject the null hypothesis H0 in this case, that stated that the two OR estimates are the same.
The weighted estimate of the two OR values combined into one is 1.0409. Note that it is only meaningful to calculate and interpret the weighted estimate if the estimates are not significantly different from each other, like in this case. The weighted OR of 1.0409 is the common OR value for the connection between age and knee osteoarthritis having adjusted for sex. The weighted estimate is significant, since 1 is not included in its 95 % confidence interval of [1.0031 : 1.0802], confirmed by the p-value of 0.0339, which is below 0.05, so we reject the null hypothesis that WE = 1.