Discrepancies in coefficients

[rebekkasl]

Hi,

I have noticed that I get discrepancies in correlation coefficients if the compare the output from the partial correlation in cor_test to the output from pcor.test in ppcor. This happens for Spearman's and Kendall's , but not for Pearson's. Is this correct?

Example:

a <- c(1, 1, 1, 2, 2, 2, 3, 4, 5, 5)
b <- c(1, 1, 2, 2, 1, 2, 2, 2, 2, 2)
c <- c(3, 2, 3, 3, 1, 4, 4, 6, 4, 1)

data <- data.frame(a, b, c)

pearson_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "pearson")
pearson_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "pearson")
print(pearson_cor)
print(pearson_pcor)

spearman_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "spearman")
spearman_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "spearman")
print(spearman_cor)
print(spearman_pcor)

kendall_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "kendall")
kendall_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "kendall")
print(kendall_cor)
print(kendall_pcor)

[IndrajeetPatil]

Reprex:

library(correlation)
a <- c(1, 1, 1, 2, 2, 2, 3, 4, 5, 5)
b <- c(1, 1, 2, 2, 1, 2, 2, 2, 2, 2)
c <- c(3, 2, 3, 3, 1, 4, 4, 6, 4, 1)

data <- data.frame(a, b, c)

pearson_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "pearson")
pearson_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "pearson")
print(pearson_cor)
#> Parameter1 | Parameter2 |    r |        95% CI | t(8) |     p
#> -------------------------------------------------------------
#> a          |          b | 0.53 | [-0.15, 0.87] | 1.78 | 0.112
#> 
#> Observations: 10
print(pearson_pcor)
#>    estimate   p.value statistic  n gp  Method
#> 1 0.5332874 0.1392674  1.667914 10  1 pearson

spearman_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "spearman")
spearman_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "spearman")
print(spearman_cor)
#> Parameter1 | Parameter2 |  rho |        95% CI |      S |     p
#> ---------------------------------------------------------------
#> a          |          b | 0.33 | [-0.40, 0.80] | 111.18 | 0.358
#> 
#> Observations: 10
print(spearman_pcor)
#>    estimate   p.value statistic  n gp   Method
#> 1 0.5358889 0.1369827  1.679318 10  1 spearman

kendall_pcor <- ppcor::pcor.test(data$a, data$b, data$c, "kendall")
kendall_cor <- correlation::cor_test(data = data, "a", "b", partial = TRUE, "kendall")
print(kendall_cor)
#> Parameter1 | Parameter2 |  tau |        95% CI |    z |     p
#> -------------------------------------------------------------
#> a          |          b | 0.30 | [-0.21, 0.69] | 1.19 | 0.236
#> 
#> Observations: 10
print(kendall_pcor)
#>   estimate    p.value statistic  n gp  Method
#> 1 0.478047 0.07277577  1.794234 10  1 kendall

^{Created on 2021-06-28 by the reprex package (v2.0.0)}

[DominiqueMakowski]

I suspect this could be due to a different treatment of the covariate (whether it is rank-transformed or not). @bwiernik what should be the right approach here?

[bwiernik]

That's probably the reason.

Internally, ppcor is doing a fairly simple procedure: (1) compute the covariance matrix using the desired method, (2) inverting that matrix, then using cov2cor() to get the partial correlations. correlation, by contrast is (1) using datawizard::adjust() to residualize the variables, then (2) computing the correlations between the residualized variables using the desired method. This is kinda-sorta using Pearson correlations for the covariates.

The ppcor approach would generally be what is expected, I think. It is of course reasonable to model different variables in a matrix using different transformations, but for the one-shot give me the whole matrix approach, using one method consistently would be better.

[DominiqueMakowski]

Yes, but ppcor's straightforward appraoch hard to make work with how correlation works internally... what if we ranktransformed the covariates before datawizard::adjusting them? though it feels a bit hacky

[bwiernik]

Let me take a look at how we might be able to vectorize some of the more esoteric types of correlations we support