KMO values always at around 0.50

[andreassoteriadesmoj]

Using numerous versions of my own data sets as well as data from various R packages, I've noticed that CAF barely leaves the 0.50 area (plus/minus, give or take). Therefore, I did a little experiment: calculate KMO for a residual matrix that has very small values, i.e. I was trying to reproduce a situation where the original correlation matrix and the model-implied correlation matrix are really close to each other. I was hoping to get a KMO near 0 (hence a CAF near 1), but got one near 0.5 instead:

res_mat <- matrix(0, 5, 5)
diag(res_mat) <- 1
res_mat [upper.tri(res_mat)] <- runif(10, 0.001, 0.01)
res_mat [lower.tri(res_mat)] <- res_mat [upper.tri(res_mat)]
res_mat <- (res_mat + t(res_mat)) / 2 # Ensures elements above and below diagonal have same precision
EFAtools::KMO(res_mat)$KMO

I think that the problem originates in the line where the diagonal of the residual matrix is set to be all ones (diag(res_mat)):

EFAtools/R/helper.R

Line 518 in f361798

diag(delta_hat) <- 1

This step is necessary so that KMO considers res_mat a correlation matrix and skips step R <- stats::cor(x, use = use, method = cor_method). However, unless I'm mistaken, the diagonal of res_mat should be zero given that we take the difference between the original correlation matrix and the model-implied correlation matrix. Let's supply a res_mat with small numbers off the diagonal and zeroes on the diagonal and calculate KMO/CAF step by step and see what happens:

res_mat.csv

res_mat <- read.csv('res_mat.csv')
res_mat <- as.matrix(res_mat)

R <- res_mat
R_i <- try(solve(R), silent = TRUE)
diag(R_i) # The diag elements will not always be all positive
S <- diag(diag(R_i)^(-1/2))
Q <- S %*% R_i %*% S
diag(Q) <- 0
diag(R) <- 0
sumQ2 <- sum(Q^2)
sumR2 <- sum(R^2)
KMO <- sumR2/(sumR2 + sumQ2)
caf <- 1 - KMO
caf
# 0.999479

As expected, the CAF is very high given that the two matrices differ by very small amounts. Looks like KMO() will need to be modified to consider as a special case an input x that is not a correlation matrix or a raw dataset for which to calculate correlations. I wouldn't know how to handle a situation where diag(R_i) has negative elements though. As x isn't a correlation matrix in this case, smoothing it with R <- psych::cor.smooth(R) (where R <- x) isn't relevant. Would one set all diagonal elements to a fixed number for example? I don't know...

Note that this issue also relates to #72.

[mdsteiner]

Yes, you are of course correct, Lorenzo-Seva, Timmerman and Kiers (2011) write "Now, we propose to express the common part accounted for by a common factor model with q common factors as 1 minus the KMO index of the estimated residual matrix..." which of course has a 0 diagonal. I don't know what I was thinking there. I think, adding a skip_checks argument to KMO might make sense here, then we can call it with the residual matrix. I'll report back here, when I pushed the fix, and of course credit you. Thanks for catching this!

[mdsteiner]

I've now looked into it a bit further. The issue is that the definition provided by Lorenzo-Seva, Timmerman and Kiers (2011) does not actually work. They define CAF as 1 - KMO(residual_matrix), but this residual matrix has a zero diagonal as they also subtract the unique variances. But such a matrix cannot be used to calculate the KMO. They solve this by setting the diagonal to 1s in the EFA.MRFA R package and this is also why we followed their implementation. But this leads to the lower CAF values, which approach .5 and thus the CAF is not really between 0 and 1. I need to think about whether an alternative would make sense or whether to leave it like this, but maybe add a note explaining this.

[andreassoteriadesmoj]

I'll report back here, when I pushed the fix, and of course credit you. Thanks for catching this!

No problem, and thanks!

I've now looked into it a bit further. The issue is that the definition provided by Lorenzo-Seva, Timmerman and Kiers (2011) does not actually work. They define CAF as 1 - KMO(residual_matrix), but this residual matrix has a zero diagonal as they also subtract the unique variances.

That's my point exactly: because res_mat is the difference between the original correlation matrix and the model-implied correlation matrix, its diagonal is 0.

such a matrix cannot be used to calculate the KMO. They solve this by setting the diagonal to 1s in the EFA.MRFA R package

You're right. For the record:

https://github.com/cran/EFA.MRFA/blob/f435b9cc40877defb020d46e207d586da561d6e3/R/hullEFA.R#L160

@UrbanoLorenzoSeva, @dnavarrogonzalez, any chance you could help us with the current issue please? In summary:

1. The calculation of CAF won't work unless the diagonal of the residual matrix diagonal is set to 1. Is there a theoretical justification for setting the diagonal to 1?
2. CAF barely exceeds 0.50 (when the differences in the residual matrix are really small). The expectation would be that CAF would have been closer to 1 when the difference between the original correlation matrix and the model-implied correlation matrix is very small.

[mdsteiner]

I think this is an issue that cannot be solved with the definition of CAF as 1 - KMO(residuals) because when the residuals become very small, partial correlations will approach approx to the negatives of the correlations and thus KMO will be approx 1/2.

Maybe the uniquenesses could be taken as diagonal entries and the matrix then renormalized, but we would still have the same issue with the KMO approaching 1/2. Not sure, whether a conceptually similar definition to the common parts accounted for would be something like 1 - sum(residuals[upper.tri(residuals)]^2) / sum(correlation_matrix[upper.tri(correlation_matrix)]^2), which would be the GFI.

In sum, if my understanding is correct, if CAF = 1 - KMO(residuals), it will just not be between 0 and 1 but a CAF of about .5 be interpreted as indication of close fit, so if we adapt the interpretation it's fine.

[andreassoteriadesmoj]

Seems like there are two separate issues here: one issue is to accurately define the range of possible values for CAF using mathematics, which is beyond the scope of EFAtools; the other issue is to ensure that EFAtools users know of this issue and not expect values anywhere near 1 (more like, expect at most 0.5).

Regarding the latter issue, I agree with you that as long as this is worded in the package documentation then it should be fine. Hopefully, the creators of the CAF will at some point address the former issue with a note to the original paper.

[dnavarrogonzalez]

Hello,

Thanks for flagging this and for the thorough analysis. I was not involved in the original proposal of the Hull method or the CAF index, but I can try to add some context from the implementation side. The diag = 1 decision was indeed a pragmatic implementation choice to allow KMO to process the residual matrix, but I don't have a theoretical justification for it beyond that. I can reach Lorenzo-Seva and let him know the issue, and see if he can weigh in on the methodological question.

[andreassoteriadesmoj]

Hello,

Thanks for flagging this and for the thorough analysis. I was not involved in the original proposal of the Hull method or the CAF index, but I can try to add some context from the implementation side. The diag = 1 decision was indeed a pragmatic implementation choice to allow KMO to process the residual matrix, but I don't have a theoretical justification for it beyond that. I can reach Lorenzo-Seva and let him know the issue, and see if he can weigh in on the methodological question.

@dnavarrogonzalez, many thanks for the quick response and for offering to reach out to Lorenzo-Seva. That would be very useful, thanks!

[andreassoteriadesmoj]

@dnavarrogonzalez, have you had the chance to reach out to Lorenzo-Seva?

[andreassoteriadesmoj]

@mdsteiner, given we've been waiting for an answer for more than a month now, it's probably a good time to start thinking of how to best handle this without input from the developer of the index. I agree with you that a change in the wording to reflect the fact that you wouldn't normally get values anywhere near 1 (anywhere above 0.5 for that matter) would be appropriate. I propose to be to as detailed as possible in the documentation, because, currently, there is no way of knowing when a CAF value is high or low relative to the maximum. For example, if 0.5 is the max, is, say, 0.47, a high or a low value? What if values tend to concentrate near 0.5 in general? It could mislead users.

I think it should be made clear that CAF is an experimental index that requires further understanding (and potentially, further development too). I’m happy to put together some wording if you agree.

[mdsteiner]

@andreassoteriadesmoj from the Lorenzo-Seva paper I would assume that close fit is around .5 and worse models will have lower CAF-values; otherwise their hull method would not work to identify the number of factors to retain as no convex hull would be spanned (but e.g., just a flat line, if all were about .5). There is no cutoff indicating close fit, or departure from it, but this is not an issue of the ~.5 upper bound but was also true for the assumed [0, 1] interval. I've added a note in the EFA() documentation telling users about the upper boundary issue. But if you think further detail is needed, you can make a suggestion.

[andreassoteriadesmoj]

@mdsteiner, I didn't know that you've added a note in the EFA() documentation, as the commit hasn't been linked to this issue. For the record, it's commit f586cf6. Think this is okay for now, as I'll need more time to have a think about this from the formula- not documentation-perspective, in the future.