Clarification on RF_cons Values in SetMethods Package

[Steffen]

Hi everyone,

I am currently working on robustness checks for fsQCA. Therefore I am using the SetMethods R package. I have created a minimum replicable example (MRE) and observed that the RF_cons value exceeds 1. As far as I understand from the paper of Oana and Schneider, the robustness fit values should not exceed 1 because they represent the overlap of the test set and the initial solution.

Here's the code I used:

```R

library('SetMethods')

df <- data.frame(

  C1 = c(1.0, 1.0, 1.0, 1.0, 0.3, 0.0, 0.3, 0.0, 0.3, 0.7),

  C2 = c(1.0, 1.0, 0.3, 1.0, 0.3, 0.3, 0.0, 1.0, 1.0, 0.3),

  OUT = c(1.0, 1.0, 1.0, 1.0, 0.2, 0.0, 0.8, 0.6, 0.9, 0.8)

)

conds <- c("C1", "C2")

IS <- minimize(data = df, outcome = "OUT", conditions = conds, incl.cut = 0.85, n.cut = 1, details = TRUE, show.cases = TRUE)

TS1 <- minimize(data = df, outcome = "OUT", conditions = conds, incl.cut = 0.85, n.cut = 3, details = TRUE, show.cases = TRUE)

TS <- list(TS1)

CF <- rob.corefit(test_sol = TS, initial_sol = IS, outcome = "OUT")

rob.fit(test_sol = TS, initial_sol = IS, outcome = "OUT")

```

I have reviewed the documentation of SetMethods but could not determine the reason for these values exceeding 1. Is this an error on my part, or is there another explanation for these results?

Has anyone else experienced something similar?

Thanks,

Steffen

[Ingo Rohlfing]

I think the following may go on here: The result is correct because it is the ratio of the consistency of the initial solution and the robust core, which is identical to the test set with the conjunction C1*C2. According to the article introducing the functions, it does not happen that the initial solution has a higher consistency score than the robust core, accounting for the maximum value of 1 for RFcons. The reproducible example suggests that this is not correct. My hunch is that it has to do with the fuzzy sets. The membership of a case in a condition is always equal to or larger than a membership in a conjunction including this condition (like C1 and C1*C2). When one plots the membership scores in an XY-plot, this means the membership scores in the condition are always in the same spot or to the right of the membership score of the conjunction. A case either stays put or moves to the right. This means that inconsistent cases for C1*C2 do not become less inconsistent; some cases that are consistent for the conjunction may become inconsistent for the condition because they cross the diagonal; some cases that are consistent for the conjunction remain consistent for the condition, but they move closer to the diagonal. Depending on how all cases move, the latter type of change may cause an increase in the consistency score when the set becomes larger. For the example below, the consistency for C1*C2 is 0.5. For C1, the consistency score is 0.79. Nothing changes for case B, but case A moves from the Y-axis to the diagonal and makes a larger contribution to the numerator of the consistency formula (also to the denominator, but the change in the contribution to the numerator is larger in relative terms).

    C1   C2   Y

A 0.8  0     0.8

B 0.6, 0.7 0.3

Regards

Ingo

[Adrian Dușa]

I was about to suggest submitting an issue to the GitHub repository of package SetMethods, at:

https://github.com/jmedzihorsky/SetMethods

But then I noticed that repo has not been updated in eight years, which suggests the package SetMethods resides solely on the developers' computers.

The next best thing would be to email the developers, particularly the maintainer Nena Oana:

https://cran.r-project.org/web/packages/SetMethods/index.html

The CRAN version is now more than one year old, which further suggests a development slowdown. This is explicable with her move from CEU to EUI, but I know Nena is still very active.

Perhaps we could return to this topic, after managing to get a response.

All the best,

Adrian

--
You received this message because you are subscribed to the Google Groups "QCA with R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qcawithr+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qcawithr/1bb17dcb-6715-4624-a9ee-ee404e370607n%40googlegroups.com.

[Steffen]

Thanks to both of you for your quick help. This helps me understand the result, and I have already reached out to the package developer.

Would the best approach still be to use the package to report the robustness results and use RF_cons to indicate the "positive" deviation of consistency between the test set and the robust core? Otherwise, I haven't found a structured robustness check for fsQCA, and most papers applying fsQCA just state that robust checks were performed without giving details on how.

Best Regards
Steffen

[Ingo Rohlfing]

I am more confident now that the function works as it should. The conceptual argument that a superset cannot have a higher consistency score than a subset is not always correct for fuzzy sets and it is not correct here. I have written short code that creates a plot illustrating what have I written before:  https://gist.github.com/ingorohlfing/812541a355eb70d4cbbf1abfaaaac72d

When you look at the data, it is the last observation that explains why the consistency of C1 is higher than the consistency of C1C2. This is the observation that remains consistent and adds more to the consistency formula of C1 than for C1C2.

The question is whether the RFcons measure is important for your robustness test. It is the ratio of two consistency scores, which may not be that relevant for you. It may be enough to present the consistency score of the robust core.

Regards

Ingo