The potential difference between the standard GWAS-by-subtraction model within Genomic SEM and
mtCOJO comes down to how beta_xy is estimated. In the standard GWAS-by-subtraction model the beta_xy is essentially a transformation of the LDSC-estimated genetic covariance matrix among the GWAS phenotypes (x and y) into a structural regression coefficient. For
mtCOJO, Zhu et al. write on p. 10 that "we can... estimate beta_xy by GSMR." However, the
software documentation indicates that beta_xy can either be provided by the user (e.g. from a collateral study), estimated with GSMR (i.e. using genome-wide significant variants as instrumental variables for Mendelian Randomization along with removal of HEIDI outliers), or estimated with "genetic correlation analysis" (i.e. the way it is estimated in the standard GWAS-by-subtraction model). My understanding is that the default within mtCOJO is to use GSMR when there are "enough" GWAS significant SNPs, and in the absence of enough GWAS significant SNPs to use genetic correlation analysis. One way to intuit the difference is that the GSMR approach would control for only the portion of shared genetic variance that is causally attributable to x, whereas the GWAS-by-subtraction ("genetic correlation analysis") approach would control for all shared variance.
Apart from potential differences in the the source of beta_xy, my reading of p.10 Zhu et al. is that mtCOJO and the standard GWAS-by-subtraction model are near equivalent. Note that both use the LDSC-derived cross-trait-intercepts to populate elements of the sampling covariance matrix used to compute the SEs of the estimates of SNP effects.
Within Genomic SEM we can fix beta_xy to some value estimated from external data (or simply assumed), or we can expand the GWAS-by-subtraction model within Genomic SEM to include multiple GWAS significant SNPs as instrumental variables for a GSMR type analysis. An advantage of the Genomic SEM approach is that, because the user can specify or modify the model, the assumptions of the model can be changed, relaxed, or tested, and the model can be extended (e.g. to subtract factors rather than individual phenotypes). We do this in great detail in the supplement to Demange.