Typically, you can't estimate a model with paths from an external predictor to a factor and all of its indicators, as it is underientified. For k indicators of a factor you have k observed SNP-indicator associations, but you would be trying to estimate k+1 associations in the model you describe. With the measurement model fixed to the estimates from the unconditional model, as is typical, you can identify the model, but the interpretation of the direct effects is somewhat nuanced. For an explication of this idea with an external GWAS phenotype, rather than SNP, see:
de la Fuente, J., Londoño-Correa, D., & Tucker-Drob, E. M. (2025). Distinguishing specific from broad genetic associations between external correlates and common factors. Bioinformatics, 41, btaf568. Link
For an example of how we've taken such an approach with SNPs (e.g. those in the APOE region for cognition), see the supplement to:
de la Fuente, J., Davies, J., Grotzinger, A. D., Tucker-Drob, E. M., & Deary. (2020). A general dimension of genetic sharing across diverse cognitive traits inferred from molecular data. Nature Human Behaviour. [de la Fuente & Davies contributed equally to this work; Tucker-Drob & Deary jointly directed this work] Link
We have not attempted to use these SNP-indicator residuals to estimate polygenic indices, and I'd be cautious with resepct to both interpretation and power of doing so.