It sounds like you have 5 copies of the same data set, where each copy of the data set has different estimated factor scores (plausible values of the latent trait).
You could think of this as a missing data problem. You have not observed the latent traits, and your IRT analysis has "imputed" them multiple (5) times. You could therefore save the 5 data sets in a list, and pass them as data to the runMI() function in semTools, which will pool results by averaging parameter estimates across the 5 data sets, as well as pool the average SE per data set with the variability of estimates across 5 data sets. I would expect this to yield nominal Type I error rates, although it would be less efficient than running a single latent regression model that includes the measurement model for the latent traits and the regression model among those latent traits (and any other variables in your model).
To do the missing-data trick, make sure you have the latest development versions of lavaan and semTools:
install.packages("lavaan", repos="http://www.da.ugent.be", type="source")
# install.packages("devtools") # if necessary, to install semTools below
devtools::install_github("simsem/semTools/semTools")
Terrence D. Jorgensen
Postdoctoral Researcher, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam