The convention for xPert is to include the factor of 1/n! in perturbation series. This is stated explicitly close to the beginning of the accompanying journal article (
https://arxiv.org/abs/0807.0824), but I see that it's easy to miss this if you're just looking at xPertDoc.nb. In xPertDoc.nb you will find documentation of the command Perturbed[expr, order] which gives you the series up to your desired order, including the factors of epsilon and 1/n!. I would recommend using Perturbed[] rather than writing out the sum yourself. Attached is an example notebook showing the correct matrix inverse property (and here is a screenshot of the contents of that notebook).