Fundamentally, the Astrometry.net code spends a lot of its time searching for matches of quadrangles in your image to quadrangles in the index files. These are searches in random places in the shape space, so it jumps around the index files. We do this using the Linux mmap() call, so if you have enough physical memory, then each time a block is requested from the index file, it gets cached in memory. Eventually, the whole file is mapped into memory, and then subsequent searches should be much faster. One thing to check in your setup would be, once you have solved a field, how long does re-solving it take?
If you can reduce the amount of sky (in RA,Dec and in scale) to search, then that memory requirement will be less. For tiny fields like 6 arcmin x 4 arcmin, if you know roughly the RA,Dec of your fields, consider the "5200-series (LITE)" files (https://portal.nersc.gov/project/cosmo/temp/dstn/index-5200/LITE/
), where the sky is split into 48 pieces. If you provide solve-field with --ra, --dec, and --radius, then it will only search the sky tiles that overlap. Similarly, if you know the image scale (--scale-low, --scale-high, --scale-units), that will restrict the number of index files that have to be searched.
You might also want to check how much time is spent detecting stars, vs searching. Turn on solve-field -v to get some details on what it's doing. The "--downsample 2" option may help. And if you don't need them, "--uniformize 0" and "--no-remove-lines" will skip some steps.