Astrometry multiprocessing

[Toni Šarić]

Have anyone tried running astrometry.net on multiprocessor?

I have tried, it partially works, but seems to me that when writing results to a fits file it gets corrupted.

For example, I created four processes for 12 files and and it solved 11 of them. I run it again, and it solves 12, next time it solves 10.

You can see one run here:

Reading input file 1 of 1: "1607744536_20201212_034216_STAR-Crab_EXP-1.0e+06_GAIN-6.0_RA-85.0_DEC-19.8_LED-20_parked-0_parkingPos-0_inMotion-1_tracking-1.fits"...
Extracting sources...
simplexy: found 163 sources.
Solving...
Reading file "none"...
Only searching for solutions within 6.3 degrees of RA,Dec (85.3139,17.4551)
Field 1 did not solve (index index-4215.fits, field objects 1-10).
Field 1 did not solve (index index-4214.fits, field objects 1-10).
Field 1 did not solve (index index-4213.fits, field objects 1-10).
  log-odds ratio 222.056 (2.74028e+96), 45 match, 0 conflict, 114 distractors, 176 index.
  RA,Dec = (85.3587,17.4774), pixel scale 18.7346 arcsec/pix.
  Hit/miss:   Hit/miss: -+++-+-+----+-+--+-+-+-+-++-+++-----+--+-+++-++----+---+--+------+------++-----+--+---+--------+++--
Field 1: solved with index index-4212.fits.
Field 1 solved: writing to file /home/toni/Documents/Astrometry/Test_1.6/B/1607744533_20201212_034213_STAR-Crab_EXP-1.0e+06_GAIN-6.0_RA-85.0_DEC-19.8_LED-20_parked-0_parkingPos-0_inMotion-1_tracking-1.solved to indicate this.
Field: /home/toni/Documents/Astrometry/Test_1.6/B/1607744536_20201212_034216_STAR-Crab_EXP-1.0e+06_GAIN-6.0_RA-85.0_DEC-19.8_LED-20_parked-0_parkingPos-0_inMotion-1_tracking-1.fits
Did not solve (or no WCS file was written).

It looks like it is messing with these two files(213 and 216) although it should have. It is supposed to solve 216 as started. But why it didn't?

I attached the full output as txt file.

[Dustin Lang]

Are you running several copies of solve-field in the background, and having them all write their log outputs to the same stdout?  I'm thinking you just have two processes whose log messages are interleaved.

Solve-field runs to different filenames are completely independent.  That's how the web service runs.

cheers,

--dustin

--
You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/f4819174-afbf-49c2-94c8-9336d5b601dan%40googlegroups.com.

[Nathaniel Paust]

I use GNU Parallel to run 100s of images at the same time.  Typically, I make a batch file with “/usr/local/bin/solve-field …..all the options….” For each image I’m solving.  Then that list of tasks get fed into parallel and it farms them out to the processors on my computer.

The screen gets a little messy with 6 simultaneous logs, but the individual files solve exactly as expected.

Dr. Nathaniel Paust

Associate Professor of Astronomy

Hall of Science 372

Whitman College

345 Boyer Ave

Walla Walla, WA 99362

Office: (509) 527-5219

Cell:    (410) 926-6842

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/CA%2BFzHBCS81ehW-52UHgkNTFV2f-MFiBW%3DzqXa-avTqw0BxV8gA%40mail.gmail.com.

[Toni Šarić]

Yes, I am running several copies of solve-field at the same time in the background, but regardless of writing output in the same logfile or don't write output at all, some files don't get solved.
I guess you are right about interleaving in log messages, but I am wondering why doesn't it solve all the files(and file that doesn't get solved is not always the same)
My code looks like this:

# import all files
files = glob.glob(src_path + "*.fits")
files_list = [files[0::3], files[1::3], files[2::3]]

num_processors = 3  # Create a pool of processors
p = Pool(processes = num_processors)  # get them to work in parallel
output = pool.map(run2, [f for f in files_list])


run2(files):
    for file in files:
        #run astrometry.net software on that file

        result = subprocess.run(solve-field, stdout=subprocess.PIPE)

BR,

Toni

[Dustin Lang]

How long do your images take to solve / what are your solve-field command-line parameters?  Given command-line parameters, the solver's behavior is deterministic, EXCEPT for the CPU time limit.  If you are not hitting the time limits, then the same set of images should always solve regardless of parallelization.

cheers,

--dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/013e4593-4c5e-4543-93db-0156ef598969n%40googlegroups.com.

[Toni Šarić]

My images takes around 4 seconds to solve.

These are the parameters:

command = [
            "solve-field", "--sigma", "200", "--overwrite", "--scale-units",
            "degwidth",
#             "--scale-low", str(9.9), "--scale-high",str(10.1),
            "--ra", str(SG_head_fix_radec.ra.value),
            "--dec",str(SG_head_fix_radec.dec.value),
            "--radius", "6.3",
            str(ffit), "--dir", os.path.dirname(ffit),
            "--rdls", "none", "--axy", "none",
            "--match", "none", "--index-xyls", "none",
            "--no-plots",
            "--new-fits", "none",
            "--tweak-order", "4",
            "--pixel-error", "0.2",
            "--crpix-x", str(x_cen), "--crpix-y",str(y_cen),
            "--continue",
            "--parity", "pos",
#             "--verify",os.path.splitext(ffit)[0]+".wcs"
        ]

[Dustin Lang]

Hmmm, so I don't know what to say.

Can you try running a simple shell script, like

solve-field ... image1.fits &

solve-field ... image2.fits &

solve-field ... image3.fits &

wait

and check whether you always get the same set of fields solving?

thanks,

--dustin

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/e84f66e4-63dd-4316-aa89-803a23ecdc80n%40googlegroups.com.

[Toni Šarić]

Hi Dustin,

the problem is solved.
I have found your previous conversation named "Cannot running multiple calls to solve-field in parallel" with Mischa Schirmer about parameter --axy none.
I also made the same "mistake".
In my defence, official documentation states the name "none" should be put for every file that doesn't need to be created, therefore I put it for axy.

So when I changed from --axy none to --temp-axy parallel running works as expected.

Thank you for your help.

BR

Toni

[Dustin Lang]

AHhhh, sorry I missed that!

--dstn

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/52c29ebf-8d66-42bf-8cf3-4f33f04b6a4an%40googlegroups.com.

[Toni Šarić]

No, I am sorry, I should have done more exhaustive search for previous conversations.

BR,

Toni

[Frank Kahle]

Hi, maybe I've missed something but after reading the post I've been rechecking some of my "faulty" images and my (parallel) pipe.

I'm calling solve-field from within python via

def solveField(img):
    cmd = "solve-field --use-sextractor --overwrite --no-remove-lines --uniformize 0 "+img+" >"+img+".log"
    os.system(cmd)

and solve the images in parallel using python multiprocessing.

So in my workflow I don't omit any solve-field-output via the none argument - as far as I understand the cmd line options;-)

However I've noticed that there are some images with a wrong wcsheader attached to the image (unfortunately I'm not allowed to upload the orininal fits image ... too large).

Imscale and orientation are both far off (code version astrometry.net 0.80; see attachment for complete image header), however the solve time indicated by the header is well below the cpu limit.

Todays refit (not working on several images in parallel; code version astrometry.net 0.82) got well solved.

Do I have to attribute this to (misconfigured) parallel processing of images?

Regards,

Frank

[tano...@gmail.com]

Hi to everyone
I recall this conversation because I'm trying to run  astrometry in python multiprocess.  

The issue is that, even if I say to use just 5 threads, the server use all the available CPU.

Here you can find the pseudo-code that I'm using and a screenshot of the CPU status

# ) List of png file in folder
files = glob.glob(png_directory + "/*.png")
# ) Split files in list for each thread
files_list = [files[0::5], files[1::5], files[2::5], files[3::5], files[4::5]]
files_list = list(filter(None,files_list))
#)  Create a pool of processors
num_processors = 5
# ) Get them to work in parallel

p = Pool(processes = num_processors)

# ) 1st loop solving procedure
print('Start First Astro-ref loop')
_ = p.map(run1, [f for f in files_list])


def run1(files):
    tweak_order = '5'

    # ) Create out and solved folder
    out_dir = os.path.dirname(files[0]) + '/Out'
    solv_dir = os.path.dirname(files[0]) + '/Solved'
    tmp_dir = '/tmp/astroref'
    if os.path.isdir(out_dir) == False:
        os.makedirs(out_dir,exist_ok=True)
    if os.path.isdir(solv_dir) == False:
        os.makedirs(solv_dir,exist_ok=True)
    if os.path.isdir(tmp_dir) == False:
        os.makedirs(tmp_dir,exist_ok=True)

    # ) Loop on all files with multiproc
    for file in files:
        if file.endswith('.png'):
            solved_file = solv_dir + '/' + os.path.basename(file).replace('.png','.fits')

        #run astrometry.net software on that file

        proc = subprocess.Popen('solve-field --dir ' + out_dir + ' --no-plots --new-fits ' + solved_file + ' --temp-dir ' + tmp_dir + ' --use-source-extractor --match none --tag-all --crpix-center --overwrite --downsample 1 --tweak-order ' + tweak_order + ' --scale-units degwidth --scale-low 0.1 --scale-high 180.0 --objs 1000 ' + file, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        # ) Waiting for ending process
        while proc.poll() == None:
            time.sleep(0.5)


    return

Any suggestion?

Cheers

Gaetano

[Dustin Lang]

Source Extractor may be using multiple cores in each of your 5 solve-field calls. 

Cheers,

Dustin 

--

You received this message because you are subscribed to the Google Groups "astrometry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astrometry+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/b5c2a9a0-1cad-45a7-8d26-230302edaa90n%40googlegroups.com.

[Gaetano Zarcone]

Oh, thanks Dustin.

So if I'd like to use just 5 threads, i should not use sextractor?

Instead, what about if I want to force to use sextractor but using 5 threads? 

Cheers

Gaetano 

[Dustin Lang]

SE has a configuration option to set the number of threads it uses - NTHREADS.

[Gaetano Zarcone]

Ok

Thank you Dustin

[Alex Whittemore]

I'll start my own thread but I'm researching a similar question myself and I just want to say, @Dustin you're a font of information and I'm learning so much from this group! Thanks for engaging so deeply!

[David W Hogg]

Dustin Lang is the absolute and very best; everything he does, he does beautifully.

To view this discussion on the web visit https://groups.google.com/d/msgid/astrometry/4f57cde1-71c6-48ec-a064-8b5d78c1e0c3n%40googlegroups.com.

--

David W. Hogg — http://cosmo.nyu.edu/hogg/