Model script failed

[Renzo Giudice]

Dear all, I am trying to calculate a travel cost map for Brazil (the whole country) at a 30m resolution but I am getting this message:

Unable to read cells: friction_dry_Tobler_slope90.tif, band 1: IReadBlock failed at X offset 0, Y offset 14278: _TIFFPartialReadStripArray:Cannot read offset/size for strile around ~14278

This indicates that there is something wrong with the friction map but I do not know what.

Is the resolution causing the problem?

Thanks in advance

Kind regards

Renzo

[Hermann Oliveira Rodrigues]

Hi, Renzo,

A bit late to provide an answer, but I believe the input map is corrupted. To test that hypothesis just load the map using a Load Map functor and save it with a different name using a Save Map. If the map is corrupted, that simple model will fail. 

Best, 

Hermann

--
You received this message because you are subscribed to the Google Groups "Dinamica EGO" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dinamica-ego...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CA%2B0Bx-0BS0bOHLp6AMe2H4gc3irods9Voz92VTGAru5eRw8i7Q%40mail.gmail.com.

[Renzo Giudice]

Dear Hermann,

Great to hear from you after such a long time, and thanks for your reply!

Following your suggestion, I have successfully loaded and saved both input maps—the friction map and the source map (municipalities) with new names, and these two simple models did not fail. I am now rerunning the script to calculate the cost map using the saved maps to see if saving them within the Dinamica EGO interface helps avoid the issue. I will get back to you once the script has finished running.

Thanks again for your help!

Best regards,
Renzo

To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CADnw22akSV_EniFfcOa16GV4cK1PZsJgtC4PvbxSOmGoRvW_aA%40mail.gmail.com.

[Renzo Giudice]

Dear Hermann,

well it seems it worked, but it is still running, for 6 days already!!

Do you think this is still normal for a 30m resolution, for the whole country (Brazil)? I am using a server which has 22 workers.

Thank you in advance!

Kind regards

Renzo

[Hermann Oliveira Rodrigues]

Hi, Renzo, 

That's a long time for a cost calculation...

Which memory management policy are you using? And how much memory do you have available in your system. If possible, try switching the memory policy to "aggressive". And you your really have tons of memory to spare, you can even try "memory only". 

Best, 

Hermann

----------------------------------------------------------------------
Hermann Rodrigues
hermann....@gmail.com
her...@csr.ufmg.br
Twitter: @horodrigues | @dinamica_ego

Centro de Sensoriamento Remoto / UFMG
https://csr.ufmg.br | https://dinamicaego.com

To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CA%2B0Bx-2fmv6T5FqfMPwBoL4cXMMCTJ8kFe99aThzKSo_WueJuQ%40mail.gmail.com.

[Renzo Giudice]

Thanks for the quick reply!

I was using the default (balanced). The RAM is 128 GB.

I have just aborted the execution of the script and restarted it after switching to the aggressive memory policy.

I hope it goes faster now.

Thanks again!

To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CADnw22a%3Dm7JjuE-StWnHd3me%3D2Od_kGhpcsHF%2Bvr4EyTybCnAQ%40mail.gmail.com.

[Renzo Giudice]

Dear Hermann,

As I mentioned in my last email, I changed the memory policy to aggressive and the model script has now been running for almost 20 hours. Is that still too long for a calc cost map? If yes, what could be causing this?

Thanks for your valuable help!

Renzo

[Hermann Oliveira Rodrigues]

It's still a long time. And it would be nice to know a few more things:

- Where are the cost calc map inputs coming from? Are they coming from a load map or are they coming from some other calculation? If they are coming from a load map, how fast is the disk/storage where they are located? Really fast, slow?

- If you hover the mouse on the progress bar at the bottom of the interface you will see what the workers are doing. And they all should be busy. Can you confirm that?

Now comes the most complicated part: you said the system where the calculation is being performed has 20 cores. Do you know anything about how these cores are organized? Do you know if your system's CPUs are organized as NUMAs?

Best,

hermann

----------------------------------------------------------------------
Hermann Rodrigues

To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CA%2B0Bx-2Ww5%2Bx%3DgyhHf%2BXD93dx6zoV_hW7Xy%2Br6_DE9J1RP09aw%40mail.gmail.com.

[Renzo Giudice]

Hi Hermann,

• The cost calc map inputs are coming from the C disk of the server. I noticed that at a coarser resolution (900 m) the script also took days to run because, apparently, the inputs were in my PC (I access the server remotely). When I saved everything in the server's C disk, the script ran in only 12 minutes (at 900 m resolution). However, at 30 m resolution is again already taking 2 days.
• All 22 workers are running the task "CCM_TDLR".

I will need to ask your last question to the IT person in charge.

Thanks again!

To view this discussion visit https://groups.google.com/d/msgid/dinamica-ego/CADnw22YeX93SCmXLgoVhr-4VMr2CWE-fbf_SCvSY%2BTwhLiJX6Q%40mail.gmail.com.