Version issue?

[steven.wangen]

Hi Joel -

I just did a clean install of grass and MDiG on the new Lucid Lynx
(needed to clean up my laptop), but I'm getting this error when I try
to run mdig:

Traceback (most recent call last):
File "./mdig.py", line 231, in <module>
main(sys.argv[1:])
File "./mdig.py", line 125, in main
mdig_config = MDiGConfig.get_config()
File "/home/steve/mdig/mdig/mdig/MDiGConfig.py", line 65, in
get_config
mdig_config = MDiGConfig()
File "/home/steve/mdig/mdig/mdig/MDiGConfig.py", line 183, in
__init__
self.updates()
File "/home/steve/mdig/mdig/mdig/MDiGConfig.py", line 217, in
updates
self['version']=mdig.version
UnboundLocalError: local variable 'mdig' referenced before assignment

I'm guessing it has something to do with the way the new configuration
files are being produced (particularly after the first install), but
wasn't quite sure - any thoughts?

Cheers!

-Steve

[Joel Pitt]

Hi Steve,

That is odd... can you check there is a line reading:

import mdig

near the top of the MDiGConfig.py file?

If there is, then can you give me the python version? (use "python --version")

--

Also, did you have any trouble with migrating the repository? If
you're starting from scratch then it won't be an issue, but best to be
aware that the repository is now merged with the GRASS DB directory
(so that people only have to track one directory of data).

Joel Pitt, PhD | http://ferrouswheel.me | +64 21 101 7308
NetEmpathy Co-founder | http://netempathy.com
OpenCog Developer | http://opencog.org
Board member, Humanity+ | http://humanityplus.org

[Steve Wangen]

Hey Joel - 

MDiG is imported, and the python version is 2.6.5.  I didn't migrate anything as I'm just starting from scratch and figured it'd be easy enough to just reload the models.  There is also a blank mdig.conf file in the .mdig folder in my home directory (not sure if that's relevant or not).  

Just curious - the GRASS DB directory - is that the user-specified folder for storing grass data (ie. ~/grassdata)?

Let me know if that helps...

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

On Tue, May 25, 2010 at 2:40 PM, Steve Wangen <i.bl...@gmail.com> wrote:
> Hey Joel -
> MDiG is imported, and the python version is 2.6.5.  I didn't migrate
> anything as I'm just starting from scratch and figured it'd be easy enough
> to just reload the models.  There is also a blank mdig.conf file in the
> .mdig folder in my home directory (not sure if that's relevant or not).

The mdig.conf file will eventually be updated with the default values
(once it gets past the error you are getting).

Can you look in the mdig/mdig/__init__.py file? There should be a
"version" variable that says "0.2.9"

> Just curious - the GRASS DB directory - is that the user-specified folder
> for storing grass data (ie. ~/grassdata)?

Yup, that's it. The mdig files are stored in their own mapsets. These
mapsets have a mdig/ sub directory with all the model files, but you
can still add models a GRASS DB directory using "mdig.py add
model_file.xml".

Also, once you've gotten passed the version error, you can edit the
config file so that you're able to run mdig outside of GRASS. If you
run MDiG inside GRASS however, then the current GRASS DB is used
instead of the mdig.conf settings.

J

[Steve Wangen]

Hi Joel - 

The init file does specify the version.  I also tried stepping through with the debugger - it looks like the mdig.version argument is fine (it returns the correct version) but when I get to line 217 (or 218) I get the error:

UnboundLocalError: "local variable 'mdig' referenced before assignment"

> /home/steve/mdig/mdig/mdig/MDiGConfig.py(218)updates()

-> self['version']=mdig.version

I'm not sure, but it looks like the 'self' reference (to the ConfigObj) might be empty at this point, and there is no 'version' element in the list to assign the 'mdig.version' value?

I'm not sure if that's it or not - I haven't been able to track it any deeper at this point, but let me know if you have any thoughts.

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

For some strange reason, adding

"import mdig" to the start of the update method fixes it!

I'm committed and pushed the change so you can just use "bzr pull"

weird!

Joel Pitt, PhD | http://ferrouswheel.me | +64 21 101 7308
NetEmpathy Co-founder | http://netempathy.com
OpenCog Developer | http://opencog.org
Board member, Humanity+ | http://humanityplus.org

[Steve Wangen]

Cool - seems to work now... up until the point where it spits this out:

Traceback (most recent call last):

  File "./mdig.py", line 231, in <module>

    main(sys.argv[1:])

  File "./mdig.py", line 127, in main

    the_action = process_options(argv)

  File "./mdig.py", line 95, in process_options

    usage()

  File "./mdig.py", line 71, in usage

    print "MDiG repository @ " + mdig.repository.db

AttributeError: 'NoneType' object has no attribute 'db'

actually when I just run './mdig.py' it returns the list of actions, it just seems that it's trying to display the repository location as well with no luck - I'm assuming it might be something similar as the attribute isn't defined?

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

Hi Steve, argh, that's another thing I fixed but hadn't pushed to bzr yet.

Try bzr pulling again...

Sorry about this, I've been cleaning up the code and although I'm
slowly adding automated testing, it doesn't catch everything yet.

Joel Pitt, PhD | http://ferrouswheel.me | +64 21 101 7308
NetEmpathy Co-founder | http://netempathy.com
OpenCog Developer | http://opencog.org
Board member, Humanity+ | http://humanityplus.org

[Steve Wangen]

TA-DA!

Seems to be up to 100% now - I'll let you know if anything goes wrong when I actually run something!

Thanks!

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Steve Wangen]

Hi Joel - 

Sorry - new problem.  It doesn't want to add models.  When I tell it to add I get 

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Steve Wangen]

Whoops - let's try that again.

When I try to add I get the error:

Traceback (most recent call last):

  File "./mdig.py", line 231, in <module>

    main(sys.argv[1:])

  File "./mdig.py", line 137, in main

    models = mdig.repository.get_models()

  File "/home/steve/mdig/mdig/mdig/ModelRepository.py", line 132, in get_models

    for loc in os.listdir(self.db):

OSError: [Errno 2] No such file or directory: ''

The sys.argv contains the file location, but I'm not sure if the 'db' variable is supposed to be the GISDBASE read in from mdig.conf?  It appears that mdig is not initializing that file correctly - I'm assuming it should be related to my grass settings, but instead it looks like it is left over from some of yours - it's contents are;

version = 0.2.9

[WEB]

host = localhost

port = 1444

[GRASS]

GRASS_WIDTH = 640

GRASS_PNG_AUTO_WRITE = TRUE

GRASS_HTML_BROWSER = firefox

GRASS_MESSAGE_FORMAT = silent

GRASS_GNUPLOT = gnuplot -persist

MAPSET = PERMANENT

GISBASE = /usr/local/grass-6.4.svn

GISDBASE = ""

LOCATION_NAME = ""

GRASS_TRUECOLOR = TRUE

GRASS_PAGER = cat

GRASS_PYTHON = python

GRASS_WISH = wish

GRASS_TRANSPARENT = TRUE

GRASS_HEIGHT = 480

[LOGGING]

ansi = false

[OUTPUT]

background_map = nz_DEM_jacques

output_width = 480

output_height = 640

Sorry to bug you again, but let me know if you have any thoughts.

Thanks!

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

Hi Steve,

Are you running this from within GRASS?

I've been running things from outside of GRASS and explicitly setting
the following two options

GISDBASE = ""
LOCATION_NAME = ""

You'll need to replace these with the actual GRASS DB directory and
the LOCATION_NAME to begin in)

MDiG can detect models in other GIS locations within the same
GISDBASE, but it needs at least one LOCATION to begin working with
GRASS commands. You should also check some of the other variables like
GISBASE to ensure it's a real directory.

I'll try to fix things so that it reads all this from the GRASS
environment when running MDiG from inside GRASS.

Cheers

Joel Pitt, PhD | http://ferrouswheel.me | +64 21 101 7308
NetEmpathy Co-founder | http://netempathy.com
OpenCog Developer | http://opencog.org
Board member, Humanity+ | http://humanityplus.org

[Steve Wangen]

Hey Joel - 

Yeah, I was trying to run it from within grass - should it just be run outside now?  I'm still not sure on the exact syntax on running outside grass - I thought it would be:

./mdig.py add [model/location] GISDBASE=[grass/database/location] LOCATION_NAME=[grass location]

but that doesn't seem to work - should the database location and the location name be specified as part of the tags?

Thanks!

-steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

Hi Steve,

Those variables should be set in mdig.conf, not on the command line! ;-)

[Steve Wangen]

Hey Joel -

Thanks for that.  Still trying to run it the ol' fashioned way!  Anyway, I think I got the mdig.conf file sorted, but now when I try to load the model (from outside grass) I get the error:

  File "/home/steve/mdig/mdig/mdig/ModelRepository.py", line 40, in add_model

    raise RepositoryException("Model doesn't define GIS Location for simulation")

mdig.ModelRepository.RepositoryException: Model doesn't define GIS Location for simulation

Which I'm assuming is due to the fact that the location isn't specified in the model xml file?  I'm not sure where to specify it though - I specified the location in the mdig.config file, but that doesn't seem to be enough.  I looked at the wiki and couldn't find anything - any hints?

Also, does the inclusion of the location information in the mdig.config file mean that this file has to be edited for each model (assuming they are run in different locations)?  Just curious...

Thanks!

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

Hi Steve,

Sorry about this, it's another change necessary for the integration
with the GRASS DB.

Basically you just have to add an element

<GISLocation>location_name</GISLocation>

After the <random> element, but before the <regions> element.

The location_name has to already exist in the GISDBASE however.

-

You don't need to change the mdig.conf as it will automatically change
the location/mapset for each model. The reason those details are in
the mdig.conf is that MDiG needs to initialise GRASS on *some
location* otherwise it'll get confused.

Cheers,

Joel Pitt, PhD | http://ferrouswheel.me | +64 21 101 7308
NetEmpathy Co-founder | http://netempathy.com
OpenCog Developer | http://opencog.org
Board member, Humanity+ | http://humanityplus.org

[Steve Wangen]

Hey Joel - 

having another issue with adding a model here - I just got back to trying, so haven't gotten any loaded at this point, but I'm getting an error when I try.  

So, I've got my GISDBASE at '/home/steve/grassdata/'.  

I'm trying to work in a location (which I've created) called 'test_landscape'. 

I've set '<GISLocation>test_landscape</GISLocation>' in the model file I'm trying to add.

In the mdig.conf file I have the variables 'GISDBASE = /home/steve/grassdata' and 'LOCATION_NAME = test_landscape'

I think that's everything.

Anyway, when I try to add the model file (running mdig outside of grass now) I get the error "ERROR: LOCATION << /home/steve/grassdata/PERMANENT >> not available".

The PERMANENT mapset is located in "/home/steve/grassdata/test_landscape/PERMANENT", so I tried changing the variable GISDBASE (in mdig.conf) to "/home/steve/grassdata/test_landscape" in order to try and get it to dig a little deeper (even though that isn't really the location of the GISDBASE), but then I get "ERROR: GRASS location dir doesn't exist: /home/steve/grassdata/test_landscape/test_landscape"

I've included the complete error messages below - it seems I'm missing something in the location specification, but I can't quite figure out what that is - any suggestions?

Thanks!

-Steve

ERROR_1:

access: No such file or directory

ERROR: LOCATION << /home/steve/grassdata/PERMANENT >> not available

Traceback (most recent call last):

  File "./mdig.py", line 231, in <module>

    main(sys.argv[1:])

  File "./mdig.py", line 160, in main

    the_action.do_me(None)

  File "/home/steve/mdig/mdig/mdig/Actions.py", line 374, in do_me

    mdig.repository.add_model(m)

  File "/home/steve/mdig/mdig/mdig/ModelRepository.py", line 49, in add_model

    if not g.change_mapset(dm.get_name(),dm.get_location(),True):

  File "/home/steve/mdig/mdig/mdig/GRASSInterface.py", line 698, in change_mapset

    result = self.run_command("g.mapset -c mapset=%s%s" % (mapset_name,loc))

  File "/home/steve/mdig/mdig/mdig/GRASSInterface.py", line 842, in run_command

    raise GRASSCommandException(commandstring,self.stderr,ret)

mdig.GRASSInterface.GRASSCommandException: GRASSCommandException (cmd:g.mapset -c mapset=8x8_CODA_scrub location=test_landscape)

ERROR_2:

20100602 09:40:39 [mdig.grass] ERROR: GRASS location dir doesn't exist: /home/steve/grassdata/test_landscape/test_landscape

Traceback (most recent call last):

  File "./mdig.py", line 231, in <module>

    main(sys.argv[1:])

  File "./mdig.py", line 140, in main

    grass_interface = GRASSInterface.get_g()

  File "/home/steve/mdig/mdig/mdig/GRASSInterface.py", line 85, in get_g

    grass_i = GRASSInterface()

  File "/home/steve/mdig/mdig/mdig/GRASSInterface.py", line 134, in __init__

    self.init_environment()

  File "/home/steve/mdig/mdig/mdig/GRASSInterface.py", line 222, in init_environment

    raise EnvironmentException()

mdig.GRASSInterface.EnvironmentException

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

Hi Steve,

First question, are you using the latest version from bzr? (just need
to check it's not something I've already fixed...)

If you are, then proceed with the below...

Is this inside or outside of GRASS?

For error 1, can you add the line:

print self.grass_vars

just before

raise GRASSCommandException(commandstring,self.stderr,ret)

in GRASSInterface.py?

Then try it again and let me know the output just before the error.
Hopefully this will indicate what part of the GRASS setup is broken. I
assume that going into the grass location manually (with GRASS) and
running g.mapset works fine?

Apologies for these hassles once again - I've been slowly building the
testing framework up to ensure things work correctly, but there's ALOT
of code so it's been taking me a while.

J

[Steve Wangen]

Hey Joel - 

I did a version pull, and it wasn't the latest, but I still got the error.  The extra line returned this:

{'GRASS_WIDTH': '640', 'GISDBASE': '/home/steve/grassdata', 'LOCATION_NAME': 'PERMANENT', 'GRASS_GUI': 'text', 'GRASS_HEIGHT': '480', 'GRASS_HTML_BROWSER': 'firefox', 'GRASS_PAGER': 'cat', 'GRASS_MESSAGE_FORMAT': 'silent', 'GRASS_WISH': 'wish', 'GISRC': None, 'GRASS_TRUECOLOR': 'TRUE', 'GRASS_TRANSPARENT': 'TRUE', 'GRASS_PNG_AUTO_WRITE': 'TRUE', 'GRASS_GNUPLOT': 'gnuplot -persist', 'MAPSET': 'test_landscape', 'GRASS_PYTHON': 'python', 'GISBASE': '/usr/local/grass-6.4.0svn'}

which indicates the location name isn't coming through - I double checked the mdig.conf file, and it is definitely set to "LOCATION_NAME = test_landscape".  

Hope that helps - I know these things are always alot more time consuming once you get past the fun stuff.  This project has at least taught me that!

-Steve

 
---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

On Thu, Jun 3, 2010 at 9:04 AM, Steve Wangen <i.bl...@gmail.com> wrote:
> I did a version pull, and it wasn't the latest, but I still got the error.
>  The extra line returned this:
> {'GRASS_WIDTH': '640', 'GISDBASE': '/home/steve/grassdata', 'LOCATION_NAME':
> 'PERMANENT', 'GRASS_GUI': 'text', 'GRASS_HEIGHT': '480',
> 'GRASS_HTML_BROWSER': 'firefox', 'GRASS_PAGER': 'cat',
> 'GRASS_MESSAGE_FORMAT': 'silent', 'GRASS_WISH': 'wish', 'GISRC': None,
> 'GRASS_TRUECOLOR': 'TRUE', 'GRASS_TRANSPARENT': 'TRUE',
> 'GRASS_PNG_AUTO_WRITE': 'TRUE', 'GRASS_GNUPLOT': 'gnuplot -persist',
> 'MAPSET': 'test_landscape', 'GRASS_PYTHON': 'python', 'GISBASE':
> '/usr/local/grass-6.4.0svn'}
> which indicates the location name isn't coming through - I double checked
> the mdig.conf file, and it is definitely set to "LOCATION_NAME =
> test_landscape".
> Hope that helps - I know these things are always alot more time consuming
> once you get past the fun stuff.  This project has at least taught me that!

Yeah - the grass_vars change and MDiG will sometimes move to the
PERMANENT mapset (and others) to do stuff... so that's okay.

Are you able to package your model and supporting files into a zip for
me to test locally?

(Also, I'm assuming you made the location using GRASS and that you've
checked it's made a PERMANENT mapset?)

J

[Steve Wangen]

Sure - here's the model stuff (should be everything) - yeah, I figured moving to the PERMANENT mapset was okay, but it's just got to get down into the 'location' directory first.  The PERMANENT does exist, it's just a little lower down the chain.  Have a look and see what you think.

-steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

On Thu, Jun 3, 2010 at 9:45 AM, Steve Wangen <i.bl...@gmail.com> wrote:
> Sure - here's the model stuff (should be everything) - yeah, I figured
> moving to the PERMANENT mapset was okay, but it's just got to get down into
> the 'location' directory first.  The PERMANENT does exist, it's just a
> little lower down the chain.  Have a look and see what you think.

Fixed in latest bzr.

Stupid mistake where I swapped the location and mapset parameters and
obviously didn't test it.

However, now your CODA file seems to be the wrong format... I'll leave
you to double check this before I start checking if it's an MDiG
issue.

[Steve Wangen]

I'll have a go - btw, it's probably not a format issue - I just neglected to include them in the files I sent over.  Whoops!

First run - got an error: mdig.ModelRepository.RepositoryException: Couldn't create mapset 8x8_CODA_scrub, it already exists in location /home/steve/grassdata/test_landscape/8x8_CODA_scrub.

Deleting the mapset is the obvious option, but that means that the -c functionality in g.region is being overridden I believe (create if it doesn't already exist).  (this becomes an issue later when I get past this point but the add command fails later AFTER creating the mapset).

Cool - then it has added!  And (I imagine you planned this) it is able to find the index file specified in the transition files in the PERMANENT mapset just fine (as it should, but you never know!).  I'll try giving it a run and see how she goes!

Thanks Joel!

-Steve

---------------------------------------------------
Steven Wangen
PhD Candidate
Bio-Protection Research Centre
Lincoln University
http://tinyurl.com/lepidulum-spread
http://www.bioprotection.org.nz
+64 (0)3 325-3838 x.8856

[Joel Pitt]

On Thu, Jun 3, 2010 at 10:58 AM, Steve Wangen <i.bl...@gmail.com> wrote:
> I'll have a go - btw, it's probably not a format issue - I just neglected to
> include them in the files I sent over.  Whoops!
> First run - got an error: mdig.ModelRepository.RepositoryException: Couldn't
> create mapset 8x8_CODA_scrub, it already exists in location
> /home/steve/grassdata/test_landscape/8x8_CODA_scrub.
> Deleting the mapset is the obvious option, but that means that the -c
> functionality in g.region is being overridden I believe (create if it
> doesn't already exist).  (this becomes an issue later when I get past this
> point but the add command fails later AFTER creating the mapset).

I'm not sure I follow that last bit? It should complain when the
mapset already exists as each
model has it's own mapset (as does each instance, but those get
created automatically).

> Cool - then it has added!  And (I imagine you planned this) it is able to
> find the index file specified in the transition files in the PERMANENT
> mapset just fine (as it should, but you never know!).  I'll try giving it a
> run and see how she goes!

Yup - the idea is that you can store maps either in the PERMANENT
mapset or in the main model mapset (i.e. not the instance mapsets that
created later). There may be some tuning required to get the main
model mapset to be correctly accessable so I'd stick with putting them
in PERMANENT for now.

J