Tree importing via uimport

[Keith Bornhorst]

Trying to import a few trees...

- OTM v1.2

Input data:

"ID","POINT_X","POINT_Y","SCIENTIFIC","ADDRESS","PLOTTYPE","PLOTLENGTH","PLOTWIDTH","POWERLINE","OWNER","DATEPLANTED","DIAMETER","HEIGHT","CANOPYHEIGHT","CONDITION","CANOPYCONDITION"

1,-82.409141,28.060768,"Acer rubrum",,,,,,,,,,,,

2,-82.408968,28.060432,"Acer rubrum",,,,,,,,,,,,

3,-82.40898,28.060737,"Acer rubrum",,,,,,,,,,,,

4,-82.408976,28.060762,"Acer rubrum",,,,,,,,,,,,

5,-82.408959,28.060737,"Acer rubrum",,,,,,,,,,,,

6,-82.408549,28.060431,"Acer rubrum",,,,,,,,,,,,

7,-82.408549,28.060522,"Acer rubrum",,,,,,,,,,,,

- "Acer rubrum" exists in treemap_species (id=4)

First attempt failed

- Updated /treemap/management/commands/uimport.py to remove unnecessary "OpenTreeMap" and fix how CHOICES were being loaded

Second attempt resulted with:

$ python manage.py uimport --verbose treemap/management/commands/data_tampa/TreeImport1.csv 1

Opening input file: treemap/management/commands/data_tampa/TreeImport1.csv

Importing 7 records

Finished data load.

Calculating new species tree counts...

Done.

Yet no rows exist in treemap_tree or treemap_plot (or any audit/pending version of those tables). The import attempt was logged to treemap_importevent. The "xxx.csv.err" file is empty.

Any thoughts on what might be wrong or how to determine where in the import process we ran into problems?

Thanks,

Keith

[Adam Hinz]

Hey Keith,

Have you loaded your species list yet? Also, there may be a
"TreeImport1.csv.err" file with rows that were explicitly rejected.

v1.3 does have an improved version of the upload script. That might
give you better results.

-Adam

On 03/14, Keith Bornhorst wrote:
> Trying to import a few trees...
>
> - OTM v1.2
>
> Input data:
> "ID","POINT_X","POINT_Y","SCIENTIFIC","ADDRESS","PLOTTYPE","PLOTLENGTH","PLOTWIDTH","POWERLINE","OWNER","DATEPLANTED","DIAMETER","HEIGHT","CANOPYHEIGHT","CONDITION","CANOPYCONDITION"
> 1,-82.409141,28.060768,"Acer rubrum",,,,,,,,,,,,
> 2,-82.408968,28.060432,"Acer rubrum",,,,,,,,,,,,
> 3,-82.40898,28.060737,"Acer rubrum",,,,,,,,,,,,
> 4,-82.408976,28.060762,"Acer rubrum",,,,,,,,,,,,
> 5,-82.408959,28.060737,"Acer rubrum",,,,,,,,,,,,
> 6,-82.408549,28.060431,"Acer rubrum",,,,,,,,,,,,
> 7,-82.408549,28.060522,"Acer rubrum",,,,,,,,,,,,
>
> - "Acer rubrum" exists in treemap_species (id=4)
>
> First attempt failed
> - Updated /treemap/management/commands/uimport.py to remove unnecessary
> "OpenTreeMap" and fix how CHOICES were being loaded
>
> Second attempt resulted with:
>
> $ python manage.py uimport --verbose
> treemap/management/commands/data_tampa/TreeImport1.csv 1
> Opening input file: treemap/management/commands/data_tampa/TreeImport1.csv
> Importing 7 records
> Finished data load.
> Calculating new species tree counts...
> Done.
>

> Yet no rows exist in *treemap_tree* or *treemap_plot* (or any audit/pending
> version of those tables). The import attempt was logged to *
> treemap_importevent*. The "xxx.csv.err" file is empty.

>
> Any thoughts on what might be wrong or how to determine where in the import
> process we ran into problems?
>
> Thanks,
> Keith
>

--

Adam Hinz
Software Engineer, Azavea
ad...@adamhinz.com | c: 608-445-8391 | Twitter: @ahinz

[Keith Bornhorst]

Yes. Single neighborhood, single zipcode, four species (one of which is "Acer rubrum"). The TreeImport1.csv.err file was created, but it was empty.

If I wanted to try an import with 1.3, how much 1.2 code do I need to overwrite with 1.3? Just uimport.py, you think? Just worried that it'll snowball into upgrading models and views and who knows what else. I'll try just uimport.py and see what happens.

Thanks,
Keith

[Bob Troester]

Keith, Hi,

I've just compared the 1.2 and 1.3 uimport.py code and as far as I can see you'll have much better success with 1.3 - once you remove the "OpenTreeMap" prefix in the "import choices" line that youalready fixed in 1.2 but which hasn't been propagated to 1.3 as yet (remembering to change "choice" to "choice_youridentifier"), AND fix the "plot.address_state = 'CA' " assignment statement (marked FIXME!) to use your own state abbreviation that has been in the code since before 1.1.

Without having checked what might be available in 1.3, I just got done fixing my 1.2 copy, and was finally successful yesterday importing one test tree, hurray!

Evidently I was repeating what Adam Hinz had just done in the 1.3 code: replacing stray tabs with spaces, initializing the variable "wrote_headers = False" and especially removing the "if i < 420: continue" that prevented the code from doing anything until it had read 420 lines of your input csv file! That little "if" is the reason you didn't see any trees imported! Happened to me too.

I would guess that that code had been entered during some testing for 1.2 and then was never used again before the release - or after it until now. Well, regressions happen, but they are especially painful to us newbys who are still trying to get up to speed.

One other thing: I was disturbed to see that your fix to remove "OpenTreeMap" from the import statement was put into release 1.2, which  was released in October, but didn't get in to 1.3. I'm used to projects where code is frozen once the release is out, and no further mods can be made in that release. If changes can be made, how can two developers be sure then that they are starting from the same code base? Is one expected to do a "git fetch" (or pull?) periodically? I get confused reading about it! (See http://stackoverflow.com/questions/292357/whats-the-difference-between-git-pull-and-git-fetch)

I used to use Subversion - I guess a Git environment is more different than I expected!
Bob

[Bob Troester]

One more thing -
Can you actually use lat, long in your input data for uimport, as you show in your example? The args for uimport are  <input_file_name, data_owner_id, base_srid, read_only> where the base srid defaults to 4326. I converted my single tree lat/long to WGS84 before running uimport, using http://www.rcn.montana.edu/resources/tools/coordinates.aspx - if one can use lat/long that will be much easier!
Bob

[Keith Bornhorst]

Thanks Bob.

I still can't get uimport to work, but I *think* we can use lat/lon values in our import data given that the samples provided (/treemap/management/commands/testdata/*.csv) do so...

[Andrew Thompson]

Re 1.2 fixes not getting in 1.3 branch: as of this (now closed) pull request: https://github.com/azavea/OpenTreeMap/pull/91 , everything is merged into 1.3 now. Sorry that took a few days, much of the OpenTreeMap development team is at the OSGeo Code Sprint this week.

Re using lat/long in uimport csv:

- Non-GIS nerdiness explanation: Yes, you can use decimal latitude and longitude in your CSV file.

- explanation for GIS nerds: Per this column definition/doc file: https://github.com/azavea/OpenTreeMap/blob/v1.2/treemap/management/commands/uimport_doc.txt , the POINT_X and POINT_Y fields ought to be decimal longitude and latitude coordinates, using the EPSG:4326 spatial reference system. Which does use the WGS84 ellipsoid. In my experience, this is a very common format one often finds lat/long coordinates in.

-Andrew

On Tue, Mar 26, 2013 at 9:47 AM, Keith Bornhorst <kborn...@gmail.com> wrote:

Thanks Bob.

I still can't get uimport to work, but I *think* we can use lat/lon values in our import data given that the samples provided (/treemap/management/commands/testdata/*.csv) do so...

--
You received this message because you are subscribed to the Google Groups "opentreemap-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentreemap-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Andrew Thompson, Community Evangelist

Azavea |  340 N 12th St, Ste 402, Philadelphia, PA
atho...@azavea.com  | T 215.701.7689  | F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea

Join our community! Google Groups:

Cicero API | OpenTreeMap | GeoTrellis | OpenDataCatalog

[jitendra shah]

I am facing the following issue to run the following command.

" python manage.py uimport --verbose ../data/tree-set1-formatted.csv 8 > ~/import.log 

Traceback (most recent call last):

  File "manage.py", line 13, in <module>

    execute_manager(settings)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 438, in execute_manager

    utility.execute()

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 379, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 191, in run_from_argv

    self.execute(*args, **options.__dict__)

  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 220, in execute

    output = self.handle(*args, **options)

  File "/var/www/otm/OpenTreeMap/treemap/management/commands/uimport.py", line 79, in handle

    self.err_writer = csv.writer(open(err_file, 'wb'))

IOError: [Errno 13] Permission denied: '../data/tree-set1-formatted.csv.err' "

[Adam Hinz]

It looks like you don't have permission to write in the "../data/" directory

> > Yet no rows exist in *treemap_tree* or *treemap_plot* (or any
> > audit/pending version of those tables). The import attempt was logged to *
> > treemap_importevent*. The "xxx.csv.err" file is empty.

> >
> > Any thoughts on what might be wrong or how to determine where in the
> > import process we ran into problems?
> >
> > Thanks,
> > Keith
> >
>

[jitendra]

Dear Adam Hinz,

So nice to get a prompt response.

In deed, I didnot use the sudo privilege I was given but had forgotten.

I have managed to upload the tree data.

I have a few issues still pending.

Shalll pose the same on this list if I cannot fix them.

regards

jitendra

(Gupteshwar is actually working on the issue. I am requesting him to join the mailing list)

--
You received this message because you are subscribed to a topic in the Google Groups "opentreemap-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opentreemap-user/ogHbjB36Gx4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opentreemap-us...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
jitendra