How to Use Your Own Data

[hawk...@electricfish.com]

Gave this a whirl with data from my neighbourhood.  It got part way and then died here:

Loading building outlines...

ERROR: Unable to open data source <data/stanley-point>

Traceback (most recent call last):

  File "/app/src/pipeline.py", line 344, in <module>

    main()

  File "/app/src/pipeline.py", line 201, in main

    outlines = load_building_outlines(

               ^^^^^^^^^^^^^^^^^^^^^^^

  File "/app/src/utils/building_outlines.py", line 29, in load_building_outlines

    v_in = grass_module("v.in.ogr", input=shapefile, output=output_name, overwrite=True)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/grass84/etc/python/grass/pygrass/modules/interface/module.py", line 625, in __init__

    self.__call__(*args, **kargs)

  File "/usr/lib/grass84/etc/python/grass/pygrass/modules/interface/module.py", line 649, in __call__

    return self.run()

           ^^^^^^^^^^

  File "/usr/lib/grass84/etc/python/grass/pygrass/modules/interface/module.py", line 838, in run

    self.wait()

  File "/usr/lib/grass84/etc/python/grass/pygrass/modules/interface/module.py", line 859, in wait

    raise CalledModuleError(

grass.exceptions.CalledModuleError: Module run `v.in.ogr input=data/stanley-point min_area=0.0001 snap=-1.0 output=nz-building-outlines-all-sources --o` ended with an error.

The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.

My folder structure looks like:

stanley-point
    BA32.aux
    BA32.aux.aux.xml
    BA32.tfw
    BA32.tif
    BA32.tif.aux.xml
    BA32.xml
    new-zealand-lidar-1m-dsm.txt
    nz-building-outlines-all-sources.gpkg
    nz-building-outlines-all-sources.txt
    nz-building-outlines-all-sources.xml

[Cameron Shorter]

Richard,

You might want to include the command you entered next time too, to help with debugging.
I'm guessing you will fix this by renaming nz-building-outlines-all-sources.gpkg to stanley-point.gpkg

(Probably an update to the instructions needed here)

I've found that asking AI to debug my logs is very helpful. (Attach relevant python and Docker files to a query):

I have run the python script attached, and have included Docker and docker-compose files attached, and got the following errors, what is the cause?

The cause of this error is that the GRASS GIS module (v.in.ogr) cannot find or open the specific GeoPackage file at the path provided.

In your previous command, the --building-dir was set to data/my-council, and the script looked inside that folder for the layer. Here, the error Unable to open data source <data/stanley-point> indicates that the path passed to the --building-dir argument is either incorrect, the file is missing, or the filename extension is not being handled as expected by the script's logic.

Here is how to debug and fix this:

1. The Missing File Extension

In the pipeline.py script, the load_building_outlines function (found in utils/building_outlines.py) takes the building-dir and attempts to open it. When using a GeoPackage, the code usually expects a path to the actual file or the directory containing it.

If the user provided --building-dir "data/stanley-point", GRASS is looking for a data source at that exact path. If the file is actually named nz-building-outlines-all-sources.gpkg inside that folder, the path is incomplete.

2. Incorrect Path Mapping (Docker vs. Local)

The most common cause in a Docker environment is a mismatch between the local file system and the container's internal volume.

• Local Path: solar-estimates/src/data/stanley-point/

• Container Path: /app/src/data/stanley-point/

If the user has not placed the GeoPackage inside the src/data/ directory (which is mapped to /app/src/data/ via the volume in docker-compose.yml), the container will not be able to "see" the data.

3. Case Sensitivity and Permissions

• Linux Case Sensitivity: Ensure the folder is actually stanley-point and not Stanley-Point. Inside the Docker container (which is Ubuntu), paths are case-sensitive.

• Permissions: On some systems (especially Linux hosts), the Docker user may not have permission to read the file if it was created by a different local user.

---

How to Fix It

The user should verify their folder structure and run the command ensuring the --building-dir points to the folder containing the GeoPackage.

Verify the file exists here:

Bash

ls src/data/stanley-point/nz-building-outlines-all-sources.gpkg

Run the corrected command: If the file is inside a folder named stanley-point, ensure the --building-dir argument points to that folder, and the --building-layer-name matches the internal layer name of the GeoPackage:

Bash

docker compose run pipeline /opt/venv/bin/python /app/src/pipeline.py \
  --area-name "stanley-point" \
  --dsm-glob "data/stanley-point/*.tif" \
  --building-dir "data/stanley-point" \
  --building-layer-name "nz-building-outlines-all-sources"

Note: If v.in.ogr still fails, it is possible the .gpkg file is corrupted or formatted in a way that the container's version of GDAL/OGR cannot read. Testing the file with ogrinfo data/stanley-point/nz-building-outlines-all-sources.gpkg inside the container would confirm if the data source is valid.

--
You received this message because you are subscribed to the Google Groups "solarmap-nz-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solarmap-nz-te...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/solarmap-nz-tech/b43de670-5936-496d-bcb9-7a9d6780eaa1n%40googlegroups.com.

[hawk...@electricfish.com]

Sorry for not including the command - this is what I ran:

docker compose run pipeline /opt/venv/bin/python /app/src/pipeline.py \

  --area-name "stanley-point" \
  --dsm-glob "data/stanley-point/*.tif" \
  --building-dir "data/stanley-point" \
  --building-layer-name "nz-building-outlines-all-sources"

Which is what the bot suggested.

I tried renaming the file as you suggested, but got the same error.

[Shreyas Rama]

The GRASS docs aren't super clear on this but I believe --building-dir needs to point to a directory containing only the outline data. The underlying GRASS module v.in.ogr, at least for Shapefiles, uses that parameter: https://grass.osgeo.org/grass-stable/manuals/v.in.ogr.html

--building-layer-name is probably worded a bit confusingly, it's just to give the internal GRASS layer a name.

[Cameron Shorter]

Hey Shreyas, welcome back.

A minor fix I think we should introduce is to use geopackage in our standard implementation instead of shapefiles.

Geopackages are more efficient, and likely would avoid this bug error.

To view this discussion, visit https://groups.google.com/d/msgid/solarmap-nz-tech/d8b3990d-1d0d-48a3-92d1-343d120d8964n%40googlegroups.com.

[hawk...@electricfish.com]

Hi Shreyas -

Which of the files in the directory should stay? I tried matching with the `shotover_country` file types but it still failed.

[Jenny Sahng]

Richard and I paired on getting it working for Stanley Point and it worked! 🙌 We were able to calculate irradiance for Stanley Point and then visualise the output geopackage it in QGIS :D 

Richard, would you mind updating the docs to make it clear what data is needed (e.g. getting building outlines from LINZ for the area in question), updating the params like `building_dir` accordingly, etc.?

To view this discussion, visit https://groups.google.com/d/msgid/solarmap-nz-tech/e5a0d3c5-8a6f-4f31-a1c7-6025f73dd47fn%40googlegroups.com.

[Cameron Shorter]

Nice work Richard, great to see you have the pipeline working.

Welcome to the inner-circle of pipeline developers. :)

To view this discussion, visit https://groups.google.com/d/msgid/solarmap-nz-tech/CALGcMYPDGSiO%2BmKRMWXf3TCrxt%2BgYsPsca5ugfL3U1ZreeODug%40mail.gmail.com.