QGIS created geopackages

[andrea antonello]

Dear all,
we are discussion offlist an issue we found with geopackages that do
not load on geopaparazzi.

I think the issue is that QGIS by default builds geopackages from
rasters using the default gdal command, i.e. the tiling scheme is
custom to optimize the resolution.

I had a look at QGIS' export dialog and there is "create options" part
in which TILING_SCHEME can be set to be GoogleMapsCompatible.

I tried to set it and the exported geopackage was empty.

Can any QGIS user of the list make this test for us please?
It would be great to have two geopackages, one with the scheme set and one not.

Thanks to anyone that might help.

Cheers,
Andrea

[Walt Ludwick]

Would love to try this -indeed, exchange of.gpkg files between QGIS on my (Mac) desktop and SMASH on my iPhone is a requirement for adoption of SMASH in my case- but i don't see any way to do this, besides that GSS server sync function that is unavailable to me at present.  Is there any way to do achieve round-trip editing (QGIS <-> SMASH 1.4.0 for iOS) of these .gpkg files, other than setting up a GSS server?

[Silvia Franceschi]

Dear Ludwick,
did you try to use a cloud system to share the data between the device
and the pc?
For example, I have an iphone and I use Dropbox.

Best regards

Silvia

> --
> You received this message because you are subscribed to the Google Groups "geopaparazzi-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to geopaparazzi-us...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/82a415b1-c1d7-4ea6-90e3-34f9a6c9d125n%40googlegroups.com.



--
ing. Silvia Franceschi
Via Roma, 64
38030 Castello di Fiemme (TN)

tel: 0039 -3384501332

[Walt Ludwick]

I likewise use Dropbox, Silvia - but SMASH app has no provision for either reading from or saving to Dropbox. Would be nice if it did!


Sent from my iPad

> On 18 Sep 2020, at 20:36, Silvia Franceschi <silvia.f...@gmail.com> wrote:
>
> Dear Ludwick,

> You received this message because you are subscribed to a topic in the Google Groups "geopaparazzi-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/geopaparazzi-users/W88I6-x5I-Y/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to geopaparazzi-us...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/CABCykq%2BPrZ8WvnCZovr02Y-jMbyj-KE8f00Rbg-GAbJfLSeBNQ%40mail.gmail.com.

[andrea antonello]

Hi Walt,
I am not sure I understand what your wish is.
a) you want to be able to EDIT a geopackage on the device and also on
your pc (moving it manually)
b) you want to have a simple way to ACCESS the same data both on the
device and on your pc

Regarding dropbox support I am not really sure about this, because
survey data can be quite large and an automatic sync of such data
might lead to some unwanted behaviors.
But we will think about it.

best,
Andrea

> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/587BAECA-31FA-426B-B4CC-D6E6B7E629D1%40valedalama.net.

[Walt Ludwick]

Yes Andrea: it is both A&B, ideally.  In context of a farm operation, the UseCase is: field worker records observations in field that then update layer(s) as appropriate in QGIS.  For this to work well, the user needs to have the Geopackage from QGIS loaded, and then record the observation to the correct layer... But if that's not possible, then the user should at least be able to record observations as features in a layer that could then be exported as a .gpkg file, which could then be imported into QGIS and integrated from desktop after the fact.  Dunno if this is consistent w/ design goals of SMASH, but we're looking for the most frictionless flow of geolocated & dateTimestamped observations -text and/or images (both ideally)- from field to QGIS.

As to mechanism of data transfer between mobile app & desktop GIS: Dropbox would be an elegant solution, but anything (besides a GSS server, which i don't have and am not eager to acquire) would be better than status quo, from my perspective.  Would be happy if i could just save to .gpkg and email the file to myself, for now... That or any other method you might suggest, i would be happy to try!   /walt

[andrea antonello]

Hi Walt,

Yes Andrea: it is both A&B, ideally.  In context of a farm operation, the UseCase is: field worker records observations in field that then update layer(s) as appropriate in QGIS.  For this to work well, the user needs to have the Geopackage from QGIS loaded, and then record the observation to the correct layer... But if that's not possible, then the user should at least be able to record observations as features in a layer that could then be exported as a .gpkg file, which could then be imported into QGIS and integrated from desktop after the fact.  Dunno if this is consistent w/ design goals of SMASH, but we're looking for the most frictionless flow of geolocated & dateTimestamped observations -text and/or images (both ideally)- from field to QGIS.

As to mechanism of data transfer between mobile app & desktop GIS: Dropbox would be an elegant solution, but anything (besides a GSS server, which i don't have and am not eager to acquire) would be better than status quo, from my perspective.  Would be happy if i could just save to .gpkg and email the file to myself, for now... That or any other method you might suggest, i would be happy to try!   /walt

a lot of things. I will try to answer and do some comments, since a lot is going on in SMASH development these days.

First thing: the GSS server. It is called server, but you could download the docker image and run it on your desktop pc without any problem. So you would not have to pay or learn anything (apart from docker). But it would not solve your usecase, since the GSS centralises the data, but then you would have to access the Postgis db directly to make custom data out of it. Sure that would not be an issue with QGIS though. :-)

Regarding your general needs, I can reveal some things we are working on right now and will be available in near future:

- geopackage geometry editing: attributes editing already works, but geometry editing is a different thing. We are trying to make it nice and with different projections support, so it is being slow. But it is what we are on right now.

- centralised serverless projects. For this we are investigating the possibility to use firebase as a server to which SMASH would be able to sync directly, without the need of the GSS. Then, with the desktop version of SMASH you would be able to access the data without any effort and export them to geopackage. 

- soon we will add geopackage support to the export options, so the above would then work

Guess that is it right now. A lot of stuff, all not funded, so if someone has the need to fund such a project, we would sure change the order of creation of the features for her/him :-D

One last thought is about an idea I have since a while now. I would love to make the device accessible as a small webserver that would expose the projects folder. Would that make your usecase simpler? I.e. connect to the device via browser and download the project?

Let the ideas flow. :-)

Cheers,

Andrea

 

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/fd1d0296-208a-4b17-9d34-e6945d23fccbn%40googlegroups.com.

[Walt Ludwick]

If there were a clickety-click way to install and run GSS on my Mac desktop, with no commandline voodoo involved, i'd try it... But i don't see this workflow explained anywhere in naive user terms.

Alternatively Andrea, to your question: YES indeed!  If my iPhone running SMASH could serve these field survey files to the web browser on my desktop, that would facilitate this UseCase greatly... So long as the files were in a format that QGIS can digest.  All standard stuff- attribute tables in .csv, related images in .jpg or .png -suchlike would serve very well for now; for this particular UseCase, no geometry required.

 

Really, whatever is the native storage format of data created in SMASH (isn't it geopackage or generic sqlite in fact?), i could probably work with that, if only i could find a way to get it from the iPhone to my Mac desktop.  Tried the export .PDF feature, but that seems to bury the file in a directory that is private to the app dev environment, not accessible to the naive user.  Am i misunderstanding something about this?

[Silvia Franceschi]

Dear Ludwick,
SMASH reads and writes some data in the native way without saving them
in the project.
In the file explore of your iphone you should find a folder called
Smash. Inside this folder you can find and place the data you want to
exchange.
Usually I use the folder data inside Smash, and there I save the data
files, gpk, shp, gpx, tiff, to be quickly available from Smash.

So basically you can send these data and save them from the email or
use the Dropbox application or an other cloud to download them in the
folder and upload them once you edit. As Andrea said there is no
automatic synchronization now, even if we are working on it, but this
"old" way works fine in iphone.

You can find more information about the Smash folder structure and the
supported data format in the manual:

https://www.geopaparazzi.org/smash/index.html

Please let me know if you can use this solution or if you need some
other information.

Regards

Silvia

> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/587BAECA-31FA-426B-B4CC-D6E6B7E629D1%40valedalama.net.

[Walt Ludwick]

This certainly helps, Silvia; in fact i was unaware that my iPhone even had a Files explorer (since 2 major releases ago, i have just learned; duh!)... So am now happily exploring the SMASH folder structure. Thanks for that!

But as to satisfying the aforementioned UseCase: do i understand correctly from this paragraph of SMASH docs that, if i put a .gpkg file there from QGIS, that i should be able to edit attribute tables in that file using SMASH? Without introducing anomalies that QGIS cannot handle? Meaning i can have RTE (Round Trip Editing) of .gpkg files between QGIS and SMASH (albeit via this somewhat frictional files trafficking process)?  If i'm reading you and the docs right, that's what it sounds like... But then the 1st msg in this thread seemed to indicate that there is some incompatibility, so i am wondering about the status of this issue. 

Anyway: i will see if i can make the RTE work, soon as i find a way to write files TO the SMASH folder structure from outside the app (since import feature only works via GSS, and since the iOS "Files" app is a read-only browser).  If you know of a way to do this, Silvia, i'd love to hear about how that is done!

[andrea antonello]

Hi Walt,

[...]

But as to satisfying the aforementioned UseCase: do i understand correctly from this paragraph of SMASH docs that, if i put a .gpkg file there from QGIS, that i should be able to edit attribute tables in that file using SMASH? Without introducing anomalies that QGIS cannot handle? Meaning i can have RTE (Round Trip Editing) of .gpkg files between QGIS and SMASH (albeit via this somewhat frictional files trafficking process)?  If i'm reading you and the docs right, that's what it sounds like...

yes that is exactly right. And we never had issues with that, so if you do, please report them, because it would be a bug.

 

But then the 1st msg in this thread seemed to indicate that there is some incompatibility, so i am wondering about the status of this issue. 

No, there is a misunderstanding. Geopackage can hold vector and raster tile data. When it comes to tile data, there is some "freedom" and we are able to handle data made in a certain way only. But we are talking about background raster data, i.e. images.

 

Anyway: i will see if i can make the RTE work, soon as i find a way to write files TO the SMASH folder structure from outside the app (since import feature only works via GSS, and since the iOS "Files" app is a read-only browser).  If you know of a way to do this, Silvia, i'd love to hear about how that is done!

Actually I test SMASH on an IPAD and I have been able to copy datasets from dropbox into the SMASH folder structure. I think you should be able to do so.

Silvia might help here also.

Ciao,

Andrea

 

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/8920c516-4cbf-4e6d-b7bd-9a1c335aaa98n%40googlegroups.com.

[Silvia Franceschi]

Hi Ludwick,
as I wrote, I usually use Dropbox:
1. browse the files in the Dropbox application
2. select the menu of the file you want to copy to Smash folder
3. select the Export option
4. select Open In..
5. select Save to Files
6. browse and select the maps folder inside the Smash folder (if it
doesn't exist please create it on your own)
7. tap Save on the top right of the window

Now you have to add it as layer in the layers section of the Smash
application. You can check if the file has been correctly saved in the
folder using the File Browser.
I never experienced that the Smash folder on the iphone was read only,
how did you open it?
Could you please try the sequence above and let me know if it works?

Thanks

Silvia

> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/8920c516-4cbf-4e6d-b7bd-9a1c335aaa98n%40googlegroups.com.

[Walt Ludwick]

Regarding your step-by-step instructions, Silvia (thanks for that!), i get as far as step 5, but then where you say "Save to Files" -presuming you mean the Files app that i have installed- there is no affordance for saving to Files.  What i see is a list of suggested apps to which i can save my chosen file -a small subset of all the apps on my iPhone, which includes neither Files nor Smash.  

Q; Could it be that there is a configuration widget hidden somewhere where permission for saving to Files might be turned on?  Or could it be that you have such permission by virtue of your role as Developer -a permission which i as ordinary User don't have?

[Silvia Franceschi]

Sorry Ludwick,

I have no super permission on my iPhone, it is definitely standard like the installation.

Could you please send me the screenshots of the step by step procedure?

Thanks 

Silvia 

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/a7249429-72e0-4aaa-bd29-7433def71fc1n%40googlegroups.com.

[Walt Ludwick]

Progress!  Some, at least.  Could not make this procedure work with the "Files" app, but i found a new app- "FE File Explorer" -that did let me save a .gpkg file to the SMASH app (see image 1 attached). 

    NB: I have saved that .gpkg file to both /Maps and /Projects, but it seems the Layers tool can only find it in /Projects. 

Then in the Layers tool, having added the two basemaps i want -Open Street Map, plus the offline map for Portugal- per image 2, i click that icon to right of the globe icon, which allows me to open the .gpkg file you see in image 1... And then i get a list of the many vector layers included in that .gpkg file. Great!  

So i tick the tickbox beside first layer in that list -as you see in image 3 below- and then clicked "OK" button a bottom right... But it brings me back to the same list of layers in image 2 below, without this new layer from the .gpkg file.  I have tried selecting other layers from the .gpkg (all of which have attribute tables in QGIS), individually and as sets, but i always get the same result.  Cannot find any way in SMASH app to view (let alone edit) any data from layers inside this .gpkg file.  Now what am i missing, i wonder?

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/CABCykqJbbZ6nSegUBeSn35w-CAUzsz-cP3z1-nYpTNasTzU4Dg%40mail.gmail.com.

[Silvia Franceschi]

Dear Ludwick,
happy to hear that you did some steps forward! :-)

> Progress! Some, at least. Could not make this procedure work with the "Files" app, but i found a new app- "FE File Explorer" -that did let me save a .gpkg file to the SMASH app (see image 1 attached).
> NB: I have saved that .gpkg file to both /Maps and /Projects, but it seems the Layers tool can only find it in /Projects.

Did you also try to browse the Smash folder using the button at the
bottom of the Layers View?

Regarding the visualization of the gpkg, did you follow the
instruction on our manual for the creation of the package?
There are some rules you have to follow to create a geopackage db
compatible with Smash.
We are now solving some issues regarding the geopackage visualization
and make the creation easier for the users. These new features will be
in the next release which will be released during this or the next
week.

If you can, could you please share with me the gpkg file so we can
test if the new version can load it?

Thanks

Silvia

> To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/CAPR4Z2%3DJ66OjKZQ_8tCHiUXEAS_jPKTjsLQcooAC7bFJpy671w%40mail.gmail.com.

[Walt Ludwick]

Thanks for the good news, Silvia; i look forward to that next release with great interest... And meanwhile, i will read those instructions that i failed to notice, about how to create a Smash-compatible geopackage.  

Also i will send you under separate cover the .gpkg file from QGIS that i'm presently working with, so maybe you can tell me some of the cross-compatibility issues that this presents, that i might avoid them in future. 

As to using the button at bottom of Layers view to browse Smash folder: i have tried, it does let me browse the full Smash directory, but still: when i select the .gpkg in /Maps directory and press save, it does not show up in the comprehensive Layers view, as before.

/walt

On Mon, Sep 21, 2020 at 8:02 AM Silvia Franceschi <silvia.f...@gmail.com> wrote:

Dear Ludwick,

happy to hear that you did some steps forward! :-)

> Progress!  Some, at least.  Could not make this procedure work with the "Files" app, but i found a new app- "FE File Explorer" -that did let me save a .gpkg file to the SMASH app (see image 1 attached).
>     NB: I have saved that .gpkg file to both /Maps and /Projects, but it seems the Layers tool can only find it in /Projects.

Did you also try to browse the Smash folder using the button at the
bottom of the Layers View?

Regarding the visualization of the gpkg, did you follow the
instruction on our manual for the creation of the package?
There are some rules you have to follow to create a geopackage db
compatible with Smash.
We are now solving some issues regarding the geopackage visualization
and make the creation easier for the users. These new features will be
in the next release which will be released during this or the next
week.

If you can, could you please share with me the gpkg file so we can
test if the new version can load it?

Thanks

Silvia

...

[andrea antonello]

Hi Walt,

Thanks for the good news, Silvia; i look forward to that next release with great interest... And meanwhile, i will read those instructions that i failed to notice, about how to create a Smash-compatible geopackage.  

Also i will send you under separate cover the .gpkg file from QGIS that i'm presently working with, so maybe you can tell me some of the cross-compatibility issues that this presents, that i might avoid them in future. 

As to using the button at bottom of Layers view to browse Smash folder: i have tried, it does let me browse the full Smash directory, but still: when i select the .gpkg in /Maps directory and press save, it does not show up in the comprehensive Layers view, as before.

just know that to make SMASH aware of your format (at least partially), today I have not been able to make the new release of SMASH :-)))))

I should be able within this week.

The problems with your geopackage is that the table names and column names contain special characters. While I had some protection for names starting with numbers, containing dashes and/or spaces, I had not thought that names might contain commas.

And for sure I wasn't aware that column names were allowed to have the same issue.

So we worked to make the system more flexible to be able to more easily fix issues that might come up in future.

The solution is:

a) wait for the upcoming release, I have been able to view your data in the upcoming version of SMASH

b) change the names of tables and columns. In the current version in the store table names can't have commas and column names can't have any fuss (so no starting number, no spaces, no comma).

Hope that helps for now. Thanks for this challenging exercise, I am sure it made the geopackage handling a bit better. :-)

Cheers,

Andrea

 

/walt

On Mon, Sep 21, 2020 at 8:02 AM Silvia Franceschi <silvia.f...@gmail.com> wrote:

Dear Ludwick,
happy to hear that you did some steps forward! :-)

> Progress!  Some, at least.  Could not make this procedure work with the "Files" app, but i found a new app- "FE File Explorer" -that did let me save a .gpkg file to the SMASH app (see image 1 attached).
>     NB: I have saved that .gpkg file to both /Maps and /Projects, but it seems the Layers tool can only find it in /Projects.

Did you also try to browse the Smash folder using the button at the
bottom of the Layers View?

Regarding the visualization of the gpkg, did you follow the
instruction on our manual for the creation of the package?
There are some rules you have to follow to create a geopackage db
compatible with Smash.
We are now solving some issues regarding the geopackage visualization
and make the creation easier for the users. These new features will be
in the next release which will be released during this or the next
week.

If you can, could you please share with me the gpkg file so we can
test if the new version can load it?

Thanks

Silvia
...

--

You received this message because you are subscribed to the Google Groups "geopaparazzi-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geopaparazzi-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/CAPR4Z2mHkKcdcsEo0D0w0cH%3D%2Bwd0Tk92D5jOtV9JNjicS9YNVQ%40mail.gmail.com.

[Walt Ludwick]

Thanks for the nuanced explanation, Andrea; this helps.  

Of course it would be great if future version(s) can pre-empt any anomalies that users might introduce in their naming of tables & columns, but meanwhile we users do best to avoid bad practice.  Including spaces in names of tables or columns is of course bad practice, as is the inclusion of commas in column names (which must be prepared for export in CSV form), so i don't do either of these things as a rule.  But this particular case involves a GIS that i've inherited from a colleague, who didn't observe the same conventions as me, i now see;  this is a BIG database (what i sent you is just one small part of it), so data conversion will take me some time. 

Meanwhile: i will certainly avoid those problematic names going forward, and will watch this space for news of the new release.

On Monday, September 21, 2020 at 3:24:44 PM UTC+1 moovida wrote:

Hi Walt,

Thanks for the good news, Silvia; i look forward to that next release with great interest... And meanwhile, i will read those instructions that i failed to notice, about how to create a Smash-compatible geopackage.  

Also i will send you under separate cover the .gpkg file from QGIS that i'm presently working with, so maybe you can tell me some of the cross-compatibility issues that this presents, that i might avoid them in future. 

As to using the button at bottom of Layers view to browse Smash folder: i have tried, it does let me browse the full Smash directory, but still: when i select the .gpkg in /Maps directory and press save, it does not show up in the comprehensive Layers view, as before.

just know that to make SMASH aware of your format (at least partially), today I have not been able to make the new release of SMASH :-)))))

I should be able within this week.

The problems with your geopackage is that the table names and column names contain special characters. While I had some protection for names starting with numbers, containing dashes and/or spaces, I had not thought that names might contain commas.

And for sure I wasn't aware that column names were allowed to have the same issue.

So we worked to make the system more flexible to be able to more easily fix issues that might come up in future.

The solution is:

a) wait for the upcoming release, I have been able to view your data in the upcoming version of SMASH

b) change the names of tables and columns. In the current version in the store table names can't have commas and column names can't have any fuss (so no starting number, no spaces, no comma).

Hope that helps for now. Thanks for this challenging exercise, I am sure it made the geopackage handling a bit better. :-)

Cheers,

Andrea

...

[andrea antonello]

Thanks for the nuanced explanation, Andrea; this helps.  

great!

 

Of course it would be great if future version(s) can pre-empt any anomalies that users might introduce in their naming of tables & columns, but meanwhile we users do best to avoid bad practice.  Including spaces in names of tables or columns is of course bad practice, as is the inclusion of commas in column names (which must be prepared for export in CSV form), so i don't do either of these things as a rule.  But this particular case involves a GIS that i've inherited from a colleague, who didn't observe the same conventions as me, i now see;  this is a BIG database (what i sent you is just one small part of it), so data conversion will take me some time. 

Meanwhile: i will certainly avoid those problematic names going forward, and will watch this space for news of the new release.

Fantastic. 

To everyone: we are experiencing some issues with saving form pictures that we are making sure to fix. 

So there is still some time to test and report issues.

Cheers,

Andrea

 

On Monday, September 21, 2020 at 3:24:44 PM UTC+1 moovida wrote:

Hi Walt,

Thanks for the good news, Silvia; i look forward to that next release with great interest... And meanwhile, i will read those instructions that i failed to notice, about how to create a Smash-compatible geopackage.  

Also i will send you under separate cover the .gpkg file from QGIS that i'm presently working with, so maybe you can tell me some of the cross-compatibility issues that this presents, that i might avoid them in future. 

As to using the button at bottom of Layers view to browse Smash folder: i have tried, it does let me browse the full Smash directory, but still: when i select the .gpkg in /Maps directory and press save, it does not show up in the comprehensive Layers view, as before.

just know that to make SMASH aware of your format (at least partially), today I have not been able to make the new release of SMASH :-)))))

I should be able within this week.

The problems with your geopackage is that the table names and column names contain special characters. While I had some protection for names starting with numbers, containing dashes and/or spaces, I had not thought that names might contain commas.

And for sure I wasn't aware that column names were allowed to have the same issue.

So we worked to make the system more flexible to be able to more easily fix issues that might come up in future.

The solution is:

a) wait for the upcoming release, I have been able to view your data in the upcoming version of SMASH

b) change the names of tables and columns. In the current version in the store table names can't have commas and column names can't have any fuss (so no starting number, no spaces, no comma).

Hope that helps for now. Thanks for this challenging exercise, I am sure it made the geopackage handling a bit better. :-)

Cheers,

Andrea

...

--

You received this message because you are subscribed to the Google Groups "geopaparazzi-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geopaparazzi-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/geopaparazzi-users/9e98e0e4-836c-40c6-a0be-7884066cc84bn%40googlegroups.com.