QGIS and GDA2020

[Nyall Dawson]

Hi all,

Just a quick heads-up, that thanks to recent funding from the ICSM, North Road are currently working on improving QGIS' compatibility and handling of the GDA2020 projections (targeted for QGIS 3.8).

I realise this is a confusing topic for many, so please feel free to ask any questions you have regarding QGIS/other opensource software handling of GDA2020 projections. I'd love to be able to assist and share the lessons we're learning to the wider community here!

Nyall

[Tyson Hillyard]

Hi Nyall,

This is brilliant. I did manage a fix for my current version of QGIS (3.6) by modifying the SRS.DB file - inserting the EPSG parameters for the transformations from GDA94 to GDA2020 into the transformation table. I wasn't successful at getting the Helmert Parameters (conformal) to work, but the grid shifts seem to work as expected. If it is of use to you i can make my SRS.DB file available to you. I would love to be able to use the conformal 7-parameter shift  but haven't devoted any time to testing why my mod is failing. 

Cheers, 

Tyson Hillyard

--
You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.
To post to this group, send email to australian-qg...@googlegroups.com.
Visit this group at https://groups.google.com/group/australian-qgis-user-group.
To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/ac0147fc-ee6f-4494-ab1c-bfdf32880f31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonah Sullivan]

Here is some background information:

 

The old way QGIS reprojected a dataset from GDA94 to GDA2020 is for the PROJ library to convert from GDA94 -> WGS84 -> GDA2020 using a 7-parameter transformation.

 

This is a bad idea because there are errors induced with each step and two steps propagates twice the error.

 

A better way to do the transformation is with a NTv2 transformation grid. The transformation grids for Oceania can be found here:

https://github.com/OSGeo/proj-datumgrid/tree/master/oceania (each grid is about 80MB).

 

The use of these transformation grids is being mainlined so that users know that there is a “recommended” transformation grid with the option to download it.

 

Another issue that needs addressing is time-dependent reference systems. Australia and USA are introducing these in 2020 and 2022 respectively. The addition of a fourth dimension requires more information about the data and a more complex way to store projection data. The method is called WKTv2.

 

More information about the improvements to the GDAL, PROJ, libgeotiff, PostGIS, and Spatialite open source toolchain can be found here.

 

The improvements to QGIS are expected to be in version 3.8, scheduled for release in June 2019.

[Nyall Dawson]

On Tuesday, 12 March 2019 16:18:15 UTC+10, Tyson Hillyard wrote:

Hi Nyall,

This is brilliant. I did manage a fix for my current version of QGIS (3.6) by modifying the SRS.DB file - inserting the EPSG parameters for the transformations from GDA94 to GDA2020 into the transformation table. I wasn't successful at getting the Helmert Parameters (conformal) to work, but the grid shifts seem to work as expected. If it is of use to you i can make my SRS.DB file available to you. I would love to be able to use the conformal 7-parameter shift  but haven't devoted any time to testing why my mod is failing. 

So the situation currently is:

1. QGIS 3.6.0 and 3.4.5 default installs now include the GDA2020 transform files. There's no longer a need to manually install these. BUT

2. The required database updates to notify QGIS of these files won't land until 3.6.1/3.4.6. When those releases come out there wont be any need to manually modify srs.db anymore - the default copy will include the necessary changes.

(for reference - this also includes the older grid transforms for AGD84/66 transformations. Not so relevant anymore, but nice nonetheless).

This "unlocks" QGIS current grid transformation capabilities with GDA 2020, BUT those existing capabilities are rather limited. For 3.8 we'll be addressing their shortcomings and making the whole transformation handling much nicer and simpler for users.

Short story: 3.6.1/3.4.6 will have partial support for GDA2020, 3.8 will have complete support.

Nyall

[Cameron Shorter]

Thanks for the heads up Nyall. I'm currently acting as a Business Analyst at  NSW Spatial Services, in a team focused on upgrading NSW systems to support GDA2020. We'll be interested to hear how you progress.

Cheers, Cameron

--
You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.
To post to this group, send email to australian-qg...@googlegroups.com.
Visit this group at https://groups.google.com/group/australian-qgis-user-group.
To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/ac0147fc-ee6f-4494-ab1c-bfdf32880f31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cameron Shorter

Technology Demystifier

Open Technologies and Geospatial Consultant

M +61 (0) 419 142 254

[middlek]

Hi Nyall, can you confirm if QGIS 3.8 will handle OTF WGS84 (3857)>GDA94>GDA2020? Many of our WA SLIP data services are published in 3857. Cheers Kylie.

[Bryce Touchstone]

I would feel almost certain it will handle WGS84 as it is so widely used (and still GPS default??).

Cheers,

Bryce Touchstone

On Thu, May 9, 2019 at 5:17 AM middlek <midd...@gmail.com> wrote:

Hi Nyall, can you confirm if QGIS 3.8 will handle OTF WGS84 (3857)>GDA94>GDA2020? Many of our WA SLIP data services are published in 3857. Cheers Kylie.

--

You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.
To post to this group, send email to australian-qg...@googlegroups.com.
Visit this group at https://groups.google.com/group/australian-qgis-user-group.

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/836bd538-af87-47a7-97de-f460d39e6dc6%40googlegroups.com.

[br...@frogmouth.net]

Do you really mean 4326? Or 3857 “Spherical Mercator”?

 

Brad

 

[Mark Arnup]

I’ve just received a new aerial photo mosaic in .ecw format in GAD2020 Zone 54. When I imported it into QGIS It wouldn’t line up, and after talking with DELWP they suggested to use the ICSM NTv2 Transformation Plugin to convert it to GDA94 Zone 54. Unfortunately, it only seems to work on vector and not raster files. I was hoping that someone can give me some advice on how to solve my issue. Thanks Mark.

On Tuesday, 12 March 2019 15:46:12 UTC+11, Nyall Dawson wrote:

[Mark Arnup]

I’ve just received a new aerial photo mosaic in .ecw format in GAD2020 Zone 54. When I imported it into QGIS It wouldn’t line up, and after talking with DELWP they suggested to use the ICSM NTv2 Transformation Plugin to convert it to GDA94 Zone 54. Unfortunately, it only seems to work on vector and not raster files. I was hoping that someone can give me some advice on how to solve my issue. Thanks Mark.

On Tuesday, 12 March 2019 15:46:12 UTC+11, Nyall Dawson wrote:

[Barrett Higman]

Hi Mark,

 

I recently mucked around with GDA2020 in QGIS (v3.6.3) and found that I had to download the ICSM-AU transformation grids and manually apply them to the specific project.

 

The transformation grid can be downloaded here: https://github.com/icsm-au/transformation_grids/blob/master/GDA94_GDA2020_conformal_and_distortion.gsb and it needs to be placed in the C:\OSGeo4W64\share\proj folder (or the equivalent for your set up).

 

You then need to open the CRS properties for the project and click on the plus button to add a Datum Transformation:

 

 

Select the Source and Destination CRS (in the case of the file downloaded in Step 1, these need to be EPSG28355 and EPSG7855 respectively). Select the transformation you want to use (there is an additional transformation grid you can download from the same location as the first):

 

 

Click OK and this transformation will be applied to the project

 

 

This should sort out the differences you see.

 

Cheers,

Barrett

--

You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.
To post to this group, send email to australian-qg...@googlegroups.com.
Visit this group at https://groups.google.com/group/australian-qgis-user-group.

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/93b7c8b8-c455-483b-ae84-d1aac4662022%40googlegroups.com.

[Nyall Dawson]

On Wednesday, 12 June 2019 16:10:22 UTC+10, Barrett Higman wrote:

Hi Mark,

 

I recently mucked around with GDA2020 in QGIS (v3.6.3) and found that I had to download the ICSM-AU transformation grids and manually apply them to the specific project.

Hey Barrett!

Thanks for the detailed write up here. I have one question for you -- the transformation grids should be available out-of-the-box on a 3.6.3 default install, so there should be no need to manually install these. How have you installed QGIS? Do you use the standard installer from qgis.org, or manually via osgeo4w?

Nyall

[Barrett Higman]

Hi Nyall,

 

I’m glad you asked as I remember reading that this was meant to be the case and it led to some head scratching for a bit.

 

I install manually from the osgeo4w installer.

 

Cheers,

Barrett

 

 

 

 

From: australian-qg...@googlegroups.com [mailto:australian-qg...@googlegroups.com] On Behalf Of Nyall Dawson

Sent: Monday, 17 June 2019 9:37 AM
To: QGIS Australia User Group

--

You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.
To post to this group, send email to australian-qg...@googlegroups.com.
Visit this group at https://groups.google.com/group/australian-qgis-user-group.

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/9bbb07fa-f586-4919-8577-8a4d9ffe0244%40googlegroups.com.

[Luke Kirkwood]

Dear All,

ISSUE: Convert text strings of years to Date field type

This is more of a continuation of a discussion to a response by Nyall to a bug report I lodged (Thanks Nyall), but was unsure if appropriate to continue the conversation on Github, so decided to approach the brains trust on the Oz QGIS list for an Australian problem.

I have a series of years as a text string for Australian archaeological sites ranging from -60000BC to 1950AD. I had tried to convert this string to a date field type, but my usual amateur method of stumbling around in GIS eludes me. I even tried manually typing the dates in the field, but for some reason the date field type in Geopackage, shapefile and spatiallite in QGIS won’t let me put a date earlier than 1/1/100. I think this is the date widget which Nyall refers to in the post linked above, but also it appears you can't type a value older than 1/1/100 into the field in the attribute table either. I’ve tried the field calculator to try to convert, but again amateur stumbling hasn’t presented a solution.

I’ve provided some sample data below and attached as CSV (saved you all from the 5000+ records we have). Basically in archaeological radiometric dating we get a date range which I have provided as oldest_date and youngest_date in the table (think of it as 95% confidence that the actual date falls within this range). These dates (-15170 for example) should be read as 15170 years before the present, where the present is defined as 1950AD.

Nyall mentioned QDateTime and googling that suggests a python solution which as an amateur is way beyond my capabilities/understanding of QGIS. If anyone has any suggestions on how to do this, would greatly appreciate it.

Cheers

Luke Kirkwood

[Ia...@jcis.net.au]

Dear Luke,

 

I think you should simply become a Historical Archaeologist – problem solved.

 

I am sure however that other archaeologists must have encountered this problem. I am not sure whether there is a QGIS archaeology group though.

 

Cheers

 

Dr Iain Stuart

JCIS Consultants

P.O. Box 2397

Burwood North

NSW, 2134

 

(02) 9701 0191
(0413) 380116 (m)

--

You received this message because you are subscribed to the Google Groups "QGIS Australia User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to australian-qgis-use...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/7F56D428-B039-4D7B-B334-A373E37DF875%40gmail.com.

[Luke Kirkwood]

LOL, that’s your solution for everything Iain. Unfortunately, I’m doomed to being a generalist, who can fake his way around a computer.

If you hear of QGIS Archaeology group though, let me know. There must be a market for it.

LK

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/00bc01d60baf%247e3d7eb0%247ab87c10%24%40jcis.net.au.

[Juan Fernando Berrío]

When the date range is so wide, usually you don't need a "date" data type anymore.  Storing the year values as integers, or as floats is sufficient.  

Most apps / python packages / etc. have a limitation on the range of dates (think MS Excel, python, pandas, etc).  Geologists, archaeologists, astrophysicists and alike frequently bump into those limits pretty fast when they use their datasets.  The common workaround is to store the years as floats, as mentioned.

Maybe you can just do that?  Or is there a reason why you need "date" data type for that range of dates?

Hope it helps.

Juan

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/0AD34C10-913E-4029-A3D9-8FA5C6820947%40gmail.com.

[Luke Kirkwood]

Thanks for your comments, Juan and I agree you are probably right about working within the confines of the tech.

I was trying to use the TimeManager plugin by Anita Graser to create a movie to show archaeological sites with different occupation dates over time.

https://anitagraser.com/projects/time-manager/

My understanding was that I need the year to be in a date format in order for this plugin to work. Again, professing my lack of skill in all things QGIS. Maybe there is a better way of achieving this.

Cheers

Luke

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/CADtK-_mEbKXBrdpe4%3DqeKicf%2B3Yqpo8bkue69L91jEvEvbJfnA%40mail.gmail.com.

[Jonah Sullivan]

I don't think that plugin requires a date column, just a column that can be converted to a date. Any date-like string should work.

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/7F16DD65-219C-4276-B829-8963DDE3FAFB%40gmail.com.

[Juan Fernando Berrío]

But Luke won't be able to produce date-like strings with the dates he has in his dataset... 

I can see the problem now Luke.  You're right, it is clearly expressed in the documentation of the plugin that it is limited by that range of years.  

Because the plugin has a few 'datetime' controls, like the slider in the GUI, and some date input boxes, then dates outside of the defined range will make it crash.  

It's not that easy to change the range of the 'date' data type.  But it is possible to take Anita's plugin, make a copy and make a few changes (it's called a 'fork' of the original project), and make it work with floats instead of dates.  

Furthermore, it will be useful for anyone with an 'undefined' time unit in their dataset (say your data is in seconds, or in thousands of years, you still can skip the whole 'date' thing and just use a number).  To get there, a few controls will need to be changed in the GUI of the plugin, and some of the code will need to be edited.  

I wish I could help you with that, but my day job that pays the bills doesn't leave too much free time at the moment (I'm snowed under, even when WFH and socially distanced). 

Maybe someone else can give you a hand with that?  I hope someone's keen on it, it will be a fun project to do and exercise the brain a bit in these days of solitude.  

Cheers

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/CAAR82xXx7BXdLNZ%2BDXEnHNRNjRHMwYbZTwzT4FNgCXKA6Zi-Ww%40mail.gmail.com.

[Tyson Hillyard]

There appears to be an archaeological mode in the the Time Manager Plugin. Have you explored that? allows years from 0001 BC  onwards. Reading the dialogue i would assume it will require your date to be 600000 BC to 001950 AD with the number of digits set to 6. see screenshot below:

Does that help at all?

Cheers,

Tyson Hillyard

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/CADtK-_k08MJ50p%2Bfg93MopW5raYpm8Lp%2Bv0nALrnObEpucNbDA%40mail.gmail.com.

[Juan Fernando Berrío]

Tyson... that's gold! 

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/CALzsxrmzwoek%2BscytrSJ7mOckK5uqR6qT%3DEKHJV%2BaHSA8mo05A%40mail.gmail.com.

[Luke Kirkwood]

Wow, thanks Tyson, I did not even see that button. To be honest, I thought it was some divider between the actual buttons. I shall give it a go and let the group know if it worked.

Cheers

Luke

On 6 Apr 2020, at 2:57 pm, Juan Fernando Berrío <juanb...@gmail.com> wrote:

Tyson... that's gold! 

On Mon, 6 Apr 2020 at 14:56, Tyson Hillyard <tyson.h...@gmail.com> wrote:

There appears to be an archaeological mode in the the Time Manager Plugin. Have you explored that? allows years from 0001 BC  onwards. Reading the dialogue i would assume it will require your date to be 600000 BC to 001950 AD with the number of digits set to 6. see screenshot below:

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/CADtK-_%3DnicuNRVN-%3D-SbcZpxHNX8StoibyjSeC4g9o3cW89_0A%40mail.gmail.com.

[Luke Kirkwood]

Well there was a bit of data processing involved, but I finally ticked all the boxes to get it to work. Here is my first draft. I’ll make a higher resolution one and submit it to the Time Manager youtube channel later today.

https://www.youtube.com/watch?v=ne4WP6JAcdo&feature=youtu.be

Thanks again for everyone’s help.

Luke

[Tyson Hillyard]

Good work!

Glad I was able to assist. I happened to be using the time manager plugin at the same time and just stumbled across the archaeological mode. 

Cheers,

Tyson

To view this discussion on the web, visit https://groups.google.com/d/msgid/australian-qgis-user-group/66A0EEF4-BC7C-428C-925E-B3E62F9B15FF%40gmail.com.