Catalog URL change to https not saving and new catalog using https disappearing

[osjam...@gmail.com]

Hello, I added this to an existing closed call (MapStore Secure Connection) as it is related, but I thought you might like a new call submitted.  The enquiry is:

Hi Lorenzo (or anyone in the group!),

We have made good progress, but have one final issue which is confusing us.

We have installed an NGINX web server/proxy on the MapStore server between the Tomcat server and user requests.  NGINX has been configured to manage the translation between an incoming https request to a local request to the Tomcat server for both the MapStore server and the GeoServer server.  This is working as required and manages the connection to the Tomcat servers on port 8080 to an https connection with a certificate over port 443.
 
However, problems occur when adding GeoServer layers into a MapStore context.

The main issue is that the http URL in a catalog (configured before the installation of NGINX) is used when MapStore makes multiple failed calls to GeoServer to load layers from GeoServer.  We have tried to resolve this in two ways.

Firstly, changing an existing catalog.  Using the catalog editor, the URL for an existing catalog is changed from the http connection to the https connection and saved with the ‘Success’ message appearing.  But when the context is closed and re-opened, the change is lost and the original http URL is shown.  But any layers added using the https connection remain visible on the map.  For these layers, attribute information is returned when querying objects on the map (the https connection seems to be used for this), but the attribute tables don’t return any features.

Secondly, adding a new catalog created using the https connection to a GeoServer workspace.  It is possible to use the catalog to load layers from GeoServer, but when the context is closed and re-opened, the catalog disappears.  But, as before, the added layers remain visible on the map.
 
Would you, or anyone in the group, have any thoughts on why the changed URL to https returns to the original http URL or why a new https catalog disappears when re-opening a context?

Many thanks

Owen

[osjam...@gmail.com]

Just an additional note to this.  I have tried to create a new catalog to another data source and the issue is replicated with this data source, so the issue seems to be specifically related to MapStore 

[Lorenzo Natali]

Hi,

It looks a common problem: the URLs returned by GeoServer contains the http (not https) URLs to their services. It is a common problem and can be solved by setting the proxy base URL in GeoServer, or (better) using the proper headers to forward the https info (e.g. X-Forwarded-Proto ) 

https://docs.geoserver.org/stable/en/user/configuration/globalsettings.html

About the maps saved, when a layer is added, it is detouched from the catalog, so changing the catalog will not have effect to the existing layers. If you want to change in bulk the URLs for your layers in the database, I think you will have to work on the database with a big string replacement.

A suggestion for the future: 

if MapStore and GeoServer are on the same domain, use `/geoserver/wms` and so on to define the catalogs. This helps to maintain the URLs of the layer when moving the installation from one server to the others.

[osjam...@gmail.com]

Hi Lorenzo,

Thanks for the advice, we will look into this.

However, this doesn't seem to resolve why new catalogs created disappear when re-opening the context, unless I have misunderstood you.  We have also created a new catalog using a source which is not our GeoServer ( http://essex.astuntechnology.com/getows.ashx?mapsource=Essex/Inspire_Brentwood&version=1.1.1&language=English) which also disappears.

Also, for further information, when querying the data on the map loaded from a new catalog to our GeoServer, results appear OK for individual features.  But, when opening the attribute table for the layer, no features are returned and the console indicates:

POST http://localhost:8080/mapstore/proxy/?url=http%3A%2F%2Fmaps.brentwood.gov.uk%3A8080%2Fgeoserver%2FBrentwoodLocalPlan%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DDescribeProcess%26IDENTIFIER%3Dgs%253APagedUnique 404

GET http://localhost:8080/mapstore/proxy/?url=http%3A%2F%2Fmaps.brentwood.gov.uk%3A8080%2Fgeoserver%2FBrentwoodLocalPlan%2Fwfs%3Fservice%3DWFS%26version%3D1.1.0%26request%3DDescribeFeatureType%26typeName%3DAQMA_Active_LDP2021%26outputFormat%3Dapplication%252Fjson 404

Is the inclusion of ' http://localhost:8080/mapstore/proxy/?' expected?  I notice that the URL is http, will your resolution to set the proxy base URL in GeoServer rectify this?

Many thanks

Owen

[Lorenzo Natali]

Hi,

Maybe we had a misunderstanding in the term disappears. 

Do you mean that you create a context, you save and open the catalog then the catalog is not present? 

The map at step 2 of the context-editing allows to edit the base map. The tools you want in the final map (included catalog) can be selected on the 3rd step. 

Because the catalog can be reached from the List of layers, from the Sidebar or from the Burger Menu), by default the catalog do not bring a container plugin. 

You should add "List of layers", "BurgerMenu" to be able to open it.

From "list of layers, the catalog is available from here ("Add Layers") 

About the failing requests , the WPS is normal. MapStore tries to find out some services from GeoServer to see if it can activate advanced features.

The other one means that you don't have WFS service available, and for this reason you will not be able to see the Attribute table 

[osjam...@gmail.com]

Hi Lorenzo, yes, we are misunderstanding each other.  The Catalog and List of Layers plugins have been added to the context and are always accessible.  But the problem occurs when we create a new catalog in the catalog plugin.  The catalog plugin is still accessible in the context when it is re-opened, but the new catalog created has gone (disappeared) from the list of catalogs available in the plugin

[osjam...@gmail.com]

Create Catalog: 

In the Catalog list: 

Re-open context and the new catalog has gone: 

[osjam...@gmail.com]

Opening an attribute table shows no features: 

[Lorenzo Natali]

About the catalog, it works for me. I attach the video. (tested on current stable 2022.02.00). I had to use https instead of http.

About the attribute table, this is caused by the proxyURL of GeoServer. If you see your server 

The URL passed by the proxy (used to avoid cross origin problems)

 http://localhost:8080/mapstore/proxy/?url=http%3A%2F%2Fmaps.brentwood.gov.uk%3A8080%2Fgeoserver%2FBrentwoodLocalPlan%2Fwfs%3Fservice%3DWFS%26version%3D1.1.0%26request%3DDescribeFeatureType%26typeName%3DAQMA_Active_LDP2021%26outputFormat%3Dapplication%252Fjson 404

Is http://maps.brentwood.gov.uk:8080/geoserver/BrentwoodLocalPlan/wfs?service=WFS&version=1.1.0&request=DescribeFeatureType&typeName=AQMA_Active_LDP2021&outputFormat=application%2Fjson

This is the address returned by DescribeLayer service to MapStore by GeoServer, used to recognize the WFS service associated to the WMS layer you added.  

And it is not reachable, so MapStore can not see what are the attributes of the feature.

[osjam...@gmail.com]

Hi Lorenzo, I will go through all the other points with my IT team.  Thanks for the video about the catalog, I have now added the new catalog with the context in editing mode and this stays in the context.  Before, I was adding the new catalog simply by opening the context and adding the catalog, but didn't realise that the catalog would only remain in the context if adding the catalog in editing mode, apologies for the user error!

[osjam...@gmail.com]

Hi Lorenzo, we have looked into this further.  I have added a new WFS catalog to the same data as the WMS catalog and loaded the WFS layer onto the map.  I have then opened the attribute data for the WFS layer and this has returned successfully.  Would this mean that WFS is working for GeoServer?  However, when this is done, the console states:

GET http://localhost:8080/mapstore/proxy/?url=https%3A%2F%2Fmaps.brentwood.gov.uk%2Fgeoserver%2FBrentwoodLocalPlan%2FAQMA_Active_LDP2021%2FBrentwoodLocalPlan%2Fwfs%3Fservice%3DWFS%26version%3D1.1.1%26request%3DGetCapabilities 404

POST http://localhost:8080/mapstore/proxy/?url=https%3A%2F%2Fmaps.brentwood.gov.uk%2Fgeoserver%2FBrentwoodLocalPlan%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DDescribeProcess%26IDENTIFIER%3Dgs%253APagedUnique 404

Are these 404 errors significant despite the attribute table being displayed?

As far as I can tell, GeoServer is enabled for WFS:

Could you please let me know if there is anything else we can check in GeoServer for WFS to work?

We are confused that WFS data can be loaded into MapStore and the attribute data displayed, but using WMS data, attribute data cannot be displayed.

Many thanks for your further help.

[Lorenzo Natali]

Hi, 

the first one is needed for showing the feature grid, the second request checks for WPS service to activate/deactivate for automatic suggestion in editing, but it is not mandatory. 

In practice this URL not responding

https://maps.brentwood.gov.uk/geoserver/BrentwoodLocalPlan/AQMA_Active_LDP2021/BrentwoodLocalPlan/wfs?service=WFS&version=1.1.1&request=GetCapabilities

This is the per-workspace WFS service, corresponding to the per-workspace WMS service, returned by the DescribeLayer. 

Anyway it has an additional slash I can not understand:

 BrentwoodLocalPlan/AQMA_Active_LDP2021/BrentwoodLocalPlan

Normally these paths are 

• /geoserver/wfs
• /geoserver/<workspace_name>/wfs
• /geoserver/<workspace_name>/<layer_name>/wfs

In this case it seems to be <workspace_name>/<layer_name>/<workspace_name>. 

Strange. Maybe you should investigate this.

--
You received this message because you are subscribed to a topic in the Google Groups "mapstore-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mapstore-developers/GxmY-WSlSxc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mapstore-develo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mapstore-developers/568ad69d-6937-441b-92f6-a6ec3f2a837fn%40googlegroups.com.

[osjam...@gmail.com]

Hi Lorenzo,

Thanks for all your help, it has been useful to gain further understanding of the mechanics of MapStore.

We were struggling to work out what was happening with the URLs, but then I decided to remove the layers from the map and then reload them and then I was able to access the attribute tables for each layer.

I then created an entirely new context, loaded layers and again was able to open the attribute tables.  The lesson I have learnt is if there are any major changes in the configuration of MapStore and, as in this case, configuration of the security for the server, any existing contexts may not work as expected.

Today, I re-opened the new context and the only issue which happened today was that at first querying the objects on the map did not work, showing no results.  This started working after closing and opening the context again, so I will test this further to see if this issue re-occurs.

Again, thanks for your help, it looks like our issues are resolved for now.

Regards

Owen

[Lorenzo Natali]

Good to know. 

I hope you will solve the rest of the problems.

Thank you for using MapStore.