Carto / Windshaft cache

[Oskar Dolch]

Hello,

I bring the old topic here again since it turned out that the cache is sometimes not refreshed even in the "official" Carto. Try to move a point in Builder to and fro couple of times and sometimes it happens that the point is not refreshed properly until zoomed in and out again. Is this issue to be addressed and fixed somehow? Or the Carto design is not intended for frequent geometry editing?

Thanks, Oskar

[Gonzalo Riestra Menéndez]

Hi Oskar,

I've been trying to reproduce it on carto.com, but it works fine for me. Could you indicate the steps in detail? 

[carto lover]

i don't know if you refer to this kind of problem

but happens also on my instance.. i have to remove cache from browser to remove error

inspecting:

{"errors":["Invalid or nonexistent map configuration token 'c2ea575ea6fa1a0fc94d456ac0cc3980'"],"errors_with_context":[{"type":"unknown","message":"Invalid or nonexistent map configuration token 'c2ea575ea6fa1a0fc94d456ac0cc3980'"}]}

[Oskar Dolch]

Hi carto lover,

I think this is a different issue.

[Oskar Dolch]

Hi Gonzalo,

see the recordings here: https://youtu.be/vyk-5MPxAns

Thanks, Oskar

[Gonzalo Riestra Menéndez]

Thanks for the video, Oskar. I've finally managed to reproduce it, so I've created a new issue to fix it: https://github.com/CartoDB/cartodb/issues/15609

I'll keep you updated.

[Javier Santana]

Hey

I don't think you can easily fix that problem. When you open a map a
request (instantiation) is done to get the url for the tiles (you'll
see a jsonp call most likely). That url is composed by a hash of sql +
style (more or less) and a timestamp (the last table update) so if a
table changes (and it changes when you add geometry) you'd need to
call that map request again in order to get new tiles for the tiles
(because the timestamp changes)

Even if you clear the browser cache, there could be other caches in
the middle with those tiles cached. You see some zoom levels are fine
but that's because they haven't been cached.

That's a known carto limit and actually was designed like that. If I
recall correctly there is a way to get tiles from a named map (what
carto embed is using under the hood) without having to "instanciate"
the map that is able to deal with map changes but still you'll need to
deal with browser cache.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cartodb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cartodb+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/6d64456e-40e1-4ab6-a359-2f15e7432044%40googlegroups.com.



--

https://tinybird.co - Data
http://javisantana.com

[Oskar Dolch]

Thanks Javier, Gonzalo,

so does it mean that Carto is not designed for "heavy" editing? Is it possible to use a GeoJSON layer in Carto? I ask since it would be a viable workaround to use a GeoJSON layer for edits, since it would be rendered on the client avoiding any cache. I've manage to do something like that recently in a Carto VL based map.

Oskar

> To unsubscribe from this group and stop receiving emails from it, send an email to car...@googlegroups.com.

[Ladislav Capek]

Dear Javier, Gonzalo,

Carto is very nice product and your team did very good work! I like your very fast and nice front and I also like the idea on back end. Thinks as Federate tables with Postgre is something really challenging. The only one think, which I do not understand, why you cannot fix this bug with Windshaft cache. We cannot use features in Builder like every day update of zipped shape files, because it simply does not update correctly the map for end user. Such bug is making many difficulties for all users.

Would you please fix it? Is proposed way posted by Oskar above working? Thank you for help.

Regards/Ladislav

[Simón Martín]

Hi all

First of all, thanks for your explanations and the video showing the problem, I would like to receive feedback always like this.

I think Javier Santana explained it very well. In the ticket, I have done my best trying to explain how it works with an example: https://github.com/CartoDB/cartodb/issues/15609#issuecomment-621864798

We cannot use features in Builder like every day update of zipped shape files, because it simply does not update correctly the map for end user. Such bug is making many difficulties for all users.

 

I want to clarify this situation happens only when you have the map loaded and the data is updated after that. So, in builder, if a new user comes to your map, s/he will always see the data updated. And of course, you will see if you refresh your page. 

HTH. Regards!

[Daniel García Aubert]

Hi Ladislav and Oskar!

We are happy to get feedback like yours to help us to improve the platform.

I've seen the video which is helpful to understand your use case.

You're editing a map in a tab of your browser and in the other one you have the published map loaded already. If you edit the map in the first tab then the published map (public view) isn't going to be updated automatically, you need to refresh the tab to get the fresh data. 

What you are seeing in the second tab is the published map with mixed content, the stalled one because it's cached and the updated data as long as you are zooming and moving into the map and the tiles weren't previously fetched. That's something that only happens when you update the data and you don't publish the map again and re-load the public view. So the workflow is: edit the map, publish it and (re)load the public view.

That's how the product is designed there is no such thing as Real-Time CARTO (yet). We can mitigate the use case making the public view a bit smarter and detect when the content is stalled and then notify the user to refresh the public view.

I hope this helps a bit.

[Ladislav Capek]

Hi Chavier, hi Gonzalo,

first of all, thank you for your fast responds! You do really great job and you explain how Carto works very well. I understand, I need to update map view.

Anyway, this solution does not work, because we are using external data synchronization with zipped shape files or Federate tables with Postgre, so we do not know when anybody would change those data. The solution is, as you mentioned, some kind of "Real-Time Carto".  Could you do any "fast hack"? ...or what would be your plan or recommendation how we could fix it?

You know, those days are some external data sets changing much faster. Even some statistical data which were stable for years are now, because of Covid_19 changing within weeks. So such feature is limitation for using Carto, which is otherwise really great product. I think, this "real time Carto" would be great advantage and help to all Carto community... Thank you.

I wish you both to have a nice Labour Day tomorrow!

Ladislav

[Javier de la Torre]

The workaround we considered was calling directly the maps api with the same map definition every 5sec... so if the data has not changed, the maps api will return the same token as before, and therefore nothing to be done. If the token changes is because the underlying data has changed, and therefore you need to retrieve new tiles with that token.

That way the map can always be outdated to a max of 5 seconds.

Of course all of this would had to be done on JS on a custom map. Implementing it on Builder to have that "auto-refresh" map will take longer of course as the user might want to decide how often to check for new maps updated and things like that. And then adding that at carto.js and many other areas.

At least this might give you a direction for a custom app solution.

Best,

Javier.

To unsubscribe from this group and stop receiving emails from it, send an email to cartodb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/7292873b-e988-4c1b-8f40-61dc3d5143b4%40googlegroups.com.

[Daniel García Aubert]

Hi!

> Anyway, this solution does not work, because we are using external data synchronization with zipped shape files or Federate tables with Postgre, so we do not know when anybody would change those data. 

If you update the data through any external sync process or even manually, and you refresh the tab where the public view is (w/o republishing the map), the map loads with the updated data.

The case of Federated Tables is different as CARTO doesn't know when the original data (in a remote database) changes, so the tiles only live for 5 minutes in our cache system. If you have a loaded public view in a browser's tab and then, after 5 minutes, you zoom in and out you'll get tiles with the updated data (w/o reloading the map). Mind that a map that uses Federated Tables might not be as performant as a regular map in CARTO as the latency between CARTO and your database can be high.

> You know, those days are some external data sets changing much faster. Even some statistical data which were stable for years are now, because of Covid_19 changing within weeks.

We have several use cases where the data is updated every hour or even less and CARTO fits really well as users usually don't keep a map open in their browsers for hours or days. Also, users usually reload the map if they want to fetch changes.

Best,

[Oskar Dolch]

Hi Javier,

how do you mean to "call directly the map api"? We have also tried to edit data from a carto.js application, and there were similar issues with refreshing the data. There is no method to refresh a carto layer (the method Invalidate seems to have been deprecated in version 4.x) and there is nothing like "reload" and carto source. I also cannot see a way to reassign a source to the layer. The only workaround we've found was to assign a dummy filter to the source and remove it immediately. This somehow worked.

So can you elaborate a bit more on this? Or did you mean using another engine than carto.js?

Thanks, Oskar

To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/7292873b-e988-4c1b-8f40-61dc3d5143b4%40googlegroups.com.