Is anyone actually successfully running open-source CartoDB in production?

[Nick Ballenger]

Hi.

I am consulting with a small non-profit that would like to run the open-source version of CartoDB on their own (cloud) infrastructure. Because they would potentially like to run it at scale (or at least know it won't catastrophically tip over under variable load), I'm attempting to run it as a collection of services rather than an impossible to scale single-host install.

Unfortunately I am repeatedly running into severe configuration and dependency problems that impact core features of the software, and trying to troubleshoot them without good ops documentation is very frustrating.

So, my questions:

• Does anyone know of any definitive operations info for running open-source CartoDB on more than one host?
• Are there any organizations or individuals out there who are successfully running it, and who might be available to connect or consult?

For what it's worth, I have a 'working' multi-container Docker solution. I put working in quotes because while it runs and performs some basic tasks, there are features failing due to (it appears to me) inter-service (or even intra-service? who knows?) communication failures. There are clearly some vital architecture details I'm missing. 

For instance, specifics of how the invalidation service is supposed to function do not appear anywhere I've looked, and it appears to be an important part of the overall application, being relied on by multiple other components. From the example CartoDB app configurations, that service claims to be available on port 3142, which is a non-standard port for almost everything. Additionally, parts of the codebase refer to invalidating the varnish cache, but the client object (obtained in the PlPython Postgres functions for dealing with "ghost tables") which hits the invalidation service endpoint is a Redis client. And, to make it more confusing, in at least one Carto company slide deck I saw the invalidation service referred to as a Node app, for which I've been able to find no source code at all.

I need someone who can confirm this thing can actually be run by anyone other than Carto. I'm extremely close to telling my client that the software simply isn't usable, and that they need to begin evaluating other geoservers.

Information sources I've looked at so far:

• The official 'CartoDB Platform Documentation' - Possibly out of date, focuses exclusively on a single-host install (so dependencies and configs are inherently unclear for multi-host configuration params)
• The official 'CARTO Developer Center' - Operations info here for on-premises relies on a pre-built CentOS/RedHat binary, and does not (so far as I have found) go into running a multi-service or multi-host setup.
• The CartoDB Google Group - Infrequently updated, so far have not found any deep dives on product architecture or ops, aside from single-issue troubleshooting.
• Stefan Verhoeven's docker-cartodb Docker project - Extremely helpful, but remains a single-container solution, so not enough info on inter-service connections
• Chris Whong's docker-cartodb project focused on HTTPS - Also extremely helpful for HTTPS related issues, but still single-container
• The Institute for Health Metrics and Evaluation's multi-container Docker project - Helpful but has less documentation than I need to really understand the stack architecture

If anyone has any leads on someone (anyone!) able to explain how to successfully operate open-source CartoDB, I would very much appreciate them.

Thanks,

Nick

[Anthony Esper]

Hey Nick

Welcome to our world. In a cloud world, having cartodb get up and running is pivotal. We currently have a cartodb development instance running on AWS EC2. I found these instructions to be very helpful.

http://michaelminn.net/tutorials/carto-aws/  

I'm going to invite you into our slack channel and hopefully together we can put this together. The support from this group and Carto team is very limited. I live in NYC where they are based and could not get anyone on the phone. I'm half tempted to go to their office. There is definitely a revenue, go to market plan for extending open source into a limited license. I wish they would see that but that is up to their business development team. In the meantime, I am offering to put our heads together. I will be sending you an email.

Tony

--

---
You received this message because you are subscribed to the Google Groups "cartodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cartodb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/98b46d76-e6b5-4abe-b7f8-adc686fe2d28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Anthony Esper Founder | Occupi NYC m: 857.540.3450 p: 718.749.9422 x102 e: aes...@occupi.nyc 159 20th St. Brooklyn, NY 11232 www.occupi.nyc

[Elahi S]

Dear Nick,

Thank you for taking the time to read this message. I am so glad I found this post and hopefully you can shine some light into the Carto-AWS installation. I been using the instructions from Micheal Minn in combination with the latest instructions from Carto. 

I got to the point in which my AWS instance works. I can upload a csv file and create a map. Everything seems to be working but it the crashes within eight hours and I have to stop and restart carto for it to work again.

The Carto instructions call for Node.js 10.x and when I follow this path it asks for Bundler 2  and if I continue the install it will not work for me. When using Node.js 12.x (not in the instructions) and I install Blunder 2 (not in the instructions) all seems to be working and I can get the login screen, upload a csv, and create a map. This however is short lived I get the message: Process has received signal: SIGTERM and then port 22: Broken pipe. I think this might be related to a faulty Node.js configuration as it relates to CartoDB-SQL-API and Windshaft-cartodb.

How did you resolve the Node.JS installation in AWS? Or I am looking in the wrong direction?

Again, thank you for your time.

Regards,

S. Sergio Elahi

To unsubscribe from this group and stop receiving emails from it, send an email to car...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/98b46d76-e6b5-4abe-b7f8-adc686fe2d28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brock Friedrich]

Hi Sergio,

Nick, Anthony, and I put our heads together last year to work through the various issues with standing up Carto.  We finished our projects and went our separate ways at the end of last year. I can't speak much to your issue directly, but I have some resources that will maybe help you out.

1. Anthony did a lot of great work adapting the popular dockerization of Carto by Sverhoeven to run on AWS EC2 behind an ALB.  His version runs all of the services in a single container on an EC2 host and has all of the Carto features working: https://github.com/la-mar/cartodb-sven-revamp.

2. Nick put a ton of work into attempting to pull the Carto components apart into separately scalable services.  He put together a lot of documentation and added a lot of comments to the code base detailing many of the Carto nuances.  His project is here: https://github.com/ruralinnovation/multi-svc-cartodb.

3. You can check out the Slack channel we used when figuring out how to stand up Carto in AWS.  There may be some discussion in there that can shed some light on various hurdles you'll likely cross along the way, though there hasn't been much discussion for the past few months.  Slack invite link: https://join.slack.com/t/carto-workspace/shared_invite/zt-f6j4nhsx-Atn6btB3cnFPDd5700PFCQ.

Both of the projects above stick to Node 10.x, so I'm not sure how much luck you'll have trying to run Node 12. Best of luck!

Thanks,

Brock Friedrich

To unsubscribe from this group and stop receiving emails from it, send an email to cartodb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/60f9a2d3-b2c3-43bc-8775-08782bb704a8o%40googlegroups.com.

[SIRAJ SERGIO ELAHI]

Brock,

Thank you so much for the response and providing these resources. Now, I been able to run Carto locally without any crashes and I will be moving the installation into the cloud.

Best of wishes! To you and family during these changing times!

Sergio Elahi

---

From: car...@googlegroups.com <car...@googlegroups.com> on behalf of brocklf...@gmail.com <brocklf...@gmail.com>
Sent: Wednesday, June 10, 2020 5:10 PM
To: car...@googlegroups.com <car...@googlegroups.com>
Subject: Re: [cartodb] Is anyone actually successfully running open-source CartoDB in production?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/CAJwOCcDq7fnwvdAxhpGkUbBePJhssfHi5%2BAZko3CC5HZkWKmpg%40mail.gmail.com.