Invalid provider Provider: postgres

[Olivier Vernin]

Hi,

I'm trying to enable postgres connector as described here https://carto.com/developers/import-api/guides/database-connectors/#the-postgresql-connector but no success so far. Can anyone help?

The initial API call to create a connector was returning that the "posgres connector wasn't enabled".

Then i ran the following commands that seem to enable to connector feature properly:

RAILS_ENV=production bundle exec rake cartodb:features:add_feature_flag['carto-connectors']
RAILS_ENV=production bundle exec rake cartodb:features:enable_feature_for__all_users['carto-connectors'] 

 

Creating the provider through the API

When calling again the import API to create the connector, I get a different error:

"get_error_text": {
    "title": "Connector invalid parameters",
    "what_about": "The connector couldn't be configured because the parameters were not valid.",
    "source": "user"
 },

in the logs, one can read 

error: Uncaught Resque failure
#<Carto::Connector::InvalidParametersError: Invalid provider Provider: postgres>

Creating the provider through the command line

When trying to create the providers, no providers are created 

RAILS_ENV=production bundle exec rake cartodb:connectors:create_providers  <~ no providers are created as the list of providers is empty

 

When trying 

RAILS_ENV=production  bundle exec rake cartodb:connectors:set_user_config["postgres","myuser","true","unlimited"].     <~ returns "Provider not found: postgres"

Thank you 

Olivier

[carto lover]

Read here

https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!topic/cartodb/Pu292VQIgFM

Then edit also config.yml to enabnle postgres mysql and etc and restart carto services

--

---
You received this message because you are subscribed to the Google Groups "cartodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cartodb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cartodb/02deb534-a787-46a8-9326-8a43d345d11do%40googlegroups.com.

[Olivier Vernin]

for the config.yaml do you mean those lines?

connectors:
    odbc:
      enabled: false
      max_rows: nil
    mysql:
      enabled: true
      max_rows: 500000
    postgresql:
      enabled: true
      max_rows: 500000
    hive:
      enabled: false
      max_rows: 500000
    sqlserver:
      enabled: false
      max_rows: 500000
    bigquery:
      enabled: false
      max_rows: 1000000

To unsubscribe from this group and stop receiving emails from it, send an email to car...@googlegroups.com.

[Nico Aliotta]

Yes these lines set them to true...
Then reboot services

[Olivier Vernin]

Still, not working despite all the very clear instructions. For info my server version is 4.38.0

Setting the connectors to `true` in app_config.yaml, only make some of them appear on the Add Datasets > Connect > database with a lock signed (see screenshot), Strangely enough postgresql doesn't appear as a connector even if set as true.

Still, my server is stuck with an empty list of providers, The same error is returned for any option like mysql.

ubuntu@geo:~/cartodb$ RAILS_ENV=production bundle exec rake cartodb:connectors:show_config[postgres]
Provider not found: postgres

When trying to create a config for either a user or an organization though the command, the same happen

ubuntu@geo:~/cartodb$ RAILS_ENV=production bundle exec rake cartodb:connectors:set_user_config[postgres,olivier2,true,100000]
Provider not found: postgres

So even if the list of connectors is set, the list of providers is simply not loaded / populated.

Do i need to activate some flags or some features in order to have the provider list loaded? 

[Olivier Vernin]

the function to create the providers doesn't do anything because the list of providers is hardcoded empty in the rb file (see below

When calling:

RAILS_ENV=production bundle exec rake cartodb:connectors:create_providers.

this runs this function

## in lib/tasks/connector_api.rake

desc "Create Connector Providers for Provider Classes"

task create_providers: :environment do

Carto::Connector.providers(all: true).keys.each do |provider_name| ## returns an empty list

unless Carto::ConnectorProvider.where(name: provider_name).exists?

puts "Creating ConnectorProvider #{provider_name}"

Carto::ConnectorProvider.create! name: provider_name

end

end

providers = Carto::Connector.providers(all: true).keys

Carto::ConnectorProvider.where.not(name: providers).each do |provider|

puts "Provider #{provider.name} is not configured in the code!"

end

end

def find_by_uuid_or_name(id_or_name, klass, name_attribute = :name)

include Carto::UUIDHelper

if uuid?(id_or_name)

klass.find(id_or_name)

else

klass.where(name_attribute => id_or_name).first

end

end

calling in lib/carto/connector.rb

def self.providers(user: nil, all: false)

providers_info = {}

provider_ids.each do |id| ## provider_ids is an empty list

next unless all || provider_public?(id)

# TODO: load description template for provider id

description = nil

enabled = if user

Connector.limits(user: user, provider_name: id)[:enabled]

else

provider = ConnectorProvider.find_by_name(id)

ConnectorConfiguration.default(provider).enabled if provider

end

providers_info[id] = {

name: provider_name(id),

description: description,

enabled: enabled

}

end

providers_info

end

calling in lib/carto/connector/providers.rb

module Carto

class Connector

PROVIDERS = [] ## the list is hard coded empty

DEFAULT_PROVIDER = nil # No default provider

class << self

def provider_class(provider_id)

provider_data provider_id

end

def provider_public?(provider_id)

provider_item provider_id, :public?

end

def provider_name(provider_id)

provider_item provider_id, :friendly_name

end

def provider_ids ## returns nothing as the PROVIDERS is empty

PROVIDERS.map &:provider_id

end

What should be in PROVIDERS?

[carto lover]

i have postgres in appconfig.yml

try to edit in this way!

[Olivier Vernin]

Still not working, i'm not missing the connectors but the providers.

@carto_lover It would help if could you share with me additional information on your setup

1) which tag version of carto are you running (or what is your last commit)?

2) Do you have any provider hard coded in lib/carto/connector/providers.rb?

3) what files are present in the folder lib/carto/connector/

4) what is the output of:  psql -U postgres carto_db_production -c "select * from connector_providers;"

Thank you for your help

Olivier

[carto lover]

/cartodb/lib/carto/connector$ dir

context.rb  errors.rb  fdw_support.rb  parameters.rb  providers  providers.rb

                  id                  |         created_at         |         updated_at         |   name

--------------------------------------+----------------------------+----------------------------+-----------

 38e08660-6b48-42c9-a61f-7239f78f4c84 | 2020-03-21 20:18:16.698101 | 2020-03-21 20:18:16.698101 | odbc

 5339b0bf-580f-4a93-8a50-a43c61822df0 | 2020-03-21 20:18:16.702302 | 2020-03-21 20:18:16.702302 | postgres

 b5cb3ae8-1a2a-4693-b78a-95ea0ad82603 | 2020-03-21 20:18:16.704981 | 2020-03-21 20:18:16.704981 | mysql

 16f43a05-4942-4cc4-9116-475e0710d910 | 2020-03-21 20:18:16.707933 | 2020-03-21 20:18:16.707933 | sqlserver

 1632f210-3cf9-40bd-9b98-49debed70590 | 2020-03-21 20:18:16.710597 | 2020-03-21 20:18:16.710597 | hive

 c1bf2d42-333e-4ba7-b4b8-707acb9673a6 | 2020-03-21 20:18:16.713271 | 2020-03-21 20:18:16.713271 | bigquery

[Olivier Vernin]

Ok, i see that's my cartodb code is missing all the necessary classes `providers/fdw/odbc/*`

@carto lover: Correct me if i'm wrong but you are running a v4.37 or lower of carto? 

On the mast branch for carto the content of PROVIDERS is empty. The PROVIDERS were removed on the 3rd of Feb 2020

[Shoaib Burq]

Did anyone figure out why this was removed? I am extremely frustrated by lack of clarity on the feature roadmap of carto and poor support on this list for open source. I know a lot of people who want to use the project but can't due to this.