oetl problems loading csv data to a document database

[Luc Evers]

testdata: https://support.spatialkey.com/spatialkey-sample-csv-data/

test.json file :

    {
      "config":{
      "log": "debug"
    },
      "begin": [
    ],
    "source" : {
     "file": { "path": "./test.csv", "lock" : false }
    },
    "extractor" : {
    "row": {}
  },
  "transformers" : [
   { "csv": { "separator": ",", "nullValue": "NULL", "skipFrom": 1, "skipTo": 2 } }
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "remote:localhost/test",
      "dbUser": "admin",
      "dbPassword": "admin",
      "dbAutoCreate": true,
      "tx": false,
      "dbType": "document",
      "classes":[{"name": "test" }],
      "indexes": [{"class":"test", "fields":["zip:STRING"], "type":"NOTUNIQUE" }]
    }
  }
}

Error MSG:
Cannot check the existance of a database in a remote server. Please use the console or the OServerAdmin class.
Why NOT?  This tool is useless if remote is not working.

Change to local

 {
  "config":{
    "log": "debug"
  },
  "begin": [
  ],
  "source" : {
    "file": { "path": "./test.csv", "lock" : false }
  },
  "extractor" : {
    "row": {}
  },
  "transformers" : [
   { "csv": { "separator": ",", "nullValue": "NULL" } }
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "plocal:/home/master/Downloads/orientdb/orientdb-community-2.1.15/databases/test",
      "dbUser": "admin",
      "dbPassword": "admin",
      "dbAutoCreate": true,
      "tx": false,
      "dbType": "document",
      "classes":[{"name": "test" }],
      "indexes": [{"class":"test", "fields":["zip:STRING"], "type":"NOTUNIQUE" }]
    }
  }
}

  Only works if I shutdown the OrientDB server.
  Result:
END ETL PROCESSOR
+ extracted 986 rows (0 rows/sec) - 986 rows -> loaded 985 documents (0 documents/sec) Total time: 965ms [0 warnings, 0 errors]

But there is no data in the database, I think because the server is not running.

Question?

   How can I load the CSV data via the ETL tool?
   Why remote is not working?

[Ivan Mainetti]

Hi,

try this json:

{
  "source": { "file": { "path": "/home/ivan/Scrivania/etl/comm18-04/data.csv" } },
  "extractor": { "csv": {} },
  "loader": {
    "orientdb": {
       "dbURL": "plocal:/home/ivan/OrientDB/db_installati/enterprise/orientdb-enterprise-2.1.15/databases/comm1804",
       "dbType": "document",
       "class": "test",

       "classes": [{"name": "test"}],
       "indexes": [{"class":"test", "fields":["zip:STRING"], "type":"NOTUNIQUE" }]
    }
  }
}

I've added the "class" parameter in the loader, like explained here and it's working for me.


Hope it helps, bye

[Luc Evers]

   Yvan,

      You solved my two problems!

      I tested your example when the server is running and the result was no problems!

      Thanks!

   

  Luc.

Il giorno lunedì 18 aprile 2016 16:16:08 UTC+2, Luc Evers ha scritto:

testdata: https://support.spatialkey.com/spatialkey-sample-csv-data/

test.json file :

    {
      "config":{
      "log": "debug"
    },
      "begin": [
    ],
    "source" : {
     "file": { "path": "./test.csv", "lock" : false }
    },
    "extractor" : {
    "row": {}
  },
  "transformers" : [
   { "csv": { "separator": ",", "nullValue": "NULL", "skipFrom": 1, "skipTo": 2 } }
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "remote:localhost/test",
      "dbUser": "admin",
      "dbPassword": "admin",
      "dbAutoCreate": true,
      "tx": false,

      "dbType": "document",

      "classes":[{"name": "test" }],
      "indexes": [{"class":"test", "fields":["zip:STRING"], "type":"NOTUNIQUE" }]
    }
  }
}

Error MSG:
Cannot check the existance of a database in a remote server. Please use the console or the OServerAdmin class.
Why NOT?  This tool is useless if remote is not working.

Change to local

 {
  "config":{
    "log": "debug"
  },
  "begin": [
  ],
  "source" : {
    "file": { "path": "./test.csv", "lock" : false }
  },
  "extractor" : {
    "row": {}
  },
  "transformers" : [
   { "csv": { "separator": ",", "nullValue": "NULL" } }

  ],
  "loader" : {
    "orientdb": {

      "dbURL": "plocal:/home/master/Downloads/orientdb/orientdb-community-2.1.15/databases/test",
      "dbUser": "admin",
      "dbPassword": "admin",
      "dbAutoCreate": true,
      "tx": false,

      "dbType": "document",

      "classes":[{"name": "test" }],
      "indexes": [{"class":"test", "fields":["zip:STRING"], "type":"NOTUNIQUE" }]
    }
  }
}

  Only works if I shutdown the OrientDB server.
  Result:
END ETL PROCESSOR
+ extracted 986 rows (0 rows/sec) - 986 rows -> loaded 985 documents (0 documents/sec) Total time: 965ms [0 warnings, 0 errors]

But there is no data in the database, I think because the server is not running.

Question?

   How can I load the CSV data via the ETL tool?
   Why remote is not working?

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/JHjCL7grdRk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luc Evers]

Still one problem, I forgot Remote:

Error MSG:
"Cannot check the existence of a remote server database. Please use the console or the OServerAdmin class."

To allow multiple access to the same database , then use Remote Mode!

But if Remote Mode is used then the Error MSG above is the problem!

Important , all questions are all about the oetl.sh program. 

Op woensdag 20 april 2016 16:15:05 UTC+2 schreef Luc Evers:

[alessand...@gmail.com]

Hi,

I did some tests and I had no problems with a remote connection and a graph database, 

but I got your exception when I used a remote connection with a document database.

Best regards,

Alessandro

[Luc Evers]

  Alessandro,

       Thanks for the test!

        Seems a problem for document databases.

   Luc.

--

[Luc Evers]

  Luca,

     Do you have a solution for this problem?

     The Etl programs are not working for document databases.

 

Op dinsdag 26 april 2016 10:48:58 UTC+2 schreef Luc Evers:

  Alessandro,

       Thanks for the test!

        Seems a problem for document databases.

   Luc.

On Tue, Apr 26, 2016 at 10:44 AM, <alessand...@gmail.com> wrote:

Hi,

I did some tests and I had no problems with a remote connection and a graph database, 

but I got your exception when I used a remote connection with a document database.

Best regards,

Alessandro

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/JHjCL7grdRk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to orient-database+unsubscribe@googlegroups.com.