Datastore export as JSON format

[Han Ju Kim]

I'm trying to export Datastore entities as JSON format. By using API or gsutil such as "gcloud datastore export", it only export as LevelDB format with many output-xxx files. Is there any tool to converting this LevelDB files to JSON format or any third party engine to do that?

[Mua Rachmann]

Hi Kim,

You might want to have a look at this - https://github.com/nisaacson/leveldb-json

regards.

Mua

On Tue, Aug 27, 2019 at 12:35 AM Han Ju Kim <hkim....@gmail.com> wrote:

I'm trying to export Datastore entities as JSON format. By using API or gsutil such as "gcloud datastore export", it only export as LevelDB format with many output-xxx files. Is there any tool to converting this LevelDB files to JSON format or any third party engine to do that?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06c074cc-cc4a-4c73-a8d9-ed7299d5421e%40googlegroups.com.

[Han Ju Kim]

Hi Mua,

I appreciate prompt feedback! I already tried that, but it showed some error and not updated quite long time period. Any help will be appreciated.

Cheers,

Han

On Tuesday, 27 August 2019 11:50:47 UTC+12, Mua Rachmann wrote:

Hi Kim,

You might want to have a look at this - https://github.com/nisaacson/leveldb-json

regards.

Mua

On Tue, Aug 27, 2019 at 12:35 AM Han Ju Kim <hkim...@gmail.com> wrote:

I'm trying to export Datastore entities as JSON format. By using API or gsutil such as "gcloud datastore export", it only export as LevelDB format with many output-xxx files. Is there any tool to converting this LevelDB files to JSON format or any third party engine to do that?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Mua Rachmann]

Hi Kim,

Can we see the logs here?

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/22e67aef-22ad-4dcd-9fdd-ecaaa89539bc%40googlegroups.com.

[Han Ju Kim]

This is my log. When I tried to "npm install -g leveldb-json", it comes up

C:\WINDOWS\system32>npm install -g leveldb-json

C:\Users\hkim\AppData\Roaming\npm\leveldb-load-from-json -> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\bin\load-from-json

C:\Users\hkim\AppData\Roaming\npm\leveldb-dump-to-json -> C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\bin\dump-to-json

> leve...@0.10.6 install C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown

> node-gyp rebuild

C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )

gyp ERR! configure error

gyp ERR! stack Error: Command failed: C:\Python37\python.EXE -c import sys; print "%s.%s.%s" % sys.version_info[:3];

gyp ERR! stack   File "<string>", line 1

gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];

gyp ERR! stack                                ^

gyp ERR! stack SyntaxError: invalid syntax

gyp ERR! stack

gyp ERR! stack     at ChildProcess.exithandler (child_process.js:294:12)

gyp ERR! stack     at ChildProcess.emit (events.js:198:13)

gyp ERR! stack     at maybeClose (internal/child_process.js:982:16)

gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)

gyp ERR! System Windows_NT 10.0.18362

gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"

gyp ERR! cwd C:\Users\hkim\AppData\Roaming\npm\node_modules\leveldb-json\node_modules\leveldown

gyp ERR! node -v v10.16.3

gyp ERR! node-gyp -v v3.8.0

gyp ERR! not ok

npm ERR! code ELIFECYCLE

npm ERR! errno 1

npm ERR! leve...@0.10.6 install: `node-gyp rebuild`

npm ERR! Exit status 1

npm ERR!

npm ERR! Failed at the leve...@0.10.6 install script.

npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:

npm ERR!     C:\Users\hkim\AppData\Roaming\npm-cache\_logs\2019-08-27T01_11_59_731Z-debug.log

C:\WINDOWS\system32>

On Tuesday, 27 August 2019 13:10:37 UTC+12, Mua Rachmann wrote:

Hi Kim,

Can we see the logs here?

On Tue, Aug 27, 2019 at 2:05 AM Han Ju Kim <hkim...@gmail.com> wrote:

Hi Mua,

I appreciate prompt feedback! I already tried that, but it showed some error and not updated quite long time period. Any help will be appreciated.

Cheers,

Han

On Tuesday, 27 August 2019 11:50:47 UTC+12, Mua Rachmann wrote:

Hi Kim,

You might want to have a look at this - https://github.com/nisaacson/leveldb-json

regards.

Mua

On Tue, Aug 27, 2019 at 12:35 AM Han Ju Kim <hkim...@gmail.com> wrote:

I'm trying to export Datastore entities as JSON format. By using API or gsutil such as "gcloud datastore export", it only export as LevelDB format with many output-xxx files. Is there any tool to converting this LevelDB files to JSON format or any third party engine to do that?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06c074cc-cc4a-4c73-a8d9-ed7299d5421e%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/22e67aef-22ad-4dcd-9fdd-ecaaa89539bc%40googlegroups.com.

[Mua Rachmann]

Okay, I think you might want to look at https://github.com/nisaacson/leveldb-json/issues/1 which is pretty similar to your logs,

Also you might want to build from source and consider a fork of the original repo - https://github.com/killix/leveldb-json 

which seems to solve the issue. Dependencies not being upgraded or so. - Basically applying - https://github.com/killix/leveldb-json/commit/35df5f3d396907e015527984aa03e13726e9e386

Can you try that and see if it solves your problem

Regards,

Mua

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7cd042ec-6ecc-4a16-9aba-889606d02191%40googlegroups.com.

[Mua Rachmann]

If that sucks, You might want to have a look at this https://github.com/junosuarez/node-level-export

Best,

Mua

[Han Ju Kim]

Great! I will have a look at. 

Still confusing, why google does not provide CSV,TXT or JSON export of datastore.

Cheers,

Han

On Tuesday, 27 August 2019 13:26:18 UTC+12, Mua Rachmann wrote:

If that sucks, You might want to have a look at this https://github.com/junosuarez/node-level-export

Best,

Mua

On Tue, Aug 27, 2019 at 2:20 AM Mua Rachmann <muara...@gmail.com> wrote:

Okay, I think you might want to look at https://github.com/nisaacson/leveldb-json/issues/1 which is pretty similar to your logs,

Also you might want to build from source and consider a fork of the original repo - https://github.com/killix/leveldb-json 

which seems to solve the issue. Dependencies not being upgraded or so. - Basically applying - https://github.com/killix/leveldb-json/commit/35df5f3d396907e015527984aa03e13726e9e386

Can you try that and see if it solves your problem

Regards,

Mua

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7cd042ec-6ecc-4a16-9aba-889606d02191%40googlegroups.com.

[Han Ju Kim]

Hi Mua,

I found some useful function of Dataflow to import JSON type file from datastore entities, I'm struggling with this now. All looks good, but exported file does not include any data at all.

https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#datastoretogcstext

Cheers,

Han

[Alex Martelli]

The error in that traceback is crystal-clear: the code uses a Python 2 `print` *statement*, making it incompatible with Python 3 (where `print` is a *function*), yet you're trying to run it with Python 3.7.

Porting the code to be compatible with Python 2 and 3 might be some work. Easier, I think, would be to install Python 2 (it's going end-of-life on Jan 1, but that's months from now) and try running that code with Pytjon 2 (2.7, of course, the last release of that family) as it's clearly meant to be run.

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7cd042ec-6ecc-4a16-9aba-889606d02191%40googlegroups.com.