How to Perform Large Deletes in the Realtime Database

[bobby....@gmail.com]

Hi,

Read you article on deleting large chunk of data via Firebase command want to know difference between
Updating a node to null
and running your new command below.
firebase database:remove /path/to/delete

I wanted to know this because I have written a housekeeping utility to to delete older data everyday. My node data structure is like below,

Articles
|_____data
|____yyyymmdd
|____pushids
|____pushids
.....
|_____statistics
|_____deleteFrom:yyyymmdd
|_____retentionDays: 60


So basically my nodejs program will read the statistics node and compute to delete data older than 60days. Retention days are different for each nodes. So for example, deletion will be like below for list of dates, I execute this via update().
Articles/data/20190115 = null

Eventually I will move this statistics housekeeping node separately.

Are there any performance advantage using the firebase command delete or does it do anything different.


defaultWriteSizeLimit
In my application, I move data regularly from local mySQL to firebase, I do that via nodejs admin service account. So does this new parameter affect if a program run admin service account as well ? Elapsed time for my updates are surely more than a minute. So do you recommend me to execute this command
firebase database:settings:set defaultWriteSizeLimit unlimited --instance <database-name>


Can you please let me know your thoughts on above two.

Also since you have brought delete. It would be good likewise if you bring a mechanism to load data to firebase. Currently I am doing it via a nodejs program(transform data to from SQL data to firebase JSON format then set/update). It would be easy if I do it via a firebase command like
firebase database:load path/to/load <JSON filename>


Thanks,
Sushanth bobby

[Kato Richardson]

Hi Sushanth,

The difference is that the CLI command will optimize the delete process and throttle out the requests a bit to reduce performance impact. Setting the node to null can potentially have a much larger impact on performance since it doesn't attempt to batch up individual keys for deletion.

More details in our blog here.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/7e2ec3d6-3e86-4cd1-b9d3-a992eb0ad33a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398