Slow read requests

[Gary Madarm]

Using the Node level package I imported a list of 3 billion records consisting of reverse email address and a short string value, e.g. key: com.gmail@user:applies value: harvard. While importing the data was pretty quick, querying the data is extremely slow. Each key is unique but keys may be similar, e.g.:

com.gmail@user:applies value: harvard

com.gmail@user:oranges value: yale

Querying a specific key on my 8 processor / 32Gb ram box takes around 7 minutes. How can I improve get or ReadStream requests to make them return a non-cached response in under 20 seconds? I use the code below to query leveldb:

const level = require('level');

const db = level('db');

db.createReadStream({gte: com.gmail@user, lt: com.gmail@usera, limit: 10})

  .on('data', function (data) {

    var school = data.key.split(':');

    var e2 = school[0].split('@');

    var dom2 = e2[0].split('.');

    dom2.reverse();

    var femail = e2[1] + "@" + dom2.join(".") ;

    var dval = decodeURIComponent(data.value);

    txt = txt + femail +',' + dval +',' + school[1]+"<br>";

    res = femail +',' + decodeURIComponent(data.value);

  })

  .on('error', function (err) {

    console.log('Oh my!', err)

  })

  .on('end', function () {

    console.log(txt)

  })

}

[Victor Costan]

We don't support any node package, so your best place for support is the package's authors.

That being said, it looks like you're reading the entire database (createReadStream) to get the information your want. LevelDB is a key-value store, so you can query each record using its key pretty easily. Querying a single key should take less than a second for most cases. The main exception is that if your database needs to be compacted right when you open it, the compaction will take some time. However, follow-up queries should still be quick.

I hope this helps,

   Victor

--
You received this message because you are subscribed to the Google Groups "leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Madarm]

Thanks Victor, based on your advice I updated the script to use a get instead. However, querying a single key still takes around a minute and a half to respond. I verified that the database finished compacting before sending the query. Any ideas how I can improve the query time for a single key?

[root-localhost ./myslql]

Aha, how about a logs?)

пн, 14 янв. 2019 г. в 11:11, Gary Madarm <gma...@gmail.com>: