Error while trying to insert a value larger or equal to 64k

[Manish Pandit]

Hi,

I am in the process of evaluating Voldemort as a KV store (MySQL
backed) plus a cache at my company. However, I started to see the
error below when the value size increased. I posted it on the bug
report as well, and got a reply from Jay (http://code.google.com/p/
project-voldemort/issues/detail?id=27). However, even after increasing
the timeouts and sizes considerably, I keep running into the same. I
am running Voldemort 0.57.1.

Here is the client config:

factory = new SocketStoreClientFactory(new ClientConfig
().setBootstrapUrls(bootstrapUrl).setMaxThreads
(100).setConnectionTimeout(1,TimeUnit.MINUTES).
setSocketBufferSize(5*1024*1024).setSocketTimeout
(1,TimeUnit.MINUTES).setRoutingTimeout(1, TimeUnit.MINUTES));

Here is the test code (gateway is my abstraction of the store client).

@Test
public void testLoadData(){
byte[] data = new byte[64*1024];
try{
gateway.put("test", "Test_Data", data);
Assert.assertTrue(1==1);
}catch(Throwable t){
logger.error(t,t);
Assert.fail();
}finally{
gateway.delete("test", "Test_Data");
}
}

Here is the error:

voldemort.store.InsufficientOperationalNodesException: No master node
succeeded!
at voldemort.store.routed.RoutedStore.put(RoutedStore.java:670)
at voldemort.store.routed.RoutedStore.put(RoutedStore.java:71)
at voldemort.store.DelegatingStore.put(DelegatingStore.java:69)
at voldemort.store.stats.StatTrackingStore.put(StatTrackingStore.java:
90)
at voldemort.store.serialized.SerializingStore.put
(SerializingStore.java:104)
at voldemort.store.DelegatingStore.put(DelegatingStore.java:69)
at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:
188)
at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:
173)

Please let me know if I am missing something - it does seem so, as I
am seeing this consistently over anything >=64k.

I appreciate your help.

-cheers,
Manish

[Geir Magnusson Jr.]

On Dec 17, 2009, at 10:30 PM, Manish Pandit wrote:

> Hi,
>
> I am in the process of evaluating Voldemort as a KV store (MySQL
> backed) plus a cache at my company.

Why mySQL and why a cache? (ok, you may need a cache when backed by MySQL.... :)

Are nodes reporting any errors?

geir

> --
>
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>

[Manish Pandit]

On Dec 18, 2:52 am, "Geir Magnusson Jr." <g...@pobox.com> wrote:
> On Dec 17, 2009, at 10:30 PM, Manish Pandit wrote:
>
> > Hi,
>
> > I am in the process of evaluating Voldemort as a KV store (MySQL
> > backed) plus a cache at my company.
>
> Why mySQL and why a cache?  (ok, you may need a cache when backed by MySQL.... :)
>
> Are nodes reporting any errors?
>
> geir
>

No errors on node. I am using the single_node_cluster configuration
for now. The reason for backing it up with MySQL is because most of
the data that'd sit in Voldemort (if I can get over this 64k issue)
will be computed via large queries and data massaging. Also, I want a
somewhat persistent store (unlike Memcache) for this solution.

-cheers,
Manish

[Manish Pandit]

On Dec 18, 9:31 am, Manish Pandit <pandit.man...@gmail.com> wrote:
>
> No errors on node. I am using the single_node_cluster configuration
> for now. The reason for backing it up with MySQL is because most of
> the data that'd sit in Voldemort (if I can get over this 64k issue)
> will be computed via large queries and data massaging. Also, I want a
> somewhat persistent store (unlike Memcache) for this solution.
>
> -cheers,
> Manish

Here is an update - when I change the storage to bdb, it works like a
charm (even as high as 10MB). However, after switching back to MySQL
it fails (with 64k).

I've no issues with BDB but was hoping to use MySQL due to our
existing infrastructure and operational know-how around it.

-cheers,
Manish

[Jay Kreps]

Hi Marish,

The error you gave is an error saying

"All the nodes are currently marked down"

What would be most helpful is the first error you got which would
explain why the node got marked down (i.e. due to timeout).

-Jay

[Coda Hale]

On Fri, Dec 18, 2009 at 11:11 AM, Manish Pandit <pandit...@gmail.com> wrote:
> Here is an update - when I change the storage to bdb, it works like a
> charm (even as high as 10MB). However, after switching back to MySQL
> it fails (with 64k).

MySQL’s BLOB type has a maximum length of 2^16 bytes, or 64KB. That's
the ceiling you're running into.

From MysqlStorageEngine.java:95-99:
> public void create() {
> execute("create table " + getName()
> + " (key_ varbinary(200) not null, version_ varbinary(200) not null, "
> + " value_ blob, primary key(key_, version_)) engine = InnoDB");
> }

The type of value_ needs to be changed from BLOB to MEDIUMBLOB (max
16MB) or LONGBLOB (max 4GB) in order to support larger values.

You'll need to run something like the following SQL:
> ALTER TABLE <table name> CHANGE value_ value_ LONGBLOB;

N.B.: This will lock that table and, depending on how many rows you
have in it, take ages to complete.

That said, even fans of MySQL will recommend against storing large
BLOBs in it. You'll need to check the JDBC driver's max_allowed_packet
setting if you're going over 16MB, and give your MySQL server rather a
lot of memory due to buffer duplication. MySQL also stores BLOBs
outside the row table, which makes for slower access.

--
Coda Hale
http://codahale.com

[Jay Kreps]

Ah, nice catch. We should probably default the table creation to that
in our code.

-Jay

[John Bäckstrand]

Another common reason for similar problems is a mysql-setting regarding maximum packet length, which is farily low per default I think. Or at least it used to be, on the order of 1MB or so.

--
John Bäckstrand

[Manish Pandit]

On Dec 18, 12:53 pm, Coda Hale <coda.h...@gmail.com> wrote:

Wow! Thanks so much!! This certainly helps.

I changed the column def and everything is running smoothly. Now I can
venture into the multi-node configuration.

-cheers,
Manish

[Manish Pandit]

On Dec 19, 7:45 pm, Manish Pandit <pandit.man...@gmail.com> wrote:
>
> Wow! Thanks so much!! This certainly helps.
>
> I changed the column def and everything is running smoothly. Now I can
> venture into the multi-node configuration.
>
> -cheers,
> Manish

Forgot to mention - I understand that BDB would be the best option
given the large blob size to handle if I go the MySQL route. I'll
evaluate with BDB as the proposed solution for my work.

Thanks again to the folks on the thread.

-cheers,
Manish