couldn't get VoldemortFilter to work
Ron Reynolds
03 23, 2010 04:23:33 AM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler ERROR: handleFetchPartitionEntries failed for request(partitions: 0
partitions: 1
store: "keys"
filter {
name: "com.audiencescience.service.user.util.VoldemortSingleKeyFilter"
data: "\312\376\272\276\...\000\036\000!\000\002\000\001\000\"\000\000\000\002\000#"
}
fetch_values: true
)
voldemort.VoldemortException: Failed to load and instantiate the filter class
at voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromRequest(AdminServiceRequestHandler.java:656)
at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleFetchPartitionEntries(AdminServiceRequestHandler.java:159)
at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleRequest(AdminServiceRequestHandler.java:114)
at voldemort.server.socket.SocketServerSession.run(SocketServerSession.java:73)
...
Caused by: java.lang.ClassNotFoundException: com.audiencescience.service.user.util.VoldemortSingleKeyFilter
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
...
at voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromRequest(AdminServiceRequestHandler.java:652)
... 6 more
i originally used an anonymous inner class (i wasn't too surprised that didn't work).
then tried a public static inner class - i could rationalize that not working as well since the outer class wasn't serialized and there might be loading issues.
but i couldn't figure out the normal public class, with or without Serializable.
public class VoldemortSingleKeyFilter implements VoldemortFilter {
private Object _key;
public VoldemortSingleKeyFilter (Object key) { _key = key; }
public boolean accept(Object key, Versioned<?> val) {
return key.equals(_key);
}
};
should this feature work? is there a flag i need to set on the server? oh, version is 0.70.1 and the connection is between a windows client and a linux server both running Java 1.6 (64-bit 1.6.0_13 on the server; 32-bit 1.6.0_17 on the client).
bhupesh bansal
setting it to true will enable class serialization and loading in remote Jvms.
Best
Bhupesh
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
Ron Reynolds
thanks. :) (pity the error message doesn't say that, tho i suppose the alternative would be to have the filter in the server's classpath, tho at this point i'm trying to keep the server as vanilla as possible).
From: bhupesh bansal <bbans...@gmail.com>
To: project-...@googlegroups.com
Sent: Tue, March 23, 2010 8:28:37 AM
Subject: Re: [project-voldemort] couldn't get VoldemortFilter to work
bhupesh bansal
the network class loading. It is waiting for some EC2 based multiple machines, multiple jvm test before we can enable it.
Its just a property change in server.properties and I think you must be setting some values already ? The alternative is to provide the filter on the server classpath which should work just fine.
Best
Bhupesh
Ron Reynolds
03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler DEBUG: Attempt to load VoldemortFilter class:com.audiencescience.service.user.util.VoldemortSingleKeyFilter
03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler WARN: NetworkLoader is experimental and should not be used for now.
03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler ERROR: handleFetchPartitionEntries failed for request(partitions: 0
(the rest is as below)
so i've added a warning to go with my error. :)
Bruce Ritchie
I never managed to get it to work when I was playing with it late last
year ... and I distrust that it'll work properly across JVM versions
and vendors in any case. YMMV.
Regards,
Bruce Ritchie
On Mar 24, 5:24 pm, Ron Reynolds <tequila...@ymail.com> wrote:
> well, this is an improvement?
>
> 03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler DEBUG: Attempt to load VoldemortFilter class:com.audiencescience.service.user.util.VoldemortSingleKeyFilter
> 03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler WARN: NetworkLoader is experimental and should not be used for now.
> 03 24, 2010 02:24:23 PM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler ERROR: handleFetchPartitionEntries failed for request(partitions: 0
> (the rest is as below)
>
> so i've added a warning to go with my error. :)
>
> ________________________________
> From: bhupesh bansal <bbansal....@gmail.com>
> To: project-...@googlegroups.com
> Sent: Tue, March 23, 2010 11:18:11 AM
> Subject: Re: [project-voldemort] couldn't get VoldemortFilter to work
>
> It was disabled for precautionary reasons, as we saw some class not found exceptions running some other tests. On further investigations we realized it was something else and not
> the network class loading. It is waiting for some EC2 based multiple machines, multiple jvm test before we can enable it.
>
> Its just a property change in server.properties and I think you must be setting some values already ? The alternative is to provide the filter on the server classpath which should work just fine.
>
> Best
> Bhupesh
>
> On Tue, Mar 23, 2010 at 8:43 AM, Ron Reynolds <tequila...@ymail.com> wrote:
>
> >is it disabled by default for security or performance reasons? i expect if no one attempts to serialize a filter (or similar) class to the server there should be no impact on performance, yes?
>
> >thanks. :) (pity the error message doesn't say that, tho i suppose the alternative would be to have the filter in the server's classpath, tho at this point i'm trying to keep the server as vanilla as possible).
>
> ________________________________
> From: bhupesh bansal <bbansal....@gmail.com>
>
>
>
>
>
> >To: project-...@googlegroups.com
> >Sent: Tue, March 23, 2010 8:28:37 AM
> >Subject: Re: [project-voldemort] couldn't get VoldemortFilter to work
>
> >>In VoldemortConfig there is a setting "enable.network.classloader" which defaults to false,
> >setting it to true will enable class serialization and loading in remote Jvms.
>
> >Best
> >Bhupesh
>
> >On Tue, Mar 23, 2010 at 7:52 AM, Ron Reynolds <tequila...@ymail.com> wrote:
>
> >>playing with the AdminClient i tried using a VoldemortFilter with fetchEntries() but any time i provided a non-null reference to my VoldemortFilter on the client i got nothing back and this on the server:
>
> >>03 23, 2010 04:23:33 AM cc: voldemort.server.protocol.admin.AdminServiceRequestHandler ERROR: handleFetchPartitionEntries failed for request(partitions: 0
> >>partitions: 1
> >>store: "keys"
> >>filter {
> >> name: "com.audiencescience.service.user.util.VoldemortSingleKeyFilter"
>
> >> data: "\312\376\272\276\...\000\036\000!\000\002\000\001\000\"\000\000\000\002\00 0#"
> >>}
> >>fetch_values: true
> >>)
> >>voldemort.VoldemortException: Failed to load and instantiate the filter class
> >> at
> >> voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromReq uest(AdminServiceRequestHandler.java:656)
> >> at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleFetchParti tionEntries(AdminServiceRequestHandler.java:159)
>
> >> at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleRequest(Ad minServiceRequestHandler.java:114)
> >> at voldemort.server.socket.SocketServerSession.run(SocketServerSession.java:73 )
> >>...
>
> >>Caused by: java.lang.ClassNotFoundException: com.audiencescience.service.user.util.VoldemortSingleKeyFilter
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>
> >>...
> >> at
> >> voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromReq uest(AdminServiceRequestHandler.java:652)
> >> ... 6 more
>
> >>i originally used an anonymous inner class (i wasn't too surprised that didn't work).
>
> >>then tried a public static inner class - i could rationalize that not working as well since the outer class wasn't serialized and there might be loading issues.
> >>but i couldn't figure out the normal public class, with or without Serializable.
>
> >>public class VoldemortSingleKeyFilter implements VoldemortFilter {
> >> private Object _key;
> >> public VoldemortSingleKeyFilter (Object key) { _key = key; }
> >> public boolean accept(Object key, Versioned<?> val) {
>
> >> return key.equals(_key);
> >> }
> >>};
>
> >>should this feature work? is there
> >> a flag i need to set on the server? oh, version is 0.70.1 and the connection is between a windows client and a linux server both running Java 1.6 (64-bit 1.6.0_13 on the server; 32-bit 1.6.0_17 on the client).
>
> >>--
> >>>>You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> >>>>To post to this group, send email to project-...@googlegroups.com.
> >>>>To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
>
> >>For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
>
> >--
> >>You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> >>To post to this group, send email to project-...@googlegroups.com.
> >>To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
>
> >For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
>
> >--
> >>You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> >>To post to this group, send email to project-...@googlegroups.com.
> >>To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
>
> >For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
Ron Reynolds
From: Bruce Ritchie <bruce....@gmail.com>
To: project-voldemort <project-...@googlegroups.com>
Sent: Wed, March 24, 2010 2:50:47 PM
Subject: [project-voldemort] Re: couldn't get VoldemortFilter to work
>
> >>For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
>
> >--
> >>You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> >>To post to this group, send email to project-...@googlegroups.com.
>
> >For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
>
> >--
> >>You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> >>To post to this group, send email to project-...@googlegroups.com.
>
> >For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> For more options, visit this group athttp://groups.google.com/group/project-voldemort?hl=en.
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
Jay Kreps
-Jay
bhupesh bansal
I think so, This one need more work I guess. Having a filter in streaming APIs is a good idea
but I am not sure the best way to go about it.
Best
Bhupesh
Bruce Ritchie
My first thoughts for this are
1. The filter would already exist on the server's classpath
2. Require a jar of the filter(s) to be uploaded prior to use (either
before startup or streamed and loaded with a custom classloader)
3. Stream the source and compile on the server.
I prefer the first option but the second I think is not undoable
either. I think this feature could have very nice potential to it if
it was rock solid stable.
Regards,
Bruce Ritchie
On Mar 25, 2:45 pm, bhupesh bansal <bbansal....@gmail.com> wrote:
> Jay,
>
> I think so, This one need more work I guess. Having a filter in streaming
> APIs is a good idea
> but I am not sure the best way to go about it.
>
> Best
> Bhupesh
>
>
>
> On Wed, Mar 24, 2010 at 11:04 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
> > Bhupesh, if this doesn't work can we just remove it?
>
> > -Jay
>
> > On Tue, Mar 23, 2010 at 11:18 AM, bhupesh bansal <bbansal....@gmail.com>
> > wrote:
> > > It was disabled for precautionary reasons, as we saw some class not found
> > > exceptions running some other tests. On further investigations we
> > realized
> > > it was something else and not
> > > the network class loading. It is waiting for some EC2 based multiple
> > > machines, multiple jvm test before we can enable it.
>
> > > Its just a property change in server.properties and I think you must be
> > > setting some values already ? The alternative is to provide the filter on
> > > the server classpath which should work just fine.
>
> > > Best
> > > Bhupesh
>
> > > On Tue, Mar 23, 2010 at 8:43 AM, Ron Reynolds <tequila...@ymail.com>
> > wrote:
>
> > >> is it disabled by default for security or performance reasons? i expect
> > >> if no one attempts to serialize a filter (or similar) class to the
> > server
> > >> there should be no impact on performance, yes?
> > >> thanks. :) (pity the error message doesn't say that, tho i suppose the
> > >> alternative would be to have the filter in the server's classpath, tho
> > at
> > >> this point i'm trying to keep the server as vanilla as possible).
>
> > >> ________________________________
> > >> From: bhupesh bansal <bbansal....@gmail.com>
> > >> To: project-...@googlegroups.com
> > >> Sent: Tue, March 23, 2010 8:28:37 AM
> > >> Subject: Re: [project-voldemort] couldn't get VoldemortFilter to work
>
> > >> In VoldemortConfig there is a setting "enable.network.classloader" which
> > >> defaults to false,
> > >> setting it to true will enable class serialization and loading in remote
> > >> Jvms.
>
> > >> Best
> > >> Bhupesh
>
> > >> On Tue, Mar 23, 2010 at 7:52 AM, Ron Reynolds <tequila...@ymail.com>
> > >> wrote:
>
> > >>> playing with the AdminClient i tried using a VoldemortFilter with
> > >>> fetchEntries() but any time i provided a non-null reference to my
> > >>> VoldemortFilter on the client i got nothing back and this on the
> > server:
>
> > >>> 03 23, 2010 04:23:33 AM cc:
> > >>> voldemort.server.protocol.admin.AdminServiceRequestHandler ERROR:
> > >>> handleFetchPartitionEntries failed for request(partitions: 0
> > >>> partitions: 1
> > >>> store: "keys"
> > >>> filter {
> > >>> name:
> > "com.audiencescience.service.user.util.VoldemortSingleKeyFilter"
> > >>> data:
>
> > "\312\376\272\276\...\000\036\000!\000\002\000\001\000\"\000\000\000\002\00 0#"
> > >>> }
> > >>> fetch_values: true
> > >>> )
> > >>> voldemort.VoldemortException: Failed to load and instantiate the filter
> > >>> class
> > >>> at
>
> > voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromReq uest(AdminServiceRequestHandler.java:656)
> > >>> at
>
> > voldemort.server.protocol.admin.AdminServiceRequestHandler.handleFetchParti tionEntries(AdminServiceRequestHandler.java:159)
> > >>> at
>
> > voldemort.server.protocol.admin.AdminServiceRequestHandler.handleRequest(Ad minServiceRequestHandler.java:114)
> > >>> at
>
> > voldemort.server.socket.SocketServerSession.run(SocketServerSession.java:73 )
> > >>> ...
> > >>> Caused by: java.lang.ClassNotFoundException:
> > >>> com.audiencescience.service.user.util.VoldemortSingleKeyFilter
> > >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > >>> ...
> > >>> at
>
> > voldemort.server.protocol.admin.AdminServiceRequestHandler.getFilterFromReq uest(AdminServiceRequestHandler.java:652)
> > >>> project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
> > .
> > >>> For more options, visit this group at
> > >>>http://groups.google.com/group/project-voldemort?hl=en.
>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "project-voldemort" group.
> > >> To post to this group, send email to project-...@googlegroups.com
> > .
> > >> To unsubscribe from this group, send email to
> > >> project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
> > .
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/project-voldemort?hl=en.
>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "project-voldemort" group.
> > >> To post to this group, send email to project-...@googlegroups.com
> > .
> > >> To unsubscribe from this group, send email to
> > >> project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
> > .
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/project-voldemort?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "project-voldemort" group.
> > > To post to this group, send email to project-...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
> > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/project-voldemort?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "project-voldemort" group.
> > To post to this group, send email to project-...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
Ron Reynolds
Subject: [project-voldemort] Re: couldn't get VoldemortFilter to work
> > .
> > >>> For more options, visit this group at
> > >>>http://groups.google.com/group/project-voldemort?hl=en.
>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "project-voldemort" group.
> > >> To post to this group, send email to project-...@googlegroups.com
> > .
> > >> To unsubscribe from this group, send email to
> > .
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/project-voldemort?hl=en.
>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "project-voldemort" group.
> > >> To post to this group, send email to project-...@googlegroups.com
> > .
> > >> To unsubscribe from this group, send email to
> > .
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/project-voldemort?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "project-voldemort" group.
> > > To post to this group, send email to project-...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/project-voldemort?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "project-voldemort" group.
> > To post to this group, send email to project-...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/project-voldemort?hl=en.
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
Bruce Ritchie
On Mar 25, 3:53 pm, Ron Reynolds <tequila...@ymail.com> wrote:
> RMI has solved loading of never-before-seen classes before using the URLClassLoader for years
I guess I discounted RMI immediately as I doubt it would be acceptable
at all for most use cases.
I've been thinking about this some more and I was thinking that
perhaps a different approach might be in order for this. What if
instead of passing in a filter class we pass in an expression that can
be parsed and used by an EL library such as MVEL? Is there any use
cases now for filters where an EL expression would not be a sufficient
replacement?
Regards,
Bruce Ritchie
Tatu Saloranta
>
> On Mar 25, 3:53 pm, Ron Reynolds <tequila...@ymail.com> wrote:
>> RMI has solved loading of never-before-seen classes before using the URLClassLoader for years
>
> I guess I discounted RMI immediately as I doubt it would be acceptable
> at all for most use cases.
Related to this: something that works "as well as RMI, just better"
that might be worth considering is Dirmi:
http://sourceforge.net/projects/dirmi/
(and see http://www.jroller.com/broneill/entry/advancing_java_remote_method_invocation
for more info)
I attended talk by its author (a very good programmer with lengthy
experience with large-scale systems, ORM etc at Amazon), and he
generally knows very well what he is doing.
So when he explain why this lib solves many of the issues JDK RMI has,
I tend to take claims seriously. It is bit surprising that the
framework has gotten so little attention (but it has been heavily used
in at least one company).
Anyway: this is just long way of saying that not only does bog
standard RMI work for lots of cases, there are even better
implementations of the same general idea, which might be worth looking
at.
-+ Tatu +-