Fetch via proxy?

[David Ongaro]

For performance reasons we can not fetch directly from our cluster in one of our datacenters. Instead we have to go over a proxy. Unfortunately I can not see any proxy setting, neither on BnP nor on server side (there is a proxy setting for the balancer, but I guess that's unrelated).

Did I overlook something? Do we have to implement that? Or do we have to install a service like https://hadoop.apache.org/docs/r2.4.1/hadoop-hdfs-httpfs on the proxy? How is this problem handled at linkedin?

Thanks and best regards

David Ongaro

[Arunachalam]

You could try setting java system level proxy ( http or https) depending on your need.

http://stackoverflow.com/questions/15927079/how-to-use-httpsurlconnection-through-proxy-by-setproperty

Thanks,

Arun.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Arunachalam]

If you don't want to change the code, you can set it at the JVM level.

http://stackoverflow.com/questions/120797/how-do-i-set-the-proxy-to-be-used-by-the-jvm

Alternatively, you can contribute the patch to the open source. But please note that if you are modifying the server, setting these proxies will affect any outbound http connections.

Thanks,

Arun.

[David Ongaro]

But wouldn't that affect all http communication? E.g. inter node communication or even client communication? I'm not sure what's running via http or directly via tcp (which probably wouldn't be affected by this property?). But since Voldemort has several REST services I guess a lot is running over http. In any case we only want to proxy the fetch traffic.

On Thursday, October 27, 2016 at 5:44:26 PM UTC-7, Arun Thirupathi wrote:

You could try setting java system level proxy ( http or https) depending on your need.

http://stackoverflow.com/questions/15927079/how-to-use-httpsurlconnection-through-proxy-by-setproperty

Thanks,

Arun.

On Thu, Oct 27, 2016 at 3:11 PM, David Ongaro <bitt...@gmail.com> wrote:

For performance reasons we can not fetch directly from our cluster in one of our datacenters. Instead we have to go over a proxy. Unfortunately I can not see any proxy setting, neither on BnP nor on server side (there is a proxy setting for the balancer, but I guess that's unrelated).

Did I overlook something? Do we have to implement that? Or do we have to install a service like https://hadoop.apache.org/docs/r2.4.1/hadoop-hdfs-httpfs on the proxy? How is this problem handled at linkedin?

Thanks and best regards

David Ongaro

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.

[Arunachalam]

Voldemort default is to use TCP for client to server and server to server communication. But you can enable the http access to server from client, I don't have much experience with that.

Thanks,

Arun.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

[David Ongaro]

That's a good link actually, because it reveals that there is also a http.nonProxyHosts JVM property, so we can avoid http communication via proxy in the same network. I guess we don't need any code changes then. But of course if you can implement it as a BnP setting we also take that ;).

On Thursday, October 27, 2016 at 5:46:02 PM UTC-7, Arun Thirupathi wrote:

If you don't want to change the code, you can set it at the JVM level.

http://stackoverflow.com/questions/120797/how-do-i-set-the-proxy-to-be-used-by-the-jvm

Alternatively, you can contribute the patch to the open source. But please note that if you are modifying the server, setting these proxies will affect any outbound http connections.

Thanks,

Arun.

On Thu, Oct 27, 2016 at 5:44 PM, Arunachalam <arunac...@gmail.com> wrote:

You could try setting java system level proxy ( http or https) depending on your need.

http://stackoverflow.com/questions/15927079/how-to-use-httpsurlconnection-through-proxy-by-setproperty

Thanks,

Arun.

On Thu, Oct 27, 2016 at 3:11 PM, David Ongaro <bitt...@gmail.com> wrote:

For performance reasons we can not fetch directly from our cluster in one of our datacenters. Instead we have to go over a proxy. Unfortunately I can not see any proxy setting, neither on BnP nor on server side (there is a proxy setting for the balancer, but I guess that's unrelated).

Did I overlook something? Do we have to implement that? Or do we have to install a service like https://hadoop.apache.org/docs/r2.4.1/hadoop-hdfs-httpfs on the proxy? How is this problem handled at linkedin?

Thanks and best regards

David Ongaro

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.