Kademlia DHT with capnp

[harlan...@googlemail.com]

I've been trying to implement a Kademlia DHT with capnproto, but I cannot work out how to get hold of a client's address.

Whilst scrolling through the various discussions, I have seen that the three-party capabilities interface is recommended for similar uses. However, it appears that the current implementation either does not exist, or would route it throught all the nodes, which since this network should scale to many nodes, would be impractical.

I have been using the Ez RPC interface, if that makes any difference.

Does anyone have any ideas?

[harlan...@googlemail.com]

Ah well, I guess I'll have to use gRPC. I much prefer the schema of capnp, but I would like to get this to work.

[Kenton Varda]

Hi Harlan,

Sorry, but I don't understand your original question. I don't see how obtaining the client's address is related to three-party handoff. I'm not sure I understand exactly what the issue is here; maybe you could provide more details?

gRPC doesn't have any concept of three-party handoff -- this is a (planned) feature that only makes sense in object-capability protocols like Cap'n Proto. So if your problem is that 3PH is not yet implemented in Cap'n Proto, I don't understand how using gRPC would solve that.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/93ce1980-9f11-4db9-91b2-607d8dc79e71%40googlegroups.com.

[harlan...@googlemail.com]

Sorry, for not making myself clear!

For a peer-to-peer system to work, there needs to be some way for nodes to be introduced to each other. This is generally done by sending over an IP address, but (in my understanding) the 3PH system would also facilitate this introduction.

3PH rather appealed to me, as it would mean that the initial connection between nodes would be a lot simpler, as I would not have to verify the addresses manually (and end up getting it wrong), nor would I have to iterate through a list, connecting to each one.

gRPC has "grpc::ClientContext::peer", which returns the URI of the remote. I was wondering if capnp had something similar.

Thanks for your help!

On Tuesday, 9 July 2019 21:42:38 UTC+1, Kenton Varda wrote:

Hi Harlan,

Sorry, but I don't understand your original question. I don't see how obtaining the client's address is related to three-party handoff. I'm not sure I understand exactly what the issue is here; maybe you could provide more details?

gRPC doesn't have any concept of three-party handoff -- this is a (planned) feature that only makes sense in object-capability protocols like Cap'n Proto. So if your problem is that 3PH is not yet implemented in Cap'n Proto, I don't understand how using gRPC would solve that.

-Kenton

On Tue, Jul 9, 2019 at 2:59 AM harlan.connor via Cap'n Proto <capn...@googlegroups.com> wrote:

Ah well, I guess I'll have to use gRPC. I much prefer the schema of capnp, but I would like to get this to work.

On Monday, 1 July 2019 22:45:56 UTC+1, Harlan Connor wrote:

I've been trying to implement a Kademlia DHT with capnproto, but I cannot work out how to get hold of a client's address.

Whilst scrolling through the various discussions, I have seen that the three-party capabilities interface is recommended for similar uses. However, it appears that the current implementation either does not exist, or would route it throught all the nodes, which since this network should scale to many nodes, would be impractical.

I have been using the Ez RPC interface, if that makes any difference.

Does anyone have any ideas?

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.

[Ian Denhardt]

With a hypothetical level-3 capnproto implementation, I don't think you
actually need this feature; you could do something like this:

interface Node {
ping @0 ();
store @1 (key :Data, val :Data);
findNode @2 (id :NodeId) -> (results :List(Result));
findValue @3 ...;
}

struct Result {
id @0 :NodeId;
node @1 :Node;
}


When doing a findNode, the receiver just hands back a direct reference
to the nodes, so you don't need to know where they are at all; routing
messages would be handled by capnproto itself.

Note that with this arrangement, it's technically possible for more than
one "Node" to live on a single actual machine. This is probably not
terribly useful in this case (except maybe for testing), but shouldn't
break anything.

Of course, none of the current implementations support level 3, so 3PH
isn't actually available.

Quoting harlan.connor via Cap'n Proto (2019-07-10 05:14:49)

> send an email to [2]capn...@googlegroups.com.
> Visit this group at [3]https://groups.google.com/group/capnproto.

> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/capnproto/93ce1980-9f11-
> 4db9-91b2-607d8dc79e71%40googlegroups.com.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [5]capnproto+...@googlegroups.com.
> Visit this group at [6]https://groups.google.com/group/capnproto.

> To view this discussion on the web visit

> [7]https://groups.google.com/d/msgid/capnproto/c1e33090-2666-4291-b2e6-
> 9c83ddbb82fd%40googlegroups.com.
>
> Verweise
>
> 1. javascript:/
> 2. javascript:/
> 3. https://groups.google.com/group/capnproto
> 4. https://groups.google.com/d/msgid/capnproto/93ce1980-9f11-4db9-91b2-607d8dc79e71%40googlegroups.com?utm_medium=email&utm_source=footer
> 5. mailto:capnproto+...@googlegroups.com
> 6. https://groups.google.com/group/capnproto
> 7. https://groups.google.com/d/msgid/capnproto/c1e33090-2666-4291-b2e6-9c83ddbb82fd%40googlegroups.com?utm_medium=email&utm_source=footer

[Kenton Varda]

Hi Harlan,

In the C++ library, you can get the client's IP address if you decompose a few classes. First, make sure you're using rpc-twoparty.h directly and not ez-rpc.h. On the server side, you'll want to copy the class capnp::TwoPartyServer and modify it a bit. When the code receives a connection -- whose type is `kj::AsyncIoStream` -- you will want to call stream->getpeername() to get the client's IP address. You can then construct a bootstrap capability to use for that specific connection, and pass the IP address into its constructor. Now your server knows the client IP!

(In theory, you're supposed to get a similar result more easily by implementing the `BootstrapFactory` interface... however, with TwoPartyVatNetwork, this interface currently doesn't tell you anything interesting except that the other side is a "client". In theory with a multi-party VatNetwork implementation, you'd expect the VatId passed to the factory function to have more meaningful information, but we don't have that at present.)

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

Visit this group at https://groups.google.com/group/capnproto.

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/c1e33090-2666-4291-b2e6-9c83ddbb82fd%40googlegroups.com.

[Harlan Connor]

Brilliant! I shall have a go at that tomorrow.

Would it be possible for me to use udp instead of tcp, or would that be pushing it too far? 

[Kenton Varda]

Hi Harlan,

You could write a custom VatNetwork implementation that uses UDP, but it'd be a big project. We don't have anything off-the-shelf for that today.

-Kenton

[harlan...@googlemail.com]

After spending a few hours reading through and trying to implement a TwoPartyServer with the right stuff, I have realised that I lack the understanding to implement this properly, and so this would take quite a while to get right.

If I decide to turn this into something that I actually rely on, the I will dedicate the time to understand exactly how to do this, but until then, I shall use gRPC, which exposes the endpoint directly to each RPC call. Whilst this will no doubt be orders of magnitude slower, and harder to maintain, this project is just my attempt to understand Kademlia (and RPC systems in general), and will almost certainly be lost in my project folder, without ever making it out to the internet.

Thanks for all of your help!

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/c1e33090-2666-4291-b2e6-9c83ddbb82fd%40googlegroups.com.