possible to create p2p large-world network with yami

[twitchyliquid64]

you know guntella? its a p2p network that route data by hopping it
from node to node.

all nodes are equal- all act as routers and clients.

my question is: is yami capable of creating a network like this, where
messages are autorouted between nodes?

thanks

[Maciej Sobczak]

Hello,

YAMI4 is capable of message delivery between communicating parties and
as such can be a foundation for arbitrarily complex distributed
system. Note that in such a p2p network the communication between
peers is a relatively simple problem that does not require any
sophisticated solution. What is a real problem here is a routing logic
that has to be implemented on top of the transport and according to
how the p2p network is supposed to work. I am sure that YAMI4 can be
effectively used as a transport for arbitrary high-level routing
scheme, but there is nothing built-in to be used out of the box as a
routing engine.
In short, there is work to be done at higher levels.

We might try to think whether it is possible to define a routing
scheme that is universal/flexible enough to be a turn-key solution for
p2p networks. If it is possible, then such a project might be an
interesting extension to the YAMI4 distribution. Or even a separate
project altogether.

Don't hesitate to describe your idea in more detail - in particular,
try to describe how you would like to use it.

Regards,

--
Maciej Sobczak * http://www.inspirel.com

[twitchyliquid64]

On May 23, 5:22 pm, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
> Don't hesitate to describe your idea in more detail - in particular,
> try to describe how you would like to use it.

Firstly, let me explain my intentions, then I shall explain my
intended solution.

I am trying to create a system/library in C++ that will allow the
creation of arbitrarily sized/complex P2P networks, from which
arbitrary blocks of data can be routed to any node. The system must be
capable of scaling to 40K nodes, so obviously efficiency, overhead,
and routing mechanisms are of a concern. Lastly, it must work on
windows and Linux. The eventual endpoint years down the track is a de-
centralised social network, for the given scale.

My Design:

All nodes have an identification number; a 64bit number unique to the
network. This is used as their 'address' of sorts.
All nodes send out a BONJOUR message, which propergates to all other
nodes within a certian hopcount. This serves a number of purposes; it
allows nodes to be aware of presence of specific nodes, and also,
forms the basis of determining the optimal path to that node.
You see, every BONJOUR message has attached to it a count, which is
incremented each time the message is propergated/flooded again. That
way, the most optimal path to every node in range can be detected by
simply inspecting these packets and remembering which socket sent us
the bonjour message with the smallest hopcount for a given node. Then,
by sending future messages through that socket, it will follow the
optimal path to the destination. Obviously, it will store other
options as well so if the path is broken, other paths are still known.
This will be stored in a hashtable for efficient lookup. The only
downside is memory usage, but even in a table that stores 40K nodes'
information will still be under a MB.

Of course, all messages sent through the network will also have a
messageID and a TTL, to prevent a single message getting stuck in a
loop and saturating bandwidth.
In terms of max. expected load, I would say about 40K nodes, each with
about 8-9 TCP connections, which amounts to processing about 30-50
small messages a second.

So the immediate decision is this: Is YAMI4 fast/scalable enough for
this? or is the parameters thingy and all the other stuff too much
overhead?
Ultimately, if Bandwidth is going to be saturated before CPU is, I
would use YAMI4.
Similarly, if CPU is going to be saturated before bandwidth then I
would have to write my own socket handling etc without the overhead of
parameters etc.
You would have a better idea of YAMI's overhead than me.

On May 23, 5:22 pm, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
>If it is possible, then such a project might be an
>interesting extension to the YAMI4 distribution.

If I write it with YAMI, have no problem writing it as an extension,
so that you can plug it into the main sourcetree if you so wish, or it
can be available to others.

[Maciej Sobczak]

On May 23, 11:55 am, twitchyliquid64 <hyper...@gmail.com> wrote:

> I am trying to create a system/library in C++ that will allow the
> creation of arbitrarily sized/complex P2P networks, from which
> arbitrary blocks of data can be routed to any node. The system must be
> capable of scaling to 40K nodes, so obviously efficiency, overhead,
> and routing mechanisms are of a concern.

No. It does not really matter how many nodes there are - what does
matter is how many connections are visible from the point of view of
the particular node.
As an analogy - there is no need for HTTP to give any special
provisions for super-scalability, yet there are billions of HTTP
interactions every day. It is perfectly possible, because they do not
happen in a single place. Same for mails, etc.
If you expect your p2p network to have the connectivity order (the
number of connections at any single node) like 5 or 10, then really
any communication approach will do.

> All nodes have an identification number; a 64bit number unique to the
> network. This is used as their 'address' of sorts.
> All nodes send out a BONJOUR message, which propergates to all other
> nodes within a certian hopcount.

[...]

You are going to reinvent some routing protocol like RIP or OSPF.

> In terms of max. expected load, I would say about 40K nodes, each with
> about 8-9 TCP connections, which amounts to processing about 30-50
> small messages a second.

This is peanuts for YAMI4, but you will also have to add expected
message payload to have a full picture.
If you say that messages are small (that is, in the order of several
kB. not MB), then you should not expect any problems.

> Ultimately, if Bandwidth is going to be saturated before CPU is, I
> would use YAMI4.
> Similarly, if CPU is going to be saturated before bandwidth then I
> would have to write my own socket handling etc without the overhead of
> parameters etc.

This will depend on the relation between your CPU and network
capacity. I have seen a system where two machines were connected with
a gigabit Ethernet and just one switch and where CPUs were fully
utilized while the network interface was working way below its
capacity. But if your network is much slower (that will be the case
with long-distance connections) or the messages are relatively big (so
that there less overhead in processing message headers), then you
might get it the other way round.

[twitchyliquid64]

> No. It does not really matter how many nodes there are - what does
> matter is how many connections are visible from the point of view of
> the particular node.
> As an analogy - there is no need for HTTP to give any special
> provisions for super-scalability, yet there are billions of HTTP
> interactions every day. It is perfectly possible, because they do not
> happen in a single place. Same for mails, etc.
> If you expect your p2p network to have the connectivity order (the
> number of connections at any single node) like 5 or 10, then really
> any communication approach will do.

Nearly all, if not all nodes will be in range. If the TTL is 5-6, and
all nodes have the default 8-9 connections, then worst-case there
would be 262 144 or more nodes in range, far exceeding intended
scalability (and TBH that many people would never use my network).
Regardless, I understand what you are saying.

> You are going to reinvent some routing protocol like RIP or OSPF.

Haha probably. Its not like its a bad idea, not to mention Ive never
liked the other structures like path folding, ring networks, chaining

etc.



>This will depend on the relation between your CPU and network
>capacity. I have seen a system where two machines were connected with
>a gigabit Ethernet and just one switch and where CPUs were fully
>utilized while the network interface was working way below its
>capacity. But if your network is much slower (that will be the case
>with long-distance connections) or the messages are relatively big (so
>that there less overhead in processing message headers), then you
>might get it the other way round.

This network is designed for home connections, over the internet (over
arbitrary distances). At the very best you might get some basic VPS's.
The majority of the overhead will be small messages, with sizes of
less that 26 bytes. The network only transfers text (a few KB at most)
so once again, not much data there.
The problem is, I have built networks that are p2p before, but Ive
never had enough people use it to get anywhere near testing the
scalability of my systems. Alot of what I predict is speculation and
timing how long it takes for computers to do things. I really have no
idea. I wish I knew a way to test it as if it was a 40K network on a
single machine.


Ok im going to give it a go writing it to use YAMI4. Ill let you know
when/if I finish in a few months if you like :P

[Maciej Sobczak]

On 24 Maj, 12:30, twitchyliquid64 <hyper...@gmail.com> wrote:

> Ok im going to give it a go writing it to use YAMI4. Ill let you know
> when/if I finish in a few months if you like :P

Sure! Just don't hesitate to share your experiences as you go without
waiting until you finish this project completely - it can be very
beneficial for everybody to learn what are your challenges and I hope
you might also benefit from some tips on how to use YAMI4 in the most
optimal way.

[twitchyliquid64]

On May 25, 7:51 am, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
> On 24 Maj, 12:30, twitchyliquid64 <hyper...@gmail.com> wrote:
>
> > Ok im going to give it a go writing it to use YAMI4. Ill let you know
> > when/if I finish in a few months if you like :P
>
> Sure! Just don't hesitate to share your experiences as you go without
> waiting until you finish this project completely - it can be very
> beneficial for everybody to learn what are your challenges and I hope
> you might also benefit from some tips on how to use YAMI4 in the most
> optimal way.

Will do.


Heyy Im trying to get the thing to compile on windows. I can compile
the core (minGW) fine, but it appears your sourcetree is missing the
required yami4-core/dll.h,
I get this error when compiling the 'cpp' src folder, using either
your VS makefile script or me manually compiling with MinGW.

Do you have a copy of this file and any other missing ones?

[Maciej Sobczak]

On 25 Maj, 09:57, twitchyliquid64 <hyper...@gmail.com> wrote:

> Heyy Im trying to get the thing to compile on windows. I can compile
> the core (minGW) fine, but it appears your sourcetree is missing the
> required yami4-core/dll.h,
> I get this error when compiling the 'cpp' src folder, using either
> your VS makefile script or me manually compiling with MinGW.
>
> Do you have a copy of this file and any other missing ones?

This file exists in src/core - the bug is in the Makefile.MinGW, which
does not copy this file to the include directory. Please try copying
it manually, it should work.

[twitchyliquid64]

On May 26, 5:06 pm, Maciej Sobczak <see.my.homep...@gmail.com> wrote:
> This file exists in src/core - the bug is in the Makefile.MinGW, which
> does not copy this file to the include directory. Please try copying
> it manually, it should work.

Thanks. Fixed.

One last question then I should be good to rock n roll.

As I understand UDP is connectionless and data integrity is not
maintained, and the packet may be lost (unlike TCP which solves all
these problems, as you know)

My question is: does YAMI4 address these issues? does YAMI4 make sure
your messages sent via UDP transport are intact?

Thanks,

Twitchyliquid64

[Maciej Sobczak]

On 27 Maj, 01:50, twitchyliquid64 <hyper...@gmail.com> wrote:

> One last question then I should be good to rock n roll.
>
> As I understand UDP is connectionless and data integrity is not
> maintained, and the packet may be lost (unlike TCP which solves all
> these problems, as you know)
>
> My question is: does YAMI4 address these issues? does YAMI4 make sure
> your messages sent via UDP transport are intact?

No, YAMI4 does not solve UDP problems. The reason for this is that
solving these problems would require implementing all the stream
integrity features of TCP, which... are already implemented in TCP.
There is no reason to implement "TCP over UDP".

What YAMI4 does, however, is ensuring the packet order within a single
bigger message. This feature exists in YAMI4 anyway and is related to
message priorities, but it can be used to address some of the issues
of UDP. That is, if you send a bigger message ("bigger" means it will
not fit in a single frame and will have to be chopped into smaller
pieces) and individual frames will get reordered when transmitted over
UDP, YAMI4 will fix that and will reconstruct the complete message
from properly ordered frames. However, it will not resend lost frames.
This means that if you decide to use UDP, you should carefully judge
the risks in your particular network environment - it might be very
well the case that in your network UDP packets are never lost and then
there is no risk in using UDP with YAMI4.
On the other hand, if you know from previous statistics that your
network loses UDP packets, then you should only use UDP for those use
cases where this brings no harm. Actually, there are many such use
cases (like: a termometer that sends current temperature every second
- there is no problem if the message will occasionally get lost), but
you should judge this carefully in the context of your target system.

Regards,

[twitchyliquid64]

Thanks. Its slowly coming along. Im about to implement the routing
system.
One quick question, does YAMI4 support IPv6?

[Maciej Sobczak]

On 5 Cze, 11:54, twitchyliquid64 <hyper...@gmail.com> wrote:

> One quick question, does YAMI4 support IPv6?

No, there is no direct support for this at the moment.

The good news, however, is that there is nothing is YAMI4 that would
prohibit it or even make it difficult and adding support for IPv6 can
be done in a way that is completely transparent to users.

[Ricardo Aguirre]

Hi Maciej,

I'm analyzing, how to build a p2p, and then I remember this thread.

Do you heard something else about this old project/intent?

By the way finally I have a free time, 

I remember that one time you asked me, if I could do something, but don't remember exactly what activity. This week I'll be free, so if I can help just let me know

Cheers


--Panzón

2012/6/6 Maciej Sobczak <see.my....@gmail.com>

[Maciej Sobczak]

Hi,

I'm analyzing, how to build a p2p, and then I remember this thread.

Do you heard something else about this old project/intent?

No, unfortunately I did not hear from the original poster and have no information on how this is evolving.
 

By the way finally I have a free time, 

I remember that one time you asked me, if I could do something, but don't remember exactly what activity. This week I'll be free, so if I can help just let me know

Thank you for your willingness to help, but in fact there are no problems that need to be solved at the moment - just use YAMI4 to build great distributed systems! :-)

[twitchyliquid64]

Ohai. I hear people talking about me :P

 

This particular iteration of this project was terminated. But since then I have partially written two other p2p large world systems.

 

The first one is partially completed and can be found among the source for pyyami; recently released on this group. I can't remember how far I got with it, but I think message routing works, which is the important part.

 

The second one is entirely written in Go. Unless your interested, I dont plan on releasing it.

[Ricardo Aguirre]

I'm interested, if you can share some code, it will be helpful.

Is my first time trying to build a p2p system; 

Right now I'm analyzing how Go can help me. 

So if you have some links, or documentation I'll appreciate

Cheers

--Panzón

2013/7/17 twitchyliquid64 <hype...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "yami4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yami4+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[twitchyliquid64]

The go version is far too complex for your needs and learning.

 

I reccommend downloading my release of pyyami (available on this group) and taking a look in my p2p implementation there. Its half complete, but short and demostrates the main principles.

 

To summerise:

-All nodes randomly connect to each other.

-All messages broadcasted have a random number ('ID') attached to it. All nodes have an index of recently-seen message IDs. If an incoming message has an ID which is in the index, that nodes drops the message; as it has already been received from another link.

-Nodes periodically broadcast ('flood') a hello message to all their links, who propergate it to all theirs and so on. Because of the previous feature, the message propergates the entire network and stops (without looping, or otherwise being repeatedly delivered). The message has a number field 'hopcount' which is incremented every time a node redelivers/rehops a message.

-By looking at the 'hopcount' field of each hello packet, any node can determine which link to send a message down to send a message down the shortest path (by picking the link with the smallest hopcount for that node, and transmitting down it).