Facilitating Phantom Testing
Ernst Rohlicek
* src/test/gencerts.sh: update for easy testing on single machine -
all over localhost with different ports (current script assumes you
have actual different hosts with correct /etc/hosts entries for each)
* src/main.c: extend to parse command line parameters
* src/main.c: add the currently only cmd parameter to set hostname ->
which config file will be loaded (could later be changed to -c
[configfile]) -> easily start multiple phantom instances on same
machine
* src/main.c: fixed a few missing newlines
Patch can be found here (expiration time 1 month):
http://pastebin.com/KrupGum8
License: As rest of Phantom source code.
Open questions:
1. As next hurdle, I get "Failed to join kademlia network\nfailed to
start kad". It tries to connect to the kademlia network, but fails
because the kaddata dir is missing kademlia data files = node lists ->
no nodes to connect to -> exits.
(If I am correct.)
So, how to generate this kademlia node list file? What format does
this have - I have seen the relevant routines in kad_contacts.c, but
which of the files generated by gencerts.sh have to be cat'ed together
to create a valid entry for a valid kid contacts data file?
If possible, I would like to alter the gencerts.sh script to also
generate these kademlia node data files.
2. What is the difference between phantom and phantomd?
3. Please add svn ignores for the generated protobuf C files and src/
test/test* files (all in there except gencerts.sh).
Greetings,
Ernst Rohlicek
Johannes Schlumberger
thank you very much for your interest in Phantom and your contribution.
> * src/test/gencerts.sh: update for easy testing on single machine -
> all over localhost with different ports (current script assumes you
> have actual different hosts with correct /etc/hosts entries for each)
> * src/main.c: extend to parse command line parameters
> * src/main.c: add the currently only cmd parameter to set hostname ->
> which config file will be loaded (could later be changed to -c
> [configfile]) -> easily start multiple phantom instances on same
> machine
> * src/main.c: fixed a few missing newlines
> Patch can be found here (expiration time 1 month):
>
> http://pastebin.com/KrupGum8
>
> License: As rest of Phantom source code.
Thanks. I have applied your patch to the C-code. For the gencerts shellscript, I
decided to have two versions of it. The old version for multiple hosts and your
single host version. There is currently a limitation, that will prohibit your
single host approach from working. In two places in server.c port 8080 is
hard-coded during tunnel creation. I have had a glimpse on it and I have an idea
on what the problem might be. If I find some time I will give it a try and fix
it, for now please be advised that this issue exists.
> Open questions:
>
> 1. As next hurdle, I get "Failed to join kademlia network\nfailed to
> start kad". It tries to connect to the kademlia network, but fails
> because the kaddata dir is missing kademlia data files = node lists ->
> no nodes to connect to -> exits.
> (If I am correct.)
I think you are right about this. As any DHT there is a bootstrapping problem if
there is no other current node running. This has to be solved by bringing up two
nodes simultaneously with pre-crafted contact information. Other nodes can then
join the DHT by knowing one of these two nodes' contact information.
> So, how to generate this kademlia node list file? What format does
> this have - I have seen the relevant routines in kad_contacts.c, but
> which of the files generated by gencerts.sh have to be cat'ed together
> to create a valid entry for a valid kid contacts data file?
I have attached one of my old kadnodes.list file as an example. The format is as
follows:
1. SHA-1 hash of the contact's communication certificate (This is the same as the
node's kademlia ID)
2. Port of the contact, IP of the contact, length of item 3, length of item 4
3. The communication certificate of the contact (this would be the -cc.pem file
for this contact)
4. The path building certificate of the contact (this would be the -pbc.pem file
for this contact)
I hope I remember this correctly.
> If possible, I would like to alter the gencerts.sh script to also
> generate these kademlia node data files.
That would be very helpful.
> 2. What is the difference between phantom and phantomd?
phantom is the phantom implementation itself.
phantomd is a quick hack to add and remove ipv6 (phantom) addresses on the
phantom tunnel device on phantom's request. It is a separate daemon for
privilege separation reasons (check phantomd.c).
> 3. Please add svn ignores for the generated protobuf C files and src/
> test/test* files (all in there except gencerts.sh).
I am not very knowledgeable about SVN, but it seems to me, SVN ignore properties
are stored locally in a repository (there is no such thing as a .cvsignore,
.gitignore) that can be part of the repository and "just work". If I am wrong
here, please correct me.
best regards,
Johannes
Ernst Rohlicek
> Hi Ernst,
> thank you very much for your interest in Phantom and your contribution.
> There is currently a limitation, that will prohibit your
> single host approach from working. In two places in server.c port 8080 is
> hard-coded during tunnel creation. I have had a glimpse on it and I have an idea
> on what the problem might be. If I find some time I will give it a try and fix
> it, for now please be advised that this issue exists.
run into this :-)
> > If possible, I would like to alter the gencerts.sh script to also
> > generate these kademlia node data files.
>
> That would be very helpful.
or whatever you think makes sense):
http://pastebin.com/3d6pQFC0
License: As rest of Phantom source code.
kadnodes.list - which refers to faui00m to the byte (checked with
hexdump).
I will try to get a testing setup running with these generated files.
>
> > 2. What is the difference between phantom and phantomd?
>
> phantom is the phantom implementation itself.
> phantomd is a quick hack to add and remove ipv6 (phantom) addresses on the
> phantom tunnel device on phantom's request. It is a separate daemon for
> privilege separation reasons (check phantomd.c).
like "this AP address should be reached via our virtual phantom TUN/
TAP interface phantom0"?
If I understand correctly, phantomd then has to be run before phantom
- and by root?
Is phantomd then the "handler" program to handle the incoming &
outgoing data packets from the TUN/TAP device - or does phantom
(without "d") already handle that?
I guess setting up the TUN/TAP devices are already the next thing
after connecting to Kad. Lets see what happens :-)
Greetings,
Ernst Rohlicek
Johannes Schlumberger
> > There is currently a limitation, that will prohibit your
> > single host approach from working. In two places in server.c port 8080 is
> > hard-coded during tunnel creation. I have had a glimpse on it and I have an idea
> > on what the problem might be. If I find some time I will give it a try and fix
> > it, for now please be advised that this issue exists.
>
> That would be great, and thank you for the pointer. Lets see when I
> run into this :-)
At the latest when you try to create the first tunnel between two nodes, I am
pretty busy for this weekend and the start of the coming week, so I will surely
not be able to fix this before the end of next week. Also I currently do not
have a running Phantom test network, so this will need some preparation time as
well. I will eventually get around to fix this, though.
> Alright, here is the script "genkadcontacts.sh" (or genkadnodelists.sh
> or whatever you think makes sense):
> I could replicate the first kad contact entry found in faui00a-
> kadnodes.list - which refers to faui00m to the byte (checked with
> hexdump).
Neat - thanks. I have added the script to the package.
> > > �2. What is the difference between phantom and phantomd?
> >
> > phantom is the phantom implementation itself.
> > phantomd is a quick hack to add and remove ipv6 (phantom) addresses on the
> > phantom tunnel device on phantom's request. It is a separate daemon for
> > privilege separation reasons (check phantomd.c).
>
> Hm, so it handles things like updating "routing tables" and such ...
> like "this AP address should be reached via our virtual phantom TUN/
> TAP interface phantom0"?
Yes, phantomd only sets and removes addresses on the tun interface by calling ip
(8). It does not do anything else. Routing table updates and such are done
automatically by ip then.
> If I understand correctly, phantomd then has to be run before phantom
> - and by root?
Yes, phantomd should be running before phantom and needs to be privileged. It
will open a command socket in your temp directory to communicate with phantom
processes. If you have multiple phantom processes running it might be necessary
to synchronize communication through this socket between the instances.
> Is phantomd then the "handler" program to handle the incoming &
> outgoing data packets from the TUN/TAP device - or does phantom
> (without "d") already handle that?
That is handled by phantom itself (have a look at the tun module (tun.{c,h}).
> I guess setting up the TUN/TAP devices are already the next thing
> after connecting to Kad. Lets see what happens :-)
Good luck, and keep asking questions here. Others might find that very helpful
when trying to reproduce this. I will answer them as good as I can.
best regards,
Johannes
Ernst Rohlicek
> Neat - thanks. I have added the script to the package.
In the gencerts-singlehost.sh script, the line
IP="localhost"
needs to be changed to
IP="127.0.0.1"
IP really means "plain IP address" here; there seems to be no name
resolving done. Otherwise it always fails with "failed to connect to
kad network".
I also added a new runtests script, see the here:
http://pastebin.com/KUb5U8A0
License: As rest of Phantom source code.
> > run into this :-)
>
> At the latest when you try to create the first tunnel between two nodes, I am
> pretty busy for this weekend and the start of the coming week, so I will surely
> not be able to fix this before the end of next week. Also I currently do not
> have a running Phantom test network, so this will need some preparation time as
> well. I will eventually get around to fix this, though.
to construct entry-path" (some say "exit path"). That is good
progress :-)
I am not sure how long this usually takes, but after a while of high
CPU usage, still no nodes have finished with path creation. Some nodes
have tried 3 times so far to create the same type of path.
Then I put some debug output before the two hard-coded port 8080s in
server.c, like so
http://pastebin.com/KUb5U8A0
and these lines already get called at this stage, so I guess this is
the issue you mentioned?
> Good luck, and keep asking questions here. Others might find that very helpful
> when trying to reproduce this. I will answer them as good as I can.
Greetings,
Ernst Rohlicek
Johannes Schlumberger
> In the gencerts-singlehost.sh script, the line
>
> IP="localhost"
>
> needs to be changed to
>
> IP="127.0.0.1"
fixed.
> I also added a new runtests script, see the here:
>
> http://pastebin.com/KUb5U8A0
>
> License: As rest of Phantom source code.
Thanks, while this might be helpful to others, I have decided not to make it
part of the official package since people's ways to test Phantom are manifold
and will often differ from this.
> Now all nodes say, "Join to kad network was sucessful" and "starting
> to construct entry-path" (some say "exit path"). That is good
> progress :-)
>
> I am not sure how long this usually takes, but after a while of high
> CPU usage, still no nodes have finished with path creation. Some nodes
> have tried 3 times so far to create the same type of path.
If nodes fail to create their path they will currently simply retry until they
succeed, which they currently never do.
If it is working correctly it should not take more than a few seconds. to create
a path. You are actually running in the 8080 problem most likely. The
connections will not be accepted on this port and so they just time out and the
path dies.
> Then I put some debug output before the two hard-coded port 8080s in
> server.c, like so
>
> http://pastebin.com/KUb5U8A0
>
> and these lines already get called at this stage, so I guess this is
> the issue you mentioned?
Yes, it is. To get single host testing running, this needs to be fixed first.
You have already come a long way though in setting it up.
Johannes
Ernst Rohlicek
>
> > http://pastebin.com/KUb5U8A0
>
> > License: As rest of Phantom source code.
>
> Thanks, while this might be helpful to others, I have decided not to make it
> part of the official package since people's ways to test Phantom are manifold
> and will often differ from this.
seen other projects benefit from having a general "resources" or "res"
sub-directory in their repository, which contains handy scripts and
examples, which users can use to save time if it fits their particular
use-care or environment, discard them, write their own scripts / files
based on them or draw useful and exemplary ideas from them.
I personally would support creating such a resources or "res" sub-
directory in the Phantom repository.
> If nodes fail to create their path they will currently simply retry until they
> succeed, which they currently never do.
> If it is working correctly it should not take more than a few seconds. to create
> a path. You are actually running in the 8080 problem most likely. The
> connections will not be accepted on this port and so they just time out and the
> path dies.
>
> > Then I put some debug output before the two hard-coded port 8080s in
> > server.c, like so
>
> > http://pastebin.com/KUb5U8A0
>
> > and these lines already get called at this stage, so I guess this is
> > the issue you mentioned?
>
> Yes, it is. To get single host testing running, this needs to be fixed first.
> You have already come a long way though in setting it up.
I am looking forward for a fix to these hard-coded routines when you
find the time as you mentioned.I will monitor the svn repository and
news feed.
Thanks and greetings,
Ernst Rohlicek
Johannes Schlumberger
> While I do respect your decision, I would like to comment that I have
> seen other projects benefit from having a general "resources" or "res"
> sub-directory in their repository, which contains handy scripts and
> examples, which users can use to save time if it fits their particular
> use-care or environment, discard them, write their own scripts / files
> based on them or draw useful and exemplary ideas from them.
>
> I personally would support creating such a resources or "res" sub-
> directory in the Phantom repository.
As soon as the script is more than a couple of lines I will consider adding a
res directory, containgin this and maybe other scripts.
> > Yes, it is. To get single host testing running, this needs to be fixed first.
> > You have already come a long way though in setting it up.
>
> Thank you for the clarification :-)
You are welcome.
> I am looking forward for a fix to these hard-coded routines when you
> find the time as you mentioned.I will monitor the svn repository and
> news feed.
I will come back to you, once this issue is fixed. A master student from UCSB is
currently setting up a phantom test environment. Once this is running, I will
give it a shot.
best regards,
Johannes
grarpamp
I've done similar things here, just been too busy to post :(
I'd bet port number flexibility, or even randomization, will happen
eventually due to the various firewalls people need to deal with,
and obscurations they may desire.
Matthew Willis
Johannes Schlumberger
> Is the test network you mentioned available to the public? I'm interested in
> trying phantom out.
You currently still have to setup your own test network if you want to play with
phantom. I plainly did not have time to work on the implementation and an easy
test network it during the last couple of months.
Johannes