How to use restful and mdp

[Lars Baumgaertner]

Hi,

I started playing around with the restful interface, trying to add some commands to it. After figuring out how to add  HANDLERS, FEATURES etc serving static content works really well.

Then I tried to add more advanced stuff using MDP. As a quick example here I copied the function from cli peercount:

static int restful_net_peercount(httpd_request *r, const char *remainder)

{

if (*remainder)

return 404;

char buf[2048];

int mdp_sockfd;

if ((mdp_sockfd = overlay_mdp_client_socket()) < 0)

return WHY("Cannot create MDP socket");

overlay_mdp_frame a;

bzero(&a, sizeof(overlay_mdp_frame));

a.packetTypeAndFlags=MDP_GETADDRS;

a.addrlist.mode = MDP_ADDRLIST_MODE_ROUTABLE_PEERS;

a.addrlist.first_sid = OVERLAY_MDP_ADDRLIST_MAX_SID_COUNT;

DEBUGF(mdprequests, "Send MDP_GETADDRS mode=MDP_ADDRLIST_MODE_ROUTABLE_PEERS first_sid=%u last_sid=%u frame_sid_count=%u",

a.addrlist.first_sid,

a.addrlist.last_sid,

a.addrlist.frame_sid_count,

a.addrlist.server_sid_count);

int ret=overlay_mdp_send(mdp_sockfd, &a,MDP_AWAITREPLY,5000);

overlay_mdp_client_close(mdp_sockfd);

if (ret){

if (a.packetTypeAndFlags==MDP_ERROR)

return WHYF(" MDP Server error #%d: '%s'",a.error.error,a.error.message);

return WHYF("Failed to send request");

}

//cli_put_long(context, a.addrlist.server_sid_count, "\n");*/

snprintf(buf,2048, "%i", a.addrlist.server_sid_count);

http_request_response_static(&r->http, 200, &CONTENT_TYPE_TEXT,buf,strlen(buf));

//http_request_response_generated(&r->http, 200, &CONTENT_TYPE_TEXT, restful_keyring_identitylist_json_content);

return 1;

}

If I just snprintf() a static number without the code from peercount it just works. But once the mdp code is in there the call fails with an internal server error. Serval log tells me the following:

INFO: [ 2442] 15:49:53.819 overlay_interface.c:654:overlay_interface_init()  Allowing a maximum of 7 packets every 5ms
INFO: [ 2442] 15:49:53.819 keyring.c:1413:keyring_inmemory_identity()  Created in-memory identity: 1326FBB259AE28052C66535471188A1F76A3EADB9DDDFD829431CC0C18C4CD5B
INFO: [ 2442] 15:50:07.702 [httpd/1] httpd.c:286:httpd_server_poll()  HTTP SERVER, ACCEPT AF_INET:127.0.0.1:60777
INFO: [ 2442] 15:50:07.702 [httpd/1] httpd.c:59:httpd_dispatch()  HTTP SERVER, GET /restful/net/peercount
ERROR:[ 2442] 15:50:12.703 [httpd/1] net_restful.c:155:restful_net_peercount()    MDP Server error #1: 'Timeout waiting for reply to MDP packet (packet was successfully sent).'
ERROR:[ 2442] 15:50:12.703 [httpd/1] http_server.c:1930:http_request_receive()  Internal failure parsing HTTP request: invalid result code -1
INFO: [ 2442] 15:50:12.703 performance_timing.c:80:fd_showstat()  5001ms (100.0%) in 1 calls (max 5001ms, avg 5001.0ms, +child avg 5001.0ms) : http_request_receive
INFO: [ 2442] 15:50:12.704 overlay_mdp.c:323:overlay_mdp_reply()  SOCK: AF_UNIX:"/tmp/s/mdp.client.2442.00000001.socket\0\0"
ERROR:[ 2442] 15:50:12.704 overlay_mdp.c:326:overlay_mdp_reply()  sendto(fd=3,len=22,addr=AF_UNIX:"/tmp/s/mdp.client.2442.00000001.socket\0\0"): No such file or directory [errno=2]
INFO: [ 2442] 15:50:12.704 overlay_mdp.c:329:overlay_mdp_reply()  Closing dead MDP client 'AF_UNIX:"/tmp/s/mdp.client.2442.00000001.socket\0\0"'

The SOCK output line was from me to compare the address with the one from the cli. Both using the same address on macOS.

What am I missing?

[Jeremy Lakeman]

Your code is running within the server's single thread, so you blocked it from processing the incoming message.

Since your code is running within the server, you have access to all static state. So you could just copy / call / refactor the existing processing code;

https://github.com/servalproject/serval-dna/blob/development/overlay_mdp.c#L1788

The API the server uses internally for MDP is a little different. If you want to send and receive packets between network peers, you need to write event driven code like this;
https://github.com/servalproject/serval-dna/blob/development/overlay_mdp_echo.c

While waiting for network traffic, you can pause your HTTP processing and specify a failure timeout, eg;

https://github.com/servalproject/serval-dna/blob/development/rhizome_restful.c#L335

And resume it again later;

https://github.com/servalproject/serval-dna/blob/development/rhizome_restful.c#L288

If you are planning on building some kind of restful routing table functions, Andrew has started writing down the design we've been discussing internally in the route-rest-api branch.

--
You received this message because you are subscribed to the Google Groups "Serval Project Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to serval-project-developers+unsub...@googlegroups.com.
To post to this group, send email to serval-project-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/serval-project-developers.
For more options, visit https://groups.google.com/d/optout.

[Lars Baumgaertner]

Okay thanks for the clarification. The single threaded design makes it complicated. I forgot that we already learned that the hard way in our big evaluation in 2016 :)

Pulling all the small bits and pieces out of the serval internals without a clear internal API is too error prone for me and as Andrew is already on the task of implementing the routing API, we will just wait a bit longer and hope for a release "really soon[tm]". Even though I only found the documentation in route-rest-api not an implementation.

At the moment Andrew seem to be the one working on all the stuff we want (iOS library, route-rest-api etc) :D

King regards,

 Lars

To post to this group, send email to serval-proje...@googlegroups.com.

[Andrew Bettison]

Hi Lars,

It certainly looks like the pieces you need are the very next ones on my to-do list:

1. build for iOS and support for Xcode projects
2. Serval DNA routing REST API (and Swift API)

I am currently working on item 1.  I hope to have it at the stage shortly that there are written instructions for importing Serval DNA into an Xcode project, together with a sample iPhone app that runs the daemon in a thread and makes requests via the REST interface.  I will follow up on this group discussion thread once it is ready for you to try, and will try to respond quickly to issues as you encounter them.

Then I will turn my attention to item 2.  Implementing the Routing API involves adding some extra functions to the daemon's in-memory routing data structure, mainly to support the "newsince" request.  It is not really obvious how to go about that without fairly deep knowledge of (a) the routing data structures and (b) the proper patterns for implementing REST requests, but I have discussed it with Jeremy and we have a pretty good plan.

I wish that the Serval DNA source code were simpler, clearer, and hence more maintainable, but unfortunately, in the C language, the pursuit of security, CPU performance, memory economy and minimal external dependencies tends to greatly complicate the code.  A modern and more expressive language like D or C++14 or even Swift or Rust might offer a solution, but we are a long way from even contemplating a transition away from C, at this stage.

--
Andrew

On 06/11/17 20:15, Lars Baumgaertner wrote:

Okay thanks for the clarification. The single threaded design makes it complicated. I forgot that we already learned that the hard way in our big evaluation in 2016 :)

Pulling all the small bits and pieces out of the serval internals without a clear internal API is too error prone for me and as Andrew is already on the task of implementing the routing API, we will just wait a bit longer and hope for a release "really soon[tm]". Even though I only found the documentation in route-rest-api not an implementation.

At the moment Andrew seem to be the one working on all the stuff we want (iOS library, route-rest-api etc) :D

King regards,

 Lars

On Monday, November 6, 2017 at 12:33:31 AM UTC+1, Jeremy Lakeman wrote:

Your code is running within the server's single thread, so you blocked it from processing the incoming message.

Since your code is running within the server, you have access to all static state. So you could just copy / call / refactor the existing processing code;

https://github.com/servalproject/serval-dna/blob/development/overlay_mdp.c#L1788

The API the server uses internally for MDP is a little different. If you want to send and receive packets between network peers, you need to write event driven code like this;
https://github.com/servalproject/serval-dna/blob/development/overlay_mdp_echo.c

While waiting for network traffic, you can pause your HTTP processing and specify a failure timeout, eg;

https://github.com/servalproject/serval-dna/blob/development/rhizome_restful.c#L335

And resume it again later;

https://github.com/servalproject/serval-dna/blob/development/rhizome_restful.c#L288

If you are planning on building some kind of restful routing table functions, Andrew has started writing down the design we've been discussing internally in the route-rest-api branch.

-- 
Andrew Bettison
Senior Software Engineer
and...@servalproject.org | Mobile +61 407185656 | Skype ajbettison