new routing protocol

2,012 views
Skip to first unread message

Lily

unread,
Jun 8, 2015, 4:40:27 PM6/8/15
to ns-3-...@googlegroups.com
Hi all,

I read some old threads about creating new routing protocol. But I am still confused.  My question is if I want to route the traffic based on the link loads, choose a widest path ( with maximum available link bandwidth) for each flow.  Should I focus on the Global routing or OLSR?  Which functions need to be modified?  And I need to monitor the link load , should I modify the channel class accordingly to record the packets sent or received then compute the traffic rate?

Thanks a lot!


Tommaso Pecorella

unread,
Jun 9, 2015, 4:12:31 PM6/9/15
to ns-3-...@googlegroups.com
Hi,

I've read some recipes in different cook books about how to prepare a cheesecake. However, I'm still a bit confused about one bit: how much the turkey have to cook on the grill ?
Does the above make sense to you ?
No ?
Good, because your question is more or less the same.

You want to develop a new routing protocol, with a given metric (residual bandwidth ?). You ask which function you need to change in OLSR or GlobalRouting... NONE. It's a new routing protocol, you can't modify an existing one.
Feel free to study them and to use them as a guideline, but that's all you can do with them.

T.

Wenrui Ma

unread,
Jun 9, 2015, 4:28:19 PM6/9/15
to ns-3-...@googlegroups.com
Thanks for your reply. Yes, I want to develop a new routing protocol with a given residual bandwidth.  I think I should spend more time to try understand one of the existed routing protocol completely.


Regards,
Wenrui Ma

--
You received this message because you are subscribed to a topic in the Google Groups "ns-3-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ns-3-users/kLRHv4D9CEU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ns-3-users+...@googlegroups.com.
To post to this group, send email to ns-3-...@googlegroups.com.
Visit this group at http://groups.google.com/group/ns-3-users.
For more options, visit https://groups.google.com/d/optout.

Jack Higgins

unread,
Jun 11, 2015, 3:40:28 AM6/11/15
to ns-3-...@googlegroups.com

Dear Tommaso ,

I perfectly get your point, there is no "one perfect recipe" for making a routing protocol. However , there are guidelines and general patterns even in the most radical routing protocols. As someone facing the same problem, some recommendations and suggestions of someone as experienced as you would be helpful to come up with these patterns.Suggestions, recommendations would be much appreciated.... something beyond the "just read the cooking book" and you can figure it out. The answer is not that obvious to us as it seems to you, is not that we do not read the tutorials or manuals, is that sometimes we do not get them 100%.... we are learning after all.... there are many things we do not know.

Here are some very "general ideas",  I have come up with in my journey as a Noob, trying to implement a routing protocol myself. Please add anything else or correct anything that seems wrong:


1- Start by defining your packet headers, although these can be defined in many ways depending on the complexity you want , its is a good idea to use the "header" class in NS3. Why packet headers first?, most protocols need them if not all and packet headers are simple enough to understand to make them a starting point. Implement the serialize and deserialize methods according to the requirements of your header (byte size of your  header info: hop count, seq number, address, zone id , etc etc etc)

2- Next would be to implement the "main routing protocol class". This class will connect all your classes in one place  and "IS A" ns3::Ipv4RoutingProtocol class. And therefore, it MUST implement the methods RouteOutput and RouteInput. As explained by Konstantinos in his page (http://personal.ee.surrey.ac.uk/Personal/K.Katsaros/clwpr.html) , "The first will provide to the transport protocol a valid route towards the destination for a specific packet. On the other hand, the latter is called by Ipv4L3Protocol when a packet is received from a NetDevice and appropriate action is taken to forward it either to upper layers (local delivery) or to other nodes (forwarding)." , in other words the sending and receiving packets points in your protocol. It s in this point that you must define the necessary "Logic" to call other , classes ,methods etc of which packets, when and how must be sent and received.

3-At this point you have the entry and exit points for your packets but you certainly will need a routing table. Define a new class to describe your routing table and the necessary methods for management (purge ,clean, insert,Lookup, etc). Some people implement the routing table in the "main routing protocol class" . If the protocol is small it should be ok, but most of the time it will make things more messy and harder to understand for other people. If a necessary example is required check the definition of the AODV tables (aodv-rtable.cc, aodv-rtable.h)


4- Define extra classes necessary for your algorithm. This might or might not be required depending of the complexity of your protocol algorithm. You can define all the necessary methods in the main class, but some "specialized classes" defined by yourself might be required for your algorithm. For example, AODV manage queues for packets while constructing routes. These queues are managed by a class in aodv-rqueue.cc file.


5- Its impossible to define all the behaviour of your routing protocol going class by class. As you progress you will need to go back and add code to all your files as required. Just start with the base of each one of them. ( adding more control methods, headers,routing  table info, etc as you progress ). I believe that is just important to keep one class per file  with a clear definition of the role they take in the protocol.


And yes.... as tommaso pointed out, you must check other protocols code to grasp the idea of how to implement a "new protocol".....

I am still understanding the role RouteOutput and RoleInput , specially when it comes to proactive protocols, how do I make the transmission of data wait until the routes are defined.... etc ..

Anyone else please share some experiences,suggestions, comments , or deeper understanding of implementing a routing protocols



Noob Jack.

Tommaso Pecorella

unread,
Jun 11, 2015, 4:46:09 AM6/11/15
to ns-3-...@googlegroups.com
Hi Jack,

I don't have anything to add, you touched all the relevant points. I wrote a post on my blog (I should update it sometimes) - http://tommy-ns-3-corner.blogspot.it/2014/03/how-to-develop-new-protocol.html - and what you wrote is very similar to what i wrote.

My reply to Lily, indeed, could have been a bit nicer. Still, the root point is the same: if one needs to develop a new protocol, he/she shouldn't modify an existing one, unless the new protocol is a slight change over an existing one, of course.

Cheers,

T.


PS: for proactive routing protocols, I'd say that RouteInput should simply trash the packets with unknown route. Otherwise it would be a reactive protocol.
Proactive protocols should have the routing tables already built. If the RT is incomplete (it may happen during the routing protocol startup or reconfiguration), packets are dropped.
Of course one can hold packets for some time in order to provide a temporary buffer for short-time routing reconfiguration, but that's not really an usual behaviour.
If you want to experiment about it, you could try to use the same technique used by AODV: if a route is unknown, the packet is sent to the loopback interface. The RouteInput can trap it and handle the local queue.

Patrick D

unread,
Jun 11, 2015, 9:28:29 AM6/11/15
to ns-3-...@googlegroups.com
I don't think programmers can help their level of sarcasm. Turkey cheesecake does seem nice though! 
I suggest you pin How to develop new protocol at the top of this mailing list because 99%(not official stat) of questions relates to this topic in some way.
I think it was a misunderstood syntax (modifying vs new).
Cheers!


Wenrui Ma

unread,
Jun 11, 2015, 10:21:28 AM6/11/15
to ns-3-...@googlegroups.com
Dear Jack,

Thanks a million. At the beginning, I am totally lost. I don't know where to start. I added fields in Packet and Node class. Then for the routing protocol,  I thought maybe the easiest way is to modify the existed routing protocol, replace some functions with my functions. I know I am supposed to extend some base classes, write my own functions, but I don't know how to do that.

Now I think I am on the right track under your guidance. Your answer is the most clear so far and will be helpful for other guys who have the same problem. Thanks again!!

Best,
Lily

Jack Higgins

unread,
Jun 11, 2015, 11:54:58 PM6/11/15
to ns-3-...@googlegroups.com

Dear Tommasso,

Thank you for your reply. I checked your blog , and it make some interesting points that I have not considered as well as some I had to learn the hard way.
As you pointed out in the post, creating a new protocol is a common question.... I suggest to make the info in your blog post a permanent link or a topic for the tutorials, I am pretty sure it would come handy to someone in the future.


Also, thank you for the tips about proactive protocol I have considered your suggestions myself (drop packets, loopback and queu) . However I still have a question about the setting of the routes for the proactive protocol in my scenario, but before asking the question I would try to analyse the OSLR code a little bit more.

Thank you for your time!

Tommaso Pecorella

unread,
Jun 12, 2015, 1:22:49 AM6/12/15
to ns-3-...@googlegroups.com
Hi,

Pinning is an option, but I guess that we'll pin a wider post with how-to links. For some strange reason they are not found or used.

@Jack: feel free to post your doubts, I think the discussion will be interesting.

Cheers,
T.

JaNa

unread,
Jun 13, 2015, 11:29:57 PM6/13/15
to ns-3-...@googlegroups.com
Hi,

@T
Thank you very much for those suggestions, tricks and tips you gave on your blog. 
However, "For some strange reason they are not found or used", is true. I never found those while I was searching. 
While I was implementing one of my protocol, I had to learn and understand things in hard way. BTW, to be honest, I am truly happy that I learned those in hard way.

@all,

This is a interesting thread. please keep posting it. This will be a great opportunity to share knowledge.

Cheers,
 

T.

--
You received this message because you are subscribed to the Google Groups "ns-3-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ns-3-users+...@googlegroups.com.

To post to this group, send email to ns-3-...@googlegroups.com.
Visit this group at http://groups.google.com/group/ns-3-users.
For more options, visit https://groups.google.com/d/optout.



--
Best Regards..

    """""""""""
:)   JaNa     :) 
    """""""""""

Xun Xiao

unread,
Jun 17, 2015, 4:35:29 PM6/17/15
to ns-3-...@googlegroups.com
I was also inspired by your post. I am also working on developing a new routing protocol now. My additional questions go to the following:

Sometime, the routing protocol also needs to proactively send some packets for building routing tables, but in ns3::Ipv4RoutingProtocol, I think RouteInput and RouteOutput are not enough. I think some socket stuffs are also needed to support the routing protocol to send/recv. For the upper layers above the routing protocol, I can imagine that Applications/users will create sockets to do network I/O, but how could we do for the Routing Protocol itself? 

Another point is that many times, I saw there are lots of NotifyInterfaceUp/Down functions are implemented while in the routing overview, it didn't mention anything about this. I was wondering if the Notify....Up/Down are necessary? 


Thanks.


On Thursday, June 11, 2015 at 9:40:28 AM UTC+2, Jack Higgins wrote:

Tommaso Pecorella

unread,
Jun 17, 2015, 5:30:40 PM6/17/15
to ns-3-...@googlegroups.com
Hi,

I'd suggest to check src/internet/model/ripng*. The fact that I wrote it is just a coincidence.

The first question (if RouteInput and RouteOutput are enough), the answer is no.
These two functions are "just" the one used by IP to find a route (i.e., forwarding). The routing table itself is updated by other means.
If the protocol is reactive, the route search will be triggered by RouteInput or RouteOutput. If the protocol is proactive, the RT will be updated by timers (usually).

One way to think to the whole system is this: You can think to the routing protocol implementation as 3 separate entities: one is the RT, one is the routing application and the third is the RouteInput/Output.
In proactive routing, there is an app that continuously do its job and keeps the RT updated.
In reactive routing the RouteInput/Output will trigger some function in the above mentioned app to fill the RT.
The RT is updated by the app, and used by RouteInput/Output.

This is an indirect answer to your question: sockets are used by the app. There's always an app implementing the protocol... and that app can open and manage its sockets, just like any normal application. Some protocols will need UDP sockets, some TCP sockets, some raw IP sockets. it depends on the particular protocol.

The second question (interface and addresses up and down): a good implementation should react to these and do something. E.g., invalidate the routing table entries for the interface that is down, starting/stopping timers, etc.
Some protocols don't react to these changes, and they should. Feel free to propose a patch if you want, it will be most welcome.
Note that a protocol should be robust against link disconnections (e.g., it shouldn't rely only on NotifyInterfaceUp/Down), but it' a good idea to be aware of them.
E.g., if I have a node with 2 interfaces, it's a good idea that if one fails I clean stuff: it's no use to keep trying to send packets on a dead interface.
At the same time, it's a good idea to start using an interface if it becomes available, even if it goes up after the protocol has been started.

Hope this clarifies a bit,

T.

Xun Xiao

unread,
Jun 17, 2015, 5:44:20 PM6/17/15
to ns-3-...@googlegroups.com


On Wednesday, June 17, 2015 at 11:30:40 PM UTC+2, Tommaso Pecorella wrote:
Hi,

I'd suggest to check src/internet/model/ripng*. The fact that I wrote it is just a coincidence.

The first question (if RouteInput and RouteOutput are enough), the answer is no.
These two functions are "just" the one used by IP to find a route (i.e., forwarding). The routing table itself is updated by other means.
If the protocol is reactive, the route search will be triggered by RouteInput or RouteOutput. If the protocol is proactive, the RT will be updated by timers (usually).

One way to think to the whole system is this: You can think to the routing protocol implementation as 3 separate entities: one is the RT, one is the routing application and the third is the RouteInput/Output.
In proactive routing, there is an app that continuously do its job and keeps the RT updated.
In reactive routing the RouteInput/Output will trigger some function in the above mentioned app to fill the RT.
The RT is updated by the app, and used by RouteInput/Output.

This is an indirect answer to your question: sockets are used by the app. There's always an app implementing the protocol... and that app can open and manage its sockets, just like any normal application. Some protocols will need UDP sockets, some TCP sockets, some raw IP sockets. it depends on the particular protocol.

This makes me very clear about the mechanism of the routing protocol, thanks a lot for your explanations. 

Actually, if there will be a small tutorial on the RigNg routing protocol corresponding to explain the modules and idea of design as you said above (like why we have these functions here, and what are their roles ...), that will be great, because now by reading the source code, guessing what are they doing is the major tasks we are digging. :) But anyway, thanks again.

Tommaso Pecorella

unread,
Jun 17, 2015, 6:11:10 PM6/17/15
to ns-3-...@googlegroups.com
On Wednesday, June 17, 2015 at 11:44:20 PM UTC+2, Xun Xiao wrote:

On Wednesday, June 17, 2015 at 11:30:40 PM UTC+2, Tommaso Pecorella wrote:
Hi,

I'd suggest to check src/internet/model/ripng*. The fact that I wrote it is just a coincidence.

The first question (if RouteInput and RouteOutput are enough), the answer is no.
These two functions are "just" the one used by IP to find a route (i.e., forwarding). The routing table itself is updated by other means.
If the protocol is reactive, the route search will be triggered by RouteInput or RouteOutput. If the protocol is proactive, the RT will be updated by timers (usually).

One way to think to the whole system is this: You can think to the routing protocol implementation as 3 separate entities: one is the RT, one is the routing application and the third is the RouteInput/Output.
In proactive routing, there is an app that continuously do its job and keeps the RT updated.
In reactive routing the RouteInput/Output will trigger some function in the above mentioned app to fill the RT.
The RT is updated by the app, and used by RouteInput/Output.

This is an indirect answer to your question: sockets are used by the app. There's always an app implementing the protocol... and that app can open and manage its sockets, just like any normal application. Some protocols will need UDP sockets, some TCP sockets, some raw IP sockets. it depends on the particular protocol.

This makes me very clear about the mechanism of the routing protocol, thanks a lot for your explanations.

You're welcome
 
 Actually, if there will be a small tutorial on the RigNg routing protocol corresponding to explain the modules and idea of design as you said above (like why we have these functions here, and what are their roles ...), that will be great, because now by reading the source code, guessing what are they doing is the major tasks we are digging. :) But anyway, thanks again.


I never thought to use it as an example, but it could be a good idea. In the meantime (who knows when I'll have time) you can study the RipNg RFC. All the functions are modeled after the RFC.

Cheers,

T.

Jack Higgins

unread,
Jun 18, 2015, 1:10:57 AM6/18/15
to ns-3-...@googlegroups.com

Dear Xun Xiau,

 NotifyUp down functions should be implemented? Of course they should be implemented, they are part of the Ipv4RoutingProtocol class you are implementing and they are there for a reason. However, as Tommaso mentioned, they are meant to be your protocol more robust and as far as the question goes..... where should I start for a new protocol? I would say implementing first interface up /down is not in my priority list until I get things moving, so in a way I am saying RoutingInput / RoutingOutput are more important and should be constructed first.

In the post I used AODV , a reactive protocol to make some examples of where to start. Ironically enough , I am also trying to implement a proactive protocol (so routing wont be triggered in RouteInput and RouteOutput).

Tommasso make some interesting points about where to start depending on your protocol type (Reactive/Proactive), I have not checked the code he suggested  yet( src/internet/model/ripng*) but his comment about how you will need to use timers  (the  routing app continusly doing his job) intrigues me.

For now my conclusion is: A proactive protocol have to start in some function, so my guess ( please correct me if I am wrong) , is to use the function DoInitialize() to  trigger the construction of the routing. (DoInitialize come from Object and as the name implies I believe it runs when the protocol class is created)

After filling the routing tables, I believe its only matter of using timers inside other functions or routeInput/ routeOutput to update the Routing Tables.

Tommaso Pecorella

unread,
Jun 18, 2015, 2:50:30 AM6/18/15
to ns-3-...@googlegroups.com
Hi both,

all is right Jack. The RipNg::DoInitialize does exactly that, and in particular:

- Start a timer for periodic dissemination of updates (unsolicited updates, RipNg has those).
- Open the sockets on all the available interfaces. While doing so, check if there are new networks to announce.
- If it founded new networks to announce, start a timer to send an announcement (they're called triggered update by RipNg).
- Add a timer to ask the neighbor routers for an update.

There are some random variables in the above process to avoid network synchronization (it's all defined in the standard).
As a reference, the periodic update unction is SendUnsolicitedRouteUpdate. Kinda small function, but it's the one.

Cheers,

T.

Jack Higgins

unread,
Jun 18, 2015, 4:21:35 AM6/18/15
to ns-3-...@googlegroups.com
Thank you for the quick response and the comments, They do give some inspiration and Ideas,  I will comment further, after digging some in the code of RipNg::DoInitialize and  trying to implement my own version of DoInitialize required for my protocol scenario.

Xun Xiao

unread,
Jun 18, 2015, 4:26:06 AM6/18/15
to ns-3-...@googlegroups.com
Continue the question, I saw that RouteInput() passes four callbacks: ucb, mcb, lcb, ecb for handling the incoming packets  (either forwarding locally or finding a route to send it out), but I didn't find where are those callbacks' implementations. Are they provided by ns3 itself?

Another questions is about the initialization of RipNg, I found sendSockets are all bound to NetDevices i.e., BindToNetDevice, but m_recvSocket was not, it only bind to Address and set its RecvCallback, I was wondering why? Thanks.

Tommaso Pecorella

unread,
Jun 18, 2015, 5:18:12 AM6/18/15
to ns-3-...@googlegroups.com
Hi,

first question: the callbacks are passed by Ipv4L3Protocol (or the IPv6 equivalent), check Ipv4L3Protocol::Receive and Ipv6L3Protocol::Receive.

Sockets... to make the long story short, if you have many interface you need this trick.
When you receive a packet, you can ask the socket the interface it has been received from. As a consequence, one socket is enough.
When you need to send a packet, things are a bit more complex: you need to have a socket bound to an interface to be sure that the packet is sent on that interface and not on another one.
Moreover, if you have to use Link-Local addresses (like in IPv6), routing goes nuts, as ALL the interfaces have a valid link-local network, even if they are different networks (dang).
As a consequence, to send packets to a Link-Local address it's always a good idea to use a socket bound to an interface. Otherwise the packet will be sent to the first available interface, which isn't exactly the right thing to do.

In IPv4 this problem is less dramatic, as there aren't Link-Local addresses and having two interfaces on the same subnet is very unlikely.
Still, if you need to send packets to a broadcast address (255.255.255.255), you'll have the same problem, and you need the same trick.

Cheers,

T.

Xun Xiao

unread,
Jun 18, 2015, 5:38:51 AM6/18/15
to ns-3-...@googlegroups.com
Thanks a lot, really understood so many, which is difficult to learn from the tutorial and manual docs. :)

Xun Xiao

unread,
Jun 19, 2015, 8:34:16 AM6/19/15
to ns-3-...@googlegroups.com


On Wednesday, June 17, 2015 at 11:30:40 PM UTC+2, Tommaso Pecorella wrote:
Hi,

I'd suggest to check src/internet/model/ripng*. The fact that I wrote it is just a coincidence.

The first question (if RouteInput and RouteOutput are enough), the answer is no.
These two functions are "just" the one used by IP to find a route (i.e., forwarding). The routing table itself is updated by other means.
If the protocol is reactive, the route search will be triggered by RouteInput or RouteOutput. If the protocol is proactive, the RT will be updated by timers (usually).

One way to think to the whole system is this: You can think to the routing protocol implementation as 3 separate entities: one is the RT, one is the routing application and the third is the RouteInput/Output.
In proactive routing, there is an app that continuously do its job and keeps the RT updated.
In reactive routing the RouteInput/Output will trigger some function in the above mentioned app to fill the RT.
The RT is updated by the app, and used by RouteInput/Output.

This is an indirect answer to your question: sockets are used by the app. There's always an app implementing the protocol... and that app can open and manage its sockets, just like any normal application. Some protocols will need UDP sockets, some TCP sockets, some raw IP sockets. it depends on the particular protocol.

Hi Tommaso, I have some further questions to ask.

If an App will be dedicated to keep the RT updated by send/recv info among Hosts/Nodes, then who is responsible for routing the packets from the App? Will they also invoke RouteOutput/RouteInput in the routing protocol? Or the App will use another internal mechanism (independent) to communicate over the network? Because at some point, we also need to call socket->send(pkt), and socket-recv() ... 

This seems a chicken-egg situation. maybe I am wrong. Thanks.

Tommaso Pecorella

unread,
Jun 19, 2015, 8:41:15 AM6/19/15
to ns-3-...@googlegroups.com
Hi,

another good question, indeed.

The answer is rather simple: nobody does. Routing protocol packets are (usually) sent as broadcast / multicast messages, and the RouteOutput will let them pass through (remember that the send socket is bound to an interface).
For the unicast packets, usually they are sent as reply to another message. Thus, a temporary route can be established.
There are some minor differences among protocols, but most of the time it happens like that. Of course there's always the exception, but one can figure it ut on a per-protocol basis. E.g., IBGP needs another IGP protocol already working (thus, it sends packets relying on an already-establiehd RT), etc.

Hope this helps,

T.

Xun Xiao

unread,
Jun 19, 2015, 9:30:53 AM6/19/15
to ns-3-...@googlegroups.com
So that means my own routing messages will also pass through RouteOutput and RouteInput, am I right? Therefore, the RT at the very initial stage at the Node should be able to provide routing info to our own routing messages, e.g., via broadcast fashion. Thanks.

Jack Higgins

unread,
Jun 25, 2015, 3:18:53 AM6/25/15
to ns-3-...@googlegroups.com
Dear Tommaso,


Maybe is kind of a dumb question but I am still struggling implementing my proactive protocol.

I ran small test based on the code in ripng.cc and oslr-routing.cc. (specifically from the doIntialize function).
My objective was to broadcast a single packet . The nodes that received it simply answer "packet received".(NS_LOG_DEBUG message)

As described in previous posts,  I did the necessary bindings with the send and receive Sockets in the doInitialize function. Soon after that I send a packet through the send socket , but  no node seemed to receive the broadcast.

Im not sure if RouteInput / RouteOutput  should be implemented at this point since I'm just  initializing by setting all the sockets and broadcasting a message. (Not handling packets sent by the application yet).
Maybe there are things I have not understand quite yet. I attached the code of the routing function .... but I wonder am I doing a correct socket intialization and sending a packet correctly?

Please , any guidance would be much appreciated.






On Thursday, June 18, 2015 at 3:50:30 PM UTC+9, Tommaso Pecorella wrote:
masp-routing-protocol.cc

Jack Higgins

unread,
Jun 25, 2015, 4:24:51 AM6/25/15
to ns-3-...@googlegroups.com


Sorry, it seems that I found the answer!

I forgot to set    socket->SetAllowBroadcast (true); in the  sending socket. Now I am getting the desired results. One little thing can cause such frustration... hehe lesson learned though.


Now Im just curious about how is the receiving socket working. Just like in ripng.cc (doInitialize function) im using InetSocketAddress local = InetSocketAddress (Ipv4Address::GetAny (), MASP_PORT_NUMBER); to get the available  (interface, address) pair. However, in my example as far as I know theres only 2 pairs in my Ipv4 object :

- Interface 0 , 127.0.0.1
- Interface 1, mainAddress


(Interface1,mainAddress) pair is bind to the sending Socket so this pair can only be used for sending. So which  pair (interface,address) am I using for the receiving socket ?  Maybe I am failing to understand some basic Socket knowledge..... 
 

Tommaso Pecorella

unread,
Jun 25, 2015, 5:01:38 PM6/25/15
to ns-3-...@googlegroups.com
Hi,

I'm glad you found the answer :)

About the other curiosity, the answer is: it's needed for multiple interfaces cases, i.e., when you have more than 2 (one of them is the Loopback).

Let's ignore the Loopback (127.0.0.1 or ::1 for a moment).
If you have just one NetDevice, then you could do everything with one socket: Rx and Tx... if you're on IPv4.
If you use IPv6 things are a bit more complex as you need to receive from multicast addresses and send from unicast, which is a bit more complex because you don't want to use "Any". IPv6 quirks.
Anyway, the generic reason is to think to what happens when you have many interfaces. Probably you could use one socket for each interface I guess, but it wasn't that easy for v6. Don't ask me why, because I don't remember. There's always another way to do it, and that one is just the one that I felt more logical. Mileage may vary.

Cheers,

T.

Jack Higgins

unread,
Jun 26, 2015, 6:46:34 AM6/26/15
to ns-3-...@googlegroups.com

Thanks for the reply!


I understand now what you mean with "that setting trick for Ipv6". Also now I see how for example, the NS3 OSLR code is using the same socket for Tx and Rx since is implemented on IpV4. 

Xun Xiao

unread,
Jul 8, 2015, 6:38:10 AM7/8/15
to ns-3-...@googlegroups.com
Hi Tommaso,

I am a little bit confused between the RouteInput() and Receive() method provided by ourselves, for example, in a routing App. 

I can understand that Receive() actually distributes different types of routing messages as in RipNg, AODV and so on, but what is RouteInput() for? According to the introduction in the tutorial, it handles all incoming packets with Ipv4header, so does it means that the routing messages will also first go through RouteInput() and then be received by the Receive()? 

My understanding is that RouteInput() is at lower level, while Receive() of ourselves stands higher. Am I correct? Thanks.

Best,

Xun Xiao

Tommaso Pecorella

unread,
Jul 8, 2015, 7:12:13 AM7/8/15
to ns-3-...@googlegroups.com
Hi,

you're totally correct. Any packet will go though RouteInput first. If it's for the local machine, then it will reach the Receive function of the appropriate socket (even the ones opened by the routing protocol itself).

Cheers,

T.

Xun Xiao

unread,
Jul 14, 2015, 10:43:50 AM7/14/15
to ns-3-...@googlegroups.com
Hi Tommaso,

Thanks for your confirmation. My further question goes to the DoInitialize (), Start() and SetIpv4 ().

I saw several routing protocols like RipNg, AODV and so on. They all initialize lots of things in different ways in the above three functions. I was wondering what is the basic way to do the initialization, and what is the order of the above three startup functions?

My understanding is: SetIpv4 () will run first, then what we should do basically at here, then what else we need to do in the rest two? Thanks a lot.

Generally, my question can be simply put as how to do Initialization of my new routing protocol .. Thanks again.

best,

Xun Xiao

Tommaso Pecorella

unread,
Jul 14, 2015, 2:47:26 PM7/14/15
to ns-3-...@googlegroups.com
Hi,

well, it's a bit strange to say, but there's no fixed rule.

In order to check who's being called first, try enabling the protocol logging, you should "see" all the functions being called. What to do in each function, on the opposite, is a matter or taste and use-cases.

Start is to avoid (it will be used soon™ for other purposes and have to be considered reserved).
Initialize is ok, but you could miss some interfaces (they can be added later).
SetIpv4/6 is "just" to set the IP layer. Could be used to initialize stuff, but I'd rather avoid that. Instead, one should use NotifyAddAddress.

Summarizing: there's no fixed rule. It mostly depends if you're pedantic in supporting interfaces and address add/removal, multiple interfaces and so on.

Hope this helps,

T.

jaasilva

unread,
Oct 20, 2015, 4:22:16 PM10/20/15
to ns-3-users
Hi everyone,

Thanks for all the great advice.

I'm implementing my own routing protocol (based in GPSR), and I found something strange.
With the help of some logging, I found that in my protocol NotifyAddAddress is called before NotifyInterfaceUp.
Is that normal? I would expect that an interface was first added, and only then an address would be added to that address.

Thanks in advance.

Konstantinos

unread,
Oct 20, 2015, 4:33:34 PM10/20/15
to ns-3-users
Not necessarily. You may have an interface "down" but that interface has an address. 
So I guess your NotifyAddAddress is called when you create the interface and assign an address to that, and when that interface becomes active you get the NotifyInterfaceUp.

Regards,
K. 

jaasilva

unread,
Oct 20, 2015, 5:30:56 PM10/20/15
to ns-3-users
First of all, thanks for the quick response.

Ok. Now, I understand that it is possible for NotifyAddAddress to be clled before NotifyInterfaceUp.

But now I have another doubt:
I'm using AODV as a base for the development of my routing protocol.
When NotifyAddAddress or NotifyInterfaceUp are called, a socket is bound to the interface hearing for packets.
Specifically, I bind the sockets to the interface broadcast address (e.g., 10.0.255.255 with mask 255.255.0.0).
When I send HELLO messages through those sockets, packets skip RouteOutput function.
But on the other end, when a node receives those packets, they are passed to the RouteInput function.
Thus, the socket callback for receiving packets is never called.
Can anyone understand why?

Thanks,
JS

Tommaso Pecorella

unread,
Oct 20, 2015, 6:05:44 PM10/20/15
to ns-3-users
Answers in-line


On Tuesday, October 20, 2015 at 11:30:56 PM UTC+2, jaasilva wrote:
First of all, thanks for the quick response.

Ok. Now, I understand that it is possible for NotifyAddAddress to be clled before NotifyInterfaceUp.

But now I have another doubt:
I'm using AODV as a base for the development of my routing protocol.
When NotifyAddAddress or NotifyInterfaceUp are called, a socket is bound to the interface hearing for packets.
Specifically, I bind the sockets to the interface broadcast address (e.g., 10.0.255.255 with mask 255.255.0.0).
When I send HELLO messages through those sockets, packets skip RouteOutput function.

RouteOutput is skipped only because you're sending packets to a broadcast address...

But on the other end, when a node receives those packets, they are passed to the RouteInput function.

That's perfectly normal.
 
Thus, the socket callback for receiving packets is never called.

Not a consequence. If packets aren't delivered to the intended callback, it's not an issue of RouteInput.
RouteInput should deliver the packets to the "lcb" function, i.e., Ipv4L3Protocol::LocalDeliver. Check if they're processed or dropped there.
 
Can anyone understand why?

Only a debugger can...

Have fun debugging,

T.

jaasilva

unread,
Oct 20, 2015, 6:15:42 PM10/20/15
to ns-3-users
Hi Tommaso,

Thanks for the clarification.
In my RouteInput those packets are delivered to the lcb callback, i.e., they are processed by RouteInput.
But, still, the receive callback for the socket is not called, e.g., socket->SetRecvCallback (MakeCallback (&RoutingProtocol::RecvGPSR, this));.
It should be called, right?

Thanks,
JS

Tommaso Pecorella

unread,
Oct 21, 2015, 2:41:45 AM10/21/15
to ns-3-users
Hi,

it depends. If there's another socket with a higher precedence, it won't be called. By higher precedence I mean a socket opened with the same (or similar) receiving properties: receive IP and port (mostly).
As an example, if you open two sockets on address X and port y, only one will receive the packet. The exact rules can be found in Ipv4EndPointDemux::Lookup.

Hope this helps,

T.

jaasilva

unread,
Oct 21, 2015, 5:22:59 AM10/21/15
to ns-3-users
Thanks. Now I understand.

Just another question:
If I bind my sockets to the interface broadcast address (e.g., 10.0.255.255 with mask 255.255.0.0), when I send packets through these sockets, those packets will skip RouteOutput.
But if I bind the sockets to Ipv4Address::GetAny(), when I send packets through these sockets, they will go through RouteOutput.
Why does this happen?

Since I'm talking about sending HELLO (beacon) messages, I use these sockets for that. And as I would expect, in RouteOutput I will have no routing table to decide anything.
How do I route these broadcast HELLO messages in RouteOuput?

Best regards,
JS

Tommaso Pecorella

unread,
Oct 21, 2015, 4:20:50 PM10/21/15
to ns-3-users
Answers in-line


On Wednesday, October 21, 2015 at 11:22:59 AM UTC+2, jaasilva wrote:
Thanks. Now I understand.

Just another question:
If I bind my sockets to the interface broadcast address (e.g., 10.0.255.255 with mask 255.255.0.0), when I send packets through these sockets, those packets will skip RouteOutput.
But if I bind the sockets to Ipv4Address::GetAny(), when I send packets through these sockets, they will go through RouteOutput.
Why does this happen?

Because "Any" is "0.0.0.0", while Broadcast is "255.255.255.255". You shouldn't send packets to "Any", that address is to receive packets from anyone :P

Since I'm talking about sending HELLO (beacon) messages, I use these sockets for that. And as I would expect, in RouteOutput I will have no routing table to decide anything.
How do I route these broadcast HELLO messages in RouteOuput?

Beacons shouldn't be routed, as they should be simply sent to one-hop neighbors :)
Usually this is done by sending them to the broadcast address with a TTL set to one, just to be sure.

Cheers,

T.

jaasilva

unread,
Oct 28, 2015, 5:42:46 AM10/28/15
to ns-3-users
Hi,

Related to this, I have a very strange problem and I can't get my head around it:
I created a specific header for my protocol (inherits from Header ns3 class) which contains a byte (uint8_t) describing the type of message.
When the header is serialized (start.WriteU8(m_type);) the message type is '1'. I check this with some logging.
But when the header is deserialized on another node (m_type = i.ReadU8 ();) it does not correspond to what was serialized on the first node.
For example, m_type is deserialized to '192' or '3'.
Does anyone has any idea why this is happening?

Best regards,
JS

Konstantinos

unread,
Oct 28, 2015, 5:50:57 AM10/28/15
to ns-3-users
The most possible cause is that you have tried to remove another header before your own.
The order of adding/removing headers should be the same.

jaasilva

unread,
Oct 28, 2015, 6:22:32 AM10/28/15
to ns-3-users
Hi Konstantinos,

If I understood well from the documentation, the addHeader method prepends a header to the packet.
If so, the remove order of the headers should be the reverse of the add, right?

Konstantinos

unread,
Oct 28, 2015, 7:27:12 AM10/28/15
to ns-3-users
Yes, that's what I meant. 

If you add headers  [ A ] [ B ] [ C ] , you need to remove them as [ C ] [ B ] [ A ]. 
So when you try to read your own, you have be sure where you add it and if other headers are read before it.

Jose Navarro Alabarta

unread,
Nov 5, 2015, 9:36:33 AM11/5/15
to ns-3-users
Hi all,

I have implemented a vehicular routing protocol based on geographical location, this implementation is for my master's thesis. This protocol uses a neighbor table instead of a routing table. This table is updated when a node receives a HELLO and ACK packet.

The other type of packets is the data packet, that it is forwarded to the destination in a hop by hop manner when the neighbour table of each node (hop) is not empty, in other words when the node has neighbors. But these packets are not reaching the destination at this moment. Does this problem could be because I am not obtaining correctly the destination node when I define a scennario? In the scennario that I define a source and receive sockets are created. Existing a callback to receive packets and a schedule to transmit packets.

In the implementation of RoutingProtocol, how can I obtain the receiving node?, thus I think it could implement the forwarding function.

I attached the source code of the routing protocol implementation.

Somebody can me give a hand?

Thank you very much in advance.
Best regards,
Jose

srr-routing-protocol.cc

Konstantinos

unread,
Nov 5, 2015, 9:48:37 AM11/5/15
to ns-3-users
Hi Jose,

The reasons why you do not receive packets may be several for bugs/errors in your protocol to disconnected network (i.e. there is not a valid path from source to destination).
The best way to solve this problem is simple. Use the debugger, use NS_LOG in a simple scenario and move incrementally. For example start with a simple 2-hop static scenario to validate that there is connection. Then start increasing complexity with more dynamic features.

Regards,
K.

Jose Navarro Alabarta

unread,
Nov 5, 2015, 10:32:31 AM11/5/15
to ns-3-...@googlegroups.com
Hi Konstantinos,

thank you for your quick response. I will follow your instructions.

Regards,
Jose

--
You received this message because you are subscribed to a topic in the Google Groups "ns-3-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ns-3-users/kLRHv4D9CEU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ns-3-users+...@googlegroups.com.
To post to this group, send email to ns-3-...@googlegroups.com.
Visit this group at http://groups.google.com/group/ns-3-users.
For more options, visit https://groups.google.com/d/optout.

jaasilva

unread,
Nov 6, 2015, 4:04:27 AM11/6/15
to ns-3-users
Hi all,

During the development of my routing protocol I discovered an interesting thing:
Broadcasts are sent using the slowest rate while unicasts are usually sent using higher rates, i.e., broadcast messages can travel farther.
This translates to the following:
In my protocol, I use broadcasts to disseminate HELLO messages (to build the neighbor/routing tables).
Thus, there can be the case that node A receives a HELLO message (a broadcast message) from node B (adding node B as a possible path to forward packets) and when node A uses node B to forward a packet, since it uses unicast for that message it will never reach node B.
This means that I am building the neighbor/routing tables using broadcast messages, but when I forward packets using unicast, those packets may never arrive their destination.

AODV also builds routing tables using broadcast messages, but I still don't understand how they solve this problem.
Does anyone understand how this can be resolved?

Regards,
JS

Tommaso Pecorella

unread,
Nov 6, 2015, 8:02:14 AM11/6/15
to ns-3-users
?!?!?!?!?!

AODV is simply smart enough. Your data packets are sent using adaptive coding, meaning that the power and data rate is set according to the distance between the two nodes (possibly also the smallest data rate, the one used by broadcasts).
Of course, if one is dumb enough to use a constant rate manager and sets the unicast data rate to be higher than the broadcast one...

T.

jaasilva

unread,
Nov 6, 2015, 8:39:36 AM11/6/15
to ns-3-users
in my example I'm creating the devices in the same way as the AODV example.
I use a constant rate manager, but I don't set data rates for unicast and broadcast, so I assume they have the default values.

  WifiMacHelper wifiMac = NqosWifiMacHelper::Default ();
  wifiMac
.SetType ("ns3::AdhocWifiMac");
 
YansWifiPhyHelper wifiPhy = YansWifiPhyHelper::Default ();
 
YansWifiChannelHelper wifiChannel = YansWifiChannelHelper::Default ();
  wifiPhy
.SetChannel (wifiChannel.Create ());
 
WifiHelper wifi = WifiHelper::Default ();
  wifi
.SetRemoteStationManager ("ns3::ConstantRateWifiManager", "DataMode",
 
StringValue ("OfdmRate6Mbps"),
 
"RtsCtsThreshold", UintegerValue (0));
 
NetDeviceContainer devices;
  devices
= wifi.Install (wifiPhy, wifiMac, nodes);

But, when I run my example, the problem that I explained happens.
Node A receives HELLO messages from node B, but when using node B to forward packets, packets don't reach node B.
HELLO messages are broadcasts and packets are forward using unicast.
Is there other explanation for this?

Best regards,
JS 

Tommaso Pecorella

unread,
Nov 6, 2015, 9:45:57 AM11/6/15
to ns-3-users
Hi,

try checking what OfdmRate are using the broadcast packets...

T. 

jaasilva

unread,
Nov 6, 2015, 10:32:09 AM11/6/15
to ns-3-users
Hi Tommaso,

Do you know where can I find that?

JS

Tommaso Pecorella

unread,
Nov 6, 2015, 11:43:16 AM11/6/15
to ns-3-users
Hi,

no idea. Perhaps the Wireshark trace ?

T.

jaasilva

unread,
Nov 11, 2015, 6:14:43 AM11/11/15
to ns-3-users
Hi everyone,

After some logging, I'm sure that both unicasts and broadcasts are transmitted at the same data rate.
And I enforce that through:
wifi.SetRemoteStationManager ("ns3::ConstantRateWifiManager", "DataMode",
 
StringValue ("OfdmRate6Mbps"),

 
"NonUnicastMode",
 
WifiModeValue(WifiMode("OfdmRate6Mbps")));

But I still don't understand how a broadcast message is received.
In my routing protocol, nodes periodically broadcast HELLO messages with some information.
In this scenario node-5 is supposed to be out of range of node 1, but node 1 still receives its broadcast message.
But when sending a direct unicast message, node 1 is not able to send the message through node 5.
I have this trace: (I have a scenario with 10 nodes, from 1 through 10)

7.79318s RoutingProtocol:SendHelloMsg(): node-5 Bcast HELLO pckt:607 215:50:0
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x258ae10, 0x2580980, -89.4754, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): sync to signal (power=1.42055e-12W)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x257d430, 0x262d100, -91.478, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): sync to signal (power=8.9577e-13W)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x25768b0, 0x25579c0, -92.606, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): sync to signal (power=6.90882e-13W)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x2579e90, 0x2576820, -96.7539, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): sync to signal (power=2.65832e-13W)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x25f7200, 0x2598ef0, -96.7539, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): sync to signal (power=2.65832e-13W)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x2580b50, 0x2587c10, -100.697, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): drop packet because signal power too Small (1.07224e-13<2.51189e-13)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x258e410, 0x25aa2b0, -101.619, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): drop packet because signal power too Small (8.67219e-14<2.51189e-13)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x25fa490, 0x2580f50, -102.308, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): drop packet because signal power too Small (7.39958e-14<2.51189e-13)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(0x2587810, 0x258e810, -102.846, OfdmRate6Mbps, 0, 0)
7.79318s YansWifiPhy:StartReceivePreambleAndHeader(): drop packet because signal power too Small (6.53779e-14<2.51189e-13)

In this first part I understand that only 5 nodes are able to receive the packet at the PHY layer.
4 of them drop the packet because signal power is to small.
 
7.7932s YansWifiPhy:StartReceivePacket(0x258ae10, 0x2580980, OfdmRate6Mbps, 0, 0)
7.7932s YansWifiPhy:StartReceivePacket(): snr(dB)=5.49054, per=8.39777e-07
7.7932s YansWifiPhy:StartReceivePacket(): receiving plcp payload
7.7932s YansWifiPhy:StartReceivePacket(0x257d430, 0x262d100, OfdmRate6Mbps, 0, 0)
7.7932s YansWifiPhy:StartReceivePacket(): snr(dB)=3.48795, per=0.00216338
7.7932s YansWifiPhy:StartReceivePacket(): receiving plcp payload
7.7932s YansWifiPhy:StartReceivePacket(0x25768b0, 0x25579c0, OfdmRate6Mbps, 0, 0)
7.7932s YansWifiPhy:StartReceivePacket(): snr(dB)=2.36003, per=0.10283
7.7932s YansWifiPhy:StartReceivePacket(): receiving plcp payload
7.7932s YansWifiPhy:StartReceivePacket(0x2579e90, 0x2576820, OfdmRate6Mbps, 0, 0)
7.7932s YansWifiPhy:StartReceivePacket(): snr(dB)=-1.78794, per=1
7.7932s YansWifiPhy:StartReceivePacket(): drop packet because plcp preamble/header reception failed
7.7932s YansWifiPhy:StartReceivePacket(0x25f7200, 0x2598ef0, OfdmRate6Mbps, 0, 0)
7.7932s YansWifiPhy:StartReceivePacket(): snr(dB)=-1.78794, per=1
7.7932s YansWifiPhy:StartReceivePacket(): drop packet because plcp preamble/header reception failed

From this part I understand that 2 nodes will drop the packet at this phase because plcp preamble/header was not received correctly.
The first reception (0x258ae10) I believe is the sending node itself, right? (because although the plcp payload is received, there is no corresponding call to the upper layers)
 
7.79332s YansWifiPhy:EndReceive(0x258ae10, 0x2580980, 0x25be7a0)
7.79332s YansWifiPhy:EndReceive(): mode=6000000, snr(dB)=5.49054, per=2.39959e-05, size=81
7.79332s AdhocWifiMac:Receive(0x25883d0, 0x2580980, 0x7ffc6349ecf0)
7.79332s RegularWifiMac:ForwardUp(0x25883d0, 0x2580980, 00:00:00:00:00:05)
7.79332s RoutingProtocol:RouteInput(0x261d430, node-7, 607, 10.0.255.255, 04-06-00:00:00:00:00:07)
7.79332s RoutingProtocol:RouteInput(): node-7 LCB pckt:607 (HELLO) to 10.0.255.255
7.79332s RoutingProtocol:RecvPacket(): node-7 Update neighbors table 10.0.0.5--215:50:0
7.79332s YansWifiPhy:EndReceive(0x257d430, 0x262d100, 0x25be6b0)
7.79332s YansWifiPhy:EndReceive(): mode=6000000, snr(dB)=3.48795, per=0.0600085, size=81
7.79332s AdhocWifiMac:Receive(0x257aa60, 0x262d100, 0x7ffc6349ecf0)
7.79332s RegularWifiMac:ForwardUp(0x257aa60, 0x262d100, 00:00:00:00:00:05)
7.79332s RoutingProtocol:RouteInput(0x26164f0, node-3, 607, 10.0.255.255, 04-06-00:00:00:00:00:03)
7.79332s RoutingProtocol:RouteInput(): node-3 LCB pckt:607 (HELLO) to 10.0.255.255
7.79332s RoutingProtocol:RecvPacket(): node-3 Update neighbors table 10.0.0.5--215:50:0
7.79332s YansWifiPhy:EndReceive(0x25768b0, 0x25579c0, 0x2623c40)
7.79332s YansWifiPhy:EndReceive(): mode=6000000, snr(dB)=2.36003, per=0.954978, size=81
7.79332s AdhocWifiMac:Receive(0x2574020, 0x25579c0, 0x7ffc6349ecf0)
7.79332s RegularWifiMac:ForwardUp(0x2574020, 0x25579c0, 00:00:00:00:00:05)
7.79332s RoutingProtocol:RouteInput(0x2612c00, node-1, 607, 10.0.255.255, 04-06-00:00:00:00:00:01)
7.79332s RoutingProtocol:RouteInput(): node-1 LCB pckt:607 (HELLO) to 10.0.255.255
7.79332s RoutingProtocol:RecvPacket(): node-1 Update neighbors table 10.0.0.5--215:50:0
7.79332s YansWifiPhy:EndReceive(0x2579e90, 0x2576820, 0x2577f90)
7.79332s YansWifiPhy:EndReceive(0x25f7200, 0x2598ef0, 0x261e2d0)
 
In the end, I see that the reference 0x2579e90 is node 1.
Despite dropping the packet because plcp preamble/header was not received correctly, it still is forwarded to the upper layers and delivered to the application.
From what I can understand from this, it shouldn't, right?
Or am I missing something?

Regards,
JS

Tommaso Pecorella

unread,
Nov 11, 2015, 11:04:45 AM11/11/15
to ns-3-users
Hi,

the code for broadcast and unicast in that code part is identical. About forwarding the packet up, you must check if this function is called:
WifiPhyStateHelper::SwitchFromRxEndOk

T.

Shehzad Rizwan

unread,
Jul 21, 2022, 1:26:21 AM7/21/22
to ns-3-users
Hi Sir, 
              I want to run your protocol srr-routing-protocol. I need srr-routing-protocol.h file. Can you provide me. I am a PhD student. Want to lean geographical location aware protocol in vanets.

Thanks

Reply all
Reply to author
Forward
0 new messages