Help needed to create a stress-testbed for Openwrt router

[Ashish Sharma]

Hi guys,

I am new to whole linux kernel as well as networking domain. I am largely an application layer developer.

I have been working on customizing Openwrt routers with the intent of creating something like 'Whatsapp for Routers' which should be able to take care of 90% of serious networking needs (including corporates) without needing to employ a sys-admin.

I am on this for an year. Made some decent progress. Deployed some of the routers on production. Now hitting some scaling issues. I am now looking to create a testbed kind of setup where we could simulate 100 (or more) parallel networking clients who connect to the router, do some random downloads & log in a file as to time taken for downloads, errors if any.

For a start, I figured https://github.com/saravana815/dhtest is a good start point. It can help me create multiple of such clients which could do complete dhcp thing with the router. I am now trying to find a way to expand the code-base to have the ability to do http calls like download & upload.

Can somebody point me to the right code I would be looking at ? If someone has prior experience doing something similar, please share your work here.

Thanks
Ashish

[Shakthi Kannan]

Hi,

--- On Mon, Oct 17, 2016 at 11:44 AM, Ashish Sharma

<pocha....@gmail.com> wrote:
| I have been working on customizing Openwrt routers

\--

Where are your code changes?

SK

--
Shakthi Kannan
http://www.shakthimaan.com

[Ashish Sharma]

There are no code changes as such. I am bundling Openwrt packages that make sense for most of the cases. I have bundled SQM & mwan3 as the first provides traffic prioritization & the second load balancing. Apart from that, I am exporting logs to the server through UDP. That setup is explained at https://gist.github.com/pocha/1adbd5e6ab01176ba4608937b94dd272

Most of the changes are related to configuration. Nothing inside the code as such. 

As for now, the next step is to build a self-serve wizard that gets launched & ask user to put values that could be fed to configuration options of SQM & mwan (upload, download speed for example) through some kind of captive portal customization. As for now, the user is suppose to login to the router dashboard himself & put those values. We are trying to simplify the task. Guess some level of code changes will come from there. 

Did I answer your question ? 

[Shakthi Kannan]

Hi,

--- On Mon, Oct 17, 2016 at 12:18 PM, Ashish Sharma

<pocha....@gmail.com> wrote:
| There are no code changes as such. I am bundling Openwrt packages that make
| sense for most of the cases.

\--

So, when you say 'testbed', is your objective is to test the
application(s) or entire OpenWRT images?

If it is the former, you can consider using multiple Docker
containers. But, if you really need to to test network routing logic,
then you can spawn multiple VMs and test the same. Search for Vagrant
and Terraform.

---

| That setup is explained
| at https://gist.github.com/pocha/1adbd5e6ab01176ba4608937b94dd272
|
| Most of the changes are related to configuration.

\--

Do you use any IT automation software (Chef, Ansible, SaltStack,
Puppet, etc.) for deploying your changes?

[Tavish Naruka]

Apache for example has 'ab' as a tool for stress testing webservers. Look for other similar tools maybe. I am guessing the clients aren't running on routers itself, and it doesn't matter if it is.

On a desktop you can get away with running multiple wget's also.

for((i=1;i<100;i++)); do wget -O /dev/null "http://myserver.com" & done

Then log as necessary based on time taken by wget.

On Mon, Oct 17, 2016 at 11:44 AM, Ashish Sharma <pocha....@gmail.com> wrote:

--
--
Mailing list guidelines and other related articles: http://lug-iitd.org/Footer

---
You received this message because you are subscribed to the Google Groups "Linux User Group @ IIT Delhi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iitdlug+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards
Tavish Naruka

[Ashish Sharma]

If it is the former, you can consider using multiple Docker
containers. But, if you really need to to test network routing logic,
then you can spawn multiple VMs and test the same. Search for Vagrant
and Terraform.

I am not a Docker/VM expert - but I dont think using the above approach,  I will be able to spawn 100s or 1000s of clients as it would be constrained by system resource to spawn each of these VM/Docker.

I am guessing there is a lightweight way to get it done. If you look at https://github.com/saravana815/dhtest it is able to act as standalone DHCP client without needing any of separate interface or virtualization which has got me interested. Hence I am more interested going this route. 

 

---
| That setup is explained
| at https://gist.github.com/pocha/1adbd5e6ab01176ba4608937b94dd272
|
| Most of the changes are related to configuration.
\--

Do you use any IT automation software (Chef, Ansible, SaltStack,
Puppet, etc.) for deploying your changes?

No. From my what I understand, all the above things are needed to deploy changes to a server. For us, we need to create a new firmware image with all the configuration changes. 

Openwrt has a way to separately plug only configuration through the router dashboard. Anything which is user specific, our user could download that from our site & then upload on his router. From the above change about launching a wizard, we are trying to automate this part. 

[Ashish Sharma]

ab is a relevant tool to stress test webserver.

What I am looking for is similar tool to stress test a router which includes following steps :-

1. A network client connection simulation which will trigger DHCP address allocation by the router.

2. Once DHCP is done, the client can now upload/download list of files (one by one) & log time taken to download. The idea is to be able to see how the router is behaving (effectively how much time taken on downloading certain files under how much load).

The issue is - tools like 'ab' - only serve the http part. All the 'ab' instance will be like 1 single network client to the router & hence will not truly simulate the real situation. 

https://github.com/saravana815/dhtest is a way to force router to allocate IP to each of the running client through DHCP. So far so good. But the tool does not have tcp/ip or http stack integrated & I thought extending it would be logical step. 

What do you think ?

To unsubscribe from this group and stop receiving emails from it, send an email to iitdlug+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards
Tavish Naruka

[Shakthi Kannan]

Hi Ashish,

--- On Tue, Oct 18, 2016 at 9:49 PM, Ashish Sharma

<pocha....@gmail.com> wrote:
| I am not a Docker/VM expert - but I dont think using the above approach, I
| will be able to spawn 100s or 1000s of clients as it would be constrained by
| system resource to spawn each of these VM/Docker.

\--

You can. It depends on your hardware.

---

| I am guessing there is a lightweight way to get it done. If you look at
| https://github.com/saravana815/dhtest it is able to act as standalone DHCP
| client without needing any of separate interface or virtualization which has
| got me interested. Hence I am more interested going this route.

\--

I am still not sure what parts of the networking stack you want to
test. If dhtest addresses your needs, then by all means use it.

---

| Openwrt has a way to separately plug only configuration through the router
| dashboard. Anything which is user specific, our user could download that
| from our site & then upload on his router. From the above change about
| launching a wizard, we are trying to automate this part.

\--

Seriously, use this opportunity to learn an IT automation tool for
configuration and deployments. It will help you in the long run. For a
start:

https://github.com/lefant/ansible-openwrt

[Tavish Naruka]

On Tue, Oct 18, 2016 at 9:53 PM, Ashish Sharma <pocha....@gmail.com> wrote:

ab is a relevant tool to stress test webserver.

What I am looking for is similar tool to stress test a router which includes following steps :-

1. A network client connection simulation which will trigger DHCP address allocation by the router.

2. Once DHCP is done, the client can now upload/download list of files (one by one) & log time taken to download. The idea is to be able to see how the router is behaving (effectively how much time taken on downloading certain files under how much load).

The issue is - tools like 'ab' - only serve the http part. All the 'ab' instance will be like 1 single network client to the router & hence will not truly simulate the real situation. 

https://github.com/saravana815/dhtest is a way to force router to allocate IP to each of the running client through DHCP. So far so good. But the tool does not have tcp/ip or http stack integrated & I thought extending it would be logical step. 

What do you think ?

You should still be able to hack together a script with tools like dhclient from busybox, create multiple virtual interfaces and get IPs for all of them with DHCP using dhclient, once you get an IP use ab (it looks like you can tell ab to bind to particular network address, which should make it connect from that address using the -B local-address parameter) to download your file (easily thousands of connections), time both of these operations. This _should_ simulate multiple clients I think.

--

Regards
Tavish Naruka

[Ashish Sharma]

Hi Tavish

> You should still be able to hack together a script with tools like dhclient from busybox,

Small clarification. dhclient needs to run from outside the router. Like a laptop/PC connected to the network. dhclient can be compiled & run on any OS. 

> create multiple virtual interfaces and get IPs for all of them with DHCP using dhclient, once you get an IP use ab (it looks like you can tell ab to bind to particular network address, which should make it connect from that address using the -B local-address parameter) to download your file (easily thousands of connections),

Question - someone else also suggested that the machine that is simulating multiple network clients will need multiple interfaces - like one interface per client. Is this a mandatory thing ? dhclient does not need multiple interface to complete dhcp process but the http stack might. 

Also - would not the system be limited by the number of interfaces my machine could support. Theoretically, a linux machine can have 1024 interfaces but does anybody has practically tried creating (& eventually using) multiple such interfaces on their laptop & see how many such interfaces they could create. 

>
> --
> Regards
> Tavish Naruka

>
> --
> --
> Mailing list guidelines and other related articles: http://lug-iitd.org/Footer
>
> ---

> You received this message because you are subscribed to a topic in the Google Groups "Linux User Group @ IIT Delhi" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/iitdlug/XDmbQ7WOqao/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to iitdlug+unsubscribe@googlegroups.com.

[Ashish Sharma]

Hi Shakthi

On Tue, Oct 18, 2016 at 11:12 PM, Shakthi Kannan <shakt...@gmail.com> wrote:

Hi Ashish,

--- On Tue, Oct 18, 2016 at 9:49 PM, Ashish Sharma
<pocha....@gmail.com> wrote:
| I am not a Docker/VM expert - but I dont think using the above approach,  I
| will be able to spawn 100s or 1000s of clients as it would be constrained by
| system resource to spawn each of these VM/Docker.
\--

You can. It depends on your hardware.

What is the maximum number of Dockers you have practically run on a laptop/PC. Also let me know the configuration of the PC to get an idea.  

---
| I am guessing there is a lightweight way to get it done. If you look at
| https://github.com/saravana815/dhtest it is able to act as standalone DHCP
| client without needing any of separate interface or virtualization which has
| got me interested. Hence I am more interested going this route.
\--

I am still not sure what parts of the networking stack you want to
test. If dhtest addresses your needs, then by all means use it.

I am looking to simulate a complete network client from outside the router, that connects to the router. negotiate DHCP & then download/upload stuff. dhclient only does DHCP negotiation. The rest I need to build. 

Could you point me to similar stuff like dhclient but that contains the http part. May be, I am able to stitch the two of them together.  

---
| Openwrt has a way to separately plug only configuration through the router
| dashboard. Anything which is user specific, our user could download that
| from our site & then upload on his router. From the above change about
| launching a wizard, we are trying to automate this part.
\--

Seriously, use this opportunity to learn an IT automation tool for
configuration and deployments. It will help you in the long run. For a
start:

  https://github.com/lefant/ansible-openwrt

I had a look. From what I understand - this tool can help deploying configuration/changes when I am connected to the same network as that of my router. 

Practically, this would never be the case. If I have to update any of my routers, there need to be an update client sitting on the router, that wakes up on periodic basis, check for updates, download (optionally show a screen that update in progress to anybody who connects to wifi at that moment) & then install updates.  

I am not sure if the resource you pointed above could help me with that. It probably can help me make a router 'ready' to be deployed to a client location.

Correct me if I am wrong. 

[Shakthi Kannan]

Hi,

--- On Wed, Oct 19, 2016 at 3:08 PM, Ashish Sharma

<pocha....@gmail.com> wrote:
| What is the maximum number of Dockers you have practically run on a
| laptop/PC. Also let me know the configuration of the PC to get an idea.

\--

As I had mentioned, it depends on hardware and your requirement.

---

| If I have to update any of my
| routers, there need to be an update client sitting on the router, that wakes
| up on periodic basis, check for updates, download (optionally show a screen
| that update in progress to anybody who connects to wifi at that moment) &
| then install updates.

\--

This is pull-based deployment. There are also push-based deployment
techniques. Please spend some time reading on the same.

Regards,

[Ashish Sharma]

On Wed, Oct 19, 2016 at 3:35 PM, Shakthi Kannan <shakt...@gmail.com> wrote:

Hi,

--- On Wed, Oct 19, 2016 at 3:08 PM, Ashish Sharma
<pocha....@gmail.com> wrote:
| What is the maximum number of Dockers you have practically run on a
| laptop/PC. Also let me know the configuration of the PC to get an idea.
\--

As I had mentioned, it depends on hardware and your requirement.

I was just asking if you had first hand experience. So if you could share some stats from your own setup/requirement for me to be able to extra/intra-polate values.

Apart from Docker, as what Tavish has suggested,  you think each Docker will eventually need a separate interface to communicate to the router ? As for now, the multiple interface approach looks scrappier to me. What do you think ?   

[Shakthi Kannan]

Hi,

--- On Wed, Oct 19, 2016 at 3:40 PM, Ashish Sharma

<pocha....@gmail.com> wrote:
| So if you could share some stats

\--

I do not have much info on your application. See the following for a start:

https://dzone.com/articles/run-1000-docker-redis-containers-in-less-than-15-m

---

| the multiple interface approach looks scrappier to me.

\--

It is hard to tell unless you test it out yourself.

[Ashish Sharma]

Hi Tavish, Shakthi (& any other networking expert here)

 I have made some progress on this & need some of your input. The progress & question is put on this Stackoverflow thread - http://stackoverflow.com/questions/40157381/will-bind-address-in-wget-over-ride-the-ip-address-of-the-interface-from-which

 Please have a look & if you could answer (here or there), that would surely help.

Ashish