File->New->Network Description File (NED)
Give it a name. Next >
Topology Generation -> Random Topology (or the other).
Make your nodes to Standard Hosts, add whatever else you need. Add your protocol where it belongs. Have a look at the tcp apps in INET to generate traffic.