Hi,
I wrote a post in my infrequently updated blog. To summarize it...
1) Study the protocol (syntax and semantic), eventually drawing the necessary sketches. I.e., packets format, state machine in the nodes, what to do when you receive a packet, when to send a packet.
2) Implement the packet format
3) Implement the sent / receive and state machine parts
4) You are already doing it during part 3... so complete and clean up the routing table.
5) Copy from another protocol the RouteInput and routeOutput functions, eventually modifying them where necessary.
6) Test it.
Suggestion: check a simple example. I'd suggest RIP-ng as it's the easiest and simplest routing protocol in ns-3.
Cheers,
T.