Tricky problem with twisted, reactor, and a simple web front end

[John Neiberger]

My boss and I finally have a VERY basic web front end working in Django that kicks off when a user goes to a certain URL. The script queries a network device and returns the output to the screen. The script is using Twisted in the background and it works great. The problem is that if a user goes to that URL a second time, we get an error that says reactor is not restartable. In order for someone to hit that URL more than once, we have to restart Apache after every run. Clearly not a scalable solution! We ultimately want to have a front end that can select from multiple scripts, provide input, etc., but I'm not sure how we'll pull it off with this setup.

Do any of you have any ideas how we could have interactive front ends to multiple scripts that use trigger in the background? If we keep running into these sorts of issues, I may have to jettison Trigger and just use pexpect, which works but is far, far more verbose. My boss thought perhaps within Django we could spawn a child process to run the Trigger/Twisted stuff, so maybe that would destroy the old process and allow us to run the script again without running into the "reactor not restartable" error all the time.

What do you think?

Thanks!

[Jathan McCollum]

First of all, I'm really happy that you're trying things like this with Trigger. I really want to get user guides for things like this documented, and having these conversations with you is a great start!

So, I'm very familiar with the issue you're running into. That's a tricky thing about Twisted, is that the reactor cannot be restarted once stopped. There is currently a feature in the works for Trigger to have a "ReactorlessCommando" class that does not stop and start the reactor on its own, and expects you to already have one.

Here is a pattern you can use:

https://gist.github.com/jathanism/4543974

And here is a pull request where our good friend Crazed was trying to do pretty much the same thing you're talking about:

https://github.com/trigger/trigger/pull/140

If you follow the thread on that pull request, you'll see that we ended up deciding together that the ReactorlessCommando is a better long-term solution for integrating into Trigger's core library.

There's another option that is built into trigger.contrib.xmlrpc that allows you to run a reactor as a daemon and define entry points using plugins that can be called remotely over XMLRPC. This is also in need of better documentation, but there is at least an example here:

https://github.com/trigger/trigger/tree/develop/examples/xmlrpc_server

Let me know which direction you'd like to go in and I'll be happy to help where I can.

--
You received this message because you are subscribed to the Google Groups "Trigger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trigger-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Jathan.
--

[John Neiberger]

Excellent! Thanks again! I'll will check into this as soon as I can. On a related note, I was talking to a guy on reddit about this and here is his response. In the context of Trigger and my situation, what do you think? I don't think he uses Trigger. I'm pretty sure he just glanced at the docs to help me out.

Hmm, if I understand the Trigger / Commando documentation correctly, it shouldn't be a problem. (Note that Commando's management of the event loop is essentially the same thing as what Crochet does: if you use that, you don't need something like Crochet.)

To solve this, you probably have two options:

1. Make your main Commando instance module-global, instead of dynamically constructing and destroying it somewhere (which is what I'm guessing is currently happening, due to the symptoms). If you make the Commandoinstance global, it will persist between Django's individual request/response handling cycles, and won't stop the reactor prematurely.

2. Use CommandoApplication instead of Commando, and let Crochet manage the reactor instead. CommandoApplication is a subclass of Commando that doesn't manage the reactor for you, so you don't have to worry about its lifecycle, or make it global, and can run the reactor however you wish (twistd, or Crochet, or manually).

All of this probably sounds a lot more complicated than it actually is. Try the first option, first: if caching or making the Commando instance module global solves it, you should be good to go, and can trust Commando to take care of the rest.

[Jathan McCollum]

Gah, CommandoApplication is actually a good choice! With the caveat that it was designed for a similar use-case, where I was front-ending Trigger with a RESTful JSON API. This is actually the pluggable base I was talking about for the XMLRPC functionality.

I just wanted to highlight that the results that come back from a CommandoApplication instance are designed to be serialized (such as to JSON), which might be better depending on your use-case.

We need more user docs!

[John Neiberger]

I'd be glad to help with the user docs if I understood any of this.  :)  I'll take a look at CommandoApplication and see if I can understand how it might work in relation to our goals. I still worry that given my experience level, I may be making this more complicated than I can handle. lol All we really want is a modular, reusable scripting platform for network auditing and automation. We have other groups in the company who have done similar things in Perl/PHP, but our group has decided that since we're starting from scratch, we'd rather go with Python. There are other tools and automation teams in the company who are also switching to python, but they focus on other things. We're just network engineers who want to improve our ability to manage and monitor our part of the network. That alone is a few thousand devices, so a better, more customized automation and auditing platform would be quite helpful. 

In the end, we foresee a platform from which we can write custom Python scripts for auditing for specific things, like the stuck TCP sockets issue I mentioned earlier, but we also want a basic web front end to make it easily accessible and to make it simple for people to launch certain scripts or queries that lend themselves to a web interface. I'm really not sure what it will look like in the end. We're sort of just diving in and trying to make it work. lol  

I really appreciate the help!

[Jathan McCollum]

I think you've come to the right place. Trigger was designed to do exactly what you're describing. I'm a network engineer who codes and Trigger has been invaluable in auditing and reusability in the past.

It occurred to me that maybe this talk I did might be a good walkthrough for you:

http://www.youtube.com/watch?v=7zZ9980X_bs

The slides are linked there too. It shouldn't be too hard to follow along in the slides based on what I'm saying in the video!

I've got a web service with a RESTful API in the works that will be leveraging the functionality we're talking about here, e.g. the long-running reactor loop, a web front end, etc. In fact, I call it out at the end of the talk. The various components found inside of trigger.contrib were a result of work I did at AOL building exactly this, which unfortunately we were not able to open source before I left the company last year.

My goal is to get more of this core functionality into Trigger, properly documented, and made as easy as possible to get up and running. I have a great love of Django, so having a Django app that knows how to talk to Trigger on the back-end ,for example, is something I really want to build (or help to build).

I look forward to working with you as you get more familiar with Trigger!  Your feedback so far has been immensely valuable!