Nexus Modules

[David Cramer]

Let's get this first post rolling here.

So we built out Nexus to provide a frontend for things like Sentry/
Gargoyle, with a couple of things in mind:

1. django.contrib.admin isnt "good enough" for everyone
- however, it's good enough for most

2. things like Gargoyle and Sentry are a huge PITA to integrate into
the admin as they're highly customized interfaces

While Nexus is still unfinished (APIs may change as we find things
arent good enough), it's mostly ready for public consumption. We've
made it pretty easy to register modules, and they should all *just
work* in an applications namespace (no collisions if you have an
existing admin setup, for example).

So bundled, you get support for django.contrib.admin. Gargoyle
includes its own nexus module, and Sentry will soon. The next task is
to build out more modules, and get existing projects to integrate.

I've talked with Daniel Lindsey (creator of Haystack) about a search
interface (so you could test your search indexes) as part of it, but
that's the only project I've approached.

Does anyone else have their own project, or want to nominate another
that we could show off some integration with?

[Mat]

Some integration with celery would be awesome, for queue info and
management. Seems like a good fit..... Also it has an interesting db
backed periodic task scheduler which might fit better in nexus than
contrib.admin

Another generic module I can think of which would be handy is a
memcache stats system, similar to https://github.com/bartTC/django-memcache-status

Really impressed with both gargoyle and nexus, we've needed something
like this for a while.

Mat

[Noah Kantrowitz]

I think my first test run is going to be a new user management module that
actually handles user profiles in a non-horible way.

--Noah

[David Cramer]

I love the memcache and celery ideas. Was just talking with Ask and I'm wondering if we can just pull in celerymon and build a module out of it.

-- 
David Cramer

[Mat]

Awesome, happy to help out, we use celery extensively, obvious modules
would probably be.

Celerymon
Periodic task scheduling management
Worker management, such as changing rate limits or which tasks excute
on which machine
Queue snapshotting, all the camera stuff.

The other idea which sprung to mind was reporting based. A module
which can give overall performance stats about an app, mainly focussed
on historical latencies of named urls, and req/s, nothing too complex,
just enough to see live load on a django app and if performance is
being degraded.

Mat

On Dec 23, 12:50 am, David Cramer <dcra...@gmail.com> wrote:
>  I love the memcache and celery ideas. Was just talking with Ask and I'm wondering if we can just pull in celerymon and build a module out of it.
>
> --
> David Cramer
>
>
>
> On Wednesday, December 22, 2010 at 4:26 PM, Mat wrote:
>
> > Some integration with celery would be awesome, for queue info and
> > management. Seems like a good fit..... Also it has an interesting db
> > backed periodic task scheduler which might fit better in nexus than
> > contrib.admin
>
> > Another generic module I can think of which would be handy is a

> > memcache stats system, similar tohttps://github.com/bartTC/django-memcache-status

[David Cramer]

Looking at django-celery, and the requirements of Celerymon, I want to
go half-way inbetween.

- I don't want to require Tornado
- I want it to run under WSGI
- I don't want a huge never ending trail in the database

So now how do we solve this?

We could store things in the database like Sentry does, summarized. We
could store stats per queue per hour or something to that extent.

We could also stick everything in CACHE_BACKEND. It'd be much easier
to use Redis, but I don't want to put a requirement on having a Redis
server.

I think the ideal situation may just be building cleanup inside of
Nexus. Potentially storing the last hours worth of jobs and always
cleaning itself up. We could then also store and compile stats down by
the hour for the last week or something, and trash or compress older
stats.

Any thoughts on this?

[David Cramer]

While all of this is still in my head, here's a rough draft plan for Celery:

- Requires django-celery

- Migrates django-celery data into its own datastore and cleans up django-celery

- Stores basic analytic data for a fixed interval (maybe last month, configurable)

- Stores last N tasks (fixed number and/or fixed interval)

- Allows all of the actionable items available in django-celery

- For cleanup, runs like a wp-ghetto cron or a server-side cron. You load a page, it asyncs a js request and runs any cleanup.

-- 
David Cramer

[Mat Clayton]

Although historical data is useful, I think the main use is seeing
what size the queues are now. This is certainly what we use the most.
Infact we have never looked at the historical data before, as it's not
setup.

Having to install a separate db, I agree would be real pain and
probably a none starter for most users. Setting up a separate wsgi
process would be a pain as well, however tornado could easily be added
to a requirements.txt and running a background process seems fair
enough, it actually seems like a better option than firing off a cron
job to tidy things up, which will more than likely be forgotten.

I'm not sure there is that much value in the historical data and
therefore something in memory in tornado seems like best solution to
me, even if it is lost of restart, also has the added advantage of
keeping the db load down.

Mat

Sent from my iPad

[David Zhou]

Yeah, we're currently leaning towards a stats panel that's mainly for showing current statuses.  For example, check out this screen shot:

http://dl.dropbox.com/u/116385/Screenshots/y7bp4dibd-94.png

-- DZ