In-memory queues

[Kevin Campbell]

There are working in-memory queues for the Aliases and Metrics now, on the kevcampb-memoryqueues branch. Our only views now are 'by_aliases' and 'requested' which lists items until the QueueMonitor can pick them up from couch and place them into the in-memory queues. If you want to replace this with another messaging method later, this is the only thing needing changed.

Queueing an item for processing is as simple as calling:

AliasQueue.enqueue(item_id)

I'll still want to check through that and check for anything needing tidied. There are still some straggling variables, from different iterations of changes, and I'm sure I still need to update some docs. Test are currently passing though.

The backend performance is looking much more respectable.

larkspur:total-impact-core kev$ time ./extras/load_test.py -s 40
Requesting items (75/120)
All items requested, now waiting for results
Results received (8/120)
Results received (8/120)
Results received (8/120)
Results received (8/120)
Results received (8/120)
Results received (42/120)
Results received (42/120)
Results received (42/120)
Results received (42/120)
Results received (42/120)
Results received (77/120)
Results received (77/120)
Results received (77/120)
Results received (77/120)
Results received (77/120)
Results received (98/120)
Results received (107/120)
Results received (107/120)
Results received (107/120)
Results received (107/120)
Results received (107/120)
All details complete
The following are not yet complete...

real    0m24.463s
user    0m1.483s
sys    0m0.307s

Heather, can you get things corrected on master such that I can run nosetests again locally? Once that's complete, I'll rebase my branch from master and merge in changes. From what I've seen so far I don't expect merging is going to take much effort.

K

[Jason Priem]

Awesome Kevin, excited to try it out! Will wait until it's merged into master.

Performance of the current version has been pretty egregious with anything more than a few items. So hopefully this will solve some of that.

J

Sent from my iPad

--
You received this message because you are subscribed to the Google Groups "total-impact-dev" group.
To post to this group, send email to total-im...@googlegroups.com.
To unsubscribe from this group, send email to total-impact-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/total-impact-dev?hl=en.

[Heather Piwowar]

Great news!

Kevin, I believe nosetests are working on master when you have the secrets.py file.  For simplicity I'll email it to you right now.

Heather

K

--