A few doubts with an implementation.

[Vineeth Sagar]

Problem:

We are developing a email-CRM. The task at hand is to fetch emails every N minutes or hours. I have received a codebase written by another guy who left the company,my job is to re-write/re-factor. The guy who previously wrote the code started a thread whenever a user logged and he put a time.sleep() inside it, There's no limit on number of threads that can be initialized, if 100 users log in, we will have 100 active threads, he used locks so that at any given time only one thread can be executed. 

Solution:

It's messy in so many ways(mentioned above) and I think writing django middleware would be appropriate for this task. So here's what I plan to do, using a Django cache I store a Python datetime object and a flag to see if there is any other request that's already retrieving the mails, I don't want multiple requests to retrieving mails at once. Is this a better idea? Are there any other ways to do this particular task?

One more doubt I have is I want the user to know that the mails are being retrieved so he doesn't refresh the page passive-aggressively. How can I do this? I want the user to know it's happening and in the background middleware should be executed, I have to keep retrieving mails and once it's completed I'll execute the view_func parameter received via the process_request hook. 

I am a junior at the company(Start up),I am fairly new to django, I keep reading the documentation and find new things daily,after picking my mind for a day this to me seems best, but please if you have any suggestion let me know. Please help me out. Thanks.

[Mikhailo Keda]

to fetch emails every N minutes or hours  - http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
I want the user to know that the mails are being retrieved - https://channels.readthedocs.io/en/latest/

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c563187e-a3ee-49c0-8c56-cb924a128891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vineeth Sagar]

Mikhailo Keda wrote:

to fetch emails every N minutes or hours  - http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
I want the user to know that the mails are being retrieved - https://channels.readthedocs.io/en/latest/

Thanks Mikhailo for the prompt reply, I'll spend a few days reading the documentation of the libraries you have pointed me to. Are web sockets supported in IE, iirc I don't think they do. Half of our clients Will use IE, not edge, good old fucked up IE. Also I was expecting a critique on my ideas. I want to weigh the pro's and con's of my approach vs clerey's(It might be clear once I read the docs.)

again thanks.

[Vijay Khemlani]

You can also use PubNub (https://www.pubnub.com/) or similar services

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/10ea3eba-a72f-4d33-8504-4658c0b810ef%40googlegroups.com.

[Bill Torcaso]

I didn't deeply consider your problem, but the aspect of doing a repetitive task at unrelated time intervals reminds me of how the Unix kernel handles the alarm() system call for multiple, unrelated processes.  For you, it will take at most one thread to handle any number of users.

1. Maintain a first-in-first-out queue, sorted by time.
2. An item in the queue has identifying info about a user, and (*this is the key fact*) a measure of time relative to the queue item before it.
3. When a new user appears, create a queue entry for that user at the appropriate spot in the time-relative queue.  That might always be at the end, but if different users get the email fetched at different intervals, the insertion spot might be in the interior of the queue.
4. When a user logs out, or by whatever criteria you chose, scan the list and remove any queue entry that refers to that user.

The API for other code to call the queue-management thread is only two things:  

• Create an entry for this user, at this repetition interval. 
  
• Remove the entry this user.

The rest of the code for the queue is internal implementation details.This is the action of the queue-management thread:

1. At start-time, create this thread.  It is idle to start with.
   
2. When a user logs in, other code will send a signal to the thread.  When a user logs out, likewise send a signal to the queue-management thread.
3. The thread calculates how long to sleep before waking up and fetching email for the entry at the head of the queue.
4. When the thread wakes up, it does the work for the item at the head of the queue.  If that work takes a long time, or has complex handling of errors during the work, you might spawn a thread to do the work outside of the queue-management thread.
5. Remove that queue-head item. Re-insert it at the place in the queue for its next execution on behalf of this user.  That will likely be at the end, but might be in the interior of the queue.
6. Loop back to #3.