Request Timed Out When Generating Huge Data

[tango ward]

Good day,

I need suggestions on how to approach this problem.

I am working on a task that needs to generate the list students per school. When the school has larger number of students, I am getting a request timed out error. I can filter the school per campus to lessen the number of students but the performance in displaying the list of students is still taking too long.

Thanks,

Jarvis

[Babatunde Akinyanmi]

Hi, 

How many records are we talking about? What code are you using to fetch the records? In other words, add more information so that it's easier to troubleshoot

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[tango ward]

Hi,

SLR.

At the moment, the largest number of students we have is 2000 in one campus. I haven't played around with the code but when I check it we're using a get_queryset function to get the data and filter it using Q. Then display it using get_context_data

Code:

class FinancialReportView(ApplicantReportMixin, ListView):
    template_name = 'education/dashboard/reports/financial_report.html'
    paginate_by = 50

    def get_queryset(self):

        student_filter = self.request.GET.get('student_filter', '')

        filter_date = self.request.GET.get('date', '')
        if not filter_date and not student_filter:
            return Recipient.objects.none()

        if student_filter in ['None', None]:
            student_filter = ''
        recipients = self.get_recipients()
        recipients = recipients.filter(enrolled=1)
        recipients = recipients.filter(enrollmentinfo__isnull=False)

        campus = self.request.GET.get('campus', '')
        if campus:
            recipients = recipients.filter(department__school=campus)
        year_level = self.request.GET.get('year_level', '')

        if year_level:
            recipients = recipients.filter(year_level=year_level)

        if student_filter and not student_filter == 'None':
            recipients = recipients.filter(
                Q(first_name__icontains=student_filter) |
                Q(last_name__icontains=student_filter) |
                Q(student_id__icontains=student_filter))
        recipients = recipients.select_related('department', 'program')
        return recipients

    def get_context_data(self, **kwargs):
        context = super(FinancialReportView, self).get_context_data(**kwargs)

        filter_form = FinancialReportFilterForm(
            data=self.request.GET,
            school_manager=self.request.user.schoolmanager,
        )
        output = []
        recipients = self.get_queryset()

        filter_date = self.request.GET.get('date', '')
        if filter_date in ['None', None]:
            filter_date = ''
        if filter_date:
            filter_date = dateutil.parser.parse(filter_date).date()
        page = self.request.GET.get('page', 1)

        for ctr, recipient in enumerate(recipients):
            total_amount = 0
            enrollment_info = recipient.get_current_enrollment_info()
            if enrollment_info:
                if filter_date:
                    invoices = enrollment_info.philsmileinvoice_set.filter(
                        due_date__lte=filter_date)
                else:
                    invoices = enrollment_info.philsmileinvoice_set.all()

                for invoice in invoices:
                    total_amount += invoice.get_remaining_balance()

                output.append([ctr, recipient, total_amount])

        if self.paginate_by:
            paginator = Paginator(output, self.paginate_by)
            try:
                output = paginator.page(page)
            except PageNotAnInteger:
                output = paginator.page(1)
            except EmptyPage:
                output = paginator.page(paginator.num_pages)
            context.update({
                'paginator': paginator
            })

        context.update({
            'outputs': output,
            'form': filter_form,
            'school_system': self.request.user.schoolmanager.school_system,
        })
        return context

On Mon, Apr 2, 2018 at 1:54 PM, Babatunde Akinyanmi <tunde...@gmail.com> wrote:

Hi, 

How many records are we talking about? What code are you using to fetch the records? In other words, add more information so that it's easier to troubleshoot

On Mon, 2 Apr 2018, 02:27 tango ward, <tango...@gmail.com> wrote:

Good day,

I need suggestions on how to approach this problem.

I am working on a task that needs to generate the list students per school. When the school has larger number of students, I am getting a request timed out error. I can filter the school per campus to lessen the number of students but the performance in displaying the list of students is still taking too long.

Thanks,

Jarvis

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAA6wQLKn98Yq9JrbQQLVncm2pkEUzaCYm_un0zZFMXQC8ewzrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2BWjgXN%2B2U5UtbjPGzTO5Aff-s%2BAeDKT%3DYg1HwFr72C4KGB4SQ%40mail.gmail.com.

[tango ward]

I tried loading the site locally to see how long will it take to load a record of 1800 students. It took about a minute and a half locally. The request timed out problem occurs when I try to generate a CSV report and PDF. Both result to timed out even locally.