Classifieds page loading is to slow

[Salima Begum]

Hi all,

We are building a Web application, We have a classifieds page that is loading too slow. How to fix this issue. how to reduce loading time of classifieds page.

Thanks

~Salima

[Benny M]

Slow loading can be sometimes be caused by non-optimized or overly complex queries. Look into prefetch_related and select_related for starters. https://docs.djangoproject.com/en/3.1/ref/models/querysets/

Django Debug Toolbar is also a great packages for examining the interaction between Django’s ORM and your database. 

Best,

Benny

On Jan 14, 2021, at 9:18 PM, Salima Begum <sali...@rohteksolutions.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMSz6bnvP51oD_dfhY0YEWiGHGq3DbZ04u%2Bz6Vg_9sBPGvjgOw%40mail.gmail.com.

[Benjamin Schollnick]

Also don’t forget to include Indexing.  

As a general rule, any field that you perform any query against, should be indexed.  

Yes, it costs a “bit” for disk space, but indexing can dramatically increase your query speed.

- Benjamin

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CH2PR14MB391358D987FFC8303F087CD5C0A70%40CH2PR14MB3913.namprd14.prod.outlook.com.

[Ram]

Hi,

We have a similar type of challenge in our project. We will definitely try indexing.

Basically we are currently doing like this.

1. Registered users can post an ad in Classifieds by uploading 4 - 6 images per ad. All the ads are placed in one postgres db table. (classifieds table)

2. Same user opens the Classifieds page where we list all the Ads posted by this user as well as other users. This page loading is taking a lot of time.

3. Each ad will display a thumbnail image, title of the ad, 2 line description, price value, distance value from nearby zip code, age of the ad, zip code

4. Since we  are calculating the distance between zip codes each Ad using a third party API, I think page loading is taking a lot of time.

4.1. I've dumb idea here where I can add a. new table, 'nearby_ads table'. As soon as the ad was submitted successfully, process the ads distance and add all the nearby ads into this table. When the classifieds page is launched, only nearby ads will be displayed. First of all I'm not sure whether this is a good idea because we are going to have two tables, one is classifieds table and the other one is a nearby_ads table.  Second of all I'm not sure what is the limit that I can set limit on how far I could calculate nearby ads ( like 10 miles or 30 miles or 50 miles radius).

5. Our goal is to present all the nearby ads based on the logged in user zip code and if a visitor opens this Classifieds page, we have to capture nearby location of the visitor and present the nearby ads.

6. Additionally registered users and visitors should be able to filter the ads based on different zip code though both are from different location

Please let me know if you recommend any advanced packages or libraries in Django or Python to achieve the above. Or indexing is the only option to reduce the page loading time? I appreciate it if you can share any recommendations and pointers.

Best Regards

~Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/0B38486A-1A58-4FFF-A7FE-52FAB24099B6%40schollnick.net.

[Benny M]

Ram,

These are all pieces to a puzzle. Indexing, ORM optimization and through tables all add up to an efficient page render. Additionally, you could consider caching some of your results, using a key like ‘postal code’ - some sort of constant that you could use as a metric/pivot. Use a cron job to populate your through tables and cache the results for a faster return. 

Benny

On Jan 16, 2021, at 5:47 PM, Ram <ram.mu...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2BOi5F1dj95n1448Q3ueTfPjGwy5H%2BAj%3DXJ6uD348P8YhnLxPw%40mail.gmail.com.

[Ram]

Thank you, Benny. We will try it and let you know.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CH2PR14MB391358D75B8DE8E42A48107CC0A50%40CH2PR14MB3913.namprd14.prod.outlook.com.

[Salima Begum]

  Hi all,

We are building website, Here I have written functionality for classifieds page. 

It is loading to slow because of We are prompting distance calculation in classifieds page. By using distance API matrix based on logged in user zip code and individual Ad zip codes from query set(database).

```

    def classifieds(request):
        global dict_time
        try:
            dict_time = {}
            email = request.session.get('email')

            # Here we are displaying the classified ads "order by date". The ads will be sorted by latest date.
            classifieds = vk_classifieds.objects.filter(class_status='1').order_by('-added_date')
           
            count = vk_classifieds.objects.filter(class_status='1').order_by('-added_date').count()
            for i in classifieds:
                # diff_time is a child method. By passing 'i' as object in diff_time.
                difrnc_date = diff_time(i)
                dict_time[i.id] = difrnc_date

            # Pagination for classifieds page.
            # classified = random.sample(list(classifieds), k=count)
            # print("classified = ", str(classified))
            # By default first page
            page = request.GET.get('page', 1)
            # print("page = ", str(page))
            # Per page setting 40 objects.
            paginator = Paginator(list(classifieds), 40)
            # print("paginator = ", str(paginator))
            classified_p = paginator.page(page)
            # print(classified_p)
        except PageNotAnInteger:
            classified_p = paginator.page(1)
        except EmptyPage:
            classified_p = paginator.page(paginator.num_pages)
        except Exception as e:
            logging.error(e)
            return render(request, "classifieds.html",
                          {"Classifieds": classified_p,
 # distance calculation
                           "distance": classifieds_dist(email),
                           'some_date': dict_time,
                           })
        return render(request, "classifieds.html", {"Classifieds": classified_p,
                                                    "distance": classifieds_dist(email),
                                                    'some_date': dict_time,
                                                    })

```

```

def classifieds_dist(email):
    global frm, km
    dict_distance = {}

    qury = vk_customer.objects.filter(email=email).first()

    # From above qury variable we are getting zip of customer.
    frm = qury.Zip
    # importing json package to calculate the distance
    url = "https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial"
    headers = {
        'Authorization': "Bearer somevalue",
        'User-Agent': "some value",
        'Accept': "*/*",
        'Cache-Control': "no-cache",
        'Postman-Token': "some value",
        'Host': "maps.googleapis.com",
        'Accept-Encoding': "gzip, deflate",
        'Connection': "keep-alive",
        'cache-control': "no-cache"
    }
    classifieds = vk_classifieds.objects.filter(class_status='1').order_by('-added_date')
    for i in classifieds:
        # while a user login through his login email we capture his complete detail into an variable which we given as "qury"
        # and after the details are stored into the variable from there we will filter his starting and destination point as zipcode

        # After login his/her based on email we are filtering in vk_customer table. Then storing in qury variable.

        # This zip is getting from vk_classifieds(model in models.py) table.
        # 'i' attribute is getting from classifieds() function.
        to = i.zip

        origin_list = [to]
        desination_list = [frm]
        # here we used api for calculating the source and destination point
        querystring = {"origins": origin_list, "destinations": desination_list, "departure_time": "now",
                       "key": "AIzaSyDhlCiMAEEfoYhkPcOyP0PLqpHsVMmYEXM"}
        # here we are passing these headers to the api

        # we are capturing the response in variable called response
        response = requests.request("GET", url, headers=headers, params=querystring)
        jsondata = response.text
        obj = json.loads(jsondata)
        list = obj['rows']
        if list:
            a = list[0].get('elements')
            obj2 = a[0].get("distance")
            if obj2 is None:
                km = "None"
            else:
                km = obj2["text"]
            dict_distance[i.id] = km
            l1.append(i.id)
            print("id = ", str(i.id))
            print("km = ", str(km))
    return dict_distance

```

Because of this loading time of classifieds page is to slow. Please help me to solve this issue. 

Thanks

~Salima

[Ryan Nowakowski]

I recommend profiling the app to figure out exactly what part is slowing you down before you do any optimization.

[Salima Begum]

Thank you everyone. We identified the issue.
Basically we are using an API which calculates distance  
between two different ads using zip code value and this help us
to show the distance of the ad from logged-in users location.
So when classifies page is being loaded, this API is called for each and every.
Ad and taking time to process distance calculation for each Ad.
So we need to avoid this somehow and we are looking for any advanced method or package to achieve this.

[Benny M]

You could try caching your results, or even store them in the table, or related table. If the ads location doesn’t change, there doesn’t seem to be a need to call the api every single time. Additionally, if the results aren’t needed right away, you could queue the job of calling the api, such that even if the api round-trip takes a while, the user can continue uninterrupted, while the queued job does the work of fetching and storing the results.

Benny

On Feb 1, 2021, at 11:07 PM, Salima Begum <sali...@rohteksolutions.com> wrote:



--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMSz6b%3DzPyf8_p8zxKw2DRMpcdmVeVXoAaSGnxo%2BLa4t2yaTQg%40mail.gmail.com.

[Salima Begum]

Thanks, How to design a database to store into a table calculated distance API . Please Help me to achieve this thank you.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CH2PR14MB3913BDBA42CF0F356DDCD8D6C0B59%40CH2PR14MB3913.namprd14.prod.outlook.com.