Dealing with large lists of objects

25 views
Skip to first unread message

Federico Capoano

unread,
May 16, 2025, 2:44:48 PMMay 16
to OpenWISP Google Summer of Code
Hi everyone,

I want to encourage all the gsoc25 contributors to keep into account the possibility that the new features you'll be working on will potentially deal with a large amount of objects which will be fetched via REST API from the database.

Keep this into account when loading data: data shall be loaded in batches and loading overlays should be used to indicate the loading is happening in the background.

For example, the firmware upgrade results for mass operations, right now we don't have any pagination, but that won't scale for mass upgrades which have thousands of upgrade operations.

This applies also for loading map data. Keep this in mind when planning your work, ensure to allocate some time to deal with improving the logic to deal with large datasets.

Best regards
Federico Capoano
OpenWISP OÜ
Kotkapoja tn 2a-10, 10615, Harju maakond, Tallinn, Estonia
VAT: EE101989729

Oliver Kraitschy

unread,
May 17, 2025, 2:54:22 AMMay 17
to Federico Capoano, OpenWISP Google Summer of Code
Hello everyone,

I suggest, when writing tests for the REST API, to add a performance test for fetching a large number of items. This gives us an impression of the current performance, lets us optimize the performance and later makes us aware of regressions which negatively affect performance.

Greetings,
Oliver

--
You received this message because you are subscribed to the Google Groups "OpenWISP Google Summer of Code" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp-gso...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openwisp-gsoc/CAAGgX6LNR1wH_QXtgrLhW-pp%3DB3TR%3DhJwHpQxKL8R4peHt-EJw%40mail.gmail.com.

Federico Capoano

unread,
May 17, 2025, 3:21:38 PMMay 17
to Oliver Kraitschy, OpenWISP Google Summer of Code
Good point, we usually add assertions for the number of queries.
Performance tests are tricky because they can yield different results on different hosts.

Best regards
Federico Capoano
OpenWISP OÜ
Kotkapoja tn 2a-10, 10615, Harju maakond, Tallinn, Estonia
VAT: EE101989729

Oliver Kraitschy

unread,
May 27, 2025, 7:32:32 AMMay 27
to OpenWISP Google Summer of Code
Yes, I know. One possibility would be to measure the time a certain action takes and then multiply that value by 1.5, for example. The resulting value could be used in an assertion. Of course this still can fail on very slow systems. I haven't thought of a better way yet.

Greetings,
Oliver

Karthik

unread,
May 28, 2025, 5:25:16 AMMay 28
to OpenWISP Google Summer of Code


Hello everyone,

I’m Karthik Suru, an aspiring contributor and GSoC 2025 applicant. I’ve been contributing to OpenWISP for the past few months, with 9 PRs successfully merged.

I’d like to inform you that I’ve started working on Issue #148 – Show node labels only after hitting a certain zoom level in the netjsongraph.js repository. This feature aims to enhance the UX by reducing visual clutter and only displaying node labels at higher zoom levels.

Looking forward to your feedback and guidance as I proceed.

Best regards,
Karthik Suru
GitHub: https://github.com/karthiksuru06
Linked in : https://www.linkedin.com/in/suru-karthik-923766321/
9390759591

Reply all
Reply to author
Forward
0 new messages