Creating an app that generates static files

39 views
Skip to first unread message

Jack Sundberg

unread,
Oct 17, 2019, 10:31:50 AM10/17/19
to Django users
Hi all,

I am currently writing an app that generates static files before serving them, and I could use some input! Here's a step-by-step example of how the app is setup:

 - the url mapping pulls a database entry
 - using the data entry, static files are created from the data (for example, a *.jpeg or a *.gltf)
 - the app serves the html template with the generated static files embedded (for example, via an <img> tag)

The issue comes with the creation of the unique static files. I can have the app create a file in the static folder, send it off, then delete the file to ready for the next request -- but the webpage will run into issues if multiple requests are made at the same time. If two people request the same url at the same time, the app will fail because both requests are trying to create/serve/delete an identical file (i.e. entry123.jpeg) at the same time. Trying to mess with the filename to avoid this won't work here either (i.e. naming entry123.01.jpeg, entry123.02.jpeg, entry123.03.jpeg, ...).

I can think of two solutions, but they both are things I'd like to avoid:
 1. have the files premade and stored -- I want to avoid this because of the massive filesize required for all these files
 2. have the static files generated in a user-specific directory -- I want to avoid this because I'd rather not have registered users on my website

Is there another solution to this? I'm still relatively new with Django, so there might be something simple I'm missing.

-Jack

vineet daniel

unread,
Oct 17, 2019, 10:56:56 AM10/17/19
to django...@googlegroups.com
This problem can happen with any stack and shouldn't be considered as django specific. What you are looking for is a solution in a layman terms.which then can be translated to django implementation.

Before that i would like to know image size and kind of image that you create. Is it some qr code or what?

Please note that working with images on the fly is always a cpu intensive operation. Ill can come up with something once you share the details with me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/190637d2-bd9b-499a-a122-1dc7adf2aca5%40googlegroups.com.

Jack Sundberg

unread,
Oct 17, 2019, 11:22:24 AM10/17/19
to Django users
Yes, I tried getting the idea across in layman terms, as I hope this helps discussion of the issue -- and then I can later translate the solution into django implementation. 

I avoided the full process in my original post for clarity. I used the *.jpeg file as an illustrative example, when I'm actually generating *.gltf and *.bin files. These files are typically a couple hundred KB in size, but I will have thousands of them (millions as I add to my database). These files are used in coordination with Verge3D  - an app built on top of Three.js for 3D rendering. Verge3D allows use of Blender for CAD, which is a python-based program that my scripts are written in. Running of these scripts are certainly cpu intensive; however I am going to get around this by using an <iframe> of this Verge3D request.

So to be more specific, my app does the following:
 - the url mapping pulls a database entry
 - the app serves the proper html template with an <iframe> embedded in it. It displays a loading screen while the 3D model is created backend.
 - the <iframe> calls a separate app using the identical data entry
 - using the data entry, static files are created from the data (Blender script creates a *.gltf and *.bin)
 - the app serves the html template with the generated static files embedded inside the <iframe>

Is this helpful? Also thank you for the quick response. I really appreciate the help.

-Jack
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.

James Gutu

unread,
Oct 17, 2019, 4:42:04 PM10/17/19
to django...@googlegroups.com
Jack,

From what I understand it looks like a conflict of uniquely named files when they are generated simultaneously.
Could you resolve this by creating a folder named from the session_id, with the file inside of that. So that if two users request the same file, there will never be a conflict, as the unique file will be created in both folders.

[user_1_session_id]/entry123.jpeg
[user_2_session_id]/entry123.jpeg

Though, this might seem like a fix, but the issue may be more fundamental than file names.
Let me know what you think.

Regards,
James

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/317c4ce6-5378-4e59-920e-c0fdcacd453a%40googlegroups.com.

Jack Sundberg

unread,
Oct 17, 2019, 6:21:45 PM10/17/19
to Django users
Thank you, James! This seems like a good fix. From reading through the documentation (here), this is how I understand the workflow -- so correct me if I'm wrong please!

 > First, I can activate Django sessions by adding 'django.contrib.sessions' to my installed apps. 
 > Django by-default stores session info in the SQLite database -- therefore, each session entry will have an ID assigned to it automatically. 
 > I can use each session_id to create a temporary folder in some directory (say /tmp) where the generated files can be made. So folder structure will look like:
            tmp/
                [session_id1]/     
                    file1.tlgf
                    file2.bin
                [session_id2]/     
                    file1.tlgf
                    file2.bin
                ....
 > when sessions data hits a certain threshold, I can call django.contrib.sessions.clearsessions which will delete expired session entries in the database
 > pull out which entries were deleted and then delete the corresponding tmp/[session_id] directory 

Is that the best way to do it?

-Jack

James Gutu

unread,
Oct 18, 2019, 3:46:05 AM10/18/19
to django...@googlegroups.com
Jack,
That sounds about right.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20e29fca-88db-4454-bb7d-d25bd8271ca3%40googlegroups.com.

vineet daniel

unread,
Oct 18, 2019, 5:00:31 AM10/18/19
to django...@googlegroups.com
Dont forget inode and files limit if you are creating too many files real quickly. 

vineet daniel

unread,
Oct 18, 2019, 5:02:40 AM10/18/19
to django...@googlegroups.com
and use ssd for maximum I/O as youd creating and deleting files a lot, move session from db to something like redis and python has concurrency libraries make use of them as well.

Jack Sundberg

unread,
Oct 18, 2019, 8:34:51 AM10/18/19
to Django users
Thank you both! This helps a ton.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages