Apple M1 Silicon + Django = weird results

[Benjamin Schollnick]

Folks,

I’m using django for an online gallery application, and I’ve seen some weird results since switching over to the M1 Mac Mini.

(Please note, I am running it under Rosetta / Intel Translation, since pillow doesn’t seem to build under M1 native).

So this could be a quirk of Rosetta, Django, or some combination of these.

I don’t think so, but it could be.

So What is happening?  

I never saw this under my 2013 intel iMac, but now under M1 (even under Rosetta 2 / Intel translation) I am receiving a damaged / altered download file roughly 6 out of 8 or 9 times (~25-33% successful).  

Just retrying the download will eventually be successful.  

This isn’t Pillow related, since the file is just being downloaded directly (no thumbnails or alterations to the file is occurring).

Does anyone have a suggestion on what to do at this time?  I’m uncertain on where to start to dig.  Although removing ranged_response, might be a starting point…

URLs:

    path("download/<uuid:d_uuid>", frontend.views.new_download, name="downloads"),

    path('thumbnails/<uuid:t_url_name>', frontend.views.thumbnails, name="raw thumbnails"),

First, I am using these function for all graphics and/or file(s) being sent:

def new_download(request, d_uuid=None):

    page = request.GET.get('page', None)

    if page is None:

        download = index_data.objects.filter(uuid=d_uuid,

                                             ignore=False,

                                             delete_pending=False)[0]

    else:

        print ("Attempting to find page %s in archive" % page)

    print("\tDownloading - %s, %s" % (download.fqpndirectory.lower(),

                                      download.name))

    return respond_as_inline(request,

                                 "%s%s%s" % (

                                     configdata["locations"]["albums_path"],

                                     os.sep,

                                     download.fqpndirectory),

                                 download.name)

def respond_as_inline(request, file_path, original_filename, ranged=False):

    # https://stackoverflow.com/questions/36392510/django-download-a-file

    # https://stackoverflow.com/questions/27712778/video-plays-in-other-browsers-but-not-safari

    # https://stackoverflow.com/questions/720419/how-can-i-find-out-whether-a-server-supports-the-range-header

    filename = os.path.join(file_path, original_filename)

    if os.path.exists(filename):

        mtype, encoding = mimetypes.guess_type(original_filename)

        if mtype is None:

            mtype = 'application/octet-stream'

        response = RangedFileResponse(request, file=open(filename, 'rb'), as_attachment=False, filename=os.path.basename(filename))

        response["Content-Type"] = mtype

        return response

    raise Http404

[Stephen J. Butler]

Looking at that code for RangedFileResponse, I'm kind of skeptical that it properly implements the Range header. Try removing that bit and see if it works.

Have you tried the download with various browsers? I wonder if Safari or Chrome released at the same time as the M1 chips is causing different Range header behavior than that code is designed to handle. So it might not be the M1 chip, but just an unrelated change in browser software that got pushed out at the same time. Speculating here.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/75E15498-69D8-47B5-94D2-821C6B35CA92%40schollnick.net.

[Benjamin Schollnick]

Have you tried the download with various browsers? I wonder if Safari or Chrome released at the same time as the M1 chips is causing different Range header behavior than that code is designed to handle. So it might not be the M1 chip, but just an unrelated change in browser software that got pushed out at the same time. Speculating here.

The only issue that I see, is that I started to use RangedFileResponse to allow the viewing of MP4, and other media files in the browser…  (

So I know it works at least in that regard.  

But for debugging purposes, I’ll switch it around…

- Benjamin

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAD4ANxWgXyHODdqsD%3DCzawv9D08zSPEwfUWda54kG%2B5XrHZmLA%40mail.gmail.com.

[Benjamin Schollnick]

Just an update.

Working my way through the problem, it appears that either the data is being written truncated to the database, or Pillow is generating a truncated image preview (which is being stored to the database).

The later is more likely, since I am seeing no other data corruption errors.

Rewriting the thumbnail delivery code, seemed to reduce the amount of downloading issues, but they are still not Non-Zero…  

But I may just have to live with this, since there are too many moving parts to be able to point a finger at any one bit of code.  (It could be rosetta 2 overly optimizing the code in some way…  etc.)

Plus, It might be time for me to just simply start rewriting the code from the ground up again, to better optimize and clean it up.  

- Benjamin

[Benjamin Schollnick]

Okay, a further update.

While rewrite the code, did seem to make a marginal difference, the corruption issue persisted intermittently.

Just today, I did some more digging, and I remembered that I turned on Jumbo Frames for my ethernet port on the M1 Mac Mini.  

Turning that off, at first glance, seems to have resolved the corruption issue…  

I’m unclear if this is an Django, Rosetta 2, Big Sur, or M1 Mac mini hardware issue (or a combination of factors).  

I can’t check, but I’d swear that I had Jumbo frames turned on with My Catalina 2013 iMac.  

Any ideas or suggestions?

- Benjamin

[Mike Dewhirst]

In principle, Django and Python and anything else you can program incuding the database are completely independent of the lower layers of the IP stack. 

I would restore the infrastructure to manufacturer's defaults and test again. 

Removing unknowns is always a good start in debugging.

Good luck

Mike

--

(Unsigned mail from my phone)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/C5A32221-1852-4DF5-A978-AD1334128C4E%40schollnick.net.

[Lane Campbell]

I'd put the code up on an x64 virtual machine and see if it behaves as expected. If the issues are still present then you know it's not M1 related. 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5ffa88f8.1c69fb81.3935b.52faSMTPIN_ADDED_MISSING%40gmr-mx.google.com.