Looking for advice on AutoPkg recipes for FileZilla

[John Lockwood]

I had previously written AutoPkg download and Munki recipes to handle both the Intel and ARM versions of FileZilla, these both previously worked.

The FileZilla website has recently made a change that means the previous standardised download URL no longer works.

The previous download URL that worked would look like this -

https://download.filezilla-project.org/client/FileZilla_%version%_macos-arm64.app.tar.bz2

This form no longer works and instead the following URL format has to be used -

https://dl1.cdn.filezilla-project.org/client/FileZilla_%version%_macos-arm64.app.tar.bz2?h=LTBBkWEMXA8gng4nKDg9eg&x=1721671357

The two main differences are -

1. A different download server is defined in the URL and this is dynamic meaning it changes to enable I presume load sharing

2. A unique token is added to the end of the URL, without this the download now fails and you are redirected back to the main download page

I have successfully created a new working download recipe for the Intel version by adding additional URLTextSearcher processor entries so I now search for and find the Version, the Server number and the Token. As mentioned this works for the Intel version of my recipe, I use all three as variables to generate the actual download URL.

One would think it would then be a simple matter of substituting arm64 instead of x86 in the relevant lines of the recipe. Using regex101.com to test suggests this should indeed be true.

However if I do this then the otherwise identical URLTextSearcher processor code fails to match for arm64 and therefore the download recipe fails.

'Cheating' and using x86 purely for the URLTextSearcher in my Arm recipe gets past those errors but the URLDownloader processor fails even though it has the apparent correct URL as per the receipt plist.

I wondered if it might be down to the URLAgent string I was using which told the FileZilla website I was a Mac and originally both recipes had Intel in the agent string but it turns out that web servers seem to ignore the processor part of the agent string and still treat all Macs regardless as Intel. (Tested by using Safari in developer mode and adjusting the agent string and the default page defaults to Intel regardless.) It is possible to download manually in a browser both ARM and Intel versions so the agent field is not blocking that.

I now use a download page which lists all available downloads including Windows and using the URLTextSearcher find and construct the correct Mac URLs.

https://dl%dlserver%.cdn.filezilla-project.org/client/FileZilla_%version%_macos-arm64.app.tar.bz2%token%

My recipes are available via the following URLs, suggestions as to what I am presumable doing wrong are much appreciated as this helps me learn for next time.

https://github.com/jelockwood/recipes/blob/master/FileZilla-Intel/FileZilla-Intel.download.recipe <--working

https://github.com/jelockwood/recipes/blob/master/FileZilla-Arm/FileZilla-Arm.download.recipe <--not-working

[Anthony Reimer]

You may be over-engineering this. The vendor has a Download page with links to both downloads:

https://filezilla-project.org/download.php?show_all=1

I would solve this with a single URLTextSearcher call with the above as the argument for url and the following string for re_pattern:

https://dl\d\.cdn\.filezilla-project\.org/client/FileZilla_[\d.]+_macos-%ARCHITECTURE%\.[^"]+

where ARCHITECTURE is an Input Variable that would be either x86 or arm64. Invoking URLDownloader will then grab the appropriate download depending on the architecture specified in your override. (I've assumed that the opening of the URL will be dl and a single digit, but you could make the regex more robust than that if you preferred.) I have not tested this in a recipe, but the regex does find the correct link.

Hope that helps.

Anthony Reimer



> On Jul 22, 2024, at 11:29 AM, John Lockwood <jeloc...@gmail.com> wrote:
>
> [△EXTERNAL]

> --
> You received this message because you are subscribed to the Google Groups "autopkg-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to autopkg-discu...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/autopkg-discuss/14671839-519f-48f9-aa3a-8bbd76aa8a7cn%40googlegroups.com.

[John Lockwood]

Thanks for your suggestion Anthony,

I agree your approach should work and is simpler. I also agree your regex will match the desired string.

I have therefore built a new recipe using your approach with a single URLTextSearcher unfortunately it failed in the same way as my previous effort. However this time I paid more attention to the error which was the same as before and is -

No match found on URL: https://filezilla-project.org/download.php?type=client Failed.

Doh! 

Whilst my Download and Munki recipes do (and did) use the new correct URL, my override recipe was unchanged and merely had the trust updated and buried inside it was the old URL. It therefore was loading the wrong page which only had a single default Intel for Mac URL. Hence it would work for Intel but not for Arm. 

I have generated a new override recipe and it now works fine including with your suggested improvement.

Many thanks!

[John Lockwood]

I have had a bit of a break running Autopkg/Munki and just come back to it and found an issue with my previous FileZilla recipe. It is now giving the same previous error of -

No match found on URL: https://filezilla-project.org/download.php?type=client Failed.

When using a Mac web browser to view the page in normal GUI mode I see it lists the two expected links but when viewing 'page source' for the same page/tab it does not contain the URLs. I suspect the URLs are being dynamically generated via javascript and rendered in Safari as text but that both view page source and more importantly Autopkg do not render javascript and hence cannot find the URLs.

My currently broken but previously working recipe is here -

https://github.com/jelockwood/recipes/tree/master/FileZilla-Arm

Any suggestions from the audience?

Many thanks.

[Gregory Neagle]

I think most of use gave up on FileZilla since they hid the links behind JavaScript.

-Greg

To view this discussion visit https://groups.google.com/d/msgid/autopkg-discuss/56821495-04ef-4a69-a79b-1f2a2e2a900bn%40googlegroups.com.

[Elliot Jordan]

This one, which requires manually adding Python's cryptography module for its custom URLProvider, appears to work:

https://github.com/autopkg/wycomco-recipes/blob/master/FileZilla/Filezilla.download.recipe

Elliot

To view this discussion visit https://groups.google.com/d/msgid/autopkg-discuss/9533A188-4785-4FB7-BEB9-73FBC6108C5F%40mac.com.

[Gregory Neagle]

I salute the amount of effort that went into that...

To view this discussion visit https://groups.google.com/d/msgid/autopkg-discuss/CADefdwiwjJ8DQLk5kemjaNBVRO6%2BETWK%2BcEnjt8_YN6zyctSNw%40mail.gmail.com.