Proposal: Add builds.txt to github.com/elixir-lang/elixir

[Wojtek Mach]

For tools like asdf, setup-beam, and similar it is very useful to get a list of all Elixir releases (included per-OTP builds). Doing this through GitHub REST API can be slightly complicated due to dealing with JSON, pagination, and rate-limiting.

Hex/Bob keeps a https://builds.hex.pm/builds/elixir/builds.txt which has none of these problems.

Since a few release ago, Elixir build are available on Elixir GitHub release. And so to keep similar things together, I'd like to propose adding something like (or exactly) such builds.txt file in the GitHub repository. This way we could, although there are no such plans at the moment, deprecate Hex/Bob-maintainted Elixir builds.

With this proposal, the way to get all Elixir builds could be simply:

    $ curl -fL https://github.com/hexpm/bob/raw/main/builds.txt

Since Elixir 1.14, the builds in GitHub releases follow the pattern elixir-otp-VSN.zip. Prior to that it was Precompiled.zip, compiled against the oldest supported OTP version for that Elixir release (https://hexdocs.pm/elixir/compatibility-and-deprecations.html#between-elixir-and-erlang-otp). I'd still add them to the builds file for informational purposes but I'm OK with not doing that either.

We could just adopt the Hex builds.txt format, here's an example:

v1.17.2 47abe2d107e654ccede845356773bcf6e11ef7cb 2024-07-06T21:18:45Z c1a0860b805ef4925ca9e97ecf9f84b3d3fc62e6a48a91c1cbe6344c3b02400b
v1.17.2-otp-25 47abe2d107e654ccede845356773bcf6e11ef7cb 2024-07-06T21:18:45Z c1a0860b805ef4925ca9e97ecf9f84b3d3fc62e6a48a91c1cbe6344c3b02400b
v1.17.2-otp-26 47abe2d107e654ccede845356773bcf6e11ef7cb 2024-07-06T21:20:26Z a2e32bf61a8dd1354ce41bed891cee72e02cbf5b4532fa190416547f422a97b4
v1.17.2-otp-27 47abe2d107e654ccede845356773bcf6e11ef7cb 2024-07-06T21:23:11Z 4e32e8761bd73916feaa24eaa98d86b1d929039e38e778b37553b0330f75b9f6

This way existing tools can easily switch to it. Since it'd be a new URL though, we have a chance to potentially improve it. Here's one idea which I believe will make it super clear what's what:

<ref_name>[-otp-VSN] <date> git:<ref> sha256:<sha>

e.g.:

v1.17.2-otp-27 2024-07-06T21:23:11Z git:47abe2d107e654ccede845356773bcf6e11ef7cb  sha256:4e32e8761bd73916feaa24eaa98d86b1d929039e38e778b37553b0330f75b9f6

(See https://github.com/hexpm/bob/pull/188#discussion_r1701912589)

My proposal is to automate this on GitHub Actions, similar to https://github.com/livebook-dev/livebook/blob/v0.14.0/.github/workflows/assets.yml#L52:L57. (It is a bit annoying though because we can't easily make sure only one GitHub job of this type runs at a time AND there could be other commits happening on main in the meantime. The best I could come up with is: https://github.com/wojtekmach/otp_builds/blob/2209f680a2f8d8e50c73509790bac565ef65b1d3/.github/workflows/update_builds.yml#L30:L67.)

[Benjamin Philip]

Reposting on googlegroups since I'm unsure if my reply was received when replied via email.

>  My proposal is to automate this on GitHub Actions, similar to https://github.com/livebook-dev/livebook/blob/v0.14.0/.github/workflows/assets.yml#L52:L57. (It is a bit annoying though because we can't easily make sure only one GitHub job of this type runs at a time AND there could be other commits happening on main in the meantime. The best I could come up with is: https://github.com/wojtekmach/otp_builds/blob/2209f680a2f8d8e50c73509790bac565ef65b1d3/.github/workflows/update_builds.yml#L30:L67.)

Why commit builds.txt to main at all? An alternate solution would be to host it on GitHub Pages, maybe at https://hexpm.github.io/bob? `builds.txt` could be deployed to https://hexpm.github.io/bob/main/builds.txt as well as
https://hexpm.github.io/bob/<ref>/builds.txt on push. Deploying a website to multiple urls is made trivial with the deploy-pages action.

This would have a number of advantages over your proposed solution:

1. Concurrent runs are not a concern - each commit is deployed to its own url for reference, and the latest of the actions is deployed to the bob/main url.
2. We do not have bot accounts committing to main, convoluting the git history. A bot account may also have security problems
3. A workflow like this would be far easier to maintain since similar workflows are fairly common (so docs, guides etc. may exist). Plus GitHub does most of the heavy lifting.

A potential concern is if the artifact for the site will expire. Looking through the upload-pages-artifact action docs, there are no expiries as of right now. However, we are at risk of GitHub deciding to expire old artifacts in the future.

-- bp

[Wojtek Mach]

I agree committing builds.txt to the source git repository is pretty unorthodox. Perhaps it’s a security malpractice to keep source and build artifacts derived from that source code; in that case, please disregard this. I thought it’s straightforward enough as opposed to relying on Hex S3 bucket for this and while at it solve https://github.com/erlef/setup-beam/issues/260. I did not know about https://github.com/actions/upload-pages-artifact, this is pretty interesting! I couldn’t find much documentation about it besides the action itself and to me it feels a bit like implementation detail of GitHub Pages. For example, do you know if there’s a programatic way to manually upload things without relying on actions/upload-pages-artifact, you know, `curl -X POST ...` or `gh release upload … $ref $file`? If there isn’t that seems like a pretty big vendor lock-in (we’re already pretty locked in to GitHub but I like to think it’s pretty portable... maybe?)

-- 
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/cd4e0dbb-d338-445c-b4fd-8bb2b4d55520n%40googlegroups.com.

[Benjamin Philip]

> I agree committing builds.txt to the source git repository is pretty unorthodox. Perhaps it’s a security malpractice to keep source and build artifacts derived from that source code;

If having the text available in source was necessary, I would use a git hook. That would address the security implications of a bot account. In this scenario, I see no need to have the list in source.

> do you know if there’s a programatic way to manually upload things without relying on actions/upload-pages-artifact

actions/upload-pages-artifact is implemented on top of actions/upload-artifact (https://github.com/actions/upload-artifact), which in turn uses this js lib from GitHub: https://github.com/actions/toolkit/tree/main/packages/artifact. So in theory you could write a js script to upload things manually.

Keep in mind that uploading the artifact is only one step of the process. You also need to deploy the page with actions/deploy-pages, which as far as I can tell, doesn't have its own API out of actions yet. I suspect you could reverse engineer something in bash from the source, but I feel that it will be pretty painful.

>  If there isn’t that seems like a pretty big vendor lock-in (we’re already pretty locked in to GitHub but I like to think it’s pretty portable... maybe?)

GitLab has its own equivalent: https://docs.gitlab.com/ee/user/project/pages/ which is a lot easier to use since they are far more open company. GitLab Pages is also supported on self-hosted instances: https://docs.gitlab.com/ee/administration/pages/source.html which gives us an extra degree of control. If at all we were to leave GitHub this could be very easily ported to GitLab. I think GitLab is a good enough solution in case things go south (I don't see us moving to cgit any time soon).

However, porting previous previous artifacts is non-trivial, so we may loose our history unless we sit down and download every version from GH.

-- bp

[Wojtek Mach]

Hey everyone,

I'd like to withdraw this proposal. We have something that already works, https://builds.hex.pm/builds/elixir/builds.txt, and per discussion above the implementation I am proposing doesn't seem like an improvement and it's probably a downgrade.

I have updated elixir-lang.org to mention builds.txt: https://github.com/elixir-lang/elixir-lang.github.com/pull/1769/files

Thanks Benjamin in particular!