Autogenerating package_index.json files (e.g. from Github Actions?)

[Matthijs Kooijman]

Hey folks,

I'm managing a few third-party Arduino cores, which I'm publishing
through a package_index.json file/URL. This works, but involves some
handwork: In addition to tagging a release in the core repository, I
also have to update the json file.

I'm considering to automate this, using a Github Action that
runs on every tag and automatically adds an entry to an existing json
file (in another repository or branch).


However, before I spend another day or two on a side project like this,
I want to ask: Does anyone know of any existing, reusable scripts for
this?


I looked around just a little bit and it *seems* like most people just
do this manually. I did find a workflow in the
https://github.com/esp8266/Arduino repository that does this, but that
looks like a fairly complex multi-script system that also generates the
zipfile along with some preprocessing and is generally a lot more
complex than I need.

I'm looking for something simple, really. Here's what I had in mind:
- I'm using github tar.gz urls for the actual platform download, which
just download a on-demand generated tarball based on a git tag (so no
need to actually generate a tarball and upload it somewhere).
- I'm thinking to store a small template json file along with the
actual core (so inside the tagged commit), which defines just a
single entry in the "platforms" array. It contains the manual fields,
like "name", "architecture" or "toolsDependencies".
- The workflow should run whenever a tag is created in the repository.
- It will then generate the download url based on the tag name, figure
out the filesize and checksum and insert all that into the template
platform entry.
- I'm storing the final json file in a separate branch, so the script
should get the current json file from that branch, merge the new
platform entry in it and commit the resulting tarball into this
branch.
- This approach makes it easy to manually fix or update any existing
entries, but still have things work automatically by default.
- Alternatively, the template json file could be stored along with the
generated final json file, but that makes things a bit more complex
when using multiple architectures in a single json file.

Gr.

Matthijs

[Alessandro Ranellucci]

Hey Matthijs :)

Would something like this work for you? It doesn’t include the git commands but maybe it’s a starter.

TAG=$(git describe --exact-match --tags)

FILE="esp8266-$TAG.zip"

PLATFORM=$(jq --arg checksum "SHA-256:$(shasum -a 256 -b "$FILE" | awk '{print $1}')" \

   --arg version "$TAG" \

   --arg size $(wc -c <"$FILE") \

   --arg url "https://github.com/esp8266/Arduino/releases/download/$TAG/$FILE" \

   --arg archiveFileName "$FILE" \

   '.version = $version | .size = $size | .checksum = $checksum | .url = $url | .archiveFileName = $archiveFileName' \

   platform_template.json)

jq --argjson p "$PLATFORM" \

   '.packages[] | select(.name == "esp8266") | .platforms += [$p]' \

   package_index.json > new_package_index.json

 - Alessandro

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.
To view this discussion on the web visit https://groups.google.com/a/arduino.cc/d/msgid/developers/20201106133745.GQ2215%40login.tika.stderr.nl.

[Owen Lyke]

Matthijs,

I am also interested in a tool like the one you are describing (to help maintain the Apollo3 core). A lot of the functionality you describe would work for us too so perhaps there could be some collaboration between us. One thing that I'd like to add is the ability to generate the archive / tarball files and upload them as assets to a GitHub release tag. Currently the files provided by GitHub do not include the contents of submodules which is problematic for us.

Please keep us informed of your next steps.

Regards,

Owen L.

To view this discussion on the web visit https://groups.google.com/a/arduino.cc/d/msgid/developers/74f474e2-2382-454d-8eec-afbbb172a1c8%40Spark.

--

Owen Lyke

[Alexander Christian]

It's not yet perfect, but it works:

https://gitlab.com/konnekting/KonnektingArduinoCore-samd

There's not manual work requiz. Update code, push to girls, wait a few moments. That's all, URL is updated.

Regards,
Alex

[MCUdude]

This is the script I'm using for all my AVR based Arduino cores. It's not perfect but it works.

https://github.com/MCUdude/MightyCore/blob/gh-pages/Boards_manager_release.sh

All I have to do is to publish a release, run git pull, execute the script and commit the new boards manager release.

[Matthijs Kooijman]

Hey folks,

thanks for all replies, nice to hear that I'm not the only one
struggling with this.

Good to see that there are already some tools out there, let me respond
a little here.

On Tue, Dec 15, 2020 at 12:25:51AM -0800, MCUdude wrote:
> This is the script I'm using for all my AVR based Arduino cores. It's not
> perfect but it works.
> https://github.com/MCUdude/MightyCore/blob/gh-pages/Boards_manager_release.sh

This looks nice to automate the generation of a tarball and the JSON,
but I would like to also automate the publishing of the JSON (and I do
not really need to generate a tarball, I'm just planning to github).
Still, seems like there's some useful bits in here to reuse.

Also, the jq tool seems nice. I was planning to write something in
Python (because that's what I like and using bash often gets messy
quickly), but maybe just keeping it simple with bash is the way to go
here.

fredag 6. november 2020 kl. 21:22:18 UTC+1 skrev Alexander Christian:

> It's not yet perfect, but it works:
>
> https://gitlab.com/konnekting/KonnektingArduinoCore-samd
>
> There's not manual work requiz. Update code, push to girls, wait a few
> moments. That's all, URL is updated.

This indeed looks like what I'd want, except it's for gitlab, uses ant
(bit cumbersome for me personally) and it replaces the existing release
rather than add to it (AFAICS). I do totally like the "json template"
approach you used, though.

Am 6. November 2020 18:58:53 MEZ schrieb 'Owen Lyke' via Developers

> One thing that I'd like to add is the ability to generate the archive
> / tarball files and upload them as assets to a GitHub release tag.
> Currently the files provided by GitHub do not include the contents of
> submodules which is problematic for us.

This is not at all relevent for me, but if I write something, I'll see
if I can keep this in mind (i.e. not make it harder than necessary to
add this later).

On Fri, Nov 6, 2020 at 10:01 AM Alessandro Ranellucci wrote:
> Would something like this work for you? It doesn’t include the git
> commands but maybe it’s a starter.

Indeed looks like something I could use. It's not a full templated
solution, but it shows how to use jq to just override specific values,
which might be more than enough to insert autogenerated values (and
saves a lot of trouble with escaping etc.).



So, thanks again for your input. Since there's nothing that fully fits
my need, I'm planning to write a github action myself, reusing parts of
all your responses where appropriate.

I think that, at least initially, I'm going to just write this as a
workflow that lives completely in my repo itself, to just see how it
works. If others want to add anything, they can just copy my workflow
and do the same. At a later moment, when things have stabilized a bit
more, I guess this workflow could be moved into its own repository, so
it can be included from different places.

AFAIU github has recently introduced "composite" actions, so the same
steps you use inside the workflow itself can now also be used as part of
a reusable action, which makes it easier to make this change (previously
reusable actions could only be javascript code or a shell script inside
docker).

I'll keep you updated of my progress!

Gr.

Matthijs

[Matthijs Kooijman]

Hey all,

so I wrote a github workflow:

https://github.com/meetjestad/mjs_boards/blob/avr/.github/workflows/publish-releases.yml

As I said before, this is just a workflow (embedded completely into the
repository, as opposed to a reusable action that lives in another repo
and is called with arguments) which is geared to my particular usecase
(using Github generated tarballs without any processing, storing the
JSON in another branch of the same repository), but I've tried to write
the workflow in a clean way to make it easier to reuse later.

Some thoughts:

- As suggested by others, I used shell scripts (embedded into the
workflow) and the `jq` tool for JSON processing. I'm pleasantly
surprised by `jq`, it looks like a solid and very complete tool for
this kind of thing.

- I put some configuration variables at the top, to make it easier to
modify the workflow for others, as well as make the code easier to
read. These variables can become parameters in a reusable action
later.

- I split the action into a "Generate tarball and download URL" step
and a "Generate and commit JSON" step, so the latter can (hopefully)
remain unchanged for a usecase where some postprocessing and/or
uploading of the tarball to a github release or other webhosting is
needed.

- The JSON template to be added is stored in the tagged commit, next to
the the workflow file (but this is configurable). E.g.:
https://github.com/meetjestad/mjs_boards/blob/avr/.github/workflows/platform_template.json

Here, I added empty values for the autogenerated fields. This is not
strictly needed (if they are omitted, they are added at the end), but
this gives a nicer order of fields in the resulting JSON file.

- Here's an example of a generated commit:
https://github.com/meetjestad/mjs_boards/commit/e6692ae9a2f3c1babf7327333d6ad0fc377f5ecb

And here's the build output that generated that commit:
https://github.com/meetjestad/mjs_boards/runs/1569701635

- The workflow currently runs whenever pushing a tag, but this could
easily be changed to run whenever building a release (might need
changes to get the tag name in another way, but it likely just works
as-is).

- When pushing a tag for a release that already exists in the JSON
(e.g. when overwriting a tag maybe), the script now bails out, to
prevent accidentally changing an existing release. If you do want to
change it, you can manually remove the old version and then tell
Github to re-run the job (in the webinterface).

- The version number to be put into the JSON file is now taken from the
platform.txt file. There is a sanity check that checks if the tag
name *ends with* the version from platform.txt. This seemed like a
good compromise to do some checking without hardcoding too much a
about the format of the actual tag names.

@Owen, you said you would be interested in such a script, but needed to
do some post-processing of the tarball and upload it to a GH release.
Maybe you could use my workflow as a basis and add the things you need?

And maybe others that have similar needs could do the same? If we keep
each other posted of progress and additional features needed, maybe in a
while we could turn our separate workflows into a reusable action (that
we could maybe collectively maintain)?

Gr.

Matthijs