Travis CI builds of machinekit-cnc

106 views
Skip to first unread message

Cameron McQuinn

unread,
Mar 21, 2020, 10:09:19 PM3/21/20
to Machinekit
What is the status of Travis CI builds for machinekit-cnc? I would love to help get this going if needed. Let me know how I can help :)

Thanks,
Cameron McQuinn

ce...@tuta.io

unread,
Mar 22, 2020, 10:00:20 AM3/22/20
to Cameron McQuinn, Machinekit
Hi,
Mar 22, 2020, 03:09 by cameron...@gmail.com:

> What is the status of Travis CI builds for machinekit-cnc? I would love to help get this going if needed. Let me know how I can help :)
>
The current status is that it is not working. And frankly have no idea if it ever worked or was just left as a something from the original split into separate repositories. I am currently trying to implement the CI/CD flow based on Github Actions. (You can read about it here: https://github.com/machinekit/machinekit-hal/issues/268 .) Main reason for this is the higher limits for Open-Source project, it is simply in the Github repository, so every fork will have it by default and Github will store the artifacts for later download.

Of course, if somebody will implement Travis CI jobs, then that's good for redundancy.

How I imagine it to work is following: The mk-cross-builder code was integrated into Machinekit-HAL repository, so in given time some job will build from it Debian images and store them in some repository (I am thinking about Github Packages as it is directly connected to Github Git repository and it is on one network with Github Actions workers). On this will be constructed the Machinekit-HAL building and packaging.

Given that Machinekit-CNC needs for its build the Machinekit-HAL (some info can be glanced from https://github.com/machinekit/machinekit-cnc/pull/93 ), I was thinking that Machinekit-CNC will add another onion layer to the Machinekit-HAL created builder images with its own build dependencies and from this get its own builder (or builder which will be capable of building both Machinekit-HAL and Machinekit-CNC).

So what needs to be done is to rework the Debian packaging in Machinekit-CNC ("debian" folder in root of the repository), build of the specific Machinekit-CNC builder (so far it can just take the Machinekit-HAL builder images from DockerHub) and the CI job.

If you have specific question, just ask. Here or on the Matrix chat.
Cern.

>
> Thanks,
> Cameron McQuinn
>
> --
> website: http://www.machinekit.io blog: http://blog.machinekit.io github: https://github.com/machinekit
> ---
> You received this message because you are subscribed to the Google Groups "Machinekit" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to machinekit+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/machinekit/0bca17f7-2840-483a-9fc1-63b5a3e642fb%40googlegroups.com.
>

Cameron McQuinn

unread,
Apr 9, 2020, 9:18:32 PM4/9/20
to Machinekit
Alright so I've been working on getting automated Docker container builds going, and I've run into a hiccup. As you mentioned, machinekit-cnc needs files from machinekit-hal to build, and the approach that I am taking to solve the problem is to build Docker containers for machinekit-cnc using the ones for machinekit-hal as a base, and installing the Debian packages generated from the mk-hal build into the mk-cnc containers. Here is where I run into a problem. The build needs header files that are provided by the machinekit-hal-dev package, which depends on machinekit-hal, but both packages provide /usr/bin/comp. When installing via apt-get install, apt detects that machinekit-hal-dev tries to overwrite /usr/bin/comp from machinekit-hal and aborts the install. This behavior can be mitigated for non sys_root environments by passing -o Dpkg::Options::="--force-overwrite" to apt. However, when those two packages are specified in a multistrap config, multistrap fails silently and does not install any of the machinekit packages.

I am not very familiar with Debian packaging, but noticed that the file debian/machinekit-hal.install in the mk-hal repo contains the line usr/bin/* which I assume sweeps up everything produced by the build under /usr/bin and puts it into the Debian package. Is there a way to exclude /usr/bin/comp from the machinekit-hal package so that we can install the machinekit-hal-dev package in multistrap systems?

You can follow along with what I have been doing at cmcquinn/machinekit-hal and cmcquinn/machinekit-cnc.

Regards,
Cameron McQuinn
> To unsubscribe from this group and stop receiving emails from it, send an email to machi...@googlegroups.com.

ce...@tuta.io

unread,
Apr 10, 2020, 7:38:08 PM4/10/20
to Cameron McQuinn, Machinekit
Hmmm,
it looks like package installation should be included in CI test run.

I am no expert at packaging, either. Debian or otherwise. (This is Zultron's area and I hope he will pop in.) What I discovered so far is that there was some discussion about functionality which would allow excluding files previously included by wildcard resolution in .install files: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=436240 , but I don't think it is currently available. (Debian developers are probably not big on set operations.)

The only thing I can come with up now, is to create script which would cure the .install files at runtime and be placed here: https://github.com/machinekit/machinekit-hal/blob/98a93c227ac2fa4e81aca40887d08b717eb926ad/debian/rules#L149 . Trying to just list all files manually fails because not all platforms create all files. (On the same note: usr/share/linuxcnc/Makefile.*) (First thing I tried: https://github.com/cerna/machinekit-hal/actions/runs/75572559 )

I think this is serious bug and should be solved (somewhat) properly. Zultron, what so you think?

BTW, didn't you want to create the build pipeline on Travis? Maybe I gave the wrong impression: The reason why I chose the Github Actions is that it can (at least I think it can) provide turn-key solution for everybody with just forking the repo on Github without the need for setting up tokens and whatnots. However, by the motto of "more stripes, more adidas" I fully support and plan to multiple CD/CI services - so when one goes under or stops the free plan, nothing really happens.

BTW, building Docker images with this logic: https://github.com/cmcquinn/machinekit-hal/blob/master/.github/workflows/docker-containers.yaml won't work because in Github Actions workflows cannot wait on each other. So the one building Docker images will run on the same time as the one building the Machinekit application. So the one building the Machinekit application will pull the older images and very probably fail. It all has to be integrated into one workflow.

Cern.


Apr 10, 2020, 03:18 by cameron...@gmail.com:

> Alright so I've been working on getting automated Docker container builds going, and I've run into a hiccup. As you mentioned, machinekit-cnc needs files from machinekit-hal to build, and the approach that I am taking to solve the problem is to build Docker containers for machinekit-cnc using the ones for machinekit-hal as a base, and installing the Debian packages generated from the mk-hal build into the mk-cnc containers. Here is where I run into a problem. The build needs header files that are provided by the machinekit-hal-dev package, which depends on machinekit-hal, but both packages provide /usr/bin/comp. When installing via apt-get install, apt detects that machinekit-hal-dev tries to overwrite /usr/bin/comp from machinekit-hal and aborts the install. This behavior can be mitigated for non sys_root environments by passing -o Dpkg::Options::="--force-overwrite" to apt. However, when those two packages are specified in a multistrap config, multistrap fails silently and does not install any of the machinekit packages.
>
> I am not very familiar with Debian packaging, but noticed that the file debian/machinekit-hal.install in the mk-hal repo contains the line usr/bin/* which I assume sweeps up everything produced by the build under /usr/bin and puts it into the Debian package. Is there a way to exclude /usr/bin/comp from the machinekit-hal package so that we can install the machinekit-hal-dev package in multistrap systems?
>
> You can follow along with what I have been doing at > cmcquinn/machinekit-hal <https://github.com/cmcquinn/machinekit-hal>>  and > cmcquinn/machinekit-cnc <https://github.com/cmcquinn/machinekit-cnc>> .
>
> Regards,
> Cameron McQuinn
>
> On Sunday, March 22, 2020 at 8:00:20 AM UTC-6, ce...@tuta.io wrote:
>
>> Hi,
>> Mar 22, 2020, 03:09 by >> cameron...@gmail.com <>>> :
>>
>> > What is the status of Travis CI builds for machinekit-cnc? I would love to help get this going if needed. Let me know how I can help :)
>> >
>> The current status is that it is not working. And frankly have no idea if it ever worked or was just left as a something from the original split into separate repositories. I am currently trying to implement the CI/CD flow based on Github Actions. (You can read about it here: >> https://github.com/machinekit/>> machinekit-hal/issues/268>> .) Main reason for this is the higher limits for Open-Source project, it is simply in the Github repository, so every fork will have it by default and Github will store the artifacts for later download.
>>
>> Of course, if somebody will implement Travis CI jobs, then that's good for redundancy.
>>
>> How I imagine it to work is following: The mk-cross-builder code was integrated into Machinekit-HAL repository, so in given time some job will build from it Debian images and store them in some repository (I am thinking about Github Packages as it is directly connected to Github Git repository and it is on one network with Github Actions workers). On this will be constructed the Machinekit-HAL building and packaging.
>>
>> Given that Machinekit-CNC needs for its build the Machinekit-HAL (some info can be glanced from >> https://github.com/machinekit/>> machinekit-cnc/pull/93>> ), I was thinking that Machinekit-CNC will add another onion layer to the Machinekit-HAL created builder images with its own build dependencies and from this get its own builder (or builder which will be capable of building both Machinekit-HAL and Machinekit-CNC).
>>
>> So what needs to be done is to rework the Debian packaging in Machinekit-CNC ("debian" folder in root of the repository), build of the specific Machinekit-CNC builder (so far it can just take the Machinekit-HAL builder images from DockerHub) and the CI job.
>>
>> If you have specific question, just ask. Here or on the Matrix chat.
>> Cern.
>>
>> >
>> > Thanks,
>> > Cameron McQuinn
>> >
>> > --
>> > website: >> http://www.machinekit.io>> blog: >> http://blog.machinekit.io>> github: >> https://github.com/machinekit>>
>> > ---
>> > You received this message because you are subscribed to the Google Groups "Machinekit" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an email to >> machi...@>> googlegroups.com <>>> .
>> > To view this discussion on the web visit >> https://groups.google.com/d/>> msgid/machinekit/0bca17f7->> 2840-483a-9fc1-63b5a3e642fb%>> 40googlegroups.com>> .
>> >
>>
>>
>
>
>
> --
> website: > http://www.machinekit.io> blog: > http://blog.machinekit.io> github: > https://github.com/machinekit
> ---
> You received this message because you are subscribed to the Google Groups "Machinekit" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to > machinekit+...@googlegroups.com> .
> To view this discussion on the web visit > https://groups.google.com/d/msgid/machinekit/d4a6d775-cb1c-4556-9cd2-807cbde080c7%40googlegroups.com <https://groups.google.com/d/msgid/machinekit/d4a6d775-cb1c-4556-9cd2-807cbde080c7%40googlegroups.com?utm_medium=email&utm_source=footer>> .
>

ce...@tuta.io

unread,
Apr 11, 2020, 5:02:39 PM4/11/20
to Cern, Cameron McQuinn, Machinekit
Hi,
I was somewhat successful in solving this issue. Description on Github: https://github.com/machinekit/machinekit-hal/issues/271

Cern.


Apr 11, 2020, 01:38 by machi...@googlegroups.com:

> Hmmm,
> it looks like package installation should be included in CI test run.
>
> I am no expert at packaging, either. Debian or otherwise. (This is Zultron's area and I hope he will pop in.) What I discovered so far is that there was some discussion about functionality which would allow excluding files previously included by wildcard resolution in .install files: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=436240 , but I don't think it is currently available. (Debian developers are probably not big on set operations.)
>
> The only thing I can come with up now, is to create script which would cure the .install files at runtime and be placed here: https://github.com/machinekit/machinekit-hal/blob/98a93c227ac2fa4e81aca40887d08b717eb926ad/debian/rules#L149 . Trying to just list all files manually fails because not all platforms create all files. (On the same note: usr/share/linuxcnc/Makefile.*) (First thing I tried: https://github.com/cerna/machinekit-hal/actions/runs/75572559 )
>
> I think this is serious bug and should be solved (somewhat) properly. Zultron, what so you think?
>
> BTW, didn't you want to create the build pipeline on Travis? Maybe I gave the wrong impression: The reason why I chose the Github Actions is that it can (at least I think it can) provide turn-key solution for everybody with just forking the repo on Github without the need for setting up tokens and whatnots. However, by the motto of "more stripes, more adidas" I fully support and plan to multiple CD/CI services - so when one goes under or stops the free plan, nothing really happens.
>
> BTW, building Docker images with this logic: https://github.com/cmcquinn/machinekit-hal/blob/master/.github/workflows/docker-containers.yaml won't work because in Github Actions workflows cannot wait on each other. So the one building Docker images will run on the same time as the one building the Machinekit application. So the one building the Machinekit application will pull the older images and very probably fail. It all has to be integrated into one workflow.
>
> Cern.
>
>
> Apr 10, 2020, 03:18 by cameron...@gmail.com:
>
>> Alright so I've been working on getting automated Docker container builds going, and I've run into a hiccup. As you mentioned, machinekit-cnc needs files from machinekit-hal to build, and the approach that I am taking to solve the problem is to build Docker containers for machinekit-cnc using the ones for machinekit-hal as a base, and installing the Debian packages generated from the mk-hal build into the mk-cnc containers. Here is where I run into a problem. The build needs header files that are provided by the machinekit-hal-dev package, which depends on machinekit-hal, but both packages provide /usr/bin/comp. When installing via apt-get install, apt detects that machinekit-hal-dev tries to overwrite /usr/bin/comp from machinekit-hal and aborts the install. This behavior can be mitigated for non sys_root environments by passing -o Dpkg::Options::="--force-overwrite" to apt. However, when those two packages are specified in a multistrap config, multistrap fails silently and does not install any of the machinekit packages.
>>
>> I am not very familiar with Debian packaging, but noticed that the file debian/machinekit-hal.install in the mk-hal repo contains the line usr/bin/* which I assume sweeps up everything produced by the build under /usr/bin and puts it into the Debian package. Is there a way to exclude /usr/bin/comp from the machinekit-hal package so that we can install the machinekit-hal-dev package in multistrap systems?
>>
>> You can follow along with what I have been doing at > cmcquinn/machinekit-hal <https://github.com/cmcquinn/machinekit-hal>>  and > cmcquinn/machinekit-cnc <https://github.com/cmcquinn/machinekit-cnc>> .
>>
>> Regards,
>> Cameron McQuinn
>>
>> On Sunday, March 22, 2020 at 8:00:20 AM UTC-6, ce...@tuta.io wrote:
>>
>>> Hi,
>>> Mar 22, 2020, 03:09 by >> cameron...@gmail.com <>>> :
>>>
>>> > What is the status of Travis CI builds for machinekit-cnc? I would love to help get this going if needed. Let me know how I can help :)
>>> >
>>> The current status is that it is not working. And frankly have no idea if it ever worked or was just left as a something from the original split into separate repositories. I am currently trying to implement the CI/CD flow based on Github Actions. (You can read about it here: >> https://github.com/machinekit/>> machinekit-hal/issues/268>> .) Main reason for this is the higher limits for Open-Source project, it is simply in the Github repository, so every fork will have it by default and Github will store the artifacts for later download.
>>>
>>> Of course, if somebody will implement Travis CI jobs, then that's good for redundancy.
>>>
>>> How I imagine it to work is following: The mk-cross-builder code was integrated into Machinekit-HAL repository, so in given time some job will build from it Debian images and store them in some repository (I am thinking about Github Packages as it is directly connected to Github Git repository and it is on one network with Github Actions workers). On this will be constructed the Machinekit-HAL building and packaging.
>>>
>>> Given that Machinekit-CNC needs for its build the Machinekit-HAL (some info can be glanced from >> https://github.com/machinekit/>> machinekit-cnc/pull/93>> ), I was thinking that Machinekit-CNC will add another onion layer to the Machinekit-HAL created builder images with its own build dependencies and from this get its own builder (or builder which will be capable of building both Machinekit-HAL and Machinekit-CNC).
>>>
>>> So what needs to be done is to rework the Debian packaging in Machinekit-CNC ("debian" folder in root of the repository), build of the specific Machinekit-CNC builder (so far it can just take the Machinekit-HAL builder images from DockerHub) and the CI job.
>>>
>>> If you have specific question, just ask. Here or on the Matrix chat.
>>> Cern.
>>>
>>> >
>>> > Thanks,
>>> > Cameron McQuinn
>>> >
>>> > --
>>> > website: >> http://www.machinekit.io>> blog: >> http://blog.machinekit.io>> github: >> https://github.com/machinekit>>
>>> > ---
>>> > You received this message because you are subscribed to the Google Groups "Machinekit" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send an email to >> machi...@>> googlegroups.com <>>> .
>>> > To view this discussion on the web visit >> https://groups.google.com/d/>> msgid/machinekit/0bca17f7->> 2840-483a-9fc1-63b5a3e642fb%>> 40googlegroups.com>> .
>>> >
>>>
>>>
>>
>>
>>
>> --
>> website: > http://www.machinekit.io> blog: > http://blog.machinekit.io> github: > https://github.com/machinekit
>> ---
>> You received this message because you are subscribed to the Google Groups "Machinekit" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to > machinekit+...@googlegroups.com> .
>> To view this discussion on the web visit > https://groups.google.com/d/msgid/machinekit/d4a6d775-cb1c-4556-9cd2-807cbde080c7%40googlegroups.com <https://groups.google.com/d/msgid/machinekit/d4a6d775-cb1c-4556-9cd2-807cbde080c7%40googlegroups.com?utm_medium=email&utm_source=footer>> .
>>
>
> --
> website: http://www.machinekit.io blog: http://blog.machinekit.io github: https://github.com/machinekit
> ---
> You received this message because you are subscribed to the Google Groups "Machinekit" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to machinekit+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/machinekit/M4aSH6m--3-2%40tuta.io.
>

Reply all
Reply to author
Forward
0 new messages