Classic packaging deprecation

[mwilhite]

Previously, we had communicated that the 3006 release would include both classic and onedir packages. In an effort to release quicker and migrate to only onedir packages quicker, we are going to stop building classic packages in 3006, and only include onedir packages. The 3005 release will be the final release where both classic and onedir packages will be built. One issue that has delayed releases in the past is the classic package builds are not fully automated. The onedir packages have a pipeline that is fully automated, which allows us to release quicker. If issues are found with the onedir packaging that prevents someone from upgrading to 3006 we can quickly release fixes for the packaging. In the next week, we will work on updating the release notes to mirror this change.

[John Nielsen]

This message raises more questions for me than it has answers. Where/how was this communicated previously? What will be the impact of this change? Will there still be e.g. .deb packages and repositories? What will upgrading to 3006 look like?

On Aug 3, 2022, at 7:43 AM, 'mwilhite' via Salt-users <salt-...@googlegroups.com> wrote:

Previously, we had communicated that the 3006 release would include both classic and onedir packages. In an effort to release quicker and migrate to only onedir packages quicker, we are going to stop building classic packages in 3006, and only include onedir packages. The 3005 release will be the final release where both classic and onedir packages will be built. One issue that has delayed releases in the past is the classic package builds are not fully automated. The onedir packages have a pipeline that is fully automated, which allows us to release quicker. If issues are found with the onedir packaging that prevents someone from upgrading to 3006 we can quickly release fixes for the packaging. In the next week, we will work on updating the release notes to mirror this change.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/6f67bc2e-4dc6-4f81-a288-5188d3879ec7n%40googlegroups.com.

[Robby Callicotte]

On Wednesday, August 3, 2022 1:26:41 PM CDT John Nielsen wrote:
> This message raises more questions for me than it has answers. Where/how was
> this communicated previously?

I can not find any announcement in the Salt Users mailing list regarding
Tiamat/Onedir, but I did find out about it during one of the Saltstack Open
Community Hour meetings. This was about a month ago.

> What will be the impact of this change? Will
> there still be e.g. .deb packages and repositories? What will upgrading to
> 3006 look like?

As far as I know, Salt will provide the Onedir based packaging via their
repos. Several OS distros are still packaging Salt via "classic" methods
(distro rpm/deb/ports etc).
--
Robby Callicotte
He/His
Timezone: US/Central

[Vaarlion]

Can we get a link, a blog post, a doc, anything about this ?
google only show 2 link which both are "WIP" runs

1. [WIP] Onedir build · saltstack/salt@1dc54b8 - GitHub
2. [WIP] Onedir build · saltstack/salt@907316b - GitHub

Onedir return nothing in the doc : https://docs.saltproject.io/en/latest/

and if it's supposed to be the "single binary" tab in beta ... WTF ?
https://repo.saltproject.io/#single-binary

From the site:

>All tools, from the salt repo, are included in the single-binary. Though, because they are contained within a single binary, they must be called differently.

>Example:

```# Make use of salt-call sudo ./salt call --local test.versions```

Available tools:

• salt master (similar to salt-master)
• salt minion (similar to salt-minion)
• salt call (similar to salt-call)
• salt ssh (similar to salt-ssh)
• salt syndic (similar to salt-syndic)
• salt cloud (similar to salt-cloud)
• salt api (similar to salt-api)
• salt pip

---
I see only drawback, this move make no sense.

1. You will be breaking every automation that isn't based on python but the binary file name, and every short hand any sys in any team ever made.
   
2. You will make it way more complicated to keep a huge park updated and iso. Do i have to download a hash file from you for every run to check if i have to download something ?
   
3. The installation will require way more step, meaning bootstrap will be slower (creating every folder by hand and copying default config file to respect the Filesystem Hierarchy Standard without salt installed yet)
4. The security is greatly diminished because GPG check will be fully optional where they are mandatory on most distro package manager. Most people won't do it in there bootstrap script...
5. It won't be possible anymore to limit what is installed on the device, and every installation of salt will come master capable, syndic capable, cloud capable...
6. What about the service file ? No longer maintained ? Not everyone is well versed in writing systemd, initd, or what ever service manager is on there os.
7. What about package requirement ? do you plan on integrating every low level library needed ? or will user have to figure it out like when they have to compile software ? for debian right now you need lsb-base (>= 3.0-6), bsdmainutils, dctrl-tools...
8. What about MAN page ?
9. Should i go on ?
   

> One issue that has delayed releases in the past is the classic package builds are not fully automated

If this is the one issue that push you to go the way google and hashicorp have gone... you really need to review your priority.

• It's not complicated to automate package creation, most company do it for a couple, and it's possible of all of the one you provide.
• Once it's automated, it usually require very little maintenance as packaging system rarely change.
• And you are no-longer a small, poorly funded team working on tiny project.
  

Single package software make sens when you are making a ~dirty~, short lived, minimal image for cloud based application. something salt isn't made for at all since it's a tool to keep a huge amount of physical and visualize server safe and ISO during there very long life in service deployment.

You failed at https://saltdocs.prodcamp.com/6
And with this move you are failing at https://saltdocs.prodcamp.com/3

Maybe it's time to share a roadmap ... You remember you have a blog right ? https://saltproject.io/blog/ Because right now, i can't see where you are heading ...

[mwilhite]

This was previously communicated in open hour, the release notes for the upcoming 3005 release notes (https://docs.saltproject.io/en/master/topics/releases/3005.html) and the new install guide https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade-to-onedir.html. The 3005 release notes do not reflect the previous notice so you can see this PR https://github.com/saltstack/salt/pull/61930 for the original statement surrounding 3007 versus 3006. It has since been updated to note that 3006 is only onedir going forward. Please also note that we used to refer to the packages as tiamat, since that is the tool we are using to build them. We shifted to using the name onedir to capture what the packages are, and not the tool we are using, so going forward the name is onedir.

The install guide and the release notes clarify the steps to upgrade to these packages. The single binary packages are different packages from the onedir packages. We are still providing rpm's/deb's,etc for users. For a user, the only changes would be to update their repo files to install the new rpm/deb's, etc and the note around pip packages installing in correct path as stated in both those docs.

You can also see the original salt enhancement proposal https://github.com/saltstack/salt-enhancement-proposals/pull/34 surrounding the original proposal to use these packages. And the packages have been in beta, until now the 3005 release.

My announcement was intended to update the community of the decision around 3005 being the final release for both classic and onedir packages. We wanted to updated before we made changes to the install guide and release notes.

[mwilhite]

I forgot to also include the 3004 blog https://saltproject.io/salt-3004-silicon-is-here/ where it was mentioned we were working on the  tiamat/onedir packages.

[Phipps, Thomas]

I think it should be mentioned as there seems to be some confusion about this. "classic" packages does not mean the deb or rpm, it means the fact that salt will not be tied to the system python. onedir will still be a rpm/deb it will just have its own python in a single dir.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/05e93fcc-d24e-4e96-ac05-34e9e3a5dc74n%40googlegroups.com.

[Vaarlion]

Ok, i've read it all, i have some question but with our slow upgrade cycle and LTS policy on OS, 95% of our infra won't leave debian 9 before also leaving salt stack as a company policy ... so we won't get supported by salt.
I try to fight against it, but at some point it's not up to me.

Most people don't have the time to follow the main meetup/talk and proposal in git, we rely on rss blogpost and this mailing list.

Throwing this subject with very little information, no link whatsoever and a clear sens of important breaking change in the mailing list was very ... "violant?" i don't have the right word but i hope the idea is there.

https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade-to-onedir.html have very little information about what will and won't be included in the pkg.
When i read:

> One issue that has delayed releases in the past is the classic package builds are not fully automated

I hope you understand that this read to me that you won't be building package anymore, not that you will build package, but with something else inside.

I personally don't like some of it.
  - IMO system deps need to be managed by OS for security reason, as i am not convinced by the replies. but this was already extensively discussed.

  - Dropping raspberry (i guess it's because onedir are way heavier?) will be a huge lost for some of our use in digital signage.

  - I am still confused about what can be patched and what can't be, because they was a lot of talk about it and it seam that the core binary won't be patch-able ?

    we have a couple of hot patch that fix bug in `config/_init_`, `pillars/_init_` and multiple utils. we'll see.

But the idea of decoupling from system's provided Python is very enticing, and the pip module was already not fun to work with so it can make it better.

Any info about the pip version ? Can unsupported os still run that, with all the drawback that come with it ?

[mwilhite]

The statement around the classic packages not being full automated was to explain one of the motivations around not providing classic packages in 3006 and the final release being in 3005. In hindsight, I think it would have been more clear to include the statement by whytewolf and the documentation. I will keep note of this for any other communications around onedir packages going forward. Thanks for the feedback.

> have very little information about what will and won't be included in the pkg.

I think you are referring to what dependencies will be included? You can look at the requirements here: https://github.com/saltstack/salt/tree/master/requirements/static/pkg  We are trying to build with python 3.9 for this release, so you can look at the py3.9 directory.

> I am still confused about what can be patched and what can't be, because they was a lot of talk about it and it seam that the core binary won't be patch-able ?

The onedir is patchable, although if the code is higher in the stack and built into the run binary it would not be. You can test your patches now with the RC2 packages.

You can also build your own packages documented in the readme here: https://gitlab.com/saltstack/open/salt-pkg/ to include any of the patches you need.

> Any info about the pip version ? Can unsupported os still run that, with all the drawback that come with it ?

We will still be providing a package on pypi, so yes an unsupported OS can run those packages. You can also use any packages provided by an OS distro, which would use the system python.

[Robby Callicotte]

On Friday, August 5, 2022 9:06:57 AM CDT 'mwilhite' via Salt-users wrote:
> We will still be providing a package on pypi, so yes an unsupported OS can
> run those packages. You can also use any packages provided by an OS distro,
> which would use the system python.

Point of clarification:
Onedir style salt will provide for pip installing new 3rd party python
modules. If your site does not allow for pip installing via external language
repo (pypi), then distro provided python modules (rpm,deb, etc.) will not work
with Onedir built Salt.

--
Robby Callicotte
He/Him/His
Timezone: America/Chicago
IRC: c4t3l | Twitter: @robbycl2v

[Dave Boucha]

Packages (RPMs, DEBs) will still be published and will be the standard way to install Salt.  They are just changing how some of the internals are installed by the packages.  You mostly won't notice any difference at all.

---

From: salt-...@googlegroups.com <salt-...@googlegroups.com> on behalf of Urs Rau <urs...@gmail.com>
Sent: Thursday, August 4, 2022 5:51 AM
To: Salt-users <salt-...@googlegroups.com>
Subject: Re: [salt-users] Classic packaging deprecation

 

⚠ External Email

To be honest, I am watching this with horror. This reads and seems to progress like somebodies garage project. Not in any way like a serious enterprise tool. 

Where are the actual discussions about the advantages and the real-life use-ability of such a new salt packaging? 

I for myself cannot see or imagine how you could honestly expect us out there in the real world to be able to just switch to such an all in one external binary 'packaging' without the proper platform deb or rpm packaging? and which one is it? tiamat, singlebin, or onedir? 

And trying to follow you guys switching between pop, heist, heist-salt, tiamat, onedir, singlebin it creates quite a messy picture. 

Which format is it going to be? Where are you going?

Glad I started looking at ansible and awx more seriously in recent months. 

Regards,

TheBigBear

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/677b81f1-9afb-4d27-b9c4-6005c3c021f0n%40googlegroups.com.

---

⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

[Peter Guspan]

Hi,

I am watching this discussion and I have a bad feeling from the start.. 

Initial announcement brought up more questions than answers.. a lot of terms are mixed together.. like Urs Rau already mentioned..

And I have some more questions.. if it's really true that the 3005 release will contain this single(bin/dir...whatever) packaged in rpm... will it also contain for stuff like default configs master/minion/rsyslog/systemd .. or directory structure for logs.. ?  Or do we have to take care of that?

What if I really don't want to have saltmaster functionality available on all minions..?

I am not always against all changes.. actually I am almost always curious about them.. and I want to test them before I judge them... so I've decided to give it a try.. here are my findings:

- 3005 is not out yet (15/08/2022) so.. because https://repo.saltproject.io/#single-binary contains only 3004.. I am testing that one

- there is a procedure for downloading and verifying this single-binary distribution... I am testing it and links are invalid... :

# Download latest salt and files required for file integrity verification
# This example covers the latest amd64 download for Linux systems
curl -fsSL https://repo.saltproject.io/salt/singlebin/3004/salt-3004_SHA512.asc -O
curl -fsSL https://repo.saltproject.io/salt/singlebin/3004/salt-3004_SHA512 -O
curl -fsSL https://repo.saltproject.io/salt/singlebin/3004/salt-3004-1-linux-amd64.tar.gz -O

... perfect... (Am I the first one who is testing that ?)

- so then I am checking link in first line which should point to singlebin file.. :

OS	Architecture	Version	Download	SHA512
Linux	amd64	v3004-1	salt-3004-1-linux-amd64.tar.gz	SHA512

.. that works!

- then I am checking the SHA512 file... broken link.. : https://repo.saltproject.io/salt/singlebin/3004/salt-3004_SHA512 ... really? it looks like there is no such directory as 3004 .. 

this page https://repo.saltproject.io/salt/singlebin/ contains now:

                                          ../
                                          3003.3-1/
                                          3003/
                                          3004-1/
                                          3004.2-1/
                                          3004rc1-1/
                                          latest/
2022-07-18 15:43:49+00:00       5.1 KB    repo.json
2022-07-18 15:43:49+00:00       4.7 KB    repo.mp

and yesterday even that 3004.2-1 was missing (got a screenshot from that somewhere).. there were just 3003 and 3004-1 + those repo.* files.. (looks like somebody is working on that.. but not completely checking his own work.. but at least some activity there)

- unpacking file... yes.. it's just a single binary.. no rpm.. whatever.. moving on.. maybe hopefully 3005 will be different.. 

- I am testing 3004-1 first.. creating services.. deploying default config for master.. configuring minion and all stuff around.. and after a while I have a working master.. with some test minions.. cool. Responses seem to be a bit sluggish.. but it's a test system and maybe I just have not much resources.. don't care right now

- day later (16/08/2022) I see a new release available.. 3004.2-1 .. testing it

- unpacking stuff.. still single binary.. no rpm around.. just replacing 3004-1 binary with 3004.2-1 .. restarting services.. and everything seems to be working.. Now I want to test pip functionality.. and suddenly finding out that I cannot install stuff like pyinotify via pip.. since I am behind proxy and it's throwing me some nasty SSLCertVerificationError.. strange.. when I use the system pip3 command with the same arguments.. everything works.. but it's still possible that I am doing something wrong... didn't have much more time to debug that yet.

- day later (16/08/2022) I see a new release available.. 3004.2-1 .. testing it.. and I am getting a LOT of warnings about: "file already exists but should not: /tmp/_ME...." .. sounds like something broken- unpacking stuff.. still single binary.. no rpm around.. just replacing 3004-1 binary with 3004.2-1 .. restarting services.. and everything seems to be working.. Now I want to test pip functionality.. and suddenly finding out that I cannot install stuff like pyinotify via pip.. since I am behind proxy and it's throwing me some nasty SSLCertVerificationError.. strange.. when I use the system pip3 command with the same arguments.. everything works.. but it's still possible that I am doing something wrong... didn't have much more time to debug that yet.

- evening of 16/08/2022 .. while writing this mail.. I see that there is RC2 of 3005 out in this singlebin form.. also discovering that in parallel there is also onedir build.. but all that is not packaged in ANY other form then just tar.gz file.. no rpm/deb so far... whatever.. testing singlebin.. still getting a lot of warnings about "file already exist but should not...." still broken... also still the same issue with pip (still it might be because of me..)

From what I see now.. it looks like not many people have tested this way of distribution and followed available documentation.. because there are some serious issues.. and if they did test it and found those issues.. then they haven't reported them.. and if they did report them.. then nobody fixed them.. Not sure which case of that it was.. but each one is worrying..

Still.. I am waiting for the official 3005 singlebin/onedir distribution.. and really hoping that it will be distributed as a regular package rpm..  Otherwise I will have to reconsider compiling/packaging from source.. and I don't like that idea either.. 

Sorry for being long.. didn't have time to write it shorter.. 

Best regards,

Peter

PS: on https://repo.saltproject.io/ .. there is not a word about onedir distribution.. only singlebin is mentioned.. 

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/BY3PR05MB80514AE1CBF9B446D145851CD96B9%40BY3PR05MB8051.namprd05.prod.outlook.com.

--

"I'm not anti-social; I'm just not user friendly"
"Microsoft is not the answer. Microsoft is the question. NO is the answer."   - Erik Naggum

[Kevin Landreth]

I understand the desire for unification and simplified deployments. But I'm finding it hard to reconcile this statement:

> All tools, from the salt repo, are included in the single-binary. Though, because they are contained within a single binary, they must be called differently.

Why? This is huge breaking change that is completely unnecessary.  A simple basename() + a lookup map should be able to map the entry points to it's respective counter parts in onebin/onedir/whatever-new-hotness.

test.py

#!/bin/env python3
import os.path
print(os.path.basename(__file__))

$ ./test.py
test.py

$ ln -s test.py test-call.py

$ ./test-call.py
test-call.py

Why can't the original entry-points be reserved and mark them as "deprecated" for a few major releases giving users a time to transition.  I haven't looked at the code but if someone got overly clever when implementing this and not taking into consideration the users of the product then this was not properly vetted or planned.  Also, in the that list provided by the "docs", it doesn't say how the normal master-invoked salt command with targeting is supposed to work?  Are those minion names reserved now?  

[Urs Rau]

Hi Peter,

You stumbled on some of the broken links on “repo.saltproject.io” it almost looks as if the links on that site are manually updated and synced. So this sort of thing can happen.

I have created a Issue giving the details of all the broken links and I am confident this will all be cleaned up quickly.

https://gitlab.com/saltstack/open/salt-pkg/-/issues/66

Regards,

TheBigBear

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAB3gpSJRbOJ-uGJcYJZ-nr1q5Hn1jGt77URs50ojXP33xOTqWA%40mail.gmail.com.