Error running mconfig

[Jeffrey Layton]

Good afternoon,

I downloaded the tarball for version 3.6.4 and when I tried running "mconfig" I get the following error:

"E: Not inside a git repository and no VERSION file found. Abort"

This is an error I haven't encountered before. Do I just need to create a file named VERSION with "3.6.4" in it?

Thanks!

Jeff

Virus-free. www.avast.com

[David Trudgian]

Jeff,

This is most commonly due to downloading GitHub's auto generated 'source code' tarball, which contains all the files in the Git repository, but is not a git repo, nor contains the extra things added when we create a release.

Check that you are using the tarball from: https://github.com/hpcng/singularity/releases/tag/v3.6.4 that is called `singularity-3.6.4.tar.gz` and not the `Source code (tar.gz) link`. There's also a sha256sums file there so you can check the hash of the tarball you have to make sure it is correct.

The correct one does have the VERSION file referred to, and the vendored libraries etc. that are needed to build outside of a git clone.

DT

[Jeffrey Layton]

Thanks - I will definitely check on the tarball. I remember getting it from github under the released Versions. I just downloaded the .tar.gz file.

Thanks!

Jeff

Virus-free. www.avast.com

[Jeffrey Layton]

I found the discrepancy. When I look under the 3.6.4 Release, under "Assets" at the bottom, it has three options for downloading the source code. The fist one is labeled "singularity-3.6.4.tar.gz", the second is "Source code (zip)" and the third is "Source code (.tar.gz)". I was using the last asset "Source code (.tar.gz)" which is significantly smaller than "Singularity-3.6.4.tar.gz". I guess the one with the version in the file name is the correct one? It's also about 2-1/2 times the size of the "Source code (.tar.gz)" download. I don't know if they are compressed differently, but this is a pretty big difference.

BTW - this is confusing. I'm used to just using the "Source (.tar.gz)" option for other packages and they install correctly. Is there a way to fix this option to make sure it has all of the needed bits? Perhaps a large warning for us noobs tell us which to download to use and not to use the other one? I would make be a Level 1 heading, perhaps in read, so it's easy to catch?

Thanks for the help!

Jeff

Virus-free. www.avast.com

[Jeffrey Layton]

I forgot to mention one thing. If you use the "Source code (.tar.gz)" download, it uncompress/untars into the sub-directory, "singularity-3.6.4". If you use the larger download "singularity-3.6.4.tar.gz" it becomes just "singularity" as the sub-directory. In my experience it's probably better to use "singularity-3.6.4" as the sub-directory so you don't overwrite a previous installation.

Jeff

Virus-free. www.avast.com

[David Trudgian]

Hi Jeff,

I found the discrepancy. When I look under the 3.6.4 Release, under "Assets" at the bottom, it has three options for downloading the source code. The fist one is labeled "singularity-3.6.4.tar.gz", the second is "Source code (zip)" and the third is "Source code (.tar.gz)". I was using the last asset "Source code (.tar.gz)" which is significantly smaller than "Singularity-3.6.4.tar.gz". I guess the one with the version in the file name is the correct one? It's also about 2-1/2 times the size of the "Source code (.tar.gz)" download. I don't know if they are compressed differently, but this is a pretty big difference.

GitHub creates those 'Source code" links automatically. It simply creates a .zip or .tar.gz of the content of the git repository at the tag for the release. If you look at the URL you'll see they come from `/archive` not `/releases`. The named `singularity-xxxxx` ones are tarballs that we create. They are produced by a `make dist` run in the code base. They are bigger because they vendor all the dependencies, so that you have a self-contained tarball that has everything you need to install Singularity without Go having to fetch stuff from the web.
 

BTW - this is confusing. I'm used to just using the "Source (.tar.gz)" option for other packages and they install correctly. Is there a way to fix this option to make sure it has all of the needed bits?

The way the mconfig based build process for Singularity works, the `Source code` auto-generated links won't work. We don't want to commit our dependencies to the git repository, and the versioning information is pulled from git history, hence the message from mconfig when you use those files. It's not uncommon for projects to provide tarballs that are different to the direct repo generated ones, e.g. autotools based projects often distribute their own .tar.gz which you can ./configure from - while working direct from the repo files requires extra steps. Unfortunately our process is a bit further of an outlier than that, and uses a build system which isn't commonly encountered. I agree it's a bit of a sharp edge for people going with the prior experience, but our docs do go through the whole procedure.
 

Perhaps a large warning for us noobs tell us which to download to use and not to use the other one? I would make be a Level 1 heading, perhaps in read, so it's easy to catch? 

I'll make a note to put something at the end of the release notes for future versions, so it sits directly above the download. For what it's worth, our INSTALL.md and guides etc. do point to obtaining the correct release tarball, but yes it'd be good to have a note here.

 > I forgot to mention one thing. If you use the "Source code (.tar.gz)" download, it uncompress/untars into the sub-directory, "singularity-3.6.4". If you use the larger download "singularity-3.6.4.tar.gz" it becomes just "singularity" as the sub-directory. In my experience it's probably better to use "singularity-3.6.4" as the sub-directory so you don't overwrite a previous installation.

This is a historical artifact. Prior to the introduction of modules to Go, the source had to be built from `$GOPATH/src/github.com/sylabs/singularity` *exactly*. This is why the extraction is to a dir named `singularity`. Now we are using Go modules we don't have this limitation. However, given the general lack of familiarity with builds using Go (outside of developers) we have hesitated to change our install docs and this path until now - so that people have had a method of installation that has remained stable. We have an issue tracking this: https://github.com/hpcng/singularity/issues/2215 - and I personally think that the next (3.8) release would be a good time to change. Most Go projects use modules now, and our code is mature enough that this is a specific change, that doesn't run the risk of complicating other things.

DT

[Priedhorsky, Reid]

On Nov 23, 2020, at 8:22 AM, David Trudgian <dtr...@sylabs.io> wrote:

GitHub creates those 'Source code" links automatically. It simply creates a .zip or .tar.gz of the content of the git repository at the tag for the release.

FWIW this is a common critique of GitHub. No project I’m aware of likes them; they would all prefer to turn them off but GitHub doesn’t allow that. It really is the worst of both worlds: you get bare-bones, maintainer-focused source code but no Git.

OpenMPI goes so far as to use their entire GitHub release notes to disavow them: https://github.com/open-mpi/ompi/releases/tag/v4.0.5

Personally, I’m baffled. GitHub is aware of the issue (it’s a common topic on their forums) but for some reason doesn’t see it as a problem or chooses not to fix it.

HTH,

Reid

—

he/his

[Jeffrey Layton]

Thanks David - the explanations are great! I think I have a good understanding of how to build SIngularity and, perhaps even better, how to build anything else I download from github :)

Thanks!

Jeff