Building Crystal from source

[Jason Frey]

Hi all,

When building Crystal from source, it seems you need a prior version to build it, which makes sense since it's written in Crystal.  However, I'm looking to build Crystal from pure source without downloading a prior binary.  To do so, I assume I'd need to start by building a very early version of Crystal, then use that to build the next version and the next along the line.  If that is so, which versions of Crystal do I need to build, in order, to get there?

Thanks,

Jason

[Chris Hobbs]

There's a tag called ruby with the last version where the compiler is ruby.

You're going to have to build each release with the previous release, and before 0.10.0 you're probably going to have to do a lot of manual interventions.

May I ask why you want to do this?

[Jason Frey]

> You're going to have to build each release with the previous release, and before 0.10.0 you're probably going to have to do a lot of manual interventions. 

Do you happen to know if I can skip any of the builds?  I don't mind going through each one, but if I can skip any that would make the job easier :)

> May I ask why you want to do this?

I work at Red Hat, and am interested in getting Crystal as an official build within the Red Hat build system.  Part of the rules with gold-signed Red Hat RPMs is that all RPMs must be rebuildable from source, and we cannot use unsigned RPMs, for security and support reasons.  For a recent release of one of our products, we built a small app with Crystal and wanted to release in production, only to find that the RPMs provided by the Crystal team are binary-only.  This is an automatic no-no for us, as we cannot verify the contents of the binary build.  We ended up just converting to Ruby for this release (it's a small app), but I really want to release with Crystal for the next release, if possible.  I believe I can get it as an official build if I can build that primordial RPM, then build the subsequent RPMs using that primordial RPM, which I'm pretty sure is how we got golang into the build system.

Jason

[Chris Hobbs]

On 05/11/17 21:53, Jason Frey wrote:

> You're going to have to build each release with the previous release, and before 0.10.0 you're probably going to have to do a lot of manual interventions. 

Do you happen to know if I can skip any of the builds?  I don't mind going through each one, but if I can skip any that would make the job easier :)

> May I ask why you want to do this?

I work at Red Hat, and am interested in getting Crystal as an official build within the Red Hat build system.  Part of the rules with gold-signed Red Hat RPMs is that all RPMs must be rebuildable from source, and we cannot use unsigned RPMs, for security and support reasons.  For a recent release of one of our products, we built a small app with Crystal and wanted to release in production, only to find that the RPMs provided by the Crystal team are binary-only.  This is an automatic no-no for us, as we cannot verify the contents of the binary build.  We ended up just converting to Ruby for this release (it's a small app), but I really want to release with Crystal for the next release, if possible.  I believe I can get it as an official build if I can build that primordial RPM, then build the subsequent RPMs using that primordial RPM, which I'm pretty sure is how we got golang into the build system.

That’s great, because it means you only have to do this once!

There might be minor releases you can skip but it’s not documented and I doubt anyone will remember, you’ll have to discover which you can skip by trial and error. I heard there may even be commits which don’t obey the “must be buildable with the previous release” rule (before my time working on crystal). I’m not sure anyone has done this “from-scratch” build in a (long) while, since for bootstrapping new platforms we always cross-compile. Unfortunately, the best I can say is “good luck”, I’m sorry that this process is such a chore. Unfortunately you’ll probably have to handle switching LLVM versions in the middle too (3.3 -> 3.5 -> 4.0 is probably your minimal path).

Please report back with progress, and it’s fantastic to hear you’re investigating Crystal at Red Hat!

Jason

On Sunday, November 5, 2017 at 3:03:16 PM UTC-5, RX14 wrote:

There's a tag called ruby with the last version where the compiler is ruby.

You're going to have to build each release with the previous release, and before 0.10.0 you're probably going to have to do a lot of manual interventions.

May I ask why you want to do this?

On 5 Nov 2017, at 7:52 pm, Jason Frey <fry...@gmail.com> wrote:

Hi all,

When building Crystal from source, it seems you need a prior version to build it, which makes sense since it's written in Crystal.  However, I'm looking to build Crystal from pure source without downloading a prior binary.  To do so, I assume I'd need to start by building a very early version of Crystal, then use that to build the next version and the next along the line.  If that is so, which versions of Crystal do I need to build, in order, to get there?

Thanks,

Jason

--
You received this message because you are subscribed to the Google Groups "Crystal" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crystal-lang...@googlegroups.com.
To post to this group, send email to crysta...@googlegroups.com.
Visit this group at https://groups.google.com/group/crystal-lang.
To view this discussion on the web visit https://groups.google.com/d/msgid/crystal-lang/b5af3d6a-cac4-4f9a-9a0e-8b6dea715bf9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

​

[Tim Uckun]

Is there any way you vang accept a signed binary from the core team? It seems like you'd be able to avoid a ton of work if some trust could be established.

[Jason Frey]

It could be compiled and signed by Linus Torvalds himself, and I don't think we'd accept it at Red Hat. 😂

Although trust is part of the reason, other major reasons are repeatability and supportability.  Red Hat keeps track of all of the srpms of everything it's ever released, allowing us to rebuild anything at any point in time should it be necessary.  From a supportability perspective we are also able to patch major bug and security fixes into any version of software supported and in use by our customers.  With 10 year support contract life cycles on RHEL, you can see how this is important.  Even though Crystal is obviously new, there's no way I can get it into the build system without this "from scratch" ability to build.

Jason

[Ary Borenszweig]

Crystal can generate LLVM IR. We could distribute the LLVM IR (.ll) of some version of the compiler. Then with some llvm binary (I think llc) you can convert that to an executable. Would that help? Or do you need to compile it using a C compiler?

[Jason Frey]

That's a good question that I'll have to find the answer to.  My gut is telling me that probably won't work out, but it can't hurt to check.

Jason

[Chris Hobbs]

It can't be right, its just a textual representation of a binary file created by an executable that red hat hasn't built. It's neither patchable in case of bugs, auditable for security, or even portable between architectures or llvm versions. I wouldn't accept it.

[Jason Frey]

Just wanted to post an update, now that I've had some time to make progress on this.  For simplicity, I've started building Crystal inside docker containers, as it's easier to rev as I find things that I'm doing wrong.  Ultimately going from a docker container to an RPM shouldn't be too difficult, and a lot of this is discovery and exploration at this point.

So, with that, I've successfully built the old ruby tag.  The Dockerfile is here: https://github.com/Fryguy/crystal-builds/blob/master/ruby/Dockerfile, and I have the image built and up on dockerhub: https://hub.docker.com/r/fryguy9/crystal.  So, if you want to try out what Crystal was like in Nov 2013 you can do:

    docker run --rm fryguy9/crystal:ruby crystal -e 'puts "Hi"'

---

Next step is to try and build 0.1.0, but I'm already running into issues.  You can see the start of it here: https://github.com/Fryguy/crystal-builds/pull/1/commits/ed4ae477f2ad2d57298d6714f75dda136ef22f9c .  The issue I hit is...

    Error in /opt/crystal-0.1.0/src/compiler/crystal.cr:1: Error Syntax error in /opt/crystal-0.1.0/src/compiler/crystal/ast.cr:41: unexpected token: :

        def class_desc : String

                       ^

    require "crystal/**"

    ^

It seems interim, untagged, builds were created between the ruby tag and the 0.1.0 tag, so I'm probably going to have to find a way to bisect my way through the commits (unless maybe someone remembers which commits led to builds), but I think I'm done for today. :)

Jason

On Sunday, November 5, 2017 at 3:03:16 PM UTC-5, RX14 wrote: