Rust required to build Gecko

[Ralph Giles]

tl;dr This is a heads-up that all gecko developers should install rust.

Next week I plan to switch our default build config to require Rust
when building Firefox.[1] If you compile Firefox from the C++ source,
please install the Rust language environment now.

You can install Rust by running `./mach bootstrap` which will download
and run the rustup installer for you.[2]

We recommend the installer at https://rustup.rs/ (despite being beta)
because it makes staying up to date and cross-compilation easy. If you
want more control, or to experiment with rust, you can install
directly from that website.

The main thing is to have up-to-date versions of the `rustc` and
`cargo` executables in the path of your build shell. Rust releases
every six weeks, just like Firefox, and we generally require the
latest stable release to compile mozilla-central. You can stay current
by running `rustup update`.

You'll still be able to build without a rust compiler by adding:

ac_add_options --disable-rust

to your mozconfig. This is a temporary work-around; we expect to
remove that option and require Rust unconditionally early next year as
non-optional features start to depend on it.

Rust language in Gecko is an important part of Project Quantum. I'm
excited to be taking this next step toward that future. We first
shipped Rust code to users in Firefox 48, so it's long past time this
aspect of our default builds matched what we release.[3]

Thanks for your attention,
-r

[1] Enabling rust is https://bugzil.la/1283898
[2] bootstrap support landed Tuesday in https://bugzil.la/1286799
[3] If you have issues with the installer or build, please file issues
blocking our tracking bug at https://bugzil.la/oxidation

[Andrew Halberstadt]

When first installing rust with ./mach bootstrap the install is
successful, but there is a message about not being able to find the
compiler immediately afterwards.

For anyone confused by this, the binaries are downloaded to ~/.cargo/bin
and adding this directory to your $PATH should fix the issue. The
bootstrapper explains this if you run it a second time, but makes no
mention of it the first time through for some reason.

-Andrew

[Ralph Giles]

On Fri, Nov 25, 2016 at 7:48 AM, Andrew Halberstadt
<ahalbe...@mozilla.com> wrote:

> For anyone confused by this, the binaries are downloaded to ~/.cargo/bin
> and adding this directory to your $PATH should fix the issue. The
> bootstrapper explains this if you run it a second time, but makes no
> mention of it the first time through for some reason.

Thanks. I've tried to address this in
https://bugzilla.mozilla.org/show_bug.cgi?id=1319860

-r

[Gerald Squelart]

Hi Ralph,

Following your instructions, rustc and friends are in ~/.cargo/bin, and I've added that path in my $PATH.

But now, `./mach build` gives me:
force-cargo-build
env: /usr/local/bin/cargo: No such file or directory
/mozilla-central/config/rules.mk:951: recipe for target 'force-cargo-build' failed
gmake[2]: *** [force-cargo-build] Error 127
/mozilla-central/config/recurse.mk:71: recipe for target 'toolkit/library/rust/target' failed
gmake[1]: *** [toolkit/library/rust/target] Error 2
gmake[1]: *** Waiting for unfinished jobs....

Sym-linking rustc from /usr/local/bin worked, but it feels wrong!


After that, building works, but shows some scary bold text:
note: link against the following native artifacts when linking against this static library
note: the order and any duplication can be significant on some platforms, and so may need to be preserved
note: library: System
note: library: c
note: library: m

What's with that?

Cheers,
g.

[Ralph Giles]

On Sun, Nov 27, 2016 at 2:46 PM, Gerald Squelart <sque...@gmail.com> wrote:

> Following your instructions, rustc and friends are in ~/.cargo/bin, and I've added that path in my $PATH.
>
> But now, `./mach build` gives me:
> force-cargo-build
> env: /usr/local/bin/cargo: No such file or directory

> [...]

> Sym-linking rustc from /usr/local/bin worked, but it feels wrong!

That does seem wrong. So you have rustc and cargo in ~/.cargo/bin, but
it was looking in /usr/local/bin instead? Was there a
/usr/local/bin/rustc previously? What does `which cargo` print?

> After that, building works, but shows some scary bold text:
> note: link against the following native artifacts when linking against this static library
> note: the order and any duplication can be significant on some platforms, and so may need to be preserved
> note: library: System
> note: library: c
> note: library: m
>
> What's with that?

The rust compiler prints this out whenever it generates a static
library, to remind you to link the dependencies into the final target.
We do that (XUL has the same dependencies) but I agree the `note`
output is distracting. I wrote a patch to suppress this by writing it
to a file instead, but didn't get it to an acceptable state to land.
If someone wants to un-bitrot it and add tests, I think that would be
the best way to resolve this.
https://github.com/rust-lang/rust/issues/31471

Thanks for sharing your results,
-r

[Michael Froman]

I tried the bootstrap on OS X (after a fresh pull from mozilla-central this morning), and I also have failures. I’ve added the following lines to my .profile:
export PATH="$HOME/.cargo/bin:$PATH"
source /Users/mfroman/.cargo/env

I continue to get errors from the bootstrap process:
The details of the failure are as follows:

CalledProcessError: Command '[u'/Users/mfroman/.cargo/bin/rustc', u'--version']' returned non-zero exit status 1

File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/mach_commands.py", line 26, in bootstrap
bootstrapper.bootstrap()
File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/bootstrap.py", line 232, in bootstrap
self.instance.ensure_rust_modern()
File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 525, in ensure_rust_modern
modern, version = self.is_rust_modern()
File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 512, in is_rust_modern
our = self._parse_version(rustc)
File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 391, in _parse_version
stderr=subprocess.STDOUT)
File "/Users/mfroman/mozilla/moz-central/python/mozboot/mozboot/base.py", line 320, in check_output
return fn(*args, **kwargs)
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)

Any thoughts? Further info:
mfroman-23602:moz-central mfroman$ which rustc
/Users/mfroman/.cargo/bin/rustc
mfroman-23602:moz-central mfroman$ rustc --version
error: no default toolchain configured

Thanks,
Michael.

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Ralph Giles]

On Mon, Nov 28, 2016 at 9:28 AM, Michael Froman <mfr...@mozilla.com> wrote:

> Any thoughts? Further info:
> mfroman-23602:moz-central mfroman$ which rustc
> /Users/mfroman/.cargo/bin/rustc
> mfroman-23602:moz-central mfroman$ rustc --version
> error: no default toolchain configured

Very mysterious. This error message usually means rustup's redirect to
the actual toolchain (in /Users/mfroman/.multirust/toolchains) is
confused. I've seen it when the executable filename has a prefix, for
example.

Make sure there's a toolchain installed under ~/.multirust, then maybe
try `rustup update` and see what that says.

If that doesn't work, try `rustup default stable`.

-r

[Gregory Szorc]

> On Nov 27, 2016, at 14:46, Gerald Squelart <sque...@gmail.com> wrote:
>
>> On Saturday, November 26, 2016 at 3:59:05 AM UTC+11, Ralph Giles wrote:
>> On Fri, Nov 25, 2016 at 7:48 AM, Andrew Halberstadt
>> <ahalbe...@mozilla.com> wrote:
>>
>>> For anyone confused by this, the binaries are downloaded to ~/.cargo/bin
>>> and adding this directory to your $PATH should fix the issue. The
>>> bootstrapper explains this if you run it a second time, but makes no
>>> mention of it the first time through for some reason.
>>
>> Thanks. I've tried to address this in
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1319860
>>
>> -r
>
> Hi Ralph,
>
> Following your instructions, rustc and friends are in ~/.cargo/bin, and I've added that path in my $PATH.
>
> But now, `./mach build` gives me:
> force-cargo-build
> env: /usr/local/bin/cargo: No such file or directory
> /mozilla-central/config/rules.mk:951: recipe for target 'force-cargo-build' failed
> gmake[2]: *** [force-cargo-build] Error 127
> /mozilla-central/config/recurse.mk:71: recipe for target 'toolkit/library/rust/target' failed
> gmake[1]: *** [toolkit/library/rust/target] Error 2
> gmake[1]: *** Waiting for unfinished jobs....
>
> Sym-linking rustc from /usr/local/bin worked, but it feels wrong!

This is a longstanding problem with the build system w.r.t. detected
programs changing out from under the build system: the build system
makes a lot of assumptions that the system doesn't change between
builds. If you do things like change the toolchain or where binaries
are installed, you'll need to perform a clobber build.

I just filed bug 1320728 to improve the build system's handling of
this scenario.

>
>
> After that, building works, but shows some scary bold text:
> note: link against the following native artifacts when linking against this static library
> note: the order and any duplication can be significant on some platforms, and so may need to be preserved
> note: library: System
> note: library: c
> note: library: m
>
> What's with that?
>
> Cheers,
> g.

[Gerald Squelart]

Ah yes, it worked after a clobber. Sorry, I should have tried that before.

> I just filed bug 1320728 to improve the build system's handling of
> this scenario.

Thank you!
g.

[Michael Froman]

See inline.

mfroman-23602:moz-central mfroman$ rustup update
info: no updatable toolchains installed
info: checking for self-updates

>
> If that doesn't work, try `rustup default stable`.

mfroman-23602:moz-central mfroman$ rustup default stable
info: syncing channel updates for 'stable-x86_64-apple-darwin'
info: downloading component 'rustc'
28.7 MiB / 28.7 MiB (100 %) 1.4 MiB/s ETA: 0 s
info: downloading component 'rust-std'
36.1 MiB / 36.1 MiB (100 %) 1.4 MiB/s ETA: 0 s
info: downloading component 'rust-docs'
7.5 MiB / 7.5 MiB (100 %) 1.4 MiB/s ETA: 0 s
info: downloading component 'cargo'
2.4 MiB / 2.4 MiB (100 %) 1.5 MiB/s ETA: 0 s
info: installing component 'rustc'
info: installing component 'rust-std'
info: installing component 'rust-docs'
info: installing component 'cargo'
info: default toolchain set to 'stable-x86_64-apple-darwin'

stable-x86_64-apple-darwin installed - rustc 1.13.0 (2c6933acc 2016-11-07)


>
> -r

Now, I’ve made it through the bootstrap successfully. Thanks for the tips! I’m not sure exactly what happened the first time. Possibly a network failure during the original download or something.

-Michael.

[Mike Hommey]

Note, you probably didn't /actually/ need a clobber. It would probably
have fixed itself after re-running mach configure.

Mike

[Ralph Giles]

I wasn't able to enable Rust by default this week. While it's working
for most developers, there are some automation jobs which aren't ready
for the change. We'll address those and continue to improve the
`./mach boostrap` installer behaviour before flipping the switch.

Thanks to everyone for your support and testing feedback!

-r

[Ralph Giles]

Today we've pushed the change to enable rust language code by default
in Firefox builds. The changes are on the autoland branch right now,
so this will affect your builds from mozilla-central or gecko-dev
starting tomorrow. This brings our default developer build in line
with what we've been doing with official builds. Thanks to everyone
who helped make this happen.

As a reminder, you'll now need `rustc` and `cargo` in the PATH of your
build environment, just like with python and the C/C++ compiler. You
can install rust with `./mach bootstrap`. After that you can stay on
the latest stable release by running `rustup update` every 6 weeks or
so. If you want more control I recommend manually running the
installer and update tool from https://rustup.rs/

More information about using Rust code in Firefox can be found at
https://developer.mozilla.org/en-US/Firefox/Building_Firefox_with_Rust_code

Cheers,

[Ben Kelly]

I tried ./mach bootstrap on a fresh m-c this morning and got:

Will try to install Rust.
Downloading rustup-init... Error running mach:

['bootstrap']

The error occurred in code that was called by the mach command. This is
either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this
error
message.

The details of the failure are as follows:

URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL
routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

> --
> You received this message because you are subscribed to the Google Groups
> "gecko-all" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gecko-all+...@mozilla.com.
> To post to this group, send email to geck...@mozilla.com.
> To view this discussion on the web visit https://groups.google.com/a/
> mozilla.com/d/msgid/gecko-all/CA%2Brf4XK6cuhDPk%
> 2B39o6SLgRpCZ9z%3DepgugEzpuB_SW0hJ05a8w%40mail.gmail.com.
>

[Simon Sapin]

On 16/12/16 17:27, Ben Kelly wrote:
> I tried ./mach bootstrap on a fresh m-c this morning and got:
>
> Will try to install Rust.
> Downloading rustup-init... Error running mach:
>
> ['bootstrap']
>
> The error occurred in code that was called by the mach command. This is
> either
> a bug in the called code itself or in the way that mach is calling it.
>
> You should consider filing a bug for this issue.
>
> If filing a bug, please include the full output of mach, including this
> error
> message.
>
> The details of the failure are as follows:
>
> URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL
> routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

What Python version do you have? (Run 'python --version') If it’s not
the latest, can you update it?

Try modifying python/mozboot/mozboot/rust.py to replace this line:

RUSTUP_URL_BASE = 'https://static.rust-lang.org/rustup'

by this line:

RUSTUP_URL_BASE =
'https://static-rust-lang-org.s3.amazonaws.com/rustup'

If that fixes the issue for you, it’s likely that your Python version
does not support SSL SNI.

--
Simon Sapin

[Ralph Giles]

On Fri, Dec 16, 2016 at 9:19 AM, Simon Sapin <simon...@exyr.org> wrote:

> RUSTUP_URL_BASE = 'https://static-rust-lang-org.s3.amazonaws.com/rustup'
>
> If that fixes the issue for you, it’s likely that your Python version does
> not support SSL SNI.

I can reproduce with python urlib2.urlopen() on an ubuntu 14.04 docker
container. I thought using the in-tree requests module to work around
that issue?

-r

[Ralph Giles]

Anyway, thanks for the suggestion, Simon. I filed
https://bugzil.la/1324040 with this fix.

In the meantime, the curl command line from https://rustup.rs/ should
work equivalently.

-r

[Simon Sapin]

On 16/12/16 19:26, Ralph Giles wrote:
> Anyway, thanks for the suggestion, Simon. I filed
> https://bugzil.la/1324040 with this fix.
>
> In the meantime, the curl command line from https://rustup.rs/ should
> work equivalently.

Oops, I wrote this earlier and forgot to click Send.

--

http://docs.python-requests.org/en/master/community/faq/#what-are-hostname-doesn-t-match-errors
says that Python 2.7.9+ supports SNI natively.

It looks like Requests can make it work on older Python if some other
libraries (including PyOpenSSL) are available:

https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484

--
Simon Sapin

[Jim Blandy]

I had a .mozconfig file that included the line:

. "$topsrcdir/build/mozconfig.common"

This turns out to cause problems: it loads build/mozconfig.rust, which says
to look for rustc in $topsrcdir/rustc/bin, leading to errors like this:

0:05.53 checking for rustc... not found
0:05.53 DEBUG: rustc: Trying /home/jimb/moz/dbg/rustc/bin/rustc
0:05.53 ERROR: Cannot find rustc
0:05.54 *** Fix above errors and then restart with\
0:05.54 "/usr/bin/gmake -f client.mk build"
0:05.54 client.mk:375: recipe for target 'configure' failed
0:05.54 gmake: *** [configure] Error 1

I remove the line loading mozconfig.common and things seem to be going well.

[J. Ryan Stinnett]

On Wed, Dec 21, 2016 at 12:23 AM, Jim Blandy <jbl...@mozilla.com> wrote:
> I had a .mozconfig file that included the line:
>
> . "$topsrcdir/build/mozconfig.common"

My understanding is that we're generally not supposed to include the
in-tree mozconfigs in our local builds, since they are free to make
various automation-only assumptions, which seems to be what happened
here. It's tempting to do it anyway though, for some abstract feeling
that the build will more closely resemble an official one. (I used to
do something like this as well!)

I believe many people have fallen into this trap over the years, most
likely because of MDN pages[1] that appeared to suggest using
automation mozconfigs directly (there's at least a warning on the page
now).

[1]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Configuring_Build_Options#Example_.mozconfig_Files

- Ryan

[Jim Blandy]

All the more reason for "mach" to be the exclusive place to put all these
smarts.

The only things I really want anyway are:

mk_add_options MOZ_OBJDIR=obj-bug
ac_add_options --enable-debug='-g3 -O0 -fno-inline'
ac_add_options --disable-optimize

[Mike Hommey]

On Wed, Dec 21, 2016 at 12:41:37AM -0600, J. Ryan Stinnett wrote:
> On Wed, Dec 21, 2016 at 12:23 AM, Jim Blandy <jbl...@mozilla.com>
> wrote:
> > I had a .mozconfig file that included the line:
> >
> > . "$topsrcdir/build/mozconfig.common"
>
> My understanding is that we're generally not supposed to include the
> in-tree mozconfigs in our local builds, since they are free to make
> various automation-only assumptions, which seems to be what happened
> here. It's tempting to do it anyway though, for some abstract feeling
> that the build will more closely resemble an official one. (I used to
> do something like this as well!)

Note that a lot of the flags set by in-tree mozconfigs are now redundant
with defaults you get without a mozconfig. While that's not entirely
true, it's a goal to have the in-tree mozconfigs to eventually *only*
contain automation-specific things (or very specific things like API
keys, update channel, etc.)

> I believe many people have fallen into this trap over the years, most
> likely because of MDN pages[1] that appeared to suggest using
> automation mozconfigs directly (there's at least a warning on the page
> now).

I filed bug 1324998 to put an end to this bad habit once and for all.

Mike

[Julian Seward]

On 21/12/16 07:55, Jim Blandy wrote:
> The only things I really want anyway are:
>
> mk_add_options MOZ_OBJDIR=obj-bug
> ac_add_options --enable-debug='-g3 -O0 -fno-inline'
> ac_add_options --disable-optimize

As a side note, I tend to use "-Og -g" as that gives much faster code
than -O0 whilst retaining a reasonable debugging experience, viz:

ac_add_options --enable-optimize="-g -Og"
ac_add_options --enable-debug-symbols
ac_add_options --enable-debug

It does require gcc 5 or later (roughly) and gdb less than a couple of
years old, but that's not much of a constraint.

>From the gcc manual:

-Og
Optimize debugging experience. -Og enables optimizations that do not
interfere with debugging. It should be the optimization level of choice
for the standard edit-compile-debug cycle, offering a reasonable level
of optimization while maintaining fast compilation and a good debugging
experience.

[Ted Mielczarek]

On Wed, Dec 21, 2016, at 04:56 PM, Julian Seward wrote:
> On 21/12/16 07:55, Jim Blandy wrote:
> > The only things I really want anyway are:
> >
> > mk_add_options MOZ_OBJDIR=obj-bug
> > ac_add_options --enable-debug='-g3 -O0 -fno-inline'
> > ac_add_options --disable-optimize
>
> As a side note, I tend to use "-Og -g" as that gives much faster code
> than -O0 whilst retaining a reasonable debugging experience, viz:
>

Just FYI,

> ac_add_options --enable-optimize="-g -Og"
> ac_add_options --enable-debug-symbols

--enable-debug-symbols is the default, so you don't need to specify it.
Additionally, --enable-debug-symbols should be adding -g to the compile
command, so you ought to be able to just use `--enable-optimize="-Og"`.

-Ted

[Ralph Giles]

On Thu, Nov 24, 2016 at 2:49 PM, Ralph Giles <gi...@mozilla.com> wrote:

> You'll still be able to build without a rust compiler by adding:
>
> ac_add_options --disable-rust

Just a heads up that I've removed this configure option in bug
1284816, so this will stop working on mozilla-central with the next
autoland merge. If you have this in your mozconfig, you'll need to
remove it and add a recent rust toolchain to your build environment.
You can do this by running `./mach bootstrap` and restarting your
shell, or running the install from https://rust-lang.org/

If you're maintaining Firefox on a tier-3 platform which doesn't have
good rust support, please be aware that Firefox 53 will be the last
upstream release where rust is optional.

Cheers,
-r

[rafet...@gmail.com]

25 Kasım 2016 Cuma 00:49:58 UTC+2 tarihinde Ralph Giles yazdı: