More Rust code

Nicholas Nethercote

10/07/17 03:30

Hi,

Firefox now has multiple Rust components, and it's on track to get a bunch
more. See https://wiki.mozilla.org/Oxidation for details.

I think this is an excellent trend, and I've been thinking about how to
accelerate it. Here's a provocative goal worth considering: "when writing a
new compiled-code component, or majorly rewriting an existing one, Rust
should be considered / preferred / mandated."

What are the obstacles? Here are some that I've heard.

- Lack of Rust expertise for both writing and reviewing code. We have some
pockets of expertise, but these need to be expanded greatly. I've heard
that there has been some Rust training in the Paris and Toronto offices.
Would training in other offices (esp. MV and SF, given their size) be a
good idea? What about remoties?

- ARM/Android is not yet a Tier-1 platform for Rust. See
https://forge.rust-lang.org/platform-support.html and
https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some
details.

- Interop with existing components can be difficult. IPDL codegen rust
bindings could be a big help.

- Compile times are high, especially for optimized builds.

Anything else?

Nick

More Rust code	Nicholas Nethercote	10/07/17 03:30	Hi, Firefox now has multiple Rust components, and it's on track to get a bunch more. See https://wiki.mozilla.org/Oxidation for details. I think this is an excellent trend, and I've been thinking about how to accelerate it. Here's a provocative goal worth considering: "when writing a new compiled-code component, or majorly rewriting an existing one, Rust should be considered / preferred / mandated." What are the obstacles? Here are some that I've heard. - Lack of Rust expertise for both writing and reviewing code. We have some pockets of expertise, but these need to be expanded greatly. I've heard that there has been some Rust training in the Paris and Toronto offices. Would training in other offices (esp. MV and SF, given their size) be a good idea? What about remoties? - ARM/Android is not yet a Tier-1 platform for Rust. See https://forge.rust-lang.org/platform-support.html and https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some details. - Interop with existing components can be difficult. IPDL codegen rust bindings could be a big help. - Compile times are high, especially for optimized builds. Anything else? Nick
Re: More Rust code	Frederik Braun	10/07/17 03:57	On 10.07.2017 12:29, Nicholas Nethercote wrote: > > What are the obstacles? Here are some that I've heard. > > - Lack of Rust expertise for both writing and reviewing code. We have some > pockets of expertise, but these need to be expanded greatly. I've heard > that there has been some Rust training in the Paris and Toronto offices. > Would training in other offices (esp. MV and SF, given their size) be a > good idea? What about remoties? FYI I have been (secretly[1]) thinking about Rust trainings in the Berlin office. AFAIU the previous trainings used the professional development budget from those who attended. Naturally, the same could be true for remoties? Though of course they'd also have to budget for travel/accomodation. [1] Maybe today is the day, I should start finding out what the demand is
Re: More Rust code	smaug	10/07/17 06:41	On 07/10/2017 01:29 PM, Nicholas Nethercote wrote: > Hi, > > Firefox now has multiple Rust components, and it's on track to get a bunch > more. See https://wiki.mozilla.org/Oxidation for details. > > I think this is an excellent trend, and I've been thinking about how to > accelerate it. Here's a provocative goal worth considering: "when writing a > new compiled-code component, or majorly rewriting an existing one, Rust > should be considered / preferred / mandated." 'considered' sounds reasonable, 'preferred' or 'mandated' requires some reasoning. > > What are the obstacles? Here are some that I've heard. > > - Lack of Rust expertise for both writing and reviewing code. We have some > pockets of expertise, but these need to be expanded greatly. I've heard > that there has been some Rust training in the Paris and Toronto offices. > Would training in other offices (esp. MV and SF, given their size) be a > good idea? What about remoties? > > - ARM/Android is not yet a Tier-1 platform for Rust. See > https://forge.rust-lang.org/platform-support.html and > https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some > details. > > - Interop with existing components can be difficult. IPDL codegen rust > bindings could be a big help. ipdl? or do you mean idl? or perhaps webidl? Also, xpconnect doesn't deal with rust, and our chrome code still heavily relies on idl+xpconnect. > > - Compile times are high, especially for optimized builds. > > Anything else? memory management. As far as I know there is no reasonable way to deal with cycle collection (which means also no reasonable way to have references to JS objects). > > Nick >
Re: More Rust code	Michael Layzell	10/07/17 08:03	On Mon, Jul 10, 2017 at 9:41 AM, smaug <sm...@welho.com> wrote: > ipdl? or do you mean idl? or perhaps webidl? > Also, xpconnect doesn't deal with rust, and our chrome code still heavily > relies on idl+xpconnect. > I have written https://bugzilla.mozilla.org/show_bug.cgi?id=1293362, which while I'm not actively working on, could be used to help close that idl+xpconnect gap a bit. I've definitely had some opposition to the idea of adding rust bindings to XPCOM interfaces because we're theoretically trying to kill them however. If we decide that getting good bindings is a thing which we want in order to to develop more stuff in rust I can polish the patches back up. memory management. As far as I know there is no reasonable way to deal with > cycle collection (which means also no reasonable way to have references to > JS objects). > I don't have my WIP cycle collection with rust XPCOM interfaces patches on that bug, but both of these things are doable with XPCOM bindings for rust.
Re: More Rust code	Sylvestre Ledru	10/07/17 08:41	I am organizing a second rust training in Paris in September or October (amongst others) The registration has been sent to the French staff last week. Will be shared with the European staff at the end of the week. Le lun. 10 juil. 2017 à 12:30, Nicholas Nethercote <n.net...@gmail.com> a écrit : > Hi, > > Firefox now has multiple Rust components, and it's on track to get a bunch > more. See https://wiki.mozilla.org/Oxidation for details. > > I think this is an excellent trend, and I've been thinking about how to > accelerate it. Here's a provocative goal worth considering: "when writing a > new compiled-code component, or majorly rewriting an existing one, Rust > should be considered / preferred / mandated." > > What are the obstacles? Here are some that I've heard. > > - Lack of Rust expertise for both writing and reviewing code. We have some > pockets of expertise, but these need to be expanded greatly. I've heard > that there has been some Rust training in the Paris and Toronto offices. > Would training in other offices (esp. MV and SF, given their size) be a > good idea? What about remoties? > > - ARM/Android is not yet a Tier-1 platform for Rust. See > https://forge.rust-lang.org/platform-support.html and > https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some > details. > > - Interop with existing components can be difficult. IPDL codegen rust > bindings could be a big help. > > - Compile times are high, especially for optimized builds. > > Anything else? > > Nick > _______________________________________________ > firefox-dev mailing list > firef...@mozilla.org > https://mail.mozilla.org/listinfo/firefox-dev >
Re: More Rust code	Zbigniew Braniecki (Gandalf)	10/07/17 09:12	Hi, This year I've been working on three new components[0] and I considered Rust for both. At the time, when asked people on #rust and #developers, the answer was that oxidation targets should have very simple APIs, because getting anything more complex than strings/ints between rust and cpp/js is non-trivial. I don't know if that changed, but I'd appreciate any tutorial on how to write a WebIDL/XPIDL module or transfer JSON between Rust and JS/CPP. Thanks, zb. [0] intl/locale/LocaleService, intl/locale/OSPreferences and intl/l10n/Localization On Mon, Jul 10, 2017 at 8:41 AM, Sylvestre Ledru <sylv...@mozilla.com> wrote:
Re: More Rust code	Bobby Holley	10/07/17 09:33	There are obvious benefits to writing new things in Rust. In general, the main barrier to doing so is the extent to which they may interact with existing C++ code, and the adequacy of our tooling for managing that interaction. In London last year, we had a meeting to discuss how we wanted to use Rust for DOM code. The consensus of that meeting was that Rust code could be useful as a backend for DOM functionality, but that it still needed to be wrapped inside a C++ shell to handle reflection, cycle collection, and interaction with the rest of the (C++-based) DOM. Our tooling (specifically around bindgen) has matured significantly since then, largely to support stylo. But there are still a lot of unsolved use cases, and some of them (like inlining) are pretty hard to solve. The costs of haphazard integration can be high, specifically to performance, safety, and complexity/maintainability. So I think we want to tread carefully here. In other words, I think we should encourage people to consider Rust for new code, but also make sure that we have a thoughtful and informed discussion about whether a use-case makes sense and how best to support it. Right now we have a module that's intended to govern decisions on these issues [1]. However, the peer list for that module seems oriented around C++ and build system expertise, and is sparse on people who have been deeply involved in the Rust integration efforts over the past year. So we could expand that group with expertise to tackle Rust integration issues. Or it could charter a committee of such experts to do so. Or some combination. Either way, I think we want some amount of steering here, and I can think of a handful of names whose input is likely required to get it right. Ehsan, as module owner, wdyt? bholley [1] https://wiki.mozilla.org/Modules/All#C.2B.2B.2FRust_usage.2C_tools.2C_and_style > dev-platform mailing list > dev-pl...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform >
Re: More Rust code	Kris Maglione	10/07/17 10:51	On Mon, Jul 10, 2017 at 08:29:59PM +1000, Nicholas Nethercote wrote: >I think this is an excellent trend, and I've been thinking about how to >accelerate it... > >- Interop with existing components can be difficult. IPDL codegen rust >bindings could be a big help. So far, this has been my biggest issue. Or, to be more specific, the lack of interop, combined with the lack of documentation for the current state of interop, or how to go about writing new Rust components, or guidelines for when and how to use it. In particular, every piece of code I've considered writing in Rust would have needed to interact with DOM and/or Spidermonkey and/or IPDL bindings, and the amount of C++ glue code I would have had to write seemed to outweigh the benefits of writing it in Rust. Combined with the fact that I would have needed to find and dig through various scattered mailing list posts and wiki pages, and then pester a bunch of people by email or IRC just to get started, I've always given up the idea pretty quickly. That said, there are things that I would love to be able to write in Rust rather than in C++, so if there's anything we can do to improve this situation, I'm very much on board. -Kris
Re: More Rust code	zbran...@mozilla.com	10/07/17 11:04	One more thought. There's a project that fitzgen told me about that aims to allow for components to communicate between JS and Rust using Streams. If we could get to the point where instead of WebIDL/XPIDL we could just plug streams between JS/CPP and Rust in Gecko, I believe the scope of Gecko components that can be written in Rust would skyrocket. zb.
Re: More Rust code	smaug	10/07/17 11:19	FWIW, that sounds rather heavy weight to me, and we should be optimizing everything.
Re: More Rust code	Xidorn Quan	10/07/17 16:49	On Mon, Jul 10, 2017, at 08:29 PM, Nicholas Nethercote wrote: > Firefox now has multiple Rust components, and it's on track to get a bunch more. See https://wiki.mozilla.org/Oxidation for details.> > I think this is an excellent trend, and I've been thinking about how to accelerate it. Here's a provocative goal worth considering: "when writing a new compiled-code component, or majorly rewriting an existing one, Rust should be considered / preferred / mandated."> > What are the obstacles? Here are some that I've heard. > > - Lack of Rust expertise for both writing and reviewing code. We have some pockets of expertise, but these need to be expanded greatly. I've heard that there has been some Rust training in the Paris and Toronto offices. Would training in other offices (esp. MV and SF, given their size) be a good idea? What about remoties?> > - ARM/Android is not yet a Tier-1 platform for Rust. See https://forge.rust-lang.org/platform-support.html and https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some details.> > - Interop with existing components can be difficult. IPDL codegen rust bindings could be a big help.> > - Compile times are high, especially for optimized builds. > > Anything else? Tooling around debug info in Rust code still isn't great. The first thing comes to my mind is crash reports. It currently doesn't always include useful panic message from Rust, see for example [1] and [2]. Also for Stylo, we generate lots of code (including using bindgen and mako template system, bindgen is usually fine, but the code generated from template can contain lots of code logic), and when the crash happens inside generated code, it is pretty hard to understand what's going wrong, because there is no useful source link, see for example [3]. There are also issues from Rust side. I've always been using an optimized debug build locally (because that runs faster), and sometimes use Visual Studio to investigate issues. C++ code works fine with this configuration, but in Rust code, I cannot see any variable information. Stack backtrace seems to work fine to me, though. [1] https://crash-stats.mozilla.com/report/index/2abff06f-d969-4ba5-845b-a98410170708[2] https://crash-stats.mozilla.com/report/index/03718a9c-9d98-4832-b8a6-026220170706[3] https://crash-stats.mozilla.com/report/index/6b7d1d78-8418-47ef-bee9-f49c20170710 Besides tooling, not just compile time, but also the link time is high. Specifically, I suspect our linking strategy for Rust code doesn't interact well with MSVC's incremental linker, which slows down this final sequential step. For that, I just did an experiment. In my local repo, when I remove all linking-related files from objdir/toolkit/library (to do a fresh linking), it takes ~1.7min for linking. When I touch one cpp file (in my experiment, layout/style/nsCSSAnonBoxes.cpp which is a reasonable small cpp file I happen to open), it takes 6s to link, and size of xul.ilk (Incremental Linker File) increases by 856KB. When I touch one rs file (in my experiment, servo/components/gecko_bindings/sugar/style_complex_color.rs which is a reasonable small rs file I happen to be familiar with), it takes ~2min to link (even longer than the fresh linking!) in addition to the compile time, and size of xul.ilk increases by 27.44MB. And after that, when I touch that cpp file again, it takes ~2.1min to link and size of xul.ilk increases by 54.64MB this time. I suspect the more times you touch rs files, the longer linking would take, and I guess this is because we link all Rust code into a single library, and then static link it into xul.dll, which makes the linker do lots of wasteful work. There could also be some linker bug involves, though. To mitigate this, we can probably avoid incremental linking when Rust code is touched. I also wonder if it is possible to avoid linking the whole gkrust into xul.dll at all and see if that makes things better. It would also be good to see if Rust developers have any advice for this situation. - Xidorn
Re: More Rust code	Nicholas Nethercote	10/07/17 17:37	On Mon, Jul 10, 2017 at 11:41 PM, smaug <sm...@welho.com> wrote: > > - Interop with existing components can be difficult. IPDL codegen rust >> bindings could be a big help. >> > > ipdl? or do you mean idl? or perhaps webidl? > bkelly suggested that item to me on Twitter :) Probably all three of those are relevant! Nick
Re: More Rust code	Nicholas Nethercote	10/07/17 17:39	On Tue, Jul 11, 2017 at 2:33 AM, Bobby Holley <bobby...@gmail.com> wrote: > > In other words, I think we should encourage people to consider Rust for > new code, but also make sure that we have a thoughtful and informed > discussion about whether a use-case makes sense and how best to support it. > Definitely! I deliberately described my "Rust should be considered / preferred / mandated" goal as "provocative" :) We should think big, and then put engineering effort in where it makes sense and will have strong benefits. > Right now we have a module that's intended to govern decisions on these > issues [1]. However, the peer list for that module seems oriented around > C++ and build system expertise, and is sparse on people who have been > deeply involved in the Rust integration efforts over the past year. > > So we could expand that group with expertise to tackle Rust integration > issues. Or it could charter a committee of such experts to do so. Or some > combination. Either way, I think we want some amount of steering here, and > I can think of a handful of names whose input is likely required to get it > right. > That module's name is "C++/Rust usage, tools, and style", and its description is "Aspects of C++ use such as language feature usage, standard library versions/usage, compiler/toolchain versions, formatting and naming style, and aspects of Rust use as needs arise" To me that feels orthogonal to the notion of encouraging the adoption of more Rust components in Firefox. That kind of encouragement feels more project-wide, and so IMO would fall under the "mozilla-toplevel" component. IIRC Brendan resigned ownership of that module a while back but he's still listed as the owner at https://wiki.mozilla.org/Modules/All#mozilla-toplevel. So that leaves things in an uncertain state. If I were the owner of that module I would consider implementing a policy something like the following: "When a person writes a new compiled-code component, or majorly rewrites an existing one, they should strongly consider writing it in Rust instead of C or C++. Such a decision will depend on the nature of the component. Components that are relatively standalone and have simple interfaces to other components are good candidates to be written in Rust, as are components such as parsers that process untrusted input." It could do with some tweaking and feedback, but in principle I don't see why such a policy couldn't be implemented right now. I don't know where this policy would be written down, though! Nick
Re: More Rust code	Nicholas Nethercote	10/07/17 18:03	On Tue, Jul 11, 2017 at 9:48 AM, Xidorn Quan <m...@upsuper.org> wrote: > Tooling around debug info in Rust code still isn't great. > The good news is that tooling issues are generally among the easier ones to solve. Even the existing Rust code that is present in Firefox will require at least some of these things to be fixed. > The first thing comes to my mind is crash reports. It currently doesn't > always include useful panic message from Rust, see for example [1] and [2]. > I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1379857 for this. It's blocking https://bugzilla.mozilla.org/show_bug.cgi?id=1348896, which is the Rust crash tracking bug. For that, I just did an experiment. In my local repo, when I remove all > linking-related files from objdir/toolkit/library (to do a fresh linking), > it takes ~1.7min for linking. > > When I touch one cpp file (in my experiment, layout/style/nsCSSAnonBoxes.cpp > which is a reasonable small cpp file I happen to open), it takes 6s to > link, and size of xul.ilk (Incremental Linker File) increases by 856KB. > > When I touch one rs file (in my experiment, servo/components/gecko_binding > s/sugar/style_complex_color.rs which is a reasonable small rs file I > happen to be familiar with), it takes ~2min to link (even longer than the > fresh linking!) in addition to the compile time, and size of xul.ilk > increases by 27.44MB. And after that, when I touch that cpp file again, it > takes ~2.1min to link and size of xul.ilk increases by 54.64MB this time. > > I suspect the more times you touch rs files, the longer linking would > take, and I guess this is because we link all Rust code into a single > library, and then static link it into xul.dll, which makes the linker do > lots of wasteful work. There could also be some linker bug involves, though. > > To mitigate this, we can probably avoid incremental linking when Rust code > is touched. I also wonder if it is possible to avoid linking the whole > gkrust into xul.dll at all and see if that makes things better. It would > also be good to see if Rust developers have any advice for this situation. > Would you mind filing a bug report for this issue? I've posted a link to this thread on the Rust Internals board: https://internals.rust-lang.org/t/dev-platform-discussion- on-getting-more-rust-code-into-firefox/5523 Nick
Re: More Rust code	Chris Peterson	10/07/17 18:04	On 7/10/17 4:48 PM, Xidorn Quan wrote: > The first thing comes to my mind is crash reports. It currently doesn't always include useful panic message from Rust, see for example [1] and [2]. Also for Stylo, we generate lots of code (including using bindgen and mako template system, bindgen is usually fine, but the code generated from template can contain lots of code logic), and when the crash happens inside generated code, it is pretty hard to understand what's going wrong, because there is no useful source link, see for example [3]. > There are also issues from Rust side. I've always been using an optimized debug build locally (because that runs faster), and sometimes use Visual Studio to investigate issues. C++ code works fine with this configuration, but in Rust code, I cannot see any variable information. Stack backtrace seems to work fine to me, though. > [1]https://crash-stats.mozilla.com/report/index/2abff06f-d969-4ba5-845b-a98410170708[2]https://crash-stats.mozilla.com/report/index/03718a9c-9d98-4832-b8a6-026220170706[3]https://crash-stats.mozilla.com/report/index/6b7d1d78-8418-47ef-bee9-f49c20170710 Looking at those crash reports' signatures, we should probably add `core::option::expect_failed` and `core::str::slice_error_fail` to Socorro's list of function names to ignore [1]. Should Socorro ignore all Rust core::* or std::* function names when searching the backtrace for a useful signature? [1] https://github.com/mozilla-services/socorro/tree/master/socorro/siglists#signatures-utilities-lists
Re: More Rust code	Chris Peterson	10/07/17 18:04	On 7/10/17 4:48 PM, Xidorn Quan wrote: > The first thing comes to my mind is crash reports. It currently doesn't always include useful panic message from Rust, see for example [1] and [2]. Also for Stylo, we generate lots of code (including using bindgen and mako template system, bindgen is usually fine, but the code generated from template can contain lots of code logic), and when the crash happens inside generated code, it is pretty hard to understand what's going wrong, because there is no useful source link, see for example [3]. > There are also issues from Rust side. I've always been using an optimized debug build locally (because that runs faster), and sometimes use Visual Studio to investigate issues. C++ code works fine with this configuration, but in Rust code, I cannot see any variable information. Stack backtrace seems to work fine to me, though. > [1]https://crash-stats.mozilla.com/report/index/2abff06f-d969-4ba5-845b-a98410170708[2]https://crash-stats.mozilla.com/report/index/03718a9c-9d98-4832-b8a6-026220170706[3]https://crash-stats.mozilla.com/report/index/6b7d1d78-8418-47ef-bee9-f49c20170710
Re: More Rust code	Bobby Holley	10/07/17 18:16	I think this is pretty uncontroversial. The high-level strategic decision to bet on Rust has already been made, and the cost of depending on the language is already sunk. Now that we're past that point, I haven't heard anyone arguing why we shouldn't opt for memory safety when writing new standalone code. If there are people out there who disagree and think they have the arguments/clout/allies to make the case, please speak up. The tradeoffs come when the code is less standalone, and we need to weigh the integration costs. This gets into questions like whether/how Rust code should integrate into the cycle collector or into JS object reflection, which is very much a technical decision that should be made by experts. I have a decent sense of who some of those experts might be, and would like to find the most lightweight mechanism for them to steer this ship.
Re: More Rust code	Julien Cristau	11/07/17 00:31	On Tue, Jul 11, 2017 at 3:04 AM, Chris Peterson <cpet...@mozilla.com> wrote: I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1379089 last week for core::option::expect_failed. Cheers, Julien
Re: More Rust code	Nicholas Nethercote	11/07/17 01:58	On Tue, Jul 11, 2017 at 11:15 AM, Bobby Holley <bobby...@gmail.com> wrote: > If I were the owner of that module I would consider implementing a policy >> something like the following: >> >> "When a person writes a new compiled-code component, or majorly rewrites >> an existing one, they should strongly consider writing it in Rust instead >> of C or C++. Such a decision will depend on the nature of the component. >> Components that are relatively standalone and have simple interfaces to >> other components are good candidates to be written in Rust, as are >> components such as parsers that process untrusted input." >> > > I think this is pretty uncontroversial. The high-level strategic decision > to bet on Rust has already been made, and the cost of depending on the > language is already sunk. Now that we're past that point, I haven't heard > anyone arguing why we shouldn't opt for memory safety when writing new > standalone code. If there are people out there who disagree and think they > have the arguments/clout/allies to make the case, please speak up. > As Gibson said: "The future is already here — it's just not very evenly distributed." The paragraph you wrote is obvious to you, but I suspect it's not obvious to everyone -- Mozilla has a lot of contributors. There may still be some Firefox developers who think that Rust something that other people do, and that isn't relevant to them or their component(s). There may be some Firefox developers who are interested in Rust, but feel intimidated or uncertain about using it. There may be some developers who are keen to use Rust, but haven't realized that we are past the experimental stage. Nick
Re: More Rust code	Gabor Krizsanits	11/07/17 03:16	On Mon, Jul 10, 2017 at 7:51 PM, Kris Maglione <kmag...@mozilla.com> wrote: > Combined with the fact that I would have needed to find and dig through > various scattered mailing list posts and wiki pages, and then pester a > bunch of people by email or IRC just to get started, I've always given up > the idea pretty quickly. > > This is by far the biggest obstacle for me. I guess the right approach here is to take the time to walk someone through from the very beginning and document the areas where it was not trivial how to progress (talking about gecko development in Rust not general Rust). And iterate that a few times. The lack of experience in the area and the sub-optimal tooling will result a huge overhead in developer time for anyone new in this area especially for remote contributors. Trying to minimize that overhead is important. Convincing managers to encourage developers to pay that overhead is also a requirement (tight deadlines will not help there). I have been flooded with work for a while, and it has been difficult to find more time for improving my Rust skills in general. Encouraging people to pick up Rust related goals and make it a priority to learn more Rust would be also important. Gabor
Re: More Rust code	Nicholas Nethercote	11/07/17 03:28	https://wiki.mozilla.org/Oxidation#Documentation has some documentation about how Rust code should be incorporated into Firefox. I don't know how accurate and thorough it is, but it looks like a good starting point. Nick On Tue, Jul 11, 2017 at 8:16 PM, Gabor Krizsanits <gkriz...@mozilla.com> wrote:
Re: More Rust code	Joshua Cranmer ��	11/07/17 05:46	On 7/10/17 5:29 AM, Nicholas Nethercote wrote: > - Interop with existing components can be difficult. IPDL codegen rust > bindings could be a big help. Rust's C++ interop story is absolutely atrocious. The FFI basically runs on C ABI, even though Rust and C++ have some similar concepts that could be exposed more cleanly (e.g., this parameters, even mapping &str or &[T] to appropriate semantics) without forcing such a step. Bindgen works a little, but really only for calling C++ from Rust, and only if the C++ code is simple enough--if the code includes headers of complexity doom (hi, std::string!), it really ends up being a game of "how can I force bindgen to ignore enough that I get access to what I need." XPIDL and WebIDL express only a subset of functionality, so they're theoretically easier to support than "generic C++17." However, XPIDL is profoundly unnatural in C++ code (the array syntax is horrendous, particularly if you want to start passing string arrays), as well as being limited in some of its vocabulary. WebIDL is more generic, but the bindings produces source code, which means that the ABI isn't exactly standardized for easy cross-language calling. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist
Re: More Rust code	Ben Kelly	11/07/17 06:27	(Picking a somewhat random response to reply here...) It would be really nice to have IPDL codegen support for rust. I considered using rust for my current task, but decided not to since I would've had to build even more boilerplate to work with IPC c++ actors. I would've ended up with more glue code than actual functional code. Another advantage of building rust IPDL codegen targets would be that we could build service oriented code in the parent process in rust. This would be an incremental improvement that could offer additional safety in the parent process without having to tackle webidle, cycle collection, etc. In particular, PBackground parent actors already cannot interface with xpcom since they are OMT and would be a good candidate here. Anyway, I think fixing these kinds of integration pain points would accelerate our ability to use rust in gecko. I would hesitate to make any kind of mandates until these issues are addressed. Thanks. Ben
Re: More Rust code	smaug	11/07/17 06:37	How is the performance when crossing Rust <-> C++ boundary? We need to make everything faster, not slower. If I understood emilio's explanation on IRC correctly having the performance of an inlined (C++) function requires handwritten rust bindings to access member variables of some C++ object. That doesn't sound too good - hard to maintain and possibly easy to forget to optimize. I don't claim to understand anything about the current setup, but has anyone written down what would be needed to have fast and easy to maintain Rust <-> C++ boundary in such way that also memory handling is easy to manage (aka, how to deal with CC/GC). I think it would be better to sort out this kind of low level issues rather soon before we have too much Rust code in tree, or perhaps we won't see much Rust usage before those issues are sorted out. (I'm looking this all from DOM point of view, where pretty much all the objects need to be cycle collectable JS holders, but perhaps Rust would fit better in code outside DOM)
Re: More Rust code	Alexis Beingessner	11/07/17 06:47	I'm currently trying to improve the C++ <-> Rust FFI story a bit. I just posted a draft proposal to add a mode to rustc that has it output the ABI details of all public types: https://internals.rust-lang.org/t/stabilizing-a-machine-readable-zprint-type-sizes/5525 This would theoretically reduce our maintenance/conversion burden at the FFI boundary.
Re: More Rust code	smaug	11/07/17 07:14	On 07/11/2017 04:27 PM, Ben Kelly wrote: > On Tue, Jul 11, 2017 at 4:57 AM, Nicholas Nethercote <n.net...@gmail.com >> wrote: > >> On Tue, Jul 11, 2017 at 11:15 AM, Bobby Holley <bobby...@gmail.com> >> wrote: >> >>> If I were the owner of that module I would consider implementing a policy >>>> something like the following: >>>> >>>> "When a person writes a new compiled-code component, or majorly rewrites >>>> an existing one, they should strongly consider writing it in Rust >> instead >>>> of C or C++. Such a decision will depend on the nature of the component. >>>> Components that are relatively standalone and have simple interfaces to >>>> other components are good candidates to be written in Rust, as are >>>> components such as parsers that process untrusted input." >>>> >>> >>> I think this is pretty uncontroversial. The high-level strategic decision >>> to bet on Rust has already been made, and the cost of depending on the >>> language is already sunk. Now that we're past that point, I haven't heard >>> anyone arguing why we shouldn't opt for memory safety when writing new >>> standalone code. If there are people out there who disagree and think >> they >>> have the arguments/clout/allies to make the case, please speak up. >>> >> >> As Gibson said: "The future is already here — it's just not very evenly >> distributed." >> >> The paragraph you wrote is obvious to you, but I suspect it's not obvious >> to everyone -- Mozilla has a lot of contributors. There may still be some >> Firefox developers who think that Rust something that other people do, and >> that isn't relevant to them or their component(s). There may be some >> Firefox developers who are interested in Rust, but feel intimidated or >> uncertain about using it. There may be some developers who are keen to use >> Rust, but haven't realized that we are past the experimental stage. >> > > (Picking a somewhat random response to reply here...) > > It would be really nice to have IPDL codegen support for rust. I > considered using rust for my current task, but decided not to since I > would've had to build even more boilerplate to work with IPC c++ actors. I > would've ended up with more glue code than actual functional code. > > Another advantage of building rust IPDL codegen targets would be that we > could build service oriented code in the parent process in rust. This > would be an incremental improvement that could offer additional safety in > the parent process without having to tackle webidle, cycle collection, > etc. In particular, PBackground parent actors already cannot interface > with xpcom since they are OMT and would be a good candidate here. OMT doesn't mean no XPCOM. You're thinking xpconnect. > > Anyway, I think fixing these kinds of integration pain points would > accelerate our ability to use rust in gecko. I would hesitate to make any > kind of mandates until these issues are addressed. > > Thanks. > > Ben >
RE: More Rust code	Jim Mathies	11/07/17 07:34	What's the debugging situation look like for Windows developers? I've heard it's pretty painful. Can we step through rust code using common tools (WinDBG/Visual Studio)? Jim -----Original Message----- From: dev-platform [mailto:dev-platform-bounces+jmathies=mozil...@lists.mozilla.org] On Behalf Of Nicholas Nethercote Sent: Monday, July 10, 2017 5:30 AM To: dev-platform <dev-pl...@lists.mozilla.org>; firefox-dev <firef...@mozilla.org> Subject: More Rust code Hi, Firefox now has multiple Rust components, and it's on track to get a bunch more. See https://wiki.mozilla.org/Oxidation for details. I think this is an excellent trend, and I've been thinking about how to accelerate it. Here's a provocative goal worth considering: "when writing a new compiled-code component, or majorly rewriting an existing one, Rust should be considered / preferred / mandated." What are the obstacles? Here are some that I've heard. - Lack of Rust expertise for both writing and reviewing code. We have some pockets of expertise, but these need to be expanded greatly. I've heard that there has been some Rust training in the Paris and Toronto offices. Would training in other offices (esp. MV and SF, given their size) be a good idea? What about remoties? - ARM/Android is not yet a Tier-1 platform for Rust. See https://forge.rust-lang.org/platform-support.html and https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some details. - Interop with existing components can be difficult. IPDL codegen rust bindings could be a big help. - Compile times are high, especially for optimized builds. Anything else? Nick _______________________________________________ dev-platform mailing list dev-pl...@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: More Rust code	Nicolas B. Pierron	11/07/17 08:46	Hi, (Answering privately until I get more manager intent to get this project as part of any long term roadmap) For your information, I am currently looking, on my spare time, at one way to get SpiderMonkey to be rewritten in Rust. () Currently SpiderMonkey is one big monolithic pieces of code. Everything in SpiderMonkey is entangled in a way which does not give much room for making simple components without thousands of bindings going both ways. Cretonne might be the exception in this haystack, but at the cost of duplicating a lot of code which already exists (which makes me even more worried security wise, even if it is written in Rust). SpiderMonkey would never switch to Rust if we have no way to see any fast-execution outcome at the end of the integration. Currently the burden of maintaining the Jit is huge. We see the cost of the integration every time we have to fix a performance bug, and every time a new language feature is added in the language. Based on these claims, I figured that there is one thing that we should rewrite first, even before the interpreter, this would be the JIT and the GC. I think the Rust community is likely to work on a GC integration in the next few years, but I do not forsee any good Jit. So I started working on a Rust project to solve the maintainability issues, the performance issues, and to make a Jit engine which would work independently of the runtime (library functions, object model, GC). I intent to release it (silently) as soon as I get a working prototype. If you know anybody who my help pushing this project forward, I would be happy to contact them privately. () From what I understand, saying that publicly would result in me getting badly evaluated by higher management layers (as it already happened with a bonus cut) for not focusing on Mozilla goals, and that even if this is a spare time project. -- Nicolas B. Pierron
Re: More Rust code	Nicolas B. Pierron	11/07/17 08:47	On 07/11/2017 03:46 PM, Nicolas B. Pierron wrote: > (Answering privately until I get more manager intent to get this project as > part of any long term roadmap) Or not so privately after all … :( -- Nicolas B. Pierron
Re: More Rust code	Brian Birtles	11/07/17 16:05	On Tue, Jul 11, 2017 at 11:34 PM, Jim Mathies <jmat...@mozilla.com> wrote: > What's the debugging situation look like for Windows developers? I've heard > it's pretty painful. Can we step through rust code using common tools > (WinDBG/Visual Studio)? > You can set breakpoints and step through rust code using Visual Studio. Sometimes you can make sense of values in the Locals/Auto/Watch panels too. (Perhaps someone has worked out how to write autoexp.dat-like debug visualizers for rust?) Then again, unoptimized rust code is very slow so you'll probably find you're building with optimizations on making most of those values useless. The biggest problem I find is that even with RUST_BACKTRACE=1 I rarely get call stacks when rust code panics. I normally have to rebuild without the panic="abort" configuration[1] and then attach a debugger in order to see the failing call stack. I suspect this only applies to rust code called from C++ (and not, e.g., when debugging Servo), and at least once I have actually seen a call stack from a panic dumped to the console, so I'm not sure what's going on there. Then there's the incredibly long build/link times. [1] https://pastebin.mozilla.org/9026883
Re: More Rust code	Mike Hommey	11/07/17 16:31	This sounds like bug 1373881. Mike
Re: More Rust code	Gabriele Svelto	12/07/17 01:37	On 10/07/2017 12:29, Nicholas Nethercote wrote: > Hi, > What are the obstacles? Here are some that I've heard. > [...] > Anything else? In the past year I wrote two tools (minidump-analyzer and pingsender) that ship with Firefox but are separate executables so both were good candidates for being written in Rust in the first place. I implemented both in C++ because of different issues: - The minidump-analyzer relies on Google Breakpad machinery that is currently C++. It's being rewritten in Rust but it's not there yet and since we needed it to quickly get more crash information going the Rust route would have simply taken too long. It is a good candidate for being rewritten in Rust in the coming 12 months though. - The pingsender talks to our telemetry infrastructure so needs HTTP/HTTPS functionality but without linking with libxul. In this case if I had a crate that offered HTTP functionality I would have gone straight to Rust but alas it wasn't there. So instead it's a blob of platform-specific C++ that dlopen()s libcurl on Mac and Linux and relies on WININET on Windows. Gabriele
Re: More Rust code	Eric Rahm	12/07/17 16:06	Interesting points. - using breakpad - was the problem that creating wrappers to access the c/c++ code was too tedious? Could bindgen help with that, if not it would be interesting gather some details about why it wouldn't work and file bugs against it. - pingsender - was something like https://hyper.rs/ not around when you were working on it or is this a case of finding the things you want can be difficult in rust-land? Either way it might be a good idea for us to put together a list of "sanctioned" crates for various tasks. -e On Wed, Jul 12, 2017 at 1:36 AM, Gabriele Svelto <gsv...@mozilla.com> wrote:
Re: More Rust code	Kyle Machulis	12/07/17 16:27	So, I've been kinda working toward the "more rust in gecko" goal, in my own space that just happened to be a good sandbox for it. Last year I was brought over to help on WebAuthn, which needs access to USB HID for things like Yubikeys. I decided to write the lowest level cross platform parts of this in Rust, because it would give us safety in a place we desperately need it (the syscall/external bindings level). Tim Taubert and JC Jones are now leading this, and we're starting to look at landing the rust parts of this project soon. I had originally planned on taking this strategy for WebMIDI, but that project has unfortunately stalled due to priorities. My goal is doing this is being able to bring in rust from the bottom-up, on a part of the system that won't be part of any tight loops, just so we can start to get an idea of what this will look like for some (not all) systems. Right now, for webauthn, we go IPDL -> C++ -> C Bindings -> Rust. Once we have this solid, the plan is to start chipping our way up the chain to the IPDL boundary, though there are definitely obstacles to that, like our calls to NSS. Note that this project is certainly not reflective of the wide scope of bringing rust everywhere in the browser: - This specifically touches system components that are in C, so we don't have to deal with C++ binding issues yet. - It's not a particularly performance sensitive API, since human activity is in the loop - It doesn't really touch many other parts of the system at the moment. This could change depending on other factors in the project, movement in the spec, etc. WebAuthn isn't the only sandbox we could do this in. There are other portions, related to hardware or the HAL, either already written or still waiting to be implemented (gamepad, webmidi, etc), that we can use to try out some simple ideas while work is happening on other parts. There's also been talking of seeing if we could possibly share code with Servo this way, which is something I've been planning on trying via the Gamepad API. Once this project was over I was planning on talking to the rest of the team about doing a writeup, and would still love to see a blog post happen when it's all done and shipped, but it seems like making it part of this conversation would also be helpful. qDot
Re: More Rust code	Selena Deckelmann	13/07/17 06:27	[moving dev-platform and firefox-dev to Bcc] What I've heard in this thread is that we have a few blockers, some pain points and bugs to file regarding more Rust integration with Firefox. Nick -- please ensure those bugs get filed, and a meta-bug entitled "Make the developer experience for Firefox + Rust great". There are some issues that Stylo needs resolved before we ship, and those bugs should probably block a Stylo tracking bug. Once we have that in place, we can work together to prioritize them. Whatever is blocking Stylo would be high priority Rust/tooling work. Also, the Stylo team is considering what performance related work will be the highest priority. We also have significant technical problems to solve regarding integration between C++/Rust/JS components. Bobby raised the issue of our decision making process around where to focus efforts around Rust integration. Ehsan is on vacation this week, so let’s pause this discussion and have a meeting next week between the module peers and those who have been working on the Stylo integration, because they have to most relevant experience. I’ll put that together and report out. -selena
Re: More Rust code	smaug	13/07/17 06:27	On 07/11/2017 04:27 PM, Ben Kelly wrote: > On Tue, Jul 11, 2017 at 4:57 AM, Nicholas Nethercote <n.net...@gmail.com
Re: More Rust code	Enrico Weigelt, metux IT consult	13/07/17 06:27	On 10.07.2017 12:29, Nicholas Nethercote wrote: Hi folks, > Firefox now has multiple Rust components, and it's on track to get a > bunch more. See https://wiki.mozilla.org/Oxidation for details. I wonder which of the pressing problems (eg. massing resource wasting, bad performance, horribly complicated build systems, hard to maintain source tree, too much in one application, etc), are actually improved by yet another language. No idea, which fancy features of rust you're so keen on, but most of what I've seen so far, has already been done in ADA (hmm, does rust have domain types and compile-time constraints?) By the way, just in case nobody noticed: recent Debian stable doesn't have rustc (for good reasons, as it's still unstable), so pretty much no chance of recent moz in Debian world (I'd guess the same w/ other stable distros) - limited to bleeding-edge distros for the next years. Rustc is so alpha, that it doesn't even compile (eg. wrong guesses on the target architecture, neeed bleeding-edge cmake, ...) I'd rather concentrate on consequent cleanups, starting with dropping bundled 3rdparty libraries, dropping various misfeatures, esoteric build systems, move audio/video stuff to separate tools that are really made for that (eg. ffmpeg or gst - yes, that gives you hw acceleration for free), move out mail protocol handling to separate service (mailfs is your friend) - same for address books, dictionaries, bookmarks finally an lightweight interface to run extensions as separate programs. When I look back the past decade, I don't recall any major improvement (okay, don't remember whether session management did exist back then), but it just got slower and slower, even w/ magnitudes faster machines since back then. Will moving to rust improve anything of these areas ? > - Lack of Rust expertise for both writing and reviewing code. We have > some pockets of expertise, but these need to be expanded greatly. Wouldn't that be a yet another big argument for keeping rust stuff optional, until that problem is settled (and rust is matured enough so become an option for production systems ?) > - ARM/Android is not yet a Tier-1 platform for Rust. See > https://forge.rust-lang.org/platform-support.html and > https://internals.rust-lang.org/t/arm-android-to-tier-1/5227 for some > details. So, is android also dropped out of the list of supported platforms, just like all recent stable gnu/linux distros already are ? > - Compile times are high, especially for optimized builds. Compile times for C++ are already very high (c++ in general is very expensive to compile, and also very complicated to write good code). How much worse is that for rust ? --mtx
Re: More Rust code	Mike Hommey	13/07/17 14:04	On Wed, Jul 12, 2017 at 04:06:39PM -0700, Eric Rahm wrote: > Interesting points. > > - using breakpad - was the problem that creating wrappers to access > the c/c++ code was too tedious? Could bindgen help with that, if not it > would be interesting gather some details about why it wouldn't work and > file bugs against it. > - pingsender - was something like https://hyper.rs/ not around when > you were working on it or is this a case of finding the things you want can > be difficult in rust-land? Either way it might be a good idea for us to put > together a list of "sanctioned" crates for various tasks. Note that pingsender is a small self-contained binary. I'm afraid writing in in rust would make it very much larger. Mike
Re: More Rust code	Nicholas Nethercote	13/07/17 14:50	On Tue, Jul 11, 2017 at 11:03 AM, Nicholas Nethercote < n.net...@gmail.com> wrote: > > >> The first thing comes to my mind is crash reports. It currently doesn't >> always include useful panic message from Rust, see for example [1] and [2]. >> > > I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1379857 for this. > It's blocking https://bugzilla.mozilla.org/show_bug.cgi?id=1348896, which > is the Rust crash tracking bug. > jryans just landed a fix for bug 1379857. Turns out that Rust panic messages weren't being captured by crash reports in content processes. Nick
Re: More Rust code	Gabriele Svelto	15/07/17 07:03	On 13/07/2017 01:06, Eric Rahm wrote: > * using breakpad - was the problem that creating wrappers to access > the c/c++ code was too tedious? Could bindgen help with that, if not > it would be interesting gather some details about why it wouldn't > work and file bugs against it. There were multiple reasons: I had no experience in writing Rust bindings and we wanted to have it ready quickly; additionally we're in the process of rewriting this part of Breakpad functionality in Rust so we decided that it would all become Rust at a later point and writing bindings wouldn't have been a good use of the time. > * pingsender - was something like https://hyper.rs/ not around when > you were working on it or is this a case of finding the things you > want can be difficult in rust-land? Either way it might be a good > idea for us to put together a list of "sanctioned" crates for > various tasks. hyper.rs was already available but it's a big dependency and I didn't feel like pulling it into mozilla-central just for the pingsender. I believe we'll have HTTP functionality available from Rust code at some point in the future so when it's ready I might as well port the pingsender to it. As Mike said though the program is very small and self-contained at this point so it would be nice to keep it that way even after a Rust rewrite if possible. Gabriele