port HTTPS Everywhere to Electrolysis

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I'm unofficially now the maintainer of HTTPS Everywhere. We apparently
looked into porting the addon to Electrolysis in order to support
Fennec three years ago and then gave up and wept in despair (see
https://trac.torproject.org/projects/tor/ticket/2471). Some of our
users have been asking us to do this because there's no version of
HTTPS-E that supports mobile at all yet, so we probably should.

Can you folks at Mozilla provide any guidance on this? Would you
recommend porting the entire extension to Jetpack in the process?

EFF is pretty busy these days, so any help you can give us with this
would be very much appreciated!

- -Yan


- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSsi60AAoJENC7YDZD/dnszw4H/AxoT9QRAqelTRbZ5l/8Vesk
k4+Oqq41IYgyd9uqzHUHnXLbyvTOzTF882O9Y+41+Lg+JFfFty/es/FYp7tqTKkf
Rbfmv257TRC/3dBd5LTcndNQ1Kcdtycux+0qKvZVkD22DL8pgL4hxngFo/MeVgmG
yJxDpR96MAljVhsdTSFX3tu1s44AEk4UTNtB7Ls5c4yj0O8JemYtE6GGHB8Mp2Ah
1GTILBSoCZouJdZ/h8lRx5JU9Nbj2kyzHxQwvsPAatdsBRS2yKXwkaDGm5R2T7uW
VzQW/6QZ9bpZm2CEXAA+Do+xJFZFcBookk3ZLwZJk4dzK6mCdRV8s9JlZEw7VY8=
=PXXR
-----END PGP SIGNATURE-----

[Garrett Robinson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/18/2013 03:24 PM, Yan Zhu wrote:
> Can you folks at Mozilla provide any guidance on this? Would you
> recommend porting the entire extension to Jetpack in the process?

We can definitely provide guidance on this process. Getting addons to
work in e10s is an ongoing project, but we definitely want popular
extensions like HTTPS Everywhere to work before we turn e10s on by
default. There is currently work being done to get the Top 10 addons
on addons.mozilla.org (Ad Block Plus, DownThemAll, etc.) working with
e10s.

If porting the entire thing to Jetpack is something you're
considering, it would be a good idea in the context of e10s. AIUI, one
of the goals is to get Jetpack to be e10s-safe so we can limit current
and future extensions use of components that are not e10s safe, or
cannot be without specialized workarounds for each component.

- -Garrett
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSsjHQAAoJEEZPConT75yuwE8P/iiO5gk3K17pNcane/OVPnGm
eE0aP4iJmnD85/uKQo/iOWuBoJG8AuEf5uugd5DJK4I3PY1YBtRSvPyU/YbhTCdU
L71ZhowPi/2YnkJ57C8TT/IcNPTew1PM4kkr9MjDK1M5hpt4aqmI/FrNUiPL9xYi
7Vx2WOFzEdsaW/7/8LJVqFr2X4j3zvyddAqsRCQnG8RpobGgaL38X/Wbq9dUuVCr
s9P8kqmySZOXVI0y+b6GUTppWs3CL8RTHDS/+DkGugsWJRY85vYVazUzfTRhBY8Z
YZNL0wUXvHe8J23B7M48Dyy858rOucK9whFrT8hGpyF92OhRWOojmkVeHW2MxowK
aSFSywGfaMM4KFmf6p3bGMO8PfAdxdUb1oj3KgmmpFSL0N98+gl2Adk4KyMuPZh/
psXRqR7MF0ytRFFoJvMEUrDd89Q+ODtw999PHTz62wznPe/qLO4WV28fDF5Hw974
6V/SwjCe2myaqx6/tfL1Gtckn+kDW37piGlv3PLiM7ksCkygD8XMEJ1m8AhC15Ks
zQczLGl+62GgGJz3Dqis6yKzH5D8xZ7o5pEIzF3w5fKuAowclielPnqEmtQR46Ru
Zm/YRbQ0Bgdgf/17/9qRhIVTqGB5EO5/DHeMw0FvTygPMPFth+JZ1vBOu2W+9Vy9
lFVoXcZczVmHPM09ASXr
=DwDt
-----END PGP SIGNATURE-----

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/18/2013 03:37 PM, Garrett Robinson wrote:
> On 12/18/2013 03:24 PM, Yan Zhu wrote:
>> Can you folks at Mozilla provide any guidance on this? Would you
>> recommend porting the entire extension to Jetpack in the
>> process?
>
> We can definitely provide guidance on this process. Getting addons
> to work in e10s is an ongoing project, but we definitely want
> popular extensions like HTTPS Everywhere to work before we turn
> e10s on by default. There is currently work being done to get the
> Top 10 addons on addons.mozilla.org (Ad Block Plus, DownThemAll,
> etc.) working with e10s.
>

Oh hi, Garrett! :)

Some EFF folks claim that HTTPS Everywhere is probably in the top 10
Firefox addons by users, though we're not in AMO (yet).

> If porting the entire thing to Jetpack is something you're
> considering, it would be a good idea in the context of e10s. AIUI,
> one of the goals is to get Jetpack to be e10s-safe so we can limit
> current and future extensions use of components that are not e10s
> safe, or cannot be without specialized workarounds for each
> component.
>

I've been wanting to rewrite the entire thing in Jetpack for a while
just so that it's easier to maintain and develop in the future;
automatic compatibility with e10s would be an extra win. I have a
feeling it might be easier to port the Chromium version of HTTPS
Everywhere to Jetpack than the current Firefox version.

Can you foresee any reason not to port to Jetpack right now? Are
Jetpack addons compatible with TBB?

- -Yan
> -Garrett _______________________________________________
> dev-tech-electrolysis mailing list
> dev-tech-e...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-electrolysis

>

- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSsjXqAAoJENC7YDZD/dnseQIH/A2Z2Ul+hOYlpj0h16xG0DUA
HdBRE9svUBXHUiIkCX8YpTpoIV0t0gmn9VvL8vTZIJMrjc8gnIw+9hYOFSsLz1Fk
vU7CLMErntFFtpra4Z/Iz13C4SySogodfJ21LCuLhMALEhnQR2O+IFrW1TSEtPDe
RRgN4IfN7D3AshP0F7nC17X/VU+aq3cGGublM2alThwqKdqqjTLRkpcykhmdx5Ip
effOpIWCz2Eq4A5j6AqsQtxOOX27WKBEQvsMO4GXjUpKPOTesmUQtIQRuQvEnlxC
n1YpTfDuhB+QCTxGYdmiKBow6LLgaQgyhtl1e0Q1twSkkZdHjgMyRM7SsD11Ahw=
=YRug
-----END PGP SIGNATURE-----

[Garrett Robinson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I've been wanting to rewrite the entire thing in Jetpack for a
> while just so that it's easier to maintain and develop in the
> future; automatic compatibility with e10s would be an extra win. I
> have a feeling it might be easier to port the Chromium version of
> HTTPS Everywhere to Jetpack than the current Firefox version.

It also might get easier if we go through with a rewrite of the
Content Policy API, which we are discussing for Q1/Q2 of next year.
IIRC there are currently some issues with the Mixed Content Blocker
and HTTPS everywhere that would ideally be resolved by such a rewrite;
additionally, handling redirects would hopefully get a lot easier.

> Can you foresee any reason not to port to Jetpack right now? Are
> Jetpack addons compatible with TBB?

I can't see why not. Asked TBB devs, and they said "no idea, assume
yes." Could be issues as Jetpack is developed and updated in Firefox
releases that diverge from the ESR the TBB is based on.

Again, we are hoping to be able to synchronize the TBB with release
Firefox in the future. That is being actively worked on, but is not
guaranteed.

- -Garrett
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSskhMAAoJEEZPConT75yuW3gP+wfSVdmcSTlkpYX1tzxkrya+
HzeEgMD2tNjioFeYRG3MOU4kYL3uiXmjJRk0PcptJ0jwGlHnVh+HpJpiQl5lgEsv
nqUvXfm8k5DTOA3vH/Tt8f1SDiSCHgckwr2uPvRtMdHQmj0ao2qM/T7buH3uQ+DF
NQfcg7G1dt3g/EYsecwZSYX2ukrPEQpoGebx0j8utgVNC9DYQqBtS0ynT/CqcIPx
3vDYsXAzc9/raNFpV7oC8yuVrcL6woOkMUFXoj6wjcjJSi4pYS9F5C4P1aIAGyts
gfn9gPKTirg09dOK5o1jxwOZ4A4TUCP2qGfdnVb8h/LAKzQSIcY2Mp82EdXrHKr9
vt+XcJ2gZN0qDhkMMurCVG5fxPxo48r3YmMldVcZFBdQhv7bxsxZr/f1iUZ96fLE
4ZquSxM/Mhd2vbF39MU/jgRkzWgVgDO7uZHRfsOVqZM4qS3WiIx0PaKvYsV+RsJR
3BnJ9j1g+RO6wU/91cbnrb8IANSx0qlzpzN+oTs+H79Aa8vLU1A4HcLEj1vpFvle
NimPG3HajRY44Q/+mIA/FLxfCmRYMIwzhDq9JXmTgNYv5oFrlf23fMQxYmmf0P8k
Iv/2FOqa7bghqT3PYoQjEZigaXvTpo+b+KBC7U4tnHjxHLOi+55qDR2jhTt9K3Ap
dSupM3pL+j9536jomfJC
=Ycf4
-----END PGP SIGNATURE-----

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I asked Mike Perry, Tor dev and one of the original creators of HTTPS
Everywhere, and he had the following concerns:

"""
In terms of specific issues with HTTPS-Everywhere, the first thing to
check is that you get access to the "http-on-modify-request" observer,
and that you can use the "redirectTo" API call on the nsIHttpChannel
(should be the subject of the observer). With that, the core
functionality should be doable.

After that, the next most likely thing to break is the context menu in
the toolbar (called the "ApplicableList" in the code, and written by
Peter IIRC) from the context of this "http-on-modify-request" observer.
In a fully e10s-locked mode, you may have difficulty updating that UI
from the same e10s process that handles the http-on-modify-request
observer redirection. If JetPack somehow enforces this already, things
might get tricky. Or maybe Mozilla has provided JetPack APIs for this
exact case, and it will actually be easier in the long run to use their
mechanism.


The risk for TBB here is if Jetpack suddenly starts depending on
features only available in the Rapid Releases, and those features are
not backported to the ESR series, then we risk losing HTTPS-Everywhere
in TBB with this move. Most likely we'll be able to simply avoid using
the very shiniest new Jetpack APIs that rely on Rapid Release, but if
the underlying implementation of "stable" Jetpack APIs change in some
key way, we could end up needing to ship two versions of the addon,
packaged with different Jetpack versions.

The best way to determine this is probably to see how well supported
Firefox 17ESR was by Jetpack and Jetpack addons. Based on this random
forum post, it sounds like the ESR series are not carefully tested with
newer Jetpack releases, and we may in fact end up needing to use
multiple versions of Jetpack to support TBB and other ESR users:
https://forums.mozilla.org/addons/viewtopic.php?f=27&t=15007

We should probably ask people we know at Mozilla to make sure this sort
of thing won't happen in the future.


Longer term (wrt the e10s switchover), I am also worried about this talk
of eliminating extension access to arbitrary XPCOM components. One of
the things that makes Firefox a good target for prototyping crazy shit
is the flexibility afforded by the fact that most things are scriptable
by default.

"""


On 12/18/2013 05:13 PM, Garrett Robinson wrote:
>> I've been wanting to rewrite the entire thing in Jetpack for a
>> while just so that it's easier to maintain and develop in the
>> future; automatic compatibility with e10s would be an extra win.
>> I have a feeling it might be easier to port the Chromium version
>> of HTTPS Everywhere to Jetpack than the current Firefox version.
>
> It also might get easier if we go through with a rewrite of the
> Content Policy API, which we are discussing for Q1/Q2 of next
> year. IIRC there are currently some issues with the Mixed Content
> Blocker and HTTPS everywhere that would ideally be resolved by such
> a rewrite; additionally, handling redirects would hopefully get a
> lot easier.
>
>> Can you foresee any reason not to port to Jetpack right now? Are
>> Jetpack addons compatible with TBB?
>
> I can't see why not. Asked TBB devs, and they said "no idea,
> assume yes." Could be issues as Jetpack is developed and updated in
> Firefox releases that diverge from the ESR the TBB is based on.
>
> Again, we are hoping to be able to synchronize the TBB with
> release Firefox in the future. That is being actively worked on,
> but is not guaranteed.
>

> -Garrett _______________________________________________
> dev-tech-electrolysis mailing list
> dev-tech-e...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-electrolysis
>

- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSsz2JAAoJENC7YDZD/dnsKC4H/1ur4aY0zUc8p04QY4mOoQtg
Glol15/N2Y0yFjgL/tKDMYc0PrkwRuGu5Tf/uRAFZ4Hf7NMluZsZMKTVdKNJROgD
/SnufeAsAJS/cGUS+CY+7Gdm2xx0K4WGu6cagbmnPOPKH7ETlLSEFdjSqjtZWZJL
EAV7McT04H/3rRxYexCpTj7g5E4dR0/KbhRbC4RZTquQCH62sHA8J6CFv9bMi/3f
xGP2eHrt6Q+B3mF064242vBegTBlTSO8bpW0ig8R8NC/Pa4YXDHk0kQS+6nJMi2G
xb8zoN0nD4ZislCIydCwO/hS5MYj53wW5Sht6fMJ9F8IcLdSCzgLmf7Y/jJQnmc=
=hK5I
-----END PGP SIGNATURE-----

[Chris Peterson]

On 12/18/13, 3:55 PM, Yan Zhu wrote:
> I've been wanting to rewrite the entire thing in Jetpack for a while
> just so that it's easier to maintain and develop in the future;
> automatic compatibility with e10s would be an extra win.

hi Yan, Thanks for reaching out!

Most Jetpack addons should work in e10s, but rewriting HTTPS Everywhere
in Jetpack is not your only option. In fact, the Jetpack APIs might not
expose all the functionality you need. You may be able to port your
addon and still be backward compatible with pre-e10s Firefox.

Our e10s documentation on MDN is a little sparse, but I'm actively
working with Mozilla tech writers to update it.

The primary porting issue is separating code that needs to analyze page
content (content scripts in the content process) from code that needs to
run in the parent ("chrome") process. You can use Message Managers to
send asynchronous messages between chrome and content processes. For
special cases, you can use Cross-Process Object Wrappers (CPOW,
pronounced "ka-pow" ;) to send *synchronous* messages from the chrome
process to content processes.

Current MDN pages:
https://developer.mozilla.org/en-US/docs/The_message_manager
https://developer.mozilla.org/en-US/docs/Cross_Process_Object_Wrappers

ttaubert's 2011 blog post about e10s:
http://timtaubert.de/blog/2011/08/firefox-electrolysis-101/

Bill's recent blog post:
http://billmccloskey.wordpress.com/2013/12/05/multiprocess-firefox/

btw, the current version of Firefox for Android no longer uses mobile
e10s. It supports some desktop plugins with minor porting. For now, the
e10s team is focusing exclusively on e10s for desktop Firefox.

When you have more questions, please free to ask detailed technical
questions on this mailing list. e10s developers can also be reached in
the #e10s IRC channel on irc.mozilla.org. We are especially interested
to get feedback from addon developers about how we can improve our e10s
APIs and documentation to make porting easier.


chris

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/19/2013 04:37 PM, Chris Peterson wrote:
> In fact, the Jetpack APIs might not expose all the functionality
> you need.

Does this imply that some XPCOM components will not be available in
Jetpack in the future? As I understand it, I can access arbitrary
XPCOM components right now through the chrome package, so in theory
any addon made with the old API's can be ported to Jetpack. It would
be really great if you guys kept this feature.

> btw, the current version of Firefox for Android no longer uses
> mobile e10s. It supports some desktop plugins with minor porting.
> For now, the e10s team is focusing exclusively on e10s for desktop
> Firefox.

Really? Could someone check whether HTTPS Everywhere works in Fennec
with this change to install.rdf?
https://gitweb.torproject.org/pde/https-everywhere.git/commitdiff/3eba359e521d6242b73ec93582fe09984162371f

Thanks,
Yan

>
> When you have more questions, please free to ask detailed
> technical questions on this mailing list. e10s developers can also
> be reached in the #e10s IRC channel on irc.mozilla.org. We are
> especially interested to get feedback from addon developers about
> how we can improve our e10s APIs and documentation to make porting
> easier.
>
>

> chris _______________________________________________

> dev-tech-electrolysis mailing list
> dev-tech-e...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-electrolysis
>

- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSs5mTAAoJENC7YDZD/dnsXOkIALJ4LdGzxWk0LKFIkPLEw3eo
/vWyvdLsCZQOwhfztrWTCQSpF306EyDWyIMHvpKY5fN8qGOcVRLGVaUSLc5gfrYU
/JQi67FcgweBbr0t7WFPRIGHIMdoy61mYAyGGrBx0rb0LPt68EFO7uoUypI5M12m
9tA1A2CutPvHQsmcJREsna3NV6bY4wWaZLOkPCDNmYCurcSNjL35odgwY/mc8qg/
wG8V2GATjpwP6mDFSR5NA90IfB7x8QFJioVf39RxtTii4W1r0SXgIX/MuCZubCBq
dyOSrLsrBKR1v8rUadgoZrtxWdbpYZU5MyKprojFvUjvTjFGLYY3MafQIDa31dI=
=uPHA
-----END PGP SIGNATURE-----

[Bill McCloskey]

Hi Yan. Thanks for getting in touch about this. As Garrett mentioned, we're
still exploring the best ways for making add-ons work in electrolysis, so
it's difficult to provide definite advice right now.

I looked at the code for HTTPS Everywhere, and it looks like it is one of
the trickier add-ons to support. I'm afraid I can't recommend switching to
using Jetpack. Jetpack doesn't have APIs to support the interfaces you're
using. For the http-on-* stuff it does allow you to register observers, but
you're still basically using the same XPCOM API as before. I don't think it
supports content policies at all; for those I think you just need to use
the XPCOM APIs. What that means is that porting your add-on to Jetpack
would not automatically make it electrolysis-compatible, since it would
still rely on XPCOM to do most of its work.

Given your limited resources, I think for now you're better off sticking
with the existing API. We're planning some blog entries in the next month
or two on how to do that. Chris already mentioned some general stuff about
multiprocess Firefox; we'll be focusing more specifically on add-ons soon.
Here are some high-level details:

Right now, add-ons run in the parent process. They can load code into child
processes using the message manager API, which Chris mentioned. The
http-on-* observers all fire in the parent process. However, they don't
have access to the DOM tree via the channel.notificationCallbacks
mechanism. It looks like that would be a problem for your add-on, since it
uses the channel's window to get the list of rewrite rules. I'm unsure how
difficult it would be to get around this. Content policies run in the child
process, since they typically need to access the DOM. So the shouldLoad
handler would need to be loaded into the child using the message manager,
and then it would need to communicate with the parent via messages.

If that sounds like a lot of work, don't despair. We're considering a lot
of other options to improve compatibility. Chris mentioned CPOWs, and there
are a few others as well. One question I have: does your Chrome extension
support all the same functionality as the Firefox add-on?

-Bill

On Wed, Dec 18, 2013 at 3:24 PM, Yan Zhu <y...@eff.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>

> Hi,
>
> I'm unofficially now the maintainer of HTTPS Everywhere. We apparently
> looked into porting the addon to Electrolysis in order to support
> Fennec three years ago and then gave up and wept in despair (see
> https://trac.torproject.org/projects/tor/ticket/2471). Some of our
> users have been asking us to do this because there's no version of
> HTTPS-E that supports mobile at all yet, so we probably should.
>

> Can you folks at Mozilla provide any guidance on this? Would you
> recommend porting the entire extension to Jetpack in the process?
>

> EFF is pretty busy these days, so any help you can give us with this
> would be very much appreciated!
>
> - -Yan
>
>

> - --
> Yan Zhu y...@eff.org
> Technologist Tel +1 415 436 9333 x134
> Electronic Frontier Foundation Fax +1 415 436 9993
> -----BEGIN PGP SIGNATURE-----
>

> iQEcBAEBCgAGBQJSsi60AAoJENC7YDZD/dnszw4H/AxoT9QRAqelTRbZ5l/8Vesk
> k4+Oqq41IYgyd9uqzHUHnXLbyvTOzTF882O9Y+41+Lg+JFfFty/es/FYp7tqTKkf
> Rbfmv257TRC/3dBd5LTcndNQ1Kcdtycux+0qKvZVkD22DL8pgL4hxngFo/MeVgmG
> yJxDpR96MAljVhsdTSFX3tu1s44AEk4UTNtB7Ls5c4yj0O8JemYtE6GGHB8Mp2Ah
> 1GTILBSoCZouJdZ/h8lRx5JU9Nbj2kyzHxQwvsPAatdsBRS2yKXwkaDGm5R2T7uW
> VzQW/6QZ9bpZm2CEXAA+Do+xJFZFcBookk3ZLwZJk4dzK6mCdRV8s9JlZEw7VY8=
> =PXXR
> -----END PGP SIGNATURE-----

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Thanks for the help! Our Chrome extension does all the essential
things that the Firefox addon does (in many fewer lines of code),
which may have caused me undue optimism about rewriting HTTPS
Everywhere to be e10s-compatible. See
https://github.com/EFForg/https-everywhere/tree/master/chromium.

> -Bill
>
>
> On Wed, Dec 18, 2013 at 3:24 PM, Yan Zhu <y...@eff.org
> <mailto:y...@eff.org>> wrote:
>
> Hi,
>
> I'm unofficially now the maintainer of HTTPS Everywhere. We
> apparently looked into porting the addon to Electrolysis in order
> to support Fennec three years ago and then gave up and wept in
> despair (see https://trac.torproject.org/projects/tor/ticket/2471).
> Some of our users have been asking us to do this because there's no
> version of HTTPS-E that supports mobile at all yet, so we probably
> should.
>
> Can you folks at Mozilla provide any guidance on this? Would you
> recommend porting the entire extension to Jetpack in the process?
>
> EFF is pretty busy these days, so any help you can give us with
> this would be very much appreciated!
>

> -Yan

>
>
> _______________________________________________
> dev-tech-electrolysis mailing list
> dev-tech-e...@lists.mozilla.org

> <mailto:dev-tech-e...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-tech-electrolysis

>
>

- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSs55JAAoJENC7YDZD/dnsbtMH/2jra3bAg1eMzFLz/G7eb8Dq
13hKIaYQFXsct2BfZoWOZ2tugwhFNbt43m9rng6seiIeNs9xfWm62r+LRZ2bMa5b
9jTk4ZZ6VzM9AkZgKLKE9lxCK+f1W7RnAPHNKtQg+lTS4RghWGMlLdxeBsO/qPLW
UlwnqIWewgHGBcjqWygN/4ePHfXx+6kyPE1apYJKjcXBbfhgrBlSlCy40NEuXDNO
3Y49DE4LTG6qPF/oEm24CgzDlQItBENeY2VLvFQPgR5pA6K0ijec3wsBLuhBkw/H
q0fbA/dbFRP5J1qa5Z48a0p2nfCB0RE7vaV+znLLAVAxn96TdWTe4fY5xSWR6yI=
=4y/H
-----END PGP SIGNATURE-----

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Actually, I don't think we use content policy anymore since y'all have
nsIHttpChannel.redirectTo. CC'ing Peter, who wrote that code to begin
with.

Sorry, there's probably a bunch of things in the codebase that should
be deleted by now. :(

So maybe porting to the new API wouldn't be so bad?

- -Yan

iQEcBAEBCgAGBQJSs6JEAAoJENC7YDZD/dnsSe4IAJEBsJp6QSlDGgCpNi9/fYnr
R1yvKcy3Q18xRWRUWe3Z17J2djC73uuSz35EfaG5gZxftJY0CMIpjnbhM9dxj5/j
WTncPgqOVexMw1uRXKYHDNfoYdeUqfAozo+lOxqE6TAH4AOEjQd4pYr9QcgvomC3
8XdZxarBavPiDxFT27S8WDyxBLg3S3Wco8zqZL45oHLsUh2GL3WbyGi9dfZZFKkM
DyrF7ozgAFcjWltQsW6n8E9aLHcc07xmWJZ0FrGHZh2ihNRxImUB0A6OoCXQDljB
GFauPQfuK24x6nNNCcHkhKYVXxcDv9FdHwjZxBcLzPfkag/Q7tRK/2x0fHCl0jI=
=e3jQ
-----END PGP SIGNATURE-----

[Brian Smith]

On Thu, Dec 19, 2013 at 5:49 PM, Yan Zhu <y...@eff.org> wrote:

> Actually, I don't think we use content policy anymore since y'all have
> nsIHttpChannel.redirectTo. CC'ing Peter, who wrote that code to begin
> with.
>
> Sorry, there's probably a bunch of things in the codebase that should
> be deleted by now. :(
>
> So maybe porting to the new API wouldn't be so bad?
>

I agree that, AFAICT, you shouldn't need nsIContentPolicy. I looked at your
code on GitHub and it doesn't seem to be using content policy anyway. That
makes things simpler.

Bill: The main issue that seems to need to be resolved is this: From the
parent process, how can I get a definitive, consistent answer to the
question "what is the origin and/or URL of the top-level document in tab
X?" Right now, There are a lot of features, like the mixed content blocker,
HTTPS Everywhere, the SSL indicator UI, client certificate support, the
permission management UI, etc. that need to make security decisions based
on the answer to that question. But, AFAICT, we have to ask the child to
get that information, and that seems to be racy and also at odds with the
sandboxing work, since we're not supposed to trust the child to give us
security-critical information.

Besides being able to query the answer to the question above, most of these
features, including HTTPS Everywhere, need to be able to associate small
amounts of data with the docshell, but with that information living in the
parent process. If we can point people to the object that lives in the
parent process that controls navigation in the top-level docshell in the
child, and if we can provide a way to extend this object by associating
arbitary data with it, and we can describe how that is accessed from an
addon, then we'll have mostly solved the most difficult problems for HTTPS
Everywhere and the other features I've mentioned above.

Because HTTPS Everywhere is a feature for securing web content, it makes
more sense for its code to run in the parent process than in the child
process. (Other addons, that aren't trying to restrict or improve the
security of web content, proably are better off running as content scripts
in the content processes.) From briefly looking at the HTTPS Everywhere
addon code, and based on some intuition of what the addon does, I would
expect that it could run 100% in the parent process without loss of
functionality, given the type of mechanism I describe above. But, before we
create/document the mechanism I describe above, then features like HTTPS
everywhere and Mixed Content Blocker are likely going to need to be split
into content-process parts and and parent-process parts, and then (ideally)
rewritten again after the above-described mechanism is in place.
Consequently, the sooner we can implement that mechanism, the less work
we'll all have to do in the long run.

Note that, because the parent process needs the child process's top-level
docshell state to be consistent while these features are making decisions,
the parent process will need to be able to prevent the child from
navigating itself while code in the parent process is executing. AFAICT,
that means every navigation mechanism in the child process probably needs
to be redone such that the child asks the parent to do the navigation and
then waits for the parent to actually force the navigation to happen, when
the parent is in a consistent state.

Cheers,
Brian

[Chris Peterson]

On 12/19/13, 5:12 PM, Yan Zhu wrote:
>> >btw, the current version of Firefox for Android no longer uses
>> >mobile e10s. It supports some desktop plugins with minor porting.
>> >For now, the e10s team is focusing exclusively on e10s for desktop
>> >Firefox.
> Really? Could someone check whether HTTPS Everywhere works in Fennec
> with this change to install.rdf?
> https://gitweb.torproject.org/pde/https-everywhere.git/commitdiff/3eba359e521d6242b73ec93582fe09984162371f

Yan, I have mixed news about Fennec. I unzipped HTTPS Everywhere's
stable release [1], applied that Fennec patch to install.rdf, and
rezipped the xpi. I could install the new xpi on desktop Firefox (and it
worked).

Firefox for Android seemed to successfully install the new xpi, but I
saw some errors in adb logcat. After restarting the browser, I can't
load any pages. I just get a blank page. The good news, though, is that
the blank pages had the redirected HTTPS URLs, so the addon is doing
*something*. :)


chris


[1] https://www.eff.org/files/https-everywhere-latest.xpi

[Bill McCloskey]

On Thu, Dec 19, 2013 at 6:14 PM, Brian Smith <br...@briansmith.org> wrote:
>
> Bill: The main issue that seems to need to be resolved is this: From the
> parent process, how can I get a definitive, consistent answer to the
> question "what is the origin and/or URL of the top-level document in tab
> X?" Right now, There are a lot of features, like the mixed content blocker,
> HTTPS Everywhere, the SSL indicator UI, client certificate support, the
> permission management UI, etc. that need to make security decisions based
> on the answer to that question. But, AFAICT, we have to ask the child to
> get that information, and that seems to be racy and also at odds with the
> sandboxing work, since we're not supposed to trust the child to give us
> security-critical information.
>

We do have an API that is supposed to give us this data. Given a <browser>
element (basically a tab), we can get the current URL via
browser.currentURI. The answer it gives is consistent in that I think the
entire frontend uses this mechanism. I don't know how the SSL indicator
works, but hopefully it uses browser.currentURI as well. Right now we get
the currentURI from the child. However, I think that's the best we can do
in a single-process model. To get a better answer, I think we would have to
use separate processes for different domains, and that's pretty far down
the road. Once we do it, though, browser.currentURI will continue to be a
workable API since it's completely controlled by the parent. Is there
anything better we can do at this point? Perhaps we need to make it easier
for C++ code to access it?

> Besides being able to query the answer to the question above, most of
> these features, including HTTPS Everywhere, need to be able to associate
> small amounts of data with the docshell, but with that information living
> in the parent process. If we can point people to the object that lives in
> the parent process that controls navigation in the top-level docshell in
> the child, and if we can provide a way to extend this object by associating
> arbitary data with it, and we can describe how that is accessed from an
> addon, then we'll have mostly solved the most difficult problems for HTTPS
> Everywhere and the other features I've mentioned above.
>

This is something that works okay but not great. It's pretty easy to have a
WeakMap that associates a <browser> element with an arbitrary piece of
data. The one flaw here is that there's an annoying feature called
"docshell swapping" that can mess up the association. We have bug 941540 on
file to fix that. Admittedly the API that we'll end up after that isn't
awesome, but it's no worse than the rest of XPCOM.

> Because HTTPS Everywhere is a feature for securing web content, it makes
> more sense for its code to run in the parent process than in the child
> process. (Other addons, that aren't trying to restrict or improve the
> security of web content, proably are better off running as content scripts
> in the content processes.) From briefly looking at the HTTPS Everywhere
> addon code, and based on some intuition of what the addon does, I would
> expect that it could run 100% in the parent process without loss of
> functionality, given the type of mechanism I describe above. But, before we
> create/document the mechanism I describe above, then features like HTTPS
> everywhere and Mixed Content Blocker are likely going to need to be split
> into content-process parts and and parent-process parts, and then (ideally)
> rewritten again after the above-described mechanism is in place.
> Consequently, the sooner we can implement that mechanism, the less work
> we'll all have to do in the long run.
>

It sounds like we already have everything we need to do everything in the
parent except for bug 941540?

> Note that, because the parent process needs the child process's top-level
> docshell state to be consistent while these features are making decisions,
> the parent process will need to be able to prevent the child from
> navigating itself while code in the parent process is executing. AFAICT,
> that means every navigation mechanism in the child process probably needs
> to be redone such that the child asks the parent to do the navigation and
> then waits for the parent to actually force the navigation to happen, when
> the parent is in a consistent state.
>

I don't really get this point. The parent and the child only communicate
via messages. Even if the child navigates, the parent won't be informed of
the navigation until it responds to messages, which basically happens when
it spins the event loop. However, I assume these add-ons must already be
aware that spinning the event loop can trigger further navigation, since
that's how single-process FF works.

The one exception to this is CPOWs, since they're synchronous. If the
parent asks for data from the child via a CPOW, it might witness navigation
that it wouldn't otherwise have seen in single-process Firefox. We just
need to be aware of that and avoid using CPOWs in any security-related code.

-Bill

[Bill McCloskey]

On Thu, Dec 19, 2013 at 9:00 PM, Bill McCloskey <
bill.mccl...@gmail.com> wrote:

>
> It sounds like we already have everything we need to do everything in the
> parent except for bug 941540?
>

Now that I think of it, one thing we may lack is a way to get to a
<browser> element from an http-on-modify observer. Is there a way to do
that?

-Bill

[Gavin Sharp]

Looks like those are passed only nsIHttpChannels. You can probably get
there via the channel's notificationCallbacks in some the common case
(get the associated docshell/DOMWindow, and then get the chrome event
handler[1] which in most cases is the <browser>), but there are likely
edge cases that wouldn't handle, and a simpler way to do this would be
nice.

[1] http://hg.mozilla.org/mozilla-central/annotate/599100c4ebfe/toolkit/components/social/MozSocialAPI.jsm#l52
is one example

On Thu, Dec 19, 2013 at 9:49 PM, Bill McCloskey

[Bill McCloskey]

On Thu, Dec 19, 2013 at 10:16 PM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> Looks like those are passed only nsIHttpChannels. You can probably get
> there via the channel's notificationCallbacks in some the common case
> (get the associated docshell/DOMWindow, and then get the chrome event
> handler[1] which in most cases is the <browser>), but there are likely
> edge cases that wouldn't handle, and a simpler way to do this would be
> nice.

It sounds like we'll need something new for e10s since that path goes
through the docshell. We'd like to be able to do this all in the parent.

-Bill

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/19/2013 07:15 PM, Chris Peterson wrote:
> On 12/19/13, 5:12 PM, Yan Zhu wrote:

>>>> btw, the current version of Firefox for Android no longer
>>>> uses mobile e10s. It supports some desktop plugins with minor
>>>> porting. For now, the e10s team is focusing exclusively on
>>>> e10s for desktop Firefox.
>> Really? Could someone check whether HTTPS Everywhere works in
>> Fennec with this change to install.rdf?
>> https://gitweb.torproject.org/pde/https-everywhere.git/commitdiff/3eba359e521d6242b73ec93582fe09984162371f
>>
>
>>

> Yan, I have mixed news about Fennec. I unzipped HTTPS Everywhere's
> stable release [1], applied that Fennec patch to install.rdf, and
> rezipped the xpi. I could install the new xpi on desktop Firefox
> (and it worked).
>
> Firefox for Android seemed to successfully install the new xpi, but
> I saw some errors in adb logcat. After restarting the browser, I
> can't load any pages. I just get a blank page. The good news,
> though, is that the blank pages had the redirected HTTPS URLs, so
> the addon is doing *something*. :)

Thanks, I reproduced the blank-page bug and hacked around it it by
changing chrome.manifest to ignore all of the HTTPS Everywhere UI
files. Hopefully this means that I'll only have to port the UI.

- -Yan

>
>
> chris
>
>
> [1] https://www.eff.org/files/https-everywhere-latest.xpi

> _______________________________________________
> dev-tech-electrolysis mailing list
> dev-tech-e...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-electrolysis
>

- --
Yan Zhu y...@eff.org
Technologist Tel +1 415 436 9333 x134
Electronic Frontier Foundation Fax +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJStOOzAAoJENC7YDZD/dnsDSEH/3kagycm1uymtdWrftJvBEDN
2vMLAlw2bs8CS8eYiTWVnqT0yodxRy/GdcgPagfVM1n4wRujhP7K3Jo8pkLv/LSw
zRb+DIiAjMyHya4eqJxFMyBBA74M35omFW7b3Ld9L3mRjNCRl862VXhnTzuS5oO0
45kPuG1UwK/UrgEu6pDvA0grlD9v3XzRrvlNTsUij6A9wf0z9+krUlkiu0Ja2aKk
cI84sA4b3Hgy0Rz6YuEqs0Pyz5ZF3hnjs4ItqbAadK89Aw9gUX0xyIY89fsvG4KH
SA66QW1w3HcTblWjLvJhnGghU6rcZu5Jjs6YdpDnYejcsRYdl3xkoWClUsMbCYk=
=9kdR
-----END PGP SIGNATURE-----

[Yan Zhu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/19/2013 06:14 PM, Brian Smith wrote:

>
> Bill: The main issue that seems to need to be resolved is this:
> From the parent process, how can I get a definitive, consistent
> answer to the question "what is the origin and/or URL of the
> top-level document in tab X?"

Does HTTPS Everywhere need to know this? It rewrites all requests from
the browser, not just those from top-level URL's [1]. For UI
convenience, we keep track of which rewrites were applied on each page
so that the user sees a list of rewrite rules that they can disable if
a page is broken with HTTPS Everywhere.

[1] Admittedly this causes some bugs with CSP, since rewriting an xhr
request from HTTP to HTTPS will sometimes prevent that resource from
loading in the top-level page. This is rare enough that we just deal
with it by disabling the rule for the xhr request when someone submits
a bug report (ugh).

But it sounds like this particular feature will be critical for
extensions that try to block third-party content, one of which is
being actively developed at EFF right now (Privacy Badger). In that
case, the operative question is whether a cookie is a third-party or
first-party cookie, which requires an API for asking whether a cookie
is being set by a response originating from a URL in a top-level document.

HTTPS Everywhere (minus SSL Observatory), for reference, just needs to
do the following:
* Rewrite any HTTP(S) request that matches one of the
insecure-to-secure-url preloaded rules, regardless of where it came from.
* Flag cookies as "secure" based on a set of
insecure-cookie-to-secure-cookie preloaded rules.h
* Show a list of the rules that were applied to each page.
* Let users enable/disable rules while the extension is running.

Also, I have no idea what a docshell is based on the limited
documentation online, but you guys should probably make sure that the
first search result isn't
https://developer.mozilla.org/en-US/docs/The_life_of_an_HTML_HTTP_request.
:)

BTW, I just want to add that I <3 everyone at Mozilla.

- -Yan
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJStOm/AAoJENC7YDZD/dns3g4IAI2MggwlvdoWhRt+XCROc9Yp
NsLwAvnVN89W8QF/8Bu/yugRgBq8ZSL+saN+KrMo/LIkJrCm27DEV5yZyagpjRiL
VQXpkB6PVyjThDsheKPYrx+st57G/312Hd7OEp2qOnRoK1OhDiNYKZnyDkGBVB3r
NZ2n1riFoIenmqZDHg1cHSuHBg2SxBimSpcXvDA4aLwXQes11ARryT4dBGaCNf/t
Eu+snvi//QxfdxvJplFOAjkbyiCA7GOghnP3XgyxsP+jKfj1bp/JYmLjnoXsocV0
RL+QR5zOXqL5vNZbV9h1a/3hA1aV2ny8/sFhqxO1ZO73+Of/lvK8uf3PRwMLwaU=
=HJ9u
-----END PGP SIGNATURE-----