Re: The future of binary embedding

[L. David Baron]

On Saturday 2012-02-25 00:53 -0800, luke.l...@gmail.com wrote:
> On Monday, March 28, 2011 10:03:11 PM UTC+1, Benjamin Smedberg wrote:
> > * Embedding Mozilla rendering into other processes is a tough
> > problem.
>
> no it's not - the python-hulahop widget uses GTK just like
> firefox does: it works perfectly, and it even provides access to
> XPCOM so that python-xpcom can get hold of the NPAPI and use DOM
> functionality in exactly the same way that javascript does [when
> xulrunner is embedded in firefox]

Has PyXPCOM been hooked up to the cycle collector yet, or do uses of
APIs that depend on cycle collection just leak memory?

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla http://www.mozilla.org/ 𝄂

[lkcl luke]

On Sat, Feb 25, 2012 at 4:59 PM, L. David Baron <dba...@dbaron.org> wrote:
> On Saturday 2012-02-25 00:53 -0800, luke.l...@gmail.com wrote:
>> On Monday, March 28, 2011 10:03:11 PM UTC+1, Benjamin Smedberg wrote:
>> >     * Embedding Mozilla rendering into other processes is a tough
>> >       problem.
>>
>>  no it's not - the python-hulahop widget uses GTK just like
>>  firefox does: it works perfectly, and it even provides access to
>>  XPCOM so that python-xpcom can get hold of the NPAPI and use DOM
>>  functionality in exactly the same way that javascript does [when
>>  xulrunner is embedded in firefox]
>
> Has PyXPCOM been hooked up to the cycle collector yet, or do uses of
> APIs that depend on cycle collection just leak memory?

david, thank you for responding on this.

... now, you see... this kind of information is entirely missing -
and it's the relegation of pyxpcom to "third party status" within
mozilla's project management - i.e. its removal from mozilla-central -
that results in things like that massive PRBool->bool substitution,
and other straightforward maintenance, being entirely missed.

so the code is just left to rot, or is left to someone who really
doesn't have the time or resources to focus 100% on keeping absolutely
without fail without exception absolutely 100% up-to-date with every
single one of the advances and modifications made to the mozilla core
codebase such that they even *know* about things like "the cycle
collector".

now, as it's a small codebase, i'm certain that someone who *does*
know about this "cycle collector", or any other code modifications,
could probably bring pyxpcom up-to-date prior to each xulrunner stable
release in very short order, especially when working alongside someone
like todd who is familiar with the code.

... but the fact that nobody has even bothered, and has pretty much
abandoned the code, leaves it as worse than useless, and a massive
drain on the time and resources of the free software community, who
have to learn or "re-learn" about each and every single API change,
each and every single mozilla core codebase change each and every
monotonous time *right* before a release.

that's massively unfair to expect members of the free software community to do.

hence why i have requested - formally on mozilla.governance - for
advice on how to go about making a formal request that the mozilla
foundation a) adopt python-hulahop as a 1st level priority project b)
reinstate pyxpcom as a 1st level priority project - preferably *back*
into mozilla-central. c) introduce unit tests that, if not passed,
will block the release of *any* xulrunner release at the time until
such time as the bugs are fixed and the unit tests pass.

the end result will actually be an *improvement* in the quality of
the xulrunner and the firefox codebase because it will result in a
different kind of testing - from a different angle with different
code-paths of the exact same core components that are utilised in
firefox.

do you happen to know to whom such formal requests should be directed?

many thanks,

l.

[L. David Baron]

On Saturday 2012-02-25 19:38 +0000, lkcl luke wrote:
> now, as it's a small codebase, i'm certain that someone who *does*
> know about this "cycle collector", or any other code modifications,
> could probably bring pyxpcom up-to-date prior to each xulrunner stable
> release in very short order, especially when working alongside someone
> like todd who is familiar with the code.

It's not. It's a massive task. (I'd guess it's roughly equivalent
in complexity to everything else in PyXPCOM. That is, I'd guess
that this would be about half the work of writing PyXPCOM from
scratch with it included. That's a very rough guess, though.)

> ... but the fact that nobody has even bothered, and has pretty much
> abandoned the code, leaves it as worse than useless, and a massive
> drain on the time and resources of the free software community, who
> have to learn or "re-learn" about each and every single API change,
> each and every single mozilla core codebase change each and every
> monotonous time *right* before a release.
>
> that's massively unfair to expect members of the free software
> community to do.

Why do you consider Mozilla employees different from other members
of the software community? Mozilla is paying its employees to be
members of the community -- and to perform the tasks where they'd
most effectively advance Mozilla's mission
( http://www.mozilla.org/about/manifesto.en.html ). There are other
companies who pay people to work on Mozilla code (e.g., in the area
of embedding, a number of Linux distributors have at various times).

> hence why i have requested - formally on mozilla.governance - for
> advice on how to go about making a formal request that the mozilla
> foundation a) adopt python-hulahop as a 1st level priority project b)
> reinstate pyxpcom as a 1st level priority project - preferably *back*
> into mozilla-central. c) introduce unit tests that, if not passed,
> will block the release of *any* xulrunner release at the time until
> such time as the bugs are fixed and the unit tests pass.

Demanding that somebody else do work for you doesn't seem like an
approach likely to succeed.

> the end result will actually be an *improvement* in the quality of
> the xulrunner and the firefox codebase because it will result in a
> different kind of testing - from a different angle with different
> code-paths of the exact same core components that are utilised in
> firefox.

That's at least a more reasonable argument. I don't believe it,
though.

I think any future in our embedding API should be based around APIs
designed to be an embedding API, not around a collection of internal
functions gathered up and labeled as an API. The problem with the
latter is that it's hard to avoid changing them, because they
weren't designed to be permanent. You're not going to turn the
latter into the former by shaking bugs out through additional
testing.

[lkcl luke]

allo david, thanks for responding here.

On Sun, Feb 26, 2012 at 4:20 AM, L. David Baron <dba...@dbaron.org> wrote:
> On Saturday 2012-02-25 19:38 +0000, lkcl luke wrote:
>>  now, as it's a small codebase, i'm certain that someone who *does*
>> know about this "cycle collector", or any other code modifications,
>> could probably bring pyxpcom up-to-date prior to each xulrunner stable
>> release in very short order, especially when working alongside someone
>> like todd who is familiar with the code.
>
> It's not.  It's a massive task.  (I'd guess it's roughly equivalent
> in complexity to everything else in PyXPCOM.  That is, I'd guess
> that this would be about half the work of writing PyXPCOM from
> scratch with it included.  That's a very rough guess, though.)

*whistles*. if you have time, could you help me to assess what's
actually involved? what code uses the "cycle collector", what patch
implemented a change-over to the functions behind "cycle collector"
etc. etc.

i note for example in xpcom/src/PyISupports.cpp that there are
functions utilised called nsMemory::Alloc and nsMemory::Free, and a
quick grep | wc shows 45 such functions in the 7,000 lines comprising
xpcom/src/*.cpp but i have no actual idea which c/c++ functions you're
referring to, yet.

>>  ... but the fact that nobody has even bothered, and has pretty much
>> abandoned the code, leaves it as worse than useless, and a massive
>> drain on the time and resources of the free software community, who
>> have to learn or "re-learn" about each and every single API change,
>> each and every single mozilla core codebase change each and every
>> monotonous time *right* before a release.
>>
>>  that's massively unfair to expect members of the free software
>>  community to do.
>
> Why do you consider Mozilla employees different from other members
> of the software community?  Mozilla is paying its employees to be
> members of the community -- and to perform the tasks where they'd
> most effectively advance Mozilla's mission
> ( http://www.mozilla.org/about/manifesto.en.html ).  There are other
> companies who pay people to work on Mozilla code (e.g., in the area
> of embedding, a number of Linux distributors have at various times).

this is a good point - i can't explain it though. i know that
something's "not right", but i don't have the... how-to-say... i
haven't been able to accurately articulate what i feel is the matter.

does that make any sense? i'm sitting here waving my hands, trying
to think how to express the frustration that i feel but the words just
aren't coming to me :)

>>  hence why i have requested - formally on mozilla.governance - for
>> advice on how to go about making a formal request that the mozilla
>> foundation a) adopt python-hulahop as a 1st level priority project b)
>> reinstate pyxpcom as a 1st level priority project - preferably *back*
>> into mozilla-central. c) introduce unit tests that, if not passed,
>> will block the release of *any* xulrunner release at the time until
>> such time as the bugs are fixed and the unit tests pass.
>
> Demanding that somebody else do work for you doesn't seem like an
> approach likely to succeed.

for "me"? no. you misunderstand.

if this was for *me* i would go and - personally - use pythonwebkit.
which i - me - *i* wrote that.
http://www.gnu.org/software/pythonwebkit. pythonwebkit does the
*exact* same job as xpcom+hulahop, from the high-level (python-based)
API perspective.

when i say the exact same job i mean that by the time you get to the
DOM/HTML5 level, all of the thousands of properties and functions, and
the hundreds of objects that are accessible to python, are EXACTLY the
same [near as damnit, and the pyjamas desktop codebase has a way of
normalising those differences seamlessly so that developers using its
SDK don't have to worry about those discrepancies].

so if this was JUST ME - if it was alllll about me, meee, meee, i
would be the kind of hypothetical egomaniac who would hypothetically
say hypothetically "f*** it, i'm not going to bother with these
[hypothetical] mozilla [hypothetical] jerks, iiii'm gonna use that
pythonwebkit code, nyer nyer nyer, [hypothetically] sod the lot of
you, nyer nyer nyer" - don't for one second think or imagine that i
ACTUALLY would say that, ok!

i - personally - have the intelligence and the capability to do that
kind of thing - to take on code that other people simply don't even
grasp, and to join it to *other* code that serves some completely
alternative purpose, and make the whole thing work.

btw it may interest you to know that pythonwebkit actually uses the
xulrunner python-based IDL file parser! i modified it to be able to
understand webkit's IDL files instead of xulrunner's IDL files! i
also adapted python-gobject's codegen.py for outputting the back-end
part of the c-based python module, joined the two together.

... but you see where i'm going with this? i neither made demands
nor expected my ego to be evaluated as part of the equation.

>>  the end result will actually be an *improvement* in the quality of
>> the xulrunner and the firefox codebase because it will result in a
>> different kind of testing - from a different angle with different
>> code-paths of the exact same core components that are utilised in
>> firefox.
>
> That's at least a more reasonable argument.  I don't believe it,
> though.

https://bugzilla.mozilla.org/show_bug.cgi?id=728645

looking back at the file history of FocusManager.cpp the bugreports
indicate that there have been changes to the FocusManager API in
exactly the area which is involved.

now the problem that _i_ have, as an outsider, is that normally i
would deploy "git bisect" techniques to track down the commit revision
with a binary search in order to isolate the commits between xulrunner
9.0 and xulrunner 10.0.2 which resulted in the segfault.

this technique is, as you know, commonly deployed as a means to
substitute for lack of unit test regression which _should_ have been
being run on a regular basis [note, pointedly, that hulahop is
*entirely* excluded from that daily continuous unit test regression
testing, which is what this is all about].

i looked for *two days* for an understanding of the mozilla codebase
and organisation in order to be able to provide that valuable
information - the two consecutive commits that did and did not result
in that segfault. could i find _anything_ which guided me on how to
do that commit bisecting? naah. i cannot find _anything_ on how to
relate commits in mozilla-central to the release cycles tree... it's
just .... waargh! :)

but anyway, that aside: can you see that between [working] xulrunner
9.0 code and [non-working] xulrunner 10.0.2 code (when using
xpcom+hulahop), there were some commits made that dealt with segfaults
on certain platforms (macosx, windows), but that somehow xulrunner on
gnu/linux does *not* have such segfaults?

but the key question is this: what do you think would have happened
if there was daily automated regression tests being carried out using
xpcom+hulahop?

do you think that xulrunner 10.0.2 would have gone out the door at
the time that it did, if this segfault had been detected at the time?

> I think any future in our embedding API should be based around APIs
> designed to be an embedding API, not around a collection of internal
> functions gathered up and labeled as an API.

david - you've entirely - i mean so completely missed the point it
actually takes my breath away.

>  The problem with the
> latter is that it's hard to avoid changing them, because they
> weren't designed to be permanent.

david: please look up what middleware technologies such as CORBA,
COM, Objective-C, gobject introspection etc. are, have achieved and
enabled, both historically (if 20 years can be considered "historic"
in the context of computing) as well as recently.

i don't wish to offend you or anything but the fact remains that
you've demonstrated a complete lack of comprehension of what XPCOM
actually brings to the table.

the summary is this: Common Object Model middleware such as
microsoft's COM, CORBA and XPCOM *are* permanent. they're a permanent
API that's so absolute "set-in-stone permanent" that 20 YEARS later,
it's possible to connect to even _proprietary_ DLLs that have a COM
interface, for example, and still use them in a modern application,
today.

even advances and improvements in COM over the past 2 decades are
done in such a way that they're always, always backwards-compatible.
backwards-compatibility is DESIGNED into the fundamental API of a good
Common Object Model's infrastructure, right at the very very beginning
when it's first created.

so please excuse me, but i'm a bit taken aback that you're commenting
on this topic when you've demonstrated a complete lack of knowledge of
the fundamentals of Common Object Model architectures. you're not
alone, though, i have to say.

>  You're not going to turn the
> latter into the former by shaking bugs out through additional
> testing.

david - i'm not being rude or anything but you just don't get it.
please review and understand what XPCOM and technology similar to it
actually is before commenting further on this subject.

i look forward to hearing from you in the near future when you will
be able to say "omg, did i _really_ say that? that's very funny!" and
we can have a bit of a laugh and share a joke about how much you
didn't know about the immense power of Common Object Model technology
(in general).

you're not alone. very very VERY few people in the Free Software
Community understand the immense power and design freedom that Common
Object Model technology gives people. some people _are_ finally
reinventing some of the concepts of COM, in the form of gobject
introspection, but that is really a very recent addition which will
take some considerable time to percolate through the free software
community.

l.

[Mook]

On 2/25/2012 8:59 AM, L. David Baron wrote:
> On Saturday 2012-02-25 00:53 -0800, luke.l...@gmail.com wrote:
>> On Monday, March 28, 2011 10:03:11 PM UTC+1, Benjamin Smedberg wrote:
>>> * Embedding Mozilla rendering into other processes is a tough
>>> problem.
>>
>> no it's not - the python-hulahop widget uses GTK just like
>> firefox does: it works perfectly, and it even provides access to
>> XPCOM so that python-xpcom can get hold of the NPAPI and use DOM
>> functionality in exactly the same way that javascript does [when
>> xulrunner is embedded in firefox]
>
> Has PyXPCOM been hooked up to the cycle collector yet, or do uses of
> APIs that depend on cycle collection just leak memory?
>
> -David
>

My (very incomplete) understanding is that, no, PyXPCOM currently knows
nothing about the cycle collector. This is mitigated somewhat by the
fact that PyDOM is dead (especially with the incoming new DOM bindings
work), plus PyXPCOM is currently unbuildable against mozilla-central,
plus it falls all over the place even if you do manage to build it.

If there are other non-DOM APIs which depend on the cycle collector,
they are not marked at all and leaks will happen just as much with
non-PyXPCOM binary components (because they wouldn't know they need to
use CC). Of course, it's not like I know that the DOM stuff needs CC
because it was clearly documented either...

--
Mook

[lkcl luke]

david, hi,

i've got over being taken aback by what you said about the APIs thing,
and it occurred to me that if i didn't provide further information and
explanation there was the risk that people reading my comments could
potentially view the response as being from a "har har i'm more
superior than you har har you're so ignorant har har you know nothing
i know everything now go away and educate yourself" complete fucking
twonce (viz: a twat _and_ a nonce, rolled into one obsequious
egoistical moron whose fingers should be broken to prevent them from
being able to type on the internet, ever again).

as i'm _not_ that kind of person, it's a responsibility and a duty for
me to clarify what i said.

you said:

"I think any future in our embedding API should be based around APIs
designed to be an embedding API, not around a collection of internal

functions gathered up and labeled as an API. The problem with the

latter is that it's hard to avoid changing them, because they
weren't designed to be permanent."

the thing is: xpcom isn't an API, it's middleware which *provides*
access to APIs. dynamically. changes to the APIs that are reflected
*through* XPCOM automatically are available.

thus, "XPCOM-the-API" requires no "labelling". it just... is. it
works, it's done. no further development required. that's the nature
of Common Object Models [as long as they've actually been designed
properly...]

thus also, any APIs that are *available* through XPCOM are *also* "not
your problem". they simply aren't. they're available - transparently
- end of story.

... do you see therefore what i was driving at?

now in addition to that, there's hulahop, which is the "next layer
up", and it's... if you will... "grounding" of both the
XPCOM-accessible NPAPI and the gecko engine into the "reality" of
python from a declarative programming perspective.

look closely at this:
http://pyxpcomext.mozdev.org/no_wrap/tutorials/hulahop/xpcom-hulahop.html

look first at the implementation of progresslistener.py. what do you
notice about it? it's nothing more than an object-orientated
"wrapper", in python, around the XPCOM-accessible
nsIWebProgressListener interface, isn't it?

so there _is_ no "API" to be "developed", here, is there?

then, come back to the main hula.py file. what do you notice? what
evidence of the existence of any mozilla-required "APIs" do you see?
what in that file requires any actual API "design"? i honestly cannot
see any.

these lines, for example:
# this is equivalent to ~/.firefox profiles subdirectory
hulahop.startup('/home/lkcl/test')

they are absolutely essential. there's no "design" required. ok,
perhaps you might pick a different name from "startup", but who
honestly cares about that?

then there's the ContentInvoker class. this is required as part of
the XPCOM API, which is set in stone, is not going to change. the
invocation and linking of ContentInvoker class instances is, again,
set in stone:

listener = xpcom.server.WrapObject(ContentInvoker(self),
interfaces.nsIDOMEventListener)

so that is inviolate, as well. no API development required.

the class Browser, which derives from the c-based python-gtk module
"WebView", is where there *might* be room for "API Development", but
there you're actually actively constrained by the _GTK_ API, or more
specifically by the python-gtk2 API, which defines how pygtk widget
APIs are written.

so absolutely no "development" of "APIs" is required *there*, either.

about the only function which looks slightly odd, to me, is this
"do_setup" function. i don't actually know the deep compelling
technical reason why that's there, and i mention that in the tutorial.
looking in the source code hulahop.c it's _still_ not blindingly
obvious, but i'm not going to worry about it. the main point is: it
appears to be a techical detail associated with the fundamentals of
python c-based modules, getting over some problem/issue that the
hulahop developers created, and, again, it has absolutely nothing to
do with xulrunner "APIs".

conceivably you could argue that the name of this function "do_setup"
could be changed, but what would be the point?

so, throughout the entire hulahop codebase there is nothing -
absolutely not one single thing - that requires "designing". the
hybrid combination of the various bits of technology *automatically*
defines the published and public API.

you create a widget, you get access to the NPAPI. wow! that's so
simple, it's actually difficult to make it complicated enough for
people to grasp the significance.

so do you see? not only is there no API designing required, but there
exists proven stable code that already *implements* the
already-automatically-technically-constrained "API". code which, if
you even tried to modify it would result in yet further (and
unnecessary) knock-on effects for the users of that code in an already
untenable situation.

anyway. i'll try tackling pyxpcom to see if i can get it
up-and-running with xulrunner-13.

l.

[Joshua Cranmer]

On 2/25/2012 11:50 PM, lkcl luke wrote:
> the summary is this: Common Object Model middleware such as
> microsoft's COM, CORBA and XPCOM *are* permanent. they're a permanent
> API that's so absolute "set-in-stone permanent" that 20 YEARS later,
> it's possible to connect to even _proprietary_ DLLs that have a COM
> interface, for example, and still use them in a modern application,
> today. even advances and improvements in COM over the past 2 decades
> are done in such a way that they're always, always
> backwards-compatible. backwards-compatibility is DESIGNED into the
> fundamental API of a good Common Object Model's infrastructure, right
> at the very very beginning when it's first created. so please excuse
> me, but i'm a bit taken aback that you're commenting on this topic
> when you've demonstrated a complete lack of knowledge of the
> fundamentals of Common Object Model architectures. you're not alone,
> though, i have to say.

While you may be coming in here preaching the gospel of the
infallibility of XPCOM, the fact is that, in many cases, we are seeking
to rid Mozilla of XPCOM, especially in the realm of the DOM. xpidl and
the XPCOM thunks are insufficient to represent all of the functionality
of the DOM IDLs, not to mention rather slow in comparative terms; the
new DOM bindings (at least, as far as I understand) will not be using
XPCOM at all.

Also, if you paid attention to some of Mozilla's history, you'll notice
that we have stopped advertising perpetual backwards-compatibility for
XPCOM's ABI, specifically so we can break it if we find doing so would
make our code faster and better. Although, it appears to have turned out
that outparam-deling QueryInterface was not a significant performance
boost...

> david - i'm not being rude or anything but you just don't get it.
> please review and understand what XPCOM and technology similar to it
> actually is before commenting further on this subject.

No offense, but claiming that one of the more long-term contributors
doesn't know what one of the foundational technologies of Mozilla is
smacks a tad bit of hubris to me.

[Benjamin Smedberg]

On 2/25/2012 3:53 AM, luke.l...@gmail.com wrote:
>
>> We never solved it fully, and each embedder has had to
>> spend lots of time tweaking things.
> perhaps it might be worthwhile for the mozilla foundation to find - and fund - the people who _do_ understand these things, rather than totally abandon existing efforts and leave entire communities completely without support:
>
> http://lists.sugarlabs.org/archive/sugar-devel/2011-June/031865.html
There is no free lunch. As I noted, finding and funding embedding
support must necessarily come at the expense of something we could be
doing instead. Overall I'm not saying that we don't want Mozilla to be
embeddable, but that there are more important things to work on, both
strategically and technically. This is not a permanent calculus, and it
can change again in the future.

Part of this decision is merely making clear that there hasn't been a
community maintaining this code effectively. If you'd like to build that
community, I encourage you to do so!

> no it's not. if you have things like XPCOM which is a dynamic "object model" interface, you don't *need* "binary compatibility" - you just need to make sure that XPCOM works.
>
> XPCOM, which was based on (inspired by) microsoft's COM, is just as misunderstood, it seems, as COM is.
>
> the benefits of Common Object Model technology are just... i cannot emphasise enough how powerful they are. this is something that the Free Software Community as a whole simply does NOT get, to its detriment. microsoft dominated computing for nearly two decades thanks to COM/DCOM. apple dominates its sector through Objective-C, at the heart of which is, yes, you guessed it: a type of Dynamic Object Model system.
objective-c and COM are fundamentally different in how they achieve
backwards compatibility. XPCOM "working" correctly will prevent you from
crashing, but it only achieves the goal of being *compatible* if you
don't change the interfaces. We have tried this, and it's clear that our
current interfaces are not suited to freezing. The XPCOM object model in
gecko does not suit the needs of binary compatibility.

We are focused primarily on the *script* compatibility, and to the
extent that we want a good embedding API in the future, it will be
designed around embedding via script, and primarily JavaScript since
that is the language of the web. We do make strong compatibility
guarantees for JS, and we should be able to do that for embedders as well.

> so do you think it would be a good start for the mozilla foundation to
> adopt python-hulahop as the beginnings of reversing this trend towards
> isolation of the mozilla foundation from absolutely everything but
> javascript, javascript, javascript and firefox, firefox, firefox?
What kind of "adoption"? I don't think that python-hulahop is
strategically important to our core goals of innovation on the internet.
So I don't think that in terms of the Mozilla mission our core community
should make maintenance and development of that code a priority. I'd be
happy to provide a place for you or somebody else to build a community
around python-hulahop, though.

--BDS

[Benjamin Smedberg]

On 2/25/2012 7:05 AM, luke.l...@gmail.com wrote:
> the first question is this: how the heck is it possible that i could create, in under 3 weeks, an entire port of pyjamas-desktop to use the MSHTML IE engine, *without* requiring access to the proprietary MSHTML Trident Engine?
The browser engine provided by Microsoft has a stable COM API. It also
does not require the embedder to handle any kind of persistent data
access: history/cache/cookie stores are handled via a single operating
system store which is shared across processes.
> the key question is this: what is it about the microsoft technology that makes this possible? this is not a rhetorical question. this is not a sarcastic question. i genuinely require that you actually think, ben, and find out the answer (hint: you already implement one. or used to. until it was pulled).
No, we didn't.

The ActiveX control provided an emulation layer which satisifed some of
the behaviors (allowing gecko to run in a process hidden behind at least
the basic set of Microsoft interfaces). But because we don't have any
form of cross-process profile sharing, the embedder either had to
configure and run their own profile directory (and think about the
consequences of only being able to use that datastore from one process
at a time) or run without any data store, which is not a well-tested
gecko configuration and will cause many websites to fail because cookies
and the cache do not behave correctly.

In addition, the primary way to access DOM nodes through the MS API is
via IDispatch, and the bridge between IDispach and COM typeinfo was
never complete and didn't match the Microsoft implementation in key
ways. The closest cross-platform equivalent we have to IDispatch is the
NPRuntime, which is pretty well tested but isn't efficient and has
inherent cycle issues.
>
> and if microsoft can do it, why can't the mozilla foundation?
We *could* do it. But it would require significant engineering effort to
implement shared profile data, solve the IDispatch-or-equivalent problem
(especially if we needed it to be safe for object cycles via the cycle
collector), and then provide real QA and release resources for the
project. Since we are not prepared to do any of that at this time, it's
better to be perfectly clear about the situation then string embedders
along thinking that they have a better solution than they actually have.

--BDS

[lkcl luke]

allo ben, thanks for taking this up.

On Mon, Feb 27, 2012 at 1:53 PM, Benjamin Smedberg
<benj...@smedbergs.us> wrote:
> On 2/25/2012 3:53 AM, luke.l...@gmail.com wrote:
>>
>>
>>> We never solved it fully, and each embedder has had to
>>>       spend lots of time tweaking things.
>>
>>  perhaps it might be worthwhile for the mozilla foundation to find - and
>> fund - the people who _do_ understand these things, rather than totally
>> abandon existing efforts and leave entire communities completely without
>> support:
>>
>>  http://lists.sugarlabs.org/archive/sugar-devel/2011-June/031865.html
>
> There is no free lunch. As I noted, finding and funding embedding support
> must necessarily come at the expense of something we could be doing instead.
> Overall I'm not saying that we don't want Mozilla to be embeddable, but that
> there are more important things to work on, both strategically and
> technically. This is not a permanent calculus, and it can change again in
> the future.
>
> Part of this decision is merely making clear that there hasn't been a
> community maintaining this code effectively. If you'd like to build that
> community, I encourage you to do so!

well... there's the problem.

there's a community *depending* on this stuff *working*, but they are
all python programmers. they shouldn't _have_ to learn c or c++ -
massive amounts of it.

and, because this code doesn't actually need "development" - just
maintenance - there's no "community" to actually build!

there's a critical CRITICAL difference between middleware and the
code that *uses* that middleware. the maintenance and development
cycles are _completely_ different.

* code that uses middleware requires continuous upkeep, development
etc. in order to stay relevant. it's also orders of magnitude larger
than the middleware it uses

* middleware does NOT require constant development. it works or it
doesn't. if it doesn't work, it was designed WRONG in the first
place. middleware DOES however require maintenance to keep up-to-date
with its dependencies (mozilla-core in this case).

... how can a community be built around something that requires only
a tiny amount of occasional maintenance?

... it can't, can it? once every year to two years, a couple of
hard-core developers get together, bash out a few updates on the
dependencies, and then go away again? that's *not* a "community", is
it, ben.

you see what i'm getting at?

just because xpcom (and hulahop) appear not to have an "active
community" does *not* mean that they are "shit" or "unmaintained" or
"should be relegated to third party status" or "removed entirely from
the mozilla codebase".

that's the mistake that's been made.

>>  no it's not.  if you have things like XPCOM which is a dynamic "object
>> model" interface, you don't *need* "binary compatibility" - you just need to
>> make sure that XPCOM works.
>>
>>  XPCOM, which was based on (inspired by) microsoft's COM, is just as
>> misunderstood, it seems, as COM is.
>>
>>  the benefits of Common Object Model technology are just... i cannot
>> emphasise enough how powerful they are.  this is something that the Free
>> Software Community as a whole simply does NOT get, to its detriment.
>>  microsoft dominated computing for nearly two decades thanks to COM/DCOM.
>>  apple dominates its sector through Objective-C, at the heart of which is,
>> yes, you guessed it: a type of Dynamic Object Model system.
>
> objective-c and COM are fundamentally different in how they achieve
> backwards compatibility. XPCOM "working" correctly will prevent you from
> crashing, but it only achieves the goal of being *compatible* if you don't
> change the interfaces.

ah.... in static programming languages, yes. in dynamic programming
languages such as python where you use the trick of lazy / dynamic
binding (based on the name) - absolutely NOT. the changing of the
interfaces is COMPLETELY irrelevant.

> We have tried this, and it's clear that our current
> interfaces are not suited to freezing.

ben - python doesn't *care* about the interface GUIDs, and as long as
the interfaces are designed by adding extra default arguments (rather
than changing the number or order of the arguments), python doesn't
even care if you _add_ to the interfaces.

even if you change the type of one of the arguments from say a
nsHTMLImage to a nsHTMLElement python *still* doesn't care!

this is really really frustrating for me to have to be explaining
this stuff because you should _know_ how it works.

but there's more. do you remember this bugreport?
https://bugzilla.mozilla.org/show_bug.cgi?id=502234

do you remember that i said that adding that "optional_arg" concept
was a really bad idea [instead of doing proper IDispatch (coclass)
stuff where you can merge / look up multiple COM interfaces]?

the reason for that is clear when you look at microsoft's COM IDL
files for MSHTML. you don't need to look at _microsoft's_ COM IDL
file(s), you can look on the Wine web site, here:

http://source.winehq.org/source/include/mshtml.idl

take a look for HTMLElement1, 2, 3 and 4, then look at line 7414. do
you notice the use of that coclass thing? what that does is it
*merges* all of the functions - each of which have the same name but
have *different* parameters (usually different numbers of parameters)
- so that you achieve the same thing that c++ has through function
overloading.

what's significant about this is that the interface UUIDs for
HTMLElement1, 2, 3 and 4 have *NEVER CHANGED*. EVER.

if a new function was ever needed, or a change to a parameter was
ever needed, they just added an extra definition (HTMLElement5) and
the advantage of that is that older (static languages, c, c++, java)
*still work*, and dynamic languages get to take care of accessing the
right thing via "lazy dynamic binding".

do you see why this is so important?

> The XPCOM object model in gecko does
> not suit the needs of binary compatibility.

again: not in a dynamic programming language is this even remotely a problem.

> We are focused primarily on the *script* compatibility, and to the extent
> that we want a good embedding API in the future, it will be designed around
> embedding via script, and primarily JavaScript since that is the language of
> the web. We do make strong compatibility guarantees for JS, and we should be
> able to do that for embedders as well.

the static-language (c,c++,java) embedders are on their own, thanks
to XPCOM's design *not* having taken on the full features of COM which
was "inspired" by.

the dynamic language embedders, when you have good "lazy / dynamic"
bindings such as that which todd (or... wait.. it was mark hammond
wasn't it?) created, simply *don't* care.

>> so do you think it would be a good start for the mozilla foundation to
>> adopt python-hulahop as the beginnings of reversing this trend towards
>> isolation of the mozilla foundation from absolutely everything but
>> javascript, javascript, javascript and firefox, firefox, firefox?
>
> What kind of "adoption"? I don't think that python-hulahop is strategically
> important to our core goals of innovation on the internet.

ok - i'll pass that on to the pyjamas community. i'll let them know
that not only has the mozilla foundation badly let down the OLPC
community but is quite prepared to abandon the pyjamas community as
well.

but before i do that, i think it might be a good idea if you ran
pyjamas-desktop and actually saw and understood what it can do.
you'll need to use debian 6.0 and do "apt-get install pyjamas" - it
specifically has to be debian 6.0 because that was the last time it
was possible to get a stable build of xulrunner 1.9, xpcom and
python-hulahop all together in the same distribution.

> So I don't think
> that in terms of the Mozilla mission our core community should make
> maintenance and development of that code a priority. I'd be happy to provide
> a place for you or somebody else to build a community around python-hulahop,
> though.

the summary is: there *is* no expertise within the community that
*is* built around python-hulahop, and you've never run pyjamas-desktop
(not that i'm aware of) so it is very hard for you to comprehend (and
even harder for me to explain) what it is that i'm on about.

so that just leaves... me. the burden of responsibility for getting
this code working is entirely on me - and i'm not very happy about it.

i've managed to hack python-xpcom into submission: i had to remove
the pyloader module because it relies on "MOZILLA_INTERNAL_API" stuff,
uses internal strings when PyXPCOM.h declares external ones; i tried
getting the Makefile.in into line with one of the js loaders
(mozJSComponentLoader) that i found: adding LIBXUL_LIBRARY=1 and
STATIC_LIBRARY=1 into pyxpcom's pyloader Makefile.in had *zero*
effect, i ended up with missing functions at linker time (ones that
were in -lxul *despite* linking with -lxul) - in the end i went "fuck
it" and commented the code out.

then i moved on to hulahop, and immediately ran into missing header
files that "make -f client.mk install" hadn't bothered to install
(large numbers of js* files). i manually copied those over, and then
immediately ran into the build error below, at which point i decided
i'd had enough.

now, these are the kinds of things that will be blindingly-obvious to
you, because you're very very familiar with the ongoing day-to-day
development.

... but we "outsiders" are *not*, and it's fucking annoying to have to
get up-to-speed _just_ the once, knowing full well that by the time
it's done, there's a good chance that there will have been *another*
release - 14, 15, 16, 17, 18 - of xulrunner - leaving absolutely NO
chance of getting this into the mainline linux distributions.

do you see what i'm getting at?

even if i did this work, getting hulahop up-and-running with each
release, it would be pointless, because the mozilla foundation "rushes
ahead" with release, release, release, release.

if you want to help, you need to include hulahop in the release cycle.
bugs in hulahop because of changes made to xulrunner should be a
*critical* block on a xulrunner release.

l.

In file included from /usr/local/include/xulrunner-13.0a1/jscntxt.h:56:0,
from
/home/lkcl/src/sugar-hulahop/./src/hulahop-web-view.cpp:33:
/usr/local/include/xulrunner-13.0a1/jsgc.h: In static member function
'static js::gc::Chunk*
js::gc::Chunk::fromPointerToNext(js::gc::Chunk**)':
/usr/local/include/xulrunner-13.0a1/jsgc.h:787:76: warning: invalid
access to non-static data member 'js::gc::Chunk::info' of NULL object
/usr/local/include/xulrunner-13.0a1/jsgc.h:787:76: warning: (perhaps
the 'offsetof' macro was used incorrectly)
In file included from
/usr/local/include/xulrunner-13.0a1/jspropertycache.h:48:0,
from /usr/local/include/xulrunner-13.0a1/jscntxt.h:57,
from
/home/lkcl/src/sugar-hulahop/./src/hulahop-web-view.cpp:33:
/usr/local/include/xulrunner-13.0a1/vm/String.h: At global scope:
/usr/local/include/xulrunner-13.0a1/vm/String.h:67:6: error: multiple
definition of 'enum js::InternBehavior'
/usr/local/include/xulrunner-13.0a1/jsatom.h:555:6: error: previous
definition here
/usr/local/include/xulrunner-13.0a1/vm/String.h:69:23: error:
conflicting declaration 'DoNotInternAtom'
/usr/local/include/xulrunner-13.0a1/jsatom.h:557:23: error:
'js::DoNotInternAtom' has a previous declaration as
'js::InternBehavior js::DoNotInternAtom'
/usr/local/include/xulrunner-13.0a1/vm/String.h:70:18: error:
conflicting declaration 'InternAtom'
/usr/local/include/xulrunner-13.0a1/jsatom.h:558:18: error:
'js::InternAtom' has a previous declaration as 'js::InternBehavior
js::InternAtom'
/usr/local/include/xulrunner-13.0a1/vm/String.h:80:91: error: default
argument given for parameter 3 of 'JSAtom*
js_AtomizeString(JSContext*, JSString*, js::InternBehavior)'
/usr/local/include/xulrunner-13.0a1/jsatom.h:573:1: error: after
previous specification in 'JSAtom* js_AtomizeString(JSContext*,
JSString*, js::InternBehavior)'
In file included from
/usr/local/include/xulrunner-13.0a1/jspropertycache.h:48:0,

[lkcl luke]

allo ben,

ok, again, as this is quite long, let me do a summary:

* i'm again a bit caught out by having to correct some things which it
didn't occur to me that you might not be aware of, apologies for that
- it always catches me off-guard and i don't really know how to handle
it [socially]. so, apologies in advance.

* python's a scripted and dynamic language, and this is GUIs.

a) speed is therefore not a critical consideration
b) the design of pyjamas apps are always ALWAYS single-threaded
c) there IS only ever one profile (hulahop.setup("~/.mozilla/hula")
d) "lazy binding" in pyxpcom takes care of interface changes. we
simply do *not* care.

* there's an entire "cool!" suggestion for an embedded approach, i
won't spoil the surprise by mentioning it here :)

* pyxpcom and hulahop do *not* require any "development": they just
"work". they *only* require "maintenance". the resource allocation
is therefore radically different.

* unit tests and regression tests which can be automated, using
hulahop (and some tricks from the pyjamas-desktop codebase), give the
mozilla foundation an "extra string to the testing bow" of the entire
xulrunner runtime.

On Mon, Feb 27, 2012 at 2:06 PM, Benjamin Smedberg
<benj...@smedbergs.us> wrote:
> On 2/25/2012 7:05 AM, luke.l...@gmail.com wrote:
>>
>> the first question is this: how the heck is it possible that i could
>> create, in under 3 weeks, an entire port of pyjamas-desktop to use the
>> MSHTML IE engine, *without* requiring access to the proprietary MSHTML
>> Trident Engine?
>
> The browser engine provided by Microsoft has a stable COM API. It also does
> not require the embedder to handle any kind of persistent data access:
> history/cache/cookie stores are handled via a single operating system store
> which is shared across processes.
>>
>> the key question is this: what is it about the microsoft technology that
>> makes this possible?  this is not a rhetorical question.  this is not a
>> sarcastic question.  i genuinely require that you actually think, ben, and
>> find out the answer (hint: you already implement one.  or used to.  until it
>> was pulled).
>
> No, we didn't.

the question was generic - ok, you got it (the answer's ActiveX), but
you've slightly misunderstood... or perhaps jumped the gun, i'm not
sure which :)

> The ActiveX control provided an emulation layer which satisifed some of the
> behaviors (allowing gecko to run in a process hidden behind at least the
> basic set of Microsoft interfaces).

yes - you've got it. Microsoft created the (entire!) ActiveX
concept, in order to allow people to be able to embed COM-enabled
controls which conformed to the standard win32 GDI interface into
*any* application.

it didn't - doesn't - matter if that ActiveX component was written 2
or 20 years ago: you grab that DLL (and its associated .tlb) and you
can use it, now and forever.

the reason why i mentioned this was not because the mozilla
foundation created an ActiveX control which installs on the *windows*
platform....

.... i wanted to advocate that the mozilla foundation work with the
free software community to bring about ActiveX-like technology... for
*GNU/Linux* (!) in exactly the same way that it created the
absolutely astounding XPCOM technology.

it's the next logical step.

but if that's too much, then the logical alternative is to rip out
XPCOM and substitute COM for Unix, instead, and then take on some of
the source code from wine [which has had to create a perfect replica
of ActiveX].

don't believe there's

> But because we don't have any form of
> cross-process profile sharing, the embedder either had to configure and run
> their own profile directory (and think about the consequences of only being
> able to use that datastore from one process at a time)

yes - that's exactly what hulahop does. what's the problem? there's
only ever one process, and there's absolutely no problem with that.

the reason why in pyjamas-desktop that that's not a problem is
because... ok, *deep breath*, this is going to need some explaining.

when you compile a pyjamas application to run in a web browser, it
runs in the [single-process, no-threaded] javascript context. with
the exception of google's "gears" plugin, javascript execution in
browser engines is *entirely* "one jobbie". this makes life reaaally
easy from a programming perspective: it's a great way to do things.
no mutexes, no shared memory, naaathing.

if you want multiple "things" going on, you do it as timers, as ajax
etc. etc. but even there, only one "thing" can ever be executed (one
timer, one onload event etc.) this is _great_, because it means that
any "complexity" is done server-side. server-side is where you do
multi-processing, multi-threading, use Rabbit-MQ and other
asynchronous blah blah even multiple AJAX queries.

and the fantastic thing about this type of design, which is very
ideally close to the best MVC-like design principles, is that the
responsiveness of the UI front-end is *not* impeded.

it's great, in other words!

now here's the kicker: pyjamas desktop is ABSOLUTELY NO DIFFERENT.

let that sink in for a moment.

in case it's not obvious what the "big deal" is, i'll explicitly tell
you: a pyjamas-desktop back-end using e.g. hulahop *still* uses
nsITimer, *still* even uses nsIXMLHttpRequest, and, critically,
*still* talks to the EXACT same back-end server-side code that the
[javascript-compiled] pyjs version of the application does.

do you see the implications here?

we've just turned a web browser engine into a world-class GUI Widget
Set which *doesn't* suffer from "blocking" on the GUI front-end
because the development paradigm forces developers to follow
close-to-ideal pure MVC design principles.

the bottom line is that there *are* no processes (other that qty 1),
and if anyone actually ever tried to do something like use
g_object_timer or g_threads they would very very quickly run into
serious trouble with race conditions, risk of over-utilisation of CPU
resources and so on _and_ they wouldn't be able to compile the
application to javascript for running in web browsers (because
g_object_timer doesn't exist in web browsers).

the reason i know this is because i tried using g_object_timer in
early versions of the pyjd hulahop port, and yes, i ran into massive
problems :)

so can you see, the requirements are completely different, here, from
anything else you've previously encountered. pyjamas and
pyjamas-desktop is a _mature_ development environment. we've _been_
through the wringer on these issues.

> or run without any
> data store, which is not a well-tested gecko configuration and will cause
> many websites to fail because cookies and the cache do not behave correctly.

yes, i can see that would be the case.

> In addition, the primary way to access DOM nodes through the MS API is via
> IDispatch, and the bridge between IDispach and COM typeinfo was never
> complete and didn't match the Microsoft implementation in key ways. The
> closest cross-platform equivalent we have to IDispatch is the NPRuntime,
> which is pretty well tested but isn't efficient and has inherent cycle
> issues.

this is python. it's a scripted programming language. we don't
care. lazy dispatch, remember?

plus, i *know* that python-comtypes on win32 actually solved this,
gains access to the MSHTML COM Interface, fires up an IWebBrowser2
ActiveX plugin, and gets on with the job.

you remember i mentioned that "lazy dispatch" stuff? each and every
DOM access (function, property) goes via the "named" lookup, each and
every time. i believe it may be cached (in memory) which is part of
the lazy dispatcher's job.

and i _believe_ that mark hammond, who i believe also wrote the
python-comtypes stuff - also made sure that python-xpcom did likewise
[used lazy dispatch].

this is GUI stuff - it's not speed critical. it just has to work,
and it _does_ work. on pythonwebkit (which is a bit different, as it
has direct python bindings, none of this COM/XPCOM stuff
unfortunately), and when using hulahop, _and_ when using COM on win32.

so i know it can be done; i know also that hulahop actually works,
actually does the job - perfectly - for which it was accidentally
designed _way_ beyond what the hulahop creators actually needed it for
(!)

ironic, huh?

>>
>>
>> and if microsoft can do it, why can't the mozilla foundation?
>
> We *could* do it. But it would require significant engineering effort to
> implement shared profile data, solve the IDispatch-or-equivalent problem
> (especially if we needed it to be safe for object cycles via the cycle
> collector), and then provide real QA and release resources for the project.

ben - hulahop _works_. it doesn't *need* "development", and it
doesn't need shared profile data. one function call -
hulahop.setup("~/.mozilla/hulahopdir") - is all that's needed, and
this has been working - as needed and as designed - since it was
written back in 2006.

all that's needed is for hulahop (and pyxpcom) to be built on the
daily build cycles, and if it breaks, someone tracks down what the
problem is, reverts the commit that broke it until they also create a
patch that updates the code in hulahop (or pyxpcom). and/or makes
damn sure that xulrunner releases *don't* go out the door until
hulahop (and pyxpcom) are patched to work.

the reason for this is because both hulahop and pyxpcom are both
_mature_ products, they're _both_ middleware, and they do *not* need
"development" - just maintenance, to ensure that they're kept
up-to-date with their dependencies (which happen to be mozilla core
libraries unfortunately).

anyway: that aside - apart from hulahop+pyxpcom not being a massive
burden (because they don't need "development", only maintenance),
there really does exist technical development advantages to having a
2nd code-path for regression and unit testing, as that focus-related
bug hints at.

l.

[Joshua Cranmer]

On 2/27/2012 8:42 AM, lkcl luke wrote:
> allo ben, thanks for taking this up.
>
> On Mon, Feb 27, 2012 at 1:53 PM, Benjamin Smedberg
> <benj...@smedbergs.us> wrote:
>> objective-c and COM are fundamentally different in how they achieve
>> backwards compatibility. XPCOM "working" correctly will prevent you from
>> crashing, but it only achieves the goal of being *compatible* if you don't
>> change the interfaces.
> ah.... in static programming languages, yes. in dynamic programming
> languages such as python where you use the trick of lazy / dynamic
> binding (based on the name) - absolutely NOT. the changing of the
> interfaces is COMPLETELY irrelevant.

No. It is irrelevant only if the interfaces are changed in such a way to
preserve backwards compatibility. Removing an interface altogether still
affects dynamic languages. Deleting methods, nontrivially changing
arguments, adding required methods for interfaces to be implemented
(say, nsITreeView) are other ways that destroy backwards compatibility
independent of language. As an example of a real change ongoing right
now that would break you, the parser interfaces are currently undergoing
complete revision (nsIContentSink is going away).

> if a new function was ever needed, or a change to a parameter was ever
> needed, they just added an extra definition (HTMLElement5) and the
> advantage of that is that older (static languages, c, c++, java)
> *still work*, and dynamic languages get to take care of accessing the
> right thing via "lazy dynamic binding". do you see why this is so
> important?

This is roughly what happened in the old model of frozen interfaces for
embedding. We made an explicit decision to trash frozen interfaces so we
could actively remove bad cruft and clean up our code. What you seem to
be arguing is that we need to go back to this model to suit the needs of
embedders; I don't think anyone disagrees with the assertion that this
is the better model for embedders compared to what we have now. However,
we still have a large technical debt of ill-designed interfaces that
need to be cleaned up, and actively maintaining backwards compatibility
is a task we cannot afford to do right now while continuing to push
ahead with all of our changes.

[lkcl luke]

On Sat, Feb 25, 2012 at 4:59 PM, L. David Baron <dba...@dbaron.org> wrote:

> Has PyXPCOM been hooked up to the cycle collector yet, or do uses of
> APIs that depend on cycle collection just leak memory?

david, hi, this had me thinking again. pyxpcom was designed, eek...
a decade ago? but i'm not sure it's entirely relevant. the logic
goes as follows:

* python objects (in c) have their own obj-ref count concept
* XPCOM objects also have their own obj-ref count concept (AddRef)
which is exactly the same as COM, which "inspired" XPCOM.
* when a python object corresponding to an XPCOM object is created,
python takes care of adding *one* and *only* one XPCOM object into the
c data structure, and calls AddRef *once* and *only* once.
* also at the same time, the python object has its PyObject_Ref
increased (by one, obviously).
* when you need that exact same XPCOM object, if mark hammond didn't
program a way to look up the python object which was *already* created
to look after it, i'll fall off my bloody chair.
* so, assuming that mark is the competent programmer that he is, when
you look up an XPCOM object for which a python object already exists,
you DO NOT increase the refcount (AddRef) on the XPCOM object, you
increase the refcount on the *PYTHON* object.
* when the python object's refcount goes to zero, *python's*
garbage-collection takes care of destroying the python object, and in
the "destructor" function (in the c-based pyxpcom module), that code
also performs an XPCOM "Release" call.

so with this in mind, i do not see how the "cycle collector" has any
relevance, because pyxpcom is merely a direct reflector of the XPCOM
objects. but i believe that todd would welcome some input - to todd,
who is the de-facto maintainer of pyxpcom - because he, as the
de-facto maintainer, would be the person with the most experience in
actually sorting it out.

esp. if it's a global function that, if not called, results in XPCOM
objects not getting properly garbage-collected after the "Release" or
something - i don't know, because i don't know what this "cycle
collector" is, or what it does.

just fyi i went through some... interesting times with webkit's
memory management. they don't have anything like XPCOM: it's entirely
c++ classes. a massive suite of virtual base classes that derive from
a class named "DOMObject". mostly :)

at the ultimate lowest level they had to implement the usual "type
id" system which lets you externally identify an object even when it's
type-cast back down to the absolute lowest object in the hierarchy
(gobject has something similar, except they do it entirely in c!).
XPCOM and COM both have something similar, except they drop a GUID
into the type field.

this type id allows you to *BACK*-typecast a c++ object up to its
correct (real) class!

on top of this, they have c++ refcounting (virtual base class, again).

as the concept of XPCOM / COM was entirely missing, they have a
templated infrastructure which allows you to map
object-of-the-bound-type to
object-of-the-webkit-type-when-typecast-down-to-the-base-class-type.

then you have to _manually_ provide a massive suite of "upcast"
functions which check the type-id and call the next "upcast" function
in the chain, after doing a switch statement to get there!

all this to handle the one-for-one representation between (say)
python objects and webkit's c++ objects, or gobjects and webkit's c++
objects.

but even here, you can _still_ do and only need to do one and only
one refcount increase on a per-object basis, because once you've
gotten into the other side (the bindings), the refcounting of *that*
system takes over.

so there are close parallels - it's still complicated as hell though :)

anyway, do let todd know about this "cycle collection" because i have
no idea if he knows about it.

l.

[Boris Zbarsky]

On 2/27/12 11:31 AM, lkcl luke wrote:
> so with this in mind, i do not see how the "cycle collector" has any
> relevance, because pyxpcom is merely a direct reflector of the XPCOM
> objects.

Many XPCOM objects are cycle collected. The problem is that if Python
is holding references the cycle collector doesn't know about, then the
cycle collector won't be able to find all cycles.

Just think about this case: a Python object A (reflecting XPCOM object
A') has a reference (possibly indirectly, via other Python) to Python
object B (reflecting XPCOM object B'). The underlying B' object has a
reference to A'.

How will the memory for all this ever get freed?

-Boris

[lkcl luke]

btw, ben, about speed of using XPCOM through pyxpcom+hulahop...

when i said we don't care [because working is a higher priority than
speed, and it's a GUI anyway], i think it helps to give some context.

for the pyjs (compiled) version of any pyjamas application, the fact
that python is a dynamic language, being translated to another dynamic
language, gives a performance hit that's so high it's difficult for
people to believe it could be that bad, just to obtain python feature
interoperability.

[with the end result that they go off and implement their *own*
python-to-javascript compiler, in some cases. this has actually
happened about four times, now!]

i spoke about this on b2g a couple of months back, when chris jones
(bless him) rather naively said that javascript was acceptable as the
de-facto "language of the web" implication being in absolutely all
circumstances because of projects such as dart, gwt and pyjamas.

if he'd ever run pyjs he'd've known that that simply isn't true,
because to even do something as simple as use the "add" operator, you
must check first if the two objects being added actually exist - which
must be done dynamically in case __getattr__ has to be used; then you
have to check the type of both objects (int? float? string?) and go
through a lookup table (!!) to find out if the two are compatible;
then you must coerce one type to the other; then _finally_ you can
actually add the two objects.

that very _very_ small and i mean small description should have you
going "oops..." and it should emphasise that maybe javascript as a
compiler target isn't such a good idea after all - not for dynamic
languages, anyway.

by contrast: for _static_ programming languages like java, c and c++,
it's great!

but for pyjamas, the performance hit is so high that when you run the
exact same application under pyjd it's a five-fold performance
improvement.

against that kind of background, and bearing in mind that that's still
using name-based lookups for absolutely each and every single access
to absolutely every single one of the functions, objects and
properties of DOM/HTML5, what we have right now looks very very good.

speed's *not* critical. we're using xulrunner+xpcom+hulahop as a GUI
widget set. a very powerful and flexible one. pyjd doesn't have to
be lightning-quick. i've installed it on 400mhz ARM9 CPUs, and it had
perfectly acceptable performance.

l.

[Asa Dotzler]

On 2/27/2012 1:14 PM, lkcl luke wrote:
> btw, ben, about speed of using XPCOM through pyxpcom+hulahop...

Luke, it seems like you've ignored a very important reply to your
ongoing attempts at educating Mozillians about the value of your
project. I urge you to re-read what Benjamin said in a previous reply:

> I don't think that python-hulahop is strategically important to our core

> goals of innovation on the internet. So I don't think that in terms of

> the Mozilla mission our core community should make maintenance and
> development of that code a priority. I'd be happy to provide a place for
> you or somebody else to build a community around python-hulahop, though.

- A

[lkcl luke]

the policy of relegating pyxpcom to "third party status" within the
mozilla codebase has resulted in these kinds of things, below.

you will recognise this as the consequences of a decision some time in
the past few weeks (which i wasn't party to, because i am an
"outsider" having no funds or time to pay 24x7 attention to each and
every decision made by the people who _are_ funded by the mozilla
foundation)

basically that decision looks like it involved deliberate splitting of
the header files into "public api" and "not public api".

so, this header file nsIProxyObjectManager.h is treated as "internal api"...

... but somebody forgot that pyxpcom needs it... because they have
relegated pyxpcom to "third party status".

they forgot that pyxpcom *needs* to be built as an "internal"
component, yet the mozilla foundation has completely failed to provide
a system for building such components, yet at the same time has
basically dumped the responsibility onto other people.

if however pyxpcom was actually part of mozilla-central, then not only
would this problem simply not be present but also it would be a damn
sight easier to find issues.

temporarily i am forced to manually seek out, copy and make available
this "nsIProxyObjectManager.h" file each and every damn time i do a
rebuild and reinstall of a particular version of the xulrunner
runtime.

that's incredibly tiresome and it's part of the *additional* burden
that the mozilla foundation's decisions are placing onto external
developers.

... developers that are *not* funded by the mozilla foundation, it's
worth once again pointing out.

right now i cannot agree more that the OLPC team made the right
decision to remove all mozilla-foundation-funded technology from their
machines.

l.

c++ -o _xpcom.o -c -I../../../dist/system_wrappers -include
../../../../config/gcc_hidden.h -DOSTYPE=\"Linux2.6\" -DOSARCH=Linux
-DMOZ_NO_MOZALLOC -I/usr/include/python2.7
-I../../../../xpcom/src/module -I. -I../../../dist/include
-I../../../dist/include/nsprpub
-I/usr/local/lib/xulrunner-devel-13.0a1/include
-I/usr/local/lib/xulrunner-devel-13.0a1/include/nsprpub
-I/usr/include/nspr -fPIC -fno-rtti -fno-exceptions -Wall
-Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy
-Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof
-Wno-long-long -pedantic -DXULRUNNER_13_0A1 -std=gnu++0x -fshort-wchar
-fno-strict-aliasing -pipe -std=gnu++0x -DNDEBUG -DTRIMMED -Os
-freorder-blocks -fno-reorder-functions -DMOZILLA_CLIENT -include
/usr/local/lib/xulrunner-devel-13.0a1/include/mozilla-config.h
-Wp,-MD,.deps/_xpcom.pp ../../../../xpcom/src/module/_xpcom.cpp
../../../../xpcom/src/module/_xpcom.cpp:68:35: fatal error:
nsIProxyObjectManager.h: No such file or directory
compilation terminated.

[Boris Zbarsky]

On 2/28/12 1:05 AM, lkcl luke wrote:
> so, this header file nsIProxyObjectManager.h is treated as "internal api"...

That header file doesn't exist anymore, because XPCOM proxies don't
exist anymore.

As you'd know from a basic 'find' or grep.

But it's more fun to rant, I guess.

> if however pyxpcom was actually part of mozilla-central

Then it would be a tax on everyone doing mozilla-central development.
That seems like a bad tradeoff to me.

> temporarily i am forced to manually seek out, copy and make available
> this "nsIProxyObjectManager.h" file each and every damn time i do a
> rebuild and reinstall of a particular version of the xulrunner
> runtime.

Good news. You can stop, because it just doesn't exist. ;)

-Boris

[lkcl luke]

https://bugzilla.mozilla.org/show_bug.cgi?id=731121

where is the information required to fix this problem?

also, why was pyxpcom excluded from the update.... my god this was _months_ ago!
https://bugzilla.mozilla.org/show_bug.cgi?id=675221

why was pyxpcom not included in that update, when the information was
fresh in everyone's minds??

oh i know - because pyxpcom has been relegated to third party status.

ok.

how is this going to get fixed? (a so that this stops happening b
these specific coding errors so i can get some working code)

On Tue, Feb 28, 2012 at 6:05 AM, lkcl luke <luke.l...@gmail.com> wrote:
> the policy of relegating pyxpcom to "third party status" within the
> mozilla codebase has resulted in these kinds of things, below.
>
> you will recognise this as the consequences of a decision some time in
> the past few weeks (which i wasn't party to, because i am an
> "outsider" having no funds or time to pay 24x7 attention to each and
> every decision made by the people who _are_ funded by the mozilla
> foundation)
>
> basically that decision looks like it involved deliberate splitting of
> the header files into "public api" and "not public api".
>

> so, this header file nsIProxyObjectManager.h is treated as "internal api"...
>

> ... but somebody forgot that pyxpcom needs it... because they have
> relegated pyxpcom to "third party status".
>
> they forgot that pyxpcom *needs* to be built as an "internal"
> component, yet the mozilla foundation has completely failed to provide
> a system for building such components, yet at the same time has
> basically dumped the responsibility onto other people.
>
> if however pyxpcom was actually part of mozilla-central, then not only
> would this problem simply not be present but also it would be a damn
> sight easier to find issues.
>

> temporarily i am forced to manually seek out, copy and make available
> this "nsIProxyObjectManager.h" file each and every damn time i do a
> rebuild and reinstall of a particular version of the xulrunner
> runtime.
>

[Kyle Huey]

On Wed, Feb 29, 2012 at 4:30 PM, lkcl luke <luke.l...@gmail.com> wrote:

> <https://bugzilla.mozilla.org/show_bug.cgi?id=731121> oh i know - because

> pyxpcom has been relegated to third party status.
>

Yes, exactly.

how is this going to get fixed? (a so that this stops happening b
> these specific coding errors so i can get some working code)
>

It's not going to get fixed unless someone steps up to fix it. And as has
been stated several times, the core Mozilla community has no interest in
doing so.

- Kyle

[lkcl luke]

On Thu, Mar 1, 2012 at 1:07 AM, Kyle Huey <m...@kylehuey.com> wrote:
> On Wed, Feb 29, 2012 at 4:30 PM, lkcl luke <luke.l...@gmail.com> wrote:
>>
>> oh i know - because pyxpcom has been relegated to third party status.
>
>
> Yes, exactly.

now i am spending considerable time spending considerable core
mozilla community's time dealing with the mess caused by that
decision.

>> how is this going to get fixed? (a so that this stops happening b
>> these specific coding errors so i can get some working code)
>
>
> It's not going to get fixed unless someone steps up to fix it.

ok, this is inappropriate to be discussing when there is still
silence from ben when i've answered the issues he raised, pointed out
that they are moot, and am waiting for him to run and install
pyjamas-desktop.

all this is with a view to getting on the same page in order to
demonstrate what the benefits of doing so are...

... but for the sake of argument, even though it risks wasting
everybody's time, i'm going to provide an answer - because you asked.

a) who is going to pay for their time?

b) who is going to train them or answer their questions?

c) how much time is it going to take to train them and answer their questions?

... it's going to be the core mozilla developers, isn't it?

how long do you think it would take the core mozilla team to keep a
small codebase like pyxpcom up-to-date vs having to continuously
answer under-informed bugreports such as this one, here:

https://bugzilla.mozilla.org/show_bug.cgi?id=731121

boris, for example, who has been extremely helpful up until now, is
getting _really_ fed up - and that's just the one bugreport!

i'm going to keep on, and on, and on, and on, and on, and on at this
until i get answers, and pyxpcom works.

then, in a couple of months, i'm going to try again. and i'll find
more bugs. and i'll raise bugreports again and again until i get
answers.

and that loop will be repeated, working from an under-informed
position where i don't have the time or money to attend mozilla
conferences, or listen to mozilla phone calls (b2g), or trawl through
mozilla documentation, or trawl through millions of lines of mozilla
source code.

are you _getting_ it yet?

i don't _like_ having to do this kind of thing. it's not fun for me
to have to point these kinds of things out, but somebody has to.

>  And as has
> been stated several times,

... and as the points which _answer_ those statements have been
ignored several times...

> the core Mozilla community has no interest in
> doing so.

kyle, that's really not helpful right now.

this is complex: the core developers don't understand the issues, and
i'm endeavouring to help them to understand.

please don't interfere.

thank you.

l.

[Justin Lebar]

> i'm going to keep on, and on, and on, and on, and on, and on at this
> until i get answers, and pyxpcom works.

Luke, I think it is unlikely that you will accomplish your goals if
you make trolling your primary strategy.

We are pretty bad at ignoring trolls, here at Mozilla, but I expect
that if you troll hard enough, we will eventually learn to ignore you.

You've tried to argue that it's in our best interests to maintain the
python-xpcom bindings. Those in this thread have heard your argument,
and they disagree. You think that's our loss, and maybe you're right!

But just because you think we're wrong doesn't give you the right to
waste everyone's time. You are not entitled to anyone's attention
here or in bugzilla.

Maintaining xpcom bindings for Python is work! You're probably right
that it would be more work for you than it would be for a core xpcom
contributor. But it's work either way, and I think you are unlikely
to convince someone to do work by threatening to be a troll.

Regards,
-Justin

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[lkcl luke]

On Thu, Mar 1, 2012 at 2:29 AM, Justin Lebar <justin...@gmail.com> wrote:
>> i'm going to keep on, and on, and on, and on, and on, and on at this
>> until i get answers, and pyxpcom works.
>
> Luke, I think it is unlikely that you will accomplish your goals if
> you make trolling your primary strategy.

i don't know what trolling is, and you are probably mistaking
trolling for pathological bug-fixing and 100% focus on getting a goal
completed irrespective of "trying to make friends".

but you are dragging the conversation off topic. i am still waiting
for ben to acknowledge the points made regarding his misunderstanding
of the situation regarding the continuous changes to XPCOM interfaces,
which are absolutely irrelevant to dynamic programming languages.

and, if that was the primary justification for destroying pyxpcom
then it is a mistake.

> You've tried to argue that it's in our best interests to maintain the
> python-xpcom bindings.  Those in this thread have heard your argument,

no they haven't. i'm not done yet.

there's more to this.

so far i've explained to chris that javascript as a compiler target
of a dynamic programming language results in an order of magniture
performance hit; to ben that he's mistaken about the binary
compatibility issue and so on.


> and they disagree.

how can you disagree with an incomplete perspective? i don't understand.

>  You think that's our loss, and maybe you're right!

maybe? there shouldn't be *any* "maybe". you're supposed to be
world leaders at the forefront of technology. there should _be_ no
"maybes".

> But just because you think we're wrong doesn't give you the right to
> waste everyone's time.  You are not entitled to anyone's attention
> here or in bugzilla.

you are assuming that this is ego-driven. where is the money that i
am being paid to do this work on behalf of others? where is it? can
you please tell me where the personal motivation is, please?

have you seen any?

mm?

you've already badly let down the OLPC community. that means that
it's now my responsibility to the pyjamas community to get this done,
and i'm PISSED that it's going to cost _me_ money and time when i've
been earning UNDER POVERTY LEVEL INCOME FOR FOUR FUCKING YEARS.

do you not fucking get it? we got EVICTED last year - myself, my
partner and my 2-year-old daughter, because i stick to free software
principles and have to turn down lucrative contracts where people in
the United States violate the Treaty of Rome's clauses on intellectual
property ownership.

you guys have absolutely no idea. i was _homeless_ for two years,
from about 2007 to 2009 for god's sake. and i _still_ kept to free
software principles and still kept working on free software.

> Maintaining xpcom bindings for Python is work!  You're probably right
> that it would be more work for you than it would be for a core xpcom
> contributor.  But it's work either way,

yes, it is. so go on. ban me. come on. go stick your fingers in
your ears. make the decision to revoke my bugzilla account (like the
webkit team did, after their deliberate bullying). see how far that
gets you.

the damage to the free software community of mark rowe's bullying was
- is - _immense_.

you want to do the same thing? go ahead. because if you don't start
taking responsibility for this instead of going "nur, nur it's not our
core focus", i _will_ absolutely pathologically drive you to that.

> and I think you are unlikely
> to convince someone to do work by threatening to be a troll.

i don't actually know what a troll is.

however i _will_ be absolutely pathological, persistent and get this done.

it would be very very good if you found someone _else_ to do the work.

anyway.

you've made me extremely angry, but let's forget that that happened, shall we?

now, back to the subject at hand.

here's where we are: i'm waiting for ben to acknowledge that the
points made about the dynamic programming language bindings to XPCOM
interfaces make no odds regarding this "binary compatibility"
argument.

he very kindly raised the point, which gives us an insight into why
the mistake was made.

now.

ben.

without prejudice.

it's okay to admit that you made a mistake.

nobody is going to think anything less of you if you say "ah, i made
an error of judgement because i didn't know how pyxpcom+hulahop works"

do you acknowledge that for dynamic programming languages which use
"lazy binding", binary compatibility with XPCOM interfaces is
irrelevant?

i'm keenly aware that all decisions have been made to destroy pyxpcom
based on the mistaken assumption that binary compatibility is
important to dynamic programming languages.

a simple yes would suffice, then we can move on to the next stage, ok?

l.

[lkcl luke]

... by the way: if you get me some money, so that i can actually focus
on this properly, you will find that my attitude COMPLETELY changes.

i won't have to worry about where next month's rent is coming from.

why are you surprised that i am angry about being backed into a corner
where i am forced to work on some free software - for the benefit of
OTHER PEOPLE?

i've BEEN through this. you remember samba? i broke the back of the
NT domains reverse-engineering... and the lead developers took the
credit, and got the contracts and funding. you've heard of the
openexchange project? libmapi? the same thing happened with the
Exchange 5.5. _again_ i got the ball rolling there - it cost me 12 to
14 hours a day for about 4 months, and i got _nothing_.

i'm sick of being taken advantage of, folks. it's not ok.

[Justin Dolske]

On 2/29/12 8:18 PM, lkcl luke wrote:

> but you are dragging the conversation off topic. i am still waiting
> for ben to acknowledge the points made regarding his misunderstanding
> of the situation regarding the continuous changes to XPCOM interfaces,
> which are absolutely irrelevant to dynamic programming languages.

No, this thread is basically already over. You've had replies from the
people experienced in this area, and your replies are ratholing in
technical details while missing the big-picture problems.

To put it more bluntly, pyxpcom simply isn't a priority for Mozilla, and
that situation is not going to change. The web has _much_ bigger battles
to fight.

> do you not fucking get it? we got EVICTED last year - myself, my
> partner and my 2-year-old daughter, because i stick to free software
> principles and have to turn down lucrative contracts where people in
> the United States violate the Treaty of Rome's clauses on intellectual
> property ownership.
>
> you guys have absolutely no idea. i was _homeless_ for two years,
> from about 2007 to 2009 for god's sake. and i _still_ kept to free
> software principles and still kept working on free software.

Kudos to you for living up to you convictions. Seriously. But your
personal issues really are not relevant to the topic at hand, nor likely
to bring people about to your point of view.

Justin

[Benjamin Smedberg]

On 3/1/2012 9:32 AM, jameso...@gmail.com wrote:
> It seems that Luke has ended up in Ben's explicit or implicit killfile.
This thread is now way off-topic. It was originally an announcement
about how we're removing *binary-stable* embedding APIs, and it has
morphed into a discussion about the relative merits of pyxpcom, which
didn't use these embedding APIs and was unaffected. We are all quite
clear that pyxpcom clients don't have to worry about binary changes
because they are insulated by pyxpcom middleware and have the benefits
of being a scripted language.

However, pyxpcom has *never* been built or maintained as part of the
regular Mozilla build/test infrastructure. It was originally in the CVS
tree because that's where you put extension projects. It has since been
moved into its own mercurial repository, and has a module owner, Todd
Whiteman (ActiveState), and mailing list:
http://listserv.activestate.com/mailman/listinfo/pyxpcom

As moderator of both the mozilla.dev.platform and mozilla.dev.embedding
forums, I request that any further discussion about the future of
pyxpcom and trying to build a community around it be moved to the
pyxpcom mailing list; it is completely offtopic for the dev.platform
list and has been thoroughly discussed in dev.embedding.

--BDS

[Bobby Holley]

>
> Does anyone have an idea about how much work it would take to maintain
> python-xpcom?
>

Lots.

I don't know much about pyxpcom, but I imagine it being pretty analogous to
XPConnect (our JS<->XPCOM bridge). XPConnect is probably the most notorious
module in Gecko - it's complicated, scary, and easy to get wrong. We
already sink a ton of engineering resources into it, and wouldn't if it
weren't so fundamental to our entire architecture.

More importantly, for a number of reasons the DOM is becoming more
intimately tied to JS: we're removing the abstraction layer that
sorta-kinda makes the DOM language-agnostic (it's too slow), and we're
writing new custom DOM bindings.

So if it isn't true already, pyxpcom won't be able to script the DOM
anymore, which makes it significantly less useful. Reversing this course
(and maintaining a separate set of python DOM bindings) is pretty much out
of the question: it incurs a performance penalty for the Web that we don't
want to pay, and it would take months and months of engineering by Boris,
Kyle, Peter, Blake, Ben, Johnny, Olli, and myself to make happen.

In a nutshell, script bindings aren't easy - this stuff is some of the most
complicated and delicate code in Gecko. So the issue isn't that Benjamin is
too lazy or proud to keep the relevant embedding APIs. The issue is that
there is a very small group of people in the world who have the knowledge
to build and maintain something like this, and they have other priorities.

bholley