proprietary packages?

[Carlos Mundi]

Hi. I make this post with the understanding that it could be controversial. I do not intend or have interest in controversy.  I'm posting to the 'dev' list rather than the 'user' group because I think discussion here will better informed and civil.

The short version of my question is, "Is it possible to write a package in the Julia language and distribute it without the source?"

I've read about the dramatic reduction of the startup time of Julia by a variety of methods, including precompiling Base. I've even tinkered a bit with the codegen Internals of the standard library. And I've noticed some discussion of precompiled packages as a possibility. All this got me wondering whether proprietary packages for Julia are possible. Proprietary need not mean commercial. What I mean by "proprietary package" is a software component implemented in Julia, loadable by Julia, with objects and methods accessible from Julia, distributed in an executable-by-Julia form without source code in the Julia language.

A few possible implementations come to mind. It sure would be nice to be have a magical Julia-to-C compiler. Another approach would be to generate the LLVM IR (one member for each specialized dispatch tuple needed) for each method of the Julia source of the package, and then load the IR from Julia -- that might already be easy. Yet another approach might be be to generate the assembly code for each target architecture (again for each specialization) and produce a shared lib which would be called from a Julia wrapper. Of course the original Julia source must not refer to any copyleft-ed code, as that would be disallowed linking.

A key thing that all these approaches have in common is that the (nondisclosed) source code is written in Julia. Writing in C just isn't as appealing as it used to be. :)

Is this already possible?  Are there plans to make it possible?

Thanks.

[Jeff Bezanson]

Part of the answer is to see that julia libraries are a lot like C++
template libraries distributed as header files. For example, if a
library implements a new numeric type, there's not much point in
distributing it in a way that prevents its code from being inlined
into user code.

An approach that handles this case is something like "p code", where
you parse each of the top-level expressions in your source file and
serialize them to binary format. Then you can write a wrapper that
loads them and calls "eval" on them. This hides the original source
text, but you can still look at the compiler's intermediate
representation.

At the other extreme, eventually you'll be able to statically compile
a whole program, and it can expose C-callable entry points. It will
look exactly like a library written in C, except that it might link
against the julia runtime. Then you will also be limited to the C ABI.

It would be kind of interesting to design something in between. That
would probably look like a shared library in a special format designed
to be loaded into the julia runtime. It would contain cached generated
native code (like sys.so), and some kind of type information in a
special data section.

Obviously I'm not excited about enabling proprietary code, but
ultimately I feel that's a decision for the code's author to make, not
for me to make.

[Carlos Mundi]

Jeff,

Thanks for your very thoughtful response.  I'm mulling it over. 

More than not excited, I'm actually a bit ashamed to even ask about proprietary extensions. That said, I've seen this play out in a commercial setting a couple times.  And I'm cautiously optimistic that the Julia community can benefit from enabling proprietary extensions.  True, someone could take the MIT-licensed core of Julia and build it into a proprietary product already. But will they want to? Probably not if keeping up with Julia features is important to their business, which presumably is the case. So that innovation never happens, or it happens completely behind closed doors.  Some might say, "So what? It's not our problem," and they'd be right. But there's a chance that more people will do more cool things with Julia in the daylight if creating proprietary packages is easy. I imagine two possible outcomes. If a proprietary package stinks, only the selfish developer was hurt and hopefully realizes that maybe sharing would have worked out better. (Call me an optimist.) If a proprietary package turns out to be the best thing since linear algebra, then we'll see mobs of FOSS developers replicating it -- something which might never have been developed without the proprietary incentive.

As I say, I'm mulling over the the technical details behind your ideas. It may take me a few days but I want to see where this could go.

Thanks!

[Stefan Karpinski]

I'm fine with people distributing proprietary packages written in Julia. What I'm not ok with is making design decisions about the language specifically to enable doing this unless those decisions are also good for the language otherwise. That is, if we can facilitate proprietary distribution with otherwise useful (or at least not detrimental) features, great, but we're not going to add detrimental features to facilitate proprietary distribution.

[Ivar Nesje]

Automatic obfuscation of a program is hard. Especially if your attacker is a knowledgeable human that does not respect the licensing agreement. I think there will be much easier to get volunteers to write a deobfuscator for the p-code, than to implement the obfuscator in the first place.

[Carlos Mundi]

@Stefan - That's perfectly reasonable. It would be completely unreasonable to inconvenience the community. This works only if it works for everyone, or at least does not get in the way.

[Carlos Mundi]

@Ivar - compilation is a form of obfuscation. And it's vulnerable to disassembly.  It's all about relative costs, as you say.  As I noted above, any good module would be greeted by an army of volunteers to replicate it in open form. And typically a clean room implementation would be even cheaper (in lifetime costs) and technically superior to a de-obfuscation.  So you could accuse me of trying to lure commercial development into the fold just so their best ideas can be picked for the greater good.  I would admit to that.  :)  

But in case I was unclear, I'm not looking for volunteers. I'm just wondering about (1) the technical feasibility; and (2) whether the reception would be hostile, constructively indifferent, or enthusiastic. Let's just rule out enthusiastic right now. :)  Constructive indifference -- "live and let live, just don't tread on anybody" -- would be ideal. Hostility by the core Julia devs would make the exercise pointless. This parallels the discussion of loadable kernel modules for Linux, and anyone who goes down this road must be self-reliant. 

Thanks.

[Tobi]

Just my 2cents on this. I am pro proprietary code in Julia for the simple reason that I am working at a company and am sure that this is a win-win situation.

The typical comercial Julia application will use some open packages and when some of these misses something one will contribute upstream. The only thing one usually don't want to open is a) "the secrete algorithm" and b) all the boring application logic which is to specific to be valuable to anyone but you don't want that competitors to take your code and start from that.

On the big picture I think that code hiding will be a key feature when companies have to decide whether to use e.g. Python or Julia.

But obviously support for this will not come from the Julia devs. At least until one hires one of them ;-)  

[Gour]

On Sun, 2 Mar 2014 00:21:17 -0800 (PST)
Tobi <tobias...@googlemail.com> wrote:

> The typical comercial Julia application will use some open packages
> and when some of these misses something one will contribute upstream.
> The only thing one usually don't want to open is a) "the secrete
> algorithm"

This one is a good one.

> On the big picture I think that code hiding will be a key feature
> when companies have to decide whether to use e.g. Python or Julia.

At the moment I'm still evaluating (Open)Dylan and possibility to create
binary executables is nice 'pro'.

Another, to me, useful use-case is simplicity of deployment - even for
open-source projects which is my primary concern to use the language -
iow. ability to easily put together all the components and ship them
within binary installer for Windows and binary package which is, afaik,
used on Mac OS.


Sincerely,
Gour

--
One must deliver himself with the help of his mind, and not
degrade himself. The mind is the friend of the conditioned soul,
and his enemy as well.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Carlos Mundi]

@ Gour - I do not see how proprietary modules would simplify deployment in any way that cannot be achieved with 100% MIT licensing.

I never became a fan of Dylan (the language; Bob is a genius) and did not know it supported binary modules.  Or do you just mean compiled whole programs?

@Tobi

I'd like to think that this could be good for everyone, and actually encourage companies to support (with paid hours) employees who want to contribute to Julia.  I'd like to be able to make a case to my management:

1. You're right, Mr. Spacely. We can't let Cogswell beat us to market again. 

2. Our product is 10x faster and twice as accurate, but they got there first.

3. We need to develop the new Embedded Sprocket 3000 smarter -- with the development speed of scripting and the runtime performance of C. 

4. My wife, Jane, told me about her friend, Julia, who can do all that and gives it away. Yes, really.

5. And the best bit is Julia is friendly to commercial development. I mean, she runs her own non-profit organization but she won't stop us.

6. In fact if we *contribute* helpful but non-secret infrastructure code to Julia under the MIT license, then some of Julia's friends might start using it and make it better.

7. But even if they don't, we still get to be Good Guys -- the board is always encouraging us to find ways to do community service, right?

8. Well this way, the company gets great PR for something we had to do anyway --- without giving up any real company secrets of course.

You see where I'd like to be headed. It's not about money. Influence, yes. :)  It may be a long way off. Or maybe not.

Anyway, talk is cheap and there's code to read (and write)!

[Gour]

On Sun, 2 Mar 2014 09:49:34 -0700
Carlos Mundi <cmu...@gmail.com> wrote:

> @ Gour - I do not see how proprietary modules would simplify
> deployment in any way that cannot be achieved with 100% MIT licensing.

Well, it's, of course, vice versa, iow. having ability to produce binary
exectutable would make it easier to hav prop. modules.

> I never became a fan of Dylan (the language; Bob is a genius) and did
> not know it supported binary modules.

I must say that I like many of its features and find it very similar, at
least in the concept, to Julia.

> Or do you just mean compiled whole programs?

Yes, I was thinking about Dylan's native code generation.


Sincerely,
Gour

--
Perform your prescribed duty, for doing so is better than not
working. One cannot even maintain one's physical body without work.