[belenix-discuss] Something that I've been working on

[Sriram Narayanan]

Hi all:

After dog fooding rpm5 for over a year, and pacman for the past two
months, I've decided to look at the next set of problems - release
engineering.

After facing enough issues with being unable to boot OI b151a7 on
various new Dell laptops, I've decided to explore release engineering
for Belenix in particular and Illumos in general.

Based on some private experiments, I made a presentation at Foss.IN
today. Here are my slides:
http://www.slideshare.net/sriramnrn/continuous-integration-for-open-source-distros-v-30

I will send links to the video of my talk once the videos are made
available online.

Next steps: Host at belenix.v12.su what ever I have done on my
personal system. The basics will take till the end of December at
least, since I'm going on leave on Dec 2 and will be back on Dec 12.

Sriram
--
Belenix: www.belenix.org
_______________________________________________
belenix-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/belenix-discuss
http://groups.google.com/group/belenix-discuss

[Irek Szczesniak]

On Thu, Nov 29, 2012 at 6:23 PM, Sriram Narayanan <sri...@belenix.org> wrote:
> Hi all:
>
> After dog fooding rpm5 for over a year, and pacman for the past two
> months, I've decided to look at the next set of problems - release
> engineering.
>
> After facing enough issues with being unable to boot OI b151a7 on
> various new Dell laptops, I've decided to explore release engineering
> for Belenix in particular and Illumos in general.
>
> Based on some private experiments, I made a presentation at Foss.IN
> today. Here are my slides:
> http://www.slideshare.net/sriramnrn/continuous-integration-for-open-source-distros-v-30
>
> I will send links to the video of my talk once the videos are made
> available online.
>
> Next steps: Host at belenix.v12.su what ever I have done on my
> personal system. The basics will take till the end of December at
> least, since I'm going on leave on Dec 2 and will be back on Dec 12.

Thank you for the update!

Irek

[Sriram Narayanan]

I should clarify, I decided to address release engineering first, since the whole non-oracle Solaris community will need to get together to solve the problem of tracking package changes, have the tools to solve posix compliance debates,  and then start to write device drivers.

Sriram

[Joerg Schilling]

Sriram Narayanan <srir...@gmail.com> wrote:

> I should clarify, I decided to address release engineering first, since the
> whole non-oracle Solaris community will need to get together to solve the
> problem of tracking package changes, have the tools to solve posix
> compliance debates, and then start to write device drivers.

Well, this is something I mention since years. I still hope there is a way to
collaborate - note that even the *BSDs that have been disintegrated since a
long time do important things together.

What we of course first need is to decide on a minimal set of consensus.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[Irek Szczesniak]

On Fri, Nov 30, 2012 at 1:13 PM, Joerg Schilling
<Joerg.S...@fokus.fraunhofer.de> wrote:
> Sriram Narayanan <srir...@gmail.com> wrote:
>
>> I should clarify, I decided to address release engineering first, since the
>> whole non-oracle Solaris community will need to get together to solve the
>> problem of tracking package changes, have the tools to solve posix
>> compliance debates, and then start to write device drivers.
>
> Well, this is something I mention since years. I still hope there is a way to
> collaborate - note that even the *BSDs that have been disintegrated since a
> long time do important things together.
>
> What we of course first need is to decide on a minimal set of consensus.

Which minimum set of consensus? The point is to go boldly forward with
the userland modernization where Illumos has failed with it's
conservative policy.

Irek

[james]

> Which minimum set of consensus? The point is to go boldly forward with
> the userland modernization where Illumos has failed with it's
> conservative policy.

Isn't that a major sticking point? Many Solaris old-timers don't
welcome such change.

It did seem to me that the OpenSolaris mechanism for different flavours
had merit, but with limited resources and a desire to be Not Linux, I
would have thought a sane approach would be to just take FreeBSD
userland and cooperate as much as possible. After all, they are using
ZFS and the approach to standardisation vs BSDism vs some progress has
always seemed to me quite measured.

[Joerg Schilling]

james <ja...@mansionfamily.plus.com> wrote:

> > Which minimum set of consensus? The point is to go boldly forward with
> > the userland modernization where Illumos has failed with it's
> > conservative policy.
>
> Isn't that a major sticking point? Many Solaris old-timers don't
> welcome such change.

Correct and for this reason, Illumos did change too much at some places already.

Solaris Old-timers like to have standard compatibility and backwards
compatibility.

It seems that the Illumos people e.g. insist in having their private broken
"od" instead of using this implementation:


http://hg.berlios.de/repos/schillix-on/file/af01495dc19c/usr/src/cmd/hdump

That is POSIX compliant and fully compatible (except for bugs) to the old Sun
closed source od.

The minmum consensus may be very tiny...

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[james]

> Correct and for this reason, Illumos did change too much at some places already.
>
> Solaris Old-timers like to have standard compatibility and backwards
> compatibility.

Well, as an outside observer who has worked on Solaris and Linux
professionally, I'd have to say:

Some of the behaviour wrt these issues has been bizarrely focussed on de
jure standardisation and backwards compatibility *for its own sake* and
one can but hope that some lessons might be learned by recent events.

I'm not joking when I suggest just throw in with FreeBSD. It is at
least a neutral venue compared to the two positions we've had with
solaris-stuck-in-the-muds and lets-be-linux-brigade, and I don't see too
much moaning about either gratuitous change or stasis one way or the
other about FreeBSD. It also has active user-space and workstation
development from PC-BSD and active NAS development too.

I use Red Hat daily at work and any non-standardisation is mildly
annoying at worst, and generally completely irrelevant and a non-issue.

Being compatible with FreeBSD might not be backwards compatible with
Solaris, or compatible with the selfish anarchy of Linux, but it is at
least compatible with something that isn't either already totally
irrelevant or rapidly becoming so.

What the free solaris kernel seems to need most is device drivers and if
the way to achieve that is in fact to sacrifice all the old de jure
standardisation for its own sake, then I'd throw my support that way
anytime: I already gave up on OI derivatives because FreeBSD actually
works on my hardware and I can have ZFS that way.

[Joerg Schilling]

james <ja...@mansionfamily.plus.com> wrote:

> > Correct and for this reason, Illumos did change too much at some places already.
> >
> > Solaris Old-timers like to have standard compatibility and backwards
> > compatibility.
>
> Well, as an outside observer who has worked on Solaris and Linux
> professionally, I'd have to say:
>
> Some of the behaviour wrt these issues has been bizarrely focussed on de
> jure standardisation and backwards compatibility *for its own sake* and
> one can but hope that some lessons might be learned by recent events.

Solaris has been focussed on backwards compatibility at the same time as POSIX
compliance. Well, this was before Sun created Indiana....

> I'm not joking when I suggest just throw in with FreeBSD. It is at

What do you understand by "just throw in with FreeBSD"?

Are you talking about adding programs from FreeBSD?

> least a neutral venue compared to the two positions we've had with
> solaris-stuck-in-the-muds and lets-be-linux-brigade, and I don't see too
> much moaning about either gratuitous change or stasis one way or the
> other about FreeBSD. It also has active user-space and workstation
> development from PC-BSD and active NAS development too.

I believe the best way is to add FreeBSD programs if the following is true at
the same time:

a) we need a program for some reason (e.g. because it is still
closed source in OpenSolaris ON)

b) FreeeBSD has such a program under the BSD license

c) The program from FreeBSD matches our needs in a sufficient
way.

If a) is true but b) and c) are not true also, we need to write new software.
This is what I did with my "od" that is based on my "hdump" from 1986.

If we on the other side have a problem with missing features, we rather should
extend existing UNIX sources.

In this area, Illumos fails for some reasons:

- People do not deliver quality: A quick "od" hack from Garret Damore that
is not POSIX compliant, that is not compatible with Svr4 and that even
does not work correctly with files > 2 GB does not belong into an
OpenSolaris continuation project.

- I software from FreeBSD is taken, it must be made working correctly
before it can be used as a closed source replacement. The i18n adds
to libc from FreeBSD obviously do not work correctly on Illumos and as
a documented result, "col -x" fails to pass jaopanese characters on
Illumos.

- Replacing other software instead of fixing bugs is a very bad idea.
Replacing troff/man because "col -x" does not work as a result from a
bug in i18n is the beginning of a destuctive hack.

- Illumos does not work on extending the current UNIX userland to become
compatible to newer POSIX versions and Illumos does not import existing
code from people that are not hired by Nexenta or similar companies.

> Being compatible with FreeBSD might not be backwards compatible with
> Solaris, or compatible with the selfish anarchy of Linux, but it is at
> least compatible with something that isn't either already totally
> irrelevant or rapidly becoming so.

Do you believe, we should add a new /usr/bsd tree in addition?

> What the free solaris kernel seems to need most is device drivers and if
> the way to achieve that is in fact to sacrifice all the old de jure
> standardisation for its own sake, then I'd throw my support that way
> anytime: I already gave up on OI derivatives because FreeBSD actually
> works on my hardware and I can have ZFS that way.

FreeBSD is nice, but I like to prevent OpenSolaris from dying.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[Irek Szczesniak]

Well, I'm strongly against importing more tools from FreeBSD. the
damage caused by Garrett D'Amore eager "adopt FreeBSD tools to fill
our needs" project has created gaping holes in
functionality/interoperability and as side effect imported all the
damn FreeBSD bugs, too. IMO the changes to tools like tr, sed and many
others must be reversed because multibyte character handling is
grossly broken now and all the other issues caused by that.

Irek

[Sriram Narayanan]

I'm back in action, and have started my work on Belenix.

Here are my own thoughts on other discussions that have come up on this thread:
- People should be able to choose what they want. If someone wants
Solaris tools + API as of b147, then that should be possible. If
someone wants FreeBSD userland and other tools, then that should be
possible too. If someone wants to be able to selectively mix and
match, then that should be possible as well. I also feel that with
some concrete steps, we can resolve all manner of issues - which
tr/sed/od/ksh should be bundled, what is POSIX compliance at a tool
level, etc.

The areas that I've not thought about too much in detail are - what
does it take to add modern device driver support to illumos, should we
take Linux drivers and make them available as separately downloadable
modules built from separate source repos, which commits from b147
onwards are acceptable/need-to-be-reverted, etc. At the moment, I'm
not skilled enough to take such calls.

What I am skilled at is the area of release engineering, and I intend
to set in place something that could potentially benefit all distros
as well as the open source software eco-system as a whole.

More updates from me as I make concrete progress.

-- Sriram

> belenix-dev mailing list
> belen...@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/belenix-dev



--
------------------------------------
Belenix: www.belenix.org
Twitter: @sriramnrn

[Joerg Schilling]

Irek Szczesniak <iszcz...@gmail.com> wrote:

> Well, I'm strongly against importing more tools from FreeBSD. the
> damage caused by Garrett D'Amore eager "adopt FreeBSD tools to fill
> our needs" project has created gaping holes in
> functionality/interoperability and as side effect imported all the
> damn FreeBSD bugs, too. IMO the changes to tools like tr, sed and many
> others must be reversed because multibyte character handling is
> grossly broken now and all the other issues caused by that.

I am not convinced that this is a result of importing things from FreeBSD.
The tools in question did e.g. pass a POSIX compliance test on Mac OS X...

There is however some kind of common thread with regard to software quality if
we look at work results from Garrett D'Amore:

- After Garret replaced the closed source i18n code in libc by a self
hacked version from FreeBSD, "col -x" no longer passes japanese
characters. This (illumos specific miss-behavior of "col") is why they
are currently discussing whether they should replace the rock solid
nroff based man subsystem by a new and not sufficiently tested
software from OpenBSD.

- The "od" OSS replacement written by Garret is full of bugs.

I would guess that if there are problems with tr and sed on Illumos, this are
problems caused by the i18n implementation they are based ob because they are
linked against the libc from Illumos rather than problems caused by the tools
themselves.

So it seems that it would be sufficient to introduce a working code review for
Illumos. Currently, a failed code review (as happened for Garret's "od") does
not prvent that piece of code from being integrated into the Illumos code base
and it seems that for the i18n replacement code no code review happened at all.

It seems to be also important to forbid code changes as a result of a so called
bug in some other software before the bug was fixed. I am convinced that it
would be sufficient to fix the behavior of the i18n implementation on Illumos
to get rid of problem in other software that is based on the correct behavior
of the i18n functions in libc.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[Moinak Ghosh]

On Sat, Dec 15, 2012 at 10:10 PM, Sriram Narayanan <srir...@gmail.com> wrote:

I'm back in action, and have started my work on Belenix.

[...]

The areas that I've not thought about too much in detail are - what
does it take to add modern device driver support to illumos, should we
take Linux drivers and make them available as separately downloadable
modules built from separate source repos, which commits from b147
onwards are acceptable/need-to-be-reverted, etc. At the moment, I'm
not skilled enough to take such calls.

Linux Driver porting is sufficiently challenging since there is zero
similarity between the internal kernel ABIs of the two OSes. Even
behavioral semantics are different. In addition stacks like USB3 and
bluetooth are missing.
These are non-trivial and complicated by the fact that not all devices
behave strictly as per the standard. Scores of device variants need
to be tested, idiosyncracies addressed etc. Biggest of all debugging
in the kernel space is an art by itself. Sometimes one will have to
forget about life to get something working.

Regards,
Moinak.
==========================
http://moinakg.wordpress.com
http://moinakg.github.com/pcompress/

[Dennis Clarke]

You should add "..and expect no reward and to have your ideas stolen from you." I think that the NetApp people have proven that to you.

By the way .. good to see you are out there alive and well. At least I hope so.

Dennis Clarke
dcl...@blastwave.org

[Moinak Ghosh]

On Mon, Dec 17, 2012 at 11:54 PM, Dennis Clarke <dcl...@blastwave.org> wrote:

> On Sat, Dec 15, 2012 at 10:10 PM, Sriram Narayanan <srir...@gmail.com>wrote:
>
> > I'm back in action, and have started my work on Belenix.
> >
> > [...]
> > The areas that I've not thought about too much in detail are - what
> > does it take to add modern device driver support to illumos, should
> we
> > take Linux drivers and make them available as separately downloadable
> > modules built from separate source repos, which commits from b147
> > onwards are acceptable/need-to-be-reverted, etc. At the moment, I'm
> > not skilled enough to take such calls.
> >
>
> Linux Driver porting is sufficiently challenging since there is zero
> similarity between the internal kernel ABIs of the two OSes. Even
> behavioral semantics are different. In addition stacks like USB3 and
> bluetooth are missing.
> These are non-trivial and complicated by the fact that not all devices
> behave strictly as per the standard. Scores of device variants need
> to be tested, idiosyncracies addressed etc. Biggest of all debugging
> in the kernel space is an art by itself. Sometimes one will have to
> forget about life to get something working.

You should add "..and expect no reward and to have your ideas stolen from you."  I think that the NetApp people have proven that to you.

By the way .. good to see you are out there alive and well. At least I hope so.

Heh, I'm fine. Been interested with HPC and storage stuff lately.

Regards,
Moinak.
--
================================
http://moinakg.wordpress.com/
http://moinakg.github.com/pcompress/

[Dennis Clarke]

> >
> Heh, I'm fine. Been interested with HPC and storage stuff lately.

I'm heads down doing programming, on Solaris. No surprise there. It's a good gig where I get to keep the IP in the software, at least everything that isn't specific to the target customers business. That is a rare agreement and it works for all involved.

So what was the last great release of BeleniX ? Is there an iso handy? Would love to have it running again as I always loved BeleniX.

Oh, by the way, I was totally caught off guard when genunix.org/com vanished one day. I had no idea where it went but one day it just vanished. I still have the domain name but the site simply went poof one day.

dc

[David Halko]

On Tue, Dec 11, 2012 at 9:03 AM, Irek Szczesniak <iszcz...@gmail.com> wrote:

On Fri, Nov 30, 2012 at 1:13 PM, Joerg Schilling
<Joerg.S...@fokus.fraunhofer.de> wrote:
> Sriram Narayanan <srir...@gmail.com> wrote:
>
>> I should clarify, I decided to address release engineering first, since the
>> whole non-oracle Solaris community will need to get together to solve the
>> problem of tracking package changes, have the tools to solve posix
>> compliance debates,  and then start to write device drivers.
>
> Well, this is something I mention since years. I still hope there is a way to
> collaborate - note that even the *BSDs that have been disintegrated since a
> long time do important things together.
>
> What we of course first need is to decide on a minimal set of consensus.

Which minimum set of consensus? The point is to go boldly forward with the
userland modernization where Illumos has failed with it's conservative policy.

 

If there was failure, it was with OpenSolaris programmers being too aggressive, and the rest of us inheriting the mess.

 

Core OS functionality (booting, services, packaging, etc.) should not be dependent upon newer GNU userland, but based upon thoroughly debugged Solaris userland. These binaries should be separate and distinct locations (perhaps: /bin, /sbin, etc.) for simple maintenance and to enable easy embedding into smaller form-factors.

 

If people wish to have a newer userland, where bugs are constantly being thrashed out, from other developers in other spheres (not interested in Solaris longevity) - then those should be put elsewhere and people should have the option to pick & choose (perhaps: /usr/bin, /usr/sbin, /usr/gnu/bin, /usr/fbsd, etc.) and those communities inherit the bugs those userlands suffer from. The core should be wary of suffering from external bugs that others may not approve fixes for or suffering from feature suppression where others may block innovation regarding.

 

There is soooo much that is plain-old broken in gawk, even though it has some GREAT additional features, virtually nothing coded in any of the standards related awk will run in gawk. A standards-compliant userland is important for software compatibility. A modern userland is important for end-user comforth. Both are aggressively needed.

 

Sun and Oracle, in OpenSolaris and Solaris 11, merged the standards-base and gnu-base userlands together without strict differentiation, contaminating the core OS services, which was perhaps the poorest engineering decision I have ever seen. I would hope others do not make the same mistake.

 

The purpose of an OS is to run software - if existing commercial software will not run under the OpenSolaris splinter, there may be little reason for the splinter to exist, with the exception of some special purpose appliance. The usefulness of a special-purpose appliance running with an OpenSolaris kernel without USB3, WiFi, or clustered-ZFS support is puzzling to me... unless it is embedded - and then modernized userland becomes less important.

 

Clearly, Illumos is driving OpenSolaris source code towards storage appliance and cloud-based hypervisor. Few in those groups understand the necessity of clustered ZFS for storage to provide good any-to-any H-A or D-R... this tells me those projects may not be long-for-this-world.

 

I guess the rest of us need to decide what WE WANT out of an OpenSolaris splinter.

 

I want a basic Desktop OS running SunRay Thin Clients, run some commercial apps, against a shared-nothing clustered back-end hypervisor. That's it. USB3 is a bonus, for faster external storage. WiFi would be a bonus for wireless storage. If no sunrays, then I need a cheap out-of-the-box thin-client alternative. I want support for SVR4 standards and a community effort to drive innovation (superset) in this arena.

 

Thanks - Dave

http://svr4.blogspot.com/

[Peter Tribble]

On Sat, Dec 15, 2012 at 4:40 PM, Sriram Narayanan <srir...@gmail.com> wrote:
> I'm back in action, and have started my work on Belenix.

One thing that I think would be hugely beneficial would be to
clarify exactly what Belenix is (or at least, how you see it).
The mission statement or elevator pitch, if you will.

--
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

[james]

> A standards-compliant userland is important for software compatibility.

The key issue here is IMO:

What do you mean 'standards-compliant'?

Do you mean:
- the de-facto standard people currently use in reality?
- the de-facto standard that is 'old Solaris', bugs and warts and all?
- strict de-jure standard from POSIX (and if so, which revision?)?

We all want compatibility. The issue is what we are compatible with,
and what it gets us.

How many awk programs are there, really, that don't use gawk as the
de-facto standard? (And where those programs are ones that people who
are considering a new system deployment will care about, rather than
both people still using an obscure Irix node on a heterogeneous compute
network).

I don't mean to pick on (g)awk particularly, just that you mentioned it.

There does not seem to be a reason to get hung up on a de-jure standard
that isn't actually used in practice (for example - because the vast
majority of almost-compliant systems in the field actually conform to a
different de-facto standard and have rather questionable commitment to
supporting the actual de-jure standard without jumping through hoops).

[Joerg Schilling]

Moinak Ghosh <moi...@belenix.org> wrote:

> Linux Driver porting is sufficiently challenging since there is zero
> similarity between the internal kernel ABIs of the two OSes. Even
> behavioral semantics are different. In addition stacks like USB3 and
> bluetooth are missing.

Correct and in addition, the VFS interface from Linux is completely different.

Also the philosophy behind setting up DMA on Linux is completely different and
causes problems inside Linux already.

> These are non-trivial and complicated by the fact that not all devices
> behave strictly as per the standard. Scores of device variants need
> to be tested, idiosyncracies addressed etc. Biggest of all debugging
> in the kernel space is an art by itself. Sometimes one will have to
> forget about life to get something working.

The only thing we could take from Linux drivers is _some_ ideas on dealing with
the hardware.

Both SunOS and recent *BSDs are based on the driver interface from BSD-4.2
(except that SunOS introduced a streams based network stack in the late 1980s).

_porting_ drivers is possible with drivers from the *BSD group but taking a
driver from Linux requires more than what I would call porting.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[Joerg Schilling]

Dennis Clarke <dcl...@blastwave.org> wrote:

> > These are non-trivial and complicated by the fact that not all devices
> > behave strictly as per the standard. Scores of device variants need
> > to be tested, idiosyncracies addressed etc. Biggest of all debugging
> > in the kernel space is an art by itself. Sometimes one will have to
> > forget about life to get something working.
>
> You should add "..and expect no reward and to have your ideas stolen from you." I think that the NetApp people have proven that to you.

Coud you explain what you have in mind here and why you mention NetApp?

NetApp has stolen the ideas for COW filesystems that I developed between 1988
and 1991and later applied for patents on my ideas.

http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

[Joerg Schilling]

David Halko <david...@gmail.com> wrote:

> If there was failure, it was with OpenSolaris programmers being too
> aggressive, and the rest of us inheriting the mess.

I would call me an "OpenSolaris programmer", are you talkig about the people
that have been employed by Sun?

In 2003, I developed a concept for a Solaris Live CD and discussed that with
Sun. In 2004 and 2005, I developed a concept for an alternative Solaris Distro
that was intended to be 100% free. I also tried to start a common project with
Sun to implement the latter idea.

Sun instead hired Ian Murdock who dod take most of my ideas but he also added
some things that caused the project to fail:

- He claimed that there was no Solaris community and tried to create one.
This way, he caused a significant damage on the existing Solaris
community.

- He believed that people are interested in a Solaris that looks like
Linux. Well, he did not ask the Solaris community.
This way, he caused further disinterest from the Solaris community.

I tried to warn, but too few people did help me arguing.

> Core OS functionality (booting, services, packaging, etc.) should not be
> dependent upon newer GNU userland, but based upon thoroughly debugged
> Solaris userland. These binaries should be separate and distinct
> locations (perhaps: /bin, /sbin, etc.) for simple maintenance and to enable
> easy embedding into smaller form-factors.

Agreed for the userland!

What do you mean by "distinct locations" in this relation?

> If people wish to have a newer userland, where bugs are constantly being
> thrashed out, from other developers in other spheres (not interested in
> Solaris longevity) - then those should be put elsewhere and people should
> have the option to pick & choose (perhaps: /usr/bin, /usr/sbin,
> /usr/gnu/bin, /usr/fbsd, etc.) and those communities inherit the bugs those
> userlands suffer from. The core should be wary of suffering from external
> bugs that others may not approve fixes for or suffering from feature
> suppression where others may block innovation regarding.

We, the Solaris community need to find people who are interested and able to
work on the UNIX userland. I am doing this since a while, but Sun disregarded
the userland (another mistake from Sun...).

If you look at the software I maintain and distrinbute, you will see that there
is a lot of enhanced UNIX software inside already. What we need to take care of
is to keep the UNIX philosophy in mind and not to be a blind Linux follower.

> Sun and Oracle, in OpenSolaris and Solaris 11, merged the standards-base
> and gnu-base userlands together without strict differentiation,
> contaminating the core OS services, which was perhaps the poorest
> engineering decision I have ever seen. I would hope others do not make the
> same mistake.

Schillix does not make that mistake ;-)

> The purpose of an OS is to run software - if existing commercial software
> will not run under the OpenSolaris splinter, there may be little reason for
> the splinter to exist, with the exception of some special purpose
> appliance. The usefulness of a special-purpose appliance running with an
> OpenSolaris kernel without USB3, WiFi, or clustered-ZFS support is puzzling
> to me... unless it is embedded - and then modernized userland becomes less
> important.

We need to work on USB3 of course and we need to work on Wifi (e.g. to
implement support for "eduroam").

> Clearly, Illumos is driving OpenSolaris source code towards storage
> appliance and cloud-based hypervisor. Few in those groups understand the
> necessity of clustered ZFS for storage to provide good any-to-any H-A or
> D-R... this tells me those projects may not be long-for-this-world.

Illumos completely ignores general software quality. I cannot speak for their
ZFS enhancements (as I did not look at them) but for the rest most of the
changes are not acceptable by me.

> I guess the rest of us need to decide what WE WANT out of an OpenSolaris
> splinter.

We need a general purpose OpenSolaris continuation that is not dominated by
companies.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[Joerg Schilling]

james <ja...@mansionfamily.plus.com> wrote:

> > A standards-compliant userland is important for software compatibility.
>
> The key issue here is IMO:
>
> What do you mean 'standards-compliant'?
>
> Do you mean:
> - the de-facto standard people currently use in reality?
> - the de-facto standard that is 'old Solaris', bugs and warts and all?
> - strict de-jure standard from POSIX (and if so, which revision?)?
>
> We all want compatibility. The issue is what we are compatible with,
> and what it gets us.

Let me mention what I am interested in:

Binary compatibility to existing (pre Oracle) Solaris installations.

Compatibility to SVID3 (Svr4 compliance) where possible.

Compatibility to all POSIX versions if possible.

The latter is something, we need to work on and to make this possible, I am a
usually attending all OpenGroup teleconferences since a year and supporting the
OpenSolaris interests.

Jörg

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni)
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

[David Halko]

On Tue, Dec 18, 2012 at 5:38 AM, Joerg Schilling <Joerg.S...@fokus.fraunhofer.de> wrote:

David Halko <david...@gmail.com> wrote:

> If there was failure, it was with OpenSolaris programmers being too
> aggressive, and the rest of us inheriting the mess.

I would call me an "OpenSolaris programmer", are you talkig about the people
that have been employed by Sun?

 

I suspect you were not "too aggressive". :-)

 

> Core OS functionality (booting, services, packaging, etc.) should not be
> dependent upon newer GNU userland, but based upon thoroughly debugged
> Solaris userland. These binaries should be separate and distinct
> locations (perhaps: /bin, /sbin, etc.) for simple maintenance and to enable
> easy embedding into smaller form-factors.

Agreed for the userland!

What do you mean by "distinct locations" in this relation?

If something is for the core-os functionality (i.e. rebooting, password management, etc.) - then these binaries should be in /bin (regular user commands) or /sbin (possibly for system adminstrative commands?)

 

If users want a different userland, it could be made available in /usr. Upstream packages should not be mixed into the core.

My intention is not to fault individuals or projects. It is just to observe the major contributors and where their "bread and butter" is.

With Red Hat bundling clustering with their hypervisor and IBM bundling clustering on ZFS with Linux in the HPC market - my concern is with the competitiveness of Illumos in their perspective arenas. I think the window for them is closing, quickly, which is a great concern to me - it will not take long for others to take off-the-shelf Linux and over-deliver what they can do, with no programming expertise, at all.

 

> I guess the rest of us need to decide what WE WANT out of an OpenSolaris
> splinter.

We need a general purpose OpenSolaris continuation that is not dominated by
companies.

 

I think we need to find a niche and operate well, in it. For me, that is SVR4.

Draw in those developers to grow community with a standards-base and provide enhanced [Solaris] functionality.

 

Dave

http://svr4.blogspot.com/

[james]

>
> Let me mention what I am interested in:
>
> Binary compatibility to existing (pre Oracle) Solaris installations.

Why?

I've enjoyed Solaris binary compatibility with database and market data
systems and old apps as much as anyone. But that was in environments
that had paid for those things and were paying for support etc. They
will not be using anything thisgroup will create for that anytime soon.

>
> Compatibility to SVID3 (Svr4 compliance) where possible.

Why?

> Compatibility to all POSIX versions if possible.

Why?

(Yes I know this is heretical on one level; but what do you get *in
practice* that actually matters?)

The problem is: by the time you attain standards compatibility, what
will be the relevance of that compatibility for ordinary people, who
just want to do their day job? And how many people will you alienate
and disenfranchise by slavish attempts to stick to de jure standards?

A de jure standard is only useful if its what people need. Has it
occured to you that a shift to a (sometimes inferior) de facto standard
might indicate that the de jure standard just isn't that useful any more?

When there was half a dozen big-iron UNIX vendors and we all had
heterogeneous networks, sure - it all mattered. But I'm really not so
sure it does anymore.

There is a level on which I think Ian was trying to save you from
yourself. Sun certainly went on and executed badly though, no question
about that.

[David Halko]

My thoughts on this, even though it was addressed primarily to Joerg.

 

On Tue, Dec 18, 2012 at 12:09 PM, james <ja...@mansionfamily.plus.com> wrote:

Let me mention what I am interested in:

Binary compatibility to existing (pre Oracle) Solaris installations.

Why?

 

I have software. There are vendors looking at moving from Solaris to a less-corporate-competitive alternative.

 

I've enjoyed Solaris binary compatibility with database and market data systems and old apps as much as anyone.  But that was in environments that had paid for those things and were paying for support etc.  They will not be using anything thisgroup will create for that anytime soon.

 

That is purely our fault. We have executed badly.

 

Compatibility to SVID3 (Svr4 compliance) where possible.

Why?

Encourage software adoption by a broader audience (i.e. ISV's)

Compatibility to all POSIX versions if possible.

Why?

Encourage software adoption by a broader audience (i.e. ISV's)

(Yes I know this is heretical on one level; but what do you get *in practice* that actually matters?)

Stability for software developers, so they do not have to revisit old code, and allow more time for them to create higher-level value.

The problem is: by the time you attain standards compatibility, what will be the relevance of that compatibility for ordinary people, who just want to do their day job?  And how many people will you alienate and disenfranchise by slavish attempts to stick to de jure standards?

That is why both is done.

 

Core-OS, standards. Legacy packages for compatibility. Bleeding-edge packages for others. PATH does wonders.

 

A de jure standard is only useful if its what people need.  Has it occured to you that a shift to a (sometimes inferior) de facto standard might indicate that the de jure standard just isn't that useful any more?

It is all about money, in the end. If you get something close for nothing, standards matter less.

 

When there was half a dozen big-iron UNIX vendors and we all had heterogeneous networks, sure - it all mattered.  But I'm really not so sure it does anymore.

Ultimately, they priced themselved out of the market, without enough innovation. Sun did the closest to innovating.

 

There is a level on which I think Ian was trying to save you from yourself.  Sun certainly went on and executed badly though, no question about that.

The idea of a network based packaging interface was goodness.

 

By creating ANOTHER packaging defacto-standard, there was little saving done. Ian should have advocated wrapping SVR4 Packaging and created a super-standard. Increasing the number of languages needed to support the "core" was probably not very wise, either. No ISV will want to move to another [proprietary] packaging standard when they are using off-the-shelf packaging tools to support multiple platforms.

 

Of course, I suspect Joerg's thoughts will be entire different from mine. ;-)

[Dennis Clarke]

> > You should add "..and expect no reward and to have your ideas stolen
> > from you." I think that the NetApp people have proven that to you.
>
> Coud you explain what you have in mind here and why you mention NetApp?
>
> NetApp has stolen the ideas for COW filesystems that I developed
> between 1988 and 1991and later applied for patents on my ideas.

NetApp seems to be the theives of the corporate software world and will use any technology they can get their hands on with the knowledge that no one can afford the legal fees to sue them. As I understand it Moinak had written a lovely compressed DVD/CD filesystem for BeleniX years ago and the nice folks at NetApp simply stole the whole idea and then went a bit further with it.

I think he had made the reasonable mistake of putting it on his resume.

In any case .. that was what I was thinking.

Dennis