Looking back on the M4 development timeline and release

[Stephane Epardaud]

I mentioned Monday I would try to summarize what worked and didn't
work so well in M4 so we could improve.

== Why M4 took "so long"

Gavin asked me why it took so long to produce M4, so I had to wonder
and come up with reasons:

I think the first mistake was thinking that anything happens during
summer. That's one of those mythical man months right there.
Experience taught me that if a team is working at 100% all year,
summer will see it drop to 20%, and if we're lucky we can bring it up
to 50% if they find something that they really enjoy, which means that
menial "someone's got to do it" tasks are not happening in summer.
Sure it's reasonable to expect people to work 100% in summer, if
they're not work-burnt, but that's not the case for us, we've been
pretty much working to the max of our abilities for a year now (at
least for Tako, Tom and I. Gavin longer), so summer was not as
productive as other months. That doesn't strike me as something
abnormal, but a fact of life we need to live with.

Next, I made the decision, on agreement with Tako and Tom (at least)
to delay work on rewriting our CLI tools to git-style until M3 was
released. And from experience I know these sorts of tasks are best
done _right at the start_ of the next dev cycle, so I told Tom to get
started with it right away. It turned out to be a long task and Tom
outdid himself with a plugin system, genericity, command-line
completion, HTML tool help generation and stuff. We have a pretty
great system now, that we can be proud of. But it took a long time. We
could have delayed this until 1.1, it was not strictly necessary. We
could argue that getting 1.0 out the door sooner could have been more
important. BUT, first impressions matter a lot, and so it could also
be the case that having this will turn out to be a major selling point
to first-time users at 1.0. We can't rewrite history, so it's too late
we have it, but even though I now see that we could have postponed it,
I am not convinced we should have postponed it. To me it's still
50/50: good or bad choice. Time will tell, I guess. In the future I
will be more cautious as to what is strictly necessary for 1.0.

Another issue was that I was away a lot, to talk about Ceylon at JUGs
and conferences. This didn't help, but at the same time it's also
necessary. We can't just write a great language and platform, we also
have to have users. And users we have to fight for: we have to
convince them. And we can't convince them with blogs. Having the best
tool is useless if people don't know it, and don't understand why it's
better than the alternatives. I go to these JUGs and start with a
largely sceptical crowd that tells me they are not convinced about
Ceylon or Red Hat's ability to deliver and why leave Java, why is it
different to Scala (the big visible language ATM), and after an hour
of presentation their attitude change completely. They are impressed
by what we achieved, especially when they ask how many people are
working on it, they are impressed by the language, the tools, the IDE,
and they are anxious to have 1.0 release to start trying it. Most will
not try it before 1.0, which is a shame, but that's what it is. We
even found good contributors that way. Most reactions are widely
enthusiastic after the presentation, which is something that wouldn't
have happened if they didn't see someone explain things face-to-face.
I'm 100% convinced we can't win the crowd without these presentations
(though it's not enough to win them, but it's a prerequisite), so I
don't regret investing this time. We also can't afford to do it after
1.0 because we want users of 1.0 as soon as we release it.

During the mythical summer month, I worked a lot on ceylon.io, which
turned out to be a lot of work that I didn't quite finish because at
some point it was suggested that getting 1.0 out the door might be
more important than having a complete SDK at 1.0. After all with our
module system we can afford to ship the complete SDK after 1.0. I can
understand and I agree to a certain point, but on the other hand,
having a complete-ish SDK at 1.0 could turn out to be a prerequisite.
Not only that but writing ceylon.io illustrated changed required in
the language, so it was beneficial, if expensive. I am still not sure
what our strategy should be regarding that SDK.

There are other things too, such as the fact that we ended up adding
more features than initially planned in M4, such as assertions and
multiple condition list, splitting the JDK into modules, type aliases
and aether. Some of those arose from my adventure writing the
ceylon.io module as a necessity to make the language better, others
from IDE needs (modular JDK) and in the case of aether, even though I
was not consulted and initially against it, I have to agree it's
important, people care a lot about Maven repository interop: it could
be one of those prerequisites to adoption. Certainly those "new"
features did take time to implement.

Not to mention we fixed a lot bugs, but that's good and required.

== What we need to improve

=== Testing

We do a lot of testing, especially in the JVM backend. Running the
tests involve testing not only the compiler, but the command-line
tools, ant tasks, API documentation generator, ceylon.language tests
and making sure that the SDK compiles, testing the CMR and stuff.

What we don't test is:

- CMR/Herd integration with the tools (including JS compiler): we do
mock HTTP servers, but that's not the real Herd. We need to figure out
a way to automate this. Perhaps a CI machine is what we need at this
point.
- IDE: that's pretty reckless and IMO suicidal. We have been hit
multiple times with old bugs that were reopened due to new code.
- Herd: pretty reckless and IMO suicidal but fairly easy to test if we
took some time to do it.
- Web IDE: we completely forgot to test this before shipping M4,
that's unacceptable.

I also noticed that at the start of a release cycle we write a lot
more thorough tests than at the end, so late features/bug fixes are
less tested.

=== Performance

We could argue that performance of Ceylon code at runtime can wait for
1.1, but performance of the tools could be an adoption prerequisite
(note that Scala has notoriously slow tools too, but that could also
be an adoption barrier for them too).

==== IDE

The IDE performance is unacceptable. We can't expect people to start
working on the SDK project with it. There are a number of things which
we can improve easily if we wanted:

- project open time: opening a project with just a single file takes
tens of seconds, especially if that project is using the SDK, because
all module imports are typechecked, then the project is built. This is
nuts. When I open a project: the project's files should be typechecked
and that's IT. It should not be compiled by the backend if none of the
source files are newer than the compiled files. Imported modules
should never be typechecked unless they are not compiled. There's zero
justification for that: after all Eclispe doesn't do this for Java
projects. If we want to open an imported module's file _then_ we can
justify to typecheck (only) that source file, but that's about it.
Whatever other need for typechecking imported modules need to be dealt
with another way. Warming up the completion should also not be done
more than once: it should be cached in an index and only invalidated
when the imported modules change. Search and stuff also needs indexes.

- editing module imports and rebuilding: we don't need a rebuild when
we import a new module, we need to fix that, it should be
instantaneous. When we remove one, we should only rebuilt those units
that imported packages from that module and possibly their dependent
units. This is something easy to do that needs to be done.

==== JVM compiler

The JVM compiler performance is unacceptable. On the SDK we take 6s of
the build (50% time) and we need to improve that. I've started looking
in the past in serialising the lazy model to load it faster than with
the model loader, but that's pretty much limited to ceylon.language
(modules that can't depend on other modules) and the improvement was
so minimal (1.5s out of 12s) that it was not merged. I think we need
to look into it more seriously. Perhaps caching and getting rid of
allocations could fix it.

==== Typechecker

The typechecker performance is unacceptable. On the SDK we take 6s of
the build (50% time) and we need to improve that. I think we need to
look into it more seriously. Perhaps caching and getting rid of
allocations could fix it.

=== Cross-project bugs/features

We have projects for typechecker, jvm, js, common, cmr, runtime,
language, ide, sdk, herd. Sometimes features/bugs appear that require
work on more than one, up to all of them in rare cases. Those are
admittedly the most annoying things to do, but we can't shy away from
doing it. We can't afford to point the finger at others or to wait for
them to fix things. Please when something requires you to get your
hands dirty in other people's work, don't hesitate to try it.
Especially the IDE, I know people are afraid of touching it, but
sometimes you just have to. If in M5 I don't see commits from
_everyone_ in the core team on the IDE, we will have failed, so please
find the courage to do it. If it's not a cross-project feature that
needs to be done, surely there must be a bug or feature in the IDE
that you experienced and want to see fixed: well now's the time to do
it.

When we try and fail, we can always try to get help from someone else,
like I did for the IDE classpath from Xavier Coulon after I realised
nobody within the core team could do better than me (and I couldn't
fix it). What we can't do is sit back and complain.

=== Community

During M4 we had many would-be contributors that offered to help, and
almost none of them ended up contributing. We need to try to find out
why. It could be our fault, or not. We did have a new contributor on
Herd from my JUG talk, so that was nice, but I want to make sure we're
not scaring new contributors away. We need to grow our community. It's
very important.

I am of the opinion that there are two sorts of contributors to
Ceylon: people that get paid for working on it, such as Gavin, Tako,
Tom or me, and others that do it in their free time, such as David,
Tomas, Ales and others. I personally think that we can't point a
finger to "free-time" contributors and tell them to fix something
ASAP. I think that free-time contributors should not be given time
constraints or menial tasks unless they are willing to do them
voluntarily, because the moment contributing on Ceylon becomes a shore
to them, we've lost them as contributors. By definition, Gavin, Tako,
Tom and me are paid to deal with time constraints and shores. That's
what we're paid for. So whenever we find a bug on code contributed by
free-time contributors and we need it fixed ASAP, then this is a job
for us, not for pointing fingers. We also should not duck out of
menial tasks by trying to delegate them to free-time contributors,
that's just nasty. Remember we're paid to do the shitty work that
needs to be done.

=== The schedule

As in every dev cycle there are bugs to fix and features to write. I
had the opinion that we should first fix all the bugs and only then
deal with features. Which meant that I spent three weeks straight (and
others too) just fixing bugs. This is boring as hell and demoralising,
but at the same time, I am not sure there's a better way to schedule
that. I still think we shouldn't introduce features while we have open
bugs.

== The things we did well

I think all the rest we did not just well but great :) I don't want to
give the impression we have a shitty team because that's absolutely
not true. I couldn't wish for a better team and wouldn't change it for
the world. That we came up with M4 at all seems nothing short of a
miracle to me. But the project is growing a lot and we need to improve
to keep up with it.

Of course I must have missed things, so if there's anything else that
we need improving (like what an ass I can be during the release
weeks), please speak up! Oh, and feel free to disagree with what I
said too (unless it's about performance and you're Gavin).
--
Stéphane Épardaud

[Tako Schotanus]

Ok, let's see if I can add something to this:

On Wed, Oct 31, 2012 at 11:53 AM, Stephane Epardaud <stephane...@gmail.com> wrote:

I mentioned Monday I would try to summarize what worked and didn't
work so well in M4 so we could improve.

== Why M4 took "so long"

Next, I made the decision, on agreement with Tako and Tom (at least)
to delay work on rewriting our CLI tools to git-style until M3 was
released. And from experience I know these sorts of tasks are best
done _right at the start_ of the next dev cycle, so I told Tom to get
started with it right away. It turned out to be a long task

Don't forget that the whole member type thing Tom did which turned out to be a monster job.

We hadn't really anticipated that either, and because of the type of problem it wasn't something we could help him with either.

 

I'm 100% convinced we can't win the crowd without these presentations
(though it's not enough to win them, but it's a prerequisite), so I
don't regret investing this time. We also can't afford to do it after
1.0 because we want users of 1.0 as soon as we release it.

Agreed

 

Not only that but writing ceylon.io illustrated changed required in
the language, so it was beneficial, if expensive. I am still not sure
what our strategy should be regarding that SDK.

Personally my first reaction would be to say that we can't release a 1.0 without an SDK.

But releasing 1.0 with a half-baked or not very well thought out SDK is much worse.

And knowing that writing a good SDK will be a lot of work means that we might have to delay the SDK until after 1.0 (but if we have them on Herd how will we prevent people from using them and starting to rely on them?)

I at least consider the design of the SDK to be something of utmost importance, not something that we can "just do iteratively until we get it right".

Of course at the same time they serve us well to get a feel for the language and to find problems (but any module serves that purpose, not only the official SDK modules)

== What we need to improve

=== Testing

We do a lot of testing, especially in the JVM backend. Running the
tests involve testing not only the compiler, but the command-line
tools, ant tasks, API documentation generator, ceylon.language tests
and making sure that the SDK compiles, testing the CMR and stuff.

What we don't test is:

We also just don't test the language itself, of course we write tests for the features, but we hardly write any *real* code.

There was also the problem that we couldn't really test the system as it should be used (for example because the modules were not uptodate on Herd or that the IDE for a long time didn't understand about inter-project dependencies), which meant that some bugs were only found when we actually got ready for the release.

I think that's also unacceptable. As much as possible our development environment should be like the final release so that each day we're using what our users will use when we release.

 

I also noticed that at the start of a release cycle we write a lot
more thorough tests than at the end, so late features/bug fixes are
less tested.

Which is why I think new features should be added at the *start* of a new cycle, but I'll go into that when you mention that below.

=== Performance

We could argue that performance of Ceylon code at runtime can wait for
1.1, but performance of the tools could be an adoption prerequisite
(note that Scala has notoriously slow tools too, but that could also
be an adoption barrier for them too).

And don't forget the bad impression Java left and that it still hasn't been able to completely get rid of

 

==== IDE

The IDE performance is unacceptable. We can't expect people to start
working on the SDK project with it. There are a number of things which
we can improve easily if we wanted:

Agreed, it's often enough to drive people crazy.

 

==== JVM compiler

The JVM compiler performance is unacceptable. On the SDK we take 6s of
the build (50% time) and we need to improve that. I've started looking
in the past in serialising the lazy model to load it faster than with
the model loader, but that's pretty much limited to ceylon.language
(modules that can't depend on other modules) and the improvement was
so minimal (1.5s out of 12s) that it was not merged. I think we need
to look into it more seriously. Perhaps caching and getting rid of
allocations could fix it.

==== Typechecker

The typechecker performance is unacceptable. On the SDK we take 6s of
the build (50% time) and we need to improve that. I think we need to
look into it more seriously. Perhaps caching and getting rid of
allocations could fix it.

In both these cases I would first try to find algorithmical hot-spots before trying to patch things with caching, because I think there's many places where we just try to do too much work. Caching will only plaster over those places making them harder too find afterwards. I guess this means lots of profiling.

=== Cross-project bugs/features

sometimes you just have to. If in M5 I don't see commits from
_everyone_ in the core team on the IDE, we will have failed, so please
find the courage to do it. If it's not a cross-project feature that
needs to be done, surely there must be a bug or feature in the IDE
that you experienced and want to see fixed: well now's the time to do
it.

This is something I just can't agree with 100%, of course, sometimes there's just nobody else to do the work and you try to do your best. But if there's tons of things to do and you know you'll probably take 10x the time someone with more experience will need then it just makes more sense to let them handle it. In my case I worked a couple of days intensively with Tomas to implement the repository stuff in the IDE, him doing the SWT stuff and me adding to and fixing the CMR at the same time. I'm still convinced that doing it that way was the most efficient way we could have used our time.

=== Community

I am of the opinion that there are two sorts of contributors to
Ceylon: people that get paid for working on it, such as Gavin, Tako,
Tom or me, and others that do it in their free time, such as David,
Tomas, Ales and others. I personally think that we can't point a
finger to "free-time" contributors and tell them to fix something
ASAP.

Agreed, but it also doesn't meant that we can't point at something they made and say "hey, this has a problem, can you fix it?"

 

I think that free-time contributors should not be given time
constraints or menial tasks unless they are willing to do them
voluntarily, because the moment contributing on Ceylon becomes a shore
to them, we've lost them as contributors.

True, but we should try to make them feel responsible for their own code, I don't think we just want people that hop in to implement some fun feature and are gone again when we ask them if they can fix something. This mean IMO that we should at least *try* to get them to fix their own stuff.

By definition, Gavin, Tako,
Tom and me are paid to deal with time constraints and shores. That's
what we're paid for. So whenever we find a bug on code contributed by
free-time contributors and we need it fixed ASAP, then this is a job
for us, not for pointing fingers.

Agreed, still we can still *try* to ask them, in a friendly way if they have time ASAP to look into it, if they don't have it or don't reply in time we'll take over. We still want people to feel they have a stake in all this, if we try to take over too soon I think people will feel that they don't have much control anyway.

 

We also should not duck out of
menial tasks by trying to delegate them to free-time contributors,
that's just nasty. Remember we're paid to do the shitty work that
needs to be done.

Isn't that what we've been doing all this time? ;)

 

=== The schedule

As in every dev cycle there are bugs to fix and features to write. I
had the opinion that we should first fix all the bugs and only then
deal with features. Which meant that I spent three weeks straight (and
others too) just fixing bugs. This is boring as hell and demoralising,
but at the same time, I am not sure there's a better way to schedule
that. I still think we shouldn't introduce features while we have open
bugs.

Here I don't agree. If after just getting out a new release, which has meant weeks of bug fixing and staying up late to put out fires the first thing we get to do is fix more bug (which were obviously not that important or we would have fixed them *before* the release) we'll just lose all motivation.

Not only that but if we spend all this time fixing *all* the bugs then after making the new features we will want to release the new version as soon as possible without having really tested any of it.

I think the best way is the other way around: we now have a couple of weeks to implement new stuff, after that we go into bugfix mode again (which will also make sure that the new features are better tested) which we all want to terminate as soon as possible so we can finally release.

And we shouldn't be doing new stuff right before a release *at all*.

Oh, the above assumes we don't find any critical bugs for which a .1 release is necessary.

== The things we did well

I think all the rest we did not just well but great :) I don't want to
give the impression we have a shitty team because that's absolutely
not true. I couldn't wish for a better team and wouldn't change it for
the world. That we came up with M4 at all seems nothing short of a
miracle to me. But the project is growing a lot and we need to improve
to keep up with it.

Of course I must have missed things, so if there's anything else that
we need improving (like what an ass I can be during the release
weeks), please speak up! Oh, and feel free to disagree with what I
said too (unless it's about performance and you're Gavin).

I'm not Gavin and it's not about performance :)

I think I'd like a bit more real planning, at least an intended release date for the new version with dates for when we switch from feature to bugfix mode and stuff like that.

In the end we might be flexible about all that, I don't mind, but I prefer that over somebody some day suddenly saying: okay we're going to release next week.

I do agree that I think we did an outstanding job, although I feel sorry for Tom who had to deal with the most horrible parts of this release this time it seems :)

So an honorable mention to him.

-Tako

[Stephane Epardaud]

On 31 October 2012 13:05, Tako Schotanus <ta...@codejive.org> wrote:
>
> Don't forget that the whole member type thing Tom did which turned out to be
> a monster job.
> We hadn't really anticipated that either, and because of the type of problem
> it wasn't something we could help him with either.

Sure but that feature was planned and it didn't take _that_ long that
I can remember, so I consider that normal and nothing we could have
done differently.

> Personally my first reaction would be to say that we can't release a 1.0
> without an SDK.
> But releasing 1.0 with a half-baked or not very well thought out SDK is much
> worse.

I think the modular SDK means we can definitely get away with that, as
long as the ones we do ship are good, and they can fall-back on the
JDK for things we didn't build yet.

> And knowing that writing a good SDK will be a lot of work means that we
> might have to delay the SDK until after 1.0 (but if we have them on Herd how
> will we prevent people from using them and starting to rely on them?)
> I at least consider the design of the SDK to be something of utmost
> importance, not something that we can "just do iteratively until we get it
> right".

On the contrary, I think the iterative way is the best way to get things right.

> Of course at the same time they serve us well to get a feel for the language
> and to find problems (but any module serves that purpose, not only the
> official SDK modules)

Right, but those are the ones we _have_ to write now, unlike "any" module.

> We also just don't test the language itself, of course we write tests for
> the features, but we hardly write any *real* code.

The SDK tests do that, or do you mean something else?

> There was also the problem that we couldn't really test the system as it
> should be used (for example because the modules were not uptodate on Herd or
> that the IDE for a long time didn't understand about inter-project
> dependencies), which meant that some bugs were only found when we actually
> got ready for the release.
> I think that's also unacceptable. As much as possible our development
> environment should be like the final release so that each day we're using
> what our users will use when we release.

I don't think I understand what you mean here. We do have more than
weekly IDE dev builds that I use, and Herd is pretty trivial to run
and use locally. Not only that but we did 3 releases of Herd in prod
during M4.

> And don't forget the bad impression Java left and that it still hasn't been
> able to completely get rid of

True, but we can't really tell if Java would have been better off
released at the first prod-usable version of 1.2. I think 1.0 and 1.1
did a lot to drive adoption, however incomplete they were.

> In both these cases I would first try to find algorithmical hot-spots before
> trying to patch things with caching, because I think there's many places
> where we just try to do too much work. Caching will only plaster over those
> places making them harder too find afterwards. I guess this means lots of
> profiling.

My guess is that most of our algorithmics issues are just caused by
recomputing things that we don't need to be.

> This is something I just can't agree with 100%, of course, sometimes there's
> just nobody else to do the work and you try to do your best. But if there's
> tons of things to do and you know you'll probably take 10x the time someone
> with more experience will need then it just makes more sense to let them
> handle it. In my case I worked a couple of days intensively with Tomas to
> implement the repository stuff in the IDE, him doing the SWT stuff and me
> adding to and fixing the CMR at the same time. I'm still convinced that
> doing it that way was the most efficient way we could have used our time.

Sure, and that may have been the best thing to do, but I don't want to
place Tomas under time constraints on what is after all his free time.
That's detrimental to us I think. Not only that but it also means you
don't grow the skills of working in the IDE. We can't afford to limit
what we do to a fixed list of areas of expertise, we need to try to
know the most of the system, even if that means being a lot less
skilled in the IDE than in the jvm compiler for example. I fixed bugs
all over the place: IDE, JS compiler, Web IDE, CMR, runtime, Herd,
tools. It's painful, but it's useful ultimately, especially when the
experts are not around. We _need_ to grow those skills.

> Agreed, but it also doesn't meant that we can't point at something they made
> and say "hey, this has a problem, can you fix it?"

"can you" sure, but not wait for them to if it can't wait. And it
works the other way around too: Tomas is very good at fixing bugs that
_I_ wrote in ceylondoc :)

> True, but we should try to make them feel responsible for their own code, I
> don't think we just want people that hop in to implement some fun feature
> and are gone again when we ask them if they can fix something. This mean IMO
> that we should at least *try* to get them to fix their own stuff.

Try, if we can wait, but not if that means alienating them. For
example on Herd I get contributions that are not necessarily complete
because they don't know the whole system yet. Most of the time I end
up finishing it and they can then see what was missing and the next
time they do an even better job than me. I prefer this over the whole
cycle of pointing them what's missing and what needs to be redone,
which can be frustrating.

>> We also should not duck out of
>> menial tasks by trying to delegate them to free-time contributors,
>> that's just nasty. Remember we're paid to do the shitty work that
>> needs to be done.
>
> Isn't that what we've been doing all this time? ;)

I've seen some ducking.

> Here I don't agree. If after just getting out a new release, which has meant
> weeks of bug fixing and staying up late to put out fires the first thing we
> get to do is fix more bug (which were obviously not that important or we
> would have fixed them *before* the release) we'll just lose all motivation.
> Not only that but if we spend all this time fixing *all* the bugs then after
> making the new features we will want to release the new version as soon as
> possible without having really tested any of it.
> I think the best way is the other way around: we now have a couple of weeks
> to implement new stuff, after that we go into bugfix mode again (which will
> also make sure that the new features are better tested) which we all want to
> terminate as soon as possible so we can finally release.
> And we shouldn't be doing new stuff right before a release *at all*.

I suppose that's a philosophical debate. What we did is bugs,
features, bugs+integration. I think that adding releases and new bugs
on top of a buggy base is just wrong. It means we end up relying on
flawed architecture to add new features, which means that the new
features introduce more bugs that way than if they were added to a
"healthy" platform.

> I think I'd like a bit more real planning, at least an intended release date
> for the new version with dates for when we switch from feature to bugfix
> mode and stuff like that.
> In the end we might be flexible about all that, I don't mind, but I prefer
> that over somebody some day suddenly saying: okay we're going to release
> next week.

I agree. Let's target January if that seems reasonable. Planning we'll
get to shortly in another email.

> I do agree that I think we did an outstanding job, although I feel sorry for
> Tom who had to deal with the most horrible parts of this release this time
> it seems :)
> So an honorable mention to him.

Honestly I feel sorry for all of us ;)
--
Stéphane Épardaud

[Festal David]

> - IDE: that's pretty reckless and IMO suicidal. We have been hit
> multiple times with old bugs that were reopened due to new code.

Max has setup the framework to add tests. For the moment I've not had
enough dedicated time to start. But implementing tests, and hopefully
also finding a way to run the compiler / model loader tests inside the
IDE, is definitely on the list of my #1 priority tasks (depending on my
availability of course).

> ==== IDE
>
> The IDE performance is unacceptable. We can't expect people to start
> working on the SDK project with it. There are a number of things which
> we can improve easily if we wanted:
>
> - project open time: opening a project with just a single file takes
> tens of seconds, especially if that project is using the SDK, because
> all module imports are typechecked, then the project is built. This is
> nuts. When I open a project: the project's files should be typechecked
> and that's IT. It should not be compiled by the backend if none of the
> source files are newer than the compiled files. Imported modules
> should never be typechecked unless they are not compiled. There's zero
> justification for that: after all Eclispe doesn't do this for Java
> projects. If we want to open an imported module's file _then_ we can
> justify to typecheck (only) that source file, but that's about it.
> Whatever other need for typechecking imported modules need to be dealt
> with another way. Warming up the completion should also not be done
> more than once: it should be cached in an index and only invalidated
> when the imported modules change. Search and stuff also needs indexes.
>
> - editing module imports and rebuilding: we don't need a rebuild when
> we import a new module, we need to fix that, it should be
> instantaneous. When we remove one, we should only rebuilt those units
> that imported packages from that module and possibly their dependent
> units. This is something easy to do that needs to be done.

During last week (when on the road for JUGs) I've tried to precise my
thoughts about how to cleanly solve this performance problem, by
defining a list of necessary steps / tasks in order to finally get an
acceptable build performance, while keeping things quite simple and
compatible with what we already have now.
As soon as I get time to make them a bit more formalized and explained,
I'll share them with you.

[Tako Schotanus]

I suppose that's a philosophical debate. What we did is bugs,
features, bugs+integration. I think that adding releases and new bugs
on top of a buggy base is just wrong. It means we end up relying on
flawed architecture to add new features, which means that the new
features introduce more bugs that way than if they were added to a
"healthy" platform.

I don't think that any of the currently open issues is about fixing flawed architecture, most of them are just bugs.

So if we have architectural changes to be made, then I agree they should come first, but to me that's almost the same as a new feature and should at least be turned into a specific issue (instead of maybe, possibly, hopefully emerging spontaneously out of the work being done fixing a bunch of related issues).

-Tako

[Stephane Epardaud]

On 31 October 2012 15:08, Tako Schotanus <ta...@codejive.org> wrote:
> I don't think that any of the currently open issues is about fixing flawed
> architecture, most of them are just bugs.

Bugs often turn out to need architecture rewrite. Like boxing/erasure
or source code mapping rewrite.
--
Stéphane Épardaud

[Gavin King]

In all this, I don't think we should forget the magnitude of the
software engineering challenge here. Almost nobody would imagine
taking on a project of this nature with a team this small, nor would I
if I didn't have a team that was so fucking _good_ at what they do.

[Tako Schotanus]

Often? Come on, that's rarely true, we've hardly ever needed to actually rewrite anything because of a bug, that normally only happens because of a new feature.

Still, we could look at all the own issues to see if there are any that we need to do right now because of architectural concerns, I just don't think it's necessary to go and fix all outstanding issues.

> --
> You received this message because you are subscribed to the Google Groups "ceylon-dev" group.
> To post to this group, send email to ceylo...@googlegroups.com.
> To unsubscribe from this group, send email to ceylon-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ceylon-dev?hl=en.
>

[Emmanuel Bernard]

1. you still writes way too long emails - my summary of your email

- we did good overall
- writing SDK + doing conferences + summer take their toll on timing
- test sucks for CMR, Wed IDE, IDE and Herd
- performance sucks for tools esp IDE, type checker and compiler
- core team stays on their petty projects, they should go fix other
projects bugs including IDE
- we somewhat scare contributors, we need to fix that
- we should fix bugs and perf? before focusing on M5 features

2. To me Ceylon will start being adopted when we can say that they can
code with it a go live. That means have a SDK. Ceylon 1.0 won't take off
unless Ceylon SDK 1.0 is here.

Emmanuel

> St�phane �pardaud

[Ivo Kasiuk]

Not only that but writing ceylon.io illustrated changed required in
the language, so it was beneficial, if expensive. I am still not sure
what our strategy should be regarding that SDK.

Personally my first reaction would be to say that we can't release a 1.0 without an SDK.

But releasing 1.0 with a half-baked or not very well thought out SDK is much worse.

And knowing that writing a good SDK will be a lot of work means that we might have to delay the SDK until after 1.0 (but if we have them on Herd how will we prevent people from using them and starting to rely on them?)

I at least consider the design of the SDK to be something of utmost importance, not something that we can "just do iteratively until we get it right".

Of course at the same time they serve us well to get a feel for the language and to find problems (but any module serves that purpose, not only the official SDK modules)

Very true. But I don't think there's a realistic way to release 1.0 with an SDK that can honestly be called mature. Not if you if you don't want to spend another year on it (and maybe not even then). But maybe that's not necessary: with Herd and Ceylon's module system we have the necessary preconditions to let the SDK evolve over time. Change and diversity are not necessarily bad, if you have the right infrastructure. 

Yes, it's absolutely amazing! But that's possibly also a reason why it's not easy to get new contributors: considering the speed at which Ceylon develops, joining the project can be quite challenging I guess.

[Tako Schotanus]

On Wed, Oct 31, 2012 at 11:25 PM, Ivo Kasiuk <i.ka...@googlemail.com> wrote:

Not only that but writing ceylon.io illustrated changed required in
the language, so it was beneficial, if expensive. I am still not sure
what our strategy should be regarding that SDK.

Personally my first reaction would be to say that we can't release a 1.0 without an SDK.

But releasing 1.0 with a half-baked or not very well thought out SDK is much worse.

And knowing that writing a good SDK will be a lot of work means that we might have to delay the SDK until after 1.0 (but if we have them on Herd how will we prevent people from using them and starting to rely on them?)

I at least consider the design of the SDK to be something of utmost importance, not something that we can "just do iteratively until we get it right".

Of course at the same time they serve us well to get a feel for the language and to find problems (but any module serves that purpose, not only the official SDK modules)

Very true. But I don't think there's a realistic way to release 1.0 with an SDK that can honestly be called mature. Not if you if you don't want to spend another year on it (and maybe not even then). But maybe that's not necessary: with Herd and Ceylon's module system we have the necessary preconditions to let the SDK evolve over time. Change and diversity are not necessarily bad, if you have the right infrastructure. 

Ok, I'm just hoping we will not end up in "module hell", where some interesting often used modules use version X of an SDK module and other popular ones X+1 and that anyone trying to use a mix of those modules is in for a world of hurt because X+1 is not backward compatible with X. (I have no idea how JBoss Modules can protect us from things like that)

-Tako

 

[Enrique]

Regarding contributors: I can only speak from my own experience, but I think that the "scary" part of joining a project of this size and complexity can be finding an entry point. When I first started working on the JS compiler, it was a small project and I got the hang of it somewhat quickly, even though I didn't fully understand how some of the other parts worked, like the typechecker; I just labelled it "magic" and got to work on what I could understand (to this day of course I don't fully understand the typechecker, but I've made my peace with that).

But the day I decided to contribute to the jvm compiler, man... that was a different story! Lots of classes, biiiiiig classes, with lots of methods calling each other, etc; it took me a while to find my way around it, in great part because there are absolutely no comments anywhere. The code is readable, yes, the names are somewhat descriptive, _when you know what you're looking at_ but at first glance it's just "my god, it's full of source!" and it takes a long time to make sense of it.

Javadoc is your friend. One-line comments are your friends. Especially when you're starting to read this amount of code; the IDE is very helpful in showing javadoc comments for methods, so that you don't have to navigate over to the method in the other class to figure out what it does; a brief description can save a lot of time because at least it can help me decide if I really need to look at the innards of that method or not.

I know that code should be self-describing with proper method/class/variable names, etc, but it really helps a lot and it is actually somewhat comforting to find comments that were made by another human being, someone who presumably wrote that code or at least understood what it does, instead of feeling like I'm looking at the remains of an ancient alien civilization that's way more advanced than ours. To the ones who dwell on that code on a daily basis, comments will seem superfluous, even dumb, but think of the people that will read that code for the first time.

Unit tests usually are also helpful, but given the nature of this project, even just looking at those can be a little intimidating for a newcomer.

[Tom Bentley]

So here are some of my thoughts on this, firstly Stefs points:
 

we've been
pretty much working to the max of our abilities for a year now (at
least for Tako, Tom and I. Gavin longer), so summer was not as
productive as other months.

I have to agree with this. Until last week I'd taken one weeks holiday (plus the odd day) since February. Plus the last few months have been crazily stressful at home too, so it s fair to say I was pretty much exhausted for this release.

Tom
outdid himself with a plugin system, genericity, command-line
completion, HTML tool help generation and stuff. We have a pretty
great system now, that we can be proud of. But it took a long time.

I agree again. In retrospect what I came up with for the tools was too ambitious to try to get done within one release cycle while also trying to get other work done. There was feature creep, and the decision late in the day to use the tool model to generate errors was a probably a mistake because it made for too many last minute bugs.

I worked a lot on ceylon.io, which
turned out to be a lot of work that I didn't quite finish because at
some point it was suggested that getting 1.0 out the door might be
more important than having a complete SDK at 1.0.

In my opinion there's this thing we call 1.0/M5, which is when we've implemented all the language features on the roadmap. And then there are the preconditions for Ceylon to be successful, which requires that 1.0 compiler + IDE, but also needs a bunch of other things, including (among others) enough useful modules for people to write applications in Ceylon.

I think the act of writing some of these modules has been a really useful way of finding some of the pain points of the language; and I can't think of a better way of finding those, either.

During M4 we had many would-be contributors that offered to help, and
almost none of them ended up contributing. We need to try to find out
why. It could be our fault, or not.

I agree we need to find out why this is. We could try emailing people who've started something but didn't end up contributing any patches and asking them. I can think of a bunch of factors, but until we know what's actually "going wrong" (if anything), I think we could end up wasting effort not tackling the real problem.

As in every dev cycle there are bugs to fix and features to write.

Bullets for this one:

• I think implementing features early and then spending time fixing the bugs introduced has a lot of precedent in other projects.
• I think we have plenty of shitty little bugs which are found at a fairly steady rate. We should try to fix at a similar rate as we go, if only to prevent a huge amount of work piling up for the week(s) before a release. Perhaps we should aim for one person doing this at any one time.
  
• There are a fair few bugs which are not trivial to fix. I think we should treat these like features, because they also introduce lots of little bugs as collateral damage.
• I think we should just be a bit more honest about how long it takes to pull a release together. We should feature freeze *at least* two weeks before we plan to have a working release. We should plan on spending days testing release candidates.  

Responding to one of Tako's points:

Don't forget that the whole member type thing Tom did which turned out to be a monster job.

It wasn't really that bad, it just got held up with other stuff (like aliases)
 
Finally, I'd like to raise a point of my own: I found that the sheer number of repositories I have to juggle now is a very real PITA. The `ant update-all` thing helps, but it still feels like we've got a lot of trees floating around. I have literally a dozen terminal windows open. I have to keep updating across multiple projects. I have to remember which projects have remote branches that I'm working on, and whether I have to push to a remote branch for each project. I have numerous RSS feeds to keep an eye on. I find all of this frustrating. I can only imagine how difficult potential contributors must find this.

Cheers,

Tom

[Tako Schotanus]

Well, I keep bringing this up once in a while, but I really think we should go to OBR (One Big Repository).

The second thing is move to something like JIRA. The problem is of course that we'd lose the integration we have with GitHub.

But this leads to an earlier suggestion to move to BitBucket, which wasn't a joke, I think we should at least investigate the possibility, their JIRA integration seems pretty good and it would make us a lot less dependent on GitHub and their particular way of doing things.

It might all seem like a big move, and it is, especially because we'd need to migrate all the issues we have in GitHub (but if we go to OBR we'd need to do so anyway). But the longer we wait the more painful it's going to be if we ever decide to move.

I wouldn't mind looking into a move like this, to see if it's feasible, if there are tools to migrate issues or if we have to write our own, etc. But only if we can agree it's worth doing.

-Tako

-Tako