Re: [sage-devel] Re: 2-year project with full-time software developer for improving coding theory in Sage

[David Joyner]

On Wed, Oct 15, 2014 at 12:47 PM, mmarco <mma...@unizar.es> wrote:
> That is great news!!!!
>

Agreed. great news!

> AFAIK, there is still some code from last year's GSoC which is not merged,
> so maybe it is a good oportunity to retake that (ticket #14973)
>
>
>
> El miércoles, 15 de octubre de 2014 17:22:40 UTC+2, Johan S. R. Nielsen
> escribió:
>>
>> Hi sage-devel
>> This is to announce and get some feedback on the start of a Sage
>> software-development project.
>>
>> This spring, Daniel Augot, Clément Pernet and I got funding from INRIA for
>> hiring a full-time software developer for two years to work on extending the
>> coding theory functionality in Sage. We have hired David Lucas, who just
>> started 1st October, and is already working on getting an overview of coding
>> theory and Sage.
>>
>> The aim of the project is to implement a much larger basis of
>> functionality for coding theory which is flexible, modular and useful for
>> the many facets of coding theory: algebra, combinatorics, engineering,
>> cryptography, etc.
>>
>> The current status is that there are many useful functions from a mostly
>> combinatorical viewpoint. From an algebraic viewpoint, though one can
>> construct the generator matrix of e.g. a Reed--Solomon code, there is no
>> specific class or functionality which investigates such a code from its
>> algebraic perspective (e.g. a fast decoder). We would like to implement such
>> things and many other.
>>
>> We would like to discuss all details with the community, and we also hope
>> that this project will get a ball rolling, reinvigorating the interest in
>> Sage for the coding theory community. Most of these discussions are better
>> taken on sage-coding-theory, since they will be quite specific to the field.
>>
>> It is my impression that such a full-time developer for Sage is a rare
>> thing, and we have been thinking on how best to proceed and communicate
>> goals and progress.
>>
>> Of course, all code will go through the Trac and thus indirectly progress
>> is tracked. However, since this will be longer-running thing, our plan is to
>> make a dedicated (small) web page with a road map as well as an overview of
>> already written functionality (which is furthermore annotated as already
>> included in Sage or not). This is not a blog, but just an overview for
>> new-comers, people interested in the development, or people who previously
>> rejected Sage for coding theory use but who might now reconsider it.
>>
>> One thing we have been thinking about how to handle well is development in
>> parallel with the ticket/review system. Since David is working full-time, he
>> will be producing a lot of code, and there will all the time be a
>> significant backlog in the review process. Our fear is that juggling a
>> cobweb of ticket dependencies might become clumsy if not handled well. Do
>> people have ideas for handling this well?
>>
>> We expect that within 2-3 months, David has made the main refactorings of
>> the existing coding theory functionality, and extended it in various ways to
>> exercise the new structure. This will most likely be more or less internal
>> (not on Trac). After this, he will start posting these changes to Trac,
>> while in parallel continuing to develop more new functionality.
>>
>> Regards,
>> Johan S. R. Nielsen
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Nathann Cohen]

Hello !

> Agreed. great news!

Indeed. I do not know if what you have in mind about coding theory somehow overlaps with combinatorial designs (in particular orthogonal arrays/BIBD :-P) but it would be nice to shake those areas of Sage anyway.

>>> Of course, all code will go through the Trac and thus indirectly progress
>>> is tracked. However, since this will be longer-running thing, our plan is to
>>> make a dedicated (small) web page with a road map as well as an overview of
>>> already written functionality (which is furthermore annotated as already
>>> included in Sage or not). This is not a blog, but just an overview for
>>> new-comers, people interested in the development, or people who previously
>>> rejected Sage for coding theory use but who might now reconsider it.
>>>
>>> One thing we have been thinking about how to handle well is development in
>>> parallel with the ticket/review system. Since David is working full-time, he
>>> will be producing a lot of code, and there will all the time be a
>>> significant backlog in the review process. Our fear is that juggling a
>>> cobweb of ticket dependencies might become clumsy if not handled well. Do
>>> people have ideas for handling this well?

Make many small tickets. The best way to not have many dependencies is to have your code integrated into the successive beta versions of Sage, this way your 'current code' will never be too far from the develop version. 

Whatever happens, please, don't come some day with 3Mb of changes. Respect our development process: write tickets, get them reviewed. Writing code takes much longer than reviewing it, so if the three of you can work by reviewing the code he writes everything will run smoothly.

Also, each commit message should begin with "trac #number:", otherwise it is a mess to know which commits belong to which ticket when you have many dependencies.

>>> We expect that within 2-3 months, David has made the main refactorings of
>>> the existing coding theory functionality, and extended it in various ways to
>>> exercise the new structure. This will most likely be more or less internal
>>> (not on Trac). After this, he will start posting these changes to Trac,
>>> while in parallel continuing to develop more new functionality.

Once more: please, don't try to shortcut the review process. Play it fair. Don't develop everything in your office only to come back later expecting us to merge everything at once without reviewing it. Please create tickets, please discuss the implementations on the tickets, please review everything. Sage's review process makes things slower, but if you are 3+1 to be interested in this project you have the manpower (and the will) to do this properly.

I will be glad to help whenever I can. It sure is a a good news.

Nathann

[Nathann Cohen]

About dependencies (again):

Working with Vincent on Combinatorial Designs (many tickets between beta releases) taught us that it is better to 'artificially' order all tickets linearly. Each ticket should have a successor and a predecessor, and they should all be linearly ordered. It is tempting to let some branches be independent, but if for some reason you find out later that they still interact somehow it will create a *LOT* of mess if you already have tickets based on those two branches.

Nathann

[Johan S. R. Nielsen]

Thanks for the feedback everyone! I'll try to comment on some of the things mentioned.

Indeed. I do not know if what you have in mind about coding theory somehow overlaps with combinatorial designs (in particular orthogonal arrays/BIBD :-P) but it would be nice to shake those areas of Sage anyway.

What we really feel is lacking right now (with respect to our own research areas) is a structured (i.e. object oriented, modular) way to deal with families of error-correcting codes, from various angles. Thus, the main priority is to introduce such a structure, as well as expected functionality for the most familiar families of codes (e.g. those everyone meets in a graduate course on coding theory). We are not combinatorists, so we know this angle poorly. Please join the discussions about structure etc. when we do this on sage-coding-theory to make sure what we come up with makes sense from that angle!
 

Make many small tickets. The best way to not have many dependencies is to have your code integrated into the successive beta versions of Sage, this way your 'current code' will never be too far from the develop version. 

Whatever happens, please, don't come some day with 3Mb of changes. Respect our development process: write tickets, get them reviewed. Writing code takes much longer than reviewing it, so if the three of you can work by reviewing the code he writes everything will run smoothly.

Also, each commit message should begin with "trac #number:", otherwise it is a mess to know which commits belong to which ticket when you have many dependencies.

Thanks, this is good advice. We are very aware of the threat of patch-bomb - this will not happen, I promise. However, in the very beginning (and only here) we will play around the OO-structure until we feel sure that it works as we would like. We'll discuss details on sage-coding-theory. Once we're preliminarily satisfied, we'll brake it into logical pieces and make tickets. After that, the new functionality will all be implemented as tickets directly.
 

Once more: please, don't try to shortcut the review process. Play it fair. Don't develop everything in your office only to come back later expecting us to merge everything at once without reviewing it. Please create tickets, please discuss the implementations on the tickets, please review everything. Sage's review process makes things slower, but if you are 3+1 to be interested in this project you have the manpower (and the will) to do this properly.

We were actually concerned that it would be regarded as too inbred if we review David's code?

 

I will be glad to help whenever I can. It sure is a a good news.

Cool :-D

Working with Vincent on Combinatorial Designs (many tickets between beta releases) taught us that it is better to 'artificially' order all tickets linearly. Each ticket should have a successor and a predecessor, and they should all be linearly ordered. It is tempting to let some branches be independent, but if for some reason you find out later that they still interact somehow it will create a *LOT* of mess if you already have tickets based on those two branches.

Hmm, this is a very interesting idea. We'll think about doing something like this.

This is great news! Is there any plan to implement semaphore codes in the near future (see the book by Jean Berstel, Dominique Perrin, Christophe Reutenauer on

"Codes and Automata")? This would be super useful!

Sorry, that's not in our road-map currently. The project will be centered around error-correcting codes, with a focus on algebraic coding theory. We'll also be touching upon the cryptographic applications (e.g. McEliece cryptosystem).

Also important: start small, fix a few bugs just to get used to the Sage development model.

Yes, this is important. That's on David's todo-list for next week :-)

Try to make this person attend a Sage Days early on.

Good idea! We'll keep an eye out for one which fits. Also, there is the GroupeUtilisateursParis which he should join.

The goal should be to have code with *positive_review*, not just to have code. There is no point in continuing to produce more code if there are 10 needs_work tickets.

 Totally agree! But there might well be 10 needing_review tickets :-)

I think it would also be a good idea to get in contact with the matroids folk who got a lot of code integrated into Sage some time ago. Or even better with Eric Gourgoulhon whos currently writing a lot of code for his project and getting it integrated into Sage piece by piece. On top of that he's not so far away from Polytechnique...

Thanks, we'll look into it, but see my clarifications on the main focus above.

Johan

[Augot Daniel]

On Thursday, October 16, 2014 5:49:08 PM UTC+2, Johan S. R. Nielsen wrote:

Thanks for the feedback everyone! I'll try to comment on some of the things mentioned.

Indeed. I do not know if what you have in mind about coding theory somehow overlaps with combinatorial designs (in particular orthogonal arrays/BIBD :-P) but it would be nice to shake those areas of Sage anyway.

What we really feel is lacking right now (with respect to our own research areas) is a structured (i.e. object oriented, modular) way to deal with families of error-correcting codes, from various angles. Thus, the main priority is to introduce such a structure, as well as expected functionality for the most familiar families of codes (e.g. those everyone meets in a graduate course on coding theory). We are not combinatorists, so we know this angle poorly. Please join the discussions about structure etc. when we do this on sage-coding-theory to make sure what we come up with makes sense from that angle!
 

Make many small tickets. The best way to not have many dependencies is to have your code integrated into the successive beta versions of Sage, this way your 'current code' will never be too far from the develop version. 

Whatever happens, please, don't come some day with 3Mb of changes. Respect our development process: write tickets, get them reviewed. Writing code takes much longer than reviewing it, so if the three of you can work by reviewing the code he writes everything will run smoothly.

Also, each commit message should begin with "trac #number:", otherwise it is a mess to know which commits belong to which ticket when you have many dependencies.

Thanks, this is good advice. We are very aware of the threat of patch-bomb - this will not happen, I promise. However, in the very beginning (and only here) we will play around the OO-structure until we feel sure that it works as we would like. We'll discuss details on sage-coding-theory. Once we're preliminarily satisfied, we'll brake it into logical pieces and make tickets. After that, the new functionality will all be implemented as tickets directly.
 

Once more: please, don't try to shortcut the review process. Play it fair. Don't develop everything in your office only to come back later expecting us to merge everything at once without reviewing it. Please create tickets, please discuss the implementations on the tickets, please review everything. Sage's review process makes things slower, but if you are 3+1 to be interested in this project you have the manpower (and the will) to do this properly.

We were actually concerned that it would be regarded as too inbred if we review David's code?

 

I will be glad to help whenever I can. It sure is a a good news.

Cool :-D

Working with Vincent on Combinatorial Designs (many tickets between beta releases) taught us that it is better to 'artificially' order all tickets linearly. Each ticket should have a successor and a predecessor, and they should all be linearly ordered. It is tempting to let some branches be independent, but if for some reason you find out later that they still interact somehow it will create a *LOT* of mess if you already have tickets based on those two branches.

Hmm, this is a very interesting idea. We'll think about doing something like this.

This is great news! Is there any plan to implement semaphore codes in the near future (see the book by Jean Berstel, Dominique Perrin, Christophe Reutenauer on

"Codes and Automata")? This would be super useful!

Sorry, that's not in our road-map currently. The project will be centered around error-correcting codes, with a focus on algebraic coding theory. We'll also be touching upon the cryptographic applications (e.g. McEliece cryptosystem).

That is part of our road-map, but inside our road map, we have that we want to discuss and interact with people interested in coding theory in Sage, to identify various needs. It's open.

But yes, most of our concerns will be centered around the algorithmic aspects of coding theory, be it for coding theory per se, cryptography (there is more than McEliece : proof of retrievability, proof of ownership, etc), theoretical computer science, math investigation.

Concerning matroids, designs, sphere packings, lattices and other domains connected to coding theory, since we have  man power, and time, we will try to be  as much as possible consistent with the existing APIs or interfaces, to avoid users  converting objects between different classes.