[Sbcl-devel] GSoC 2013 Mentoring - Unicode

[Tom Emerson]

Hello all,

I registered on the site, but wanted to send a message to the list as well: I'm interested in mentoring GSoC this summer, particular in areas related to character encoding and Unicode. I have over 16 years of hands-on experience with Unicode and complex character encodings, have been programming in Lisp for over 25 years, and have been doing a lot of NLP work in non-English languages, in Lisp, for the last two. I have experience with both CCL's and SBCL's character set implementations, and am working on a CDR to try and standardize the representation of Unicode character names across implementations (e.g., SBCL and CCL use different naming conventions: I've modified my local SBCL to use CCL's, just to make my code easier to move between implementations.)

Many thanks.

    -tree

--
Tom Emerson
trem...@gmail.com
http://www.dreamersrealm.net/tree

[Christophe Rhodes]

Tom Emerson <trem...@gmail.com> writes:

> I registered on the site, but wanted to send a message to the list as well:
> I'm interested in mentoring GSoC this summer, particular in areas related
> to character encoding and Unicode. I have over 16 years of hands-on
> experience with Unicode and complex character encodings, have been
> programming in Lisp for over 25 years, and have been doing a lot of NLP
> work in non-English languages, in Lisp, for the last two.

Great! Welcome. I have a work-in-progress local branch which
storesmore of the necessary information about code points from
UnicodeData.txt and implements normalization, and might hand that branch
over to a student (or use it as introductory reading material) for the
SoC, if a sufficiently interested one shows up. (If not, well, I'll
keep on working on it in my own slow way -- but would also be open to
direct collaboration).

I think the next step is for you to visit the sbcl page on melange and
request your addition as a mentor -- I don't see you on my list of
actions.

Cheers,

Christophe

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Sbcl-devel mailing list
Sbcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-devel

[Tom Emerson]

On Tue, Apr 16, 2013 at 3:13 PM, Christophe Rhodes <cs...@cantab.net> wrote:

Great!  Welcome.  I have a work-in-progress local branch which

storesmore of the necessary information about code points from
UnicodeData.txt and implements normalization, and might hand that branch
over to a student (or use it as introductory reading material) for the
SoC, if a sufficiently interested one shows up.  (If not, well, I'll
keep on working on it in my own slow way -- but would also be open to
direct collaboration)

I'd certainly be interested in working on it with you: having all the normalization forms supported efficiently has been on my personal task list for a while.

 

 

I think the next step is for you to visit the sbcl page on melange and
request your addition as a mentor -- I don't see you on my list of
actions.

I could have sworn I had, but I'll do it again. Thanks.

    -tree

[Christophe Rhodes]

Tom Emerson <trem...@gmail.com> writes:

> On Tue, Apr 16, 2013 at 3:13 PM, Christophe Rhodes <cs...@cantab.net> wrote:
>
>> Great! Welcome. I have a work-in-progress local branch which
>> storesmore of the necessary information about code points from
>> UnicodeData.txt and implements normalization, and might hand that branch
>> over to a student (or use it as introductory reading material) for the
>> SoC, if a sufficiently interested one shows up. (If not, well, I'll
>> keep on working on it in my own slow way -- but would also be open to
>> direct collaboration)
>
> I'd certainly be interested in working on it with you: having all the
> normalization forms supported efficiently has been on my personal task list
> for a while.

OK; I've got round to pushing a branch (volatile, could be rebased at
any instant, at least if I can work out how) to github. It supports
NFD and NFKD semi-efficiently; there are some low-hanging fruit to
improve them (by precomputing the recursive decomposition at build-time
rather than decomposing recursively at run-time, for example; also by
doing a first pass just checking codepoints).

It does not yet support NFC or NFKC; I'm still contemplating coming up
with a viciously clever indexing scheme for the primary composition
lookup (hashing pairs of build-time allocated integers somehow to lookup
compositions in a table). However, the NFD/NFKD support is tested using
the Unicode normalization test vectors, so improvements to it can be
made and tested with a reasonable amount of confidence.

The tree's at
<https://github.com/csrhodes/sbcl/tree/unicode-improvements>. Let me
know how/whether it works for you.

Best,

[Christophe Rhodes]

Christophe Rhodes <cs...@cantab.net> writes:

> It does not yet support NFC or NFKC; I'm still contemplating coming up
> with a viciously clever indexing scheme for the primary composition
> lookup (hashing pairs of build-time allocated integers somehow to lookup
> compositions in a table). However, the NFD/NFKD support is tested using
> the Unicode normalization test vectors, so improvements to it can be
> made and tested with a reasonable amount of confidence.

I have not come up with a viciously clever indexing scheme for primary
composition; I've contented myself with a fairly simple scheme, and also
not implementing any of the optimizations (obvious or otherwise) in the
normalization area -- not even the various quick-check properties for
the various normalization forms. I dare say I'll get to them
eventually, but in the absence of substantial impending time I have gone
ahead and merged the normalization work into master. Still plenty of
low-hanging fruit...

Cheers,

Christophe

------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d