[racket] Anyone familiar with mercurial? Add syntax coloring to github?

[John Clements]

It looks like adding syntax coloring for Racket to GitHub is as easy as changing

filenames = ['*.scm']

to

filenames = ['*.scm', '*.rkt']

in line 39 of

pygments-main/pygments/lexers.functional.py

in the "pygments" project on BitBucket.

Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.

Does someone here have the mercurial experience to make this edit?

John

[Tony Garnock-Jones]

On 2011-06-01 6:56 PM, John Clements wrote:
> Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.

Done. I've filed a pull request. See the change here:
https://bitbucket.org/tonyg/pygments-main/overview

Regards,
Tony
_________________________________________________
For list-related administrative tasks:
http://lists.racket-lang.org/listinfo/users

[John Clements]

On Jun 1, 2011, at 4:57 PM, Tony Garnock-Jones wrote:

> On 2011-06-01 6:56 PM, John Clements wrote:
>> Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.
>
> Done. I've filed a pull request. See the change here:
> https://bitbucket.org/tonyg/pygments-main/overview

Nice! With luck, we didn't just break pygments :).

I think we'll have to wait for someone to re-generate the lexer mapping to see this take effect.

Thanks again,

John

[Eli Barzilay]

Two hours ago, Tony Garnock-Jones wrote:
> On 2011-06-01 6:56 PM, John Clements wrote:
> > Unfortunately, it uses mercurial, and I don't really want to learn
> > yet another source control system unless I have to.
>
> Done. I've filed a pull request. See the change here:
> https://bitbucket.org/tonyg/pygments-main/overview

How about

filenames = ['*.scm', '*.ss', '*.rkt*']

?

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

[Tony Garnock-Jones]

On 2011-06-01 10:07 PM, Eli Barzilay wrote:
> How about
> filenames = ['*.scm', '*.ss', '*.rkt*']
> ?

Whoops! I should have thought of that. I've just added another patch to
my repo; hope they see it if/when they pick up the pull request.

Tony

[Greg Hendershott]

Also, how about changing line 38 from

aliases = ['scheme', 'scm']

to

aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']

?

(Or maybe this would be getting to the point where there should be a
distinct new RacketLexer class, complete with a `builtins' keyword
list from e.g. Quack, and so on.)

[Vincent St-Amour]

At Thu, 2 Jun 2011 06:30:00 -0400,

Greg Hendershott wrote:
> aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']

rktl would probably fit in there too.

Vincent

[Eli Barzilay]

An hour ago, Vincent St-Amour wrote:
> At Thu, 2 Jun 2011 06:30:00 -0400,
> Greg Hendershott wrote:
> > aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']
>
> rktl would probably fit in there too.

(This is just in case someone takes it on more seriously: IIRC,
another issue with pygments was either ignoring square brackets, or
highlighting them as errors.)

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

[John Clements]

On Jun 2, 2011, at 6:43 AM, Eli Barzilay wrote:

> An hour ago, Vincent St-Amour wrote:
>> At Thu, 2 Jun 2011 06:30:00 -0400,
>> Greg Hendershott wrote:
>>> aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']
>>
>> rktl would probably fit in there too.
>
> (This is just in case someone takes it on more seriously: IIRC,
> another issue with pygments was either ignoring square brackets, or
> highlighting them as errors.)

Gosh, sounds complicated. I don't think any of us could handle that!

John (goad, goad) Clements

[Greg Hendershott]

So I followed up on this.

https://bitbucket.org/greghendershott/pygments-main/changeset/240e51e2da13b079482f6b61c215280224f89f06

~~~~~
Add RacketLexer.

Previously Racket files were handled by SchemeLexer. Instead, use a
proper RacketLexer, which handles Racket more appropriately:

1. Treat square brackets like parentheses.
2. Expanded list of keywords.
3. Different file extensions, MIME types, etc.
4. Handle #:keyword arguments.
5. Handle more number literals (e.g. #xFF, #o777, 2e2, #e232, etc.).
6. Handle #| ... |# multiline comments (although NOT nested).
~~~~~

Before I give them a pull request, I wanted to give folks here a
chance to critique it (or even flatly veto it).

Again, the goal is that eventually this would flow through to GitHub,
and improve the readability of Racket repos and gists.

> _________________________________________________
> For list-related administrative tasks:
> http://lists.racket-lang.org/listinfo/users

____________________
Racket Users list:
http://lists.racket-lang.org/users

[Sam Tobin-Hochstadt]

Wonderful! This will make the github experience much nicer.

--
sam th
sa...@ccs.neu.edu

[Greg Hendershott]

I spent more time on this, including trying to recognzie all the
variations of numbers like these:

(values
;; #b
#b1.1
#b-1.1
#b1e1
#b1/1
#b0/1
#b1e-1
#b101
#b2 ;highlight as error

;; #d
#d-1.23
#d1.123
#d1e3
#d1e-22
#d1/2
#d-1/2
#d1
#d-1
#dZ ;highlight as error

;; No # reader prefix -- same as #d
-1.23
1.123
1e3
1e-22
1/2
-1/2
1
-1

;; #e
#e-1.23
#e1.123
#e1e3
#e1e-22
#e1
#e-1
#e1/2
#e-1/2
#eZ ;highlight as error

;; #i always float
#i-1.23
#i1.123
#i1e3
#i1e-22
#i1/2
#i-1/2
#i1
#i-1
#iZ ;highlight as error

;; #o
#o777.777
#o-777.777
#o777e777
#o777e-777
#o3/7
#o-3/7
#o777
#o-777
#08 ;highlight as error

;; #x
#x-f.f
#xf.f
#x-f
#xf
#xG ;highlight as error

)

In this way (Python):

## numbers: Keep in mind Racket reader hash prefixes,
## which can denote the base or the type. These don't map
## neatly onto pygment token types; some judgment calls
## here. Note that none of these regexps attempt to
## exclude identifiers that start with a number, such as a
## variable named "100-Continue".

# #b
(r'#b[-+]?[01]+\.[01]+', Number.Float),
(r'#b[01]+e[-+]?[01]+', Number.Float),
(r'#b[-+]?[01]/[01]+', Number),
(r'#b[-+]?[01]+', Number.Integer),
(r'#b\S*', Error),

# #d OR no hash prefix
(r'(#d)?[-+]?\d+\.\d+', Number.Float),
(r'(#d)?\d+e[-+]?\d+', Number.Float),
(r'(#d)?[-+]?\d+/\d+', Number),
(r'(#d)?[-+]?\d+', Number.Integer),
(r'#d\S*', Error),

# #e
(r'#e[-+]?\d+\.\d+', Number.Float),
(r'#e\d+e[-+]?\d+', Number.Float),
(r'#e[-+]?\d+/\d+', Number),
(r'#e[-+]?\d+', Number),
(r'#e\S*', Error),

# #i is always inexact-real, i.e. float
(r'#i[-+]?\d+\.\d+', Number.Float),
(r'#i\d+e[-+]?\d+', Number.Float),
(r'#i[-+]?\d+/\d+', Number.Float),
(r'#i[-+]?\d+', Number.Float),
(r'#i\S*', Error),

# #o
(r'#o[-+]?[0-7]+\.[0-7]+', Number.Oct),
(r'#o[0-7]+e[-+]?[0-7]+', Number.Oct),
(r'#o[-+]?[0-7]+/[0-7]+', Number.Oct),
(r'#o[-+]?[0-7]+', Number.Oct),
(r'#o\S*', Error),

# #x
(r'#x[-+]?[0-9a-fA-F]+\.[0-9a-fA-F]+', Number.Hex),
# the exponent variation (e.g. #x1e1) is N/A
(r'#x[-+]?[0-9a-fA-F]+/[0-9a-fA-F]+', Number.Hex),
(r'#x[-+]?[0-9a-fA-F]+', Number.Hex),
(r'#x\S*', Error),

Talk about brain burn. :)

Also, Jens pointed out I should also handle curly braces. () = [] = {}

https://bitbucket.org/greghendershott/pygments-main

[Robby Findler]

Its great to see someone working on this!

And a terrible shame that you cannot reuse the syntax colorer we
already have! (but you may find the source code in
collects/syntax-color/scheme-lexer.rkt to be useful?)

Robby

[Greg Hendershott]

> And a terrible shame that you cannot reuse the syntax colorer we
> already have! (but you may find the source code in
> collects/syntax-color/scheme-lexer.rkt to be useful?)

Yes I suppose there's duplication among DrRacket, Quack, pygments, et
al. A central web service could be a neat way to go in some
always-connected future.

Meanwhile, Jens helped me again (thanks!) pointing me to how to
generate the `keywords' and `built-ins' lists using
`namespace-mapped-symbols'. I incorporated that. Noticed and fixed a
few bugs.

I'll marinade in it for a day or two, then consider giving pygments a
pull request.

BTW I'm trying to keep in mind this is a syntax highlighter, not
checker. I want something better than the Scheme highlighter, but
sometimes "the perfect is the enemy of the good". Trying to handle
every number literal variation perfectly, is me probably already
trying too hard.

[Greg Hendershott]

I submitted a pull request,
<https://bitbucket.org/birkenfeld/pygments-main/pull-request/94/add-lexer-for-racket-language>.

On Mon, Aug 13, 2012 at 11:25 AM, Greg Hendershott

[John Clements]

On Aug 14, 2012, at 7:36 PM, Greg Hendershott wrote:

> I submitted a pull request,
> <https://bitbucket.org/birkenfeld/pygments-main/pull-request/94/add-lexer-for-racket-language>.

Many thanks for this work.

I clicked on "Like" for your pull request, in hopes that it would help speed the process… but the website is just spinning on me. Ah well. I'm guessing that once this is in the pygments-main repo, we can request that github update?

John

[Greg Hendershott]

> I clicked on "Like" for your pull request, in hopes that it would help speed the process… but the website is just spinning on me. <

Thanks. It probably won't hurt if Georg Brandl sees more votes.

Also, I linked to this mailing list thread in the pull request comments.

> I'm guessing that once this is in the pygments-main repo, we can request that github update? <

I'm probably the last person to ask about how the process will work.
This is my first open-source contribution, ever. And I wouldn't have
predicted my first contribution toward Racket would be by way of
Python and Mercurial. So what do I know. :)

Seriously, I'll follow up periodically with pygments. When that's
pulled, I'll focus on Github and Bitbucket.

On Fri, Aug 17, 2012 at 11:15 AM, John Clements

[Greg Hendershott]

Update: Georg accepted the pull request into pygments a few minutes ago.

Next: I'll keep an eye on the progress pulling into Github and Bitbucket.

On the Github side, I *think* the relevant repo is
<https://github.com/tmm1/pygments.rb>. It looks like they pull from
pygments every 1-3 months. The most recent pull was about 2 months
ago: <https://github.com/tmm1/pygments.rb/commit/7b35292815a709d151add6ef8d2c2f7b20a5c749>.
As a result, I'm hopeful the next pull will be soon.

On Fri, Aug 17, 2012 at 9:12 PM, Greg Hendershott

[Matthias Felleisen]

Thank you very much! -- Matthias