[racket] Anyone familiar with mercurial? Add syntax coloring to github?

14 views
Skip to first unread message

John Clements

unread,
Jun 1, 2011, 6:56:04 PM6/1/11
to users Users
It looks like adding syntax coloring for Racket to GitHub is as easy as changing

filenames = ['*.scm']

to

filenames = ['*.scm', '*.rkt']

in line 39 of

pygments-main/pygments/lexers.functional.py

in the "pygments" project on BitBucket.

Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.

Does someone here have the mercurial experience to make this edit?

John

Tony Garnock-Jones

unread,
Jun 1, 2011, 7:57:50 PM6/1/11
to John Clements, users Users
On 2011-06-01 6:56 PM, John Clements wrote:
> Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.

Done. I've filed a pull request. See the change here:
https://bitbucket.org/tonyg/pygments-main/overview

Regards,
Tony
_________________________________________________
For list-related administrative tasks:
http://lists.racket-lang.org/listinfo/users

John Clements

unread,
Jun 1, 2011, 8:13:23 PM6/1/11
to Tony Garnock-Jones, users Users

On Jun 1, 2011, at 4:57 PM, Tony Garnock-Jones wrote:

> On 2011-06-01 6:56 PM, John Clements wrote:
>> Unfortunately, it uses mercurial, and I don't really want to learn yet another source control system unless I have to.
>
> Done. I've filed a pull request. See the change here:
> https://bitbucket.org/tonyg/pygments-main/overview

Nice! With luck, we didn't just break pygments :).

I think we'll have to wait for someone to re-generate the lexer mapping to see this take effect.

Thanks again,


John

Eli Barzilay

unread,
Jun 1, 2011, 10:07:48 PM6/1/11
to Tony Garnock-Jones, users Users, John Clements
Two hours ago, Tony Garnock-Jones wrote:
> On 2011-06-01 6:56 PM, John Clements wrote:
> > Unfortunately, it uses mercurial, and I don't really want to learn
> > yet another source control system unless I have to.
>
> Done. I've filed a pull request. See the change here:
> https://bitbucket.org/tonyg/pygments-main/overview

How about

filenames = ['*.scm', '*.ss', '*.rkt*']

?

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

Tony Garnock-Jones

unread,
Jun 1, 2011, 10:12:29 PM6/1/11
to Eli Barzilay, users Users, John Clements
On 2011-06-01 10:07 PM, Eli Barzilay wrote:
> How about
> filenames = ['*.scm', '*.ss', '*.rkt*']
> ?

Whoops! I should have thought of that. I've just added another patch to
my repo; hope they see it if/when they pick up the pull request.

Tony

Greg Hendershott

unread,
Jun 2, 2011, 6:30:00 AM6/2/11
to Tony Garnock-Jones, John Clements, Eli Barzilay, users Users
Also, how about changing line 38 from

aliases = ['scheme', 'scm']

to

aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']

?

(Or maybe this would be getting to the point where there should be a
distinct new RacketLexer class, complete with a `builtins' keyword
list from e.g. Quack, and so on.)

Vincent St-Amour

unread,
Jun 2, 2011, 8:44:17 AM6/2/11
to Greg Hendershott, Eli Barzilay, John Clements, users Users
At Thu, 2 Jun 2011 06:30:00 -0400,

Greg Hendershott wrote:
> aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']

rktl would probably fit in there too.

Vincent

Eli Barzilay

unread,
Jun 2, 2011, 9:43:04 AM6/2/11
to Vincent St-Amour, users Users, John Clements
An hour ago, Vincent St-Amour wrote:
> At Thu, 2 Jun 2011 06:30:00 -0400,
> Greg Hendershott wrote:
> > aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']
>
> rktl would probably fit in there too.

(This is just in case someone takes it on more seriously: IIRC,
another issue with pygments was either ignoring square brackets, or
highlighting them as errors.)

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

John Clements

unread,
Jun 2, 2011, 12:49:17 PM6/2/11
to Eli Barzilay, users Users

On Jun 2, 2011, at 6:43 AM, Eli Barzilay wrote:

> An hour ago, Vincent St-Amour wrote:
>> At Thu, 2 Jun 2011 06:30:00 -0400,
>> Greg Hendershott wrote:
>>> aliases = ['scheme', 'scm', 'ss', 'racket', 'rkt']
>>
>> rktl would probably fit in there too.
>
> (This is just in case someone takes it on more seriously: IIRC,
> another issue with pygments was either ignoring square brackets, or
> highlighting them as errors.)

Gosh, sounds complicated. I don't think any of us could handle that!


John (goad, goad) Clements

Greg Hendershott

unread,
Aug 12, 2012, 6:36:20 PM8/12/12
to John Clements, Eli Barzilay, users Users
So I followed up on this.

https://bitbucket.org/greghendershott/pygments-main/changeset/240e51e2da13b079482f6b61c215280224f89f06

~~~~~
Add RacketLexer.

Previously Racket files were handled by SchemeLexer. Instead, use a
proper RacketLexer, which handles Racket more appropriately:

1. Treat square brackets like parentheses.
2. Expanded list of keywords.
3. Different file extensions, MIME types, etc.
4. Handle #:keyword arguments.
5. Handle more number literals (e.g. #xFF, #o777, 2e2, #e232, etc.).
6. Handle #| ... |# multiline comments (although NOT nested).
~~~~~

Before I give them a pull request, I wanted to give folks here a
chance to critique it (or even flatly veto it).

Again, the goal is that eventually this would flow through to GitHub,
and improve the readability of Racket repos and gists.
> _________________________________________________
> For list-related administrative tasks:
> http://lists.racket-lang.org/listinfo/users
____________________
Racket Users list:
http://lists.racket-lang.org/users

Sam Tobin-Hochstadt

unread,
Aug 12, 2012, 6:50:02 PM8/12/12
to Greg Hendershott, Eli Barzilay, John Clements, users Users
Wonderful! This will make the github experience much nicer.
--
sam th
sa...@ccs.neu.edu

Greg Hendershott

unread,
Aug 13, 2012, 8:46:10 AM8/13/12
to Sam Tobin-Hochstadt, us...@racket-lang.org
I spent more time on this, including trying to recognzie all the
variations of numbers like these:

(values
;; #b
#b1.1
#b-1.1
#b1e1
#b1/1
#b0/1
#b1e-1
#b101
#b2 ;highlight as error

;; #d
#d-1.23
#d1.123
#d1e3
#d1e-22
#d1/2
#d-1/2
#d1
#d-1
#dZ ;highlight as error

;; No # reader prefix -- same as #d
-1.23
1.123
1e3
1e-22
1/2
-1/2
1
-1

;; #e
#e-1.23
#e1.123
#e1e3
#e1e-22
#e1
#e-1
#e1/2
#e-1/2
#eZ ;highlight as error

;; #i always float
#i-1.23
#i1.123
#i1e3
#i1e-22
#i1/2
#i-1/2
#i1
#i-1
#iZ ;highlight as error

;; #o
#o777.777
#o-777.777
#o777e777
#o777e-777
#o3/7
#o-3/7
#o777
#o-777
#08 ;highlight as error

;; #x
#x-f.f
#xf.f
#x-f
#xf
#xG ;highlight as error

)

In this way (Python):

## numbers: Keep in mind Racket reader hash prefixes,
## which can denote the base or the type. These don't map
## neatly onto pygment token types; some judgment calls
## here. Note that none of these regexps attempt to
## exclude identifiers that start with a number, such as a
## variable named "100-Continue".

# #b
(r'#b[-+]?[01]+\.[01]+', Number.Float),
(r'#b[01]+e[-+]?[01]+', Number.Float),
(r'#b[-+]?[01]/[01]+', Number),
(r'#b[-+]?[01]+', Number.Integer),
(r'#b\S*', Error),

# #d OR no hash prefix
(r'(#d)?[-+]?\d+\.\d+', Number.Float),
(r'(#d)?\d+e[-+]?\d+', Number.Float),
(r'(#d)?[-+]?\d+/\d+', Number),
(r'(#d)?[-+]?\d+', Number.Integer),
(r'#d\S*', Error),

# #e
(r'#e[-+]?\d+\.\d+', Number.Float),
(r'#e\d+e[-+]?\d+', Number.Float),
(r'#e[-+]?\d+/\d+', Number),
(r'#e[-+]?\d+', Number),
(r'#e\S*', Error),

# #i is always inexact-real, i.e. float
(r'#i[-+]?\d+\.\d+', Number.Float),
(r'#i\d+e[-+]?\d+', Number.Float),
(r'#i[-+]?\d+/\d+', Number.Float),
(r'#i[-+]?\d+', Number.Float),
(r'#i\S*', Error),

# #o
(r'#o[-+]?[0-7]+\.[0-7]+', Number.Oct),
(r'#o[0-7]+e[-+]?[0-7]+', Number.Oct),
(r'#o[-+]?[0-7]+/[0-7]+', Number.Oct),
(r'#o[-+]?[0-7]+', Number.Oct),
(r'#o\S*', Error),

# #x
(r'#x[-+]?[0-9a-fA-F]+\.[0-9a-fA-F]+', Number.Hex),
# the exponent variation (e.g. #x1e1) is N/A
(r'#x[-+]?[0-9a-fA-F]+/[0-9a-fA-F]+', Number.Hex),
(r'#x[-+]?[0-9a-fA-F]+', Number.Hex),
(r'#x\S*', Error),

Talk about brain burn. :)

Also, Jens pointed out I should also handle curly braces. () = [] = {}

https://bitbucket.org/greghendershott/pygments-main

Robby Findler

unread,
Aug 13, 2012, 8:53:08 AM8/13/12
to Greg Hendershott, us...@racket-lang.org
Its great to see someone working on this!

And a terrible shame that you cannot reuse the syntax colorer we
already have! (but you may find the source code in
collects/syntax-color/scheme-lexer.rkt to be useful?)

Robby

Greg Hendershott

unread,
Aug 13, 2012, 11:25:37 AM8/13/12
to Robby Findler, us...@racket-lang.org
> And a terrible shame that you cannot reuse the syntax colorer we
> already have! (but you may find the source code in
> collects/syntax-color/scheme-lexer.rkt to be useful?)

Yes I suppose there's duplication among DrRacket, Quack, pygments, et
al. A central web service could be a neat way to go in some
always-connected future.

Meanwhile, Jens helped me again (thanks!) pointing me to how to
generate the `keywords' and `built-ins' lists using
`namespace-mapped-symbols'. I incorporated that. Noticed and fixed a
few bugs.

I'll marinade in it for a day or two, then consider giving pygments a
pull request.

BTW I'm trying to keep in mind this is a syntax highlighter, not
checker. I want something better than the Scheme highlighter, but
sometimes "the perfect is the enemy of the good". Trying to handle
every number literal variation perfectly, is me probably already
trying too hard.

Greg Hendershott

unread,
Aug 14, 2012, 7:36:05 PM8/14/12
to us...@racket-lang.org
I submitted a pull request,
<https://bitbucket.org/birkenfeld/pygments-main/pull-request/94/add-lexer-for-racket-language>.

On Mon, Aug 13, 2012 at 11:25 AM, Greg Hendershott

John Clements

unread,
Aug 17, 2012, 11:15:57 AM8/17/12
to Greg Hendershott, us...@racket-lang.org

On Aug 14, 2012, at 7:36 PM, Greg Hendershott wrote:

> I submitted a pull request,
> <https://bitbucket.org/birkenfeld/pygments-main/pull-request/94/add-lexer-for-racket-language>.

Many thanks for this work.

I clicked on "Like" for your pull request, in hopes that it would help speed the process… but the website is just spinning on me. Ah well. I'm guessing that once this is in the pygments-main repo, we can request that github update?

John

Greg Hendershott

unread,
Aug 17, 2012, 9:12:31 PM8/17/12
to John Clements, us...@racket-lang.org
> I clicked on "Like" for your pull request, in hopes that it would help speed the process… but the website is just spinning on me. <

Thanks. It probably won't hurt if Georg Brandl sees more votes.

Also, I linked to this mailing list thread in the pull request comments.

> I'm guessing that once this is in the pygments-main repo, we can request that github update? <

I'm probably the last person to ask about how the process will work.
This is my first open-source contribution, ever. And I wouldn't have
predicted my first contribution toward Racket would be by way of
Python and Mercurial. So what do I know. :)

Seriously, I'll follow up periodically with pygments. When that's
pulled, I'll focus on Github and Bitbucket.

On Fri, Aug 17, 2012 at 11:15 AM, John Clements

Greg Hendershott

unread,
Aug 23, 2012, 12:59:44 PM8/23/12
to us...@racket-lang.org
Update: Georg accepted the pull request into pygments a few minutes ago.

Next: I'll keep an eye on the progress pulling into Github and Bitbucket.

On the Github side, I *think* the relevant repo is
<https://github.com/tmm1/pygments.rb>. It looks like they pull from
pygments every 1-3 months. The most recent pull was about 2 months
ago: <https://github.com/tmm1/pygments.rb/commit/7b35292815a709d151add6ef8d2c2f7b20a5c749>.
As a result, I'm hopeful the next pull will be soon.

On Fri, Aug 17, 2012 at 9:12 PM, Greg Hendershott

Matthias Felleisen

unread,
Aug 23, 2012, 1:15:34 PM8/23/12
to Greg Hendershott, us...@racket-lang.org

Thank you very much! -- Matthias
Reply all
Reply to author
Forward
0 new messages