Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.

[Caml-list] updating position in an ocamllex lexer

Skip to first unread message

Sébastien Hinderer

Jul 28, 2015, 11:52:14 AM7/28/15
Dear all,

I have to maintain a lexer which did not, so far, compute token
positions during the lexing step. The positions were computed
later, which was not very eficient. So I am now trying to compute
positions during the lexing and have a problem to figure out what
to do for one token:

| ['\n'] [' ' '\t' '\r' '\011' '\012' ]* { ... }

When this token is met and contains just "\n", calling new_line on the
lexbuf is enough to keep further position information accurate.

However, if the recognized token is, say, "\n\t", then it seems that the
column for further tokens will be incorrect.

I assumed that I had to manually update the current position and added
code like so:

let s = Lexing.lexeme lexbuf in
let l = String.length s in
let t = TCommentNewline (tokinfo lexbuf) in
(* Adjust the column manually *)
Lexing.new_line lexbuf;
let lcp = lexbuf.lex_curr_p in
lexbuf.lex_curr_p <- { lcp with
pos_cnum = lcp.pos_bol + l - 1;

But that does not seem to work.

Does somebody know how such tokens should be handled, please?

Many thanks in advance for any help,


Caml-list mailing list. Subscription management and archives:
Beginner's list:
Bug reports:
0 new messages