Use now working positive lookahead to simplify template lexer

[Thue Janus Kristensen]

Using the new positive lookahead, which I recently fixed, it is possible to simplify the template lexer. An example (also attached):

Index: development/lexer/smarty_internal_templatelexer.plex
===================================================================
--- development/lexer/smarty_internal_templatelexer.plex        (revision 3412)
+++ development/lexer/smarty_internal_templatelexer.plex        (working copy)
@@ -156,7 +156,7 @@
 qmark = /\s*\?\s*/
 constant = /([_]+[A-Z0-9][0-9A-Z_]*|[A-Z][0-9A-Z_]*)(?![0-9A-Z_]*[a-z])/
 id = /[0-9]*[a-zA-Z_]\w*/
-literaltexttoend = '[\S\s]+?(SMARTYldel/?literalSMARTYrdel|<\?)'
+literaltexttoend = '[\S\s]+?(?=SMARTYldel/?literalSMARTYrdel|<\?)'
 literalstart = 'SMARTYldelliteralSMARTYrdel'
 literalend = 'SMARTYldel/literalSMARTYrdel'
 stripstart = 'SMARTYldelstripSMARTYrdel'
@@ -546,18 +546,7 @@
   $this->token = Smarty_Internal_Templateparser::TP_LITERAL;
 }
 literaltexttoend {
-  $lenght_literal = strlen($this->smarty->left_delimiter.$this->smarty->right_delimiter)+7;
-  if (substr($this->value,-2,2) === '<?') {
-    $this->token = Smarty_Internal_Templateparser::TP_LITERAL;
-    $this->value = substr($this->value,0,-2);
-  } else if (substr($this->value,-$lenght_literal,$lenght_literal) === $this->smarty->left_delimiter.'literal'.$this->smarty->right_delimiter) {
-    $this->token = Smarty_Internal_Templateparser::TP_LITERAL;
-    $this->value = substr($this->value,0,-$lenght_literal);
-  } else {
-    assert(substr($this->value,-$lenght_literal-1,$lenght_literal+1) === $this->smarty->left_delimiter.'/literal'.$this->smarty->right_delimiter);
-    $this->token = Smarty_Internal_Templateparser::TP_LITERAL;
-    $this->value = substr($this->value,0,-$lenght_literal-1);
-  }
+  $this->token = Smarty_Internal_Templateparser::TP_LITERAL;
 }
 all {
   $this->compiler->trigger_template_error ("missing or misspelled literal closing tag");


Generally, I think it is possible to simplify almost all lexer rules which currently use substr.

Regards, Thue