I'm getting non-deterministic lexing errors from Gherkin. When parsing a particular file, *sometimes* Gherkin throws Gherkin::Lexer::LexingError, and *sometimes* it doesn't, without the input changing in any way. I pasted the stack trace at the end of this message.
The symptoms I'm seeing are exactly as described in this unresolved issue<https://github.com/cucumber/gherkin/issues/182>. But I don't think my problem has to do with character encoding.
I tried setting force_ruby=true in gherkin/i18n.rb, as suggested in the issue comments, but that just gave me a different error:
No lexer was found for en (no such file to load -- gherkin/lexer/en).
> Supported languages are listed in gherkin/i18n.json. > (Gherkin::I18n::LexerNotFound)
I spent a couple hours altering my Cucumber tests to try to narrow down what token is causing the problem, but I was unsuccessful. However, I did manage to narrow down one of my step definition files into something that is triggering a LexingError *every time*. And yet the code appears to be lexically valid. That file is also pasted below with the stack trace.
Of particular interest are the last two step definitions -- I wasn't having any of the non-deterministic problems until I added them.
If I remove either the *last* step definition, or *any combination of the first three*, then the lexer error goes away (deterministically, I believe). Very strange...
Does anyone have any idea what is lexically wrong with my code? If not, how can I debug the lexer? I want to see exactly how the lexer is parsing the tokens. Are there any docs that describe how to do that?
Thank you,
Trevor
*Versions* OS X: 10.8.2 Ruby: 1.8.7 (2012-02-08 patchlevel 358) gcc: i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) Cucumber: 1.2.1
*Stack Trace* features/shopping_cart.feature: Lexing error on line 5: ''. See http://wiki.github.com/cucumber/gherkin/lexingerror for more information. (Gherkin::Lexer::LexingError) /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140 :in `scan' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140 :in `transition_table' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:127 :in `build_transition_map' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:123 :in `transition_map' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:90: in `initialize' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67: in `new' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67: in `push_machine' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:102 :in `event' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:56: in `event' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:48: in `comment' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb: 22:in `scan' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb: 22:in `scan' /Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:32: in `parse' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/feature_file .rb:37:in `parse' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:28:in `load' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:26:in `each' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:26:in `load' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:14:in `features' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:1 70:in `features' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:4 6:in `run!' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb: 43:in `execute!' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb: 20:in `execute' /Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/cucumber:14 /usr/bin/cucumber:19:in `load' /usr/bin/cucumber:19
*Code* When /^I dismiss the Favorites tooltip$/ do wait_for { screen_exists("Venues") } sleep(STEP_PAUSE) # Wait for the transition animation tap_center end
When /^I choose the venue "(.*?)"$/ do |venue| wait_for { screen_exists("Venues") } touch("searchBarTextField") keyboard_enter_text(venue) done wait_for { query("tableViewCell index:0", :text).first.to_s.strip ==venue
}
touch("tableViewCell index:0") end
When /^I go to the Order screen$/ do wait_for { screen_exists("Welcome") } touch("buttonLabel marked:'Order'") end
When /^I choose "(.*?)" from the "(.*?)" tab$/ do |item, tab| end
When /^I choose "(.*?)" with modifier "(.*?)" from the "(.*?)" tab$/ do | item, modifier, tab| end
On Sunday, 11 November 2012 at 03:11, Trevor Harmon wrote:
> Hi,
> I'm getting non-deterministic lexing errors from Gherkin. When parsing a particular file, sometimes Gherkin throws Gherkin::Lexer::LexingError, and sometimes it doesn't, without the input changing in any way. I pasted the stack trace at the end of this message.
> The symptoms I'm seeing are exactly as described in this unresolved issue (https://github.com/cucumber/gherkin/issues/182). But I don't think my problem has to do with character encoding.
> I tried setting force_ruby=true in gherkin/i18n.rb, as suggested in the issue comments, but that just gave me a different error:
> > No lexer was found for en (no such file to load -- gherkin/lexer/en). Supported languages are listed in gherkin/i18n.json. (Gherkin::I18n::LexerNotFound)
> I spent a couple hours altering my Cucumber tests to try to narrow down what token is causing the problem, but I was unsuccessful. However, I did manage to narrow down one of my step definition files into something that is triggering a LexingError every time. And yet the code appears to be lexically valid. That file is also pasted below with the stack trace.
> Of particular interest are the last two step definitions -- I wasn't having any of the non-deterministic problems until I added them.
> If I remove either the last step definition, or any combination of the first three, then the lexer error goes away (deterministically, I believe). Very strange...
Are you seriously putting step definitions (ruby code) in your .feature files or did I misunderstand completely here?
> Does anyone have any idea what is lexically wrong with my code? If not, how can I debug the lexer? I want to see exactly how the lexer is parsing the tokens. Are there any docs that describe how to do that?
> Thank you,
> Trevor
> Versions
> OS X: 10.8.2
> Ruby: 1.8.7 (2012-02-08 patchlevel 358)
> gcc: i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
> Cucumber: 1.2.1
> Code
> When /^I dismiss the Favorites tooltip$/ do
> wait_for { screen_exists("Venues") }
> sleep(STEP_PAUSE) # Wait for the transition animation
> tap_center
> end
> When /^I choose the venue "(.*?)"$/ do |venue|
> wait_for { screen_exists("Venues") }
> touch("searchBarTextField")
> keyboard_enter_text(venue)
> done
> wait_for { query("tableViewCell index:0", :text).first.to_s.strip == venue }
> touch("tableViewCell index:0")
> end
> When /^I go to the Order screen$/ do
> wait_for { screen_exists("Welcome") }
> touch("buttonLabel marked:'Order'")
> end
> When /^I choose "(.*?)" from the "(.*?)" tab$/ do |item, tab|
> end
> When /^I choose "(.*?)" with modifier "(.*?)" from the "(.*?)" tab$/ do |item, modifier, tab|
> end
> -- > -- Rules --
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].
> 2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style > 3) If you have a question, don't reply to an existing message. Start a new topic instead.
> You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cukes@googlegroups.com. To unsubscribe from this group, send email to cukes+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
On Nov 11, 2012, at 1:25 AM, Aslak Hellesøy <aslak.helle...@gmail.com> wrote:
> Are you seriously putting step definitions (ruby code) in your .feature files or did I misunderstand completely here?
Sorry, I was a bit unclear about that. The lexing error occurs when I run Cucumber on one of my .feature files, but it is caused by something in my step definitions. In other words, I can change my step definitions and make the lexing error go away (or come back again) without touching the .feature file.
On Sun, Nov 11, 2012 at 2:18 PM, Trevor Harmon <tre...@vocaro.com> wrote:
> On Nov 11, 2012, at 1:25 AM, Aslak Hellesøy <aslak.helle...@gmail.com>
> wrote:
> > Are you seriously putting step definitions (ruby code) in your .feature
> files or did I misunderstand completely here?
> Sorry, I was a bit unclear about that. The lexing error occurs when I run
> Cucumber on one of my .feature files, but it is caused by something in my
> step definitions. In other words, I can change my step definitions and make
> the lexing error go away (or come back again) without touching the .feature
> file.
Oh I see :-)
Can you create a minimal project on Github so others can reproduce this?
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].
> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style > 3) If you have a question, don't reply to an existing message. Start a new
> topic instead.
> You received this message because you are subscribed to the Google Groups
> Cukes group. To post to this group, send email to cukes@googlegroups.com.
> To unsubscribe from this group, send email to
> cukes+unsubscribe@googlegroups.com. For more options, visit this group at
> https://groups.google.com/d/forum/cukes?hl=en
On Mon, Nov 12, 2012 at 6:35 PM, Trevor Harmon <tre...@vocaro.com> wrote:
> On Nov 11, 2012, at 6:32 AM, aslak hellesoy <aslak.helle...@gmail.com>
> wrote:
> > Can you create a minimal project on Github so others can reproduce this?
> Yes, I'm still working on stripping down to a reproducible test case I can
> share.
> In the meantime, I'm curious… What could possibly cause non-determinism in
> a lexer?
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].
> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style > 3) If you have a question, don't reply to an existing message. Start a new
> topic instead.
> You received this message because you are subscribed to the Google Groups
> Cukes group. To post to this group, send email to cukes@googlegroups.com.
> To unsubscribe from this group, send email to
> cukes+unsubscribe@googlegroups.com. For more options, visit this group at
> https://groups.google.com/d/forum/cukes?hl=en
On Sunday, November 11, 2012 6:33:10 AM UTC-8, Aslak Hellesøy wrote:
> Can you create a minimal project on Github so others can reproduce this?
I copied my steps and step definitions to another machine and installed the dependent gems, but the lexer error went away. :(
If I could just see how the lexer is tokenizing my code, I might be able to figure out what's wrong. Any pointers on how to hack the Cucumber source to do that?
For reference, here is the stack trace:
features/shopping_cart.feature: Lexing error on line 4: ''. See http://wiki.github.com/cucumber/gherkin/lexingerror for more information. (Gherkin::Lexer::LexingError)
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140 :in `scan'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140 :in `transition_table'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:127 :in `build_transition_map'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:123 :in `transition_map'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:90: in `initialize'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67: in `new'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67: in `push_machine'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:102 :in `event'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:56: in `event'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:48: in `comment'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb: 22:in `scan'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb: 22:in `scan'
/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:32: in `parse'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/feature_file .rb:37:in `parse'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:28:in `load'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:26:in `each'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:26:in `load'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/feat ures_loader.rb:14:in `features'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:1 70:in `features'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:4 6:in `run!'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb: 43:in `execute!'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb: 20:in `execute'
/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/cucumber:14
/usr/bin/cucumber:19:in `load'
/usr/bin/cucumber:19
I'm getting clues that the lexer error might be triggered by a comment at the end of a feature file. Something like:
Feature: Registration Scenario: Successful registration Given I am on the Register screen When I register as a new user # TODO: Finish this later
If I add a comment like this to one of my scenarios, I get the lexer error about 1 out of every 5 runs. If I remove the comment, I can run it 10 times without errors.
On Mon, Nov 12, 2012 at 8:29 PM, Trevor Harmon <tre...@vocaro.com> wrote:
> I'm getting clues that the lexer error might be triggered by a comment at
> the end of a feature file. Something like:
> Feature: Registration
> Scenario: Successful registration
> Given I am on the Register screen
> When I register as a new user
> # TODO: Finish this later
Comments at the end of the file should deterministically cause a parse
error (not a lexing error).
> If I add a comment like this to one of my scenarios, I get the lexer error
> about 1 out of every 5 runs. If I remove the comment, I can run it 10 times
> without errors.
> Thoughts?
I think the only way to fix this is a minimal reproducible example.
Guesswork will get us nowhere.
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].
> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style > 3) If you have a question, don't reply to an existing message. Start a new
> topic instead.
> You received this message because you are subscribed to the Google Groups
> Cukes group. To post to this group, send email to cukes@googlegroups.com.
> To unsubscribe from this group, send email to
> cukes+unsubscribe@googlegroups.com. For more options, visit this group at
> https://groups.google.com/d/forum/cukes?hl=en
> I'm getting clues that the lexer error might be triggered by a comment at the end of a feature file. Something like:
> Feature: Registration
> Scenario: Successful registration
> Given I am on the Register screen
> When I register as a new user
> # TODO: Finish this later
> If I add a comment like this to one of my scenarios, I get the lexer error about 1 out of every 5 runs. If I remove the comment, I can run it 10 times without errors.
> Thoughts?
Have we established what platform / editor the OP is using?
I have a feeling the last time this happened it was a Windows editor saving non-UTF8 characters into the file.
