Non-deterministic lexing errors -- how to debug?

[Trevor Harmon]

Hi,

I'm getting non-deterministic lexing errors from Gherkin. When parsing a particular file, sometimes Gherkin throws Gherkin::Lexer::LexingError, and sometimes it doesn't, without the input changing in any way. I pasted the stack trace at the end of this message.

The symptoms I'm seeing are exactly as described in this unresolved issue. But I don't think my problem has to do with character encoding.

I tried setting force_ruby=true in gherkin/i18n.rb, as suggested in the issue comments, but that just gave me a different error:

No lexer was found for en (no such file to load -- gherkin/lexer/en). Supported languages are listed in gherkin/i18n.json. (Gherkin::I18n::LexerNotFound)

I spent a couple hours altering my Cucumber tests to try to narrow down what token is causing the problem, but I was unsuccessful. However, I did manage to narrow down one of my step definition files into something that is triggering a LexingError every time. And yet the code appears to be lexically valid. That file is also pasted below with the stack trace.

Of particular interest are the last two step definitions -- I wasn't having any of the non-deterministic problems until I added them.

If I remove either the last step definition, or any combination of the first three, then the lexer error goes away (deterministically, I believe). Very strange...

Does anyone have any idea what is lexically wrong with my code? If not, how can I debug the lexer? I want to see exactly how the lexer is parsing the tokens. Are there any docs that describe how to do that?

Thank you,

Trevor

Versions

OS X: 10.8.2

Ruby: 1.8.7 (2012-02-08 patchlevel 358)

gcc: i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

Cucumber: 1.2.1

Stack Trace

features/shopping_cart.feature: Lexing error on line 5: ''. See http://wiki.github.com/cucumber/gherkin/lexingerror for more information. (Gherkin::Lexer::LexingError)

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140:in `scan'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:140:in `transition_table'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:127:in `build_transition_map'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:123:in `transition_map'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:90:in `initialize'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67:in `new'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:67:in `push_machine'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:102:in `event'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:56:in `event'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:48:in `comment'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb:22:in `scan'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/lexer/i18n_lexer.rb:22:in `scan'

/Library/Ruby/Gems/1.8/gems/gherkin-2.11.5/lib/gherkin/parser/parser.rb:32:in `parse'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/feature_file.rb:37:in `parse'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/features_loader.rb:28:in `load'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/features_loader.rb:26:in `each'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/features_loader.rb:26:in `load'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime/features_loader.rb:14:in `features'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:170:in `features'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/runtime.rb:46:in `run!'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb:43:in `execute!'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/../lib/cucumber/cli/main.rb:20:in `execute'

/Library/Ruby/Gems/1.8/gems/cucumber-1.2.1/bin/cucumber:14

/usr/bin/cucumber:19:in `load'

/usr/bin/cucumber:19

Code

When /^I dismiss the Favorites tooltip$/ do
  wait_for { screen_exists("Venues") }
  sleep(STEP_PAUSE)  # Wait for the transition animation
  tap_center
end

When /^I choose the venue "(.*?)"$/ do |venue|
  wait_for { screen_exists("Venues") }
  touch("searchBarTextField")
  keyboard_enter_text(venue)
  done
  wait_for { query("tableViewCell index:0", :text).first.to_s.strip == venue }
  touch("tableViewCell index:0")
end

When /^I go to the Order screen$/ do
  wait_for { screen_exists("Welcome") }
  touch("buttonLabel marked:'Order'")
end

When /^I choose "(.*?)" from the "(.*?)" tab$/ do |item, tab|
end

When /^I choose "(.*?)" with modifier "(.*?)" from the "(.*?)" tab$/ do |item, modifier, tab|
end

[Aslak Hellesøy]

On Sunday, 11 November 2012 at 03:11, Trevor Harmon wrote:

Hi,

I'm getting non-deterministic lexing errors from Gherkin. When parsing a particular file, sometimes Gherkin throws Gherkin::Lexer::LexingError, and sometimes it doesn't, without the input changing in any way. I pasted the stack trace at the end of this message.

The symptoms I'm seeing are exactly as described in this unresolved issue. But I don't think my problem has to do with character encoding.

I tried setting force_ruby=true in gherkin/i18n.rb, as suggested in the issue comments, but that just gave me a different error:

No lexer was found for en (no such file to load -- gherkin/lexer/en). Supported languages are listed in gherkin/i18n.json. (Gherkin::I18n::LexerNotFound)

I spent a couple hours altering my Cucumber tests to try to narrow down what token is causing the problem, but I was unsuccessful. However, I did manage to narrow down one of my step definition files into something that is triggering a LexingError every time. And yet the code appears to be lexically valid. That file is also pasted below with the stack trace.

Of particular interest are the last two step definitions -- I wasn't having any of the non-deterministic problems until I added them.

If I remove either the last step definition, or any combination of the first three, then the lexer error goes away (deterministically, I believe). Very strange...

Are you seriously putting step definitions (ruby code) in your .feature files or did I misunderstand completely here?

Aslak 

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 
You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
 
 

[Trevor Harmon]

On Nov 11, 2012, at 1:25 AM, Aslak Hellesøy <aslak.h...@gmail.com> wrote:

> Are you seriously putting step definitions (ruby code) in your .feature files or did I misunderstand completely here?

Sorry, I was a bit unclear about that. The lexing error occurs when I run Cucumber on one of my .feature files, but it is caused by something in my step definitions. In other words, I can change my step definitions and make the lexing error go away (or come back again) without touching the .feature file.

Trevor

[aslak hellesoy]

Oh I see :-)

Can you create a minimal project on Github so others can reproduce this?

Aslak

Trevor

[Trevor Harmon]

On Nov 11, 2012, at 6:32 AM, aslak hellesoy <aslak.h...@gmail.com> wrote:

> Can you create a minimal project on Github so others can reproduce this?

Yes, I'm still working on stripping down to a reproducible test case I can share.

In the meantime, I'm curious… What could possibly cause non-determinism in a lexer? Is Cucumber multithreaded?

Trevor

[aslak hellesoy]

On Mon, Nov 12, 2012 at 6:35 PM, Trevor Harmon <tre...@vocaro.com> wrote:

On Nov 11, 2012, at 6:32 AM, aslak hellesoy <aslak.h...@gmail.com> wrote:

> Can you create a minimal project on Github so others can reproduce this?

Yes, I'm still working on stripping down to a reproducible test case I can share.

In the meantime, I'm curious… What could possibly cause non-determinism in a lexer?

No idea - very few people have seen this problem.

 

Is Cucumber multithreaded?

Nope

[Trevor Harmon]

On Sunday, November 11, 2012 6:33:10 AM UTC-8, Aslak Hellesøy wrote:

Can you create a minimal project on Github so others can reproduce this?

I copied my steps and step definitions to another machine and installed the dependent gems, but the lexer error went away. :(

If I could just see how the lexer is tokenizing my code, I might be able to figure out what's wrong. Any pointers on how to hack the Cucumber source to do that?

For reference, here is the stack trace:

features/shopping_cart.feature: Lexing error on line 4: ''. See http://wiki.github.com/cucumber/gherkin/lexingerror for more information. (Gherkin::Lexer::LexingError)

Thanks,

Trevor

[Trevor Harmon]

I'm getting clues that the lexer error might be triggered by a comment at the end of a feature file. Something like:

Feature: Registration
  Scenario: Successful registration
    Given I am on the Register screen
    When I register as a new user
    # TODO: Finish this later

If I add a comment like this to one of my scenarios, I get the lexer error about 1 out of every 5 runs. If I remove the comment, I can run it 10 times without errors.

Thoughts?

[aslak hellesoy]

On Mon, Nov 12, 2012 at 8:29 PM, Trevor Harmon <tre...@vocaro.com> wrote:

I'm getting clues that the lexer error might be triggered by a comment at the end of a feature file. Something like:

Feature: Registration
  Scenario: Successful registration
    Given I am on the Register screen
    When I register as a new user
    # TODO: Finish this later

Comments at the end of the file should deterministically cause a parse error (not a lexing error).

 

If I add a comment like this to one of my scenarios, I get the lexer error about 1 out of every 5 runs. If I remove the comment, I can run it 10 times without errors.

Thoughts?

I think the only way to fix this is a minimal reproducible example. Guesswork will get us nowhere.

[Matt Wynne]

Have we established what platform / editor the OP is using?

I have a feeling the last time this happened it was a Windows editor saving non-UTF8 characters into the file.

cheers,

Matt

--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com

Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne

[Trevor Harmon]

On Nov 12, 2012, at 2:14 PM, Matt Wynne <ma...@mattwynne.net> wrote:

> Have we established what platform / editor the OP is using?
>
> I have a feeling the last time this happened it was a Windows editor saving non-UTF8 characters into the file.

I'm using Sublime Text on a Mac. 'file' reports ASCII text for all input files.

Trevor

[Trevor Harmon]

Replying to this old thread because I think I found a solution: Use Ruby 1.9.3 instead of Ruby 1.8.7.