Intermittent test failure, "incompatible encoding regexp match"

[Bryan Stearns]

I'm seeing an intermittent failure running Calagator tests (actually, from loading the Rails environment to run tests with "bundle exec rake"):

/Users/stearns/rad/calagator/app/helpers/application_helper.rb:44:in `match':

incompatible encoding regexp match (Windows-31J regexp with UTF-8 string) (Encoding::CompatibilityError)

It's this line here -- https://github.com/calagator/calagator/blob/0b66bdbd307549860309c2defb90d5269e4428eb/app/helpers/application_helper.rb#L42

where ApplicationHelper.source_code_version_raw is trying to match the output from a backtick'd git command (which has UTF-8 encoding) with a regex that has a /s switch on it:

        m = s.match(/^Date: (.+?)$/s)

That /s means "Use Windows 31J encoding" (and has since Ruby 1.9.0). I haven't had to mess with Ruby encodings (especially on Windows) so I don't know whether the /s is important -- I'm on Mac, and removing the /s gets rid of the failure.

In trying to isolate the failure, I've gone from consistent success with an unrelated uncommited change (to a Rakefile task that isn't even being run!) to having consistent failure after simply committing the change, then consistent success after "git reset HEAD^" (to un-commit it again). I've looked, and the encoding of s is Encoding:UTF-8 in both cases (that is, in situations where it's failing consistently or succeeding consistently.

Any suggestions? Anyone know why the /s is needed? Can someone who develops on Windows look to see what encoding a backtick'd git command returns nowadays?

Thanks,

...Bryan

[Reid Beels]

I don’t have the ability to check on Windows at the moment, but it seems that explicitly converting the returned string to UTF-8 before running the regexp might be a way to skirt this.

--
You received this message because you are subscribed to the Google Groups "Calagator Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to calagator-develo...@googlegroups.com.
To post to this group, send email to calagator-...@googlegroups.com.
Visit this group at http://groups.google.com/group/calagator-development.
For more options, visit https://groups.google.com/d/optout.

[Bryan Stearns]

Thanks, Reid. Explicitly reencoding to UTF-8 got me the same error :-(, but reencoding to Encoding::Windows_31J succeeds, sort of*.

While playing with this, I noticed that gitx (which I almost always use to do git commits) converts normal apostrophes to curly quotes: as I was playing with this, my last commit message was "Fix tests’ Sunspot/Solr running check". If I look at the reencoded (to W31J) string, the message is "Fix tests\x{8166} Sunspot/Solr running check"; this doesn't really matter since all we're only pulling the timestamp out of it, but still... I don't like the idea of forcing Windows_31J encoding, and I'm reluctant to remove the /s without more understanding of the impact on Windows...

-- so, for now, I'll just make sure not to use gitx on Calagator. If anyone knows why the /s is needed, please pipe up, because I'm sure this'll bite someone else someday.

...Bryan

On Sunday, July 27, 2014 8:08:23 PM UTC-7, Reid Beels wrote:

I don’t have the ability to check on Windows at the moment, but it seems that explicitly converting the returned string to UTF-8 before running the regexp might be a way to skirt this.

On July 27, 2014 at 6:25:23 PM, Bryan Stearns (bryans...@gmail.com) wrote:

I'm seeing an intermittent failure running Calagator tests (actually, from loading the Rails environment to run tests with "bundle exec rake"):

/Users/stearns/rad/calagator/app/helpers/application_helper.rb:44:in `match':

incompatible encoding regexp match (Windows-31J regexp with UTF-8 string) (Encoding::CompatibilityError)

It's this line here -- https://github.com/calagator/calagator/blob/0b66bdbd307549860309c2defb90d5269e4428eb/app/helpers/application_helper.rb#L42

where ApplicationHelper.source_code_version_raw is trying to match the output from a backtick'd git command (which has UTF-8 encoding) with a regex that has a /s switch on it:

        m = s.match(/^Date: (.+?)$/s)

That /s means "Use Windows 31J encoding" (and has since Ruby 1.9.0). I haven't had to mess with Ruby encodings (especially on Windows) so I don't know whether the /s is important -- I'm on Mac, and removing the /s gets rid of the failure.

In trying to isolate the failure, I've gone from consistent success with an unrelated uncommited change (to a Rakefile task that isn't even being run!) to having consistent failure after simply committing the change, then consistent success after "git reset HEAD^" (to un-commit it again). I've looked, and the encoding of s is Encoding:UTF-8 in both cases (that is, in situations where it's failing consistently or succeeding consistently.

Any suggestions? Anyone know why the /s is needed? Can someone who develops on Windows look to see what encoding a backtick'd git command returns nowadays?

Thanks,

...Bryan

--
You received this message because you are subscribed to the Google Groups "Calagator Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to calagator-development+unsub...@googlegroups.com.

[Reid Beels]

Subversion spelunking time! Out of curiosity, I dug into the early history of this bit of code…

Calagator has had its revision in the footer since our first day of development on January 19, 2008. Igal added it in revision 26 using Subversion’s keyword substitution feature.

Three months later, on Mar 2, 2008, support for git-svn revisions was added and the svn support changed to use a shell command. The regular expressions with /s were there from the start in revisions 236 and 250.

For context: Ruby 1.8.7 was released ninety days later, on May 31.

Over time, as this logic moved from the layout into application_helper.rb, got Mercurial support (?!), and underwent some refactoring, the original /s remained.

I would hazard a guess that Igal’s original intention with the /s was probably not to invoke a somewhat-obscure Japanese code page, but instead was a holdover from perl, where it’s used to treat multi-line strings as if they were single-line.

So: I propose we change the /s to Ruby’s equivalent /m and see how things work. (It also might be a good time to drop support for Mercurial and SVN revisions.) We don’t ever want this revision-parsing code to get in the way people trying to work on Calagator, so we may want to widen the set of exceptions caught by the surrounding begin…rescue block.

~Reid

To unsubscribe from this group and stop receiving emails from it, send an email to calagator-develo...@googlegroups.com.

[Bryan Stearns]

Excellent spelunking, Reid! I'd figured out that /s was added to Ruby for 1.9.0, but hadn't gone back far enough to figure out whether Calagator used it before that...

I'll have a pull request shortly that does what you suggest.

...Bryan