[QUIZ] Whiteout (#34)

[Ruby Quiz]

The three rules of Ruby Quiz:

1. Please do not post any solutions or spoiler discussion for this quiz until
48 hours have passed from the time on this message.

2. Support Ruby Quiz by submitting ideas as often as you can:

http://www.rubyquiz.com/

3. Enjoy!

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Perl programmers have all the fun. They have an entire section of the CPAN
devoted to their playing around. The ACME modules are all fun little toys that
have interesting if rarely very useful effects.

This week's Ruby Quiz is to port ACME::Bleach to Ruby. I won't make you go
hunting through the CPAN to figure it out though. Here's how our version will
work:

1. Make a Ruby file that is both an executable and a library. We'll call
it "whiteout".
2. When "whiteout" is executed, it should take ARGV to be a list of Ruby
source code files to alter in-place. (You may save backup copies if you
like, but the original file should be changed.) Here are the changes:
a. A Shebang line, if present is to be passed through the filter
without any changes.
b. The script should then add the line: require "whiteout"
c. The entire rest of the file should be made invisible. You might do
this by converting the rest of the file to whitespace, as
ACME::Bleach does.
3. When "whiteout" is required, the original code must be executed with no
change in functionality.

Let's show those Perl guys that we know how to have a good time too!

[Dave Burt]

> 3. When "whiteout" is required, the original code must be executed with
> no
> change in functionality.

That means that a file that is just "require 'whiteout'" and then a bunch of
whitespace should run like the original program before it was whited out.

Perl has something called "source filters" (see perl doco "perlfilter"),
which apply a transformation to following source code, kind of like a
preprocessor. This would be an ideal way of dealing with this situation, and
avoid doing File.read($0).

Is there a way to do that in Ruby? Can a C extension concievably alter
following code before it is parsed? I think I'm expecting a simple "no".

Cheers,
Dave

[Florian Groß]

Dave Burt wrote:

> Is there a way to do that in Ruby? Can a C extension concievably alter
> following code before it is parsed? I think I'm expecting a simple "no".

If you are allowed to alter the shebang then you can append -ryourlib
and steal STDIN from Ruby and use eval() to execute the filtered code.

[James Edward Gray II]

On Jun 3, 2005, at 11:40 AM, Dave Burt wrote:

>> 3. When "whiteout" is required, the original code must be
>> executed with
>> no
>> change in functionality.
>>
>
> That means that a file that is just "require 'whiteout'" and then a
> bunch of
> whitespace should run like the original program before it was
> whited out.

Correct.

> Perl has something called "source filters" (see perl doco
> "perlfilter"),
> which apply a transformation to following source code, kind of like a
> preprocessor. This would be an ideal way of dealing with this
> situation, and
> avoid doing File.read($0).

Yeah, this is an interesting if probably evil feature. I wasn't sure
how feasible something like this was, or wasn't in Ruby. I would
like to see a source filter library for Ruby, even if it is bad style.

James Edward Gray II

[James Edward Gray II]

Hmm, I hadn't considered something like this. Since:

#!/usr/local/bin/ruby -rwhiteout

is basically:

#!/usr/local/bin/ruby

require "whiteout"

I'm okay with it if you want to try something like this.

James Edward Gray II

[Eric Hodel]

I believe ParseTree can provide some of that for you.

--
Eric Hodel - drb...@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

[Logan Capaldo]

The problem with doing source filters in ruby is that 'use' doesn't
really work the same way as require. Things that are 'use'd get looked
at BEFORE perl reads the rest of the file. This means that this will
let you do things like have invalid syntax before the source filter
gets to it. In contrast, require in ruby (and in perl btw) occurs more
on the runtime side of the compile --- runtime scale. Same thing with
ruby's BEGIN blocks. Which is why perl will let you do this:

$ cat test.pl
#!/usr/bin/env perl
BEGIN { exit 0; }
dasdadadaddada = 33423q3412 -,_1+e=3

$ perl test.pl
$

Even more amusing:
$ perl -c test.pl
test.pl syntax OK

In ruby however...
$ cat test.rb
#!/usr/bin/env ruby
BEGIN { exit 0 }
fsdnddfqw,+-};

$ ruby test.rb
test.rb:2: syntax error
fsdnddfqw,+-};
^

You can possibly do it with soemthing like this:
$ cat runaway.rb
exit 0

$ ruby -rrunaway test.rb
$

But to use this to create a source filter requires smarter people than me.

[Ryan Leavengood]

This was a fun one. If I would consider anything my Ruby forte, text
processing would be it. So this was right up my alley. I learned a good
bit too. For example, Fixnum#to_s can take a radix representing the base
you want the number converted to in the String. String#to_i does the
same thing, just in the opposite direction.

I first wrote a simple binary conversion that was inspired by what I
could figure out from the original Perl ACME::Bleach (which wasn't too
much since I'm not a Perl hacker and it was somewhat obfuscated.) Then I
thought I could probably one-up that by making a ternary conversion. I
considered trying higher radixes, but found at least on my editor (Vim)
that only spaces, tabs and newlines were truly "invisible." So ternary
it was, as shown in the code below.

Since I had written two conversions, I decided to make things
interesting and randomly choose which one I used when creating the
files. That should thoroughly confuse people who try and decode any
files that have been "whited out" without knowing the code :)

Anyhow, here is the code (if this weren't a Ruby Quiz I would make this
code much more compact and obfuscated):

# Ruby Quiz: Whiteout (#34)
# Solution by Ryan Leavengood
#
# There are two ways of "whiting out", one that uses a binary
# encoding of spaces and tabs on each line (preserving the
# original newlines), and a ternary encoding that makes newlines
# part of the code and encodes any of the original newlines. The
# method of encoding is chosen at random. In theory other
# non-printable characters could be added to increase the radix
# used for encoding, but I think the best cross-platform "whiting
# out" can be had using spaces, tabs and newlines.

REQUIRE_LINE = "require 'whiteout'"

class WhiteoutBinary
attr_reader :id

WHITESPACE = " \t"
DIGITS = '01'

def initialize
@id = " \t\t"
end

def paint_on(paper)
paper.map do |line|
line.chomp.unpack('b*')[0].tr(DIGITS, WHITESPACE)
end.join("\n")
end

def rub_off(paper)
paper.map do |line|
[line.chomp.tr(WHITESPACE, DIGITS)].pack('b*')
end.join("\n")
end
end

class WhiteoutTernary
attr_reader :id

WHITESPACE = " \t\n"
DIGITS = '012'
# This allows up to 22222 ternary, which is 242 decimal, enough
# for most of ASCII
DIGIT_LENGTH = 5
RADIX = 3

def initialize
@id = " \t\t\t"
end

def paint_on(paper)
paper.join.gsub(/./m) do |c|
c[0].to_s(RADIX).rjust(DIGIT_LENGTH,'0')
end.tr(DIGITS, WHITESPACE)
end

def rub_off(paper)
paper.join.tr(WHITESPACE, DIGITS).gsub(/.{#{DIGIT_LENGTH}}/) do |d|
d.to_i(RADIX).chr
end
end
end

bottle_holder = [WhiteoutBinary.new, WhiteoutTernary.new]

if $0 == __FILE__
ARGV.each do |filename|
wo_name = "#{filename}.wo"
File.open(wo_name, 'w') do |file|
whiteout = bottle_holder[rand(2)]
paper = IO.readlines(filename)
if paper[0] =~ /^\s*#!/
file.print paper.shift
end
file.puts REQUIRE_LINE
file.puts whiteout.id
file.print whiteout.paint_on(paper)
end
File.rename(filename, filename+'.bak')
File.rename(wo_name, filename)
end
else
paper = IO.readlines($0)
paper.shift if paper[0] =~ /^\s*#!/
paper.shift if paper[0] =~ /^#{REQUIRE_LINE}/
id = paper.shift.chomp
whiteout = bottle_holder.find {|bottle| bottle.id == id}
if whiteout
eval whiteout::rub_off(paper)
else
puts "Error: This does not appear to be a valid whiteout file!"
exit(1)
end
end
__END__

Ryan Leavengood

[Christian Neukirchen]

Ryan Leavengood <mrc...@netrox.net> writes:

> Anyhow, here is the code (if this weren't a Ruby Quiz I would make this
> code much more compact and obfuscated):

And this is my solution, with no time spend on robustness or error
handling. It is more space efficient, though, as I use base 4:

unless caller.empty?
eval File.read($0). # or extract from caller...
gsub(/\A.*\0/m, '').
tr(" \n\t\v", "0123").
scan(/\d{4}/m).map { |s| s.to_i(4) }.
pack("c*")
else
require 'fileutils'
ARGV.each { |file|
code = File.read file
FileUtils.copy file, file + ".dirty"
File.open(file, "w") { |out|
code.gsub!(/\A#!.*/) { |shebang|
out.puts shebang
''
}
out.puts 'require "whiteout"'
out.print "\0"
code.each_byte { |b|
out.print b.to_s(4).rjust(4).tr("0123", " \n\t\v")
}
}
}
end

--
Christian Neukirchen <chneuk...@gmail.com> http://chneukirchen.org

[Ara.T.Howard]

my solution tried to strike a balance between being readable and user friendly
(usage message, etc.) and succicntness. the basic idea is that whiteout.rb is
a self modifying program that stores the whited-out files in it's __END__
section as yaml using the expanded path of the original source file as the
key. this has the nice side effect that all sources remain quite readable
within the whiteout.rb __END__ section. eg:

jib:~/tmp > ls
a.rb b.rb whiteout.rb

jib:~/tmp > cat whiteout.rb
#!/usr/bin/env ruby
require 'yaml'

this, prog, *paths = [__FILE__, $0, ARGV].flatten.map{|x| File::expand_path x}
usage = "#{ prog } file [files]+"

f = open this, 'r+'
s, pos = f.gets, f.pos until s =~ /^__END__$/
srcs = YAML::load f

if prog == this
abort usage if paths.empty?
abort "#{ prog } must be writable" unless File::stat(this).writable?
paths.each do |path|
s, b = IO::read(path).split(%r/(^\s*#\s*!.*\n)/o).reverse.first 2
srcs[path] = s
open(path,'w'){|o| o.puts b, "require 'whiteout'\n"}
end
f.seek pos and f << srcs.to_yaml and f.truncate f.pos
else
eval srcs[prog]
end

__END__
---
{}

jib:~/tmp > cat a.rb
#!/usr/bin/env ruby
p 42

jib:~/tmp > cat b.rb
#!/usr/bin/env ruby
p 'forty-two'

jib:~/tmp > ruby a.rb
42

jib:~/tmp > ruby b.rb
"forty-two"

jib:~/tmp > whiteout.rb a.rb b.rb

jib:~/tmp > cat a.rb
#!/usr/bin/env ruby
require 'whiteout'

jib:~/tmp > cat b.rb
#!/usr/bin/env ruby
require 'whiteout'

jib:~/tmp > ruby a.rb
42

jib:~/tmp > ruby b.rb
"forty-two"

jib:~/tmp > cat whiteout.rb
#!/usr/bin/env ruby
require 'yaml'

this, prog, *paths = [__FILE__, $0, ARGV].flatten.map{|x| File::expand_path x}
usage = "#{ prog } file [files]+"

f = open this, 'r+'
s, pos = f.gets, f.pos until s =~ /^__END__$/
srcs = YAML::load f

if prog == this
abort usage if paths.empty?
abort "#{ prog } must be writable" unless File::stat(this).writable?
paths.each do |path|
s, b = IO::read(path).split(%r/(^\s*#\s*!.*\n)/o).reverse.first 2
srcs[path] = s
open(path,'w'){|o| o.puts b, "require 'whiteout'\n"}
end
f.seek pos and f << srcs.to_yaml and f.truncate f.pos
else
eval srcs[prog]
end

__END__
---
"/home/ahoward/tmp/b.rb": "p 'forty-two'\n"
"/home/ahoward/tmp/a.rb": "p 42\n"

all and all quite fun!

cheers.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple. My religion is kindness.
| --Tenzin Gyatso
===============================================================================

[Brian Schröder]

Hello Group,

first let me thank you for this quiz. It is a hillarious idea. Never
would have come up with this by myself and it is really cool to see
the sourcecode vanish and reappear as if by magic.

I first thought to modify the whitout library such, that it includes a
loaded program, but then came to the solution to encode the file in
the whitespace. The simplest encoding / decoding routines are:

# Encode a string into whitespace
def encode(string)
string.unpack('B*')[0].tr('01', " \n")
end

# Decode a whitespace encoded string
def decode(string)
[string.tr(" \n", '01')].pack('B*')
end

But these make the files quite big. So I added two thing:
- Encoding into a eight-sign code with eight different whitespace
symbols that ruby doesn't choke on. This may show as non-whitespace in
something like an editor or "less", but using "cat" its only
whitespace.
- On the fly zipping of the files before encoding.

The encoding of a encoded file is detected by a four byte whitespace
header, so everything works out automatically.

I also added a nice user interface using optparse. Though there is
some optparse bashing going on at the moment I think it is really a
great library.

You can find my solution at:
http://ruby.brian-schroeder.de/quiz/whiteout/

Have fun with the solution and please don't come after me if you have
converted all your ruby code to whitespace and can't get it back.

--
http://ruby.brian-schroeder.de/

Guitar chords in different tunings: http://chordlist.brian-schroeder.de/guitar/

[Dominik Bathon]

Here is my solution. It is quite similar to all the others already posted.

I use Zlib::Deflate to compress the source file, then the bytes are
converted to base 3 and represented by spaces, tabs and newlines.
This way the result is approx. 3 times bigger than the source.

Dominik

The code:

require "zlib"

def encode_to_ws(str)
str=Zlib::Deflate.deflate(str, 9)
res=""
str.each_byte { |b| res << b.to_s(3).rjust(6,"0") }
res.tr("012", " \t\n")
end

def decode_from_ws(str)
raise "wrong length" unless str.length%6 == 0
str.tr!(" \t\n", "012")
res=""
for i in 0...(str.length/6)
res << str[i*6, 6].to_i(3).chr
end
Zlib::Inflate.inflate(res)
end

if $0 == __FILE__
if File.file?(f=ARGV[0])
str=IO.read(f)
File.open(f, "wb") { |out|
if str =~ /\A#!.*/
out.puts $&
end
out.puts 'require "whiteout"'
out.print encode_to_ws(str)
}
else
puts "usage #$0 file.rb"
end
else
if File.file?($0)
str=File.read($0)
str.sub!(/\A(#!.*)?require "whiteout".*?\n/m, "")
eval('$0=__FILE__')
eval(decode_from_ws(str))
else
raise "required whiteout from non-file"
end
end

[Matthew D Moss]

I thought of adding more bits by using other whitespace characters, but
in the end stuck with space and tab, since everything will render them
as whitespace (whereas almost any other whitespace I tried showed up as
garbage in one editor/reader or another). Also, I didn't use linefeed
nor carriage return, since non-binary transmission from one system to
another could potentially break the file (ie, eol conversion).

Anyway, here's my solution. I tried to keep it short and simple but
should still be easily understood.

#!/usr/bin/env ruby

Bits = '01'
Blank = " \t"

def shebang(f)
f.pos = 0 unless f.gets =~ /^#!/
end

def confuse(fname)
File.open(fname, File::RDWR) do |f|
shebang f
f.pos, data = f.pos, f.read
f.puts "require '#{File.basename($0, '.*')}'"
f.write data.unpack('b*').join.tr(Bits, Blank)
f.truncate f.pos
end
end

def clarify(fname)
File.open(fname, File::RDONLY) do |f|
shebang f
f.gets # skip require 'whiteout'
eval [f.read.tr(Blank, Bits)].pack('b*')
end
end

if __FILE__ == $0
ARGV.each { |fname| confuse fname }
else
clarify($0)
end

[Matthew D Moss]

I just realized I can shave this down by one expression by removing
"f.truncate f.pos". I put that in there to esure that, if the
"confused" version were shorter, the file would have the proper length.
However, since the file is always expanded by this algorithm, there's
no need for truncate.