Newbie help: File searching and writing output to new file

[drew]

Hi people,

I've done a bit of searching through this group, and found some things
that I thought would help me, but unfortunately they haven't.

I'm just trying to write a short script that will search through web
server log files and write out lines from the original files that
contain a certain term to a new file. I did have the "puts" line of my
code working, but after a few changes that doesn't work either!

What am I doing wrong?? :

There are a bunch of "ex2005##.log" files in the current directory.

--------------------
#!c:\ruby\bin\ruby

def find_redir
found = []
Dir['ex*.log'].each do |file|
file.each_line do |line|
#found.push line if line =~ /redir/
found.push line if line["redir"]
#puts line if line["redir"]
end
File.open("found-redir-lines.txt", "w+") do |o|
o.write found.join
end
#p found.join
end
end

def show_500_errors
f = File.open("found-redir-lines.txt", "r")
f.each do |line|
puts line if line["500"]
end
end

find_redir()
#show_500_errors()
-------------------

The second function is to go through the new file and find 500 server
errors, which I guess I could have done in the first function, but not
to worry.

So why does this produce a create a "found-redir-lines.txt" file, but
leave it blank ??

Thanks in advance, rubyists
Drew

[Daniel Sheppard]

Your problem in find_redir that you were opening the file for write for
each of the input files. When you open the file for write, it zero's out
the file. I'm guessing that the last file that you open doesn't have any
matches.

This will do the what you want:

def find_redir
File.open("found-redir-lines.txt", "w") do |out|
Dir['ex*.log'].each do |in|
in.grep(/redir/) { |line| out.puts(line) }
end
end
end

That way, you're also not keeping all the lines in memory - you're just
outputting each one as you find it.

> --------------------
> #!c:\ruby\bin\ruby
>
> def find_redir
> found = []
> Dir['ex*.log'].each do |file|
> file.each_line do |line|
> #found.push line if line =~ /redir/
> found.push line if line["redir"]
> #puts line if line["redir"]
> end

> #p found.join

> File.open("found-redir-lines.txt", "w+") do |o|
> o.write found.join
> end

> end
> end
>
> def show_500_errors
> f = File.open("found-redir-lines.txt", "r")
> f.each do |line|
> puts line if line["500"]
> end
> end
>
> find_redir()
> #show_500_errors()
> -------------------

#####################################################################################
This email has been scanned by MailMarshal, an email content filter.
#####################################################################################

[William James]

drew wrote:
> Hi people,
>
> I've done a bit of searching through this group, and found some things
> that I thought would help me, but unfortunately they haven't.
>
> I'm just trying to write a short script that will search through web
> server log files and write out lines from the original files that
> contain a certain term to a new file. I did have the "puts" line of my
> code working, but after a few changes that doesn't work either!
>
> What am I doing wrong?? :
>
> There are a bunch of "ex2005##.log" files in the current directory.
>
> --------------------
> #!c:\ruby\bin\ruby
>
> def find_redir
> found = []
> Dir['ex*.log'].each do |file|
> file.each_line do |line|

"file" is a string, the name of a file. It isn't a file handle.
You are not reading from a file.

ruby -e"ARGF.each_line{|x|puts x if x[/redir/]}" ex*.log >out.txt

[William James]

William James wrote:

> ruby -e"ARGF.each_line{|x|puts x if x[/redir/]}" ex*.log >out.txt

ruby -ne"puts $_ if /redir/" ex*.log >out.txt

[William James]

ruby -ne"print if /redir/" ex*.log >out.txt

[drew]

Thanks very much Daniel and William, I will try Daniel's fix soon.

Even though I know this is a trivial thing to do, I was deliberately
trying to do it in a script rather than from the command line, as that
way I can learn more ruby :->

So William, thank you for your suggestions, I will try to remember them
for when I am a master rubyist in years to come - I love the way each
post contained a more condensed version of the one-liner, as you
thought about it more ... the wonder of Ruby :-> (and the CLI helps a
lot too)

thanks very much people,
drew

[drew]

OK - William, you were right - the Dir[*.log].each statement only
returns a list of file names, not file handles, so I (and Daniel) was
simply searching the filenames, which of course wasn't returning any
results.

So I tried William's command-line versions, and they work, but I would
still like to know how to do the equivalent in a script. How do you
convert a list of filenames into actual file handles so you can open
them ? I guess I'll go and have a look in the Ruby docs.

So thanks William, your solutions worked; and Daniel, you made some
good suggestions (i.e. I was doing the wrong thing by zeroing the file
every time I opened a new input file, and it was nicer to write
directly to the file rather than into an array first), but you were
caught by the same assumption as I made - that Dir.each gives you
openable file handles, not just a directory listing !

Thanks guys,
Drew

[drew]

Found the extra bit that takes file names and reads them. Obviously,
it's File.read, and you just need to pop it in the right place ...

The answer (as many good answers do) came from Why's Poignant Guide,
Chapter 4, Part 3 (Chaining Delusions Together)
http://poignantguide.net/ruby/chapter-4.html

The key part of the script now reads:

def find_redir
File.open("found-redir-lines.txt", "w") do |out|

Dir['ex*.log'].each do |file_name|

# new bit below
file = File.read( file_name )

file.grep(/redir/) { |line| out.puts(line) }
# and can also be any of these ...
#file.each_line { |line| o.puts line if line["redir"] }
#file.each_line { |line| o.puts line if line =~ /redir/ }
end
end
end

So thanks people, thanks _why, and thanks Ruby !

drew

[Daniel Sheppard]

> "file" is a string, the name of a file. It isn't a file handle.
> You are not reading from a file.

eep. What he said. It seems I'll never learn to test my code before
posting...

def find_redir
File.open("found-redir-lines.txt", "w") do |out|

Dir['ex*.log'].each do |in|
File.open(in) { |f| f.grep(/redir/) { |line| out.puts(line) } }
end
end
end

> ruby -ne"print if /redir/" ex*.log >out.txt

Yeah, but that's cheating - you changed the behaviour. What if that
found-redir-lines.txt file was important for something else?

ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;a=$_;f.puts a;i+=1;puts a if i<2;end}" ex*.log
ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;a=$_;f<<a;i+=1;puts a if i<2;end}" ex*.log
ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;f<<$_;print if(i+=1)<2;end}" ex*.log
ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|f<<$_&&(print
if(i+=1)<2)if/redir/}" ex*.log

And I think that's about as much golf as I'm capable of playing.

[Daniel Sheppard]

Bah, stupid mail mangling programs.

ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;a=$_;f.puts a;i+=1;puts a if i<2;end}" ex*.log

ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;a=$_;f<<a;i+=1;puts a if i<2;end}" ex*.log

ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|if
/redir/;f<<$_;print if(i+=1)<2;end}" ex*.log

ruby -ne "i=0;open('found-redir-lines.txt','w'){|f|f<<$_&&(print
if(i+=1)<2)if/redir/}" ex*.log

ruby -ne
"i=0;open('found-redir-lines.txt','w'){|f|f<<$_&&(i+=1)<2&&print
if/redir/}" ex*.log

Is how those lines where meant to be read

And if they get mangled again... there'll be trouble.