Text Parser
ajocius
I admit it, I'm a hack. As a test engineer I'm faced with many
application problems. Most of the time, I take something that already
works, back-engineer it and use it. Now some of my colleagues swear that
TCL can be an excellent text parser. Seems like today we collect so much
data, but can't seem to manipulate like we would like. Can someone suggest
a TCL script that can parse comma separated data? I want to take this data
and arrange it so I can use it. Any examples would be fantastic. The data
looks something like this.
Socket, Time Stamp, Serial Number, Product Code, Date Stamp, test1,
results1, test2, results2...etc (over 300 tests and data)
the above repeats itself over 50,000 times per shift.
The file ends in a .dat
Tony
suchenwi
For heavy-duty CSV, with escaped commas, quotes, newlines and all
complications, there's a cvs module in Tcllib.
For simple cases, which yours seems to be, it can be done right out of
the box.
Imagine you read a line like this from a file:
(ru) 8 % set headerline {Socket,TimeStamp,this,that,foo,bar,grill}
Socket,TimeStamp,this,that,foo,bar,grill
Splitting on commas turns it into a list:
(ru) 9 % set headers [split $headerline ,]
Socket TimeStamp this that foo bar grill
Data lines also may come from a file:
(ru) 10 % set dataline {4711,11:55:59,hello,world,and,once,again}
4711,11:55:59,hello,world,and,once,again
Now with [foreach] you can assign the fields of the comma-split data
line to the headers:
(ru) 11 % foreach $headers [split $dataline ,] break
And each field can now be retrieved by a variable of its name:
(ru) 12 % set this
hello
(ru) 13 % set Socket
4711
(ru) 14 % set grill
again
Isn't it easy? :-)
Cameron Laird
Richard, from my reading of ajocius' description, I suspect he might
like a warm-up which reads a file:
set filename my_file.dat
set fp [open $filename]
set content [read $fp]
close $fp
foreach line [split $content \n] {
puts "One line is '$line'."
}
ajocius, do you understand what to do with these examples Richard and
I offer you? You might like to read <URL: http://wiki.tcl.tk/660 >.
Once we understand your background better, we can provide more precise
help.
suchenwi
After opening, read the first line for the headers, then all the other
lines for the content,
ajocius
Thank you for the advice and support. I believe I can work through
this. Keep checking, should have something in the next day or two. I'm not
much of a TCL programmer (actually not at all), but will give it a go. If
this works you'll have a convert in this group very quickly.
Tony
"Cameron Laird" <cla...@lairds.us> wrote in message
news:8aj5m3-...@lairds.us...
Jeff Godfrey
news:25CdnYVqy6EkiQ3Z...@insightbb.com...
> Can someone suggest a TCL script that can parse comma separated
> data? I want to take this data and arrange it so I can use it. Any
> examples would be fantastic. The data looks something like this.
Hi Tony,
As others have already stated, Tcl is more than up to the task of
manipulating text-based data in any way you might see fit. That said
though, as you are specifically interested in manipulating CSV data,
there is a wonderful freeware (actually cardware) tool called "CSVed"
that's designed to do just that. It loads CSV files into an on-screen
"grid" and then allows you to manipulate the data much like a typical
database application.
I'm definitely not trying to steer you away from Tcl for your task as
that will be time well-spent, but CSVed might at least be good to have
in your toolbox. I don't think you mentioned your platform, but CSVed
is a Windows application.
You can read about it and download it from here:
http://home.hccnet.nl/s.j.francke/t2t/text2table.htm
Jeff
ajocius
I checked the CVS app and it appears to be a manual operation. It has
lots of functionality, but I didn't see where I can open a .dat file and
automatically manipulate columns. Am I wrong?
Tony
"Jeff Godfrey" <jeff_g...@pobox.com> wrote in message
news:K_Akg.6585$o4....@newsread2.news.pas.earthlink.net...
ajocius
Well I got a book, titled Graphical Applications Tcl &Tk by Eric
Foster - Johnson. I read the first 100 or so pages and am working to better
understand TCL. I downloaded TCL from a URL >>> www.activestate.com . So
far so good. Could send you a sample of the .dat file. My address is
ajo...@insightbb.com . Can you recommend a text for learning Tcl? The
book I have was published in 1997, so I don't know how relavant the material
is with todays TCL engine.
Tony
"Cameron Laird" <cla...@lairds.us> wrote in message
news:8aj5m3-...@lairds.us...
Cameron Laird
ajocius <ajo...@insightbb.com> wrote:
>Cameron/Richard,
> Well I got a book, titled Graphical Applications Tcl &Tk by Eric
>Foster - Johnson. I read the first 100 or so pages and am working to better
>understand TCL. I downloaded TCL from a URL >>> www.activestate.com . So
>far so good. Could send you a sample of the .dat file. My address is
>ajo...@insightbb.com . Can you recommend a text for learning Tcl? The
>book I have was published in 1997, so I don't know how relavant the material
>is with todays TCL engine.
.
.
I think Eric's book <URL: http://wiki.tcl.tk/113 > will serve you.
It's almost unique among Tcl books <URL: http://wiki.tcl.tk/book >
for its emphasis on Windows, rather than Unix or cross?platform
programming, and also on Tk rather than Tcl. Also, it's almost ten
years old, almost hilariously ancient in a software world where
books regularly fall out of favor in a matter of *months*.
However, the Windows focus might serve you well, Tcl respects
cross-version compatibility to an extent other languages make hard
to imagine, and I like *Graphical Applications ...*--more so, in
fact, than probably any other person likely to post here.
So: what you learn from *Graphical Applications ...* remains utterly
relevant to today's Tcl engine. Its sins are almost entirely those
of omission. It won't tell you, for example, about tcllib <URL:
http://wiki.tcl.tk/tcllib >, which includes the csv module <URL:
http://wiki.tcl.tk/csv > recommended to you.
www.activestate.com packages Tcl in several forms; without more de-
tails, it's hard to know which you chose and installed for yourself.
Many of use learned Tcl wholly or largely from on-line materials,
including the ActiveTcl distribution documentation <URL:
http://ASPN.ActiveState.com/ASPN/docs/ActiveTcl/8.4/at.pkg_index.html >,
tutorials <URL: http://wiki.tcl.tk/1304 >, and so on.
I would think that a five-line sample of your .dat might have enough
content to be worth sharing, while also small enough to fit this
forum well.
ajocius
It seems as the REAL tcl programmers learned their craft from an on-line
source. A colleague of mine, Laszlo who is a UNIX programmer in
Szombathely, Hungary never picked up a TCL book, but can write some
wonderful scripts used on our FIS (factory information system). I'll send
him an e-mail and tell him I'm finally taking his advice on TCL. He'll be
very happy. The links you gave me are stocked full of goodies.
Instead of 5 lines I'll drop in 3 because of the length of each record.
The structure is as follows:
Socket Number, Barcode, Time-stamp, Date Stamp, Test-Time, test1, results1,
test2, results2....testn, resultn, (Pass=1/Fail=0). My intent is to put a
heading in row1, Element 1=Socket, Element 2=Barcode, Element3=Time-Stamp,
Element4=Date-Stamp, Element 5=Test Time.....then Broadcast Code OK T/F,
Char 1 - Label, Char 2 - Label, Char 7 - Label, ....etc, until the last
element P/F. Finally placing data in rows 2 thru nth, under its
respective column header.
Tony
2,89FDVWM161500325,5:07:14 PM,05-31-2006,459.7664,Broadcast Code OK
T/F,"True",Char 1 - Label,8,Char 2 - Label,9,Char 7 - Label,"M",Char 8 -
Label,1,Char 9 - Label,6,Char 10, 11, 12 - Label,150,Char 13 - Label,0,Char
14 - Label,0,Char 15 - Label,0,Char 16 - Label,0,Top Barcode 22 Chars,P,Char
14 = Num 14,0,Char 15 = Num 15,6,Char 16 = Num 16,0,Char 17 = Num 17,5,Char
18 = Num 18,3,Char 19 = Num 19,0,Char 20 = Num 20,0,Char 21 = Num 21,0,Char
22 = Num 22,1,Cust PN OK ?,"Part Num OK",B2 to B3 Ground,6.1,Diag Fail
Test,1,Software ID,"AA0448AA09",Amadeus Flash,"AA0384AA10",Flash
Checksum,"95A5",Amadeus Upgrade,"AA0384AB10",Read Memory 1,"39363131",Read
Memory 2,"30334536",Read Memory 3,"3030",Keyboard light
test,1,POWER,"04",TRACK DOWN,"08",SEEK UP ^,"10",EJECT ^,"02",AM
Button,"80",FM1 button,"01",FM2 button,"02",4 CD button,"04",5
SCAN,"08",EQ,"08",DIR ^,"01",DIR Down,"80",Audio,"40",Preset 6,"40",Preset
5,"20",Preset 4,"10",Preset 3,"01",Preset 2,"02",Preset 1,"04",<<
Button,"01",>> Button,"02",RPT Button,"04",RDM button,"08",MARK
button,"10",INFO button,"20",DE Diag Mode,"076171",DE Diag Mode,"076171",VOL
MAX Status,255,DE Diag Mode,"076171",DE Diag Mode,"076171",VOL MIN
Status,0,LF Track 3 dB,0.508,RF Track 3 dB,0.510,LF SINAD Track 3,61.891,RF
SINAD Track 3,61.659,LF 40 Hz tone dB,2.294,RF 40 Hz tone dB,2.246,LF 10K
tone dB,-0.175,RF 10K tone dB,-0.153,LF 20K tone dB,-1.756,RF 20K tone
dB,-1.696,LF Track 11 dB,0.017,RF Track 11 dB,-55.310,LF FADE Front
dB,0.954,RF FADE Front dB,0.954,LR FADE Front dB,-51.041,RR FADE Front
dB,-50.891,LF FADE Rear dB,-50.865,RF FADE Rear dB,-51.010,LR FADE Rear
db,0.930,RR FADE Rear dB,0.947,LF MAX Vol dB,17.039,RF MAX Vol dB,17.039,LR
MAX Vol dB,17.019,RR MAX Vol dB,17.036,Get Band Freq,"62",LF FM High
dB,-6.069,RF FM High dB,-47.934,LF SINAD FM High,43.133,FM Level At FM
High,54,LF dB AM HIGH,-1.424,RF dB AM HIGH,-1.422,LF Freq AM HIGH,984,LF
SINAD AM HIGH,47.518,RF SINAD AM HIGH,47.527,AM Level At AM HIGH,40,Get Band
Freq -Stopping,"76",AM Level At AM LOW,49,LF dB AM LOW,-0.471,RF dB AM
LOW,-0,LF SINAD AM LOW,44.128,RF SINAD AM LOW,44.317,Vol Step 17,"7C",SWC
Volume Change,"00",Vdc At B6,12.800,Vdc at B12,9.518,Vdc at A12,4.659,Write
Component To Memory,"61C21001",Write Broadcast Code To
Memory,"61C41001",Write PML To Memory,"61C11001",Write Shift To
Memory,"61C11001",Write Year To Memory,"61C11001",Write Julian Date To
Memory,"61C31001",Write Seq Number To Memory,"61C41001",Verify EEPROM,0,Read
Stby current,0.739,Enable volt OFF,-0.06,1
1,89FDVWM161510380,5:07:34 PM,05-31-2006,480.6483,Broadcast Code OK
T/F,"True",Char 1 - Label,8,Char 2 - Label,9,Char 7 - Label,"M",Char 8 -
Label,1,Char 9 - Label,6,Char 10, 11, 12 - Label,151,Char 13 - Label,0,Char
14 - Label,0,Char 15 - Label,0,Char 16 - Label,0,Top Barcode 22 Chars,P,Char
14 = Num 14,0,Char 15 = Num 15,6,Char 16 = Num 16,0,Char 17 = Num 17,5,Char
18 = Num 18,3,Char 19 = Num 19,1,Char 20 = Num 20,0,Char 21 = Num 21,0,Char
22 = Num 22,1,Cust PN OK ?,"Part Num OK",B2 to B3 Ground,15.2,Diag Fail
Test,1,Software ID,"AA0448AA09",Amadeus Flash,"AA0384AA10",Flash
Checksum,"95A5",Amadeus Upgrade,"AA0384AB10",Read Memory 1,"39363131",Read
Memory 2,"30334536",Read Memory 3,"3030",Keyboard light
test,1,POWER,"04",TRACK DOWN,"08",SEEK UP ^,"10",EJECT ^,"02",AM
Button,"80",FM1 button,"01",FM2 button,"02",4 CD button,"04",5
SCAN,"08",EQ,"08",DIR ^,"01",DIR Down,"80",Audio,"40",Preset 6,"40",Preset
5,"20",Preset 4,"10",Preset 3,"01",Preset 2,"02",Preset 1,"04",<<
Button,"01",>> Button,"02",RPT Button,"04",RDM button,"08",MARK
button,"10",INFO button,"20",DE Diag Mode,"076171",DE Diag Mode,"076171",VOL
MAX Status,255,DE Diag Mode,"076171",DE Diag Mode,"076171",VOL MIN
Status,0,LF Track 3 dB,0.465,RF Track 3 dB,0.456,LF SINAD Track 3,60.149,RF
SINAD Track 3,44.641,LF 40 Hz tone dB,2.256,RF 40 Hz tone dB,2.211,LF 10K
tone dB,-0.248,RF 10K tone dB,-0.249,LF 20K tone dB,-1.891,RF 20K tone
dB,-1.858,LF Track 11 dB,-0.022,RF Track 11 dB,-38.904,LF FADE Front
dB,0.901,RF FADE Front dB,0.886,LR FADE Front dB,-50.344,RR FADE Front
dB,-50.323,LF FADE Rear dB,-50.335,RF FADE Rear dB,-50.327,LR FADE Rear
db,0.902,RR FADE Rear dB,0.893,LF MAX Vol dB,16.996,RF MAX Vol dB,16.975,LR
MAX Vol dB,16.991,RR MAX Vol dB,16.982,Get Band Freq,"62",LF FM High
dB,-6.087,RF FM High dB,-48.411,LF SINAD FM High,42.855,FM Level At FM
High,55,LF dB AM HIGH,-1.548,RF dB AM HIGH,-1.569,LF Freq AM HIGH,984,LF
SINAD AM HIGH,47.513,RF SINAD AM HIGH,47.505,AM Level At AM HIGH,41,Get Band
Freq -Stopping,"76",AM Level At AM LOW,49,LF dB AM LOW,-0.435,RF dB AM
LOW,-0,LF SINAD AM LOW,43.577,RF SINAD AM LOW,39.413,Vol Step 17,"7C",SWC
Volume Change,"00",Vdc At B6,12.746,Vdc at B12,9.497,Vdc at A12,4.659,Write
Component To Memory,"61C21001",Write Broadcast Code To
Memory,"61C41001",Write PML To Memory,"61C11001",Write Shift To
Memory,"61C11001",Write Year To Memory,"61C11001",Write Julian Date To
Memory,"61C31001",Write Seq Number To Memory,"61C41001",Verify EEPROM,0,Read
Stby current,0.717,Enable volt OFF,-0.04,1
5,89FDVWM161510395,5:08:07 PM,05-31-2006,583.7165,Broadcast Code OK
T/F,"True",Char 1 - Label,8,Char 2 - Label,9,Char 7 - Label,"M",Char 8 -
Label,1,Char 9 - Label,6,Char 10, 11, 12 - Label,151,Char 13 - Label,0,Char
14 - Label,0,Char 15 - Label,0,Char 16 - Label,0,Top Barcode 22 Chars,P,Char
14 = Num 14,0,Char 15 = Num 15,6,Char 16 = Num 16,0,Char 17 = Num 17,5,Char
18 = Num 18,3,Char 19 = Num 19,1,Char 20 = Num 20,0,Char 21 = Num 21,0,Char
22 = Num 22,1,Cust PN OK ?,"Part Num OK",B2 to B3 Ground,6.0,Diag Fail
Test,1,Software ID,"AA0448AA09",Amadeus Flash,"AA0384AA10",Flash
Checksum,"95A5",Amadeus Upgrade,"AA0384AB10",Read Memory 1,"39363131",Read
Memory 2,"30334536",Read Memory 3,"3030",Keyboard light
test,1,POWER,"04",TRACK DOWN,"08",SEEK UP ^,"10",EJECT ^,"02",AM
Button,"80",FM1 button,"01",FM2 button,"02",4 CD button,"04",5
SCAN,"08",EQ,"08",DIR ^,"01",DIR Down,"80",Audio,"40",Preset 6,"40",Preset
5,"20",Preset 4,"10",Preset 3,"01",Preset 2,"02",Preset 1,"04",<<
Button,"01",>> Button,"02",RPT Button,"04",RDM button,"08",MARK
button,"10",INFO button,"20",DE Diag Mode,"076171",DE Diag Mode,"076171",VOL
MAX Status,255,DE Diag Mode,"076171",DE Diag Mode,"076171",VOL MIN
Status,0,LF Track 3 dB,0.724,RF Track 3 dB,0.749,LF SINAD Track 3,60.396,RF
SINAD Track 3,60.286,LF 40 Hz tone dB,2.513,RF 40 Hz tone dB,2.505,LF 10K
tone dB,-0.031,RF 10K tone dB,0.056,LF 20K tone dB,-1.775,RF 20K tone
dB,-1.571,LF Track 11 dB,0.236,RF Track 11 dB,-55.144,LF FADE Front
dB,1.164,RF FADE Front dB,1.187,LR FADE Front dB,-50.469,RR FADE Front
dB,-50.484,LF FADE Rear dB,-50.414,RF FADE Rear dB,-50.328,LR FADE Rear
db,1.161,RR FADE Rear dB,1.160,LF MAX Vol dB,17.252,RF MAX Vol dB,17.273,LR
MAX Vol dB,17.249,RR MAX Vol dB,17.248,Get Band Freq,"62",LF FM High
dB,-5.857,RF FM High dB,-46.502,LF SINAD FM High,41.544,FM Level At FM
High,55,LF dB AM HIGH,-1.508,RF dB AM HIGH,-1.484,LF Freq AM HIGH,985,LF
SINAD AM HIGH,48.617,RF SINAD AM HIGH,48.467,AM Level At AM HIGH,40,Get Band
Freq -Stopping,"76",AM Level At AM LOW,49,LF dB AM LOW,-0.347,RF dB AM
LOW,-0,LF SINAD AM LOW,44.261,RF SINAD AM LOW,44.326,Vol Step 17,"7C",SWC
Volume Change,"00",Vdc At B6,12.785,Vdc at B12,9.508,Vdc at A12,4.662,Write
Component To Memory,"61C21001",Write Broadcast Code To
Memory,"61C41001",Write PML To Memory,"61C11001",Write Shift To
Memory,"61C11001",Write Year To Memory,"61C11001",Write Julian Date To
Memory,"61C31001",Write Seq Number To Memory,"61C41001",Verify EEPROM,0,Read
Stby current,0.759,Enable volt OFF,-0.05,1
Note that the last number, 0=Fail or 1=Pass indicates whether a
product passed its tests.
"Cameron Laird" <cla...@lairds.us> wrote in message
news:gq4gm3-...@lairds.us...
Cameron Laird
.
.
I'm not sure where the boundary is in what I'm seeing between
data, and artifact-of-posting. Does a data record embed new-
lines, as the ones here appear to do? Does a data record
always begin with five blank characters?
I bet we can parse and process all this stuff more easily
than you imagine.
Your mention of FIS certainly interests me; most of my favorite
uses of Tcl are in industrial control or process control or
related areas.
ajocius
that 5 black spaces preceded a socket. Can't explain this. The number used
in the begining can be an integer from 0 to 15. I'm using National
Instruments TestStand to get the data on an audio tester. We are using
multi-threaded features of TestStand.
Tony
"Cameron Laird" <cla...@lairds.us> wrote in message
news:hihgm3-...@lairds.us...
joramo
for CSV parsing, you can use LogParser, see a GUI made in Tcl for that
application: http://mini.net/tcl/15440
Jorge.
Cameron Laird
ajocius <ajo...@insightbb.com> wrote:
>I sent you a file of 33 records. I didn't know/pay attention to the fact
>that 5 black spaces preceded a socket. Can't explain this. The number used
>in the begining can be an integer from 0 to 15. I'm using National
>Instruments TestStand to get the data on an audio tester. We are using
>multi-threaded features of TestStand.
.
.
I've lost track of what you want. When I run this program
set fn _903523May06.dat
set fp [open $fn]
set content [read $fp]
close $fp
set lines [split $content \n]
set line_number 0
foreach line $lines {
# Splitting on \n leaves an empty line at the end.
if {![string length $line]} continue
set parts [split $line ,]
catch {unset elements}
# Skip over timestamp and such.
array set elements [lrange $parts 5 end-3]
puts "Data from line #$line_number:"
parray elements
incr line_number
}
against the log you e-mailed me, I get a display that to
me looks at least suggestive.
ajocius
Whoa, thats was awesome. I can't believe your program did all that it
did with just about 15 lines. I have to spend some time picking this apart.
Sorry for the confusing requests. I'm like a kid in a candy store. Every
evening for the past week or so, been spending some time with TCL. What
blows me away is so few lines and so much movement of the data. I see all
kinds of uses for this tool. Wanted to thank you for your effort and hard
work. Does TCL or TK have a newsletter somewhere or magazine? Want more of
this stuff. Thank you, Thank you very much.
Tony
"Cameron Laird" <cla...@lairds.us> wrote in message
news:7iqlm3-...@lairds.us...
Michael A. Cleverly
> Whoa, thats was awesome. I can't believe your program did all that it
> did with just about 15 lines. I have to spend some time picking this apart.
> Sorry for the confusing requests. I'm like a kid in a candy store. Every
> evening for the past week or so, been spending some time with TCL. What
> blows me away is so few lines and so much movement of the data. I see all
> kinds of uses for this tool. Wanted to thank you for your effort and hard
> work. Does TCL or TK have a newsletter somewhere or magazine? Want more of
> this stuff. Thank you, Thank you very much.
Take a look at Bryan Oakley's tclscripting.com website. If you're feeling
like a kid in a candy store with comp.lang.tcl, visit the Tcl'ers Wiki
(http://wiki.tcl.tk) and you'll be like a kid on Christmas morning... :-)
Michael
Cameron Laird
ajocius <ajo...@insightbb.com> wrote:
>Cameron,
> Whoa, thats was awesome. I can't believe your program did all that it
>did with just about 15 lines. I have to spend some time picking this apart.
>Sorry for the confusing requests. I'm like a kid in a candy store. Every
>evening for the past week or so, been spending some time with TCL. What
>blows me away is so few lines and so much movement of the data. I see all
>kinds of uses for this tool. Wanted to thank you for your effort and hard
>work. Does TCL or TK have a newsletter somewhere or magazine? Want more of
>this stuff. Thank you, Thank you very much.
.
.
You're welcome. Yes, the local ethos puts a lot of emphasis on
getting big results with simple means; I'm pleased that the
example at hand strikes you that way.
While we've tried various things in the past, the current most
consistent periodic outlets are Bryan Oakley's Tcl Scripting
<URL: http://www.tclscripting.com/ > and "Tcl-URL!" <URL:
http://wiki.tcl.tk/228 >. Given just a little time, we'll over-
whelm you with other nuggets of insight and excitement (the Chat,
the Wiki, ...).
I should mention that we try to make "Regular Expressions" <URL:
http://regularexpressions.com/ > readable by a general audience,
and often mention Tcl.
There are indeed ALL kinds of uses for Tcl.
Schelte Bron
> # Splitting on \n leaves an empty line at the end.
You won't get the empty line at the end if you read the file with
the -nonewline option:
set content [read -nonewline $fp]
Schelte
--
set Reply-To [string map {nospam schelte} $header(From)]