Fwd: ShelveLogger 4.0 Transition Notes. [updated]
Silver Wav
Date: Thu, Feb 18, 2010 at 7:39 PM
Subject: ShelveLogger 4.0 Transition Notes. [updated]
To: Shelve Firefox Add-on <shelve-fir...@googlegroups.com>
The next version of ShelveLogger will incorporate some great new
features (thanks to the developer of Shelve).
Note1: If your are new, ignore this thread and just use the new
how-to: "ShelveLogger QuickStart version 4.0 " (to be posted later
this week).
Note2: The easy option is to use the new ShelveLogger.html and shelve
templates provided, rename the old one to ShelveLogger3.html and
reference it while shelving to the new one.
But if you have been using ShelveLogger.html and want to continue to
access your old data in version 4.0... read on.
I have 5 years of data in ShelveLogger.html so I needed a way to
update the file format from the old to the new, here is how I did it.
---
Made a copy of your old ShelveLogger.html just in case :)
If you check the screen shots1 and 2, you can see the old and new formats.
The main issue is changing the records from the present which has 1
table row (normally and 1 more for each clip or note) to the new
format in which the whole record has only 1 table row (the clips etc.,
are incorporated as in one table cell along with the title.
--- Old
The old quick and dirty way of doing this was to use a text editor and
find/replace, plus importing into a spreadsheet to move the columns of
data around. Indeed that's exactly what I did last time when moving to
Shelve from Slogger:
Quote:
"I had 4.5 years of Slogged data in one large xml file, so I first
converted it to a format that Shelve was happy with, I saved the whole
thing in the html complete format. I used a text editor to clean up
the data part of the log and chop it into records starting with the
date and ending with a newline (I removed any other newlines from the
data first). This allowed me to import the records into a spreadsheet
and re-sort them into descending date order."
This will work but its painful and there is a better way...
--- New
The following posts will detail how to manage the format changes with
a lot more ease using awk.
I will post my solution to that next but the basics are as follows
Terminal:
silverwave@blue:~/data/Shelve$ awk -f sw1.awk body.html
---
Where:
body.html - is a file with just your table data no header info.
sw1.awk - is the awk script file:
BEGIN {
FS="\n"
RS=""
}
{
#print $1 ", " $2 ", " $3
print $0
}
---
I did use a text editor to add a new blank line between each record.
Add blank line between records:
Find: <tr>
Reblace: \n<tr>
---
I have selected awk as it will will allow complete control over any
substitutions needed and can reference different fields of the record
with ease.
Details to follow.
Silver Wav
Date: Sun, Feb 21, 2010 at 12:24 AM
Subject: Re: [shelve] Re: ShelveLogger 4.0 Transition Notes. [updated]
To: Shelve Firefox Add-on <shelve-fir...@googlegroups.com>
Run script clean up - it will print 5 and 6 field records from
body.html to body2.html.
Check you have the same number of lines in each.
If not you need to identify the problem records and clean them up.
#Note: Search in body2.html for NFgt6 and NFlt5, these are printed if
awk found records with greater that 6 newlines, or less than 5
newlines.
Usage:
---
Terminal:
silv@blue:~/data/Shelve$ awk -f sw1cleanup.awk body.html >body2.html
---
Where:
body.html - The file with your table data (no header info).
sw1cleanup.awk - The awk script file:
body2.html - The new file created.
---
On Sun, Feb 21, 2010 at 12:14 AM, Silver Wav <silv...@googlemail.com> wrote:
> The New version 4.0 of ShelveLogger is attached.
> I have some new streamlined awk scripts to aid the transition from 3.0
> to 4.0, to be posted soon.
>
>
> On Sun, Feb 21, 2010 at 12:04 AM, SilverWave <silv...@googlemail.com> wrote:
>> After extensive testing I am in the process of updating this thread.
>> Details follow:
>>
>> ShelveLogger Version 4.0 (for shelve 1.22+)
>> by silverwav - with filtering and better JavaScript care of the Shelve
>> Developer.
>> http://groups.google.com/group/shelve-firefox-addon/
>>
>> Use with this template:
>>
>>
>> <tr>
>> <td class="date" nowrap="nowrap">%Y-%M-%D %h:%m</td>
>> <td class="description">
>> <div class="title">%[tcf]</div>
>> <div class="sup" class="clip">%{clip?}</div>
>> <span class="sup" class="host"> [%{hostbasename?}] </span>
>> <span class="sup" class="keyw">Keywords: %{keywords?}</span>
>> <div class="sup" class="note">%{note?}</div>
>> </td><td class="links">
>> <a class="url" href="%{url?}">url</a>
>> <a class="local" href="file://%o">local</a></td></tr>
>>> 1.Screenshot-Old-format.png
>>> 77KViewDownload
>>>
>>> 2.Screenshot-New-format.png
>>> 103KViewDownload
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Shelve Firefox Add-on" group.
>> To post to this group, send email to shelve-fir...@googlegroups.com.
>> To unsubscribe from this group, send email to shelve-firefox-a...@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/shelve-firefox-addon?hl=en.
>>
>>
>
Silver Wav
Date: Sun, Feb 21, 2010 at 12:39 AM
Subject: Re: [shelve] Re: ShelveLogger 4.0 Transition Notes. [updated]
To: Shelve Firefox Add-on <shelve-fir...@googlegroups.com>
A sceenshot of the new Version 4.0 format is attached
Silver Wav
Date: Mon, Feb 22, 2010 at 4:57 PM
Subject: Re: [shelve] Re: ShelveLogger 4.0 Transition Notes. [updated]
To: Shelve Firefox Add-on <shelve-fir...@googlegroups.com>
The New version 4.2 of ShelveLogger is attached.
Just click to view and to test the the new filtering etc..
Try ticking all the boxes and these searches: bbc; dist; 23:11; 935;
http:..n; http:......b;
This is very cool, very powerful :D
---
I have added a new JavaScript function to enable multiple classes to
be used per element.
This was the last outstanding Todo item which impacted the final file format.
Now that I know the final features and file format to aim for, we are
good to go...
...next up, the new awk scripts to transform ShelveLogger3 data to the
new ShelveLogger4 format.
Cheers.
Silver Wav
Date: Tue, Feb 23, 2010 at 7:29 PM
Subject: [shelve] Re: ShelveLogger 4.0 Transition Notes. [updated]
To: Shelve Firefox Add-on <shelve-fir...@googlegroups.com>
sw2.awk (ShelveLogger 4.2 Transition script).
Usage:
---
Terminal:
silv@blue:~/data/Shelve$ awk -f sw2.awk body.html >body2.html
---
Where:
body.html - The file with your table data (no header info).
sw2.awk - The awk script file:
body2.html - The new file created.
code:
---
BEGIN {
FS="\n"
RS=""
}
{
#sw1.awk - Check data
# 1st run sw1.awk -
# This will print 5 and 6 field records from body.html to body2.html.
# Check you have the same number of lines in each.
# If not you need to identify the problem records and clean them up
(or adjust the sw2.awk script to take account of them).
# Note: Search in body2.html for NFgt6 and NFlt5, these are printed
if awk found records with greater that 6 newlines, or less than 5
newlines.
#sw2.awk - Transform data
# 2nd run sw2.awk -
# This will print 5 and 6 field records from body.html to body2.html
and alter them to the new format.
# Transform ShelveLogger data from version 3.0 to 4.2
# After the update you will need to use the new 4.2 ShelveLogger
template for Shelve 1.22+ to save new records correctly.
if(NF == 5)
{
print ""
# tr
if(length($1) > 0) print $1
# date
if(length($2) > 0)
{
swsdate = substr($2, 4, 30)
swstime = substr($2, 32, 2)
#print $2
print "<td class=\"date\""swsdate":"swstime"</td>"
}
# description
print "<td class=\"description\">"
# Title
if(length($3) > 0)
{
swl = length($3)
sws = substr($3, 4, swl-8)
#print $3
print "<div"sws"</div>"
}
# host
sub(/www./,"",$4)
split($4, a, "/")
#print $4
#a[3]
print "<span class=\"sup host\"> ["a[3]"] </span>"
# links
print "</td><td class=\"links\">"
# url
if(length($4) > 0)
{
swl = length($4)
sws = substr($4, 7, swl-11)
#print $4
print "<a class=\"url\""sws
}
# Local
if(length($5) > 0)
{
swl = length($5)
sws = substr($5, 7, swl)
#print $5
print "<a class=\"local\""sws
}
}
if(NF == 6)
{
print ""
# tr
if(length($1) > 0) print $1
# date
if(length($2) > 0)
{
swsdate = substr($2, 4, 30)
swstime = substr($2, 32, 2)
#print $2
print "<td class=\"date\""swsdate":"swstime"</td>"
}
# description
print "<td class=\"description\">"
# Title
if(length($3) > 0)
{
swl = length($3)
sws = substr($3, 4, swl-8)
#print $3
print "<div"sws"</div>"
}
# clip
if(length($6) > 0)
{
swl = length($6)
sws = substr($6, 59, swl-68)
#print $6
print "<div class=\"sup clip\""sws"</div>"
}
# host
sub(/www./,"",$4)
split($4, a, "/")
#print $4
#a[3]
print "<span class=\"sup host\"> ["a[3]"] </span>"
# links
print "</td><td class=\"links\">"
# url
if(length($4) > 0)
{
swl = length($4)
sws = substr($4, 7, swl-11)
#print $4
print "<a class=\"url\""sws
}
# Local
if(length($5) > 0)
{
swl = length($5)
sws = substr($5, 7, swl)
#print $5
print "<a class=\"local\""sws
}
}
}
---
On Feb 22, 4:57 pm, Silver Wav <silver...@googlemail.com> wrote:
> The New version 4.2 of ShelveLogger is attached.
> Just click to view and to test the the new filtering etc..
>
> Try ticking all the boxes and these searches: bbc; dist; 23:11; 935;
> http:..n; http:......b;
> This is very cool, very powerful :D
> ---
> I have added a new JavaScript function to enable multiple classes to
> be used per element.
> This was the last outstanding Todo item which impacted the final file format.
>
> Now that I know the final features and file format to aim for, we are
> good to go...
> ...next up, the new awk scripts to transform ShelveLogger3 data to the
> new ShelveLogger4 format.
>
> Cheers.
>
> On Sun, Feb 21, 2010 at 12:39 AM, Silver Wav <silver...@googlemail.com> wrote:
> > A sceenshot of the new Version 4.0 format is attached
>
> > On Sun, Feb 21, 2010 at 12:24 AM, Silver Wav <silver...@googlemail.com> wrote:
> >> Run script clean up - it will print 5 and 6 field records from
> >> body.html to body2.html.
> >> Check you have the same number of lines in each.
> >> If not you need to identify the problem records and clean them up.
> >> #Note: Search in body2.html for NFgt6 and NFlt5, these are printed if
> >> awk found records with greater that 6 newlines, or less than 5
> >> newlines.
>
> >> Usage:
>
> >> ---
> >> Terminal:
> >> silv@blue:~/data/Shelve$ awk -f sw1cleanup.awk body.html >body2.html
>
> >> ---
> >> Where:
> >> body.html - The file with your table data (no header info).
> >> sw1cleanup.awk - The awk script file:
> >> body2.html - The new file created.
> >> ---
>
> >> On Sun, Feb 21, 2010 at 12:14 AM, Silver Wav <silver...@googlemail.com> wrote:
> >>> The New version 4.0 of ShelveLogger is attached.
> >>> I have some new streamlined awk scripts to aid the transition from 3.0
> >>> to 4.0, to be posted soon.
>
> ShelveLogger.html
> 19KViewDownload
>
> ShelveLogger4.2.png
> 227KViewDownload