Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Problem parsing HTML

11 views
Skip to first unread message

Ninja Li

unread,
Nov 24, 2009, 11:42:12 AM11/24/09
to
Hi,

I am trying to parse HTML from the website http://biz.yahoo.com/c/e.html
using HTML::TreeBuilder module and generate a comma-delimited file.
However, I am getting an extra "," at the first line and I would also
like to get rid of the "," at the end of the each line.

Please advise why that happens and the fix. The code is at the end
of the post.

Thanks in advance.

Nick
-----------------------------------------------
Soure code:

use strict;
use LWP::Simple;
use HTML::Tree;
use warnings;

my $url = 'http://biz.yahoo.com/c/e.html';
my $content = get($url);

my $tree = HTML::TreeBuilder->new_from_content($content);

my @tr = $tree->look_down('_tag' => 'tr',
sub { $_[0]->as_text !~ m/My Yahoo!/ &&
$_[0]->as_text !~ m/Welcome/i &&
$_[0]->as_text !~ m/Economic Calendar/i
&&
$_[0]->as_text !~ m/Last Week/i });

foreach my $tr (@tr)
{
if ($tr)
{
my @detail = $tr->look_down('_tag' => 'td');

foreach my $detail (@detail)
{
print $detail->as_text . ",";
}
print "\n";
}
else
{
warn "No detail data";
}
}

$tree->delete;

Dan Rumney

unread,
Nov 24, 2009, 12:29:44 PM11/24/09
to

> my @detail = $tr->look_down('_tag' => 'td');
>
> foreach my $detail (@detail)
> {
> print $detail->as_text . ",";
> }
> print "\n";

There's your problem.

For every element in @detail, you print that element, plus a comma.
You've programmed it to put a comma after every $detail, so you're going
to get a comma after the last $detail on each line.

Try

> my @detail = $tr->look_down('_tag' => 'td');

> print join(',',@detail)."\n";


Dan

Ninja Li

unread,
Nov 24, 2009, 1:25:36 PM11/24/09
to

Dan,

I tried your solution but I got error messages.

Thanks.

Nick

J�rgen Exner

unread,
Nov 24, 2009, 1:46:32 PM11/24/09
to
Ninja Li <nickl...@gmail.com> wrote:
>On Nov 24, 12:29�pm, Dan Rumney <n...@danrumney.co.uk> wrote:
>> > � � � � � �my @detail = $tr->look_down('_tag' => 'td');
>>
>> > � � � � � �foreach my $detail (@detail)
>> > � � � � � �{
>> > � � � � � � �print $detail->as_text . ",";
>> > � � � � � �}
>> > � � � � � �print "\n";
>>
>> There's your problem.
>>
>> For every element in @detail, you print that element, plus a comma.
>> You've programmed it to put a comma after every $detail, so you're going
>> to get a comma after the last $detail on each line.
>>
>> Try
>>
>> �> � � � � � my @detail = $tr->look_down('_tag' => 'td');
>> �> � � � � � print join(',',@detail)."\n";
>
> I tried your solution but I got error messages.

And? Are you keeping those error messages a secret? Kind of difficult to
correct code without knowing the code _and_ the error messages.

jue

Ninja Li

unread,
Nov 24, 2009, 2:13:02 PM11/24/09
to
On Nov 24, 1:46 pm, Jürgen Exner <jurge...@hotmail.com> wrote:
> jue- Hide quoted text -
>
> - Show quoted text -

Jurgen,

Thanks for pointing this out. Here is error message after code
change. The new code is at the end of the post:

Thanks.

------------------------------------
HTML::Element=HASH(0xdd3228)
HTML::Element=HASH(0xdee864),HTML::Element=HASH
(0xdee924),HTML::Element=HASH(0xdee9a8),HTML::Element=HASH
(0xdeea44),HTML::Element=HASH(0xdeeb04),HTML::Element=HASH
(0xdeebac),HTML::Element=HASH(0xdeec54),HTML::Element=HASH
(0xdf2bd8),HTML::Element=HASH(0xdf2c80)
HTML::Element=HASH(0xdf2d58),HTML::Element=HASH
(0xdf2dd0),HTML::Element=HASH(0xdf2e0c),HTML::Element=HASH
(0xdf2eb4),HTML::Element=HASH(0xdf2f2c),HTML::Element=HASH
(0xdf2f8c),HTML::Element=HASH(0xdf2fec),HTML::Element=HASH
(0xdf304c),HTML::Element=HASH(0xdf30ac)
HTML::Element=HASH(0xdf313c),HTML::Element=HASH
(0xdf31b4),HTML::Element=HASH(0xdf31f0),HTML::Element=HASH
(0xdf32a4),HTML::Element=HASH(0xdf331c),HTML::Element=HASH
(0xdf337c),HTML::Element=HASH(0xdf33dc),HTML::Element=HASH
(0xdf343c),HTML::Element=HASH(0xdf349c)
HTML::Element=HASH(0xdf352c),HTML::Element=HASH
(0xdf35a4),HTML::Element=HASH(0xdf35e0),HTML::Element=HASH
(0xdf3694),HTML::Element=HASH(0xdf370c),HTML::Element=HASH
(0xdf376c),HTML::Element=HASH(0xdf37cc),HTML::Element=HASH
(0xdf382c),HTML::Element=HASH(0xdf388c)
HTML::Element=HASH(0xdf391c),HTML::Element=HASH
(0xdf3994),HTML::Element=HASH(0xdf39d0),HTML::Element=HASH
(0xdf3a84),HTML::Element=HASH(0xdf3afc),HTML::Element=HASH
(0xdf6480),HTML::Element=HASH(0xdf64e0),HTML::Element=HASH
(0xdf6540),HTML::Element=HASH(0xdf65a0)
HTML::Element=HASH(0xdf6630),HTML::Element=HASH
(0xdf66a8),HTML::Element=HASH(0xdf66e4),HTML::Element=HASH
(0xdf6798),HTML::Element=HASH(0xdf6810),HTML::Element=HASH
(0xdf6870),HTML::Element=HASH(0xdf68d0),HTML::Element=HASH
(0xdf6930),HTML::Element=HASH(0xdf6990)
HTML::Element=HASH(0xdf6a20),HTML::Element=HASH
(0xdf6a98),HTML::Element=HASH(0xdf6ad4),HTML::Element=HASH
(0xdf6b28),HTML::Element=HASH(0xdf6ba0),HTML::Element=HASH
(0xdf6c00),HTML::Element=HASH(0xdf6c60),HTML::Element=HASH
(0xdf6cc0),HTML::Element=HASH(0xdf6d20)
HTML::Element=HASH(0xdf6db0),HTML::Element=HASH
(0xdf6e28),HTML::Element=HASH(0xdf6e64),HTML::Element=HASH
(0xdf6f0c),HTML::Element=HASH(0xdf6f84),HTML::Element=HASH
(0xdf6fe4),HTML::Element=HASH(0xdf7044),HTML::Element=HASH
(0xdf70a4),HTML::Element=HASH(0xdf7104)
HTML::Element=HASH(0xdf7194),HTML::Element=HASH
(0xdf720c),HTML::Element=HASH(0xdf7248),HTML::Element=HASH
(0xdf72f0),HTML::Element=HASH(0xdf7368),HTML::Element=HASH
(0xdf73c8),HTML::Element=HASH(0xdf7428),HTML::Element=HASH
(0xdfae6c),HTML::Element=HASH(0xdfaecc)
HTML::Element=HASH(0xdfaf5c),HTML::Element=HASH
(0xdfafd4),HTML::Element=HASH(0xdfb010),HTML::Element=HASH
(0xdfb064),HTML::Element=HASH(0xdfb0dc),HTML::Element=HASH
(0xdfb13c),HTML::Element=HASH(0xdfb19c),HTML::Element=HASH
(0xdfb1fc),HTML::Element=HASH(0xdfb25c)
HTML::Element=HASH(0xdfb2ec),HTML::Element=HASH
(0xdfb364),HTML::Element=HASH(0xdfb3a0),HTML::Element=HASH
(0xdfb3f4),HTML::Element=HASH(0xdfb46c),HTML::Element=HASH
(0xdfb4cc),HTML::Element=HASH(0xdfb52c),HTML::Element=HASH
(0xdfb58c),HTML::Element=HASH(0xdfb5ec)
HTML::Element=HASH(0xdfb67c),HTML::Element=HASH
(0xdfb6f4),HTML::Element=HASH(0xdfb730),HTML::Element=HASH
(0xdfb7d8),HTML::Element=HASH(0xdfb850),HTML::Element=HASH
(0xdfb8b0),HTML::Element=HASH(0xdfb910),HTML::Element=HASH
(0xdfb970),HTML::Element=HASH(0xdfb9d0)
HTML::Element=HASH(0xdfba60),HTML::Element=HASH
(0xdfbad8),HTML::Element=HASH(0xdfbb14),HTML::Element=HASH
(0xdfbb68),HTML::Element=HASH(0xdfbbe0),HTML::Element=HASH
(0xdfbc40),HTML::Element=HASH(0xdfbca0),HTML::Element=HASH
(0xdfbd00),HTML::Element=HASH(0xdfbd60)
HTML::Element=HASH(0xdfbdf0),HTML::Element=HASH
(0xdfe75c),HTML::Element=HASH(0xdfe798),HTML::Element=HASH
(0xdfe84c),HTML::Element=HASH(0xdfe8c4),HTML::Element=HASH
(0xdfe924),HTML::Element=HASH(0xdfe984),HTML::Element=HASH
(0xdfe9e4),HTML::Element=HASH(0xdfea44)
HTML::Element=HASH(0xdfead4),HTML::Element=HASH
(0xdfeb4c),HTML::Element=HASH(0xdfeb88),HTML::Element=HASH
(0xdfec3c),HTML::Element=HASH(0xdfecb4),HTML::Element=HASH
(0xdfed14),HTML::Element=HASH(0xdfed74),HTML::Element=HASH
(0xdfedd4),HTML::Element=HASH(0xdfee34)
HTML::Element=HASH(0xdfeec4),HTML::Element=HASH
(0xdfef3c),HTML::Element=HASH(0xdfef78),HTML::Element=HASH
(0xdff020),HTML::Element=HASH(0xdff098),HTML::Element=HASH
(0xdff0f8),HTML::Element=HASH(0xdff158),HTML::Element=HASH
(0xdff1b8),HTML::Element=HASH(0xdff218)
HTML::Element=HASH(0xdff2a8),HTML::Element=HASH
(0xdff320),HTML::Element=HASH(0xdff35c),HTML::Element=HASH
(0xdff3b0),HTML::Element=HASH(0xdff428),HTML::Element=HASH
(0xdff488),HTML::Element=HASH(0xdff4e8),HTML::Element=HASH
(0xdff548),HTML::Element=HASH(0xdff5a8)
HTML::Element=HASH(0xdff638),HTML::Element=HASH
(0xdff6b0),HTML::Element=HASH(0xdff6ec),HTML::Element=HASH
(0xe040e0),HTML::Element=HASH(0xe04158),HTML::Element=HASH
(0xe041b8),HTML::Element=HASH(0xe04218),HTML::Element=HASH
(0xe04278),HTML::Element=HASH(0xe042d8)
HTML::Element=HASH(0xe04368),HTML::Element=HASH
(0xe043e0),HTML::Element=HASH(0xe0441c),HTML::Element=HASH
(0xe044d0),HTML::Element=HASH(0xe04548),HTML::Element=HASH
(0xe045a8),HTML::Element=HASH(0xe04608),HTML::Element=HASH
(0xe04668),HTML::Element=HASH(0xe046c8)
HTML::Element=HASH(0xe04758),HTML::Element=HASH
(0xe047d0),HTML::Element=HASH(0xe0480c),HTML::Element=HASH
(0xe04860),HTML::Element=HASH(0xe048d8),HTML::Element=HASH
(0xe04938),HTML::Element=HASH(0xe04998),HTML::Element=HASH
(0xe049f8),HTML::Element=HASH(0xe04a58)
HTML::Element=HASH(0xe04ae8),HTML::Element=HASH
(0xe04b60),HTML::Element=HASH(0xe04b9c),HTML::Element=HASH
(0xe04c44),HTML::Element=HASH(0xe04cbc),HTML::Element=HASH
(0xe04d1c),HTML::Element=HASH(0xe04d7c),HTML::Element=HASH
(0xe04ddc),HTML::Element=HASH(0xe04e3c)
HTML::Element=HASH(0xe04ecc),HTML::Element=HASH
(0xe04f44),HTML::Element=HASH(0xe04f80),HTML::Element=HASH
(0xe04fd4),HTML::Element=HASH(0xe089d8),HTML::Element=HASH
(0xe08a38),HTML::Element=HASH(0xe08a98),HTML::Element=HASH
(0xe08af8),HTML::Element=HASH(0xe08b58)

-------------------------
Source Code:

use strict;
use LWP::Simple;
use HTML::Tree;
use warnings;

my $file = "economic_calendar.dat";
unlink($file);

open FILE, ">$file" or die $!;

my $url = 'http://biz.yahoo.com/c/e.html';
my $content = get($url);

my $tree = HTML::TreeBuilder->new_from_content($content);

my @tr = $tree->look_down('_tag' => 'tr',
sub { $_[0]->as_text !~ m/My Yahoo!/ &&
$_[0]->as_text !~ m/Welcome/i &&
$_[0]->as_text !~ m/Economic Calendar/i
&&
$_[0]->as_text !~ m/Last Week/i });

foreach my $tr (@tr)
{
if ($tr)
{

my @detail = $tr->look_down('_tag' => 'td');
print join(',',@detail)."\n";
}

else
{
warn "No detail data";
}
}

close FILE;

$tree->delete;

Tad McClellan

unread,
Nov 24, 2009, 2:32:38 PM11/24/09
to
Ninja Li <nickl...@gmail.com> wrote:
> Hi,
>
> I am trying to parse HTML from the website http://biz.yahoo.com/c/e.html
> using HTML::TreeBuilder module and generate a comma-delimited file.


> use HTML::Tree;

> my @tr = $tree->look_down('_tag' => 'tr',


If the date that you want to scrape is in an HTML table,
then TableExtract is likely to make for prettier and more
robust code:


----------------------------------
#!/usr/bin/perl
use warnings;


use strict;
use LWP::Simple;

use HTML::TableExtract;

my $html = get 'http://biz.yahoo.com/c/e.html';

my @headers = (
'Date',
"Time",
'Statistic',
'For',
'Actual',
'Briefing Forecast',
'Market Expects',
'Prior',
"Revised\nFrom",
);

my $te = HTML::TableExtract->new( headers => \@headers );
$te->parse($html);

foreach my $ts ( $te->tables ) {
foreach my $row ($ts->rows) {
my $csv = join ',', @$row;
print "$csv\n";
}
}
----------------------------------


--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"

Ninja Li

unread,
Nov 24, 2009, 3:52:48 PM11/24/09
to
On Nov 24, 2:32 pm, Tad McClellan <ta...@seesig.invalid> wrote:

> Ninja Li <nickli2...@gmail.com> wrote:
> > Hi,
>
> >    I am trying to parse HTML from the websitehttp://biz.yahoo.com/c/e.html

Tad,

Thanks for your response. HTML::TableExtract looks to be a better
option for dealing with HTML. I tried to apply your code to another
web link, however I didn't get any output. Please advise what might be
wrong. The code is at the end of the post.

Thanks.

Nick

------------------------

use warnings;
use strict;
use LWP::Simple;
use HTML::TableExtract;

my $html = 'http://www.earnings.com/conferencecall.asp?client=cb';

my @headers =
(
'SYMBOL',
'COMPANY',
'EVENT TITLE',
'WEBCAST',
'TRANSCRIPT',
'TIME'

Tad McClellan

unread,
Nov 24, 2009, 4:19:24 PM11/24/09
to

[ Please do not full quote.
Quote only enough to provide context for your comments.
Please see the Posting Guidelines that are posted here frequently.
]

Ninja Li <nickl...@gmail.com> wrote:
> On Nov 24, 2:32 pm, Tad McClellan <ta...@seesig.invalid> wrote:

[snip]

>> my $html = get 'http://biz.yahoo.com/c/e.html';

^^^
^^^

[snip]

>> --
>> Tad McClellan
>> email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"


You are not supposed to quote .sigs either...


> however I didn't get any output.

> my $html = 'http://www.earnings.com/conferencecall.asp?client=cb';


Try this:

print $html;

and see if that doesn't give you hint...

Ninja Li

unread,
Nov 24, 2009, 4:48:32 PM11/24/09
to
Tad,

Maybe I am too slow on this, print $html only shows the web link,
right? I can confirm it does contain valid data. Could you give more
details?

Thanks.

Nick

Tad McClellan

unread,
Nov 24, 2009, 5:03:13 PM11/24/09
to
Ninja Li <nickl...@gmail.com> wrote:

> print $html only shows the web link,


And what _should_ it show?

It should contain HTML, not a URL.

Why does it contain the URL instead of HTML that was fetched
from that URL?


> I can confirm it does contain valid data.


If $html does not contain <html>, then it does NOT contain valid data.


> Could you give more
> details?


Did you see the part of my code that I underlined in my followup?

Does your code have the part that was underlined?

You will smack your forehead when you see the mistake you've made...

:-)

Ninja Li

unread,
Nov 24, 2009, 5:44:16 PM11/24/09
to
On Nov 24, 5:03 pm, Tad McClellan <ta...@seesig.invalid> wrote:

Thanks a million. A smack on the head is required and deserved for me.

Nick

0 new messages