[PHP-WIN] reading lines from a file
George Pitcher
going to be supplied to me on a fairly regular basis.
George
> -----Original Message-----
> From: Svensson, B.A.T. (HKG) [mailto:B.A.T.S...@lumc.nl]
> Sent: 11 February 2003 10:56 am
> To: George Pitcher; php-w...@lists.php.net
> Subject: RE: [PHP-WIN] reading lines from a file
>
>
> What do you want to do? Parse the file into tab-delimited data
> in order to bulk load it into the database?
>
> > -----Original Message-----
> > From: George Pitcher [mailto:george....@ingenta.com]
> > Sent: Tuesday, February 11, 2003 11:41 AM
> > To: php-w...@lists.php.net
> > Subject: [PHP-WIN] reading lines from a file
> >
> >
> > Hi,
> >
> > I have a huge file that I want to get into a db format.
> >
> > There are over 5000 records and I need to create fieldnames on the fly.
> >
> > Each record comprises a number of lines (fields) containing 2 elements
> > (fieldname [first 5 chars (which need rtrimming)], value [rest
> of line, some
> > with lots of text]).
> >
> > Some fieldnames are duplicated within a record so I need to
> rename these by
> > using a counter which is reset for each record (there are at least 5
> > fieldnames that are affected this way - so 5 counters).
> >
> > Some values contain subfields so I need to be able to handle
> these, but this
> > doesn't look frightening.
> >
> > And I would like to end up with an array with keys (fieldnames)
> and values.
> >
> > But I'm stuck at the first hurdle because I can't find out how
> to put a line
> > into a string.
> >
> > Any pointers?
> >
> > George in Oxford
> >
> >
> > ===
> >
> > George Pitcher
> > HERON Technical Manager
> > Ingenta plc
> > 23-38 Hythe Bridge Street, Oxford, OX1 2ET
> > T +44 (0)1865 799137 direct
> > T +44 (0)1865 799000 switchboard
> > F +44 (0)1865 799134
> > E george....@ingenta.com
> >
> > www.ingenta.com
> > Ingenta: Empowering the exchange of academic and professional content
> > online.
> >
> >
> > --
> > PHP Windows Mailing List (http://www.php.net/)
> > To unsubscribe, visit: http://www.php.net/unsub.php
> >
>
B.A.T. Svensson
B.A.T. Svensson
Chris Kranz
manual) and then break up each line into an array to be added to the db.
This is fairly basic stuff, you just need to look in the php manual, it
sounds like you don't really have a problem as such, you just want
someone to tell you what the code would be?
chris kranz
fatcuban.com
-----Original Message-----
From: George Pitcher [mailto:george....@ingenta.com]
Sent: 11 February 2003 11:59
To: Svensson, B.A.T. (HKG); php-w...@lists.php.net
Subject: RE: [PHP-WIN] reading lines from a file
Yes, I suppose I do.
I want to be able to take my file, work out a complete of fieldnames and
then create an array which could then be used to, initially, create a
table
in MySQL.
I'm not using MySQL for my main system which is based on FileMaker, but
will
be moving later this year to MySQL, so it's not a wasted effort.
I know that I can export the MySQL data and push that into FileMaker.
Below (in between ==== lines, is a sample record for parsing. This may
help
you understand the problem I have.
=======================================================
NEWRC -T&F eBook data rec.1534 |@ 27/01/2003
VISBN 0203445570
OSBN 0-203-44557-0
EBK1 0203548310
EBK2 0203651057
EBK3 020375381X
EBK4 0203273567
OTOP
OTART
OTI Creative Mathematics
OSUB
USAU Upitis |f Rena |u Queens University |c Kingston, Ontario |x Canada
USAU Phillips |f Eileen |u Teacher |c Vancover |x Canada
USAU Higginson |f William |u Queens University |c Kingston |x Canada
EDIT
INTRO
FORWD Pimm |f David |u The Open University x| UK
TRANS
2BIND 0415164621 |P 70.00 |U 100.00 |B Hb |A |M 1 |T Creative
Mathematics
VPR 11.99
USPRI 17.99
OTBPR 19.99 |U 25.95 |D 17/07/1997
USBDG
MFDT
ERT GB US CA AU NZ ZA IE
ORT
NRT
TTY
OBIND eB
PDUE 11/03/2002
VIZE
VEXT 192
OILLS 40 illustrations
LEVEL
OSER
OIMP RoutledgeFalmer
LIBPR
OALT
OPPQ
OCATL This book shows how creative maths can <I>really</I> work.
Exploring
the ways in which maths skills can be learned through cross-curricular
activities based on visual arts and music, the book presents maths as a
meaningful and exciting subject which holds no fears for children.<BR>
The
authors recognise that while maths-phobia prevails in our increasingly
mathematicised world, attitudes and approaches to teaching the subject
need
to be reviewed, and issues such as gender stereotyping, which encourage
maths-apathy, need to be tackled at an early stage.<BR> Within this
collection of classroom-based stories are detailed examples of
integrative
mathematic projects; these will give teachers the confidence to try out
cross-curricular activities in their classes. The book also provides
support
with difficult areas such as assessment, planning and development.<BR>
Fascinating to read in its own right this book will appeal to the
specialist
and non-specialist alike.<BR>
OCONC This book shows how creative maths can <I>really</I> work.
Exploring
the ways in which maths skills can be learned through cross-curricular
activities based on visual arts and music, the book presents maths as a
meaningful and exciting subject which holds no fears for children.<BR>
The
authors recognise that while maths-phobia prevails in our increasingly
mathematicised world, attitudes and approaches to teaching the subject
need
to be reviewed, and issues such as gender stereotyping, which encourage
maths-apathy, need to be tackled at an early stage.<BR> Within this
collection of classroom-based stories are detailed examples of
integrative
mathematic projects; these will give teachers the confidence to try out
cross-curricular activities in their classes. The book also provides
support
with difficult areas such as assessment, planning and development.<BR>
Fascinating to read in its own right this book will appeal to the
specialist
and non-specialist alike.<BR>
OCATS Exploring the ways in which maths skills can be learned through
cross-curricular projects on arts and music, this book presents maths as
a
meaningful and exciting subject which holds no fear for children.
OQUO 'This is a fascinating book ... which gives detailed case
studies.' |I
Infant Equipment
OCNT
FCONT
OWRIT
VEDIT 1
BICAC U
BSUB1 J
BSUB2 JD
BSUB3
BICAV RF
SUBJ1 SB150000
SUBJ2
SUBJ3
VPROD EB
VACDE WEB
CREAT MFK - AUTO |D 2002 07 19
================================================================
the use of '|' denotes a subfield. My plan is to name the subfield by
the
name of the field followed by '_' and the subfield identifier, so that
the
one on the last line would appear as 'CREAT_D, and that way I'll be sure
to
properly tag those that are belonging to the 'counter-affected' fields.
All help is appreciated.
Cheers
George Pitcher
All help is appreciated.
Cheers
George
Vladimir Galkov
$f_array = file (FILE_NAME);
for ($i=0; $i<count($f_array); $i++)
{
$arr_str = explode(";", $f_array[$i]);
//***********
// work here with $arr_str as array of str elements (if separator ";")/ if
else use ereg or other way
//***************
}
"George Pitcher" <george....@ingenta.com> сообщил/сообщила в новостях
следующее: news:HHEIKBOENJPJNLHFCCCCA...@ingenta.com...
Jill Ramonsky
fgets()
It's in the section Filesystem functions. (What a surprise)
You might like the function file() also.
Jill
-----Original Message-----
From: George Pitcher [mailto:george....@ingenta.com]
Sent: Tuesday, February 11, 2003 12:25 PM
To: Chris Kranz; php-w...@lists.php.net
Subject: RE: [PHP-WIN] reading lines from a file
Chris,
I couldn't find any readline function in the manual - did I miss it?
B.A.T. Svensson
* decide what data in this file that you need
* create a data model based on this data
* sufficiently normalize your data
* implement a database schema according to the normalized data
* write a parser that parse data according to the RDBMS schema.
* load data in the RDBMS
* finished
That's what you need to do, and as far I can understand none
of the above activities are related to PHP at this moment.
However, it should be fairly easy to write a parser for
this kind of record layout. I see no major problem with
it.
I would recommend you to consult a professional programmer,
or at least a programmer with some experience with C/C++ or
perl or whatever, and he will for sure be able to make a
parser for you according to some specification that you
can work out together with him/her. (I've seen worse
record layout in my days. ;)
I am sorry for not being able to give you the answer
you wish for.
//Anders
P.S. I am just a bit curious on how big this "huge file" is? D.S.
George Pitcher
Thanks - I'll see if I can persuade one of the PERL people here to have a
go.
Ta.
george
George Pitcher
I couldn't find any readline function in the manual - did I miss it?
It isn't as simple as you say, because, to start with, I don't know how many
fields there are going to be, I don't know how I'm going to build my array
to record-level sets.
I'm not really looking for someone to provide the code (snippets accepted
though) but I would appreciate being pointed in the right direction.
B.A.T. Svensson
It is as bad recommendation to suggest it. And you just don't do that.
> -----Original Message-----
> From: Jill.R...@Aculab.com [mailto:Jill.R...@Aculab.com]
> Sent: Tuesday, February 11, 2003 1:27 PM
> To: php-w...@lists.php.net
> Subject: RE: [PHP-WIN] reading lines from a file
>
>
> Yes
>
> fgets()
> It's in the section Filesystem functions. (What a surprise)
>
> You might like the function file() also.
>
> Jill
>
> -----Original Message-----
> From: George Pitcher [mailto:george....@ingenta.com]
> Sent: Tuesday, February 11, 2003 12:25 PM
> To: Chris Kranz; php-w...@lists.php.net
> Subject: RE: [PHP-WIN] reading lines from a file
>
>
> Chris,
>
> I couldn't find any readline function in the manual - did I miss it?
>
Jill Ramonsky
If someone asks me "What's the function that reads a line from a file?" I'm
happy to answer it.
What they want to use that function for is not my problem - nor yours.
In this forum we can hopefully get answers to questions along the lines of
"How do you do XYZ?" from each other. I know it's bad form to ask "What's
the algorithm for XYZ?", but that wasn't actually asked.
Furthermore, if you read the text _BELOW_ (which is what you ACTUALLY
replied to, and I have included in full, despite the usual netiquette),
you'll find no mention of parsing or record-based files. That was another
email in the same thread. There was no reason to reply to MINE. Your reply
_is not_ a reply to _my_ post.
If you want to talk more about your philosophy please could you start a new
thread, and leave me out of it.
Jill
-----Original Message-----
From: Svensson, B.A.T. (HKG) [mailto:B.A.T.S...@lumc.nl]
B.A.T. Svensson
But I don't understand why you take this personally?
Maybe I should have written "one" instead of "you"?
Anyhow;
First; If a programmer suggest to parse an "huge file" via a
web based script; that programmer is an idiot and should be
fired. Period.
Secondly; weather or not you are right, by maintaining the
very idea of giving "help" to this person you encourage a
very bad solution and at the same time you confuse and make
waste of that persons time. Reading that persons questions/
answer it fairly easy to understand that (s)he has less
experience with programming.
In fact you was not helping this person at all, but rather
contributing to create a mess for this person.
Following the tread it is quite obviously that the people who
answer/"helps" in the thread don't care to investigate what
the person who asked the question actually needed
What a person might ask for it not always whet that person needs.
If you think you don't have this time to investigate what
people really need. Try to avoid to answer then, because it
wont help them, but rather confuse them.
Because why else would one like to answer? Exhibition purposes?
Ignatius Reilly
I trust that I am not the only one to be curious to learn more concerning
your remark.
I routinely parse files similar to those of Mr Pitcher, using PHP from the
Windows command line.
So far I have never had any problem generating CSV files containing about a
million rows (Celeron 750, 256 MB).
Would you care to explain in more details why PHP is a MUST DON'T to do
this?
Thanks
Ignatius
____________________________________________
B.A.T. Svensson
> learn more concerning your remark.
[...]
> Would you care to explain in more details why PHP is
> a MUST DON'T to do this?
Hi,
Thanx for your comment.
One can talk about things like load balance for scalability,
and other boring subjects, but I agree that it is of course
hard to establish what is meant with a "good" design without
knowing the purposes. But, my general view goes this:
"Of course you can use a hammer to nail a screw, but
a screw driver will do the job much easier and better."
ARGUMENT OF COUPLING & COHESION:
By using for instance PHP for parsing a large text files,
you are stealing resources from the web server (assuming
PHP runs as an instance within Apache), resources better
used elsewhere for other purposes.
A single application (the parser) can be executed in
its own processing space and given independent priority,
this will favoring parallelism with multi CPU system;
if the parser is smart enough written it might even
utilize several CPU's, possible running with low priority
as a background task.
If the system become over loaded, then parser
application can easily be moved onto another system.
Anyway, the main idea is that you wants to keep the
pears and apples separated from each other.
By the way; You're fired! :)
//Anders
George Pitcher
Nope, I never mentioned the command line at all, but it is probably how I
would have done it if I hadn't passed it over to the PERL guys.
It would have been useful to go through a parsing routine just to learn how
its done (never know when it might be a useful trick).
Even so, I still have plenty on my plate migrating my Filemaker/Lasso system
over to MySQL/PHP/Java.
Cheers
George
> -----Original Message-----
> From: Chris Kranz [mailto:ch...@kranz.com]
> Sent: 11 February 2003 4:30 pm
> To: 'Php Win32 list'
> Subject: RE: [PHP-WIN] reading lines from a file
>
>
> Didn't he just say parse it via command line tho?
>
> This wouldn't take away resources from the web server?
>
> Personally, large batches like this shouldn't be done on your live
> webserver anyway. No matter how you accomplish it, your taking away a
> lot of system resources. Best way to do it, is to parse it on a separate
> computer, then just do a simple db dump with the results.
>
> In this case, is PHP any worse a language than a CGI or c++ program?
> Other than being a lot easier to code (in my opinion :p).
>
> I've done the same thing, on many occasions, but I don't run the scripts
> on my live server, that's just asking for trouble, especially if you
> make a mistake... (thinks back to his recursive directory delete
> function)
>
> Hehe :)
>
> chris kranz
> fatcuban.com
>
>
> -----Original Message-----
> From: Svensson, B.A.T. (HKG) [mailto:B.A.T.S...@lumc.nl]
> Sent: 11 February 2003 15:13
> To: Ignatius Reilly
> Cc: Php Win32 list
> Subject: RE: [PHP-WIN] reading lines from a file
>
Ignatius Reilly
Well, of course the parser will consume full resources during several
minutes. But I trust our old fellow George would do the parsing on a
separate machine, and only do the LOAD DATA LOCAL on the production machine
at night or such.
My question was more as to whether there is a fundamental performance
difference for file parsing between PHP at the command line and other
languages such as Perl (which I don't know, so you won't have the
opportunity to fire me because you won't hire me in the first place :-)
Cheers
Ignatius (self-employed idiot, currently pondering about firing himself)
____________________________________________
----- Original Message -----
From: "Svensson, B.A.T. (HKG)" <B.A.T.S...@lumc.nl>
To: "Ignatius Reilly" <ignatiu...@free.fr>
Cc: "Php Win32 list" <php-w...@lists.php.net>
Sent: Tuesday, February 11, 2003 4:12 PM
Subject: RE: [PHP-WIN] reading lines from a file
Chris Kranz
This wouldn't take away resources from the web server?
Personally, large batches like this shouldn't be done on your live
webserver anyway. No matter how you accomplish it, your taking away a
lot of system resources. Best way to do it, is to parse it on a separate
computer, then just do a simple db dump with the results.
In this case, is PHP any worse a language than a CGI or c++ program?
Other than being a lot easier to code (in my opinion :p).
I've done the same thing, on many occasions, but I don't run the scripts
on my live server, that's just asking for trouble, especially if you
make a mistake... (thinks back to his recursive directory delete
function)
Hehe :)
chris kranz
fatcuban.com
-----Original Message-----
From: Svensson, B.A.T. (HKG) [mailto:B.A.T.S...@lumc.nl]
Sent: 11 February 2003 15:13
To: Ignatius Reilly
Cc: Php Win32 list
Subject: RE: [PHP-WIN] reading lines from a file
[...]
Hi,
Thanx for your comment.
//Anders
--
B.A.T. Svensson
this in PHP is because PHP is what you are good/best at?
Also excuse my limited ignorant knowledge about PHP, but
my impression about PHP is that it is runs as an instance
of the web server or something? (Please, correct me if I
made any fundamental mistake here!)
When it comes to perl vs PHP, I believe both of these
languages are interpreted in the host evirionment, and I
can't see any reason why there should be any significant
performance difference between these two.
On the other hand I can imagine an performance difference
on doing this with C/C++, which would be my natural choose
when writing a cmd line parser.
> -----Original Message-----
> From: Ignatius Reilly [mailto:ignatiu...@free.fr]
> Sent: Tuesday, February 11, 2003 5:09 PM
> To: Svensson, B.A.T. (HKG)
B.A.T. Svensson
in general I agree with you.
My remark were against using a web server for this, running
as a thread under it - or whatever. If somebody do that,
then they waste unnecessary system resources.
Maybe I had a to strong opinion in the first letter; writing
in a haste at work while occupied with other stuff might not
always be good for your way of expressing your self.
Design and implementations is a matter of compromises
between contradicting requirements, and if one does
not have knowledge about the pre requirements, then
one easily can make a faulty judgment about a design.
However, there are some few underlying design principles
that once should try to follow if possible - and in most
case they are. As a general role; if I need to scarify
performance for maintenance in a none time critical section,
then I will scarify performance without hesitation.
Bad performance can always be fixed with faster
hardware later on, but how do you fix bad design?
In using PHP in kind of the same fashion as a script
languages, I have no objection against; is this even
possible? And you are right PHP is no worse than anything
else in that respect.
But the best thing would be if we didn't had to parse files
at all. :) But those things only happens in programming utopia
I have heard.
But I don't agree with your point that PHP necessarily would
be easier than C++, my personal opinion is rather the other
way around. :)
> -----Original Message-----
> From: Chris Kranz [mailto:ch...@kranz.com]
Sven Schnitzke
I understand your points (well - mostly) but
----- pleeez don't get too fundamental over that !! ------
This thread is up to 21postings by now and
if I knew a better NG I'd suggest a follow up there!
This one is for helping people get going with PHP in
WIN environment, and as far as I am asked with as
little judgement as possible.
Cheer up and be confident in the judgement of the
respectively responsible. There are aspects to consider
unmentioned and irrelevant w/resp to this NG. So I'd
suggest to keep cool.
Cheers,
--
Sven Schnitzke (another self-employed one on the edge of being fired ...)
B.A.T. Svensson
I don't need that, because I don't use PHP. ;)
> But then again, doing the parser would be a much more time-consuming
> process. And even worse, we're on the wrong mailing list for that... :)
I guess you are quite right on that one too.
Per Lundberg
> Also excuse my limited ignorant knowledge about PHP, but
> my impression about PHP is that it is runs as an instance
> of the web server or something? (Please, correct me if I
> made any fundamental mistake here!)
Yes and no. It can be run both as a SAPI module (running in Apache,
IIS, Netscape or another web server) or as a stand-alone process in what
is called CGI and CLI modes. There is even wrappers for writing Gtk+
applications with PHP using these modes (the wrapper is called PHP-GTK I
think).
I would say PHP is just as "all-round" as Perl actually, and I have
personally switched from Perl to PHP completely because I prefer PHP's
more "C-like" syntax. Maybe you should Read The Fine Manual, Anders?
:-)
> On the other hand I can imagine an performance difference
> on doing this with C/C++, which would be my natural choose
> when writing a cmd line parser.
But then again, doing the parser would be a much more time-consuming
process. And even worse, we're on the wrong mailing list for that... :)
--
Best regards,
Per Lundberg / Capio ApS
Phone: +46-18-4186040
Fax: +46-18-4186049
Web: http://www.capio.com
Neil Smith
I think chris probably meant not readline but fgets() and / or file()
fgets() reads a specified chunk size into a string or other variable.
file() reads a file line by line (separator is the newline character: :
array file (string filename [, int use_include_path])
To be honest, if you will receive an unknown number of fields, you might be
better transforming the whole lot to XML and storing the XML en-bloc in the
database - unless you require to search on each field from the off.
You can parse the XML to fieldsets at a later date when you've established
just *what* fields are going to be sent. At least this way you can fool
around with XSL to display at least the common information in each file,
but still retain the oddities for later analysis, without having to set up
a DB with dozens of possibly redundant fields.
HTH - my 2cv err - 2c lol
Neil Smith.
At 23:59 11/02/2003 +0000, you wrote:
>From: "George Pitcher" <george....@ingenta.com>
>To: "Chris Kranz" <ch...@kranz.com>,
> <php-w...@lists.php.net>
>Date: Tue, 11 Feb 2003 12:25:14 -0000
>Subject: RE: [PHP-WIN] reading lines from a file
>
>Chris,
>
>I couldn't find any readline function in the manual - did I miss it?
>
B.A.T. Svensson
> respectively responsible. There are aspects to consider
> unmentioned and irrelevant w/resp to this NG. So I'd
> suggest to keep cool.
Basically I don't care how people implement things: as long
as I don't have to clean it up. So it not my business, but
on the other hand if you ask for my opinion... ;)
> Sven Schnitzke (another self-employed one on the edge of being fired ...)
Well,t he business need a clean up anyway, hehe. :)
//Aners