dev-make D3 on HP-UX
hnw
System Release Information
==========================
D3 Release Version 7.1.0.HP
Implementation. . . . . . HP9000
System Serial Number. . . 11029593
System ID Number . . . . 20146173
Release . . . . . . . . . D3/UNIX: HP9000
Unix Information. . . . . HP-UX;pick0:HPUX;B.11.00;A;123931537
We have been tasked with exporting a bunch of data for a customer
from our distant past.
We have been a jBase house for forever, and our Pick skill set stops
at R83
As some of the files have over a million records, and the way the
customer wants them extracted will multiply that by 10 or more and as
the customer is in Central America, our first choice would be to dump
this data to a pseudo floppy, ftp it and then read and massage it in
jBase.
Option 2 would be to convert to flat files in situ.
Some questions:
What is the right syntax for dev-make on hpux? We have tried all sorts
of combinations without success. Most look something like
:dev-make -t Tape -a "/tmp/floppy,p"
dev-make: Creation failed. [-003] Invalid Entity ID
Failing success with dev-make, can we edit the pick0 item and create a
new device that way ?
Will that require a reboot, and does a reboot require the compressed
abs file which I think I clobbered in my fumbling ?
Is there a faster way to write sequential files than to open "","unix:/
path", create a regular pick item and write it ?
Mark Hogden
Ross Ferris
Try something like:
chg-device 0 name=/YourPathGoesHere/FileName type=floppy
density=pseudo
Provided you are dropping out <2Gb of data you should be fine ... bad
things happen if you go over 2Gb (which is why D3 supports compressed
files for backup, BUT you probably wouldn't be able to read those
.... oh, hang on, you want to WRITE out a flat file (rather than t-
dump or something) ...
OK, just open the file as you have suggested, and build up an array
that you can then MATWRITE out ---> maybe 1 "file" for every 10,000
records?
You can stitch these back together at the other end ... maybe convert
@AM to char(9) for a tab delimited file, perhaps @VM to "|" and @sm to
"~"
HTH
hnw
The CHG-DEVICE worked a treat. I can now T-LOAD on my end and use our
stock jBase tools for the exports.
Thanks
Mark
Tony Gravagno
read/write from OS-level files?
OPEN "/tmp/d3datashare" TO FV ELSE STOP
WRITE REC ON FV,ID
...
If the DBMS systems are in the same physical area, jBase can read that
through jEDI without having to create a hashed file. With Samba (or
similar) shares you should be able to do everything from BASIC with no
data transport at all.
If the source and target systems are really far away, then sure, you
can t-dump. But also consider just writing all data from D3 as above,
then using tar/ftp. Why? On the receiving end you'll have a
directory with file/items, rather than a t-dump to unpack first. That
might be attractive.
HTH
Tony Gravagno
Nebula Research and Development
TG@ remove.pleaseNebula-RnD.com
Nebula R&D sells mv.NET and other Pick/MultiValue products
worldwide, and provides related development services
remove.pleaseNebula-RnD.com/blog
Visit PickWiki.com! Contribute!
http://Twitter.com/TonyGravagno
RJ
is needed with the ItemId inserted as the first field of the file before
writing.
BobJ
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:5p25q557he6umqi31...@4ax.com...
Ross Ferris
just
COPY MYFILE *
TO:(/tmp/d3datashare
> "Tony Gravagno" <address.is.in.po...@removethis.com.invalid> wrote in
> messagenews:5p25q557he6umqi31...@4ax.com...
Tony Gravagno
Ross Ferris
<address.is.in.po...@removethis.com.invalid> wrote:
> If you're using the OS file system there is no hashing.
My comment should have been "were NOT a problem", in reference to the
point RJ made - not sure where hashing came into the equation, but
interestingly enough I have had cause to "play" with a directory with
a few million files in it under windows.
Performance is EXCELLENT (instant!) .... but never, Ever, EVER open a
directory with this many files in explorer!!!
RJ
"Ross Ferris" <ro...@stamina.com.au> wrote in message
news:3777a17e-8e07-438a...@n7g2000prc.googlegroups.com...
> If the item ID's were a problem, then why write a program even ....
> just
>
> COPY MYFILE *
> TO:(/tmp/d3datashare
Because WriteSeq could be considerably faster if the file is huge.
Actually that's a difficult thing to determine given the paucity of
information about how MS does some things. Certainly ReadSeq and WriteSeq
are a lot faster than using the native hash for even medium sized files.
BobJ
RJ
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:7l16q5huo70m9iec0...@4ax.com...
> If you're using the OS file system there is no hashing.
Is that true? If you are using a native file system and addressing an Item
by name then how is it located? If it is a search, even a sophisticated
search, then it is going to be very slow if the file is heavily populated.
BobJ
Kevin Powick
> If you are using a native file system and addressing an Item
> by name then how is it located?
In the native/OS file system every D3 item is an actual file, not an
item within a larger file, so it's easily and quickly located by the
native OS. Ever notice how in a native file system made up of
hundreds of thousands of files in hundreds of directories, you can
gain instant access to any file?
While direct access should be close to instant, sorting on an
attribute will be slower for D3 items residing in a native/OS
directory.
--
Kevin Powick
Tony Gravagno
First, D3 doesn't use WriteSEQ, it's the same BASIC Read, Write,
Delete, and ClearFile that are used for hashed files. This allows
your D3 code to be consistent independent of the underlying datastore.
Compare this to platforms that do actually use WriteSeq or UWrite,
essentially hard-coding a program to a specific environment. I just
wish TigerLogic would appreciate their own technology a little better
and hype OSFI for doing this the way jBase made a big deal of jEDI ...
it's essentially the same thing.
Second, native/OS file operations are not faster than hashed files.
The DBMS needs to make a read request of the OS where the DBMS cannot
control caching or flushing. Writes must be immediate and new reads
must be performed on every operation. Sure, the OS can cache data to
avoid disk activity but the data transfer into DBMS memory is still
required on every operation. Every DBMS will handle the IO
differently but the challenges will be the same.
Third and finally, as to the speed of hashing, I think the DBMS
hashing algorithm is as efficient as *nix or Windows use of inodes.
Both DBMS and OS environments seem equally fast in finding data, where
there is proper memory allocation and defragmentation of both DBMS and
OS. I haven't seen any research done to compare them. I suspect
doing so would be extremely difficult.
T
RJ
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:jl57q51mnebri704a...@4ax.com...
> Bob - you're mixing terms and also being inaccurate, sorry my friend.
>
> First, D3 doesn't use WriteSEQ, it's the same BASIC Read, Write,
> Delete, and ClearFile that are used for hashed files. This allows
> your D3 code to be consistent independent of the underlying datastore.
> Compare this to platforms that do actually use WriteSeq or UWrite,
> essentially hard-coding a program to a specific environment. I just
> wish TigerLogic would appreciate their own technology a little better
> and hype OSFI for doing this the way jBase made a big deal of jEDI ...
> it's essentially the same thing.
I don't use D3 and I'm surprised anyone does. QM is a fraction of the
cost and Martin has heard the word "support."
QM and jBase make good use of OpenSeq. I suspect others do also.
>
> Second, native/OS file operations are not faster than hashed files.
> The DBMS needs to make a read request of the OS where the DBMS cannot
> control caching or flushing. Writes must be immediate and new reads
> must be performed on every operation. Sure, the OS can cache data to
> avoid disk activity but the data transfer into DBMS memory is still
> required on every operation. Every DBMS will handle the IO
> differently but the challenges will be the same.
Never said they were. Did say that WriteSeq is faster than OS write for
very large sets of data. Won't be true for D3 because
no WriteSeq.
>
> Third and finally, as to the speed of hashing, I think the DBMS
> hashing algorithm is as efficient as *nix or Windows use of inodes.
> Both DBMS and OS environments seem equally fast in finding data, where
> there is proper memory allocation and defragmentation of both DBMS and
> OS. I haven't seen any research done to compare them. I suspect
> doing so would be extremely difficult.
Can't argue with that. Optimizing microseconds doesn't pay anyway.
BobJ
Tony Gravagno
> I don't use D3 and I'm surprised anyone does. QM is a fraction of the
>cost and Martin has heard the word "support."
Then what the heck are you doing providing advice in this thread?
Incorrect advice at that? Please Bob
I think the only problem with TigerLogic Technical Support these days
is that there is no one marketing the TigerLogic Technical Support
department. As always, TL "Lack Of Marketing Department" is their own
worst enemy. (I'm not talking about the people charged with
implementing Marketing decisions, I mean the people who are supposed
to be making Marketing decisions.) People in the field seem to have
no clue that things are different there from the "weird" several years
ago. They're doing some good things over there, if only they'd bother
to tell people about it.
> Did say that WriteSeq is faster than OS write for
> very large sets of data.
What? WriteSeq _is_ OS write for some non-D3 platforms.
> Won't be true for D3 because no WriteSeq.
What? Which part of what I wrote did you miss? D3 and the other DBMS
products do exactly the same thing, but D3 doesn't require you to use
a different command for it. With a single D3 program I can use a
local hashed file, an OS file, or a file across the internet, and not
change a line. With OpenSeq that's not the case.
> QM and jBase make good use of OpenSeq. I suspect others do also.
OpenSeq is a legacy command that should never have been implemented
except that engineers at the time probably couldn't figure out a way
to use the same Open syntax to refer to a non-hashed file. So now
it's been re-implemented across other MV platforms purely for
compatibility, and developers are stuck with it. And you're here
arguing for it like it's a feature.
OpenSeq is like a pen that only writes on Tuesdays. Why use it when
your other Open pen can (or should) write every day of the week?
T
Ross Ferris
and OPEN/READ/WRITE, especially for VERY LARGE (multi-Mb "files" -
and there actually IS an equivalent of OPENSEQ/READSEQ in D3!)
with the ...SEQ commands you can build or read an item 1 "line" at a
time whereas straight OPEN/READ/WRITE commands read the entire OS
"file" in ....
I believe QM provides the same functionality with DIR type files,
while in UV they would be type 1 or 19
RJ
with TL is that only a fool would do business with them. If you are stuck
with a D3 application that works the MAYBE staying with them pays. But
porting to something else would probably pay larger dividends over time.
I'll say this much for TL, they have never hidden their intent or their
motives.
BobJ
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:qde8q59hetg8jribf...@4ax.com...
Jeff Caspari
> motives.
> BobJ
Which are?
RJ
Jeff Caspari
"RJ" <nob...@nowhere.com> wrote in message news:hOmpn.3855$Ek4....@newsfe24.iad...
Thanks, Bob. You made it sound nefarious so I wanted to clarify for fairness.
RJ
"Jeff Caspari" <mu...@idt.net> wrote in message news:4ba611b4$0$22537$607e...@cv.net...
Kevin Powick
> To be fair Tony, there IS a difference between OPENSEQ/READSEQWRITESEQ
> and OPEN/READ/WRITE
I agree that Tony seems to have confused reading/writing of items vs.
sequential file access of OS level files -- Which is why there are
different commands for these operations in products other than D3.
> and there actually IS an equivalent of OPENSEQ/READSEQ in D3!)
What is the D3 equivalent function/statement in BASIC? I do not know
of one, and have scanned the manual again hoping to find it.
--
Kevin Powick
Ross Ferris
>
> What is the D3 equivalent function/statement in BASIC? I do not know
> of one, and have scanned the manual again hoping to find it.
>
> --
> Kevin Powick
Hmmm, I actually had to go and look at this one ... Visage goes &
hides all of the platform specific stuff so we just have a standard
interface for all platforms to "do stuff". Anyway, a quick look at the
generated platform specific code highlighted the "hidden magic" - the
%Cfunctions
Like so many features of D3, I believe these little gems have laid
dormant for years - at one stage we looked at writing our own
extensions, but ultimately decided against it. Anyway the following
link should get you started if you don't have the REF account loaded
(or if you use D3/Windows, 'cause their own tools [REF] still don't
work in the FSI) http://www.d3ref.com/index.php?token=basic.cfunc
Enjoy!
Ross Ferris
Stamina Software
Vsiage - Better by Design!
Tony Gravagno
Unlike some of our colleagues, when I'm completely wrong I'll admit it
quickly and openly. You're right Kevin. I haven't been giving
OpenSeq the proper credit for special functionality and have been
ignoring the sequential access feature altogether. I'm thinking
OpenSeq and related statements would be used to iterate through a long
OS file one line at a time. This is not the same as D3 open/read
which will read the OS file all at once.
The equivalent in D3 would be to either use %functions, or to do this
(off the top of my head):
EXECUTE "QSELECT /tmp somefile"
LOOP
READNEXT LINE ELSE STOP
* do something with the line
UNTIL 0 DO REPEAT
All that said, I'll still stick by using D3 OSFI as an alternative to
device operations. This had nothing to do with the digression on
OpenSeq. I never mentioned OpenSeq in my original proposal. Bob said
"WriteSeq could be considerably faster" and I digressed because D3
doesn't have WriteSeq so it can't possibly be faster in D3. Given
that fact it's tough to know if it's faster or not. As to whether
it's faster in platforms that support it, I'd guess that's probably
true.
Thanks for keeping me honest guys.
T
RJ
but never dishonest .
BobJ
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:9u5dq55eak57f6sof...@4ax.com...
Kevin Powick
<address.is.in.po...@removethis.com.invalid> wrote:
> EXECUTE "QSELECT /tmp somefile"
> LOOP
> READNEXT LINE ELSE STOP
> * do something with the line
> UNTIL 0 DO REPEAT
Only problem I see with this is, again, memory issues with large
files. You're still effectively reading in the whole file/item.
--
Kevin Powick
Ross Ferris
Kevin,
To be fair TG did say "off the top of my head", rather than "off with
my head"! As I said, the %functions provide the desired functionality,
WITHOUT having to read a multi-GB file entirely into RAM.
I think I also understand WHY TG has a preference preference for using
OSFI references .... because it means that there is nothing "differnt"
about an OS file from an application perspective ... the same is true
for the openDB product --> as I understand it you can read/write to
ANY ODBC compliant database table as if it were a local D3 file ... I
haven't used personally, but I know one of our VAR's in NZ uses to
consolidate data from multiple legacy systems
Kevin Powick
> To be fair TG did say "off the top of my head", rather than "off with
> my head"!
I didn't think I was unfairly jumping on Tony's program. Just
pointing out that as a method of dealing with "lines" within an OS
file, it to suffers from the same memory issues as other BASIC
statements and using the OSFI.
> As I said, the %functions provide the desired functionality,
Oh, yes. In my zeal to make a laughing stock out of Tony, I forgot to
thank you for pointing out the % functions as an option. My
apologies.
--
Kevin Powick
Ross Ferris
> Oh, yes. In my zeal to make a laughing stock out of Tony, I forgot to
> thank you for pointing out the % functions as an option. My
> apologies.
>
> --
> Kevin Powick
Which bit are you apologizing for (don't answer that ;-)
Tony Gravagno
Respected colleagues, the OP said he had a lot of records, not big
records. OSFI is still a likely candidate for his purposes, perhaps
not for others.
Best,
T