UniVerse 32-bit 2GB file size limit

[Mike Preece]

Hello,

 

Can someone please tell me, for sure, what the 2GB file size limit is?

 

Is it the amount of data that can be written to a file?

 

...or...

 

Is it the amount of physical space allocated to the file (including for overflow)?

 

...or...

 

Something else.

 

I just want to confirm that I am doing the right thing by monitoring the results of a *nix "ls -la" command - and that files will go "pop" if/when the number of bytes shown there approach/exceed 2,147,483,648. I am only interested in non-distributed, static hashed files, btw.

 

Thanks,

Mike.

 

 

 

[Martin Phillips]

Hi Mike,

 

It is the physical size of the file, including overflow and unused space. In the case of a dynamic file, both portions are limited to 2Gb but that doesn’t mean you can store 4Gb of data.

 

 

Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton NN4 6DB, England
+44 (0)1604-709200

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

[Anthony Youngman]

On 04/01/13 17:13, Martin Phillips wrote:
> Hi Mike,
>
>
>
> It is the physical size of the file, including overflow and unused
> space. In the case of a dynamic file, both portions are limited to 2Gb

> but that doesn�t mean you can store 4Gb of data.
>
Bear in mind that a 32-bit int goes up to 2Gb if it's signed, 4Gb if it
isn't. So - assuming 32-bit signed arithmetic - 2Gb is the largest size
a file can be before any attempt to indicate your position in the file
wraps around to be negative.

If you think that's bad, I worked with SCO Unix, and that was restricted
to 2Gb partitions, presumably for the same reason. I had to upgrade
several BIOSes when the disks were upgraded beyond 4Gb, etc etc. A lot
of things in computing blow up when they hit a 2Gb or 4Gb limit.

Cheers,
Wol

[Kevin King]

I would definitely be more interested in a file approaching the 2Gb limit vs. one that has breached it.  Give yourself a fighting chance to do something before things get crazy, right?

Maybe schedule something like this to monitor nightly and send the results via email?

find /path/to/directory -size +3145728

The -size of 3145728 is that many blocks at 512-byte blocks (AIX) which is roughly 1.5G. 

-K

 

 

 

--

[Saralinda Christopher-Evans]

Does any system other than Ult Plus have POVF at TCL, or does this not take care of that?

---

From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Kevin King
Sent: Friday, January 04, 2013 1:34 PM
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit

[Dick Thiot]

Saralinda,

 

I think that POVF originated with the early Microdata systems and was available through R83 for sure.  It is also available on D3/Windows and mvBase (and maybe D3/Linux too) and may be on others as well.  I suspect that it is less common on systems that utilize the underlying O/S for file storage like QM or jBASE rather than carving out a block of the host as a virtual disk.

 

Dick.

[Tony Gravagno]

Since you mention it: POVF is not used in the D3 Windows FSI. As you said, this is because FSI uses the underlying file system for files, so the pool of "frames" for file space is only limited to disk space. And since D3 files can reside anywhere, not just the local drives, then there's virtually no limit. D3 Linux still uses overflow and thus POVF is valid. However, we used to judge total available space by POVF, and since we can now store data in the host OS, POVF only provides part of the information needed now for complete system management. The D3 Windows VME does have POVF for the blob, but user files generally should not be in the VME. These days it's more for system data, spooler files, and other housekeeping.

 

Tony Gravagno                

Nebula Research and Development       

TG@ remove.pleaseNebula-RnD.com   

http://Nebula-RnD.com/blog    

Visit http://PickWiki.com! Contribute! 

http://Twitter.com/TonyGravagno         

<ad>Nebula R&D provides D3/mvBase licenses, support, and other services to end-users and VARs</ad>

[Saralinda Christopher-Evans]

Have been on Ultimate, then Ult Plus PICK for a long time. Learning from these posts. --Saralinda

---

From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Tony Gravagno
Sent: Friday, January 04, 2013 5:33 PM
To: mvd...@googlegroups.com
Subject: RE: [mvdbms] UniVerse 32-bit 2GB file size limit

--

[Tony Gravagno]

My first Pick system in 1982 was Ultimate and I worked on them for many years. If you ever need to migrate from Ultimate you may find that D3 is the closest match. It supports Ultimate verbs, file and system maintenance is virtually the same, and BASIC supports nuances like the //syntax in Executes.

HTH

T

[Saralinda Christopher-Evans]

Cool. Good to know. Thank you. --Saralinda

---

From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Tony Gravagno

Sent: Friday, January 04, 2013 7:45 PM

To: mvd...@googlegroups.com
Subject: RE: [mvdbms] UniVerse 32-bit 2GB file size limit

--

[MikeR]

I thought all of the 2GB limit went away when the file type changed. Microsoft introduced ntfs and on unix extfs and more recently some even more fancy file types.

After all, if there was a 2GB file limit, how come I can store my 20GB virtual machine on a drive??


Mike

[Charlie Noah]

I didn't know D3 supported Ultimate // executes. I may never use it, but it's good to know.

Happy New Year, Tony!
Charlie

--

[Bob Wyatt]

Several of the mvdbms products have remained as 32-bit applications, so the 2GB limit remained – it was all that they could address. Some later evolved into 64-bit compatible mvdbms applications (so they could run on a 64-bit implementation), but were still a 32-bit application at its core, thereby retaining the 2GB file size limit. And now we have true 64-bit mvdbms products that have the 2GB restriction lifted.

The underlying Operating System support for > 2GB files had to evolve before the mvdbms products could evolve.

--

[Saralinda Christopher-Evans]

Does that mean that other PICK / MultiValue flavors do not support execute statements within programs? Thank you, Saralinda

 

 

---

From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Charlie Noah
Sent: Saturday, January 05, 2013 6:41 AM
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit

[Rick Weiser]

All flavors support some version of EXECUTE.

[Charlie Noah]

Hi Saralinda,

Some flavors may not support the // form of execute. An example is

EXECUTE 'SP-EDIT UR ': HOLD.FILE.NO // IN. < TEXT OUT. > SCREEN

which in other flavors might be

DATA TEXT
EXECUTE 'SP-EDIT UR ': HOLD.FILE.NO CAPTURING SCREEN

At least I think I remember that IN. passes data rather than a select list. Someone please correct me if I'm wrong - it's been a lot of years since I've worked with Ultimate. I still have an old Honeywell box that ran Ultimate somewhere out in the garage, though. Don't ask me why I still have it.

Charlie

[Tony Gravagno]

From: Charlie Noah

>I didn't know D3 supported Ultimate // executes. I may never use it,
but it's good to know.
>Happy New Year, Tony!

Happy New Year to you Charlie, and everyone here!

The syntax of D3 Execute is as follows:

execute tcl.exp {[stacking|,//in.<] exp} {[passlist|,//select.<]
list.var} {[rtnlist|,//select.>] list.var } {[returning|setting] var}
{[capturing|,,//out.>] var}

Pick style syntax is first followed by optional Ultimate style
(",//op.>"). There's no reason for non-Ultimate people to use the more
complex syntax but it's there for Ultimate migrations - like a lot of
features are in competitor platforms just for D3 migrations.

( Were the Jersey people on drugs with that syntax? )

T

[Saralinda Christopher-Evans]

Interesting. What is the UR for? What does one assign a specific HOLD.FILE.NO?

 

Thank you,

Saralinda

 

---

From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Charlie Noah

Sent: Saturday, January 05, 2013 10:41 AM

[Ed Clark]

For the SP-EDIT command the U option allows you to view/edit spooler entries created by accounts other than the one you are logged in to, assuming you have SYS2 permission on your account. The R command overrides the settings the spooler entry was created with, so for example is the job was created with:
SP-ASSIGN HSF3
which holds it in the spooler and suppresses output, you could
SP-ASSIGN F5
SP-EDIT R 3
and then spool the job in SP-EDIT an it would print to queue#5.
You would get the hold file number from a LISTPEQS listing, or from system(20) if you had just printed it yourself.

There is online documentation for D3 at http://www.tigerlogic.com/tigerlogic/pick/support/documentation/d3nt/90/refman/index.htm. A lot of the commands are the same as in Ultimate so if you don't have manuals for ultimate, that might be a starting point.

On Jan 5, 2013, at 2:38 PM, Saralinda Christopher-Evans <saralinda.chr...@gmail.com> wrote:

> Interesting. What is the UR for? What does one assign a specific HOLD.FILE.NO?
>
> Thank you,
> Saralinda
>

> From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of Charlie Noah
> Sent: Saturday, January 05, 2013 10:41 AM
> To: mvd...@googlegroups.com
> Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit
>
> Hi Saralinda,
>
> Some flavors may not support the // form of execute. An example is
>
> EXECUTE 'SP-EDIT UR ': HOLD.FILE.NO // IN. < TEXT OUT. > SCREEN
>
> which in other flavors might be
>
> DATA TEXT
> EXECUTE 'SP-EDIT UR ': HOLD.FILE.NO CAPTURING SCREEN
>
> At least I think I remember that IN. passes data rather than a select list. Someone please correct me if I'm wrong - it's been a lot of years since I've worked with Ultimate. I still have an old Honeywell box that ran Ultimate somewhere out in the garage, though. Don't ask me why I still have it.
>
> Charlie
>
> On 01-05-2013 9:23 AM, Saralinda Christopher-Evans wrote:
>> Does that mean that other PICK / MultiValue flavors do not support execute statements within programs? Thank you, Saralinda
>>
>>

[Charlie Noah]

Thanks for following up on this, Ed. Nicely done, too!

Charlie

[Charlie Noah]

They were probably on the same drugs Dick appeared to be on at times. No
disrespect to Dick, he just seemed to be coming way out of left field
sometimes.

Charlie

[Saralinda Christopher-Evans]

Thank you.

I remembered, after I asked, that I knew the U, but I simply didn't ever
deal with these other things (one person shop).

I just joined this group and am amazed at the information provided in the
last 24 hours. The future only knows if I will find a job using PICK,
although it will be nice to use the skills I've developed for the last 25
years. I'm a a lot more aware of these online MV groups since my job search
began. What an amazing resource! The nice thing about the google group seems
to be that my posts don't seem to be broadcast to the web.

Thanks again.

[Saralinda Christopher-Evans]

My first examples included the //select type of statements as that was the
example I saw displayed in the code written by The Software Group used for
one department and written by one of the attorneys for a different
department when he was a UT student.

During the last few years it was simply things like (although I don't think
this was originally an option - upgrades):

EXECUTE \SELECT XXX WITH XXX \, PASSLIST LIST, RTNLIST LIST, CAPTURING
S.RSLT

(S.RSLT just made up variable short for "screen result" to avoid screen
display -- of course could have used the captured "screen" results if I was
parsing it to make a report to parse and shoot out in email, move to a UNIX
file or print)

All of this is wonderful. Thank you.

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of

Charlie Noah
Sent: Saturday, January 05, 2013 8:55 PM
To: mvd...@googlegroups.com

[Charlie Noah]

Hi Saralinda,

I wish you the best of luck on the job front. MV jobs seem to be rather
scarce these days, although there are some out there. I've been looking
for over 2 years, but I have a limitation you (hopefully) don't have.
I'm disabled and need to work remotely. Everyone talks about it, but few
employers are willing to jump into it. I expect I'm not even counted in
the unemployment numbers anymore. :-(

Regards,
Charlie

[Saralinda Christopher-Evans]

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of
Charlie Noah
Sent: Sunday, January 06, 2013 2:51 AM
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit

Hi Saralinda,

[Saralinda Christopher-Evans]

Wow. Good luck to you too. So sorry.

You're right. I've seen a few since October, but not a lot. Lot more in the
DFW area than in the Austin area. I'm probably going to end up retooling. I
have not limited myself to only applying for MV jobs, and I definitely
haven't been looking at telecomuting jobs.

--Saralinda

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of

Charlie Noah
Sent: Sunday, January 06, 2013 2:51 AM
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit

Hi Saralinda,

[Jan Van Schalkwyk]

This thread has gone sooo way of topic?

[Charlie Noah]

Jan,

Although it did go off-topic, there were teaching moments here, and several people (myself included) learned something they didn't know about MV, and I've been at it 34 years. Do you have a problem with that?

Charlie

--

[Jan Van Schalkwyk]

No, I only have a problem with OFF Topic discussions.

Nothing stops you from starting a NEW thread with a NEW subject line.

Peace Bro!

[geneb]

On Mon, 7 Jan 2013, Jan Van Schalkwyk wrote:

> No, I only have a problem with OFF Topic discussions.
> Nothing stops you from starting a NEW thread with a NEW subject line.
>

Topic drift happens and it's often educational or interesting.

Also, don't top post. :)

g.

--
Proud owner of F-15C 80-0007
http://www.f15sim.com - The only one of its kind.
http://www.diy-cockpits.org/coll - Go Collimated or Go Home.
Some people collect things for a hobby. Geeks collect hobbies.

ScarletDME - The red hot Data Management Environment
A Multi-Value database for the masses, not the classes.
http://www.scarletdme.org - Get it _today_!

[Anthony Youngman]

On 05/01/13 15:23, Saralinda Christopher-Evans wrote:
> Does that mean that other PICK / MultiValue flavors do not support
> execute statements within programs? Thank you, Saralinda
>

And you're aware of the difference between PERFORM and EXECUTE?

(That said, as an old PI programmer, even I'm not sure which is which -
if it was important I'd have to look it up. iirc PERFORM keeps the same
execution context, and EXECUTE creates a new one.)

Cheers,
Wol

[Ed Clark]

Difference between EXECUTE and PERFORM vary wildly by platform.
On d3/Reality, and most of the platforms that came from pick licensees (so I expect also ultimate), EXECUTE and PERFORM are exact synonyms. They create a new execution context.
On universe in ideal/pick/reality/in2 flavors, EXECUTE is like EXECUTE on D3, but PERFORM doesn't create a new execution context. In universe information and pipen flavors, EXECUTE acts like PERFORM and does not create a new context either. These features can be modified in any flavor with a $options statement. There are also differences in what optional clauses EXECUTE and PERFORM can take. Generally on universe, PERFORM isn't allowed to have CAPTURING/RETURNING or list-passing clauses.
On unidata, EXECUTE works like universe's PERFORM, but allows clauses like CAPTURING.

For example, given a simple 2-line program:
EXECUTE "SELECT MD 'BASIC' "
EXECUTE "LIST MD"

universe in ideal/pick/in2/reality flavors will list the entire MD, but in information and IN2 flavors, and on unidata, will list only the item 'BASIC'
D3 will list only 'BASIC'. It creates a new context for the execute, but passes back the external select list for a later execute to use.
You would expect mvbase to work like D3, but in this case it works like universe pick and prints the whole MD. mvbase creates a separate context, but doesn't pass back the external list.

"Execute context" involves what select lists are available, what @variables are available, and whether or not output will go to the printer by default if the EXECUTEing program has its printer on.

In universe, some @variables are "stacked" and some aren't. For example, the @RECUR1 variable is stacked. An EXECUTEd program gets a clean copy of it, but a PERFORMed program shares the parent's copy. On universe, the @USER1 variable is not stacked--it retains it's value through execute levels. On unidata, almost all @variables are non-stacked, because universe never creates a new execution context. But the @RECUR1 variable on unidata is specifically stacked.

[Saralinda Christopher-Evans]

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of
Anthony Youngman
Sent: Monday, January 07, 2013 3:10 PM
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] UniVerse 32-bit 2GB file size limit

--
You received this message because you are subscribed to the "Pick and
MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com To unsubscribe, email to:
mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

Don't think Ult Plus had PERFORM. Doesn't mean it didn't. I'd just never
seen it used. Thank you, Saralinda

[Saralinda Christopher-Evans]

Wonderful lesson (cleared out everything because of the top/bottom stacking
thing). Thank you. What is a $options statement? I used $INCLUDES to pull in
things I'd set up somewhere else such as file definitions (who wants lines
of that to read through in pgm after pgm) or common variables (clear screen,
null, etc.).

[Kevin Powick]

On Monday, 7 January 2013 18:44:18 UTC-5, Saralinda Christopher-Evans wrote:

 What is a $options statement?

http://www.tigerlogic.com/tigerlogic/pick/support/documentation/d3nt/90/refman/index.htm#pickbasic-flashbasic/dollar_options_statement.htm

--

Kevin Powick 

[Ed Clark]

$options on D3 is somewhat different from $options on universe.
On D3 $options is mostly about variable name compatibility. If you were migrating from R83 to D3 and had some code like:

001 DIM STATUS(5)
002 STATUS(3)=45
003 CRT STATUS(3)

this would not compile on D3, because D3 has a function STATUS(). STATUS has become a reserved word. If you want to keep using STATUS() as an array, you can add
$OPTIONS R83
After which you can use STATUS() as an array, but you also lose the usage of the STATUS() function. You also loose all the other functions introduced in D3 like fmt(), dquote(), minimum(), etc. For this reason you probably don't want to use $OPTIONS in D3.

$options on universe is mostly about emulation. For example $OPTIONS PERF.EQ.EXEC will compile PERFORM as EXECUTE. $OPTIONS LOCATE.R82 changes the sort order used by "AR" and "DR" in the LOCATE statement. $OPTIONS TIME.MILLISECOND controls whether TIME() returns fractional seconds or not.

There are several dozen $OPTIONS options on universe, while there are effectively only 3 on d3.

[Don Robinson]

Gene,

 

I usually find your posts interesting and useful but I disagree with "Also, don't top post. :)"

 

All email programs that I've seen, put you at the top and push the existing text down so it's natural to put new stuff at the top.

 

Also, with bottom posting you have to look for the new stuff because it's mixed in with the old.

 

But, I can handle it either way. ;-)

 

Cheers,

Don Robinson

 

-- You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+unsub...@googlegroups.com

[Art Martz]

On 01/08/2013 09:55 AM, Don Robinson wrote:

Gene,

 

I usually find your posts interesting and useful but I disagree with "Also, don't top post. :)"

 

All email programs that I've seen, put you at the top and push the existing text down so it's natural to put new stuff at the top.

 

Also, with bottom posting you have to look for the new stuff because it's mixed in with the old.

This is a religious war that depends on how you read emails. The best solution is to trim quoted text so that it all fits in a window without paging up or down, that way it doesn't really matter.
Art

[geneb]

On Tue, 8 Jan 2013, Don Robinson wrote:

> Gene,
> �
> I usually find your posts interesting and useful but I disagree with "Also, don't top post. :)"
> �
> All email programs that I've seen, put you at the top and push the existing text down so it's natural to put new stuff at the top.
> �
> Also, with bottom posting you have to look for the new stuff because it's mixed in with the old.
> �
> But, I can handle it either way. ;-)

> �

It was partly a joke. However, top posting makes discussions really hard
to follow because it's the exact reverse of how a conversation flows. I
know that many email programs (incorrectly) stick you at the top when
replying to a message, but that doesn't mean you can't overcome the
problem. (well except with Outlook. That sucker is a _nightmare_ on
wheels) I also understand that I'm tilting at windmills - especially
amongst the mouth-breathing MBA set, but it's a fun windmill to tilt at on
occasion. :)

[Anthony Youngman]

On 08/01/13 14:55, Don Robinson wrote:
> Gene,
>
> I usually find your posts interesting and useful but I disagree with
> "Also, don't top post. :)"

A. This is why.

Q. Why shouldn't you top post?

>
> All email programs that I've seen, put you at the top and push the
> existing text down so it's natural to put new stuff at the top.
>

You're not using the same mail client as me ...

That trend of putting the cursor at the top was started by Microsoft. It
has ALWAYS been recognised that you should add to the bottom.

> Also, with bottom posting you have to look for the new stuff because
> it's mixed in with the old.

Which is how it's supposed to be. It's also supposed to encourage people
to trim because in the old days (and even in modern days, with download
limits) people don't want to have to pay to download the same stuff
multiple times.

>
> But, I can handle it either way. ;-)
>

As can most of us :-) But I don't like it when top posting leads to a
massive email with bugger all new content

> Cheers,
> Don Robinson

Cheers,
Wol

[Jan Van Schalkwyk]

To be or not to be

Hot and sunny in Australia with many fires raging :-(

[MikeR]

Interesting stuff - errm - not!

Could we return to the OP topic about a 2GB limit?

In terms of the filesystem, the 2GB limit only applies if you have a FAT32 filesystem. Since windows has NTFS and Linus extfs - there is no 2GB file limit.

But is there some limit within the hashing mechanism of the MV database?

[Anthony Youngman]

It's nothing to do with the hashing mechanism - and everything to do
with the file pointer.

The 2Gb limit applies on NTFS, and I believe it doesn't necessarily
apply on FAT32. 32-bit *applications* use a 32-bit file pointer (by
default) and as such are limited to 2Gb files.

So you need "special" files using a 64-bit file pointer to have a UV
file greater than 2Gb. Oh for the old PI system of segmented files :-)

Lots of 32-bit stuff breaks at 2 or 4Gb - file systems, bioses (who
remembers 6Gb disks rendering systems unbootable?) etc etc. Our old SCO
box the filesystem was limited to 2Gb, not just the files...

Cheers,
Wol

[Symeon Breen]

No this is the most significant piece of text in this window, and
therefore is at the top - you have already read what is below beforehand,
probably many times, so no need to read it all again in order to get to the
important thing which is my reply that I have just given to you and placed
at the top of the message :)

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of
Anthony Youngman
Sent: 08 January 2013 23:15
To: mvd...@googlegroups.com

--
You received this message because you are subscribed to the "Pick and
MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com To unsubscribe, email to:

mvdbms+un...@googlegroups.com

For more options, visit http://groups.google.com/group/mvdbms

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2221 / Virus Database: 2637/5518 - Release Date: 01/08/13

[John Lorentz]

On Wed, Jan 9, 2013 at 4:55 AM, Symeon Breen <sym...@gmail.com> wrote:
> No this is the most significant piece of text in this window, and
> therefore is at the top - you have already read what is below beforehand,
> probably many times, so no need to read it all again in order to get to the
> important thing which is my reply that I have just given to you and placed
> at the top of the message :)

Totally disagree.

Why should I have to scan all the way through the many lines of text
that you've carried forward, just to find the one bit that you're
responding to?

John

[Symeon Breen]

In that scenario top or bottom posting is the same !


I think the issue you are referring to here is trimming of the original
text.

-----Original Message-----
From: mvd...@googlegroups.com [mailto:mvd...@googlegroups.com] On Behalf Of
John Lorentz
Sent: 09 January 2013 16:20
To: mvd...@googlegroups.com
Subject: Re: [mvdbms] Top verses bottom posting. (Was UniVerse 32-bit 2GB
file size limit)

--
You received this message because you are subscribed to the "Pick and
MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com To unsubscribe, email to:
mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms
-----
No virus found in this message.
Checked by AVG - www.avg.com

Version: 2012.0.2221 / Virus Database: 2637/5519 - Release Date: 01/08/13

[John Lorentz]

On Wed, Jan 9, 2013 at 8:29 AM, Symeon Breen <sym...@gmail.com> wrote:
>
> I think the issue you are referring to here is trimming of the original
> text.

However, I've found that people who generally top-post also generally
don't trim. (Such as in your previous post.)

I only top-post if the previous e-mail I'm responding to has a
formatted layout that doesn't allow for in-line responses.

John

[MikeR]

> Could we return to the OP topic about a 2GB limit?
>
> In terms of the filesystem, the 2GB limit only applies if you have a
> FAT32 filesystem. Since windows has NTFS and Linus extfs - there is no
> 2GB file limit.
>
> But is there some limit within the hashing mechanism of the MV database?

It's nothing to do with the hashing mechanism - and everything to do
with the file pointer.

The 2Gb limit applies on NTFS, and I believe it doesn't necessarily
apply on FAT32. 32-bit *applications* use a 32-bit file pointer (by
default) and as such are limited to 2Gb files. 

 

Your post confuses me greatly!

I can store large files in Windows on NTFS - I have 20GB virtual machines - and this is 32bit windows

Similarly, I can store large files on 32bit Linux on ext2fs - ripped DVD and virtual machines for example.

I know that I cannot copy this to a fat32 filesystem (such as a usb stick)

If the 2GB limit is nothing to do with the database engine and only to do with the file system then, it would appear, we are discussing a problem that was solved 10 years ago. 

Regards

Mike

 

[Anthony Youngman]

On 09/01/13 17:39, MikeR wrote:
> I can store large files in Windows on NTFS - I have 20GB virtual
> machines - and this is 32bit windows
> Similarly, I can store large files on 32bit Linux on ext2fs - ripped DVD
> and virtual machines for example.
> I know that I cannot copy this to a fat32 filesystem (such as a usb stick)
>
> If the 2GB limit is nothing to do with the database engine and only to
> do with the file system then, it would appear, we are discussing a
> problem that was solved 10 years ago.

Yes it is a problem that was solved many years ago. The problem isn't
the file system - it is that a lot of APPLICATIONS haven't been updated.

A lot of applications still use the old 32-bit routines to access files.
As such, they can't get beyond 2Gb (or 4Gb if the long is unsigned).

And if the file format only allocates four bytes for storing position
information (as I guess must be true of the default UV files) then
equally those files can only go to 2 or 4Gb for the same reason. Which
is why UV has 64BIT files that use a long-long offset pointer.

Cheers,
Wol

[MikeR]

On Thursday, January 10, 2013 12:19:24 AM UTC, Wol wrote:

On 09/01/13 17:39, MikeR wrote:
> I can store large files in Windows on NTFS - I have 20GB virtual
> machines - and this is 32bit windows
> Similarly, I can store large files on 32bit Linux on ext2fs - ripped DVD
> and virtual machines for example.
> I know that I cannot copy this to a fat32 filesystem (such as a usb stick)
>
> If the 2GB limit is nothing to do with the database engine and only to
> do with the file system then, it would appear, we are discussing a
> problem that was solved 10 years ago.

Yes it is a problem that was solved many years ago. The problem isn't
the file system - it is that a lot of APPLICATIONS haven't been updated.

A lot of applications still use the old 32-bit routines to access files.

I am getting confused here!

Firstly, the discussion was about filesystems and database engines. You now introduce applications. What is an application (in the context of a UV/QM/jBase structure)?

Clearly, if you are implying that UV is the 'application' and this has not been updated then I understand your point. And the problem is inbuilt in the database engine.

Secondly, you say that 'a lot of applications' - how does one identify these applications?

Thanks

Mike

[Anthony Youngman]

On 11/01/13 11:29, MikeR wrote:
>
>
> On Thursday, January 10, 2013 12:19:24 AM UTC, Wol wrote:
>
> On 09/01/13 17:39, MikeR wrote:
> > I can store large files in Windows on NTFS - I have 20GB virtual
> > machines - and this is 32bit windows
> > Similarly, I can store large files on 32bit Linux on ext2fs -
> ripped DVD
> > and virtual machines for example.
> > I know that I cannot copy this to a fat32 filesystem (such as a
> usb stick)
> >
> > If the 2GB limit is nothing to do with the database engine and
> only to
> > do with the file system then, it would appear, we are discussing a
> > problem that was solved 10 years ago.
>
> Yes it is a problem that was solved many years ago. The problem isn't
> the file system - it is that a lot of APPLICATIONS haven't been
> updated.
>
> A lot of applications still use the old 32-bit routines to access
> files.
>
> I am getting confused here!

Sorry

>
> Firstly, the discussion was about filesystems and database engines. You
> now introduce applications. What is an application (in the context of a
> UV/QM/jBase structure)?

The application is whatever uses the file system, ie UV or QM or jBase
themselves.

> Clearly, if you are implying that UV is the 'application' and this has
> not been updated then I understand your point. And the problem is
> inbuilt in the database engine.
>
> Secondly, you say that 'a lot of applications' - how does one identify
> these applications?

Basically, anything that crashes or corrupts a file when the file goes
over the magic 2/4Gb limit, and the integer being used to track position
is a 32-bit int.

This first came to the fore in the Win95 era, as 6Gb drives started
appearing. Anything predating then (and probably a fair bit of stuff
after, too :-( is a candidate for this problem.

I guess there's still an awful lot of software out there that suffers
from this, but mostly we don't notice because, to be honest, how often
do we get files over 2Gb? The odd app (eg UV :-) may have a lot, but
most apps would probably never get near it.
>
> Thanks
> Mike
>
Cheers,
Wol

[John Lorentz]

On Fri, Jan 11, 2013 at 9:41 AM, Anthony Youngman <wolw...@gmail.com> wrote:
> I guess there's still an awful lot of software out there that suffers
> from this, but mostly we don't notice because, to be honest, how often
> do we get files over 2Gb? The odd app (eg UV :-) may have a lot, but
> most apps would probably never get near it.

I've worked on several applications that have had files well over 2GB,
which is how I learned about using distributed files on PI/Open and
UV.

Unfortunately, the first time I encountered the problem with a file
going over 2GB was the call log on a 9-1-1 system. It's not good when
the operators suddenly couldn't save any info in the system from
emergency calls.

John

[Martin Phillips]

Hi all,

Since QM has now appeared in this discussion, let me set out the QM position on this though I shall wander about a bit first...

As others have already said, the file size limit comes from two places; the operating system and the application/database. Most
modern operating systems support files well over 2Gb by supplying APIs that allow positioning to addresses passed as 64 bit values.

The original poster posed the question specifically for UniVerse. UV files normally use 32 bit pointers internally and are therefore
limited to 2Gb (many things treat addresses as signed values so the 4Gb limit apparently offered by 32 bit addressing doesn't tend
to apply). Long ago, UV introduced a file format variation that can be selected on creating the file to use 64 bit pointers
internally. This means that, so long as the operating system supports files over 2Gb, so does UV.

So, what about QM?

Since 2005, QM has supported files over 2Gb and, unlike UV last time I checked, this includes sequential files (yes, we had a user
who wanted to import a 6Gb text file). Internally, QM uses 32 bit pointers but they are group numbers, not file byte offsets. This
means that we can, with the largest group size, support hashed files up to 16Tb, more than enough for all but the most avid data
hoarder.


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton NN4 6DB, England
+44 (0)1604-709200

[Mike Preece]

Hi Mike,

 

I've attached a pdf from Rocket all about changing from 32-bit to 64-bit on UV. I hope this clarifies.

 

Best regards,

Mike.