DBserver 1024 File Limitation Issue
Nathan Robeson
open under Windows 2000.
We have a large app using 1gb of memory and needs to open >1024 dbf files.
Registry setting or Windows issue??
Nathan
Phil McGuinness
Why would you want to key 1024 files open......... There has to be a better
way.
Imagine if you had this many Indexes open as well. I thought 254 was too
much.
Phil
---
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:usbkjar...@news.supernews.com...
Gary Stark
Nathan Robeson wrote:
Neither.
Redesign the application so that it doesn't need all those files.
--
g.
Gary Stark
gst...@RedbacksWeb.com
http://RedbacksWeb.com
Ed Ratcliffe
> We have a large app using 1gb of memory and needs to open >1024 dbf
files.
Surely you are not editing data in all of those files at the same time.
An urgent redesign is in order here.
Ed
Stephen Quinn
Suggest you move to 4th Normal Form instead of 5th Normal Form.
Those single field to a DBFs are killers on system resources aren't they<g>
--
HTH
Steve Quinn
andrea bruni
I think w2000 limit 1024 files for windows/session...
andrea
"Nathan Robeson" <nat...@voicecue.com> ha scritto nel messaggio
news:usbkjar...@news.supernews.com...
Nathan Robeson
not open so many files, you are respectfully incorrect.
This application which is sold for several Millions of Dollars is one of the
most actively used VO applications out in the field.
What does it do you might ask. It does call processing for Prepay and Post
Paid Billing 24 hours a day for some of the largest Domestic and
International wireless carriers. (Yes some of those Mobil phones you used
today went through a VO platform!)
Now why open so many files?
Because each time a person uses his wireless phone, the call comes to our
platform and opens several DBFs (depending upon the nature of the call) and
keeps them open for the duration of the call for specific reasons.Among
these reasons is the fact that every Inbound Port has its own set of open
files thus being totally object oriented. There are multiple workstations
called "Intelligent Peripheral" (IP) and each one can handle over 1000 calls
each at one time.
Why use DBFs? I'm glad you asked that question. Well, first of all, files
opened on the server use Advantage Database Server which for this
application is ideal.
Why not use Oracle or SQL based Data Retrieval? The problem with using SQL
type databases is that an SQL query (which Advantage can also do quite well)
gets processed at the Server instead of the Client.
Processing at thee Server sounds good you say, isn't that a better way of
doing it?
No, not really for this application. You see if the Server's CPU
utilization goes too high, calls will stop processing for brief periods of
time. This could never be allowed. By manual file pointer manipulation, the
local CPU gets used thus assuring that the Server CPU is running cold.
Because Advantage is so efficient across a LAN or WAN, and because we have
done our homework through the years, we have found that we can process a
very large number of calls (>100 per second ), In fact our competition can't
figure out how we process so many CPS, so you see there are good reasons for
what we are doing.
We have expanded our IP to allow for more calls to be processed. Each Client
now has 1 GIG of memory, Dual CPUs running quite fast and is NEBs compliant
(Telephony Grade Hardware that is earthquake and fire resistant etc.).
Now back to my original question, and please don't tell me not to keep so
many files open.
What is causing the 1024 DBserver Open Files Limitation?
Nathan
"Phil McGuinness" <hey...@sherlock.com.au> wrote in message
news:aq4lio$6g1hf$1...@ID-88745.news.dfncis.de...
Ed Ratcliffe
> not open so many files, you are respectfully incorrect.
Perhaps. But please do not be so certain.
> local CPU gets used thus assuring that the Server CPU is running cold.
And one possible alternate solution would be array servers.
That would keep it on the local CPU and OOP compliant.
> What is causing the 1024 DBserver Open Files Limitation?
Windows may have an upper limit on the number of open files.
Ed
Nathan Robeson
think it is probably is VO itself.
Nathan
"Ed Ratcliffe" <Gryphon...@telus.net> wrote in message
news:DXyx9.6$1F....@news0.telusplanet.net...
Ed Ratcliffe
> I am able to FOpen() in the same platform >10000 files so I'm
starting to
> think it is probably is VO itself.
You are probably right, there must be a limit on the space allocated to
store alias's, buffers etc. But back to my statement, there must be
another way. The thought of keeping that many files open is scary. In
this app of yours you must be collecting limited information about each
call, not the entire conversation. With my limited imagination I can
envision about 1 or 2 K per call. Simple to hold in memory and write out
at call termination.
Ed
pt
show it as 250, but I don't think that's right.
Maybe you can talk to Grafx and they'll do up a special version for ya.
pt
"Ed Ratcliffe" <Gryphon...@telus.net> wrote in message
news:%Dzx9.11$U93....@news2.telusplanet.net...
Nathan Robeson
Nathan
"Ed Ratcliffe" <Gryphon...@telus.net> wrote in message
news:%Dzx9.11$U93....@news2.telusplanet.net...
pt
pt
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:usdlr15...@news.supernews.com...
Phil McGuinness
snip[ To all those people who tell me I should redesign our application so
as to not open so many files, you are respectfully incorrect. ]
Maybe not........
One thing we do to you reduce the number of DBF open is use a Flex memo with
a DBF structure in each memo.
That is Flex 312 add on (should be Free... ask Brian) library allows you to
have Dbf type structures with Records etc in the memo and treat it like a
database. We also have reasons like you for a particular design
cosideration.
Now the assumption is you need the headache of the rewrite which I am sure
you do not.
I think you are correct in 1024 work area issue and from memory this is a VO
imposed limit as I think the DBserver type works areas started around this
from memory. The following code snippet demonstrates this.
When you open these databases are you using DBserver the Work areas start at
1023 up but if you open the Databases with a USE or lets say not using the
DBserver the work areas start at 1 an have a maximum work area space of
1022...
Maybe this is your problem. Therefore if I am correct then switching some
of you code to use the DBserver method would fix your problem. How you get
around this with ADS is another whole issue.
FUNCTION Start()
LOCAL oDB AS dbserver
LOCAL cDataDBF := 'test_data.dbf' AS STRING
LOCAL aData := { "a a", "a ab", "a-a", "a-ab" } AS ARRAY
SetDefault("c:\temp\")
DBCREATE( cDataDBF, {{ "myData", "C", 6, 0 }}, 'DBFCDX')
USE ( "c:\temp\" + cDataDBF ) NEW SHARED // <---- startting work area 1
oDB := DBServer{cDataDBF, DBSHARED} // <= Starting work area 1023
IF oDB:Used
// made it
endif
Phil McGuinness
------------------
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:usda7kd...@news.supernews.com...
Gary Stark
Nathan Robeson wrote:
> I am able to FOpen() in the same platform >10000 files so I'm starting to
> think it is probably is VO itself.
It probably is, as the workareas used by VO start at #1023 and decrement as
more are invoked.
But I still believe that there are other ways to do this. Ed has made a couple
of suggestions ...
>
>
> Nathan
>
> "Ed Ratcliffe" <Gryphon...@telus.net> wrote in message
> news:DXyx9.6$1F....@news0.telusplanet.net...
> > Nathan,
> >
> > > not open so many files, you are respectfully incorrect.
> > Perhaps. But please do not be so certain.
> >
> > > local CPU gets used thus assuring that the Server CPU is running cold.
> > And one possible alternate solution would be array servers.
> > That would keep it on the local CPU and OOP compliant.
> >
> > > What is causing the 1024 DBserver Open Files Limitation?
> > Windows may have an upper limit on the number of open files.
> >
> > Ed
> >
> >
> >
--
Stephen Quinn
> What is causing the 1024 DBserver Open Files Limitation?
RDD most probably - legacy of coming from a 16bit world (Clipper was 250).
Lack of foresight by the developers maybe.
Couldn't you use ArrayServers and dump their contents when the call is
complete??
--
HTH
Steve Quinn
Nathan Robeson
command.
The Alias shows as the Filename_1 up to FileName_1023 when using DBServer{}
> When you open these databases are you using DBserver the Work areas start
at
> 1023 up but if you open the Databases with a USE or lets say not using the
I am not sure where you get this information.
After peeking under the covers in the SDK:
----------------------------------------------------------------------------
--------------------------------------
FUNCTION __ConstructUniqueAlias ( cFileName AS STRING ) AS SYMBOL PASCAL
LOCAL cTryNewAlias AS STRING
LOCAL w AS DWORD
IF SLen(cFileName) == 0
cFileName := "NOFILENAME"
ELSEIF ( w := At( ".", cFileName ) )>0
cFileName := SubStr( cFileName, 1, w-1 )
ENDIF
w := 1
cTryNewAlias := Upper(cFileName)
DO WHILE VODBSymSelect( String2Symbol( cTryNewAlias ) )>0
cTryNewAlias := Upper(cFileName)+"_"+AllTrim( Str( w++ ) )
ENDDO
RETURN String2Symbol( cTryNewAlias )
----------------------------------------------------------------------------
--------------------------------------
As you can see it is very inefficient as you open more files it has to
transverse all the used aliases to find an empty one on every open.
However, this does not seem to be the problem. It seems that the problem is
somewhere in the VODBUseArea() where no source is provided.
From the DbServer Init() Method:
...
lRetCode := VODBUseArea( FALSE, rddList, oFileSpec:FullPath,
Symbol2String( symAlias ) , lShared, lReadOnly)
...
I believe the text value of symAlias is irrelevant and that in VODBUseArea()
rejects it when its internal table is greater than 1024.
Nathan
"Phil McGuinness" <hey...@sherlock.com.au> wrote in message
news:aq6pct$790u1$1...@ID-88745.news.dfncis.de...
Phil McGuinness
VO...
USE ( "c:\temp\" + cDataDBF ) ALIAS 'aaa' NEW SHARED // <---- startting
work area 1
USE ( "c:\temp\" + cDataDBF ) ALIAS 'bbb' NEW SHARED // <---- startting
work area 2
oDB := DBServer{cDataDBF, DBSHARED} // <= Starting work area 1023
oDB := DBServer{cDataDBF, DBSHARED} // <= Starting work area 1022
So the USE's count up from 1 and DBserver counts down from 1023 and I guess
they 'blowup' when they meet.
IF we could set the starting work area for the Dbserver at say 4000 you
would achieve what you want.
I have not worked how to do this yet.
Phil McGuinness
---
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:use84an...@news.supernews.com...
Nathan Robeson
For example on a Pre Pay call, the account gets decremented.
Nathan
"Stephen Quinn" <ste...@brutecom.com.au> wrote in message
news:aq6t06$75vsq$1...@ID-88745.news.dfncis.de...
Ed Ratcliffe
> No Steve because the dbfs also get written to after each call.
A perfect place for an array server.
> For example on a Pre Pay call, the account gets decremented.
Again why not an array server, and delayed DB write.
Ed
Phil McGuinness
After looking at the SDK I think you are in trouble.
VO calls the VODBSetSelect() function un the CAOVRT20.DLL which allocates
the Work area number.
VODBSetSelect(-1) allocates the TOP most number and each call after
that reduces this number down.
You can also set the number by passing it in like VODBSetSelect(900)
however you cannot change the number beyond 1024
Phil McGuinness
------------------
AA := VODBSetSelect(900)
BB := VODBSetSelect(1020)
BB := VODBSetSelect(1025)
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:use84an...@news.supernews.com...
Stephen Quinn
> IF we could set the starting work area for the Dbserver at say 4000 you
> would achieve what you want.
> I have not worked how to do this yet.
The workarea is a BIT pattern on a DWORD, so unless you have the RDD source
it's unlikely you'll be able to change it.
SDK says
--------------
STRUCT _WORKAREASTATUS
MEMBER uiSelect AS DWORD
MEMBER atomAlias AS SYMBOL
MEMBER DIM achDriver[MAXDRIVERNAME] AS BYTE
MEMBER lRecno AS LONG
MEMBER lLastRec AS LONG
// etc..
and in the INIT
-------------------
wCurrentWorkArea := VODBGetSelect() // Get
Current WA
SELF:wWorkArea := VODBSetSelect(-1) // Get New WA
for this DBServer
VODBSetSelect( INT( _CAST( wCurrentWorkArea ) ) ) // Put the old WA
back
--
HTH
Steve Quinn
Stephen Quinn
> IF we could set the starting work area for the Dbserver at say 4000 you
> would achieve what you want.
> I have not worked how to do this yet.
Nathan Robeson
I beleive your correct. I'll probably need to talk to Brian about this.
Nathan
"Phil McGuinness" <hey...@sherlock.com.au> wrote in message
news:aq7j5s$74d1a$1...@ID-88745.news.dfncis.de...
Nathan Robeson
As I mentioned to Phil, it looks like its behind the covers.
Nathan
"Stephen Quinn" <ste...@brutecom.com.au> wrote in message
news:aq7kon$7bn6t$1...@ID-88745.news.dfncis.de...
Dave Francis
Along the lines of using memo files to hold several logical DBFs, in
the past, we've compressed several low usage files into a single DBF
with just a rectype and buffer field. Depending on the rectype, the
buffer field is split into smaller fields. Report generators can cope
with this scheme quite well, but I imagine solutions like this cause
you grief in peripheral apps.
HTH
Dave Francis
"Nathan Robeson" <nat...@voicecue.com> wrote in message news:<usgclta...@news.supernews.com>...
Robert van der Hulst
On Mon, 4 Nov 2002, at 11:07:31 [GMT -0600] (which was 18:07 where I live)
you wrote about: 'DBserver 1024 File Limitation Issue'
> What is causing the 1024 DBserver Open Files Limitation?
As far as I know there is a static table in the Workarea structure of
the VO Runtime that holds the RDD info, and this table has a fixed
size of 1024.
The developers probably never expected anyone to use so many tables.
--
Robert van der Hulst
AKA Mr. Data
Vo2Jet & Vo2Ado Support
www.sas-software.nl
mrd...@sas-software.com
Nathan Robeson
many ports.
Thanks
Nathan
"Robert van der Hulst" <mrd...@sas-software.com> wrote in message
news:1303381921.2...@sas-software.nl...
Phil McGuinness
Is it possible your application can run multiple times and therefore have
say 2 x 1024 or even run on two machines co-operatiing on the incoming data.
Just a thought.
Phil McGuinness
------------------
"Nathan Robeson" <nat...@voicecue.com> wrote in message
news:uslbdi4...@news.supernews.com...
> I guess they never expected peopler to use multi threaded comm apps with
> many ports.
>
> Thanks
>
> Nathan
> >
Gary Stark
On a similar note, do individual thre processes hold references to their own
internal workarea table? If so (but I doubt they do) that could be an
alternative approach for Nathan, although I'm yet to see any convincing argument
from him to continue using this many tables.
Phil McGuinness wrote:
--
Stephen Quinn
> I guess they never expected peopler to use multi threaded comm apps with
> many ports.
This has no bearing on the number of DBFs that VO allows though
You could have used text files and hit a different limitation.
--
HTH
Steve Quinn
Dirk Herijgers
Another question : Is this limitation of 1024 tables for 1 process or is this
limitation of 1024 for each thread into a process ?
So, If the process has a main-thread and a sub-thread, can each thread has his own
1024 tables ???
Dirk
Phil McGuinness
apply.
My suggestion was if the application was started twice on the same machine
or another pointing to the same set of data would this double the workareas.
Phil
----
"Dirk Herijgers" <di...@fujitron.be> wrote in message
news:cacnsucs8jkn8j2ju...@4ax.com...
Robert van der Hulst
On Fri, 08 Nov 2002, at 13:44:37 [GMT +0100] (which was 13:44 where I live)
> Another question : Is this limitation of 1024 tables for 1 process or is this
> limitation of 1024 for each thread into a process ?
> So, If the process has a main-thread and a sub-thread, can each thread has his own
> 1024 tables ???
All VO Threads share the same Work Area table
Nathan Robeson
open files without any issue whatsoever. The old File Handles= restraint are
a dinosaur from the old DOS days.
Again, in a very active multithreaded comm application, there are times when
for speed reasons and OOPS design, that a file should stay open during the
duration of the connection. Opening and closing files takes time on a
network, and with a good operating system and lots of memory, there is no
reason to limit this to 1024 files.
Nathan
"Stephen Quinn" <ste...@brutecom.com.au> wrote in message
news:aqf0f3$9ks7a$1...@ID-88745.news.dfncis.de...
Nathan Robeson
We have entertained this possibility especially since the workstations have
2 CPUs. But this would be a royal PITA for the on sight engineers to have to
monitor and maintain. In a telephony environment, you need to pass very
stringent failover and alarming testing and a multiple instance application
creates headaches. Another problem is that the application is using several
third part device drivers to talk to the hardware and there are issues
having them work properly with multiple copies running on the same host.
Nathan
"Phil McGuinness" <hey...@sherlock.com.au> wrote in message
news:aqeqdu$972f7$1...@ID-88745.news.dfncis.de...
Stephen Quinn
The point being that you chose DBFs - there's a limit as you've found out.
Change to something that doesn't have any limitations.
If you have found no problems/limitations with text files then use them
instead.
I still think you'd be better off using ArrayServers or possibly memory
mapped files.
Another thing to look at is the SDK
- there's skeleton RDD code there to create your own implementation.
In SDT (2001/02) there's an RDD implementation (HIDDEN.RDD by Uwe Holz) use
that as a base for your own.
----------------------------------------------------------
Your app needs to be fixed now doesn't it??
Waiting for the VO Development team to change the RDD and test it, and it
will need testing so as not to break current code, doesn't help you now if
the patch doesn't come out for 6 months.
That is if they decide to even make the changes, after all so far your the
only developer asking for this change.
Anyone else need more than 1024 open DBFs??
--
HTH
Steve Quinn
Nathan Robeson
Nathan
"Stephen Quinn" <ste...@brutecom.com.au> wrote in message
news:aqhjtg$a79h1$1...@ID-88745.news.dfncis.de...
Rob's Ozemail
How 'real-time' does your app need to be? Unless it's for mobile phone hire
like you get at airports etc. could pre and post paid have a bit of slack in
them?
What I'm thinking of is buffering the CDRs (into memory if poss, or to disk
if you have to) after you get to the file limit, and then re-reading them
when you have the file limit down? Could you pass them through an FEP and
do any processing there?
Just a thought, not knowing the architecture of your app....
Good to see VO being used in Telecoms!
--
Rob Grattan
R&D Software Pty. Ltd.
Email: r...@rndsoftware.com.au
Fixed: (+61) (0)7-5449-1919
Mobile: (+61) (0)412-499753
Fax: (+61) (0)7-3009-0560
P.O. Box 126
Eumundi
QLD 4562
Australia
Nathan Robeson
The CDRs (Call Detail Records) are one thing but there is also the issue of
decrementing the account or refilling the account with a Pre-paid Card or
credit Card. Yes there are ways of closing files more often but then there
is the possibility that a lot of people want to the same thing at any given
time and the workarea is unavailable at that time.
If I don't take a muti-threaded approach to things there is no problem at
all as all the Subscribers can use the same workareas synchronously.
Nathan
"Rob's Ozemail" <bgra...@ozemail.com.au> wrote in message
news:aqnfqk$mlc$1...@newsreader.mailgate.org...
Robert van der Hulst
On Mon, 11 Nov 2002, at 10:32:24 [GMT -0600] (which was 17:32 where I live)
> The CDRs (Call Detail Records) are one thing but there is also the issue of
> decrementing the account or refilling the account with a Pre-paid Card or
> credit Card. Yes there are ways of closing files more often but then there
> is the possibility that a lot of people want to the same thing at any given
> time and the workarea is unavailable at that time.
> If I don't take a muti-threaded approach to things there is no problem at
> all as all the Subscribers can use the same workareas synchronously.
What about using some kind of FIFO queue (Array ?) to store the
requests and have one thread do all the DBF related processing from
this queue.
Nathan Robeson
This could work for CDRs but Refill Cards and Credit Card Replenishment
require real time interaction.
Nathan
"Robert van der Hulst" <mrd...@sas-software.com> wrote in message
news:196360942906....@sas-software.nl...
Robert van der Hulst
On Mon, 11 Nov 2002, at 12:36:59 [GMT -0600] (which was 19:36 where I live)
> This could work for CDRs but Refill Cards and Credit Card Replenishment
> require real time interaction.
Well if you would use it for CDRs you may reduce the number of open
files. It is something to start with. I also don't know how
complicated and timeconsuming your lookups/calculations are, but I can
imagine a system that has a dozen or so separate threads managing the
database access.
CMc
> Hi Nathan,
>
> On Mon, 4 Nov 2002, at 11:07:31 [GMT -0600] (which was 18:07 where I live)
> you wrote about: 'DBserver 1024 File Limitation Issue'
>
> > What is causing the 1024 DBserver Open Files Limitation?
>
> As far as I know there is a static table in the Workarea structure of
> the VO Runtime that holds the RDD info, and this table has a fixed
> size of 1024.
> The developers probably never expected anyone to use so many tables.
>
Thanks,
Chris
Johan Nel
Yes this is still the limitation. You need to externalise your connections to tables into a shared collection that handles your interface to the database. Very unlikely that any process ever will need 1024 open tables to perform a task.
Regards,
Johan Nel
Vulcan VIP
Durban, South Africa.
CMc
Thanks,
Chris
Johan Nel
No, what I do is to have a STATIC var that basically works on the following principle:
STATIC LOCAL aDbServers AS ARRAY
aDbServers := {}
Then I have a class/function/method that interacts with my open DbServers
FUNCTION ServerOpen(sAlias AS STRING)
IF (nPos := Ascan(aDbServers, {|a| a[1] == sAlias})) > 0
oDb := aDbServer[nPos, 2]
aDbServer[nPos, 3] += 1
ELSE
oDb := <Open your server>
Aadd(aDbServers, {sAlias, oDb, 1})
ENDIF
RETURN oDb
FUNCTION ServerClose(sAlias AS STRING)
IF (nPos := Ascan(aDbServers, {|a| a[1] == sAlias})) = 0
// Throw error the server is not open
ELSE
aDbServers[nPos, 3] -= 1
IF aDbServers[nPos, 3] <= 0
ADel(aDbServers, nPos)
ENDIF
ENDIF
RETURN lClosed
FUNCTION ServerGet(sAlias)
nPos := AScan(aDbServers, {|a| a[1] == sAlias})
RETURN aDbServers[nPos, 2]
Basically all my interactions with servers are handled by the above three FUNCTIONS that keep track of how many references to each one is active. You have only 1 "workarea" for each DbServer.
HTH,
Johan
JohnMartens
This in case I made some code addition and forget to close one or more
servers. Doesnt take long to do so.
FOR wAlias := 1 UPTO 1024
IF .NOT. ( ALIAS(wAlias) == NULL_STRING )
dwMethresult += 1
ENDIF
NEXT
John
Johan Nel schreef op 29-1-2015 om 14:58:
Grzegorz R.
Sorry to dig old subject out but just recently I have been hit with a problem that when on one PC many windows with many DBF's have been opened (alhough far from 1024) another PC could not add a record to a particular shared DBF - Append() yelded FALSE. As soon as I closed some windows on another PC all returned to proper work. Where should I seek for error - is it a local app (VO 2.8) or server (QNAP file sever) limit. If it's is the app setup like in old CLIPPER (Set files = xxx) where I can change it?
Best regards
Gregory
Wolfgang Riedmann
> Sorry to dig old subject out but just recently I have been hit with a
> problem that when on one PC many windows with many DBF's have been
> opened (alhough far from 1024) another PC could not add a record to a
> particular shared DBF - Append() yelded FALSE. As soon as I closed
> some windows on another PC all returned to proper work. Where should
> I seek for error - is it a local app (VO 2.8) or server (QNAP file
> sever) limit. If it's is the app setup like in old CLIPPER (Set files
> = xxx) where I can change it?
closing a table on another PC it sounds really like a limit on your NAS
(probably Linux, but maybe also Windows Storage Server if it is a
highend NAS).
Wolfgang
--
Grzegorz R.
Thank you. I thought so but just wanted to be sure...
Gregory