Memory allocation in 5a9, NDBM related
Frederick True
I've been using with 4.036 under the most recent release (5a9) and I'm
seeing some strange behavior with respect to memory allocation when using
large NDBM files bound to associative arrays. Not having had the time to
mull over the entire 5a9 distribution source with more than a casual glance,
I'm hoping maybe someone else has noticed this problem and can explain
what's going on before I dig too far.
The core of the script is basically akin to:
--
require NDBM_File;
tie(%ARR,NDBM_File,'/scratch/temp/bigfile.0594',0x202,0640);
main: while (<>) {
chop; # strip record separator
($key,$data) = unpack('a10a5',$_);
$qvar=$ARR{$key};
#... additional analysis and conditional output here...
}
--
So basically, for each input record I fetch a datum from a prebuilt NDBM
file, process it, and do some output. The input file can be as large as
several million records; the NDBM file is on the order of 10k records.
The problem is that when this script is run, its resident memory size creeps
up rapidly until there is no remaining memory (around 100MB), it prints an
"Out of memory!" and dies. Why this happens is a mystery to me, since the
data is being fetched from the NDBM file and, for each iteration of the main
loop, is assigned to the same variable. The cache size of the NDBM package
is set to the default value (64 if I remember correctly), and %ARR contains
nothing before the 'tie' is done. The script functions properly, in that the
correct values are fetched from the NDBM file, and correct output is
produced up until the point where memory is exhausted.
The same script (using dbmopen) works fine under 4.036, and runs in about
1300k rss.
Is there any new magic going on in the NDBM_File package which might cause
all accessed records to remain in an ever-growing cache somewhere, or is
something not being freed correctly? Perhaps the operation of 'tie' in
conjunction with the NDBM_File package is other than I think it is?
If you have any clues, please let me know. For the curious, my config.sh is
attached below for 5a9 on a Sun 4.1.3 server.
Regards,
Fred
--config.sh--
#!/bin/sh
#
# This file was produced by running the Configure script. It holds all the
# definitions figured out by Configure. Should you modify one of these values,
# do not forget to propagate your changes by running "Configure -der". You may
# instead choose to run each of the .SH files by yourself, or "Configure -S".
#
# Configuration time: Sun May 22 01:38:22 EDT 1994
# Configured by: ft
# Target system: sunos maxwell 4.1.3_u1 2 sun4m
extensions=' ext/dbm/NDBM_File.xs ext/dbm/ODBM_File.xs ext/dbm/GDBM_File.xs ext/dbm/SDBM_File.xs ext/posix/POSIX.xs'
d_eunice='undef'
d_xenix='undef'
eunicefix=':'
Mcc='Mcc'
awk='/bin/awk'
bash=''
bison='/usr/local/bin/bison'
byacc='byacc'
cat='/bin/cat'
chgrp=''
chmod=''
chown=''
compress=''
cp='/bin/cp'
cpio=''
cpp='/usr/lang/cpp'
csh='/bin/csh'
date='/bin/date'
echo='/bin/echo'
egrep='/bin/egrep'
emacs=''
expr='/bin/expr'
find='/bin/find'
flex=''
gcc=''
grep='/bin/grep'
inews=''
ksh=''
less=''
line='/bin/line'
lint=''
ln='/bin/ln'
lp=''
lpr=''
ls=''
mail=''
mailx=''
make=''
mkdir='/bin/mkdir'
more=''
mv='/bin/mv'
nroff='/bin/nroff'
perl='/bin/perl'
pg=''
pmake=''
pr=''
rm='/bin/rm'
rmail=''
sed='/bin/sed'
sendmail=''
sh=''
shar=''
sleep=''
smail=''
sort='/bin/sort'
submit=''
tail=''
tar=''
tbl=''
test='test'
touch='/bin/touch'
tr='/bin/tr'
troff=''
uname='/bin/uname'
uniq='/bin/uniq'
uuname=''
vi=''
zcat=''
hint='previous'
myuname='sunos maxwell 4.1.3_u1 2 sun4m '
osname='sunos'
osvers='4.1.3_u1'
Author=''
Date='$Date'
Header=''
Id='$Id'
Locker=''
Log='$Log'
RCSfile='$RCSfile'
Revision='$Revision'
Source=''
State=''
afs='false'
memalignbytes='8'
bin='/usr/local/bin'
binexp='/usr/local/bin'
installbin='/usr/local/bin'
byteorder='4321'
cc='cc'
gccversion=''
ccflags='-DDEBUGGING'
cppflags=' -DDEBUGGING'
ldflags=''
lkflags=''
optimize='-O'
cf_by='ft'
cf_time='Sun May 22 01:38:22 EDT 1994'
contains='grep'
cpplast=''
cppminus=''
cpprun='/usr/lang/cpp'
cppstdin='/home/maxwell/ft/build/perl/perl5alpha9/cppstdin'
d_access='define'
d_bcmp='define'
d_bcopy='define'
d_bzero='define'
d_casti32='define'
castflags='0'
d_castneg='define'
d_charsprf='define'
d_chsize='undef'
d_const='undef'
cryptlib=''
d_crypt='define'
d_csh='define'
d_dosuid='undef'
d_dup2='define'
d_fchmod='define'
d_fchown='define'
d_fcntl='define'
d_flexfnam='define'
d_flock='define'
d_getgrps='define'
d_gethent='define'
aphostname=''
d_gethname='undef'
d_phostname='undef'
d_uname='define'
d_getpgrp2='undef'
d_getpgrp='define'
d_getprior='define'
d_htonl='define'
d_isascii='define'
d_killpg='define'
d_link='define'
d_lstat='define'
d_memcmp='define'
d_memcpy='define'
d_memmove='undef'
d_memset='define'
d_mkdir='define'
d_msg='define'
d_msgctl='define'
d_msgget='define'
d_msgrcv='define'
d_msgsnd='define'
d_open3='define'
d_portable='undef'
d_readdir='define'
d_rewinddir='define'
d_seekdir='define'
d_telldir='define'
d_rename='define'
d_rmdir='define'
d_safebcpy='define'
d_safemcpy='undef'
d_select='define'
d_sem='define'
d_semctl='define'
d_semget='define'
d_semop='define'
d_setegid='define'
d_seteuid='define'
d_setlocale='define'
d_setpgid='define'
d_setpgrp2='undef'
d_bsdpgrp=''
d_setpgrp='define'
d_setprior='define'
d_setregid='define'
d_setresgid='undef'
d_setresuid='undef'
d_setreuid='define'
d_setrgid='define'
d_setruid='define'
d_setsid='define'
d_shm='define'
d_shmat='define'
d_voidshmat='undef'
d_shmctl='define'
d_shmdt='define'
d_shmget='define'
d_oldsock='undef'
d_socket='define'
d_sockpair='define'
sockethdr=''
socketlib=''
d_statblks='define'
d_stdstdio='define'
d_index='undef'
d_strchr='define'
d_strctcpy='define'
d_strerrm='define'
d_strerror='undef'
d_sysernlst=''
d_syserrlst='define'
d_symlink='define'
d_syscall='define'
d_system='define'
d_time='define'
timetype='long'
clocktype='long'
d_times='define'
d_truncate='define'
d_usendir='undef'
i_ndir='undef'
ndirc=''
ndirlib=''
ndiro=''
d_vfork='define'
d_voidsig='define'
signal_t='void'
d_volatile='undef'
d_charvspr='define'
d_vprintf='define'
d_wait4='define'
d_waitpid='define'
cccdlflags=''
ccdlflags=''
dldir='ext/dl'
dlobj='dl_sunos.o'
dlsrc='dl_sunos.c'
lddlflags=''
shlibsuffix='.so'
usedl='define'
gidtype='gid_t'
groupstype='int'
h_fcntl='false'
h_sysfile='true'
i_dbm='define'
d_dirnamlen='undef'
i_dirent='define'
i_dlfcn='define'
i_fcntl='undef'
i_gdbm='define'
i_grp='define'
i_memory='define'
i_ndbm='define'
i_neterrno='undef'
i_niin='define'
i_sysin='undef'
d_pwage='define'
d_pwchange='undef'
d_pwclass='undef'
d_pwcomment='define'
d_pwexpire='undef'
d_pwquota='undef'
i_pwd='define'
i_sdbm='define'
i_stdarg='undef'
i_stddef='define'
i_string='define'
strings='/usr/include/string.h'
i_sysdir='define'
i_sysfile='define'
d_voidtty=''
i_bsdioctl=''
i_sysioctl='define'
i_syssockio=''
i_sysndir='undef'
i_sysselct='undef'
i_sgtty='undef'
i_termio='undef'
i_termios='define'
i_systime='define'
i_systimek='undef'
i_time='undef'
timeincl='/usr/include/sys/time.h '
i_unistd='define'
i_utime='define'
i_varargs='define'
i_varhdr='varargs.h'
i_vfork='define'
intsize='4'
lib='/usr/local/lib'
libexp='/usr/local/lib'
libc='/usr/lib/libc.so.1.9'
libpth=' /lib /usr/lib /usr/ucblib /usr/local/lib'
plibpth=''
xlibpth='/usr/lib/386 /lib/386'
libs='-lnsl -lgdbm -ldbm -ldl -ldld -lm -lposix'
lns='/bin/ln -s'
lseektype='off_t'
d_mymalloc='define'
mallocobj='malloc.o'
mallocsrc='malloc.c'
malloctype='char *'
usemymalloc='y'
installmansrc='/usr/local/man/man1'
manext='1'
mansrc='/usr/local/man/man1'
mansrcexp='/usr/local/man/man1'
huge=''
large=''
medium=''
models='none'
small=''
split=''
mydomain=''
myhostname='maxwell'
phostname='hostname'
c=''
n='-n'
groupcat=''
hostcat='cat /etc/hosts'
passcat=''
orderlib='false'
ranlib='/usr/bin/ranlib'
package='perl5'
spackage=''
installprivlib='/usr/local/lib/perl5'
privlib='/usr/local/lib/perl5'
privlibexp='/usr/local/lib/perl5'
prototype='undef'
ptrsize='4'
randbits='31'
installscript='/usr/local/bin'
scriptdir='/usr/local/bin'
scriptdirexp='/usr/local/bin'
sig_name='ZERO HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG STOP TSTP CONT CLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH LOST USR1 USR2'
sharpbang='#!'
shsharp='true'
spitshell='cat'
startsh='#!/bin/sh'
stdchar='unsigned char'
sysman='/usr/man/man1'
uidtype='uid_t'
nm_opt=''
runnm='true'
usenm='true'
incpath=''
mips=''
mips_type=''
usrinc='/usr/include'
defvoidused='15'
voidflags='15'
yacc='yacc'
yaccflags=''
PATCHLEVEL=0
CONFIG=true
--
Fred True "My name is Ozymandias, King of Kings:
AT&T Global Systems Technology Group Look on my works, ye Mighty,
f...@maxwell.att.com and despair!"
ft...@attmail.com -P. B. Shelley