Abnormal termination of impute

[Pascal Maradan]

Hi iveWare group,

I'm having a problem with the imputation macro which gives an "Abnormal termination of impute" but I can't find exactly where the problem is.

I'm running it with Sas 9.4 on Linux x64 server.

Could somebody help and explain how to debug this kind of error please.

Thanks a lot

Pascal Maradan

Federal Office of Statistics (Switzerland).

Here's the setup file :

title EFH_GASTWS multiple imputation;
datain efh_gastws_imp;
dataout ive.impgastws_efh all;
count gastws gazwoh gastwe_w sup_stwks;
categorical GASTWS_h gbaups ;
transfer egid;
minrsqd .01;
iterations 10;
multiples 10;
perturb Sir;
seed 20150629;
run;

The macro call and the Log :

3156    option mprint nosymbolgen nomlogic;
3157    %impute(name=impute_efh, dir=&iveLog);
MPRINT(SETUP):  ;
MPRINT(IMPUTE):  ;
MPRINT(GETKEYS):   data _null_;
MPRINT(GETKEYS):   array keys(*) $ 32 key1 key2 ( "DATAIN" , "DATAOUT" );
MPRINT(GETKEYS):   length key $ 32;
MPRINT(GETKEYS):   do while (1);
MPRINT(GETSET):   length line $ 32767;
MPRINT(GETSET):   retain maxlen 32767;
MPRINT(GETSET):   retain reclen 0 recptr 1 quote "";
MPRINT(GETSET):   comment = 0;
MPRINT(GETSET):   complete = 0;
MPRINT(GETSET):   line = "";
MPRINT(GETSET):   linelen = 0;
MPRINT(GETSET):   do until (complete);
MPRINT(GETSET):   if (recptr gt reclen) then do;
MPRINT(GETSET):   if (linelen gt 0) then do;
MPRINT(GETSET):   if (substr(line, linelen, 1) ne ' ') then do;
MPRINT(GETSET):   linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = ' ';
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   do while (recptr gt reclen);
MPRINT(GETSET):   if (end) then do;
MPRINT(GETSET):   if (linelen gt 0) then call symput("msg", "Setup line incomplete");
MPRINT(GETSET):   stop;
MPRINT(GETSET):   end;
MPRINT(GETSET):   infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set" _infile_=record end=end;
MPRINT(GETSET):   input;
MPRINT(GETSET):   if (record ne "") then do;
MPRINT(GETSET):   reclen = length(record);
MPRINT(GETSET):   recptr = 1;
MPRINT(GETSET):   do while (recptr le reclen and substr(record, recptr, 1) le ' ');
MPRINT(GETSET):   recptr = recptr + 1;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   if (comment) then do;
MPRINT(GETSET):   if (recptr + 1 le reclen) then do;
MPRINT(GETSET):   if (substr(record, recptr, 2) eq "*/") then do;
MPRINT(GETSET):   recptr = recptr + 1;
MPRINT(GETSET):   substr(record, recptr, 1) = ' ';
MPRINT(GETSET):   comment = 0;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   if (comment) then recptr = recptr + 1;
MPRINT(GETSET):   end;
MPRINT(GETSET):   else if (recptr + 1 le reclen) then do;
MPRINT(GETSET):   if (substr(record, recptr, 2) eq "/*") then do;
MPRINT(GETSET):   recptr = recptr + 2;
MPRINT(GETSET):   comment = 1;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   if (not comment) then do;
MPRINT(GETSET):   if (quote ne "") then do;
MPRINT(GETSET):   if (substr(record, recptr, 1) eq quote) then quote = "";
MPRINT(GETSET):   linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = substr(record, recptr, 1);
MPRINT(GETSET):   end;
MPRINT(GETSET):   else if (substr(record, recptr, 1) eq "'" or substr(record, recptr, 1) eq '"') then do;
MPRINT(GETSET):   quote = substr(record, recptr, 1);

MPRINT(GETSET):   linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = substr(record, recptr, 1);
MPRINT(GETSET):   end;
MPRINT(GETSET):   else if (substr(record, recptr, 1) eq ';') then do;
MPRINT(GETSET):   if (substr(line, 1, 1) eq '*') then linelen = 0;
MPRINT(GETSET):   else do;
MPRINT(GETSET):   if (linelen eq 0) then do;
MPRINT(GETSET):   call symput("msg", "Setup line empty");
MPRINT(GETSET):   stop;
MPRINT(GETSET):   end;
MPRINT(GETSET):   if (substr(line, linelen, 1) ne ' ') then linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = ';';
MPRINT(GETSET):   complete = 1;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   else if (substr(record, recptr, 1) gt ' ') then do;
MPRINT(GETSET):   linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = substr(record, recptr, 1);
MPRINT(GETSET):   end;
MPRINT(GETSET):   else do;
MPRINT(GETSET):   if (linelen gt 0) then do;
MPRINT(GETSET):   if (substr(line, linelen, 1) ne ' ') then do;
MPRINT(GETSET):   linelen = linelen + 1;
MPRINT(GETSET):   substr(line, linelen, 1) = ' ';
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   recptr = recptr + 1;
MPRINT(GETSET):   end;
MPRINT(GETSET):   end;
MPRINT(GETSET):   line = trim(substr(line, 1, linelen));
MPRINT(GETKEYS):  ;
MPRINT(GETKEYS):   key = scan(line, 1, " ");
MPRINT(GETKEYS):   key = upcase(key);
MPRINT(GETKEYS):   keylen = length(key);
MPRINT(GETKEYS):   do _i_ = 1 to dim(keys);
MPRINT(GETKEYS):   if (keylen ge length(keys(_i_))) then do;
MPRINT(GETKEYS):   if (substr(key, 1, length(keys(_i_))) eq keys(_i_)) then do;
MPRINT(GETKEYS):   line = left((substr(line, keylen+1)));
MPRINT(GETKEYS):   line = substr(line, 1, length(line) - 1);
MPRINT(GETKEYS):   call symput(keys(_i_), trimn(line));
MPRINT(GETKEYS):   leave;
MPRINT(GETKEYS):   end;
MPRINT(GETKEYS):   end;
MPRINT(GETKEYS):   end;
MPRINT(GETKEYS):   end;
MPRINT(GETKEYS):   run;

NOTE: The infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set" is:
      Filename=/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set,
      Owner Name=t7rck,Group Name=ldap,
      Access Permission=-rw-rw----,
      Last Modified=26. Mai 2016 12.05 Uhr,
      File Size (bytes)=252

NOTE: 12 records were read from the infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set".
      The minimum record length was 4.
      The maximum record length was 39.
NOTE: DATA statement used (Total process time):
      real time           0.59 seconds
      cpu time            0.01 seconds

MPRINT(IMPUTE):  ;
Get metadata.

MPRINT(METADATA):   proc datasets nolist;
MPRINT(METADATA):   delete vars1 codes1 / memtype=data;
MPRINT(METADATA):   delete vars1 codes1 / memtype=view;
MPRINT(METADATA):   run;

NOTE: The file WORK.CODES1 (memtype=DATA) was not found, but appears on a DELETE statement.
NOTE: The file WORK.VARS1 (memtype=VIEW) was not found, but appears on a DELETE statement.
NOTE: The file WORK.CODES1 (memtype=VIEW) was not found, but appears on a DELETE statement.
NOTE: Deleting WORK.VARS1 (memtype=DATA).

NOTE: PROCEDURE DATASETS used (Total process time):
      real time           0.09 seconds
      cpu time            0.01 seconds

MPRINT(METADATA):   data variables;
MPRINT(METADATA):   attrib Name label="Variable name" Type label="Variable type" Frame label="Variable codeframe" Label label="Variable label"
;
MPRINT(METADATA):   datafile = open("efh_gastws_imp", "i");
MPRINT(METADATA):   nvars = attrn(datafile, "nvars");
MPRINT(METADATA):   do Var = 1 to nvars;
MPRINT(METADATA):   Name = varname(datafile, var);
MPRINT(METADATA):   if (vartype(datafile, var) eq "C") then Type = "char";
MPRINT(METADATA):   else Type = "num";
MPRINT(METADATA):   Frame = varfmt(datafile, var);
MPRINT(METADATA):   if (substr(Frame, length(Frame), 1) eq ".") then Frame = substr(Frame, 1, length(Frame)-1);
MPRINT(METADATA):   Label = varlabel(datafile, var);
MPRINT(METADATA):   keep Var Name Type Frame Label;
MPRINT(METADATA):   output;
MPRINT(METADATA):   end;
MPRINT(METADATA):   rc = close(datafile);
MPRINT(METADATA):   run;

NOTE: The data set WORK.VARIABLES has 8 observations and 5 variables.
NOTE: DATA statement used (Total process time):
      real time           0.20 seconds
      cpu time            0.01 seconds

MPRINT(METADATA):   data vars1;
MPRINT(METADATA):   set variables;
MPRINT(METADATA):   frame = "";
MPRINT(METADATA):   run;

NOTE: There were 8 observations read from the data set WORK.VARIABLES.
NOTE: The data set WORK.VARS1 has 8 observations and 5 variables.
NOTE: DATA statement used (Total process time):
      real time           0.06 seconds
      cpu time            0.01 seconds

MPRINT(METADATA):   data _null_;
MPRINT(METADATA):   set vars1;
MPRINT(METADATA):   length line $ 1024;
MPRINT(METADATA):   length oldframe $ 32;
MPRINT(METADATA):   length oldtype $ 8;
MPRINT(METADATA):   retain oldtype oldframe '' _code_ 0;
MPRINT(METADATA):   file "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.met";
MPRINT(METADATA):   if (_n_ eq 1) then do;
MPRINT(METADATA):   put 'standard;';

MPRINT(METADATA):   put 'variables';
MPRINT(METADATA):   end;
MPRINT(METADATA):   line = 'name=' || name;
MPRINT(METADATA):   if (label ne '') then line = trim(line) || ' label="' || tranwrd(trim(label), '"', '""') || '"';
MPRINT(METADATA):   type = trim(type);
MPRINT(METADATA):   if (type ne oldtype) then do;
MPRINT(METADATA):   oldtype = type;
MPRINT(METADATA):   if (type eq 'char') then line = trim(line) || ' type=char';
MPRINT(METADATA):   else line = trim(line) || ' type=float';
MPRINT(METADATA):   end;
MPRINT(METADATA):   if (_n_ eq 1) then line = trim(line) || ' width=8';
MPRINT(METADATA):   line = trim(line) || ';';
MPRINT(METADATA):   put @3 line;
MPRINT(METADATA):   run;

NOTE: The file "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.met" is:
      Filename=/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.met,
      Owner Name=t7rbd,Group Name=ldap,
      Access Permission=-rw-rw----,
      Last Modified=26. Mai 2016 12.54 Uhr

NOTE: 10 records were written to the file "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.met".
      The minimum record length was 9.
      The maximum record length was 31.
NOTE: There were 8 observations read from the data set WORK.VARS1.
NOTE: DATA statement used (Total process time):
      real time           0.33 seconds
      cpu time            0.01 seconds

MPRINT(IMPUTE):  ;
Check setup.
MPRINT(EXECUTE):   data _null_;
MPRINT(EXECUTE):   call system("/appl/bfs/IVEware/srclib/iveset /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set
/sas");
MPRINT(EXECUTE):   run;

NOTE: DATA statement used (Total process time):
      real time           0.05 seconds
      cpu time            0.00 seconds

MPRINT(IMPUTE):  ;
Input data.
MPRINT(IMPUTE):   filename input "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.inp";
MPRINT(IMPUTE):   data _null_;
MPRINT(IMPUTE):   set efh_gastws_imp;
MPRINT(IMPUTE):   file "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.dat" recfm=n lrecl=256;
MPRINT(IMPUTE):   if (_N_ eq 1) then do;
MPRINT(IMPUTE):   mdata = .;
MPRINT(IMPUTE):   put mdata rb8.;
MPRINT(IMPUTE):   end;
MPRINT(IMPUTE):   if GBAUPS le .Z then GBAUPS = .;
MPRINT(IMPUTE):   if GASTWS le .Z then GASTWS = .;
MPRINT(IMPUTE):   if GAZWOH le .Z then GAZWOH = .;
MPRINT(IMPUTE):   if sup_stwks le .Z then sup_stwks = .;
MPRINT(IMPUTE):   if GASTWe le .Z then GASTWe = .;
MPRINT(IMPUTE):   if GASTWS_h le .Z then GASTWS_h = .;
MPRINT(IMPUTE):   if GASTWe_w le .Z then GASTWe_w = .;
MPRINT(IMPUTE):   put GBAUPS rb8. GASTWS rb8. GAZWOH rb8. sup_stwks rb8. GASTWe rb8. GASTWS_h rb8. GASTWe_w rb8. ;
MPRINT(IMPUTE):   run;

NOTE: UNBUFFERED is the default with RECFM=N.

NOTE: The file "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.dat" is:
      Filename=/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.dat,
      Owner Name=t7rck,Group Name=ldap,
      Access Permission=-rw-rw----,
      Last Modified=26. Mai 2016 12.54 Uhr

NOTE: There were 981080 observations read from the data set WORK.EFH_GASTWS_IMP.
NOTE: DATA statement used (Total process time):
      real time           0.72 seconds
      cpu time            0.46 seconds

Execute Imputation.
MPRINT(EXECUTE):   data _null_;
MPRINT(EXECUTE):   call system("/appl/bfs/IVEware/srclib/impute /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set
/sas");
MPRINT(EXECUTE):   run;

NOTE: DATA statement used (Total process time):
      real time           1.54 seconds
      cpu time            0.00 seconds

MPRINT(IMPUTE):  ;
Copy log.
MPRINT(COPYLOG):   data _null_;
MPRINT(COPYLOG):   infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.log" length=len end=test;
MPRINT(COPYLOG):   do until (test eq 1);
MPRINT(COPYLOG):   input line $varying133. len;
MPRINT(COPYLOG):   put line $varying133. len;
MPRINT(COPYLOG):   end;
MPRINT(COPYLOG):   put;
MPRINT(COPYLOG):   output;
MPRINT(COPYLOG):   run;

NOTE: The infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.log" is:
      Filename=/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.log,
      Owner Name=t7rck,Group Name=ldap,
      Access Permission=-rw-rw----,
      Last Modified=26. Mai 2016 12.54 Uhr,
      File Size (bytes)=328

                         IVEware Statistical Procedures
             Survey Research Center, Institute for Social Research
                             University of Michigan
                        Version 2.0, Copyright (c) 2009

Thu May 26 12:54:00 2016
  Begin setup checking
  Read setup
  Write setup
  End setup checking
NOTE: 10 records were read from the infile "/project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.log".
      The minimum record length was 0.
      The maximum record length was 66.
NOTE: DATA statement used (Total process time):
      real time           0.22 seconds
      cpu time            0.00 seconds

MPRINT(IMPUTE):  ;
Abnormal termination of impute. Check log.

[Peter Solenberger]

Dear Pascal,

You could try running the imputation outside the SAS framework by executing in Linux the system call SAS was trying to execute when Impute failed.

/appl/bfs/IVEware/srclib/impute /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set

That might give you more information.

You're using an old version of IVEware. I'm not sure I could recreate it to debug your problem. We have a new version on the verge of release. If running Impute alone doesn't indicate a solution to your problem, I'd suggest installing the new version from the following URL.

http://www.isr.umich.edu/src/smp/ive/dev/

With a small change to your options command, you should be able to run your setup. The change is to refer to the sas subdirectory of srclib, not the srclib directory itself. In your case that would be

/appl/bfs/IVEware/srclib/sas

Let me know if either suggestion solves the problem.

Peter

--
You received this message because you are subscribed to the Google Groups "IVEware" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iveware+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pascal Maradan]

Hi Peter,

Thanks a lot for your quick reply.

Effectivelly, running the program outside SAS did give an explicit error which makes us able to go one step forward.

     Error: Can't read /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/efh_gastws_imp.dat

The dat file (specified into the .set file) doesn't exist and now we have to fix why this file has not been created.

I'll let you know when we'll have some new information.

Thanks again

Cheers

Pascal :)

[Peter Solenberger]

Dear Pascal,

I realize that I made a mistake. The Linux command needs a "/sas" parameter to tell Impute that SAS is the data source, as in the following.

/appl/bfs/IVEware/srclib/impute /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set /sas

Please try again.

Peter

On 5/30/16 10:13 AM, Pascal Maradan wrote:

Hi Peter,

Thanks a lot for your quick reply.

Effectivelly, running the program outside SAS did give an explicit error which makes us able to go one step forward.

     Error: Can't read /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/efh_gastws_imp.dat

The dat file (specified into the .set file) doesn't exist and now we have to fix why this file has not been created.

I'll let you know when we'll have some new information.

Thanks again

Cheers

Pascal :)

Le jeudi 26 mai 2016 21:11:40 UTC+2, Peter Solenberger a écrit :

[Pascal Maradan]

Thank you Peter for this update... I should have seen this myself :)

I've re-executed the impute step but now we get "segmentation fault" error :

>/appl/bfs/IVEware/srclib/impute /project/prd/shape/gwi/progs/Stochastische_Einsetzungen/IVEware/impute_efh.set /sas

Wed Jun  1 10:03:40 2016
  Begin execution
  Read setup
  Read data
Wed Jun  1 10:03:41 2016
  Mult 1, Iter 1, Imp 1
  Impute GASTWe
  Update data
  Linear regression
  Correct data
Wed Jun  1 10:03:41 2016
  Mult 1, Iter 1, Imp 2
  Impute GASTWS
  Update data
  Poisson regression
Segmentation fault

Problem is probably into data, but do you have a suggestion to fix this ?

Thanks a lot for your help.

Cheers

Pascal :)

[Pascal Maradan]

Here's the .set file :

title EFH_GASTWS multiple imputation;
datain efh_gastws_imp;
dataout ive.impgastws_efh all;
count gastws gazwoh gastwe_w sup_stwks;

categorical gbaups ;
transfer egid GASTWS_h gastws gkats gheizs genhzs gwwvs genwws;

[Peter Solenberger]

Dear pascal,

There may be a problem with the data, but the unhelpful message indicates a problem with how Impute handled it. I suggest that you try the development version from the URL I sent. This may be a problem we've found and fixed. If you get a segmentation fault with the development version, is your data confidential? Could you send me the input dataset or a subset that produces the error?

Peter

[Trivellore Raghunathan]

I wonder whether the count variables are not integers. You can do a frequency distribution to check it out.

Raghu