Syntax file: line length limit for inline data? (SPSS 19, Mac OS X)
Rick Strong
New user, please forgive if this is really obvious!
I am trying to set up a syntax file to read a single wide record of data
and seem to have come up against a line length limit. The record
consists of 1 row of 548 variables (in between BEGIN DATA and END DATA.)
and is being truncated consistently at 1024 characters. The result is
that there is a mismatch between the number of variables specified in
the DATA LIST statement and the number of values provided.
The aim is to provide this file as output from a program I've written
and to allow users to import this data into SPSS, creating or adding to
existing data.
Is there a specific line length limit, and is it possible to work around
it using inline data? Is there perhaps a better way for me to do what
I'm trying to do? This is the general form of what I'm now writing:
DATA LIST LIST /var_1 var_2 ... var_n.
BEGIN DATA
value_1,value_2, ... value_n
END DATA.
LIST.
VARIABLE LABELS
varname_1 "label_1"
/ varname_2 "label_2"
.
.
.
/ varname_n "label_n".
Thanks for any suggestions!
Best,
Rick Strong
Art Kendall
however I presume your are not typing all of that into syntax.
perhaps you should have it in a .txt or .asc file and use the wizard
under <file> to write and paste your syntax.
<help> says
>
> LRECL Subcommand (FILE HANDLE command)
>
> LRECL specifies the length of each record in the file. When you
> specify IMAGE under UNIX, OS/2, or Microsoft Windows, or 360 for
> IBM360 EBCDIC data files, you must specify LRECL. You can specify a
> record length greater than the default (8,192) for an image file, a
> character file, or a binary file. The maximum record length is
> 2,147,483,647. Do not use LRECL with MULTIPUNCH.
>
Art Kendall
Social Research Consultants
Rick Strong
hoping to write just one file to simplify things. Is there some sort of
equivalent way to specify length of inline data or am I stuck with 1024
characters (if that's really the limit)?
Art Kendall
saying sounds very unusual.
It may be that there is a character limit in a syntax file.
But what is the problem with reading a file from the disk as would
usually be done with files with that many variables.
an external disk file can have a length just over 2 billion bytes
(usually that number of characters, e.g., in English).
How were you planning to put it into a syntax file?
If you are writing a program, and you know how to use them there are
free .dll files so you can read or write files as SPSS system files. I
do not know if there are such things for MAC but I would be surprised if
there are not.
there is some example syntax below the signature. It works under Windows 7.
open a new instance of SPSS. Paste the syntax into a syntax window. Run it.
Does the disk file resemble what you are trying to do?
Does the dataset "readback" look like what you want to end up with?
Art Kendall
Social Research Consultants
*generate some data that sounds like the situation.
input program.
vector x (600,f3).
loop id = 1 to 3.
loop #p = 1 to 600.
compute x(#p) = rnd(rv.normal(500,100)).
end loop.
end case.
end loop.
end file.
end input program.
execute.
* save the simulated data to a disk file in character mode.
dataset name original.
save translate /outfile = 'c:\project\wide.csv'/type = CSV /fieldnames.
* read the disk file back in.
GET DATA
/TYPE=TXT
/FILE="C:\project\wide.csv"
/DELCASE=LINE
/DELIMITERS=","
/ARRANGEMENT=DELIMITED
/FIRSTCASE=2
/IMPORTCASE=ALL
/VARIABLES=
x1 F3.0
x2 F3.0
x3 F3.0
x4 F3.0
x5 F3.0
x6 F3.0
x7 F3.0
x8 F3.0
x9 F3.0
x10 F3.0
x11 F3.0
x12 F3.0
x13 F3.0
x14 F3.0
x15 F3.0
x16 F3.0
x17 F3.0
x18 F3.0
x19 F3.0
x20 F3.0
x21 F3.0
x22 F3.0
x23 F3.0
x24 F3.0
x25 F3.0
x26 F3.0
x27 F3.0
x28 F3.0
x29 F3.0
x30 F3.0
x31 F3.0
x32 F3.0
x33 F3.0
x34 F3.0
x35 F3.0
x36 F3.0
x37 F3.0
x38 F3.0
x39 F3.0
x40 F3.0
x41 F3.0
x42 F3.0
x43 F3.0
x44 F3.0
x45 F3.0
x46 F3.0
x47 F3.0
x48 F3.0
x49 F3.0
x50 F3.0
x51 F3.0
x52 F3.0
x53 F3.0
x54 F3.0
x55 F3.0
x56 F3.0
x57 F3.0
x58 F3.0
x59 F3.0
x60 F3.0
x61 F3.0
x62 F3.0
x63 F3.0
x64 F3.0
x65 F3.0
x66 F3.0
x67 F3.0
x68 F3.0
x69 F3.0
x70 F3.0
x71 F3.0
x72 F3.0
x73 F3.0
x74 F3.0
x75 F3.0
x76 F3.0
x77 F3.0
x78 F3.0
x79 F3.0
x80 F3.0
x81 F3.0
x82 F3.0
x83 F3.0
x84 F3.0
x85 F3.0
x86 F3.0
x87 F3.0
x88 F3.0
x89 F3.0
x90 F3.0
x91 F3.0
x92 F3.0
x93 F3.0
x94 F3.0
x95 F3.0
x96 F3.0
x97 F3.0
x98 F3.0
x99 F3.0
x100 F3.0
x101 F3.0
x102 F3.0
x103 F3.0
x104 F3.0
x105 F3.0
x106 F3.0
x107 F3.0
x108 F3.0
x109 F3.0
x110 F3.0
x111 F3.0
x112 F3.0
x113 F3.0
x114 F3.0
x115 F3.0
x116 F3.0
x117 F3.0
x118 F3.0
x119 F3.0
x120 F3.0
x121 F3.0
x122 F3.0
x123 F3.0
x124 F3.0
x125 F3.0
x126 F3.0
x127 F3.0
x128 F3.0
x129 F3.0
x130 F3.0
x131 F3.0
x132 F3.0
x133 F3.0
x134 F3.0
x135 F3.0
x136 F3.0
x137 F3.0
x138 F3.0
x139 F3.0
x140 F3.0
x141 F3.0
x142 F3.0
x143 F3.0
x144 F3.0
x145 F3.0
x146 F3.0
x147 F3.0
x148 F3.0
x149 F3.0
x150 F3.0
x151 F3.0
x152 F3.0
x153 F3.0
x154 F3.0
x155 F3.0
x156 F3.0
x157 F3.0
x158 F3.0
x159 F3.0
x160 F3.0
x161 F3.0
x162 F3.0
x163 F3.0
x164 F3.0
x165 F3.0
x166 F3.0
x167 F3.0
x168 F3.0
x169 F3.0
x170 F3.0
x171 F3.0
x172 F3.0
x173 F3.0
x174 F3.0
x175 F3.0
x176 F3.0
x177 F3.0
x178 F3.0
x179 F3.0
x180 F3.0
x181 F3.0
x182 F3.0
x183 F3.0
x184 F3.0
x185 F3.0
x186 F3.0
x187 F3.0
x188 F3.0
x189 F3.0
x190 F3.0
x191 F3.0
x192 F3.0
x193 F3.0
x194 F3.0
x195 F3.0
x196 F3.0
x197 F3.0
x198 F3.0
x199 F3.0
x200 F3.0
x201 F3.0
x202 F3.0
x203 F3.0
x204 F3.0
x205 F3.0
x206 F3.0
x207 F3.0
x208 F3.0
x209 F3.0
x210 F3.0
x211 F3.0
x212 F3.0
x213 F3.0
x214 F3.0
x215 F3.0
x216 F3.0
x217 F3.0
x218 F3.0
x219 F3.0
x220 F3.0
x221 F3.0
x222 F3.0
x223 F3.0
x224 F3.0
x225 F3.0
x226 F3.0
x227 F3.0
x228 F3.0
x229 F3.0
x230 F3.0
x231 F3.0
x232 F3.0
x233 F3.0
x234 F3.0
x235 F3.0
x236 F3.0
x237 F3.0
x238 F3.0
x239 F3.0
x240 F3.0
x241 F3.0
x242 F3.0
x243 F3.0
x244 F3.0
x245 F3.0
x246 F3.0
x247 F3.0
x248 F3.0
x249 F3.0
x250 F3.0
x251 F3.0
x252 F3.0
x253 F3.0
x254 F3.0
x255 F3.0
x256 F3.0
x257 F3.0
x258 F3.0
x259 F3.0
x260 F3.0
x261 F3.0
x262 F3.0
x263 F3.0
x264 F3.0
x265 F3.0
x266 F3.0
x267 F3.0
x268 F3.0
x269 F3.0
x270 F3.0
x271 F3.0
x272 F3.0
x273 F3.0
x274 F3.0
x275 F3.0
x276 F3.0
x277 F3.0
x278 F3.0
x279 F3.0
x280 F3.0
x281 F3.0
x282 F3.0
x283 F3.0
x284 F3.0
x285 F3.0
x286 F3.0
x287 F3.0
x288 F3.0
x289 F3.0
x290 F3.0
x291 F3.0
x292 F3.0
x293 F3.0
x294 F3.0
x295 F3.0
x296 F3.0
x297 F3.0
x298 F3.0
x299 F3.0
x300 F3.0
x301 F3.0
x302 F3.0
x303 F3.0
x304 F3.0
x305 F3.0
x306 F3.0
x307 F3.0
x308 F3.0
x309 F3.0
x310 F3.0
x311 F3.0
x312 F3.0
x313 F3.0
x314 F3.0
x315 F3.0
x316 F3.0
x317 F3.0
x318 F3.0
x319 F3.0
x320 F3.0
x321 F3.0
x322 F3.0
x323 F3.0
x324 F3.0
x325 F3.0
x326 F3.0
x327 F3.0
x328 F3.0
x329 F3.0
x330 F3.0
x331 F3.0
x332 F3.0
x333 F3.0
x334 F3.0
x335 F3.0
x336 F3.0
x337 F3.0
x338 F3.0
x339 F3.0
x340 F3.0
x341 F3.0
x342 F3.0
x343 F3.0
x344 F3.0
x345 F3.0
x346 F3.0
x347 F3.0
x348 F3.0
x349 F3.0
x350 F3.0
x351 F3.0
x352 F3.0
x353 F3.0
x354 F3.0
x355 F3.0
x356 F3.0
x357 F3.0
x358 F3.0
x359 F3.0
x360 F3.0
x361 F3.0
x362 F3.0
x363 F3.0
x364 F3.0
x365 F3.0
x366 F3.0
x367 F3.0
x368 F3.0
x369 F3.0
x370 F3.0
x371 F3.0
x372 F3.0
x373 F3.0
x374 F3.0
x375 F3.0
x376 F3.0
x377 F3.0
x378 F3.0
x379 F3.0
x380 F3.0
x381 F3.0
x382 F3.0
x383 F3.0
x384 F3.0
x385 F3.0
x386 F3.0
x387 F3.0
x388 F3.0
x389 F3.0
x390 F3.0
x391 F3.0
x392 F3.0
x393 F3.0
x394 F3.0
x395 F3.0
x396 F3.0
x397 F3.0
x398 F3.0
x399 F3.0
x400 F3.0
x401 F3.0
x402 F3.0
x403 F3.0
x404 F3.0
x405 F3.0
x406 F3.0
x407 F3.0
x408 F3.0
x409 F3.0
x410 F3.0
x411 F3.0
x412 F3.0
x413 F3.0
x414 F3.0
x415 F3.0
x416 F3.0
x417 F3.0
x418 F3.0
x419 F3.0
x420 F3.0
x421 F3.0
x422 F3.0
x423 F3.0
x424 F3.0
x425 F3.0
x426 F3.0
x427 F3.0
x428 F3.0
x429 F3.0
x430 F3.0
x431 F3.0
x432 F3.0
x433 F3.0
x434 F3.0
x435 F3.0
x436 F3.0
x437 F3.0
x438 F3.0
x439 F3.0
x440 F3.0
x441 F3.0
x442 F3.0
x443 F3.0
x444 F3.0
x445 F3.0
x446 F3.0
x447 F3.0
x448 F3.0
x449 F3.0
x450 F3.0
x451 F3.0
x452 F3.0
x453 F3.0
x454 F3.0
x455 F3.0
x456 F3.0
x457 F3.0
x458 F3.0
x459 F3.0
x460 F3.0
x461 F3.0
x462 F3.0
x463 F3.0
x464 F3.0
x465 F3.0
x466 F3.0
x467 F3.0
x468 F3.0
x469 F3.0
x470 F3.0
x471 F3.0
x472 F3.0
x473 F3.0
x474 F3.0
x475 F3.0
x476 F3.0
x477 F3.0
x478 F3.0
x479 F3.0
x480 F3.0
x481 F3.0
x482 F3.0
x483 F3.0
x484 F3.0
x485 F3.0
x486 F3.0
x487 F3.0
x488 F3.0
x489 F3.0
x490 F3.0
x491 F3.0
x492 F3.0
x493 F3.0
x494 F3.0
x495 F3.0
x496 F3.0
x497 F3.0
x498 F3.0
x499 F3.0
x500 F3.0
x501 F3.0
x502 F3.0
x503 F3.0
x504 F3.0
x505 F3.0
x506 F3.0
x507 F3.0
x508 F3.0
x509 F3.0
x510 F3.0
x511 F3.0
x512 F3.0
x513 F3.0
x514 F3.0
x515 F3.0
x516 F3.0
x517 F3.0
x518 F3.0
x519 F3.0
x520 F3.0
x521 F3.0
x522 F3.0
x523 F3.0
x524 F3.0
x525 F3.0
x526 F3.0
x527 F3.0
x528 F3.0
x529 F3.0
x530 F3.0
x531 F3.0
x532 F3.0
x533 F3.0
x534 F3.0
x535 F3.0
x536 F3.0
x537 F3.0
x538 F3.0
x539 F3.0
x540 F3.0
x541 F3.0
x542 F3.0
x543 F3.0
x544 F3.0
x545 F3.0
x546 F3.0
x547 F3.0
x548 F3.0
x549 F3.0
x550 F3.0
x551 F3.0
x552 F3.0
x553 F3.0
x554 F3.0
x555 F3.0
x556 F3.0
x557 F3.0
x558 F3.0
x559 F3.0
x560 F3.0
x561 F3.0
x562 F3.0
x563 F3.0
x564 F3.0
x565 F3.0
x566 F3.0
x567 F3.0
x568 F3.0
x569 F3.0
x570 F3.0
x571 F3.0
x572 F3.0
x573 F3.0
x574 F3.0
x575 F3.0
x576 F3.0
x577 F3.0
x578 F3.0
x579 F3.0
x580 F3.0
x581 F3.0
x582 F3.0
x583 F3.0
x584 F3.0
x585 F3.0
x586 F3.0
x587 F3.0
x588 F3.0
x589 F3.0
x590 F3.0
x591 F3.0
x592 F3.0
x593 F3.0
x594 F3.0
x595 F3.0
x596 F3.0
x597 F3.0
x598 F3.0
x599 F3.0
x600 F3.0
id F1.0.
CACHE.
EXECUTE.
DATASET NAME readback WINDOW=FRONT.
.
Jon Peck
But this is not a good idea. Putting a large quantity of data in a text file (or Excel, or ...) will lead to better organization and a lower likelihood of user errors. And it is no harder than putting the data inline.
A FILE HANDLE may sometimes still be necessary for text files read as data, but IIRC, the default limit is 8K bytes, which will only rarely be exceeded.
HTH,
Jon Peck
Rick Strong
like that are a bit odd in these days of 32 and 64 bit variables and
cheap memory.
What I'm trying to do, and which I guess isn't possible right now, is to
have my report writer generate a single syntax file that represents one
case, with information to populate the variable view if needed as well
as the data view. I'd like my client to run the report, then open the
generated file with SPSS, and be able to add the case to existing data,
and/or use the information to start a new dataset. (I see that if the
data were smaller I could get away with this!)
From what you say I gather I really do need to write 2 files for each
report, one syntax file and one delimited data file. I can do it, but it
makes the user work a bit more than I prefer.
Best,
Rick
Art Kendall
bytes.
An important aspect of syntax files is that they be readable by people.
This is important in drafting syntax, having people do quality assurance
review, getting help, etc. Most people use a monitor to look at
syntax. A very wide syntax would put a very heavy burden on short term
cognitive retention.
If you really do not want to directly create an SPSS system file by
using the DLLs and distributed it, why not just produce 3 files, but
have the user only deal with the main syntax file.
1 the data file mydata.csv. it would look like the file in the previous
simulation I posted earlier.
2 the long set of syntax to read that as a CSV file.
Longread.sps.Generate it once using the wizard and <paste>
3 a syntax file that includes an INSERT for the long set of syntax.
Mysyntax.sps
longread .sps would look like the syntax in the simulation I posted earlier.
* read the disk file back in.
GET DATA
/TYPE=TXT
/FILE="mydata.csv"
/DELCASE=LINE
/DELIMITERS=","
/ARRANGEMENT=DELIMITED
/FIRSTCASE=2
/IMPORTCASE=ALL
/VARIABLES=
x1 F3.0
...
CACHE.
EXECUTE.
DATASET NAME mydata WINDOW=FRONT.
mysyntax.sps would look like this. (untested check INSERT and PRINTACK
in <help>)
printback off.
INSERT file = 'longread.sps'.
printback both.
whatever commands you want.
[longread.sps could also be made more elaborate by using PRESERVE and
RESTORE in the syntax.
HOST command to give OS command to assign a logical name to a folder.
and the CD specification on the INSERT command.]
Then the user would do something like this.
Set up SPSS something like this to set it to look at the last folder
used for a file. Chances are very high that this already set.
Open SPSS <edit> <options><file locations> set the radio button that
says <last folder used>.
exit SPSS.
download your 3 files.
click on the syntax file.
Art Kendall
Social Research Consultants
Ben Pfaff
> Thanks! Parenthetically, best practices are one thing, but hard limits
> like that are a bit odd in these days of 32 and 64 bit variables and
> cheap memory.
If GNU PSPP has the features that you need, you could use it
instead. PSPP does not have any fixed line length limits.
--
Ben Pfaff
http://benpfaff.org
Rick Strong
Windows-based libraries as my system is all Java. I did find a Java
library that writes .sav files but the licensing fee is fairly high.
I see your point about the syntax file size limits. I would never write
incredibly long and unintelligible lines of code myself but I understand
things like that do happen :-)
I'll give the multiple-file approach a try, thanks!
Best,
Rick
Rick Strong
take advantage of this.
Best,
Rick
Ben Pfaff
> What I'm trying to do, and which I guess isn't possible right now, is
> to have my report writer generate a single syntax file that represents
> one case, with information to populate the variable view if needed as
> well as the data view. I'd like my client to run the report, then open
> the generated file with SPSS, and be able to add the case to existing
> data, and/or use the information to start a new dataset. (I see that
> if the data were smaller I could get away with this!)
Why does this require long lines? Break each case into multiple
lines and use the multi-record syntax for DATA LIST. If no
individual string fields are over 1024 bytes long, this should
work fine as-is. Even if individual string fields are wider, you
can write them as shorter segments and concatenate.
Art Kendall
whatever DLLs are called in JAVA, since SPSS uses JAVA.
I copied this post to Jon Peck the developer central guru he most likely
can answer whether there are input output whatevers that work with JAVA.
In straight SPSS, did you understand the post about a user only having
to click on one syntax file to deal with syntax and data in other
files. All the user would see is
Did you understand the idea of using the text wizard one time to create
SPSS syntax that you can put into another stream of syntax via the
INSERT command? Depending what you put in the user-specific syntax file,
you might only have to send that once, the user would only have to click
on the user-specific syntax file each time you sent a new CSV file.
Art Kendall
Social Research Consultants
Rick Strong
how to set it up in my report writing code. It's a little slower going
than I'd like, only because I'm pretty new with SPSS and syntax file
capabilities.
Rick Strong
Forgive my limited knowledge of the syntax, what would this look like?
Ben Pfaff
Here's the multiline syntax example for DATA LIST from the PSPP
manual:
DAT LIS FIL="survey.dat"
/ID 1-5 NAME 7-36 (A) SURNAME 38-67 (A) MINITIAL 69 (A)
/Q01 TO Q50 7-56
/.
Defines the following variables:
* `ID', a numeric variable, in columns 1-5 of the first record.
* `NAME', a 30-character long string variable, in columns 7-36
of the first record.
* `SURNAME', a 30-character long string variable, in columns
38-67 of the first record.
* `MINITIAL', a 1-character short string variable, in column 69
of the first record.
* Fifty variables `Q01', `Q02', `Q03', ..., `Q49', `Q50', all
numeric, `Q01' in column 7, `Q02' in column 8, ..., `Q49' in
column 55, `Q50' in column 56, all in the second record.
Cases are separated by a blank record.
Data is read from file `survey.dat' in the current directory.
This example shows keywords abbreviated to their first 3 letters.
Suppose you have two 80-byte strings A and B and a third 34-byte
string C. You can concatenate them into a new 194-byte string D
with:
STRING /D(A194).
COMPUTE D=CONCAT(A,B,C).
--
"I admire him, I frankly confess it; and when his time comes
I shall buy a piece of the rope for a keepsake."
--Mark Twain
Rick Strong
Thanks! I have a more general question RE PSPP vs SPSS - how compatible
are their syntax files? Is it practical to consider using PSPP as a test
environment to develop files that will be used in SPSS?
Bruce Weaver
>
> Thanks! I have a more general question RE PSPP vs SPSS - how compatible
> are their syntax files? Is it practical to consider using PSPP as a test
> environment to develop files that will be used in SPSS?
I believe they are quite compatible (if not perfectly so) for procedures that are included in PSPP. But see the following:
http://www.gnu.org/software/pspp/manual/html_node/Not-Implemented.html#Not-Implemented
--
Bruce Weaver
bwe...@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/Home
"When all else fails, RTFM."
Ben Pfaff
> Thanks! I have a more general question RE PSPP vs SPSS - how
> compatible are their syntax files? Is it practical to consider using
> PSPP as a test environment to develop files that will be used in SPSS?
PSPP is intended to be as compatible as possible with SPSS. The
differences can be lumped roughly into three categories:
intentional enhancements (which you can disable with
--syntax=compatible), features not yet implemented, and bugs.
--
"Let others praise ancient times; I am glad I was born in these."
--Ovid (43 BC-18 AD)