run-program and binary input
Bart W
I'd like to run an external program (an image conversion program) and
give it some binary data (a JPEG image) as standard input, and save the
result in a file.
I'm using CMUCL 20a, whose manual entry for extensions:run-program at
http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc45
says
----
:input ... If specified as :stream, then the process-input slot contains
an output stream. Anything written to this stream goes to the program as
input.
----
Now
(let ((proc (ext:run-program "/bin/cat" nil
:wait nil
:input :stream
:output "file.jpg")))
(stream-element-type (ext:process-input proc)))
prints CHARACTER. So the output stream, to which I'm supposed to write
the bytes, is a character stream.
So the question is: how do I get the /binary/ input (which I have as an
array of unsigned bytes) into the program's input?
Any advice appreciated.
Bart
Raymond Toy
Good question. One possible way is to take each byte and use code-char
and write out the individual characters (or strings). If you're using
the unicode version, be sure *default-external-format* is :latin1 (or
:iso8859-1). Or instead of specifying :stream, create a binary stream
and pass that as the :input parameter. (Not sure if that will work or not.)
Ray
Bart Wage
Raymond> Bart W wrote:
>> Dear all,
>>
>> I'd like to run an external program (an image conversion program)
>> and give it some binary data (a JPEG image) as standard input,
>> and save the result in a file.
>>
>> I'm using CMUCL 20a, whose manual entry for
>> extensions:run-program at
>> http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc45
>> says
>>
>> ----
>>
>> :input ... If specified as :stream, then the process-input slot
>> contains an output stream. Anything written to this stream goes
>> to the program as input.
>>
>> ----
>>
>> Now
>>
>> (let ((proc (ext:run-program "/bin/cat" nil :wait nil :input
>> :stream :output "file.jpg"))) (stream-element-type
>> (ext:process-input proc)))
>>
>> prints CHARACTER. So the output stream, to which I'm supposed to
>> write the bytes, is a character stream.
>>
>> So the question is: how do I get the /binary/ input (which I have
>> as an array of unsigned bytes) into the program's input?
>>
Raymond> Good question. One possible way is to take each byte and
Raymond> use code-char and write out the individual characters (or
Raymond> strings). If you're using the unicode version, be sure
Raymond> *default-external-format* is :latin1 (or :iso8859-1). Or
Raymond> instead of specifying :stream, create a binary stream and
Raymond> pass that as the :input parameter. (Not sure if that will
Raymond> work or not.)
Hello Raymond,
Thanks for the suggestions. Unfortunately, they do not work.
As for the first one:
(defun in-out (byte-array output-file program &optional arguments)
(let ((*default-external-format* :latin1))
(with-input-from-string (input-stream (with-output-to-string (s)
(loop for b across byte-array
do (write-char (code-char b) s))))
(ext:run-program program arguments
:input input-stream
:output output-file
:if-output-exists :supersede))))
(in-out (kmrcl:read-file-to-usb8-array "binary-input-file")
"binary-output-file"
"/bin/cat")
File binary-output-file is written, but contains quite different bytes
than the binary-input-file.
As for the second one: I haven't got the code at hand, but I tried it
and Lisp complains about not being able to do READ-LINE on a binary
stream.
Any other suggestions? Surely it must be possible to pass a stream of
bytes to a program somehow?
Thanks, Bart
Raymond Toy
>
> Any other suggestions? Surely it must be possible to pass a stream of
> bytes to a program somehow?
Sorry for the delay. I see now that if you specify a stream, cmucl will
basically write out the data more or less directly from the string,
which contains 16-bit characters. This isn't what we want.
run-program needs to be updated somehow.
But until then, perhaps some variant of the following will work. I
tested it very briefly.
(defun in-out2 (byte-array output-file program &optional arguments)
(let ((proc (ext:run-program program arguments
:input :stream
:output output-file :wait nil
:if-output-exists :supersede)))
(loop for x across byte-array
do (write-char (code-char x) (process-input proc)))
(close (process-input proc))
(process-close proc)))
Ray
Bart W
Raymond> Bart Wage wrote:
>>
> Any other suggestions? Surely it must be possible to pass a stream of
>> bytes to a program somehow?
Raymond> Sorry for the delay. I see now that if you specify a
Raymond> stream, cmucl will basically write out the data more or
Raymond> less directly from the string, which contains 16-bit
Raymond> characters. This isn't what we want.
Raymond> run-program needs to be updated somehow.
Raymond> But until then, perhaps some variant of the following will
Raymond> work. I tested it very briefly.
Thanks, that worked! My problems are solved, but here are some more
points:
1. A similar problem exists with binary output and :output :stream.
2. The documentation (at
http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc45)
does not mention that the :output argument may be specified as a stream,
but it may and the comments in the source code (run-program.lisp) say so
too.
3. But passing a stream as :output to run-program, that stream is closed
when the process finishes. I think that's a bug.
Bart
Raymond Toy
>>>>>> "Raymond" == Raymond Toy <toy.r...@gmail.com> writes:
>
> Raymond> Bart Wage wrote:
> >>
>> Any other suggestions? Surely it must be possible to pass a stream of
> >> bytes to a program somehow?
>
> Raymond> Sorry for the delay. I see now that if you specify a
> Raymond> stream, cmucl will basically write out the data more or
> Raymond> less directly from the string, which contains 16-bit
> Raymond> characters. This isn't what we want.
>
> Raymond> run-program needs to be updated somehow.
>
> Raymond> But until then, perhaps some variant of the following will
> Raymond> work. I tested it very briefly.
>
> Thanks, that worked! My problems are solved, but here are some more
Great! Here's another variant that is better and probably much faster:
(defun in-out2 (byte-array output-file program &optional arguments)
(let ((proc (ext:run-program program arguments
:input :stream
:output output-file :wait nil
:if-output-exists :supersede)))
(write-vector byte-array (process-input proc))
(close (process-input proc))
(process-close proc)))
The byte array has to be some kind of specialized array.
> points:
>
> 1. A similar problem exists with binary output and :output :stream.
I suspected that, but I haven't yet tried it. I think a similar
technique can be used as shown above.
>
> 2. The documentation (at
> http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc45)
> does not mention that the :output argument may be specified as a stream,
> but it may and the comments in the source code (run-program.lisp) say so
> too.
I'll update that soon.
>
> 3. But passing a stream as :output to run-program, that stream is closed
> when the process finishes. I think that's a bug.
>
I'll have to think about that. Thanks for the information.
Ray
Rob Warnock
+---------------
| Bart W wrote:
| > 3. But passing a stream as :output to run-program, that stream is closed
| > when the process finishes. I think that's a bug.
|
| I'll have to think about that.
Please do think about it very carefully before changing anything.
While unconditionally closing the stream is not always the correct
behavior, neither is leaving the stream open!! One can easily construct
use cases for RUN-PROGRAM that prefer one behavior over the other.
[Hint: Think "pipe"...]
Perhaps the behavior should be controllable by a new keyword option,
but the default should *definitely* remain the same as it is now!
-Rob
-----
Rob Warnock <rp...@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Raymond Toy
> Raymond Toy <toy.r...@gmail.com> wrote:
> +---------------
> | Bart W wrote:
> | > 3. But passing a stream as :output to run-program, that stream is closed
> | > when the process finishes. I think that's a bug.
> |
> | I'll have to think about that.
> +---------------
>
> Please do think about it very carefully before changing anything.
>
> While unconditionally closing the stream is not always the correct
> behavior, neither is leaving the stream open!! One can easily construct
> use cases for RUN-PROGRAM that prefer one behavior over the other.
> [Hint: Think "pipe"...]
[sorry for the delay]
For the pipe use-case, wouldn't you just use the capabilities of
run-program to create the necessary streams for you if you're expecting
them to be closed by run-program anyway?
I assume that if someone wants to pass in a stream, he probably wants to
control it too. The typical (?) use would be using with-open-file and
that complains if run-program closes the stream.
>
> Perhaps the behavior should be controllable by a new keyword option,
> but the default should *definitely* remain the same as it is now!
Yes, I wouldn't want to gratuitously break existing programs, unless
absolutely necessary.
Ray
P.S. Perhaps this is better discussed on the cmucl list instead?