Check the hashbang (magic number) in a executable file.

[hongy...@gmail.com]

See the following testing:

$ dd if=/usr/bin/find bs=1 count=2 2>/dev/null |od -bc
0000000 177 105
177 E
0000002

$ dd if=/usr/bin/ls bs=1 count=2 2>/dev/null |od -bc
0000000 177 105
177 E
0000002

$ ascii -t \# \!
2/3 35 0x23 0o43 00100011
2/1 33 0x21 0o41 00100001

So, it seems that for a non-script executable file, the magic number is always the following sequence:

$ ascii -t o177 o105
7/15 127 0x7F 0o177 01111111
4/5 69 0x45 0o105 01000101

Amy I right?

Regards,
HZ

[Eli the Bearded]

In comp.unix.shell, hongy...@gmail.com <hongy...@gmail.com> wrote:
> See the following testing:

man file
man magic

Maybe something like

https://openpreservation.org/blogs/magic-editing-and-creation-primer/

Or just find non-compiled version of the magic file.

Elijah
------
last attempted to edit a magic file twenty plus years ago

[Keith Thompson]

You seem to be assuming that a file's "magic number" is always 2 bytes.

man 5 elf

--
Keith Thompson (The_Other_Keith) Keith.S.T...@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

[David W. Hodgins]

On Fri, 17 Sep 2021 19:54:42 -0400, hongy...@gmail.com <hongy...@gmail.com> wrote:
> So, it seems that for a non-script executable file, the magic number is always the following sequence:
>
> $ ascii -t o177 o105
> 7/15 127 0x7F 0o177 01111111
> 4/5 69 0x45 0o105 01000101
>
> Amy I right?

See https://wiki.osdev.org/ELF#Header

Regards, Dave Hodgins

--
Change dwho...@nomail.afraid.org to davidw...@teksavvy.com for
email replies.

[hongy...@gmail.com]

On Saturday, September 18, 2021 at 11:04:42 AM UTC+8, David W. Hodgins wrote:
> On Fri, 17 Sep 2021 19:54:42 -0400, hongy...@gmail.com <hongy...@gmail.com> wrote:
> > So, it seems that for a non-script executable file, the magic number is always the following sequence:
> >
> > $ ascii -t o177 o105
> > 7/15 127 0x7F 0o177 01111111
> > 4/5 69 0x45 0o105 01000101
> >
> > Amy I right?
> See https://wiki.osdev.org/ELF#Header

Thank you and all replies here. I rechecked and confirmed it as the following:

$ dd if=/usr/bin/find bs=1 count=4 2>/dev/null | hd -c
00000000 7f 45 4c 46 |.ELF|
0000000 177 E L F
0000004

$ man 5 elf | grep -i 'magic'
EI_MAG0 The first byte of the magic number. It must be filled with ELFMAG0. (0: 0x7f)
EI_MAG1 The second byte of the magic number. It must be filled with ELFMAG1. (1: 'E')
EI_MAG2 The third byte of the magic number. It must be filled with ELFMAG2. (2: 'L')
EI_MAG3 The fourth byte of the magic number. It must be filled with ELFMAG3. (3: 'F')

Best, HZ

[Lew Pitcher]

On Fri, 17 Sep 2021 18:32:49 -0700, Keith Thompson wrote:

> "hongy...@gmail.com" <hongy...@gmail.com> writes:
>> See the following testing:
>>
>> $ dd if=/usr/bin/find bs=1 count=2 2>/dev/null |od -bc
>> 0000000 177 105
>> 177 E
>> 0000002
>>
>> $ dd if=/usr/bin/ls bs=1 count=2 2>/dev/null |od -bc
>> 0000000 177 105
>> 177 E
>> 0000002
>>
>> $ ascii -t \# \!
>> 2/3 35 0x23 0o43 00100011
>> 2/1 33 0x21 0o41 00100001
>>
>> So, it seems that for a non-script executable file, the magic number is always
>> the following sequence:
>>
>> $ ascii -t o177 o105
>> 7/15 127 0x7F 0o177 01111111
>> 4/5 69 0x45 0o105 01000101
>>
>> Amy I right?
>
> You seem to be assuming that a file's "magic number" is always 2 bytes.
>
> man 5 elf

Hongy also seems to be assuming that a "non-script executable file" only
consists of an ELF executable. Under Linux, this is not true, as almost
/any/ file can be executed using the binfmt_misc facility. This facility
allows the sysop to define new "magic numbers" for "non-script executable"
files (such as a+x WAV files, or a+x pdf files) and the interpreter that
the kernel will automagically invoke to execute that sort of file.

See /usr/src/linux/Documentation/binfmt_misc.txt for details, or
https://linuxgazette.net/118/lg_tips.html#tips.2 for my short summary.

--
Lew Pitcher
"In Skills, We Trust"

[Lew Pitcher]

Example of binfmt_misc use:
As root:
mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
echo :audio:M::RIFF::/usr/bin/paplay: >/proc/sys/fs/binfmt_misc/register

As user:
12:56 $ ls -l HELLO.WAV
-rw-r--r-- 1 lpitcher users 25956 Feb 23 2020 HELLO.WAV
12:56 $ chmod a+x HELLO.WAV
12:56 $ ./HELLO.WAV
(audio file HELLO.WAV plays on speakers)

[Ben Bacarisse]

Some of the cases could be argued to be scripts and not "non-script
executable files". Here, the audio player can be seen as an interpreter
for a sound script!

--
Ben.

[Lew Pitcher]

Ahhh, but not a script in the usual Unix sense of the word.

Given your proposed interpretation, then even ELF files become scripts,
"interpreted" by ld.so. After all, the binfmt_misc process is exactly
what happens under the (kernel) covers when exec() invokes an ELF file.

So, perhaps /all/ executables are "scripts" of some sort.

[Ben Bacarisse]

Oh, sure. I was stretching the term, but I do think "data interpreted
by some other running program" is a reasonable extension to the idea.

> Given your proposed interpretation, then even ELF files become scripts,
> "interpreted" by ld.so. After all, the binfmt_misc process is exactly
> what happens under the (kernel) covers when exec() invokes an ELF
> file.

Surely control eventually transfers to the code? The "data", once
loaded and resolved, are executable by the hardware. That's the core
distinction, isn't it?

> So, perhaps /all/ executables are "scripts" of some sort.

--

Ben.

[Lew Pitcher]

With ld.so, yes. But that's not a /requirement/.

> The "data", once
> loaded and resolved, are executable by the hardware. That's the core
> distinction, isn't it?

ELF files aren't binary images; they contain a large number of data
elements that need external processing to resolve into executable
binary data. ld.so provides that external processing, interpreting
elements (such as relocation and external linkage tables) in order
to prepare an executable binary. Even then, the whole "program"
isn't loaded directly into memory; ld.so handles page-fault requests,
providing binary image data for pages to be demand-loaded. Even the
error messages from ld.so ("Bad ELF interpreter") call ld.so an
"interpreter".

>> So, perhaps /all/ executables are "scripts" of some sort.




--

[hongy...@gmail.com]

$ ls /usr/src/linux/Documentation/binfmt_misc.txt
ls: cannot access '/usr/src/linux/Documentation/binfmt_misc.txt': No such file or directory

> or
> https://linuxgazette.net/118/lg_tips.html#tips.2 for my short summary.

Thank you for your wonderful and elaborate tips.

HZ

[hongy...@gmail.com]

$ sudo mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
mount: /proc/sys/fs/binfmt_misc: none already mounted on /sys/fs/bpf.

[Lew Pitcher]

On Sat, 18 Sep 2021 17:00:56 -0700, hongy...@gmail.com wrote:

> On Saturday, September 18, 2021 at 11:36:19 PM UTC+8, Lew Pitcher wrote:

[snip]

>> Hongy also seems to be assuming that a "non-script executable file" only
>> consists of an ELF executable. Under Linux, this is not true, as almost
>> /any/ file can be executed using the binfmt_misc facility. This facility
>> allows the sysop to define new "magic numbers" for "non-script executable"
>> files (such as a+x WAV files, or a+x pdf files) and the interpreter that
>> the kernel will automagically invoke to execute that sort of file.
>>
>> See /usr/src/linux/Documentation/binfmt_misc.txt for details,
>
> $ ls /usr/src/linux/Documentation/binfmt_misc.txt
> ls: cannot access '/usr/src/linux/Documentation/binfmt_misc.txt': No such file or directory

Did you install the appropriate kernel source code?

/usr/src/linux usually symlinks to the Linux kernel source code (which is
named with the full kernel version, something like /usr/src/linux-4.4.276)
and the Documentation subdirectory of that source code directory contains
the binfmt_misc.txt file.

If you don't want to install the kernel source code, you might look at
https://lwn.net/Articles/679310/
which contains a version of the binfmt_misc.txt file

>
>> or
>> https://linuxgazette.net/118/lg_tips.html#tips.2 for my short summary.
>
> Thank you for your wonderful and elaborate tips.

You're welcome

[Lew Pitcher]

Try
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
as root

I don't often mount binfmt_misc manually. I automount it through
an /etc/fstab entry that looks like:

none /proc/sys/fs/binfmt_misc binfmt_misc defaults 0 0


[snip]

HTH

[Chris Elvidge]

On 19/09/2021 03:42 pm, Lew Pitcher wrote:
> On Sat, 18 Sep 2021 17:00:56 -0700, hongy...@gmail.com wrote:
>
>> On Saturday, September 18, 2021 at 11:36:19 PM UTC+8, Lew Pitcher wrote:
> [snip]
>>> Hongy also seems to be assuming that a "non-script executable file" only
>>> consists of an ELF executable. Under Linux, this is not true, as almost
>>> /any/ file can be executed using the binfmt_misc facility. This facility
>>> allows the sysop to define new "magic numbers" for "non-script executable"
>>> files (such as a+x WAV files, or a+x pdf files) and the interpreter that
>>> the kernel will automagically invoke to execute that sort of file.
>>>
>>> See /usr/src/linux/Documentation/binfmt_misc.txt for details,
>>
>> $ ls /usr/src/linux/Documentation/binfmt_misc.txt
>> ls: cannot access '/usr/src/linux/Documentation/binfmt_misc.txt': No such file or directory
>
> Did you install the appropriate kernel source code?
>
> /usr/src/linux usually symlinks to the Linux kernel source code (which is
> named with the full kernel version, something like /usr/src/linux-4.4.276)
> and the Documentation subdirectory of that source code directory contains
> the binfmt_misc.txt file.

I found :
/usr/src/linux-5.13.13/Documentation/admin-guide/binfmt-misc.rst

>
> If you don't want to install the kernel source code, you might look at
> https://lwn.net/Articles/679310/
> which contains a version of the binfmt_misc.txt file
>
>>
>>> or
>>> https://linuxgazette.net/118/lg_tips.html#tips.2 for my short summary.
>>
>> Thank you for your wonderful and elaborate tips.
>
> You're welcome
>


--

Chris Elvidge
England

[David W. Hodgins]

The binfmt_misc directory in /sys is created when the kernel module binfmt_misc is
loaded.

https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/binfmt-misc.rst

In systemd based systems, the module is loaded by systemd if systemd-binfmt.service
has been enabled.

https://www.man7.org/linux/man-pages/man5/binfmt.d.5.html

In non-systemd based systems it's done by the first service that needs it in
the initd startup script. For example wine normally has ...
if [ -e /proc/sys/fs/binfmt_misc/windows ]; then
failure
echo
gprintf "Wine Registration already enabled!\n"
exit 1
fi
/sbin/modprobe binfmt_misc &>/dev/null
RETVAL=$?
echo ':windows:M::MZ::/usr/bin/wine:' >/proc/sys/fs/binfmt_misc/register
echo ':windowsPE:M::PE::/usr/bin/wine:' >/proc/sys/fs/binfmt_misc/register

[Spiros Bousbouras]

On Sun, 19 Sep 2021 14:42:14 -0000 (UTC)
Lew Pitcher <lew.p...@digitalfreehold.ca> wrote:

> If you don't want to install the kernel source code, you might look at
> https://lwn.net/Articles/679310/
> which contains a version of the binfmt_misc.txt file

There is also man update-binfmts

[Kenny McCormack]

In article <0E1AooXb...@bongo-ra.co>,

I found it at:

https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/Pedantic

[Ben Bacarisse]

So what characterises the distinction you were using between script and
non-script executable files when you proposed sound files as an example
of the latter?

I agree it's woolly, but if you don't like my take on it, where is your
line?

--
Ben.

[Lew Pitcher]

TBH, I don't really /have/ a line. If I were pressed, I might say that I
make a distinction between executables that use the "shebang", and
executables that do not. To me (my personal opinion) a "shebang" clearly
indicates a "script".

Just my (friendly) 2 cents worth

[Lew Pitcher]

I should point out that, in Linux, the kernel portion of the exec() syscall
is implemented in fs/exec.c. It invokes several fs/binfmt_*.c modules to handle
the executable exec() was passed. These binfmt_* modules include
binfmt_aout.c which handles the a.out format executables
binfmt_elf.c which handles ELF executables
binfmt_em86.c which handles X86 code execution on Alpha processors
binfmt_flat.c which handles uClinux FLAT format binaries
binfmt_elf_fdpic.c which handles uClinux FDPIC format binaries
binfmt_misc.c which handles user-defined executables
and
binfmt_script.c which handles "scripts starting with #!"

[David W. Hodgins]

On Sun, 19 Sep 2021 21:26:54 -0400, Lew Pitcher <lew.p...@digitalfreehold.ca> wrote:
> TBH, I don't really /have/ a line. If I were pressed, I might say that I
> make a distinction between executables that use the "shebang", and
> executables that do not. To me (my personal opinion) a "shebang" clearly
> indicates a "script".

Based on what I've been taught ...

A program contains machine language instructions (and some data), usually created by
an assembler, and if needed, a compiler. It usually uses a dynamic linker to patch
in addresses needed for relocation and handle segment loading, but otherwise is
processed directly by the cpu.

A script is interpreted as it's executed. The interpreter generates the machine
language instructions dynamically, based on the contents of the script. The hashbang
just allows automatic selection of the interpreter to use when selecting the file for
processing. Some interpreters allow the generated machine language to be stored for
later execution. Those compiled scripts are now programs.

While binfmt allows a data file to include automatic resolution of what program or
interpreter to use allowing clicking on the file, the data file does not contain
any machine instructions. It's still just data.

[Janis Papanagnou]

I very much agree with what you've described here, but one detail;
'program' we called the textual "recipe"; we have C-programs (to be
compiled) or shell-programs (~"shell-Scripts") to be interpreted, we
have assembler-programs (the text, not the assembled machine code).

Janis

[hongy...@gmail.com]

On Sunday, September 19, 2021 at 10:42:19 PM UTC+8, Lew Pitcher wrote:
> On Sat, 18 Sep 2021 17:00:56 -0700, hongy...@gmail.com wrote:
>
> > On Saturday, September 18, 2021 at 11:36:19 PM UTC+8, Lew Pitcher wrote:
> [snip]
> >> Hongy also seems to be assuming that a "non-script executable file" only
> >> consists of an ELF executable. Under Linux, this is not true, as almost
> >> /any/ file can be executed using the binfmt_misc facility. This facility
> >> allows the sysop to define new "magic numbers" for "non-script executable"
> >> files (such as a+x WAV files, or a+x pdf files) and the interpreter that
> >> the kernel will automagically invoke to execute that sort of file.
> >>
> >> See /usr/src/linux/Documentation/binfmt_misc.txt for details,
> >
> > $ ls /usr/src/linux/Documentation/binfmt_misc.txt
> > ls: cannot access '/usr/src/linux/Documentation/binfmt_misc.txt': No such file or directory
> Did you install the appropriate kernel source code?
>
> /usr/src/linux usually symlinks to the Linux kernel source code (which is
> named with the full kernel version, something like /usr/src/linux-4.4.276)
> and the Documentation subdirectory of that source code directory contains
> the binfmt_misc.txt file.

I've installed the kernel source code, but still can't find this file:

werner@X10DAi-00:~$ dpkg --get-selections | grep linux-source
linux-source install
linux-source-5.4.0 install
werner@X10DAi-00:~$ dpkg -L linux-source-5.4.0 | grep -i binfmt_misc.txt
werner@X10DAi-00:~$

[hongy...@gmail.com]

On Monday, September 20, 2021 at 12:13:27 AM UTC+8, Chris Elvidge wrote:
> On 19/09/2021 03:42 pm, Lew Pitcher wrote:
> > On Sat, 18 Sep 2021 17:00:56 -0700, hongy...@gmail.com wrote:
> >
> >> On Saturday, September 18, 2021 at 11:36:19 PM UTC+8, Lew Pitcher wrote:
> > [snip]
> >>> Hongy also seems to be assuming that a "non-script executable file" only
> >>> consists of an ELF executable. Under Linux, this is not true, as almost
> >>> /any/ file can be executed using the binfmt_misc facility. This facility
> >>> allows the sysop to define new "magic numbers" for "non-script executable"
> >>> files (such as a+x WAV files, or a+x pdf files) and the interpreter that
> >>> the kernel will automagically invoke to execute that sort of file.
> >>>
> >>> See /usr/src/linux/Documentation/binfmt_misc.txt for details,
> >>
> >> $ ls /usr/src/linux/Documentation/binfmt_misc.txt
> >> ls: cannot access '/usr/src/linux/Documentation/binfmt_misc.txt': No such file or directory
> >
> > Did you install the appropriate kernel source code?
> >
> > /usr/src/linux usually symlinks to the Linux kernel source code (which is
> > named with the full kernel version, something like /usr/src/linux-4.4.276)
> > and the Documentation subdirectory of that source code directory contains
> > the binfmt_misc.txt file.
> I found :
> /usr/src/linux-5.13.13/Documentation/admin-guide/binfmt-misc.rst

This file comes from linux-doc package:

$ apt-file search binfmt-misc.rst
linux-doc: /usr/share/doc/linux-doc/admin-guide/binfmt-misc.rst.gz
linux-doc: /usr/share/doc/linux-doc/html/_sources/admin-guide/binfmt-misc.rst.txt