Mange your IMSAI8080esp from the command line

[TheHighNibble]

Note: "command line":

- meaning the Windows/Linux/macOS command line of another computer

- not the CP/M command line of the IMSAI8080esp

• The attached file is a python script (only tested to work with python3)
  • it has two (2) module dependencies
    • requests
    • websocket-client
  • these need to be installed (with pip) for this script to work
• It provides a command line interface to many of the same features of the desktop UI (webfrontend) for the following devices:
  • SYS:
  • CFG: (access to the config files)
  • DSK:
  • x:DSK:
  • LIB:
  • MAN:
  • CPA:
• the script can be made executable and renamed whatever you like
• if you've changed the network name of your IMSAI8080esp
  • edit the script, changing the following line as required
  • baseurl = 'http://imsai8080'
• run the script with no parameters and it will show basic help
• run the script as follows to see all the detailed help:
  • imsai.py help all

Why? Because sometimes I want to remotely manage my IMSAI without the desktopUI. 

Why a script? This version can probably be considered Alpha or early Beta right now. When its field tested I'd like to package it as an executable.

Why python? I've never used it before, the alternative was JavaScript. It turned out pretty well from my perspective.

Enjoy; discuss; provide feedback and suggestions.

Dave.

[udo....@freenet.de]

This is really cool. While playing around I ran into this problem:

udos-mbp:esp32 udo$ ./imsai.py man:

8080_Assembly.pdf

CP-M 80 Kermit.pdf

CP-M_2.2.pdf

Traceback (most recent call last):

  File "./imsai.py", line 391, in <module>

    sections[sect](args[1:])

  File "./imsai.py", line 188, in man_f

    print(fmt.format(m))

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 13: ordinal not in range(128)

[udo....@freenet.de]

The next in the manual list would be Cromemco D+7A IO, looks like it has a character with upper bit set in the name.

[TheHighNibble]

Can you check your python version using:

    python --version
I am using 3.9.5 and as noted above this script was only tested using python3

From the little I know, python3 handles unicode "natively", but python 2.x does not.

You may need to edit the first line of the script to read:

#!/usr/bin/env python3

assuming you have python3 installed, and it's not already your default python version (macOS still ships only with python 2.7, and I see you are on your mbp)

[TheHighNibble]

Updated script attached. Now lets you:

• remotely update the firmware : 
  • imsai.py sys: --update imsaisim_esp32.bin
• remotely update the webfrontend (etc...): 
  • imsai.py sys: --update update.bin

[udo....@freenet.de]

TheHighNibble schrieb am Donnerstag, 8. Juli 2021 um 00:49:26 UTC+2:

Can you check your python version using:

    python --version
I am using 3.9.5 and as noted above this script was only tested using python3

From the little I know, python3 handles unicode "natively", but python 2.x does not.

You may need to edit the first line of the script to read:

#!/usr/bin/env python3

assuming you have python3 installed, and it's not already your default python version (macOS still ships only with python 2.7, and I see you are on your mbp)

Right, by default OSX uses python 2. I have installed python 3 and modified the script to execute that and now

it works:

udos-mbp:esp32 udo$ ./imsai.py man:

8080_Assembly.pdf

CP-M 80 Kermit.pdf

CP-M_2.2.pdf

Cromemco D+7A IO.pdf

Cromemco Dazzler Games.pdf

Cromemco JS-1 Joystick.pdf

Cromemco_88_ACC.pdf

Cromemco_88_CCC.pdf

Cromemco_Dazzler.pdf

IMSAI_8080.pdf

Memon80 Users Manual.pdf

Z80_R_Zaks.pdf

 

[John Kennedy]

Is there a specific JSON module to import? On both a Mac and PC, the script fails on launch at line d_sys = sys_get.json()?

[John Kennedy]

Ah.. maybe the Python environment is just fine but the call isn't successfully reaching the imsai device on my network for some reason (I am looking at it right now in the web browser, but, you know.. computers. Who really knows how they work?)

[TheHighNibble]

The json module should already be available under python3.

Did you use pip3 to install both the requests and the websocket-client modules?

If you used pip, then they got installed for python 2.x

[John Kennedy]

I did use pip3, yes. Doing some experiment from REPL proves that the JSON module is present and working.

Further,

sys_get.text returns all the good stuff (wifi details, port names, devices, ROMs..)

sys_get,json() throws the wobbly.

So I have narrowed it down:

r = requests.get('https://api.github.com/events')

r.json()

works perfectly. So that's ok. All the tools are installed.

And:

r = requests.get('http://imsai8080/manual')

r.json()

works but

r = requests.get('http://imsai8080/system')

r.json()

doesn't. 

Therefore I must have put something bad in the system config file while editing it. I will check!

[John Kennedy]

Yeah! Fixed it! I had copied and pasted the suggested new settings for the serial ports but they used spaces rather than =, and that didn't make the JSON happy.

i.e.

These settings from your docs:

#SIO 2 Channel B, Ports 34/35 connectted to UART1/RS232-2 (default)

sio2a_upper_case  0

sio2a_strip_parity   0

sio2a_drop_nulls    1

sio2a_baud_rate   19200

should be:

#SIO 2 Channel B, Ports 34/35 connectted to UART1/RS232-2 (default)

sio2a_upper_case=0

sio2a_strip_parity=0

sio2a_drop_nulls=1

sio2a_baud_rate=19200

[John Kennedy]

I remember at the start of the project I wondered if there was going to be a way to control some of the LEDs remotely, that way I could use the IMSAI to display the time, or twitter followers or something.

Getting closer :-)

[udo....@freenet.de]

johntk...@gmail.com schrieb am Freitag, 9. Juli 2021 um 15:56:37 UTC+2:

Yeah! Fixed it! I had copied and pasted the suggested new settings for the serial ports but they used spaces rather than =, and that didn't make the JSON happy.

i.e.

These settings from your docs:

#SIO 2 Channel B, Ports 34/35 connectted to UART1/RS232-2 (default)

sio2a_upper_case  0

sio2a_strip_parity   0

sio2a_drop_nulls    1

sio2a_baud_rate   19200

should be:

#SIO 2 Channel B, Ports 34/35 connectted to UART1/RS232-2 (default)

sio2a_upper_case=0

sio2a_strip_parity=0

sio2a_drop_nulls=1

sio2a_baud_rate=19200

No, that should be white space (blank(s) or tab(s)), the = will not work. Because the syntax is wrong these

commands will be ignored and the default setting used, which is 0 for all flags and baud rate 0 is

unlimited.

Had a look at my config, I used tabs, don't know if json is unhappy with spaces.

[John Kennedy]

ok! I'll give it another go, using tabs.

[villa...@gmail.com]

John, did you resolve your JSON issue

I have same problem.

Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license()" for more information.
>>>
================== RESTART: /home/colin/Downloads/imsai.py ==================

Traceback (most recent call last):

  File "/home/colin/Downloads/imsai.py", line 42, in <module>
    d_sys = sys_get.json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 889, in json
    self.content.decode(encoding), **kwargs
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 1923 (char 1922)

Colin

[John Kennedy]

Not really! If I use a tab,  it fails. If I use a single space, it fails.

[John Kennedy]

I tried tabs - it failed.

On Friday, July 9, 2021 at 10:19:51 AM UTC-7 udo....@freenet.de wrote:

[John Kennedy]

Oh wait a minute!

I had added the serial stuff to boot.conf, when I needed to add it to system.conf

SMH sorry - it's all working now. 

Colin, make sure you are adding the serial settings to system.conf and using a tab.

[TheHighNibble]

|   I had added the serial stuff to boot.conf, when I needed to add it to system.conf

Right! I had come to the same conclusion but had been testing it for the past hour, to be sure.

ThenI come back to the forum to post some polite questions - and see that you've found your own solution (see polite!) :-)

[udo....@freenet.de]

johntk...@gmail.com schrieb am Freitag, 9. Juli 2021 um 15:59:32 UTC+2:

I remember at the start of the project I wondered if there was going to be a way to control some of the LEDs remotely, that way I could use the IMSAI to display the time, or twitter followers or something.

Getting closer :-)

If one could send something to the console device then we could remote execute CP/M commands

with such a python script. 

[TheHighNibble]

A quick and dirty solution is attached.

• added a tty: device. See the help
• it just jams the text you provide down the websocket for TTY: as if it has been typed (at 10 characters per second, followed by a <cr>)
• there are caveats:
  • you don't see the command you send typed on the console as the "echo" is to the websocket and this is ignored
  • this wont work correctly if the TTY: device on the desktop UI is connected at the time as it breaks the websocket being used by TTY: 
  • won't work if anything other than tty: is your console (with webtty in the HAL) 

The better solution will be to implement a new interface for console commands (eg. con:, aka webcon in the HAL) that only accepts input and doesn't capture output (or something like that, low priority in my mind as the solution I've already provided here can cover enough use cases).

[villa...@gmail.com]

OK, making progress! On windows and linux I get the same problem 

If I comment out line 35 #sys_get = requests.get(baseurl + '/system')     and  line 42 #d_sys = sys_get.json()

then all remaining commands are working perfectly .  E.g can eject disk etc

If enabled I get :-

C:\Users\Admin\AppData\Local\Microsoft\WindowsApps>python imsai.py man:

Traceback (most recent call last):

  File "imsai.py", line 42, in <module>

    d_sys = sys_get.json()

  File "C:\Users\Admin\AppData\Roaming\Python\Python38\site-packages\requests\models.py", line 900, in json

    return complexjson.loads(self.text, **kwargs)

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 357, in loads

    return _default_decoder.decode(s)

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 353, in raw_decode

    obj, end = self.scan_once(s, idx)

json.decoder.JSONDecodeError: Invalid control character at: line 1 column 1923 (char 1922)

SAME COMMAND with commented out

C:\Users\Admin\AppData\Local\Microsoft\WindowsApps>python imsai.py man:

        8080 Assembly.pdf

        CP-M 2.2.pdf

        CP-M 80 Kermit.pdf

        Cromemco 88 ACC.pdf

        Cromemco 88 CCC.pdf

        Cromemco D+7A�IO.pdf

        Cromemco Dazzler Games.pdf

        Cromemco Dazzler.pdf

        Cromemco JS-1 Joystick.pdf

        IMSAI 8080 User Manual.pdf

        Memon80 Users Manual.pdf

        Z80 R Zaks.pdf

Unfortunately my software knowledge is very limited to pursue, it's not a big problem in as much it is very useful as is but if any helpers I will try and sort out.

Thanks for all help thus far.

 PS Imsai at latest revision 

[John Kennedy]

If the script works when you comment out the line referencing SYSTEM, that means that (like me) you have something that isn't in a format that the python script can parse using JSON.

To find out what it might be: 

1. On the IMSAI GUI, click on SYS: 

2. Then click on the cog in the top right to open the Config Files window.

3. Click on boot.conf

The contents of boot.conf will be displayed in the right-most pane.

There must NOT be any mention of SIO ports. If you accidentally added that setting to the boot.conf file, then there is your problem - take it out. That's the error I made.

The SIO port stuff goes into the system.conf file not the boot.conf file.

If you didn't, then something else in the file - perhaps the formatting from a previous experiment - is causing the JSON parser to fail.

Make sure any parameters are separated by = and not tabs or spaces. Feel free to share a copy of it BUT REMOVE YOUR PRIVATE NETWORK DETAILS.

[villa...@gmail.com]

Thank you John for your help and perseverance/patience with me. ALL IS WORKING!!

My boot.conf was all enabled on the following 4 items

#SIO1.portA.device=WEBTTY,UART0

#SIO1.portB.device=VIOKBD

#SIO2.portA.device=MODEM

#SIO2.portB.device=UART1

So I commented them all out and all worked correctly as far as imsai.py was concerned.  Will test further tomorrow my other Imsai progs (Pi 4 terminal on UC1: and kermit

Thank you Dave for the program and John for sorting me out !!

[John Kennedy]

That seems to work! I can whip up some .COM programs to accept a number as a param and set the programmable LEDs! :-)

[John Kennedy]

Someone did all the hard work, and wrote a Turbo Pascal program to output to Z80's ports.

https://groups.google.com/g/rc2014-z80/c/vO9DLwYiMy8

Admittedly although the Turbo Pascal program works, I can't quite get the python script to trigger it. Maybe parameters aren't sent by the script? I will tinker.

[John Kennedy]

Yup, that was it, the extra params are ignored by the Python. Quickly hacking the script  - 

  msg = args[0] + " " + args[1] + " " + args[2]

 

means that when I enter 

./imsai.py tty: i:remote 255 15 

(where remote is REMOTE.COM - the CP/M program compiled from the Turbo Pascal)

then half the LEDs turn on :-) :-) :-)

[TheHighNibble]

Good catch, a more robust solution is to use the line: msg = ' '.join(args)

I've attached a new copy of the script that includes this mod along with a few other improvements including:

• lets you know if the IMSAI is unreachable on the network
• gives you some idea where in the boot.conf file a problem stopped things producing good JSON

[John Kennedy]

Thanks - works!

[TheHighNibble]

New version:

• adds LPT: device that dumps the contents of the printer to stdout

Works fine on Z80PACK imsaisim, not so useful on IMSAI8080esp until the next firmware update (v.1.8.0) is available (next few days)

The reason is that on the ESP32 the printer file can't be read by another process while it is open, so v1.8.0 firmware closes the file after 0.5-1 sec of inactivity.

About time I get this onto a GitHub repo for some proper version control.

[udo....@freenet.de]

Cool, but how can I clean the print buffer? Tried reset printer in the Web GUI, but that cleans the paper only, the buffer contents still is there. Lots of stuff I printed into it sometimes ;-)

[TheHighNibble]

Currently the only way is to restart the simulator/reboot the ESP32, depending on your platform.

Once I release the v1.8.0 firmware for the IMSIA8080esp then any coldboot of CP/M will clear the buffer, or if you send 0x80 to the printer (as the CP/M BIOS does when it boots).

I will look at updating the LPT: reset button on the webUI to trigger this also.

If I do this, then the imsai.py script can have this feature added also.

The code for imsaiaim on the PC platforms will need to be modified to follow the same behaviour.

[udo....@freenet.de]

Unfortunately what I printed still is there if I completely switch it off and later on again.

For some reason it won't clean the printer file.

[TheHighNibble]

Yes, you are correct. This feature will be useless until I release V1.8.0

All because the esp-idf libraries for file handling are not thread aware. So I have resorted to using a timeout period after which I have to close the printer file. 

[TheHighNibble]

With the V1.8.0 firmware released, the LPT: device function of the imsai.py tool is now useful, and I have added a `--reset` option (see attached for the latest version)

[Mike Kostersitz]

Just grabbed the latest version of the script. 

When I try to upload a dsk to the library 

I get this error, this is on my Mac. I also am not able to drag and drop the dsk to the LIB: folder in the web UI, which I thought should work.

mikek@Onyx imsai8080 % python3 imsai.py lib: --upload ~/Downloads/wargames.dsk
LIB: UPLOAD /Users/mikek/Downloads/wargames.dsk

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/mikek/imsai8080/imsai.py", line 494, in <module>
    sections[sect](args[1:])
  File "/Users/mikek/imsai8080/imsai.py", line 189, in lib_f
    lib_file = requests.put(baseurl + '/library?' + lib_name,
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 132, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

Any idea what could be wrong?