I think I have figured out the YottaDB IO issues (? bugs ?)

[Kevin Toppenberg]

I have previously posted about how YottaDB doesn't seem to be handling IO correctly or consistently.  See my other posts about this issue.  

To summarize, I found two issues (?bugs?) that I will call the UNSAVED NOCANONICAL and the UNSAVED NOECHO issues.

Some terminology:

• Canonical is a term defined by the IO / TTY system in Unix, and it means that the TTY system provides some very basic line editing functionality.  Thus, when in canonical mode, and if the user types "CATZ<BACKSPACE>S", then when code reads from the input buffer, it will receive "CATS". But if the mode has been changed to noncanonical, the read will return exactly what the user typed, e.g.  "CATZ<BACKSPACE>S".
• Echo is a term again defined by the TTY system to mean that when the user types a character, the TTY system will directly output this character to the output channel (i.e. the screen of the terminal).  The is completely separate from what the program might want to do with the characters.  So if the program were to ask the user to type a secret password, and ECHO is on, the password would be printed out in plaintext as it is typed in. 

Examples of the problem:

Canonical issue:

USE $P:(NOCANONICAL) HANG 3 READ *X:0.1 WRITE !,X,!

If user pressed BACKSPACE during the hang, then X should be filled with 127, but instead it return -1 (nothing read).

Echo issue

USE $P:(NOECHO) HANG 5 READ X#3 WRITE !,X,!

ABC   <---- user types this during the HANG, TTY puts onto screen

ABC   <--- this is my code putting output to screen.  

[Kevin Toppenberg]

I hit post accidentally.  I wasn't finished with post, so continuing.  

QUESTION: What is the problem?

ANSWER:  The TTY settings are not being stored in local yottadb structures.  Later, after a READ is completed, resetterm() is called which pulls from the local structures.  Since the settings were not saved there, it reverts back to prior values, ignoring last USE command.    For the ECHO issue, yottadb was setting a mask value, which presumably was used elsewhere in managing characters coming in.  But NOECHO/ECHO settings are never sent to the TTY system, so it will continue to write directly to the screen.  Again, ignoring parameter in USE command.  

Below is some analysis I did of code.  I did step-by-step evaluation of test code, setting key breakpoints and examined the values of data structures.  I was after a few weeks of doing this that I figured out the issue. 

Working test code:

TEST

  use $P:(NOCANONICAL:NOECHO)

  w "[" HANG 10 w "].. Enter key:" READ *X:0.1 write X,!

  quit

Significant execution events

• Launch execution via DO TEST^TMGTEST

• resetterm() called.  

  • Stack is

    • resetterm(io_desc * iod) 

    • dm_read(mval * v) 

    • op_dmode() 

    • tkok3() 

    • [Unknown/Just-In-Time compiled code]

  • At end of function, we have flag set indicating that setterm() is needed: ttptr->setterm_done_by = 0;  

• Op_use() called

  • Stack is

    • op_use(mval * v, mval * p) 

    • [Unknown/Just-In-Time compiled code]

  • This sets up ’nl’ variable (logical record for the passed name ‘/dev/pts/1’), which holds a pointer to io data structures. 

  • active_device (global scope var) set to nl->iod

  • Io_curr_device (global scoped var) set to nl->iod->pair

  • This next calls iott_use, passing in nl->iod->pair.out received as iod pointer

• iott_use(iod) called

  • tt_ptr set from iod-> dev_sp  (Device Structure Pointer)

  • Current tty settings are queried via tcgetattr(), into variable ‘t’ (type ‘termios’ e.g. ‘term-io-struct’).  NOTE: These are the real settings from the TTY system.  

  • Local var mask_in set from iod->pair.in->dev_sp->term_ctrl

  • Local var mask_term set from iod->pair.in->dev_sp->mask_term. This seems to be the terminator chars used to indicate end of input.  

  • Because of NOCANONICAL, we have this code.

    • case iop_nocanonical:

    •   tt_ptr->canonical = FALSE;

    •   t.c_lflag &= ~(ICANON);

  • NOTE: setting t.c_lflag will result in this being sent to TTY system.  But state of ‘t’ is not saved in global scope *iod.   tt_ptr->canonical is set, but this is not later used.  mask_in, mask_term will be saved later, but they are not set here. 

  • Next, because of NOECHO, we have this code

    • case iop_noecho:

    •   mask_in |= TRM_NOECHO;

    •   break;

  • NOTE: we set mask_in, which will be saved later, but ‘t’ is not modified, so this will not be sent to the TTY system. 

  • Next we send the terminal settings, stored in local var ‘t’ to the TTY system

    • Tcsetattr(tt_ptr->fildes, TCSANOW, &t, status, save_errno, CHANGE_TERM_TRUE);

  • Next we save local mask_in and mask_term into global variables

    • temp_ptr->term_ctrl = mask_in;

    • memcpy(&temp_ptr->mask_term, &mask_term, SIZEOF(io_termmask));

  • Comment: We need to save state here ...

    • because resetterm() later needs to restore state set here.

    • resetterm() uses ttptr->ttio_struct as its source of TTY state

    • THEREFORE, t should be saved back into iod

• Continuing, we get to iott_rdone (i.e. ‘read one’)

  • This calls setterm, checking ttPtr->setterm_done_by==0, indicating need.

  • setterm() called

    • This uses tt_ptr->canonical as the indicator of state.  If FALSE (as set above) found, then code as follows.  Here ‘t’ is a local variable.

      • t.c_lflag &= ~(ICANON | ECHO);

      • t.c_cc[VTIME] = 8;

      • t.c_cc[VMIN] = 1;

      • t.c_iflag &= ~(ICRNL);Set regardless of tt_ptr->canonical

    • Next we send ‘t’ to TTY system.  NOTE: nothing here is saved in state of global var iod etc.  

      • Tcsetattr(tt_ptr->fildes, TCSANOW, &t, status, save_errno, CHANGE_TERM_TRUE);

    • Next flag is set showing that setterm has been done.

      • tt_ptr->setterm_done_by = process_id;

    • COMMENT: We do not want to save the state here.  Because settterm() is called by iott_rdone(), the TTY system is being just temporarily modified, with the idea being to restore this in resetterm().

  • Code for reading in 1 character from IO channel done next.

  • At the end of function, resetterm() is executed.

    • RESETTERM_IF_NEEDED(io_ptr, EXPECT_SETTERM_DONE_TRUE);

• resetterm() called:

  • Local var ‘t’ is obtained from global var’s ttio_struct, and passed to TTY system

    • t = *ttptr->ttio_struct;

    • At runtime, this is binary 1000101000111011

      • bold bit means CANONICAL, at variance from last USE command

    • Tcsetattr(ttptr->fildes, TCSANOW, &t, status, save_errno, CHANGE_TERM_FALSE);

    • ttptr->setterm_done_by = 0;  ← flag set to indicate setterm() will be needed again in the future.  

Summary of how state of TTY system is apparently saved

• *ttptr->ttio_struct This is the usual source of ‘t’.  It is the raw data to send to TTY

  • Used in resetterm()

  • ttio_struct.c_lflag takes the following items

    • local mode flags  (note: values below are OCTAL)

    • ISIG 0001 Enable signals.  

    • ICANON 0002 Canonical input (erase and kill processing).  

    • XCASE 0004

    • ECHO 0010 Enable echo.  

    • ECHOE 0020 Echo erase char as err-correcting backspace

    • ECHOK 0040 Echo KILL.  

    • ECHONL 0100 Echo NL.  

    • NOFLSH 0200 Disable flush after interrupt or quit.  

    • TOSTOP 0400 Send SIGTTOU for background output.  

• *ttptr->term_ctrl  ← stores mask_in

  • NOTE: Several confusing similar terms exist and it is tricky to distinguish them.

    • mask, mask_in, ttptr->mask_term.mask[i]

  • NOTE: term_ctrl seems to be the place that SHOULD be holding the TTY state.  I see many places in the code where it is modified and bit set, but I find only few places that it is used. It does seem to be the internal representation that yottadb uses to store the logical state, but it is NOT directly passed to the TTY system.

    • I see this  → term_setup(term_ctrl); //found in io_init(), but I think this ‘term_ctrl’ is different.  In term_setup(), this input param is called “ctrlc_enable”,and seems to be simple boolean.

  • term_ctrl is a bit flag, taking the following items

    • TRM_MODIFIERS 0

    • TRM_EDITMODE 1

    • TRM_NOTYPEAHD 2

    • TRM_READSYNC 4

    • TRM_PROMPT 8

    • TRM_PASTHRU 16

    • TRM_ESCTRMOVR 32

    • TRM_ESCAPE 64

    • TRM_NOECHO 128

    • TRM_CONVERT 256

• **ttptr->canonical -- boolean to store if canonical

  • in setterm(), this variable is tested, and depending on values, c_cflag and c_cc are set

  • in iott_open(),iott_use() the value is set based on user parameters

  • in zshow_devices(), used to output zshow "D"

Changes made to fix issues -- See attached iott_use.c My changes are marked by //kt

• iott_use() -- added:

  • *(temp_ptr->ttio_struct) = t;  //save t (holding TTY settings) back into ttio_struct

• I also modified these code blocks in iott_use(). I am setting flat in 't', so that it will be sent to the TTY system.

  • case iop_echo:

  •   mask_in &= (~TRM_NOECHO);

  •   t.c_lflag |= ECHO;  //kt ADDED

  •   break;

  • case iop_noecho:

  •   mask_in |= TRM_NOECHO;

  •   t.c_lflag &= (~ECHO);  //kt ADDED

  •   break;

Changing the IO function of yottadb is a really big issue.  So I have been quite anxious and nervous about this.  I have tried to consider any adverse consequences that might arise from these changes but can never anticipate every possible outcome.   I would REALLY appreciate someone checking my work and giving me feedback.   

Thanks

Kevin

[Sam Habiel]

I had to work on this code recently, so I am not surprised that you found bugs in that code. I think your changes sound correct; as we had to make similar changes.

So you didn't say: Does this actually fix your problem?

--Sam

--
You received this message because you are subscribed to the Google Groups "Everything MUMPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everythingmum...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/352d0b04-6328-46aa-90ae-3f6ff0e22e54n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Toppenberg]

Yes, it does.

What I am concerned about is, does it introduce other errors. For example, sometimes when I am messing around with the settings, suddenly I can’t type at the command prompt anymore, and I have to close the terminal window. I haven’t been able to reproduce the steps that caused that to happen.

Kevin

[Sam Habiel]

Do me a small favor: Play around with various terminal programs in VistA (Screenman, Listman, your debugger) on the new compiled version. If they all work, I would say it's probably safe and we can move on to the next steps.

--Sam

[Kevin Toppenberg]

Sam,

I'm happy to do that.  The only problem is that I have been developing in an otherwise empty virtual machine.  I was able to do all my IO testing in yottadb -dir without even setting up database files.  But if I am going to test my new yottadb executable against a vista system, I need an area in which to test.  So I downloaded your vehu docker container and got that running so I could get to the command prompt in mumps and at a room prompt inside the container.  But when I tried swapping the working yottadb for my newly compiled yottadb, I get an error about glibc.   My VM where I compiled has version 2.4, and the vehu container has 2.34.  I tried doing an update via yum, and it actually downgraded it to 2.28.  ChatGPT tells me it is probably not worth trying to compile a newer version of glibc, since so many things depend on this an I likely will break other elements of my docker container. 

Do you happen to have any instructions about how I could go about getting vista working in my VM?  I know there are many steps, and it has been many years since I got our system going.

Merry Christmas

Kevin

[Sam Habiel]

I will give you detailed instructions another day. Briefly:

1. Log in as root into the VEHU image

2. Install all the packages in YDB README

3. Git clone the branch you intend to compile/install

4. Compile/install

5. Log in as vehu and kill all VistA processes and rundown the databases

5. Remove the symbolic link in ~vehu/lib/gtm and repoint it to your new installation

6. Restart the container

7. Go into VistA and test

Do you have a Git branch I can try?

--Sam

--

You received this message because you are subscribed to the Google Groups "Everything MUMPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everythingmum...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/64da5add-31c0-4843-afce-51e9cee8dfaen%40googlegroups.com.

[Kevin Toppenberg]

Sam,

Thanks for those instructions.  I'll try to work on them.  I'm visiting family in the Seattle area for the holidays.  The time I have to work on this is unpredictable.  But there is no rush.

I did not fork the yottadb source, as I don't want it showing up as a repository on my github page.  I only modified one file, iott_use.  I included it as an attachment to a post on the forum.  I will also attach it to this message.  Do a search in the file for "//kt" to see the changes I made.  

Merry Christmas,

Kevin

[Sam Habiel]

I tested for you and everything seems to work okay in VistA. Tested screenman, Listman, VPE, access/verify code hiding (which uses the "ECHO" parameters).

Here are the instructions for you to try for yourself:

docker run -d --name kev-test worldvista/vehu
docker exec -it kev-test su -
yum install sudo file cmake make gcc git curl tcsh {libconfig,libicu,ncurses,elfutils-libelf,readline}-devel binutils ca-certificates
cd /tmp/
git clone https://gitlab.com/YottaDB/DB/YDB
cd YDB
exit
docker cp iott_use.c kev-test:/tmp/YDB/sr_unix/iott_use.c
docker exec -it kev-test su -
cd /tmp/YDB
git diff
mkdir build
cd build
cmake ..
make -j16
sudo make install
cd ./yottadb_r203/
sudo ./ydbinstall --posix --octo --aim --gui
exit
docker exec -it kev-test su - vehu
mumps -r HALTALL^ZSY
pkill rocto
pkill yottadb
mupip rundown -r '*
rm lib/gtm
ln -s /usr/local/lib/yottadb/r203 ./lib/gtm
exit
docker restart kev-test
docker exec -it kev-test su - vehu
mumps -dir

Do you have a simple test case that shows how your problem got fixed?

  

--Sam

[Kevin Toppenberg]

Thanks you for this information.

Here is a test for the NOECHO issue:

u $P:(noecho) w "Type 3 chars now:[" h 5 w "] <-- if anything here, then NOECHO failed." read x#3 w !,"Output controlled value=[",x,"]",!

Here is a test for the NOCANONICAL issue:

use $p:(nocanonical) w "Type BACKSPACE key: " hang 3 read *x:0.1 write !,$s(x=127:"SUCCESS",1:"FAILED"),". Got [",x,"]",!

I'm not sure how to automate these tests, however.

Kevin

[Kevin Toppenberg]

I have found an issue with my modification:  If I execute: USE $P:(NOECHO) from the command line, yottadb works OK, but if I leave yottadb, the TTY system is left in NOECHO mode, and the linux command-line does not show commands as they are typed in.  To fix this, I have to blindly type the command to reenter yottadb, and type (with visible chars shown) USE $P:(ECHO), and then halt out of yottadb.    I think to fix this problem, I need to query and save off the state of the TTY system when yottadb starts, and then restore this before exiting. 

And actually there is a 2nd issue.  After USE $P:(NOECHO), the left-right arrow for line editing at the yottadb command prompt stops working.  Entering USE $P:(ECHO) seems to restore functionality.  I don't know why this would be behaving this way.  I'll have to look more into this.

Kevin

[Sam Habiel]

Kevin, iott_close.c has the code upon YottaDB exit.

--Sam

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/fde23469-ddf6-4fe3-9e9e-f54730ce699an%40googlegroups.com.