Need help setting up YDBTest

[Kevin Toppenberg]

I am trying to set up the YDB test environment (https://gitlab.com/YottaDB/DB/YDBTest). 

I'm reading the instructions at the bottom of this same page: https://gitlab.com/YottaDB/DB/YDBTest

I have tried to modify the .cshrc file as follows:

#!/usr/local/bin/tcsh
###########################################################
#
# Copyright (c) 2024 YottaDB LLC and/or its subsidiaries.
# All rights reserved.
#
#       This source code contains the intellectual property
#       of its copyright holder(s), and is made available
#       under a license.  If you do not know the terms of
#       the license, please stop and do not read further.
#
###########################################################

# This file sets up the cshell environment

# WARNING: this file is a template only. It should be copied into your $HOME directory and modified to suit you

set autolist
set backslash_quote
set dextract
set dunique
set filec
set lineedit
set matchbeep=nomatch
set history=1000
unset histdup  # I have occasionally had trouble, if this is set to 'erase', with the previous command not getting saved
set savehist=(1000 merge)   # if you append 'lock' then you need to make a mechanism to remove lock file when stuck
set histfile=~/.csh_history
unset autologout
unset glob
set path = ( $path /sbin/ )   # Required on Debian

set HOST=$HOST:ar
setenv HOST $HOST:ar

# Exit from .cshrc if not interactive
# Be aware that gtmtest sometimes invokes tcsh using -c or << which are considered 'interactive'.
# So don't print stuff below unless you're prepared for it to appear in the output of some tests.
if ( ! $?edit ) goto quit

bindkey "^R" i-search-back
bindkey "\e[1;5D" backward-word  # ctrl-left
bindkey "\e[1;5C" forward-word   # ctrl-right

alias ls 'ls --color=auto'
alias ll 'ls -alF'
alias la 'ls -A'

alias grep 'grep --color=auto'
alias fgrep 'fgrep --color=auto'
alias egrep 'egrep --color=auto'

# Handy git shortcuts
alias gitlog 'git log --graph --pretty=format:"%C(auto)%ad %h%d %s" --date="short"'
alias gitup 'git fetch upstream && git rebase upstream/master'
alias gitsquash 'git log --pretty=format:"%C(auto)%h%d:%n%B" origin/master..HEAD && git reset --soft origin/master && git status && echo "Squashed to origin/master. Now check and commit."'

# USER-SPECIFIC LOCATIONS for YottaDB's gtmtest
setenv mailid kdt...@secretserver.com   # Replace with your email here
setenv build_id 1648   # Your build number: any 9xx no.; assigned by YottaDB on their servers: avoids clobbering others' builds
setenv verno V${build_id}_R203   # Change Rxxx to the revision of yottadb you will install and test against
setenv work_dir /opt/worldvista/EHR/compile   # Where you wish to check out YDB, YDBTest, etc.
setenv gtm_root /usr/library   # Where your (and others') installed binaries go
setenv gtm_test /opt/worldvista/EHR/compile/YDBTest   # Where to hold a copy of $work_dir/YDBTest to run from: it needn't be under $gtm_root
setenv tst_dir /opt/worldvista/EHR/compile/test_results   # Where to put the output of your test
setenv r /opt/worldvista/EHR/compile/test_results   # Where to create symlink that points to latest test results directory; short name for easy access
#setenv gtm_test_com_individual[5~ $work_dir/YDBTest-dev/com   # Use your own tester-command scripts, not main YDBTest checkout (useful for when you are developing YDBTest/com separately from writing
#setenv force_gtm_test_com_individual   # gtmtest normally re-runs itself from gtmtest.csh in the test source dir. Prevent this.

# Other settings you're less likely to want to change
setenv tst_image dbg   # Select default production/debug build image (dbg/pro) to test against; change later with 'ver' alias
# Ensure gtm_curpro is at least a valid build - set if not already set by /etc/cshrc
if ! ($?gtm_curpro) setenv gtm_curpro $verno   # Subst for current production release used to createdb.csh and by some tests

# set args for gtmtest alias -- select useful ones for a developer
setenv gtmtest_args "-fg -noencrypt -k -nomail -dontzip -stdout 2"
# TEMPLATE: to prevent randomized test settings, use the following for settest_args and/or gtmtest_args
#   (all settings are necessary to avoid test errors when --norandomsettings is included)
setenv settest_args "$gtmtest_args -norandomsettings -nospanreg -noco -jnl nobefore -env eall_noinverse=1"

# The following settings are only used if you build using YDBDevOps (private to Yottadb developers)
#setenv ydb_cc_choose_clang 1; setenv CC clang   # Force clang build: it's faster. But be warned: this causes GT.M builds to fail.
#setenv ydb_cc_choose_asan 0   # asan catches address errors but slows down testing by about 10% (according to Sam)

# Set up the environment for running YottaDB's gtmtest
source $gtm_test_com_individual/test_env.csh

# Other Convenience settings

set cdpath = ( $work_dir )   # for convenient access to your repositories
if ( "$HOST" == "berwyn" && $shlvl == 2) then
  cleantests   # example auto-run on your machine: clean all but most recent day's test results the first time tcsh runs
endif

# Show git branch in prompt
setenv GIT_BRANCH "sh -c 'git branch --no-color 2> /dev/null' | sed -e '/^[^*]/d' -e 's/* \(.*\)/\ [\1]/'"
alias precmd 'set prompt="%{\033[34m%}%n@%B%m%b %B%{\033[31m%}%~%b%{\033[36m%}`$GIT_BRANCH`%{\033[0m%}%# "'
set promptchars=">#"   # use '>' prompt for users; '#' for root

# Optionally uncomment this to avoid loss of history if you close your terminal window with your mouse by:
#   Merging history every minute (at next command) so that history from concurrent terminal window is saved
#   It would be better to save every user command, but avoid precmd or postcmd for reasons given here:
#   cf. https://unix.stackexchange.com/questions/88838/preserve-tcsh-history-in-multiple-terminal-windows
#set tperiod = 1   # Most frequent is 1 minute
#alias periodic 'history -M; history -S'

quit:
# Init the cshdebug.csh debugger if it was left on by the parent shell
if ( $?gtm_test_com_individual ) then
        source $gtm_test_com_individual/cshdebug.csh init "$0" "$*"
endif

I am getting lost in the instructions at this point:

Get a clone of the YDBTest repository:

cd $work_dir git clone https://gitlab.com/YottaDB/DB/YDBTest.git

gtmtest runs from a copy of the test source that must be at $gtm_test. Make this copy as follows:

mkdir -p $gtm_root/gtm_test/T${build_id} tsync

tsync is an alias (defined by .cshrc via test_env.csh), that creates a copy of your test source directory. Your test source directory is taken to be the git repository of your current directory. Strictly, you need to run tsync before running a test, every time you change the source code; but the test-running aliases (e.g. gtmtest) that are defined in test_env.csh will do this for you. Here is a simplified version of tsync:

alias tsync 'rsync -av --delete `git rev-parse --show-toplevel`/ $gtm_test/$gtm_testver/'

The part I don't understand is that in the .cshrc template, $gtm_root is set to /usr/library.  I don't understand if I am supposed to change this or not.  The comment says "Where your (and others') installed binaries go".  Installed by whom?  Linux?  Me?  YottaDB?  I left this at the default value. 

The instructions then says " gtmtest runs from a copy of the test source that must be at $gtm_test. Make this copy as follows:"  But the included command says:
mkdir -p $gtm_root/gtm_test/T${build_id}

This would seem to be installing to $gtm_root, (i.e. /usr/library), not to the $gtm_test.

Can someone help me sort this out?

Thanks

Kevin

[Sam Habiel]

Don't try to install it. It will take a lot of time. Just use the docker container which has things mostly set-up.

--Sam

--
You received this message because you are subscribed to the Google Groups "Everything MUMPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everythingmum...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/f30c3fb6-35aa-4ebc-ae2b-fc2b273ee9f4n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Toppenberg]

Sam,

Thank you for your reply.  Narayanan started looking at the merge request I provided, and it failed multiple tests.  So I will need to be able to step through the tests with the debugging functionality of vscode.  I haven't worked with docker a huge amount, but my understanding is that it is a sandbox.  And if a sandbox, how can it interact with my modified code?

But at your recommendation, I will try going through the steps in the dockerfile instructions section and see if I can make any headway there.

Kevin

[Sam Habiel]

To use your custom version of YottaDB (from the YDBTest README):

To build this image with a custom version of YottaDB, clone
`https://gitlab.com/YottaDB/DB/YDB.git`, and then run the following:

```sh
cd YDB
docker build -f Dockerfile-test -t ydbtest2 .
docker run --init -it -v <local directory>:/testarea1/ --cap-add PERFMON ---rm ydbtest2 -t r200
```

You can use the docker run command to run various tests; or you can enter the test system using "-shell". Look at the README for instructions.

--Sam

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/2321ee51-f9aa-4725-bdcd-d1b6456f280bn%40googlegroups.com.

[Kevin Toppenberg]

OK, it has been some time, but I'm back to this know.  All this is difficult for me to understand, so I am going to write up my progress here and see how it goes.

I'm following the instructions here: https://gitlab.com/YottaDB/DB/YDBTest#using-the-test-system-with-docker

First, it says:

"To build, you need to build from YDBTest root folder, not the docker folder. In this example, we tag our built image to be called "registry.gitlab.com/yottadb/db/ydbtest" (but any name would work).
docker build -f docker/Dockerfile -t registry.gitlab.com/yottadb/db/ydbtest ."

Turns out that "build" is a specific term in the world of Docker that neophytes might not know.  It starts a very length compilation process of building the docker image from the bottom up, from what I can tell.  It finally finished overnight.  It would be nice if the instructions were more clear that this is not needed, if a pull command was previously done.

docker pull registry.gitlab.com/yottadb/db/ydbtest
 

It's probably just that I'm not very familiar with Docker. 

I start the sample test with this command:
docker run --init -it -v /tmp/testarea1/:/testarea1/ --cap-add PERFMON --rm registry.gitlab.com/yottadb/db/ydbtest -t r200

With some digging, I have discovered that this image automatically runs gtmtest, and that the "-t r200" is passed to it.  If I do this:

docker run --rm -it registry.gitlab.com/yottadb/db/ydbtest -shellt -shell   

I get a shell and from there I can type:   gtmtest -t r134   and this will launch a test cycle. 

'gtmtest' is aliased to /usr/library/gtm_test/T999/com/gtmtest.csh -nomail -fg -env gtm_ipv4_only=1 -stdout 2

HOWEVER, the test seems to hang when launched this way, as shown below. 

$> docker run --rm -v /tmp/testarea1/:/testarea1/ -it registry.gitlab.com/yottadb/db/ydbtest -shell
Try running gtmtest -t r134
Type 'ver' to switch versions
1f103bb5aca2:~> gtmtest -t r134
Created /testarea1/tst_V999_R999_dbg_01_250408_230401_HXH
Test Version            :: V999_R999
Test Image              :: dbg
Test Mail List          :: gtmtest
Test Output Directory   :: /testarea1/tst_V999_R999_dbg_01_250408_230401_HXH
Test Source Directory   :: T999

r134 test starts...
PASS from ydb757
... <----  hangs here.    I finally killed it after about 30 minutes of apparent inactivity. 

Moving on....

The instructions say:

To run against a copy of YDBTest on your file system (e.g. you use it with VSCode, and want to autoload the changes), you can do the following (the volume -v argument's left hand side is the full path of the YDBTest git repository, the right hand side of the colon is a fixed path that is known by the docker scripts which you must not change):
docker run --init -it -v <local directory>:/testarea1/ -v <full path to YDBTest>:/YDBTest --cap-add PERFMON ---rm registry.gitlab.com/yottadb/db/ydbtest -t r200

I interpret this to mean that if I want to make a custom version of the YDBTest system, and have the docker image use my custom version instead of the one that the docker image contains, then I would do this.    HOWEVER, in my case, I haven't made any changes to the YDBTest system.  So I don't think I want to do this.

Jumping forward, I see next:

To build this image with a custom version of YottaDB, clone https://gitlab.com/YottaDB/DB/YDB.git, and then run the following:

cd YDB

docker build -f Dockerfile-test -t ydbtest2 .

docker run --init -it -v <local directory>:/testarea1/ --cap-add PERFMON ---rm ydbtest2 -t r200

This looks like what I need.  The normal YDB folder on my system is in /opt/worldvista/EHR/compile/YDB  and the folder with my modified code is in /opt/worldvista/EHR/compile/TMG-YDB.     So to create a custom docker image I did this:


cd /opt/worldvista/EHR/compile/TMG-YDB

docker build -f Dockerfile-test -t tmgtest .

This shows that the tmgtest image is created.

>$ docker images
REPOSITORY                               TAG        IMAGE ID       CREATED         SIZE
tmgtest                                  latest     cf060ef0ba22   2 minutes ago   5.3GB
registry.gitlab.com/yottadb/db/ydbtest   latest     00c1b1754514   25 hours ago    4.44GB
debian                                   bookworm   d84448199470   3 weeks ago     117MB

This is all the time I have for today. 

QUESTION:

-- using my tmgtest docker image, I will be able to ask the image to run a particular test.  This will launch gtmtest against my code, and it will likely confirm the various tests which my modified code fails.  But how to debug?   My perception is that gtmtest is a script that interacts with ydb executable in a command-line fashion.  But I need to be in debug mode inside that ydb executable to see what is going on. Normally, when I do step-by-step debugging, I use VSCode which launches the executable via a debugger.

So I see the situation like this:

[YDBTest] --> script --> control ydb via command-line interface (CLI)

And when I debug with VSCode, I have this.

[VSCode] --> launch debugger, telling it to run ydb executable (via launch.json)

To marry these two approaches, I need to modify my launch.json such that the test is done via the debugger. 

But how to find the various tests??

Has anyone already solved this problem?

Thanks

Kevin

[Sam Habiel]

FYI, I just tried gtmtest -t r134 on my system inside of the docker shell, and the output is as follows:

```

bf55c9350e4c:~> gtmtest -t r134
Created /testarea1/tst_V999_R999_dbg_01_250409_120645_z05

Test Version            :: V999_R999
Test Image              :: dbg
Test Mail List          :: gtmtest

Test Output Directory   :: /testarea1/tst_V999_R999_dbg_01_250409_120645_z05

Test Source Directory   :: T999
 
r134 test starts...
PASS from ydb757

PASS from ydb775
PASS from ydb782
PASS from ydb772
PASS from ydb785
PASS from ydb734
PASS from ydb828
PASS from ydb831
PASS from ydb555
PASS from ydb546
PASS from ydb758
PASS from ydb557
PASS from ydb840
PASS from ydb781
PASS from ydb846
PASS from ydb845
r134 test DONE.
00:00:01:28
Wed Apr 9 12:08:15 PM UTC 2025 PASSED r134

```

Answering your questions:

A. using my tmgtest docker image, I will be able to ask the image to run a particular test.  This will launch gtmtest against my code, and it will likely confirm the various tests which my modified code fails.  But how to debug?

The process of using the test system is as follows:

1. Run the failing test.

2. Navigate to directory /testarea1/<random name in the test>/<test_suite>_0/<test_name>

3. Diff the .cmp (expected output) and .log (actual output) files. See where they exactly differ.

4. Read the source code of the test in <test_suite>/u_inref/<test_name>.csh and see what command exactly caused that problem.

5. Construct a simple test case based on #4

6. Use a debugging facilities with the new version vs the old version with the test case in #5

You may find out that the error is so obvious that you can skip some steps. Other times the error is way way deep in Linux system libraries that it's hard to figure out why the behavior changed.

--Sam

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/19d7a1a9-f852-42b8-9178-e80aa47d7817n%40googlegroups.com.

[Kevin Toppenberg]

On Wednesday, April 9, 2025 at 8:19:51 AM UTC-4 Sam Habiel wrote:

FYI, I just tried gtmtest -t r134 on my system inside of the docker shell, and the output is as follows:

I see that it works for you.  I don't know why it doesn't for me.  And it probably doesn't matter right now.

 

Answering your questions:

A. using my tmgtest docker image, I will be able to ask the image to run a particular test.  This will launch gtmtest against my code, and it will likely confirm the various tests which my modified code fails.  But how to debug?

The process of using the test system is as follows:

1. Run the failing test.

2. Navigate to directory /testarea1/<random name in the test>/<test_suite>_0/<test_name>

3. Diff the .cmp (expected output) and .log (actual output) files. See where they exactly differ.

4. Read the source code of the test in <test_suite>/u_inref/<test_name>.csh and see what command exactly caused that problem.

5. Construct a simple test case based on #4

6. Use a debugging facilities with the new version vs the old version with the test case in #5

You may find out that the error is so obvious that you can skip some steps. Other times the error is way way deep in Linux system libraries that it's hard to figure out why the behavior changed.

--Sam

Thank you for this reply. 

Below is the list of errors Narayanan reported when he tested my code. 

gtmtest.csh -t v54003 -st C9K08003315
gtmtest.csh -t expect -st C9K05003274
gtmtest.csh -t expect -st D9F05002548
gtmtest.csh -t expect -st gtm4661
gtmtest.csh -t expect -st gtm7658
gtmtest.csh -t expect -st gtm7994
gtmtest.csh -t expect -st gtm8223
gtmtest.csh -t expect -st gtm8232
gtmtest.csh -t expect -st terminal
gtmtest.csh -t expect -st terminal5
gtmtest.csh -t expect -st zshowempterm
gtmtest.csh -t io -st gtm8239
gtmtest.csh -t io -st zshow_principal
gtmtest.csh -t r120 -st ctrlchandler
gtmtest.csh -t r120 -st readcmdrecallhist
gtmtest.csh -t r120 -st readtimeout
gtmtest.csh -t r124 -st ydb343
gtmtest.csh -t r124 -st ydb346mr
gtmtest.csh -t r124 -st ydb350
gtmtest.csh -t r126 -st find_timer_assert
gtmtest.csh -t r128 -st v63006C
gtmtest.csh -t r130 -st ydb568
gtmtest.csh -t r132 -st ydb612
gtmtest.csh -t r132 -st ydb712
gtmtest.csh -t r136 -st ydb940
gtmtest.csh -t r200 -st ydb1029
gtmtest.csh -t r200 -st ydb401
gtmtest.csh -t r200 -st ydb88_14
gtmtest.csh -t r200 -st ydb88_15
gtmtest.csh -t r200 -st ydb88_16
gtmtest.csh -t r202 -st rlsiglongjmp-ydb1065
gtmtest.csh -t v52000 -st D9H02002642
gtmtest.csh -t v55000 -st GTM6940
gtmtest.csh -t v63003 -st gtm8889
gtmtest.csh -t v63004 -st gtm8777
gtmtest.csh -t v63005 -st gtm8962
gtmtest.csh -t v63010 -st gtm9178
gtmtest.csh -t v70002 -st ctrap_acsii-gtmde201390

I'm going to start with the first one, following Sam's instructions.

So I ran this code.  tmgtest is the docker image I built to work against my modified code TMG-YDB

 docker run --init -it -v /tmp/testarea1/:/testarea1/ --cap-add PERFMON --rm tmgtest  -t v54003 -st C9K08003315

This gives this output:

Created /testarea1/tst_V999_R999_dbg_01_250409_162912_d7g

Test Version            :: V999_R999
Test Image              :: dbg
Test Mail List          :: gtmtest

Test Output Directory   :: /testarea1/tst_V999_R999_dbg_01_250409_162912_d7g
Test Source Directory   :: T999

v54003 test starts...
PASS from C9K08003315
v54003 test DONE.
00:00:00:17
Wed Apr 9 04:29:39 PM UTC 2025 PASSED v54003

It looks like is PASSED, which is not what I expected.  Let's look at the test details. 

/tmp/testarea1/tst_V999_R999_dbg_01_250409_162912_d7g$ ls -Al
total 40
-rw-rw-r-- 1 kdt0p laughlin_results   25 Apr  9 12:29 cleanup.csh
-rw-rw-r-- 1 kdt0p laughlin_results  414 Apr  9 12:29 config.log
drwxrwxr-x 2 kdt0p laughlin_results 4096 Apr  9 12:29 debugfiles
-rw-rw-r-- 1 kdt0p laughlin_results  157 Apr  9 12:29 ipcs.b4subtest
-rw-rw-r-- 1 kdt0p laughlin_results   25 Apr  9 12:29 PID
-rw-rw-r-- 1 kdt0p laughlin_results    0 Apr  9 12:29 relinkctl.b4subtest
-rw-rw-r-- 1 kdt0p laughlin_results  451 Apr  9 12:29 report.txt
-rw-rw-r-- 1 kdt0p laughlin_results   55 Apr  9 12:29 submitted_tests
-rw-rw-r-- 1 kdt0p laughlin_results  205 Apr  9 12:29 tests.out
drwxrwxr-x 3 kdt0p laughlin_results 4096 Apr  9 12:29 utilobj
drwxrwxr-x 3 kdt0p laughlin_results 4096 Apr  9 12:29 v54003_0

[0.019s] kdt0p@ubuntu64:/tmp/testarea1/tst_V999_R999_dbg_01_250409_162912_d7g$ cd v54003_0/
[0.004s] kdt0p@ubuntu64:/tmp/testarea1/tst_V999_R999_dbg_01_250409_162912_d7g/v54003_0$ ls -Al
total 60
-rw-rw-r-- 1 kdt0p laughlin_results    0 Apr  9 12:29 check_space.out
-rw-rw-r-- 1 kdt0p laughlin_results    0 Apr  9 12:29 chmod_C9K08003315.out
-rwxrwxr-x 1 kdt0p laughlin_results   25 Apr  9 12:29 cleanup.csh
-rw-rw-r-- 1 kdt0p laughlin_results  758 Apr  9 12:29 config.log
-rw-rw-r-- 1 kdt0p laughlin_results   17 Apr  9 12:29 du.out
-rw-rw-r-- 1 kdt0p laughlin_results   22 Apr  9 12:29 outstream.cmp.original
-rw-rw-r-- 1 kdt0p laughlin_results  108 Apr  9 12:29 outstream.log
-rw-rw-r-- 1 kdt0p laughlin_results   62 Apr  9 12:29 outstream.log_actual
-rw-rw-r-- 1 kdt0p laughlin_results   62 Apr  9 12:29 outstream.log_m
-rw-rw-r-- 1 kdt0p laughlin_results 6490 Apr  9 12:29 settings.csh
-rw-rw-r-- 1 kdt0p laughlin_results 6490 Apr  9 12:29 settings.csh.v54003
-rw-rw-r-- 1 kdt0p laughlin_results   44 Apr  9 12:29 subtest_exclude_list
drwxrwxr-x 2 kdt0p laughlin_results 4096 Apr  9 12:29 testfiles
-rw-rw-r-- 1 kdt0p laughlin_results   97 Apr  9 12:29 timing.subtest
-rw-rw-r-- 1 kdt0p laughlin_results   74 Apr  9 12:29 v54003.log

Below is the output of the .cmp file and the .log file

[0.008s] kdt0p@ubuntu64:/tmp/testarea1/tst_V999_R999_dbg_01_250409_162912_d7g/v54003_0$ cat v54003.log
v54003 test starts...
PASS from C9K08003315
v54003 test DONE.
00:00:00:17

[0.007s] kdt0p@ubuntu64:/tmp/testarea1/tst_V999_R999_dbg_01_250409_162912_d7g/v54003_0$ cat outstream.cmp.original
PASS from C9K08003315

They are not exactly the same, but it does say that they passed. 

"4. Read the source code of the test in <test_suite>/u_inref/<test_name>.csh and see what command exactly caused that problem."

I can't see this in the directory.  Am I overlooking it?

 I try running the next test:

[0.004s] kdt0p@ubuntu64:/opt/worldvista/EHR/compile/TMG-YDB$ docker run --init -it -v /tmp/testarea1/:/testarea1/ --cap-add PERFMON --rm tmgtest  -t expect -st C9K05003274
Created /testarea1/tst_V999_R999_dbg_01_250409_164215_W1O

Test Version            :: V999_R999
Test Image              :: dbg
Test Mail List          :: gtmtest

Test Output Directory   :: /testarea1/tst_V999_R999_dbg_01_250409_164215_W1O
Test Source Directory   :: T999

expect test starts...   <----- hung here

So far it has been about 15 minutes.  I'm going to let it run an hour before aborting it. 

I think I want to get the docker technology out of the mix.  I'm going to try getting YDBTEST working in my machine.....

Kevin

[Sam Habiel]

>  docker run --init -it -v /tmp/testarea1/:/testarea1/ --cap-add PERFMON --rm tmgtest  -t v54003 -st C9K08003315

By default, if a test passes, the test output is deleted. You need to pass "-keep" to keep the test output. If it fails, then you will see a directory called C9K08003315 that contains the .cmp and .log files.

> I can't see this in the directory.  Am I overlooking it?

cd YDBTest

less v54003/u_inref/C9K08003315.csh

> expect test starts...   <----- hung here

When an expect test hangs, then you already know it's broken. There is no need for you to wait. I debug expect test failures by cding into the directory that the test output is in and then watch the output.

--Sam

--
You received this message because you are subscribed to the Google Groups "Everything MUMPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to everythingmum...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/10c7fb47-e9a4-42c6-8302-73593be0e183n%40googlegroups.com.

[Kevin Toppenberg]

Sam,

Thanks for the reply.

Right now, I am going to try working on getting the YDBTest working outside of docker.  I know that for those comfortable with Docker, that this seems crazy.  But to me it just seems like one more layer of technology that I don't fully understand, and which may or may not be part of the problem.   For example, is the hang because of an error with "expect"?  Maybe?  What about the other hang I had that didn't involve expect?  How would I go about figuring out what is causing the hang if it is from a docker issue? 

So now I am trying to get my .cshrc file setup, based on the template provided in the instructions here: https://gitlab.com/YottaDB/DB/YDBTest

There may be a small bug on the line that shows:

source $gtm_test_com_individual/test_env.csh

I think maybe this should be:

source $gtm_test/com/test_env.csh

Questions:

What do I put for the following line?

setenv build_id "1648"   # Your build number: any 9xx no.; assigned by YottaDB on their servers: avoids clobbering others' builds

When Narayanan communicated with me, he reference my code changes with a number of "!1648".  In the email, this is a URL, and clicking it takes me to https://gitlab.com/YottaDB/DB/YDB/-/merge_requests/1648  This page shows the merge request I submitted.  So I guess my number is "1649"?? I'm not sure what the "!" is in there for.  Also "1648" doesn't match the specified the description on the line above.  So is this correct or not?

When I use this in my .cshrc file, and launch via tcsh, I get this error

VERSION-E-VERNOTEXIST : Directory /usr/lib/V1648_R202/dbg does not exist. Exiting...

Is this a directory that I am supposed to supply?  I don't really want my testing to put stuff into /usr/lib.  Did this come from $gtm_root?

There is a line in .cshrc that I changed from /usr/library, to /usr/lib

setenv gtm_root /usr/lib   # Where your (and others') installed binaries go

Any advice about how to get this going?  Again, I don't just want to use the docker instance if possible.

Kevin

[Sam Habiel]

> Any advice about how to get this going?

I am sorry. It's quite complex to get it going... It usually takes around 2 days to set-up with assistance. You are on your own if you want to do that.

--Sam

To view this discussion visit https://groups.google.com/d/msgid/everythingmumps/31c92bf1-80d5-4185-92ef-9eef693c9d2en%40googlegroups.com.

[Kevin Toppenberg]

On Wednesday, April 9, 2025 at 3:12:52 PM UTC-4 Sam Habiel wrote:

> Any advice about how to get this going?

I am sorry. It's quite complex to get it going... It usually takes around 2 days to set-up with assistance. You are on your own if you want to do that.

Well, I'll have to wrestle with it on my own with docker, or I'll have wrestle with it on my own without docker.  So as the song goes... "Alone again ... naturally."

I'm going to investigate this issue of buid_id and how it interacts with the environment.   FYI, I am working in a VM that is a copy of my live production system.

I have decided to compare the .cshrc file that works (partly) in the docker image, and compare that with what I have.

docker's .cshrc shows the following (selected lines)

  setenv gtm_root /usr/library

 setenv verno V999_R999

 setenv gtm_dist $gtm_root/$verno/dbg # change later  ==> /usr/library/V999_R999/dbg

 setenv gtm_tools $gtm_root/$verno/tools   ==> /usr/library/V999_R999/tools

 setenv gtm_inc $gtm_root/$verno/inc       ==> /usr/library/V999_R999/inc

 setenv gtm_verno $verno                ==> V999_R999

 setenv gtm_ver $gtm_root/$verno      ==> /usr/library/V999_R999/

 setenv gtm_test_v6_dbcreate_rand_ver $verno  ==> V999_R999

in my .cshrc (based on provided template), I have:

# Your build number: any 9xx no.; ...

setenv build_id "1648" 

 

# Change Rxxx to the rev of yottadb you will install and test against <--- This is the only line using $build_id

setenv verno V${build_id}_R202  

# Subst for current prod release used to createdb.csh and by some tests <--- this is the only use of $verno

if ! ($?gtm_curpro) setenv gtm_curpro $verno  

What about those other vars that are basedon verno? 

In my system, I included the setup for my production environment.

setenv vista_home "/opt/worldvista/EHR"
setenv VH ${vista_home}                 # a short name for paths, etc - temp
setenv gtm_dist "${VH}/m"   <--- end result is /opt/worldvista/EHR/m

gtm_tools <-- I don't have this

My goal is to set the env vars such that YDBTest can operate against my usual ydb in a manner similar to that done in the docker image.  So what is found at the various locations in the docker image?

Here is the contents from the docker image:
4237965baa8a:/usr/library/V999_R999> ls -Al
dr-xr-xr-x 1 root root  4096 Apr  8 23:37 dbg
drwxr-xr-x 1 root root 36864 Apr  8 23:38 inc
drwxr-xr-x 1 root root 73728 Apr  7 23:08 src
drwxr-xr-x 1 root root  4096 Apr  8 23:37 tools

and

4237965baa8a:/usr/library/V999_R999/tools> ls -al
drwxr-xr-x 1 root root  4096 Apr  8 23:37 .
drwxr-xr-x 1 root root  4096 Apr  7 23:08 ..
-rwxr-xr-x 1 root root  6827 Apr  8 23:37 buildaux.csh
-rw-r--r-- 1 root root  1600 Apr  8 23:37 buildaux_dse.csh
-rw-r--r-- 1 root root  3744 Apr  8 23:37 buildaux_gde.csh
-rw-r--r-- 1 root root  1907 Apr  8 23:37 buildaux_gtcm_gnp_server.csh
-rw-r--r-- 1 root root  1743 Apr  8 23:37 buildaux_gtcm_pkdisp.csh
-rw-r--r-- 1 root root  1803 Apr  8 23:37 buildaux_gtcm_play.csh
-rw-r--r-- 1 root root  1835 Apr  8 23:37 buildaux_gtcm_server.csh
-rw-r--r-- 1 root root  1773 Apr  8 23:37 buildaux_gtcm_shmclean.csh
-rwxr-xr-x 1 root root  5298 Apr  8 23:37 buildaux_gtmcrypt.csh
-snip-
-rwxr-xr-x 1 root root  3787 Apr  8 23:37 gtmsrc.csh
-rwxr-xr-x 1 root root  1002 Apr  8 23:37 gtm_version_dirname.csh
-rwxr-xr-x 1 root root  2443 Apr  8 23:37 is_icu_symbol_rename.csh
-rwxr-xr-x 1 root root  5893 Apr  8 23:37 lintgtm.csh
-rwxr-xr-x 1 root root  3216 Apr  8 23:37 lintimage.csh
-rwxr-xr-x 1 root root  2131 Apr  8 23:37 mkutf8dir.csh
-rw-r--r-- 1 root root  2613 Apr  8 23:37 offset.awk
-rwxr-xr-x 1 root root  4841 Apr  8 23:37 offset.csh
-rwxr-xr-x 1 root root  2185 Apr  8 23:37 runall_cc_many.csh
-rwxr-xr-x 1 root root  3582 Apr  8 23:37 runall_cc_one.csh
-rwxr-xr-x 1 root root 17936 Apr  8 23:37 runall.csh
-rw-r--r-- 1 root root  1999 Apr  8 23:37 set_library_path.csh
-rw-r--r-- 1 root root  4273 Apr  8 23:37 stripmine.awk
-rwxr-xr-x 1 root root   797 Apr  8 23:37 versions.csh
-rw-r--r-- 1 root root  3579 Apr  8 23:37 xtrgtmtypes.awk

and

4237965baa8a:/usr/library/V999_R999/dbg> ls -al
dr-xr-xr-x 1 root root     4096 Apr  8 23:37 .
drwxr-xr-x 1 root root     4096 Apr  7 23:08 ..
-r--r--r-- 1 root root     3094 Apr  8 23:37 _CONVBASEUTIL.m
-r--r--r-- 1 root root     9624 Apr  8 23:37 _CONVBASEUTIL.o
-r--r--r-- 1 root root      500 Feb 17 19:23 custom_errors_sample.txt
-r--r--r-- 1 root root     2858 Apr  8 23:37 _DATE.m
-r--r--r-- 1 root root    15968 Apr  8 23:37 _DATE.o
-r--r--r-- 1 root root    11271 Apr  8 23:37 DECOMMENT.m
-r--r--r-- 1 root root    21424 Apr  8 23:37 DECOMMENT.o
-r--r--r-- 1 root root     2596 Apr  8 23:37 _DH.m
-r--r--r-- 1 root root     6248 Apr  8 23:37 _DH.o
-r--r--r-- 1 root root      849 Apr  8 23:37 _D.m
-r--r--r-- 1 root root     1880 Apr  8 23:37 _D.o
-r--r--r-- 1 root root     1572 Apr  8 23:37 _DO.m
-r--r--r-- 1 root root     3976 Apr  8 23:37 _DO.o
-r-xr-xr-x 1 root root    21464 Apr  8 23:37 dse
-snip-
-r-xr-xr-x 1 root root     1391 Apr  8 23:37 ydb
-r--r--r-- 1 root root     9527 Feb 17 19:23 ydbcrypt_interface.h
-r--r--r-- 1 root root    47671 Apr  8 23:37 _YDBENV.m
-r--r--r-- 1 root root    92944 Apr  8 23:37 _YDBENV.o
-r--r--r-- 1 root root     7318 Apr  8 23:37 ydb_env_set
-r--r--r-- 1 root root     1614 Apr  8 23:37 ydb_env_unset
-rwxr-xr-x 1 root root    74552 Apr  8 23:37 ydbinstall
-r--r--r-- 1 root root    26420 Apr  8 23:37 _YDBJNLF.m
-r--r--r-- 1 root root    48496 Apr  8 23:37 _YDBJNLF.o
-r--r--r-- 1 root root     7159 Apr  8 23:37 _YDBPROCSTUCKEXEC.m
-r--r--r-- 1 root root    14184 Apr  8 23:37 _YDBPROCSTUCKEXEC.o
lrwxrwxrwx 1 root root        9 Apr  8 23:37 ydbsh -> ./yottadb
-r--r--r-- 1 root root    20657 Feb 17 19:23 ydb_tls_interface.h
-r--r--r-- 1 root root    11360 Apr  8 23:37 _YGBLSTAT.m
-r--r--r-- 1 root root    23584 Apr  8 23:37 _YGBLSTAT.o
-r-xr-xr-x 1 root root    21464 Apr  8 23:37 yottadb
-rw-r--r-- 1 root root      240 Apr  8 23:37 yottadb.pc
-r--r--r-- 1 root root     2790 Apr  8 23:37 _ZMVALID.m
-r--r--r-- 1 root root     3696 Apr  8 23:37 _ZMVALID.o
-r--r--r-- 1 root root     5690 Apr  8 23:37 _ZSHOWVTOLCL.m
-r--r--r-- 1 root root    14368 Apr  8 23:37 _ZSHOWVTOLCL.o\

and

4237965baa8a:/usr/library/V999_R999/inc> ls -al
drwxr-xr-x 1 root root  36864 Apr  8 23:38 .
drwxr-xr-x 1 root root   4096 Apr  7 23:08 ..
-rw-rw-r-- 1 root root    622 Feb 17 19:23 add_inter.h
-rw-rw-r-- 1 root root    647 Feb 17 19:23 advancewindow.h
-rw-rw-r-- 1 root root   4813 Feb 17 19:23 aio_shim.h
-rw-rw-r-- 1 root root  15917 Feb 17 19:23 alias.h
-rw-rw-r-- 1 root root    556 Feb 17 19:23 alloc_reg.h
-rw-rw-r-- 1 root root   4625 Feb 17 19:23 alternate_sighandling.h
-rw-rw-r-- 1 root root  23885 Feb 17 19:23 anticipatory_freeze.h
-rw-rw-r-- 1 root root   3554 Feb 17 19:23 arit.h
-rw-rw-r-- 1 root root    931 Feb 17 19:23 arlinkdbg.h
-rw-rw-r-- 1 root root    485 Feb 17 19:23 ast.h
-rw-rw-r-- 1 root root    857 Feb 17 19:23 aswp.h
-rw-rw-r-- 1 root root    935 Feb 17 19:23 auto_zlink.h
-rw-rw-r-- 1 root root    863 Feb 17 19:23 bg_update.h
-rw-rw-r-- 1 root root    688 Feb 17 19:23 bit_clear.h
-rw-rw-r-- 1 root root    494 Feb 17 19:23 bit_op.h
-rw-rw-r-- 1 root root    680 Feb 17 19:23 bit_set.h
-rw-rw-r-- 1 root root    704 Feb 17 19:23 bm_getfree.h
-snip-

So if I am going to have a gtm_tools var, it should point to a directory with the files below (and others), which I have picked as sample files in $gtm_tools in the docker image.

-rwxr-xr-x 1 root root  6827 Apr  8 23:37 buildaux.csh

-rwxr-xr-x 1 root root  2185 Apr  8 23:37 runall_cc_many.csh

-rw-r--r-- 1 root root  3579 Apr  8 23:37 xtrgtmtypes.awk

A search on my system for the first file finds these:
> find . -name buildaux.csh
./TMG-YDB/sr_unix/buildaux.csh
./YDB/sr_unix/buildaux.csh
./r203_gold_YDB/sr_unix/buildaux.csh

> find . -name runall_cc_many.csh
./TMG-YDB/sr_unix/runall_cc_many.csh
./YDB/sr_unix/runall_cc_many.csh
./r203_gold_YDB/sr_unix/runall_cc_many.csh

> find . -name xtrgtmtypes.awk
./TMG-YDB/sr_unix/xtrgtmtypes.awk
./YDB/sr_unix/xtrgtmtypes.awk
./r203_gold_YDB/sr_unix/xtrgtmtypes.awk

So I think I will point gtm_tools to /opt/worldvista/EHR/compile/TMG-YDB/sr_unix/

setenv gtm_root /usr/library
 setenv verno V999_R999
 setenv gtm_dist $gtm_root/$verno/dbg # change later  ==> /usr/library/V999_R999/dbg
 setenv gtm_tools $gtm_root/$verno/tools   ==> /usr/library/V999_R999/tools
 setenv gtm_inc $gtm_root/$verno/inc       ==> /usr/library/V999_R999/inc
 setenv gtm_verno $verno                ==> V999_R999
 setenv gtm_ver $gtm_root/$verno      ==> /usr/library/V999_R999/
 setenv gtm_test_v6_dbcreate_rand_ver $verno  ==> V999_R999



... out of time....

Kevin

[David Wicksell]

Kevin,

 I think that Sam was trying to tell you that getting the YottaDB test suite up and running with Docker is

_much_ easier than doing so on bare metal. Which is true, it is much easier with Docker.

- David Wicksell