Update to DIRAC-7.3

[zhangxm]

Hello,

According to the page:
https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-7.3-(v7r3)#initial-update,
I tried to update python2 DIRACv7r3p1 to python3 one.

I failed during the initial update step:

[samtest.ihep.ac.cn]> update DIRAC==v7.3
Software update can take a while, please wait ...
[ERROR] Failed to update the software
Error in _read: timed out SSLTimeoutError('timed out',)


Any hints to find out where did it stop or other ways to work around?


Best regards,

Xiaomei

[Federico Stagni]

Hi Xiaomei,

sorry for the late reply (I was off for a few days, if you want a faster reply I suggest you write in https://github.com/DIRACGrid/DIRAC/discussions).

That might simply be a timeout issue, which we have corrected in later patches of DIRAC 7.3. Try with 7.3.8, which is the latest: https://github.com/DIRACGrid/DIRAC/releases (and https://pypi.org/project/DIRAC/)

Cheers,

Federico

--
You received this message because you are subscribed to the Google Groups "diracgrid-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to diracgrid-for...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/6f79351d-210a-fdfe-281b-ca8ed2e74e82%40ihep.ac.cn.

[zhangxm]

Hi Federico,

I got the same error with DIRAC==v7.3.8:

[samtest.ihep.ac.cn]> update DIRAC==v7.3.8

Software update can take a while, please wait ...
[ERROR] Failed to update the software
Error in _read: timed out SSLTimeoutError('timed out',)

Best regards,

Xiaomei

[Federico Stagni]

Mmm I see. I think I know what it is, I will provide a PR after the BiLD meeting.

Cheers,

Federico

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/00a0f862-aba0-b2e3-bd4a-0ff92831d70b%40ihep.ac.cn.

[Federico Stagni]

Hi Xiaomei,

can you please upgrade first to py2 v7r3p8, and only then to py3 7.3.8? Also, use the most recent 7.3.8 client.

Cheers,

Federico

[zhangxm]

Hi Federico,

Look like timeout=600 is still not enough, I changed all the related timeout to 900, then go down with another error:

[samtest.ihep.ac.cn]>  update DIRAC==v7.3.8
Software update can take a while, please wait ...
[ERROR] Failed to update the software

Failed to install DIRACOS2 with message ERROR: Invalid requirement: '{IHEPDIRAC}[server]'

It looks like can't update from IHEPDIRAC, only valilla DIRAC?

Best regards,

Xiaomei

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/CAPaK5Jn1oU%3D2eUCi-TzA%2Bmsp8DPMowAE_Tj2uXfa7BZ_ZzceoA%40mail.gmail.com.

[Federico Stagni]

Hi Xiamei,

seems https://github.com/DIRACGrid/DIRAC/pull/5600 would help

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/584fa2b9-22b3-0722-70ec-8c4b23a955e2%40ihep.ac.cn.

[Chris Burr]

Hi Xiamei,

Yeah, #5600 is definitely needed. Sorry about that! 😅

What is the state of the IHEPDIRAC extension? Niavely I would expect you
to be updating to IHEPDIRAC==vX.Y.Z however I don't see a package for
IHEPDIRAC: https://pypi.org/project/IHEPDIRAC/

If IHEPDIRAC is a legacy thing that you don't need anymore you'll have
to remove it from the CS first.

Cheers,

Chris

On 2021-11-17 13:07, Federico Stagni wrote:
> Hi Xiamei,
> seems https://github.com/DIRACGrid/DIRAC/pull/5600 would help
>
> Il giorno mar 16 nov 2021 alle ore 15:46 zhangxm <zha...@ihep.ac.cn>
> ha scritto:
>
>> Hi Federico,
>>
>> Look like timeout=600 is still not enough, I changed all the related
>> timeout to 900, then go down with another error:

>> [samtest.ihep.ac.cn [1]]> update DIRAC==v7.3.8

>> Software update can take a while, please wait ...
>> [ERROR] Failed to update the software
>> Failed to install DIRACOS2 with message ERROR: Invalid requirement:
>> '{IHEPDIRAC}[server]'
>>
>> It looks like can't update from IHEPDIRAC, only valilla DIRAC?
>>
>> Best regards,
>> Xiaomei
>>
>> 在 2021-11-12 0:33, Federico Stagni 写道:
>>
>> Hi Xiaomei,
>> can you please upgrade first to py2 v7r3p8, and only then to py3
>> 7.3.8? Also, use the most recent 7.3.8 client.
>>
>> Cheers,
>> Federico
>>
>> Il giorno gio 11 nov 2021 alle ore 09:54 Federico Stagni
>> <sta...@gmail.com> ha scritto:
>>
>> Mmm I see. I think I know what it is, I will provide a PR after the
>> BiLD meeting.
>>
>> Cheers,
>> Federico
>>
>> Il giorno gio 11 nov 2021 alle ore 03:47 zhangxm
>> <zha...@ihep.ac.cn> ha scritto:
>>
>> Hi Federico,
>>
>> I got the same error with DIRAC==v7.3.8:
>>

>> [samtest.ihep.ac.cn [1]]> update DIRAC==v7.3.8

>> Software update can take a while, please wait ...
>> [ERROR] Failed to update the software
>> Error in _read: timed out SSLTimeoutError('timed out',)
>>
>> Best regards,
>> Xiaomei
>>
>> 在 2021-11-10 18:36, Federico Stagni 写道:
>>
>> Hi Xiaomei,
>> sorry for the late reply (I was off for a few days, if you want a
>> faster reply I suggest you write in
>> https://github.com/DIRACGrid/DIRAC/discussions).
>>
>> That might simply be a timeout issue, which we have corrected in
>> later patches of DIRAC 7.3. Try with 7.3.8, which is the latest:
>> https://github.com/DIRACGrid/DIRAC/releases (and
>> https://pypi.org/project/DIRAC/)
>>
>> Cheers,
>> Federico
>>
>> Il giorno ven 5 nov 2021 alle ore 04:01 zhangxm <zha...@ihep.ac.cn>
>> ha scritto:
>> Hello,
>>
>> According to the page:
>>
> https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-7.3-(v7r3)#initial-update,
>>
>> I tried to update python2 DIRACv7r3p1 to python3 one.
>>
>> I failed during the initial update step:
>>

>> [samtest.ihep.ac.cn [1]]> update DIRAC==v7.3

>> Software update can take a while, please wait ...
>> [ERROR] Failed to update the software
>> Error in _read: timed out SSLTimeoutError('timed out',)
>>
>> Any hints to find out where did it stop or other ways to work
>> around?
>>
>> Best regards,
>>
>> Xiaomei
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "diracgrid-forum" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to diracgrid-for...@googlegroups.com.
>> To view this discussion on the web visit
>>
> https://groups.google.com/d/msgid/diracgrid-forum/6f79351d-210a-fdfe-281b-ca8ed2e74e82%40ihep.ac.cn.
>
> --
> You received this message because you are subscribed to the Google
> Groups "diracgrid-forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to diracgrid-for...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/diracgrid-forum/00a0f862-aba0-b2e3-bd4a-0ff92831d70b%40ihep.ac.cn

> [2].

> --
> You received this message because you are subscribed to the Google
> Groups "diracgrid-forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to diracgrid-for...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/diracgrid-forum/CAPaK5Jn1oU%3D2eUCi-TzA%2Bmsp8DPMowAE_Tj2uXfa7BZ_ZzceoA%40mail.gmail.com
> [3].

>
> --
> You received this message because you are subscribed to the Google
> Groups "diracgrid-forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to diracgrid-for...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/diracgrid-forum/584fa2b9-22b3-0722-70ec-8c4b23a955e2%40ihep.ac.cn
> [4].

>
> --
> You received this message because you are subscribed to the Google
> Groups "diracgrid-forum" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to diracgrid-for...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/diracgrid-forum/CAPaK5JmDwYSn2Jiw%3DB%3DOMMFYQCsB9rAT23ULzdC4gmNQ3c-EDw%40mail.gmail.com
> [5].
>
>
> Links:
> ------
> [1] http://samtest.ihep.ac.cn
> [2]
> https://groups.google.com/d/msgid/diracgrid-forum/00a0f862-aba0-b2e3-bd4a-0ff92831d70b%40ihep.ac.cn?utm_medium=email&amp;utm_source=footer
> [3]
> https://groups.google.com/d/msgid/diracgrid-forum/CAPaK5Jn1oU%3D2eUCi-TzA%2Bmsp8DPMowAE_Tj2uXfa7BZ_ZzceoA%40mail.gmail.com?utm_medium=email&amp;utm_source=footer
> [4]
> https://groups.google.com/d/msgid/diracgrid-forum/584fa2b9-22b3-0722-70ec-8c4b23a955e2%40ihep.ac.cn?utm_medium=email&amp;utm_source=footer
> [5]
> https://groups.google.com/d/msgid/diracgrid-forum/CAPaK5JmDwYSn2Jiw%3DB%3DOMMFYQCsB9rAT23ULzdC4gmNQ3c-EDw%40mail.gmail.com?utm_medium=email&utm_source=footer

[zhangxm]

Hi,

After fixing #5600 and uploading IHEPDIRAC package to pypi, updating is
working now. But after restarting, the services and agents in
/opt/dirac/runit lost connection. Here is the example of Configuration
Server log:
"DIRAC installation path overridden to /opt/dirac/pro
./run: line 11:exec: dirac-service: can't find"

How to fix that?

Best regards,
Xiaomei

[Chris Burr]

I suspect the bashrc file is outdated and needs to be updated:
https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-7.3-(v7r3)#bashrc

On 2021-11-19 16:10, Federico Stagni wrote:
> Hi Xiaomei,
> have you gone through
> https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#rename-of-scripts ?
>
> Cheers,
> Federico

[zhangxm]

Hi,

Before updating,  I have already done the "rename" and "update bashrc".

(1) Here is part of "run":

"

[[ "service" = "agent" ]] && renice 20 -p $$
exec dirac-service Configuration/Server --cfg /opt/dirac/etc/Configuration_Server.cfg < /dev/null

"

(2) Here is the bashrc script used:"

# DIRAC bashrc file, used by service and agent run scripts to set environment
# Find which DIRAC installation to use
default_dirac="/opt/dirac/pro"

# If "pro" points to a Python 3 version there will be a Linux-x86_64 folder
system_arch="$(uname -s)-$(uname -m)"
if [ -d "${default_dirac}/${system_arch}" ]; then
    default_dirac="${default_dirac}/${system_arch}"
fi

if [[ -n "${DIRAC:-}" ]] && [[ "${default_dirac}" != "${DIRAC:-}" ]]; then
    echo DIRAC installation path overridden to ${DIRAC}
fi

export DIRAC=${DIRAC:-${default_dirac}}
unset default_dirac system_arch

# CAs path for SSL verification
export X509_CERT_DIR=${X509_CERT_DIR:-"/etc/grid-security/certificates"}
export X509_VOMS_DIR=${X509_VOMS_DIR:-"/etc/grid-security/vomsdir"}
export X509_VOMSES=${X509_VOMSES:-"/etc/vomses"}

if [[ -d "${DIRAC}/diracos" ]]; then
    py3_based=0
    . $DIRAC/diracos/diracosrc
else
    py3_based=1
    . $DIRAC/diracosrc
fi

# Configuration which is only required for legacy installations
if [[ "${py3_based}" -eq 0 ]]; then
    export PYTHONUNBUFFERED=yes
    export PYTHONOPTIMIZE=x
    export DIRACSCRIPTS=$DIRAC/scripts
    export TERMINFO=$DIRACOS/usr/share/terminfo:/usr/share/terminfo:/etc/terminfo
    export RRD_DEFAULT_FONT=$DIRACOS/usr/share/fonts/DejaVuSansMono-Roman.ttf

........

"

Best regards,

Xiaomei

在 2021-11-19 23:10, Federico Stagni 写道:

Hi Xiaomei,

have you gone through https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#rename-of-scripts ?

Cheers,
Federico

On 19/11/21 16:07, zhangxm wrote:

[zhangxm]

Hi,

If I run it by hand, it is working:

(base) [dirac@samtest dirac]$ /opt/dirac/bashrc

(base) [dirac@samtest dirac]$ exec dirac-service Configuration/Server --cfg /opt/dirac/etc/Configuration_Server.cfg < /dev/null
2021-11-22 03:32:15 UTC Configuration/Server INFO: Loading Configuration/Server
2021-11-22 03:32:15 UTC Configuration/Server INFO: Trying to Configuration/Server from CS defined path DIRAC/ConfigurationSystem/Service/ConfigurationHandler.py
2021-11-22 03:32:15 UTC Configuration/Server NOTICE: Loaded module Configuration/Server
2021-11-22 03:32:15 UTC Configuration/Server INFO: Initializing Configuration/Server
2021-11-22 03:32:15 UTC Configuration/Server INFO: Found 1 initialization methods
2021-11-22 03:32:15 UTC Configuration/Server INFO: Initializing Configuration Service URL is dips://samtest.ihep.ac.cn:9135/Configuration/Server
2021-11-22 03:32:15 UTC Configuration/Server INFO: Starting configuration service as master
2021-11-22 03:32:15 UTC Configuration/Server INFO: Starting purge slaves thread
2021-11-22 03:32:15 UTC Configuration/Server ALWAYS: Listening at dips://samtest.ihep.ac.cn:9135/Configuration/Server

but not with runsvctrl.

Regards,

Xiaomei

[Chris Burr]

Hi Xiaomei,

Ah I should have noticed earlier, the problem is with:

> "DIRAC installation path overridden to /opt/dirac/pro
> ./run: line 11:exec: dirac-service: can't find"

Somehow you have `$DIRAC` defined, and defined incorrectly. If I look at
LHCb's configuration service that is using Python 3 and modify the run
file to be:

> #!/bin/bash
> echo "CBURR1: DIRAC=$DIRAC"
> rcfile=/opt/dirac/bashrc
> [[ -e $rcfile ]] && source ${rcfile}
> echo "CBURR2: DIRAC=$DIRAC"
> exec 2>&1

> [[ "service" = "agent" ]] && renice 20 -p $$
> exec dirac-service Configuration/Server --cfg
> /opt/dirac/etc/Configuration_Server.cfg < /dev/null

I see the log prints this when the service is restarted:

> CBURR1: DIRAC=
> CBURR2: DIRAC=/opt/dirac/pro/Linux-x86_64
> 2021-11-22 09:26:21 UTC Configuration/Server INFO: Loading
> Configuration/Server
> 2021-11-22 09:26:21 UTC Configuration/Server INFO: Trying to

> Configuration/Server from CS defined path
> DIRAC/ConfigurationSystem/Service/ConfigurationHandler.py

Can you try and figure out where `DIRAC=/opt/dirac/pro` is coming from?

Cheers,

Chris

> https://groups.google.com/d/msgid/diracgrid-forum/68988b52-d1d1-5462-84ba-b49dc19b6f06%40ihep.ac.cn
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/diracgrid-forum/68988b52-d1d1-5462-84ba-b49dc19b6f06%40ihep.ac.cn?utm_medium=email&utm_source=footer

[zhangxm]

Hi Chris,

Thank you for clues.  It is found out to be a problem of runsvdir-start
which is not correctly running. After restarting it, it is working.

Another problem found is mysql. With the new bashrc, mysql client from
Dirac has problems to connect the original mysql server:
(base) [dirac@samtest ~]$ mysql -version
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/tmp/mysql.sock' (2)
(base) [dirac@samtest ~]$ mysql --version
mysql  Ver 8.0.27 for Linux on x86_64 (conda-forge)

The one in /bin/mysql is able to connect.
[dirac@samtest ~]$ mysql -uDirac -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14522
Server version: 8.0.26 MySQL Community Server - GPL

I am not sure if it is caused by the different version ( 8.0.26 vs.
8.0.27 ), or something else?


Best regards,
Xiaomei

[Federico Stagni]

Hi,

what message do you get when trying to connect? And what is the version of the MySQL server?

Federico

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/6a6d1162-8e43-2c03-e624-b35b642f93ac%40ihep.ac.cn.

[zhangxm]

Hi,

Our mysql server is using 8.0.26:

[dirac@samtest ~]$ mysql --version

mysql  Ver 8.0.26 for Linux on x86_64 (MySQL Community Server - GPL)
[dirac@samtest ~]$ rpm -qa|grep "mysql"
mysql80-community-release-el7-3.noarch
mysql-community-libs-8.0.26-1.el7.x86_64
mysql-community-client-plugins-8.0.26-1.el7.x86_64
mysql-community-client-8.0.26-1.el7.x86_64
mysql-community-server-8.0.26-1.el7.x86_64
mysql-community-common-8.0.26-1.el7.x86_64
mysql-community-libs-compat-8.0.26-1.el7.x86_64

It failed with mysql client under dirac:

[dirac@samtest ~]$  /opt/dirac/versions/v7.3.8-1637279175/Linux-x86_64/bin/mysql -uDirac -p
Enter password:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

But it is working with mysql client under /bin:

[dirac@samtest ~]$ /bin/mysql -uDirac -p

Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 14537

Server version: 8.0.26 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit
Bye

Best regards,

Xiaomei

[zhangxm]

In jobMonitoring log, there is an error about connecting mysql:

2021-11-25 01:05:40 UTC WorkloadManagement/JobMonitoring ERROR: Cannot connect to the DB  MySQL Error ( 1131 : Could not connect: (2002, "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)"))
2021-11-25 01:05:40 UTC WorkloadManagement/JobMonitoring ERROR: Error while initializing WorkloadManagement/JobMonitoring: Can't connect to DB: Can not connect to DB 'JobDB', exiting...

If I specified 127.0.0.1 or samtest.ihep.ac.cn, the mysql command is working, but the jobMontioring service still can't connect to the DB.

mysql -uDirac -h samtest.ihep.ac.cn -p

IHEP configuration in CS is like these:

 
  Databases
  {
    User = Dirac
    Password = xxxxx
    RootUser = root
    RootPwd = xxxxx
    Host = samtest.ihep.ac.cn
    Port = 3306
  }

Regards,

Xiaomei

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/9d47f65d-8197-dca5-dbfd-5d9a7b7b236b%40ihep.ac.cn.

[zhangxm]

Hi,

I found the reason. It is because of different socket (my.sock) locations.  The mysql under DIRAC packages is asking for /tmp/my.sock and the mysql server is using /var/lib/mysql/mysql.sock.

Therefore, the soft link solved the problems.

Best regards,

Xiaomei

To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/7386a955-89f1-f7f0-c5ab-22ed66fd9bf8%40ihep.ac.cn.