Generation of proxy certificate with VOMS attached on cloud

Igor Pelevanyuk

unread,

Jul 22, 2021, 5:23:04 AM7/22/21

to diracgrid-forum

Hello,

We have virtual organization spd. When I generate proxy on my local machine and try to upload data to EOS using root protocol everything works. But when I submit jobs to cloud, jobs cant submit data to EOS with error:

Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/nica/spd/dirac/spd.nica.jinr/vo/test/inner.out; Operation not permitted

I checked certificate which works on my local machine:

$ voms-proxy-info2 --all
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=6869618326/CN=428092401
issuer : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=6869618326
identity : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=6869618326
type : RFC compliant proxy
strength : 1024 bits
path : /tmp/x509up_u1002
timeleft : 22:44:06
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO spd.nica.jinr extension information ===
VO : spd.nica.jinr
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk
issuer : /C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms01.jinr.ru
attribute : /spd.nica.jinr/Role=NULL/Capability=NULL
timeleft : 22:44:06
uri : lcgvoms01.jinr.ru:15003

But when I go to a virtual machine, log in there as plt00 user, and set X509_USER_PROXY to point on /scratch/tmp?????? certificate I can not submit data with the same error. A check of the certificate does not show any issues:

$ voms-proxy-info2 --all
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=743984388/CN=825427583/CN=5826101989
issuer : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=743984388/CN=825427583
identity : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk/CN=743984388/CN=825427583
type : RFC compliant proxy
strength : 1024 bits
path : /scratch/tmp0s9DWC
timeleft : 19:41:02
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO spd.nica.jinr extension information ===
VO : spd.nica.jinr
subject : /C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Igor Pelevanyuk
issuer : /C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms01.jinr.ru
attribute : /spd.nica.jinr/Role=NULL/Capability=NULL
timeleft : 19:41:03
uri : lcgvoms01.jinr.ru:15003

Somehow certificates look different for EOS.

So I checked proxy with openssl. The working one:

# openssl x509 -in /tmp/x509up_u1003 -text -noout

Certificate:

Data:

Version: 3 (0x2)

Serial Number: 1889812439 (0x70a43fd7)

Signature Algorithm: sha256WithRSAEncryption

Issuer: C=RU, O=RDIG, OU=users, OU=jinr.ru, CN=Igor Pelevanyuk, CN=9908250618

Validity

Not Before: Jul 22 08:12:57 2021 GMT

Not After : Jul 23 08:11:57 2021 GMT

Subject: C=RU, O=RDIG, OU=users, OU=jinr.ru, CN=Igor Pelevanyuk, CN=9908250618, CN=1889812439

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (1024 bit)

Modulus:

00:a2:fb:86:cc:db:df:bf:11:07:9e:54:be:ff:e9:

a0:56:ab:57:83:04:46:9b:49:2c:b8:5f:a3:75:53:

e6:a7:50:fc:a1:51:14:8d:3e:45:cb:41:40:45:b9:

8a:c5:bb:d4:98:ab:d5:77:34:c7:c7:9f:51:eb:aa:

f6:09:85:b2:a5:68:7d:de:dd:70:a5:1b:f9:77:f8:

7b:da:5b:38:92:cb:cb:4e:fe:d8:0d:6f:0d:24:a0:

eb:71:ee:de:cf:10:93:1b:c3:ce:13:f8:b0:47:6c:

64:1c:a5:fe:7b:71:30:58:4f:c5:26:e1:ea:13:d3:

aa:6c:69:f1:f6:80:ff:5e:f9

Exponent: 65537 (0x10001)

X509v3 extensions:

1.3.6.1.4.1.8005.100.100.5:

0...U. 0...0...0...0......0d.b0\.Z0X1.0...U....RU1

0...U.1.0...U....users1.0...U....jinr.ru1.0...U....Igor Pelevanyuk..&..`0^.\0Z1.0...U....RU1

............sH..s.P .>.0"..20210722081757Z..20210723081157Z0i0g.r.ru0

+.....Edd.1Y0W.).'spd.nica.jinr://lcgvoms01.jinr.ru:150030*.(/spd.nica.jinr/Role=NULL/Capability=NULL0..50....

+.....Edd

0...U.E1.0...U....RU1......&.0

0...U.094113Z0Z1.0...U....RU1-Intensive Grid CA0..

..........0......hosts1.0...U....jinr.ru1.0...U....lcgvoms01.jinr.ru0.."0

.......rsj0....\u...P.$[.|#A.z.f..h.E.M.4.=.z...4...l...ld.uh.....n....o....EJ...P......+......bg.9*.%o.9........-....V...h.s..m.U...v..y#..b...?l..U(..(.t.-......*;t.. Z+...R......x....n..Bl1....}...0...../.me...E..>...G.}.2.T}l..H..~.O.....^!e8.

0...U..L..7.#.........0...0...U.......0.0...U...........0...`.H...B........0...U....0...lcgvoms01.jinr.ru0...U.......w1-f...$...:...7...0m..U.#.f0d.....}...a...!h.~`..pY.I.G0E1.0...U....RU1

..RDIG1'0%..U....Russian Data-Intensive Grid CA...0=..U...604..rdig-ca@grid.kiae.ru..http://ca.grid.kiae.ru/RDIG/0?..+........3010/..+.....0..#http://ca.grid.kiae.ru/RDIG/ca.html06..U.../0-0+.).'.%http://ca.grid.kiae.ru/RDIG/cacrl.pem0Z..U. .S0Q0..

............:##.X)..e.U..Jk.^...@... ..+F9.]2@.(-..W!..Dqy.!.o.......pYn1...............K.'.../.'.+x.u.N...a.H,.7..C.....7.u .w.......&.".......#=f:#.....2@.{.F...I#..<.l.......O..[/6.V^3...!..:....b...&...p^[.L..H.\}......e.=..........\...S=.........F, ....@mE..S..v.w^..%I...wE.QHz..9.z......7. \. `?..{9vB.F...3}~.>....X`H..>W..#.l0..+32n..

X509v3 Key Usage: critical

Digital Signature, Key Encipherment, Data Encipherment

X509v3 Basic Constraints: critical

CA:FALSE

X509v3 Authority Key Identifier:

0.

Proxy Certificate Information: critical

Path Length Constraint: infinite

Policy Language: Inherit all

Signature Algorithm: sha256WithRSAEncryption

80:cb:98:a9:fb:9a:c7:00:5b:9d:e4:e4:a9:ec:57:d6:8f:93:

13:b6:40:5e:20:59:00:ed:07:d0:70:5e:ea:a7:f8:32:fc:fe:

f2:9b:e8:56:a0:1e:e8:25:7a:0c:08:78:e5:60:64:2f:73:8e:

be:6c:11:3a:4b:18:8e:8c:01:45:ec:dd:66:b6:15:58:64:3c:

14:27:66:81:65:d9:ca:c3:d0:9d:81:ba:06:0b:da:c0:fd:00:

b7:7e:b2:05:07:d3:03:cc:02:41:66:bf:b4:b6:a9:7a:f4:b7:

f2:ce:a3:0d:bb:39:c9:7e:29:47:53:26:14:7c:17:a6:65:6f:

68:fe

And here is failing one:

# openssl x509 -in /scratch/tmpShQVww -text -noout

Certificate:

Data:

Version: 3 (0x2)

Serial Number:

38:33:36:30:39:39:38:39:35:32

Signature Algorithm: sha256WithRSAEncryption

Issuer: C=RU, O=RDIG, OU=users, OU=jinr.ru, CN=Igor Pelevanyuk, CN=743984388, CN=2749424050

Validity

Not Before: Jul 22 07:05:59 2021 GMT

Not After : Jul 23 07:20:58 2021 GMT

Subject: C=RU, O=RDIG, OU=users, OU=jinr.ru, CN=Igor Pelevanyuk, CN=743984388, CN=2749424050, CN=33566894

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

Public-Key: (1024 bit)

Modulus:

00:9c:f6:7a:e6:d2:92:fa:f4:ba:af:6d:03:da:9a:

93:6c:e5:67:b9:1f:83:46:46:a0:63:aa:fe:fb:9e:

a3:08:17:98:bd:08:bd:cc:4e:76:c5:3c:7f:e3:f5:

32:04:d8:9a:a2:2d:f0:de:e3:3a:46:6f:48:3e:90:

21:f3:03:cb:ae:d0:ba:c3:ba:12:ce:79:04:1e:f7:

37:55:db:67:66:a9:67:34:cd:c6:c7:67:83:42:7c:

93:bc:16:46:4b:8c:70:c0:b4:1b:94:a4:54:d4:a5:

95:2d:63:b6:3d:60:19:e9:3e:a9:1e:d7:e3:02:a5:

e5:cc:5c:af:9b:5f:58:3a:85

Exponent: 65537 (0x10001)

X509v3 extensions:

X509v3 Key Usage: critical

Digital Signature, Key Encipherment, Data Encipherment

Proxy Certificate Information: critical

Path Length Constraint: infinite

Policy Language: Inherit all

Signature Algorithm: sha256WithRSAEncryption

33:ae:a9:d5:5c:7a:26:6c:37:56:34:d9:26:ce:27:82:4a:83:

bf:c9:e1:f0:fb:c2:f3:38:72:66:c7:56:96:75:f6:c9:dd:b9:

3e:95:f8:94:60:ed:4c:6b:d2:42:56:3f:7f:ab:41:30:4e:9b:

35:9f:e7:54:54:bd:ee:68:11:f6:4d:4b:21:ef:ee:f0:65:39:

c4:d7:e6:5f:a9:2f:63:79:da:a0:4a:69:76:2d:f8:f9:74:0a:

2e:25:77:0a:56:4b:20:4e:23:17:42:f6:47:c7:a9:5f:a7:19:

27:2c:f6:0f:6a:57:b4:71:f8:52:24:98:85:8b:39:15:08:21:

42:87

I am not a specialist in x509 certificates, but these outputs look different for me. For dirac-proxy-info and voms-proxy-info they are the same.

Can you point to me what component is responsible for generating proxy on clouds? Maybe someone saw the same behavior?

Kind regards,

Igor Pelevanyuk

Simon Fayer

unread,

Jul 22, 2021, 9:07:21 AM7/22/21

to Igor Pelevanyuk, diracgrid-forum

Hi Igor,

On Thu, Jul 22, 2021 at 02:23:04AM -0700, Igor Pelevanyuk wrote:
> Somehow certificates look different for EOS.

> And here is failing one:
> # openssl x509 -in /scratch/tmpShQVww -text -noout

> Certificate: ...

This certificate from the openssl output seems to be missing the VOMS
extension. I would check the following things:

- Does your VM image have an /etc/vomses directory? If it does, does it
have the correct entries in and is it readable by all users?
- Are you able to get the pilot log (I think this is normally stored in
the scratch directory and called pilot.out)? If you can find that, are
there any VOMS related warnings/errors in there?

We've seen this before in a few (non-cloud) places and it's normally caused
by issues contacting the VOMS servers. There is no special cloud component
for setting up the user proxy, the DIRAC JobAgent does this as part of the
job initialisation as it would for any other grid job.

Regards,
Simon

Igor Pelevanyuk

unread,

Jul 22, 2021, 10:06:32 AM7/22/21

to diracgrid-forum

Hello Simon,

VM image does not have vomses. So during vm-bootstrap I copy all from /cvmfs/grid.cern.ch/etc/grid-security to /etc/grid-security.

After that I add specific vomses and vomsdir in the following way:

echo "/C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms01.jinr.ru" > /etc/grid-security/vomsdir/spd.nica.jinr/lcgvoms01.jinr.ru.lsc

echo "/C=RU/O=RDIG/CN=Russian Data-Intensive Grid CA" >> /etc/grid-security/vomsdir/spd.nica.jinr/lcgvoms01.jinr.ru.lsc

echo '"spd.nica.jinr" "lcgvoms01.jinr.ru" "15003" "/C=RU/O=RDIG/OU=hosts/OU=jinr.ru/CN=lcgvoms01.jinr.ru" "spd.nica.jinr" "24"' > /etc/grid-security/vomses/spd.nica.jinr

All files are readable by plt00. Maybe I should put vomsdir and vomses into different directory
About pilot.out: there are some lines related to VO, but it is only parameters for commands like:

2021-07-22 07:19:18 UTC DEBUG [Pilot] PARAMETER [('--debug', ''), ('--setup', 'JINR-Production'), ('-r', 'v7r0p27'), ('-l', 'DIRAC'), ('-e', 'VMDIRAC'), ('--MaxCycles', '10'), ('--configurationServer', 'dips://dirac-conf.jinr.ru:9135/Configuration/Server'), ('--Name', 'JINR'), ('--name', 'CLOUD.JINR.ru'), ('--userEnvVariables', 'DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True'), ('-Q', 'CentOS7_spd'), ('--cert', ''), ('--certLocation', '/scratch/plt00/etc/grid-security'), ('-o', '/Resources/Computing/CEDefaults/SubmitPool=mpdPool'), ('-o', '/Resources/Computing/CEDefaults/VirtualOrganization=spd.nica.jinr'), ('-o', '/Resources/Computing/CEDefaults/NumberOfProcessors=1'), ('-o', '/Resources/Computing/CEDefaults/WholeNode=True'), ('-o', '/Systems/WorkloadManagement/Production/Agents/JobAgent/StopAfterFailedMatches=0'), ('-o', '/Systems/WorkloadManagement/Production/Agents/JobAgent/CEType=Pool')]

2021-07-22 07:19:57 UTC INFO [CheckCECapabilities] Executing command dirac-resource-get-parameters -S CLOUD.JINR.ru -N JINR -Q CentOS7_spd -o /DIRAC/Security/UseServerCertificate=yes

2021-07-22 07:19:58 UTC DEBUG [CheckCECapabilities] Return code of dirac-resource-get-parameters -S CLOUD.JINR.ru -N JINR -Q CentOS7_spd -o /DIRAC/Security/UseServerCertificate=yes: 0

The only error is related to:

2021-07-22 07:19:56 UTC dirac-install [ERROR] Requirements installation script /scratch/plt00/scripts/dirac-externals-requirements failed. Check /scratch/plt00/scripts/dirac-externals-requirements.err

There is one thing: submitPool. Clouds do not work if this is not configured, so I had to hardcode it tom mpdPool in CloudDirector. As far as I know, that option does not change anything. But maybe I am wrong.

Kind regards,

Igor Pelevanyuk

Daniela Bauer

unread,

Jul 22, 2021, 10:15:43 AM7/22/21

to Igor Pelevanyuk, diracgrid-forum, Matt Doidge

As an aside:

If you would like to add a VO to /cvmfs/grid.cern.ch please open a GGUS ticket for it and/or email (with all the config details) Matt Doidge at Lancaster (in the cc). It really is no problem to add them (same comment is probably also true for Andrei T ;-).

Regards,

Daniela

--
You received this message because you are subscribed to the Google Groups "diracgrid-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to diracgrid-for...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/diracgrid-forum/aa411d57-f9d4-4dd3-8464-8ed5b9ef325bn%40googlegroups.com.

--

Sent from my guinea pig enhanced living room

-----------------------------------------------------------
daniel...@imperial.ac.uk
HEP Group/Physics Dep
Imperial College
London, SW7 2BW
Tel: Working from home, please use email.
http://www.hep.ph.ic.ac.uk/~dbauer/

Simon Fayer

unread,

Jul 22, 2021, 10:53:39 AM7/22/21

to Igor Pelevanyuk, diracgrid-forum

Hi Igor,

On Thu, Jul 22, 2021 at 07:06:32AM -0700, Igor Pelevanyuk wrote:
> The only error is related to:
> 2021-07-22 07:19:56 UTC dirac-install [ERROR] Requirements installation
> script /scratch/plt00/scripts/dirac-externals-requirements failed. Check
> /scratch/plt00/scripts/dirac-externals-requirements.err

Hmm, that's a bit dubious, could you please send me the entire pilot.out
and the .err file (if it exists) off-list?

> There is one thing: submitPool. Clouds do not work if this is not
> configured, so I had to hardcode it tom mpdPool in CloudDirector. As far as
> I know, that option does not change anything. But maybe I am wrong.

I think that's fine... If that was wrong, it just wouldn't pick-up any jobs
at all.

Regards,
Simon

Igor Pelevanyuk

unread,

Oct 25, 2021, 3:25:36 AM10/25/21

to diracgrid-forum

Hello everyone,

This issue was quite tough. Now it is clear that the reason is not DIRAC itself, but rather EOS configuration.

My user is different since I participate in many VOs. It looks like EOS can not correctly map me to the right VO. I was able to overcome this issue by requesting a new certificate and registering it to SPD VO.

I think the problem is mostly related to our local EOS configuration.

Thanks, Simon and Daniela for the help!

Kind regards,

Igor Pelevanyuk

Reply all

Reply to author

Forward