Failed to download metadata from /Shibboleth.sso/DiscoFeed

[Asbjørn KU-IT]

Hello

We are using Shibboleth (WAYF) for authentication.

We are encountering the following error when reaching the login page:

"Failed to download metadata from /Shibboleth.sso/DiscoFeed"

Does anyone know what this error means and how to fix it?`

You can see the error live here: https://t-dataverse.ku.dk/loginpage.xhtml?redirectPage=dataverse.xhtml

We use Ansible for installation.

Regards

Asbjørn

University of Copenhagen

[Paul Boon]

Hi Asbjørn,

At DANS we have changed the Dataverse code (in our fork) to not use that 'discoFeed' endpoint.

Our IdP has its' own WAYF and the dropdown list would contain only on entry.

Regretfully I don't remember how we got that discoFeed working, because we had it working at some tests we did years ago, and the person changing the configs then doesn't work with us anymore.

If you have the same problem as we have; just one IdP as 'proxy' for the others, it would be useful If I created an Issue and PR for that special case.

Also, I am interested in how to 'fix' the discoFeed endpoint.

Regards,

Paul Boon (DANS)

---

From: datave...@googlegroups.com <datave...@googlegroups.com> on behalf of Asbjørn KU-IT <a...@adm.ku.dk>
Sent: Tuesday, December 5, 2023 9:19 AM
To: Dataverse Dev <datave...@googlegroups.com>
Subject: [Dataverse-Dev] Failed to download metadata from /Shibboleth.sso/DiscoFeed

 

--
You received this message because you are subscribed to the Google Groups "Dataverse Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/1f1b100e-5819-46ad-b1c5-94eea44ac420n%40googlegroups.com.

[Philip Durbin]

Hi Asbjørn,

I'm a little confused. Was Shibboleth/DiscoFeed ever working properly? Or has it been broken (not working) the whole time?

Our docs do try to explain that DiscoFeed is supposed to be working: https://guides.dataverse.org/en/6.1/installation/shibboleth.html#verify-discofeed-and-metadata-urls

On your installation is seems like neither https://t-dataverse.ku.dk/Shibboleth.sso/DiscoFeed nor https://t-dataverse.ku.dk/Shibboleth.sso/Metadata are working (both 404s) so we might need to retrace your steps (or Ansible's steps) to see if something was missed.

I hope this helps,

Phil

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/DBBPR02MB1046388FDDD9EF0A456122A0FD985A%40DBBPR02MB10463.eurprd02.prod.outlook.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Asbjørn KU-IT]

Hi Philip

Thanks for replying.

No, our Shibboleth was never working. It's been broken the entire time.

We've based our Ansible installation on the dataverse-ansible repository and made adjustments to the code.

We've read the official documentation several times.

I can share any number of YAML files from our Ansible, or preferably would you or a colleague be available for a remote bugfixing session?

We are running RHEL8 and Dataverse 6.0.

[Don Sizemore]

Hello,

I wouldn't expect Dataverse-Ansible's implementation to result in a working Shibboleth installation. It's stubbed out, barely-tested if at all (pull requests welcomed), and each institution's IDP is something of a snowflake.

That said, I'd be happy to screen-share some time and take a look. If you're not seeing DiscoFeed I'd start by looking in Shibboleth's warn_log or equivalent.

Thank you,

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/41827df1-ff4d-4dd2-9689-8e2ee0e081can%40googlegroups.com.

[Philip Durbin]

Oh, one gotcha is SELinux. You probably saw this in the guides: https://guides.dataverse.org/en/6.1/installation/shibboleth.html#disable-or-reconfigure-selinux .As a quick test you could try disabling it.

We definitely need that /Shibboleth.sso/Metadata path working.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/CAPfMOaytS6nGK3ZbJKrFLXj9ABuTgSLTP4HZW%3DOmd6vt5iZG1Q%40mail.gmail.com.

[Don Sizemore]

oh, yes. for SELinux and Shibboleth:

# Allow httpd

sudo /usr/sbin/setsebool -P httpd_can_network_connect 1
sudo /usr/sbin/setsebool -P httpd_read_user_content 1

# Allow httpd to connect to Shib socket
sudo grep httpd_t /var/log/audit/audit.log |/usr/bin/audit2allow -M allow_httpd_shibd_sock
sudo /usr/sbin/semodule -i allow_httpd_shibd_sock.pp

# Allow httpd to read /var/cache/shibboleth
sudo /usr/sbin/semanage fcontext -a -t httpd_sys_content_t "/var/cache/shibboleth(/.*)?"

sudo /usr/sbin/restorecon -vR /var/cache/shibboleth

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/CABbxx8HfsCGQf8uRbCL_uzDSzjB5nvAUVhhm3OYQSgOnqrvk4A%40mail.gmail.com.

[Asbjørn KU-IT]

Happy New Year!

We would be very grateful to have a session with you, @Don Sizemore.

Let us know when you may be available, our agendas are open for any timeslot and date to resolve this issue for us.

For now we have SELinux set to permissive, as this:

- name: Set SELinux in permissive mode, logging actions that would be blocked

  ansible.posix.selinux:

    policy: targeted

    state: permissive

[Philip Durbin]

I'm confused. Was it SELinux? In permissive mode is everything working?

Also, why are we on the dev list? :) Next time, let's use the main list.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/ef42434d-61f6-42a8-b746-7581a05cf584n%40googlegroups.com.

[Asbjørn KU-IT]

Hello Philip

We had a great session with Don and we got some errors from the shib warnings log fixed. The only logs we have now is:

2024-01-08 12:39:46 WARN Shibboleth.DEPRECATION : MetadataGenerator handler
2024-01-08 12:39:47 WARN Shibboleth.AttributeExtractor.XML : attribute mappings are reloadable; be sure to restart web server when adding new attribute IDs

However, it did not fully resolve our issues.

We have SELinux set to permissive and that is OK according to Don.

Do you have any pointers for us on how to resolve this?

The dev list was a mistake on my behalf. I thought I wrote to the other channel. If this can be moved to the other channel, that is the right thing to do.

[Philip Durbin]

No, that's ok. Let's just stay on this list for this problem. Let me check in with Don about the state of things. We both hang out at https://chat.dataverse.org if you ever want to pop in.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/874283b6-3fdf-4bac-a546-7154f1880cddn%40googlegroups.com.

[Asbjørn KU-IT]

We talked with our idm group/manager as Don suggested and he helped us fix some stuff:

- In shibboleth2.xml our tag for "include" in the MetadataProvider tag was <include> instead of uppercase <Include>

- We can get the necessary files /Metadata and /DiscoFeed on the localhost domain using:  "curl -sk https://localhost/Shibboleth.sso/DiscoFeed"

- The files are denied using our site URL

- Tomorrow we will try to replace "localhost" with our site URL in the config files and see what happens, as we suspect this is now our problem. We identified three places, we have inserted "localhost". In Apache/httpd, Payara and the Dataverse installer arguments. Is there any one of these that are the right ones to change, or should we just apply our trial'n'error approach as suggested?

Regards

Asbjørn

[Don Sizemore]

Hello,

When you say "denied" does your Apache proxy include

  <IfModule mod_shib>
    # don't pass paths used by Shibboleth to Payara
    ProxyPassMatch ^/Shibboleth.sso !
    ProxyPassMatch ^/shibboleth-ds !
    ProxyPassMatch ^/shibboleth-sp !

    <Location /shib.xhtml>
      AuthType shibboleth
      ShibRequestSetting requireSession 1
      require shibboleth
    </Location>
  </IfModule>

The Payara siteUrl setting needs to remain the public FQDN of your service, https://t-dataverse.ku.dk

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/c81a365b-3731-4f0b-bfb4-95cbf2eabe36n%40googlegroups.com.

[Asbjørn KU-IT]

We have this:

<IfModule mod_shib>

  # don't pass paths used by Shibboleth to Glassfish

  ProxyPassMatch ^/Shibboleth.sso !

  ProxyPassMatch ^/shibboleth-ds !

  ProxyPassMatch ^/shibboleth-sp !

  <Location /shib.xhtml>

    AuthType shibboleth

    ShibRequestSetting requireSession 1

    require shibboleth

  </Location>

</IfModule>

# pass everything else to Glassfish

ProxyPass / ajp://localhost:8009/

# ProxyPass / http://localhost:8080/

Great, thanks.

We have set up full siteURL all those places the dataverse-ansible repo had "payara.siteurl" mentioned.

Do we need to set this in default.config Dataverse installer arguments also?

The dataverse-ansible repo has two different variants of the file, one with "localhost" and a template with "{{ servername }}" variable, but we cannot find this variable defined in defaults/main.yml.

[Asbjørn KU-IT]

by that I mean this part

[glassfish]

HOST_DNS_ADDRESS = {{ payara_hostname }}

The two variants:

https://github.com/search?q=repo%3AGlobalDataverseCommunityConsortium%2Fdataverse-ansible%20%5Bglassfish%5D&type=code

[Don Sizemore]

You all wrote your own Ansible role, but the end result is that your Payara dataverse.siteUrl jvm-option needs to point to the https://hostname of your service.

files/default.config can likely be cleaned up; I've made a note to remove it.

The ProxyPassMatch lines are what you need to enable access to Shibboleth.sso/, when you say these URIs are "denied" what error are you seeing?

Thank you,

Don

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/6cc4a9f8-c934-43bd-bf34-cdaab0b39fc8n%40googlegroups.com.

[Asbjørn KU-IT]

Yes, that is true. You are right. 

We get "404 Not Found"

Example: https://t-dataverse.ku.dk/Shibboleth.sso/Metadata

However when we curl the same URL replacing " t-dataverse.ku.dk" with "localhost" from inside the terminal of our Payara webserver, the expected content are delivered.

I will check the value of the Payara "dataverse.siteUrl jvm-option" and get back to you on that when we have our "asadmin" command fixed.

[Asbjørn KU-IT]

These are our JVM options

-Ddataverse.fqdn=datavrsweb01tl.unicph.domain
-Ddataverse.siteUrl=http://${dataverse.fqdn}:8080

Could this be the reason to our problems? So it does not say "localhost". The FQDN works, however browser reports the URL as insecure.

Our "real" domain is https://t-dataverse.ku.dk

This file "idpselect_config.js" has this "localhost" value as seen below, is this a problem?

this.dataSource = window.location.hostname === 'localhost' ? '/resources/dev/sample-shib-identities.json' : '/Shibboleth.sso/DiscoFeed'; // where to get the data from (dev vs. prod)

[Donald Sizemore II]

you want both jvm-options using the public address of your service - fqdn and siteurl. the latter prepended with https://

don

painstakingly pecked on my iphone.

On Jan 15, 2024, at 05:30, Asbjørn KU-IT <a...@adm.ku.dk> wrote:

These are our JVM options

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/6bd46311-baad-4bc5-be3a-1cfdcdc93102n%40googlegroups.com.

[Philip Durbin]

Yes, to reiterate what Don said, siteUrl should be your real domain: https://t-dataverse.ku.dk

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/D77D014E-4ED4-4C6E-97EB-04632BFE1277%40gmail.com.

[Asbjørn KU-IT]

Hello

We got it working now. TBH we don't know what really fixed it. We did several things: updated the JVM options fqdn and siteUrl, changed apache config file (removed SSL, point to port 80) and did complete reinstall of everything including operating system on the virtual machines.

Thank you so much for your guidance @Don and @Philip.

We feel very relieved.

[Philip Durbin]

We're relieved as well! Good job sticking with it!

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-dev/e4b7bda2-8c2d-4ac0-aff5-a61b55303d6fn%40googlegroups.com.