Failure to deploy BitCurator on Ubuntu 22.04.1

[matthew.phi...@gmail.com]

Hi All,

I have followed the quickstart guide for BitCurator version 4.3.0 and later to deploy in
Ubuntu 22.04LTS, but it is failing. Looking at the saltstack.log (attached with username and device name redacted), I don't understand why? It looks like many of the tools have been installed. It also fails when trying to install with addon mode. Any thoughts or advice appreciated.

Kind regards,

Matthew

[co...@digitalsleuth.ca]

Hi Matthew,

I've reviewed the log and it appears as if there was some sort of permissions issue when attempting to modify the group to which your chosen user was assigned.

Would you be able to provide me the command line you used to install, and let me know if the user you chose was destined to have a different group-id than that of the user-id?

I'll do some testing tonight, but knowing the parameters of your install might help me to narrow down an issue quicker.

Cheers!

Corey

[matthew.phi...@gmail.com]

Hi Corey,

Thank you for reviewing the log. I ran the installer on the command line using the simple install option 'sudo bitcurator install' and also tried addon mode with 'sudo bitcurator install --addon' and both failed to complete.

Something I should have mentioned in my initial message is that I am trying to install it on a VM that uses my Active Directory (AD) login credentials from my organisation - not a local Linux account on the VM. Would that impact the group-id? My account has sudo privileges.

Thanks,

Matthew

[co...@digitalsleuth.ca]

Hi Matthew,

It is very possible that this could be an issue, especially if the AD user doesn't have permissions within the VM to affect the VM user. Can you tell me what the uid, gid, and groups are on the VM you were trying to set it up on, and the SID (just the RID - last 3-4 digits) for your AD credentials?

I don't need usernames, just looking for a conflict or unintentional match between the uids for both.

For linux, please run the following:

`id`

You can sanitize the username, but if possible please leave the uid, gid, and group for the username

For windows:

`wmic useraccount get sid`

And you can remove everything prior to the last dash (just need the digits after the last dash)

In the meantime, I'll try running some tests and see if there's anything else that can be done.

[matthew.phi...@gmail.com]

Hi Corey,

I ran 'id' but the results are not suitable for sharing publicly as it includes all of the AD groups my username is in (there are many).

Discussing this with my ICT colleague, I have been advised that the VM is AD-joined and that by default Linux joined to an Active Directory domain will detect the default group as "domain users". To prevent others from accessing my home directory in the Linux VM they changed group ownership to root instead of "domain users".

Does that help?

[co...@digitalsleuth.ca]

Hi Matthew,

I've tried a few different ways to come up with a solution for this, however given the complexities of the account policies your system may have, the first solution I could suggest might cause your system to be dropped from the domain.

This would involve us changing the "user" state to set the "allow_gid_change" option to True. This means it would force change of the users primary Group ID in the system, which could cause authentication issues.

The second option would be to disconnect the system from the domain, run the install / update, then rejoin. It may work, but it may also cause the same Group ID issue as mentioned above.

The third option would be to set up a new VM which has never been joined to the domain, then connect it. All configurations and UID/GID's can be updated and modified at the time of joining, and shouldn't impact your system or permissions.

Since I don't currently have an AD domain to test with, the first two options are relative speculation, but could prove to be valid options. However, the easiest one is the third option, even though it may not be ideal since you already have a system joined to the domain.

I'll keep looking into some options, but it may take a few more days.

Hopefully this helps!

[Peter Brotherton]

Hi Corey,

I have the same issue when using my AD login credentials and I noticed that my user does not exist in /etc/passwd

I could be wrong about this because I don't know saltstack but I think that salt['user.list_users'] is only including users from/etc/passwd which I think means in this code the else block is executed:

https://github.com/BitCurator/bitcurator-salt/blob/main/bitcurator/config/user/user.sls#L8

{% set all_users = salt['user.list_users']() %}
{% if user in all_users %}
...
{% else %}

bitcurator-user-{{ user }}:
  group.present:
    - name: {{ user }}
  user.present:
    - name: {{ user }}
    - gid: {{ group }}
    - fullname: BitCurator
    - shell: /bin/bash
    - home: {{ home }}
    - password: $6...
{% endif %}

I assume the intention of this else clause is for creating a new user but in the case the user already exists it trips up when trying to change the gid. Even if this were changed to force the gid change I think it would still trip up when trying to set the other user properties such as password?

If I am right about this then I think it will cause problems anywhere that all_user is used or in any other function that relies on https://docs.python.org/3/library/pwd.html

Cheers,

Peter

[co...@digitalsleuth.ca]

Hi Peter,

The all_users variable is set when we use SaltStack to list all users in the current environment by using the 'user.list_users' salt command ( {% set all_users = salt['user.list_users']() %})

If the name provided by the pillar above it ({% set user = salt['pillar.get']('bitcurator_user', 'bcadmin') %}) or provided by the --user option in the bitcurator-cli exists in that list of users, then the remainder of the state only confirms / sets the Name and Home directory of the user, it does not try to change the GID.

This also includes the basic "sudo bitcurator install" as the current running user is passed as the bitcurator_user pillar when none is selected.

{% if user in all_users %}

bitcurator-user-{{ user }}:
  user.present:
    - name: {{ user }}
    - home: {{ home }}
{% else %}

If the user does NOT exist (as in not in the list of all users on the system), then the else statement is triggered and the user is to be created.

From the original log, the user was set to be created because the current user did not exist in the list of users. As such, the currently logged in and running user did not have the authority to create a user with a specific group, likely due to their AD credentials not having sufficient privilege.

bitcurator-user-[USERNAME REDACTED]:
  group.present:
    - name: [USERNAME REDACTED]
  user.present:
    - name: [USERNAME REDACTED]
    - gid: [USERNAME REDACTED]

    - fullname: BitCurator
    - shell: /bin/bash

    - home: /home/[USERNAME REDACTED]

- Changing gid (xxxx -> [USERNAME REDACTED]) not permitted, set allow_gid_change to True to force this change. Note that this will not change file ownership.

Salt does use the PWD module to identify the users on the system (in a linux based environ at least - https://github.com/saltstack/salt/blob/2e30ac0a027e63ac6f5e27b2fc088a25e09113e2/salt/modules/useradd.py#L873) as pwd.getpwall(), but we're using it only to list the users on the system, and to check if the chosen user (either by --user or bcadmin) exists. If they do exist, the GID is not changed.

The issue then becomes, did the user actually exist before the installation, and if so, why wasn't it detected?

Matthew: Can you tell me if the user that you were logged in as was created in AD first, then logged in with the AD credentials, or did the account exist in the VM first, then added to AD?

Also, can you run the following command on your BitCurator system and tell me if the user you're logged in as exists in the list which is returned from the command? I don't need the actual names (as I know they're sensitive), I just need to know if your username exists in the list (yes or no)

The command: salt-call --local users.list_users

[Peter Brotherton]

Hi Corey,

Apologies I didn't mention that I work at the same institution as Matthew and we are testing using the same VM configuration. I tried that command without and with sudo and these are the results:

$ salt-call --local users.list_users
[WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion?
'users.list_users' is not available.

$ sudo salt-call --local users.list_users
'users.list_users' is not available.

As for the question "Can you tell me if the user that you were logged in as was created in AD first, then logged in with the AD credentials, or did the account exist in the VM first, then added to AD?"

Our accounts existed in AD before the VM's were provisioned, we logged into the VM with the existing AD credentials. 

Cheers,

Peter

[Corey Forman]

Hi Peter,

Sorry, that should be user.list_users , not USERS.list_users. That's a typo on my part.

---

From: 'Peter Brotherton' via BitCurator Users <bitcurat...@googlegroups.com>
Sent: Tuesday, February 14, 2023 5:58:10 PM
To: BitCurator Users <bitcurat...@googlegroups.com>
Subject: Re: Failure to deploy BitCurator on Ubuntu 22.04.1

 

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/dc493799-ea3c-4a96-a834-4738560a0b26n%40googlegroups.com.

[Peter Brotherton]

Hi Corey,

That command worked and returned a list. My username is not in the list.

Cheers,

Peter

[co...@digitalsleuth.ca]

Hi Peter, sorry for the delay in getting back to you.

Since your current user, isn't in the list of users, this means that SaltStack thinks the user doesn't exist, tries to create it, finds that the user does exist and, when trying to create the group, it's trying to modify the GID because it doesn't match what it thinks the UID is, but it can't. Unfortunately since I don't have an AD environment available to test, anything I try to configure wouldn't guarantee success for you.

If you have a test environment that you could use within an AD environment, I can generate some scenarios which may work for you. Otherwise, are you able to create the machine separately and then join it to AD?

[Peter Brotherton]

Hi Corey,

Thanks for looking into this but that's okay we are going to use the VirtualBox Virtual Appliance method with the ova file instead. I know this method is recommended for testing but given the difficulty we are having with the regular install we are going to try the VirtualBox ova method in a  production environment. Do you see any major issues with this?

Thanks,

Peter

[co...@digitalsleuth.ca]

Hi Peter,

Actually the OVA isn't specifically for testing, as it contains a full install of BitCurator, just as you would get if you were to use a physical environment. I think this would be perfectly fine in your environment and don't see any issues, provided your environment is suited to working with virtual machines.

Additionally, I have been testing an idea based around the logic of user creation that I think might resolve your previous issue. Once it's tested and released, I'll let you know :)

Corey