[docker] How to expose "felix/load"

Giacomo Petillo

unread,

Apr 15, 2021, 1:13:50 PM4/15/21

to dotCMS User Group

Hi all,

exists a way to expose the "/data/share/felix/load" folder?

If is possible, after a gradle build, the jar can be automatically deployed without the manual upload, or exists another way?

The following settings give "Permission denied" error, seems that the folders owner is "1000", but the dotcms is started by "1000000" user.

...
volumes:
# Abilita se hai il file starter.zip in locale
- ${PWD}/starter.zip:/srv/dotserver/tomcat-8.5.32/webapps/ROOT/starter.zip
- ${PWD}/log4j2.xml:/srv/templates/dotcms/OVERRIDE/WEB-INF/log4j/log4j2.xml

- ${PWD}/felix/load:/data/shared/felix/load
- cms-shared:/data/shared

...

Nathan Keiter

unread,

Apr 15, 2021, 1:41:05 PM4/15/21

to dot...@googlegroups.com

I'm not familiar with that exact path, but you might be on a different version than we are.

It sounds like maybe a server configuration error?

The "user" that runs dotCMS should have full access to /felix/load/ (read,write,delete)

{INSTALL-ROOT}/dotserver/{TOMCAT-ROOT}/webapps/ROOT/WEB-INF/felix/load/

That said, there's no built-in way to deploy OSGI plugins other than manual upload or push publishing.

You could always write your own plugin that provides an endpoint to do it, assuming you get that permissions error fixed first so that dotCMS can write to that folder.

If you have control over your local network and local server file systems you could mount it on your SAN using a symlink.

Nathan I. Keiter | Lead Network Applications Programmer | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>
________________________________
From: dot...@googlegroups.com <dot...@googlegroups.com> on behalf of Giacomo Petillo <giacomo...@gmail.com>
Sent: Thursday, April 15, 2021 1:13 PM
To: dotCMS User Group
Subject: [dotcms] [docker] How to expose "felix/load"

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
________________________________

Hi all,
exists a way to expose the "/data/share/felix/load" folder?

If is possible, after a gradle build, the jar can be automatically deployed without the manual upload, or exists another way?

The following settings give "Permission denied" error, seems that the folders owner is "1000", but the dotcms is started by "1000000" user.
...
volumes:

# Abilita se hai il file starter.zip<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstarter.zip&c=E,1,8nXx8JA_wJ2ubsEH34unBfr8LCMfAGaC76G6VCLejsIL3c1GdUZxyu4gzMQWoxc_56O7sB7dxEzNRwgka__spPj7UITf1B-nJMXRipTB&typo=1&ancr_add=1> in locale
- ${PWD}/starter.zip<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstarter.zip&c=E,1,AHzgu6Z_r8m1wQ63BTvdzX-f1xQttPpZ9vsSwiLoXp8rxGC4I89Fg0bZ8CkS4ov6b0rPhitjrcq1HEbUE-a6DV7Vb6N-td-uL23dQ5ND7jVCNg,,&typo=1&ancr_add=1>:/srv/dotserver/tomcat-8.5.32/webapps/ROOT/starter.zip<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstarter.zip&c=E,1,dggf_WdrQ8rAadlx8hU72jf-r18YX3FtaH2VqMu982Gqbvdax_l2aG8pNbC3cYFD9SYnDmOLO0MG3diNGV8By8goIKH-GnBc9iUptXCHbhc,&typo=1&ancr_add=1>

- ${PWD}/log4j2.xml:/srv/templates/dotcms/OVERRIDE/WEB-INF/log4j/log4j2.xml
- ${PWD}/felix/load:/data/shared/felix/load
- cms-shared:/data/shared
...

--
http://dotcms.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fdotcms.com&c=E,1,xQoe2KuPiiJIGXRJPoYNMRav-ejXis6Sb0xOAIxgZOUVUbLpVcLS36x-Wg_IEI7y9ExkfA7UNbAt0Cg8oVDMOcnSBQKiD0eEe4fWX2NwvwK7Np-R&typo=1> - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com<mailto:dotcms+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e375579b-a6d6-44a4-a499-facaa4a1f076n%40googlegroups.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgroups.google.com%2fd%2fmsgid%2fdotcms%2fe375579b-a6d6-44a4-a499-facaa4a1f076n%2540googlegroups.com%3futm_medium%3demail%26utm_source%3dfooter&c=E,1,ePOAfOwv98BB_Bu49lcffzhtKUa-9cgDzKLwHdz79EKpgKFTF4bMrIFzWCg6VljA8qd6aDr-6sxZEpF38a_SnM0bGs1zRMkgKMgeAskyM8arR-6xpJEownkVj0c,&typo=1>.

Nathan Keiter

unread,

Apr 15, 2021, 1:43:42 PM4/15/21

to dot...@googlegroups.com

Oh, sorry. I see now you are using docker. My answer was concerning a standard install. I'm not familiar with docker installs.

Nathan I. Keiter | Lead Network Applications Programmer | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>
________________________________

From: Nathan Keiter
Sent: Thursday, April 15, 2021 1:41 PM
To: dot...@googlegroups.com
Subject: Re: [dotcms] [docker] How to expose "felix/load"

Giacomo Petillo

unread,

Apr 15, 2021, 1:48:35 PM4/15/21

to dotCMS User Group

Yes Nathan,

this is a docker for a dev environment.

This page explains what I'm trying to do: https://dotcms.com/docs/latest/docker-image-configuration-options#dotCMSDockerVolumeMapping

...docker-compose.yml...

- ${PWD}/data/logs:/srv/dotserver/tomcat-8.5.32/logs:rw

....

...

dotcms_1 | 2021-04-15 17:42:30,716 FelixStartLevel DEBUG RollingFileAppender$Builder(fileName="/srv/dotserver/tomcat-8.5.32/logs/dotcms-sitesearch.log", filePattern="/srv/dotserver/tomcat-8.5.32/logs/archive/dotcms-sitesearch-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=20971520)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=10, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{dd/MM/yy HH:mm:ss:SSS z}] %5p %c{2}: %m%n), name="SITESEARCH-FILE", Configuration(/srv/dotserver/tomcat-8.5.32/webapps/ROOT/WEB-INF/log4j/log4j2.xml), Filter=null)
dotcms_1 | 2021-04-15 17:42:30,718 FelixStartLevel ERROR Unable to create file /srv/dotserver/tomcat-8.5.32/logs/dotcms-sitesearch.log java.io.IOException: Permission denied

...

Falzone, Chris

unread,

Apr 19, 2021, 8:50:18 AM4/19/21

to dot...@googlegroups.com

So a couple things ...

1) Just map the /data/shared directory -- you need to do this anyway to persist the assets. Your felix/load directory will be under this. On my instance I mapping everything to a local directory: /data
'/data/shared:/data/shared'
2) If you ever need to adjust the osgi-extra.conf (this is the exported packages button in the UI), you need to map this file directly. Note in this example I am using a different version of tomcat than comes with stock dotCMS, more on that later. What I did was spin the container locally, bashed into the container and then copied the base file to my server ...
'/data/felix/osgi-extra.conf:/srv/dotserver/tomcat-8.5.60/webapps/ROOT/WEB-INF/felix/osgi-extra.conf'
3) Permissions ... I attempted to run the stock dotCMS container, but I ran into multiple issues with running not as the root user. Maybe you can figure it out, but after a few weeks or so of mucking around with it and then finally working with dotCMS support (MOTIV now), we ended up switching to MOTIV's version of the dotCMS container. This makes sure we are not running the container as root, which is not allowed by our security guys. That being said, on my servers, I had to create the mounted directories and set their permissions very specifically. This also has the added benefit of regular java and tomcat upgrades, which is something else our security team is very interested in (read: demands).

Here is my full puppet module if it helps. I am sure you could do the same in whatever orchestration you are using. I am going to hide/delete some things as I am not sure if the MOTIV containers are exactly free for all to use without their support package. We are using version 20.11.1. I also provide this knowing that I am very green at containerized deployments and I might not be "doing it right". We are running docker on ec2 instances with the goal of transitioning to kubernetes after our upgrade is finished. This is also not running a production workload yet, still working out the kinks of the upgrade and transition from running the bare release on 5.0.3 to running docker containers on 20.11.1.

# Installs all Docker containers to run dotCMS
class dotcms(
  String $cms_heap_size      = '3g',
  String $cms_version        = '20.11.1-master-v2.0',
  String $config_repo_branch = 'master', # only relevant to our static config plugin
  String $data_device        = '/dev/nvme1n1',
  String $db_host            = $facts['dotcms']['db_host'], # grabs this from SSM Params
  String $db_password        = $facts['dotcms']['db_password'],
  String $dotcms_user        = '1000000000', # this is the magic user id used in the container
  String $license_file       = '',
  String $role               = 'dev', # this is something our custom plugins use to know what environment they are in.
) {
  data_volume { '/data':  # this just creates the EBS Volume Mount
    device => $data_device,
    owner  => $dotcms_user,
    group  => $dotcms_user,
  } -> file { # Create the docker mount directories
    default:
      ensure => directory,
      owner  => $dotcms_user,
      group  => $dotcms_user,
    ;
    '/data/plugins':;
    '/data/felix':;
    '/data/license':;
    '/data/shared':;
    '/data/local':;
    '/data/esdata':
      owner => '1000',
      group => '1000',
  } -> file {
    default:
      ensure => directory,
      owner  => $dotcms_user,
      group  => $dotcms_user,
    ;
    '/data/plugins/static':; # separate .../static due to Puppet not being able to do mkdir -p
    '/data/plugins/osgi':;
    '/data/shared/assets':; # The container seems to want to create this as root unless we do it first.
  } -> file { # pipe in the license and osgi-extra files
    default:
      ensure => present,
      owner  => $dotcms_user,
      group  => $dotcms_user,
      mode   => '0644',
    ;
    '/data/felix/osgi-extra.conf':
      source => 'puppet:///modules/dotcms/osgi-extra.conf',
    ;
    '/data/license/license.zip':
      source => "puppet:///modules/dotcms/${license_file}",
    ;
  } -> exec { 'clone-dotcms-config':
    # This is our Configuration plugin, it just has the settings for the cache and the like
    cwd     => '/data/plugins/static',
    command => '/usr/bin/git clone g...@github.com:aquent/dotcms.config.git',
    unless  => '/usr/bin/test -d /data/plugins/static/dotcms.config',
  } -> exec { 'checkout-dotcms-config-branch':
    cwd     => '/data/plugins/static/dotcms.config',
    command => "/usr/bin/git checkout ${config_repo_branch} ; /usr/bin/git pull",
  } -> exec { 'clone-dotcms-hotfix-19895':
    # TODO:  This should be removed after an upgrade to dotCMS v21.2+
    # See the github issue for details, broken upgrade only from 5.0.3 - https://github.com/dotCMS/core/issues/19895 
    cwd     => '/data/plugins/static',
    command => '/usr/bin/git clone g...@github.com:aquent/dotcms-hotfix-GIT-19895.git',
    unless  => '/usr/bin/test -d /data/plugins/static/dotcms-hotfix-GIT-19895',
  } -> exec { 'checkout-dotcms-hotfix-19895-branch':
    # TODO:  This should be removed after an upgrade to dotCMS v21.2+
    cwd     => '/data/plugins/static/dotcms-hotfix-GIT-19895',
    command => '/usr/bin/git checkout main ; /usr/bin/git pull',
  } -> docker_network { 'dotcms':
    ensure => present,
  } -> docker::run { 'elasticsearch':
    image    => 'docker.elastic.co/elasticsearch/elasticsearch:7.9.1',
    net      => ['dotcms'],
    username => '1000:1000',
    env      => [
      'cluster.name=elastic-cluster',
      'discovery.type=single-node',
      'data=',
      'bootstrap.memory_lock=true',
      'ES_JAVA_OPTS=-Xmx1G -Duser.timezone=US/Eastern',
    ],
    volumes  => [
      # Persist Index Data -- Note that mapping it in the dotCMS instance didn't help.  
      '/data/esdata:/usr/share/elasticsearch/data',
    ]
  } -> docker::run { 'dotcms':
    image    => "(Insert MOTIV's Container Repo here):${cms_version}",
    net      => ['dotcms'],
    ports    => '8080:8080',
    username => "${dotcms_user}:0", # GID has to be 0 to make /data mount work correctly, possibly an ec2/puppet thing, but this got me going.
    env      => [
      "CMS_HEAP_SIZE=${cms_heap_size}",
      'CMS_JAVA_OPTS=-Duser.timezone=US/Eastern',
      "PROVIDER_DB_DNSNAME=${db_host}",
      "PROVIDER_DB_PASSWORD=${db_password}",
      'PROVIDER_ELASTICSEARCH_ENDPOINTS=http://elasticsearch:9200',
      'ES_ADMIN_PASSWORD=pass',
      "SERVER_ROLE=${role}",
    ],
    volumes  => [
      # Persist Assets and felix/load
      '/data/shared:/data/shared',
      # Persist Elasticsearch Index -- didn't work in my experience
      '/data/local:/data/local',
      # Static plugins - deployed after restart
      '/data/plugins/static:/plugins/static',
      # OSGI Plugins - deployed after restart
      '/data/plugins/osgi:/plugins/osgi',
      # Persist and set the OSGI Exported Packages File
      '/data/felix/osgi-extra.conf:/srv/dotserver/tomcat-8.5.60/webapps/ROOT/WEB-INF/felix/osgi-extra.conf',
      # Apply License Pack
      '/data/license/license.zip:/data/shared/assets/license.zip'
    ]
  }
}

Hope that helps!

--
http://dotcms.com - Open Source Java Content Management

---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/7e221767-b2eb-4745-9cf5-465510e2487cn%40googlegroups.com.