Error with `update_assets` for vendor NPM asset (Owl Carousel)

[Zachary Trabookis]

Getting the following error when running `update_assets` on a native install of Hawthorn.

edxapp@server:~/edx-platform$ paver update_assets cms --settings=aws --theme-dirs=/edx/app/ew-themes/edx-platform/ --themes educateworkforce

We are using latest 2.3.4 version of Owl Carousel 2 and include this in the vendor files (edx-platform/common/static/common/{css|js}/vendor/) from the Owl /dist directory.

https://www.npmjs.com/package/owl.carousel

python manage.py cms --settings=aws collectstatic --ignore "fixtures" --ignore "karma_*.js" --ignore "spec" --ignore "spec_helpers" --ignore "spec-helpers" --ignore "xmodule_js" --ignore "geoip" --ignore "sass" --noinput > /dev/null

2019-07-18 14:07:25,708 INFO 21874 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125

Traceback (most recent call last):

  File "manage.py", line 121, in <module>

    execute_from_command_line([sys.argv[0]] + django_args)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line

    utility.execute()

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute

    output = self.handle(*args, **options)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle

    collected = self.collect()

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect

    for original_path, processed_path, processed in processor:

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/require/storage.py", line 172, in post_process

    for path in super_class.post_process(paths, dry_run, **options):

  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/theming/storage.py", line 292, in post_process

    for name, hashed_name, processed in super_class.post_process(paths.copy(), dry_run, **options):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pipeline/storage.py", line 39, in post_process

    for name, hashed_name, processed in super_class.post_process(paths.copy(), dry_run, **options):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 246, in post_process

    for name, hashed_name, processed, _ in self._post_process(paths, adjustable_paths, hashed_files):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 303, in _post_process

    content = pattern.sub(converter, content)

  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/theming/storage.py", line 225, in converter

    force=True, hashed_files=hashed_files,

  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/theming/storage.py", line 180, in _url

    return super(ThemeCachedFilesMixin, self)._url(hashed_name_func, processed_asset_name, force, hashed_files)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 141, in _url

    hashed_name = hashed_name_func(*args)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 352, in _stored_name

    cache_name = self.clean_name(self.hashed_name(name))

  File "/edx/app/edxapp/edx-platform/openedx/core/storage.py", line 20, in hashed_name

    out = super(PipelineForgivingStorage, self).hashed_name(name, content, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 100, in hashed_name

    if not self.exists(filename):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/files/storage.py", line 392, in exists

    return os.path.exists(self.path(name))

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 53, in path

    return super(StaticFilesStorage, self).path(name)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/files/storage.py", line 405, in path

    return safe_join(self.location, name)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/_os.py", line 78, in safe_join

    'component ({})'.format(final_path, base_path))

django.core.exceptions.SuspiciousFileOperation: The joined path (/edx/var/edxapp/staticfiles/common/css/vendor/owl.video.play.png) is located outside of the base path component (/edx/var/edxapp/staticfiles/studio)

Captured Task Output:

---------------------

---> pavelib.assets.update_assets

---> pavelib.prereqs.install_node_prereqs

---> pavelib.assets.process_xmodule_assets

---> pavelib.prereqs.install_python_prereqs

xmodule_assets common/static/xmodule

mkdir_p path('common/static/common/js/vendor')

mkdir_p path('common/static/common/css')

mkdir_p path('common/static/common/css/vendor')

/bin/cp -rf node_modules/backbone.paginator/lib/backbone.paginator.js common/static/common/js/vendor

/bin/cp -rf node_modules/backbone/backbone.js common/static/common/js/vendor

/bin/cp -rf node_modules/bootstrap/dist/js/bootstrap.bundle.js common/static/common/js/vendor

/bin/cp -rf node_modules/hls.js/dist/hls.js common/static/common/js/vendor

/bin/cp -rf node_modules/jquery-migrate/dist/jquery-migrate.js common/static/common/js/vendor

/bin/cp -rf node_modules/jquery.scrollto/jquery.scrollTo.js common/static/common/js/vendor

/bin/cp -rf node_modules/jquery/dist/jquery.js common/static/common/js/vendor

/bin/cp -rf node_modules/moment-timezone/builds/moment-timezone-with-data.js common/static/common/js/vendor

/bin/cp -rf node_modules/moment/min/moment-with-locales.js common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/owl.carousel.js common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/README.md common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/owl.carousel.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/LICENSE common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.carousel.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.video.play.png common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.theme.default.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/ajax-loader.gif common/static/common/js/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.carousel.css common/static/common/css/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.theme.default.css common/static/common/css/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.theme.green.css common/static/common/css/vendor

/bin/cp -rf node_modules/owl.carousel/dist/assets/owl.theme.green.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/picturefill/dist/picturefill.js common/static/common/js/vendor

/bin/cp -rf node_modules/requirejs/require.js common/static/common/js/vendor

/bin/cp -rf node_modules/underscore.string/dist/underscore.string.js common/static/common/js/vendor

/bin/cp -rf node_modules/underscore/underscore.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/accessibilityPolicy.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseHealthCheck.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/assets.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/runtime.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/assets.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/assets.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/common.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18nMessages.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/accessibilityPolicy.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/editImageModal.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/editImageModal.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/common.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/accessibilityPolicy.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseOutlineHealthCheck.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/editImageModal.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseHealthCheck.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseOutlineHealthCheck.min.css common/static/common/css/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18nMessages.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/runtime.min.js common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/common.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseOutlineHealthCheck.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/courseHealthCheck.min.js.map common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18n/messages/fr.json common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18n/messages/zh_CN.json common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18n/messages/ar.json common/static/common/js/vendor

/bin/cp -rf node_modules/@edx/studio-frontend/dist/i18n/messages/es_419.json common/static/common/js/vendor

/bin/cp -rf node_modules/which-country/index.js common/static/common/js/vendor

/bin/cp -rf node_modules/sinon/pkg/sinon.js common/static/common/js/vendor

/bin/cp -rf node_modules/squirejs/src/Squire.js common/static/common/js/vendor

---> pavelib.assets.webpack

python manage.py lms --settings=aws print_setting STATIC_ROOT 2>/dev/null

python manage.py cms --settings=aws print_setting STATIC_ROOT 2>/dev/null

python manage.py lms --settings=aws print_setting WEBPACK_CONFIG_PATH 2>/dev/null

NODE_ENV=production STATIC_ROOT_LMS=/edx/var/edxapp/staticfiles STATIC_ROOT_CMS=/edx/var/edxapp/staticfiles/studio $(npm bin)/webpack --config=webpack.prod.config.js

python manage.py cms --settings=aws compile_sass cms  --theme-dirs /edx/app/ew-themes/edx-platform/ --themes educateworkforce arm belizeproject bmw caregiver dlsc greermade mahcc meep nccte nwgrc ptc scelect scjru ts

python manage.py cms --settings=aws collectstatic --ignore "fixtures" --ignore "karma_*.js" --ignore "spec" --ignore "spec_helpers" --ignore "spec-helpers" --ignore "xmodule_js" --ignore "geoip" --ignore "sass" --noinput > /dev/null

Build failed running pavelib.assets.update_assets: Subprocess return code: 1

This seems to be the issue with the CSS file /edx-platform/common/static/common/css/vendor/owl.carousel.css looking for relative owl.video.play.png file. 

https://github.com/OwlCarousel2/OwlCarousel2/blob/2.3.4/dist/assets/owl.carousel.css#L160

.owl-carousel .owl-video-play-icon { background: url("owl.video.play.png") no-repeat; }

When collectstatic runs the file actually resides at /edx/var/edxapp/staticfiles/common/js/vendor/owl.video.play.png rather than the /css/vendor path in staticfiles. This is because anything in the vendor files that's not .css or .css.map extension gets placed in the NPM_JS_VENDOR_DIRECTORY directory path.

https://github.com/edx/edx-platform/blob/open-release/hawthorn.master/pavelib/assets.py#L602-L610

Also I have specified a /dist/ path in NPM_INSTALLED_LIBRARIES so all files get copied across. What if a vendor had the same filename as another vendor library? Wouldn't overrides occur. Has anyone looked into creating separate directories per vendor within the destination path to avoid this conflict?
https://github.com/edx/edx-platform/blob/open-release/hawthorn.master/pavelib/assets.py#L53-L69

[Ernesto Sanchez]

Hi,

why do not you move the theme directory to /edx/app/edxapp/theme and try again?

[Zachary Trabookis]

Ernesto,

This is not a theme issue. It's more of including javascript vendor library using NPM. The process_npm_assets is where the NPM vendor files get copied up to `edx-platform/common/static/common/{css|js}/vendor` directories according to the file extension. Really trying to figure our edX's policy on including new javascript libraries in the platform. 

Zach