Django-compressor manifest.json problem on Heroku

[Guy Tamir]

Hey all,

I'm trying to add django-compressor into my app.

I'm deploying to Heroku.

This is the code in my Settings file

AWS_QUERYSTRING_AUTH = False
AWS_STORAGE_BUCKET_NAME = os.environ.get('BUCKET_NAME')

conn = S3Connection(AWS_ACCESS_KEY, AWS_SECRET_KEY,host="s3-us-west-1.amazonaws.com")
STATIC_ROOT = 'static'
COMPRESS_ROOT = 'static'



COMPRESS_CSS_FILTERS = ['compressor.filters.yuglify.YUglifyCSSFilter','compressor.filters.cssmin.CSSMinFilter']
COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter'] #,'compressor.filters.jsmin.SlimItFilter'
COMPRESS_OUTPUT_DIR = 'compressed'
COMPRESS_CSS_BACKEND = 'django_compressor.css.CssCompressor'
COMPRESS_JS_BACKEND = 'django_compressor.js.JsCompressor'
COMPRESS_OFFLINE = True
COMPRESS_ENABLED = True


if DEBUG:
    STATIC_URL = '/static/'
else:
    AWS_S3_CUSTOM_DOMAIN = os.environ.get('COMPRESS_URL') # cdn.test.com
    STATIC_URL =  'http://' + AWS_S3_CUSTOM_DOMAIN + '/'
    STATICFILES_STORAGE = 'myapp.CachedS3BotoStorage.CachedS3BotoStorage'
    COMPRESS_STORAGE = 'myapp.CachedS3BotoStorage.CachedS3BotoStorage'


COMPRESS_URL = STATIC_URL

I ran the compress command locally and then deployed to Heroku.

When I see my deployed pages I can see in the source files that the browser is using the compressed css and js file like expected.

My problem is that when I see my logs from Heroku, each time that I refresh a page, it seems like Heroku goes and fetches the manifest.json file from my S3 bucket.

This obviously slows down the requests..

Logs from Heroku:

2014-12-10T16:01:14.087225+00:00 app[web.1]: DEBUG:boto:Token: None
2014-12-10T16:01:14.086183+00:00 app[web.1]: DEBUG:boto:auth
_path=/test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.137618+00:00 heroku[router]: at=info method=GET path="/login/" host=test-dollar-social.herokuapp.com request_id=bede0d95-e9b6-4453-95b3-438606fc7e62 fwd="93.172.158.119" dyno=web.1 connect=1ms service=194ms status=200 bytes=13248
2014-12-10T16:01:14.051410+00:00 app[web.1]: DEBUG:boto:Using secret key provided by client.
2014-12-10T16:01:14.051322+00:00 app[web.1]: DEBUG:boto:Using access key provided by client.
2014-12-10T16:01:14.051990+00:00 app[web.1]: DEBUG:boto:path=/compressed/manifest.json
2014-12-10T16:01:14.052126+00:00 app[web.1]: DEBUG:boto:auth_path=/test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.052405+00:00 app[web.1]: DEBUG:boto:Path: /compressed/manifest.json
2014-12-10T16:01:14.052316+00:00 app[web.1]: DEBUG:boto:Method: HEAD
2014-12-10T16:01:14.052490+00:00 app[web.1]: DEBUG:boto:Data:
2014-12-10T16:01:14.052605+00:00 app[web.1]: DEBUG:boto:Headers: {}
2014-12-10T16:01:14.052734+00:00 app[web.1]: DEBUG:boto:Host: test-dollarsocial.s3.amazonaws.com
2014-12-10T16:01:14.052828+00:00 app[web.1]: DEBUG:boto:Params: {}
2014-12-10T16:01:14.052970+00:00 app[web.1]: DEBUG:boto:establishing HTTPS connection: host=test-dollarsocial.s3.amazonaws.com, kwargs={'timeout': 70}
2014-12-10T16:01:14.053082+00:00 app[web.1]: DEBUG:boto:Token: None
2014-12-10T16:01:14.053224+00:00 app[web.1]: DEBUG:boto:StringToSign:
2014-12-10T16:01:14.053227+00:00 app[web.1]: HEAD
2014-12-10T16:01:14.053228+00:00 app[web.1]:
2014-12-10T16:01:14.053230+00:00 app[web.1]:
2014-12-10T16:01:14.053231+00:00 app[web.1]: Wed, 10 Dec 2014 16:01:14 GMT
2014-12-10T16:01:14.053233+00:00 app[web.1]: /test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.085443+00:00 app[web.1]: DEBUG:boto:path=/compressed/manifest.json
2014-12-10T16:01:14.086354+00:00 app[web.1]: DEBUG:boto:Method: HEAD
2014-12-10T16:01:14.086473+00:00 app[web.1]: DEBUG:boto:Path: /compressed/manifest.json
2014-12-10T16:01:14.086576+00:00 app[web.1]: DEBUG:boto:Data:
2014-12-10T16:01:14.086685+00:00 app[web.1]: DEBUG:boto:Headers: {}
2014-12-10T16:01:14.053367+00:00 app[web.1]: DEBUG:boto:Signature:
2014-12-10T16:01:14.053370+00:00 app[web.1]: AWS XXXXXXXXXXXX:XXXXXXXXXXXX
2014-12-10T16:01:14.086965+00:00 app[web.1]: DEBUG:boto:Params: {}
2014-12-10T16:01:14.086802+00:00 app[web.1]: DEBUG:boto:Host: test-dollarsocial.s3.amazonaws.com
2014-12-10T16:01:14.087538+00:00 app[web.1]: DEBUG:boto:StringToSign:
2014-12-10T16:01:14.087540+00:00 app[web.1]: HEAD
2014-12-10T16:01:14.087541+00:00 app[web.1]:
2014-12-10T16:01:14.087543+00:00 app[web.1]:
2014-12-10T16:01:14.087567+00:00 app[web.1]: /test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.087918+00:00 app[web.1]: DEBUG:boto:Signature:
2014-12-10T16:01:14.087921+00:00 app[web.1]: AWS XXXXXXXXXXXXX
:XXXXXXXXXXXXXXX
2014-12-10T16:01:14.099581+00:00 app[web.1]: DEBUG:boto:path=/compressed/manifest.json
2014-12-10T16:01:14.099629+00:00 app[web.1]: DEBUG:boto:auth_path=/test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.099784+00:00 app[web.1]: DEBUG:boto:Method: GET
2014-12-10T16:01:14.099870+00:00 app[web.1]: DEBUG:boto:Path: /compressed/manifest.json
2014-12-10T16:01:14.099956+00:00 app[web.1]: DEBUG:boto:Data:
2014-12-10T16:01:14.087544+00:00 app[web.1]: Wed, 10 Dec 2014 16:01:14 GMT
2014-12-10T16:01:14.100045+00:00 app[web.1]: DEBUG:boto:Headers: {}
2014-12-10T16:01:14.100129+00:00 app[web.1]: DEBUG:boto:Host: test-dollarsocial.s3.amazonaws.com
2014-12-10T16:01:14.100214+00:00 app[web.1]: DEBUG:boto:Params: {}
2014-12-10T16:01:14.100394+00:00 app[web.1]: DEBUG:boto:Token: None
2014-12-10T16:01:14.100532+00:00 app[web.1]: DEBUG:boto:StringToSign:
2014-12-10T16:01:14.100535+00:00 app[web.1]: GET
2014-12-10T16:01:14.100537+00:00 app[web.1]:
2014-12-10T16:01:14.100538+00:00 app[web.1]:
2014-12-10T16:01:14.100540+00:00 app[web.1]: Wed, 10 Dec 2014 16:01:14 GMT
2014-12-10T16:01:14.100541+00:00 app[web.1]: /test-dollarsocial/compressed/manifest.json
2014-12-10T16:01:14.100695+00:00 app[web.1]: DEBUG:boto:Signature:
2014-12-10T16:01:14.100697+00:00 app[web.1]: AWS XXXXXXXXXXXX:XXXXXXXXXX

What I expected to happen is for heroku to cache the manifest file and not to fetch it each request..

Any ideas what I'm doing wrong?

Best,

Guy

[Collin Anderson]

Hi,

Do you have django caching configured? If not, then caching will only be per-process and each process will need to fetch the information. (I'm assuming CachedS3BotoStorage uses django's cache.)

Collin

[Guy Tamir]

I haven't configured django caching, how do I do that?

Also - CachedS3BotoStorage is this:

from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage


from django.core.files.base import File


class CachedS3BotoStorage(S3BotoStorage):
    """
    S3 storage backend that saves the files locally, too.
    """
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class("compressor.storage.CompressorFileStorage")()


    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

...

[Collin Anderson]

Hi Guy,

I haven't used django_compressor, but it might help to configure COMPRESS_OFFLINE_MANIFEST.

Collin

[Guy Tamir]

Hi,

The COMPRESS_OFFLINE_MANIFEST is to set the name of the manifest, default is manifest.json

this issue isn't that the server can't find the manifest, it finds the correct one but downloads (I think) it every request..

...