App won't start after adding uploadfs and s3 configuration

[Joseph Bader]

Hey There,

I'm new to Apostrophe and have been loving it. Right now I'm trying to set up uploadfs to push uploaded media to S3. Unfortunately though, when I add the uploadfs block to app.js, starting the app hangs at "Ran initial Browserify asset bundling."  If I take out the uploadfs block, everything starts fine. I'm wondering if I'm missing something here.

Here are my config files (sensitive info removed). These are the only files I've modified to get uploadfs to work:

app.js

var site = require('apostrophe-site')({

  // This line is required and allows apostrophe-site to use require() and manage our NPM modules for us.
  root: module,
  shortName: 'www',
  hostName: 'www',
  title: 'www',
  sessionSecret: 'apostrophe sandbox demo party',
  adminPassword: 'demo',
  address: '0.0.0.0',
  port: 3000,

  // Force a2 to prefix all of its URLs. It still
  // listens on its own port, but you can configure
  // your reverse proxy to send it traffic only
  // for URLs with this prefix. With this option
  // "/" becomes a 404, which is supposed to happen!

  // prefix: '/test',

  // If true, new tags can only be added by admins accessing
  // the tag editor modal via the admin bar. Sometimes useful
  // if your folksonomy has gotten completely out of hand
  lockTags: false,

  // Give users a chance to log in if they attempt to visit a page
  // which requires login
  secondChanceLogin: true,

  locals:  require('./lib/locals.js'),

  // you can define lockups for areas here

  // Here we define what page templates we have and what they will be called in the Page Types menu.

  // For html templates, the 'name' property refers to the filename in ./views/pages, e.g. 'default'
  // refers to '/views/pages/default.html'.

  // The name property can also refer to a module, in the case of 'blog', 'map', 'events', etc.

  pages: {
    types: [
      { name: 'default', label: 'Default (Two Column)' },
      { name: 'home', label: 'Home Page' },
      { name: 'contact', label: 'Contact' }
    ]
  },

  // These are the modules we want to bring into the project.
  modules: {
    // Styles required by the new editor, must go FIRST
    'apostrophe-editor-2': {},
    'apostrophe-ui-2': {},
    'apostrophe-browserify': {
      files: ['./public/js/modules/_site.js']
    },
    uploadfs: {
      backend: 's3',
      // Get your credentials at aws.amazon.com
      secret: 'SECRET_REMOVED',
      key: 'KEY_REMOVED',
      // You need to create your bucket first before using it here
      // Go to aws.amazon.com
      bucket: 'BUCKET_NAME_REMOVED',
      // I recommend creating your buckets in a region with
      // read-after-write consistency (not us-standard)
      region: 'us-west-2'
      // Required if you use copyImageIn
      // tempPath: __dirname + '/temp',
      // imageSizes: [
      //   {
      //    name: 'small',
      // width: 320,
      //    height: 320
      //  },
      //  {
      //    name: 'medium',
      //    width: 640,
      //    height: 640
      //  },
      //  {
      //    name: 'large',
      //    width: 1140,
      //    height: 1140
      //  }
      // ],
      // Render up to 4 image sizes at once. Note this means 4 at once per call
      // to copyImageIn. There is currently no built-in throttling of multiple calls to
      // copyImageIn
      // parallel: 4
    }
  },

  // These are assets we want to push to the browser.
  // The scripts array contains the names of JS files in /public/js,
  // while stylesheets contains the names of LESS files in /public/css
  assets: {
    scripts: [ 'vendor/interact', 'site-compiled' ],
    stylesheets: ['site']
  }
});

package.json

{
  "name": "www",
  "version": "0.0.1",
  "description": "DESCRIPTION_REMOVED",
  "main": "app.js",
  "scripts": {
    "start": "node app.js"
  },
  "repository": {
    "type": "git",
    "url": ""
  },
  "author": "AUTHOR_REMOVED",
  "license": "BSD",
  "dependencies": {
    "apostrophe-browserify": "^0.5.5",
    "apostrophe-editor-2": "0.5.x",
    "apostrophe-site": "0.5.x",
    "apostrophe-ui-2": "0.5.x",
    "uploadfs":  "1.1.x"
  }
}

[Tom Boutell]

Hmm, that's strange. Are you absolutely sure that if you just comment out the uploadfs block, and make *no* other changes, it starts up fine?

--
You received this message because you are subscribed to the Google Groups "apostrophenow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apostropheno...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

THOMAS BOUTELL, DEV & OPS

P'UNK AVENUE | (215) 755-1330  |  punkave.com

[Tom Boutell]

I see that you specified that actually. Let me try it in the sandbox.

[Tom Boutell]

Oh, I see the problem.

You have added it in the "modules" section of app.js. You don't want to do that because it is not an apostrophe module (although yes, uploadfs is a module in the npm sense, which could be confusing...)

You want it one level higher, as a peer of modules, not inside it.

It is interesting that a nonexistent module being configured hangs rather than crashing in an informative fashion. We should look at that (:

[Joseph Bader]

OK great! Thanks for clarifying. It seems to be working now. Perhaps the "modules" section should be called "apoModules" or something to that effect?

[Tom Boutell]

In A2 0.6, the concept of an Apostrophe module (as opposed to an npm module) will be pushed very much to the foreground, and there will only be a tiny handful of options that don't belong in "modules" somewhere, so this will be less of an issue.

[Joseph Bader]

Excellent!

Back to the original problem: Now that I have the app running, it isn't uploading to S3 when I add an image through the "Add content" menu. I enabled logging in S3 but nothing is generated there. I just get a NoSuchKey error when trying to view the image.

[Tom Boutell]

Just did this successfully in the a2 sandbox.

My guess is that your bucket was created in the wrong region. It must match your "region" property, you must have one, and it may not be "us-standard". (us-standard is actually a CDN of sorts, it pushes files to both west and east coasts, and for that reason it doesn't promise you can access what you just uploaded right away, which is a problem for a CMS. (: )

If it doesn't match a redirect status code comes back and knox can't deal with it.

[Tom Boutell]

FYI, while I was investigating this, I decided to upgrade knox (the S3 API module we use in uploadfs). We were using an ancient release. The unit tests all continue to pass with the latest knox, there don't seem to be any API changes that matter for us. So it shouldn't impact anyone adversely.

[Joseph Bader]

I believe my region is correct. Amazon has changed their naming in their web interface though and I'm not sure how this impacts configuration.

I selected Oregon when setting up S3 and in the configuration I set it as us-west-2.

--
You received this message because you are subscribed to a topic in the Google Groups "apostrophenow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/apostrophenow/cC9VbAsUazM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to apostropheno...@googlegroups.com.

[Tom Boutell]

Not sure then, but this definitely worked for me. How about sharing your latest iteration of app.js?

[Tom Boutell]

It could be that the credentials you're using don't actually have write access for that particular bucket.

[Joseph Bader]

Is there any way to get verbose feedback or logs from uploadfs that would contain the response from Amazon?

[Tom Boutell]

Not baked in, you could start doing some console.log work in s3.js.

[Joseph Bader]

I'll try that. In the meantime, here is a terminal error I'm getting when I try to upload an image with the Apostrophe admin interface:

.write(string, encoding, offset, length) is deprecated. Use write(string[, offset[, length]][, encoding]) instead.

[Joseph Bader]

I ran trace-deprecation. I hope this helps:

Joes-MBP:www joe$ node --trace-deprecation app.js

Ran initial Browserify asset bundling.

Listening on 0.0.0.0:3000

Trace: .write(string, encoding, offset, length) is deprecated. Use write(string[, offset[, length]][, encoding]) instead.

    at Buffer.write (buffer.js:400:17)

    at MultipartParser.initWithBoundary (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/node_modules/formidable/lib/multipart_parser.js:61:17)

    at IncomingForm._initMultipart (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/node_modules/formidable/lib/incoming_form.js:308:10)

    at IncomingForm._parseContentType (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/node_modules/formidable/lib/incoming_form.js:250:12)

    at IncomingForm.writeHeaders (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/node_modules/formidable/lib/incoming_form.js:129:8)

    at IncomingForm.parse (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/node_modules/formidable/lib/incoming_form.js:97:8)

    at /Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/lib/middleware/multipart.js:126:12

    at noop (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/lib/middleware/multipart.js:22:3)

    at multipart (/Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/lib/middleware/multipart.js:76:5)

    at /Users/joe/Working/cnverg/www/node_modules/apostrophe-site/node_modules/appy/node_modules/express/node_modules/connect/lib/middleware/bodyParser.js:57:9