Reize images on the fly

[n...@boozt.com]

Hi,

I'm trying to implement a solution were I resize on the fly the images on my Google Storage bucket.

There's this documentation but it's not really clear, at least some of the steps are omitted.

Does anyone has more information or an example of a simple implementation like this?

Thanks

[pj]

https://github.com/muesli/smartcrop

[Kyle Finley]

This is what I'm using:

package graphql

import (

  "fmt"

  "golang.org/x/net/context"

  "google.golang.org/appengine/blobstore"

  aefile "google.golang.org/appengine/file"

  "google.golang.org/appengine/image"

)

func servingURL(ctx context.Context, filename string, size int, crop bool) (string, error) {

  dbn, err := aefile.DefaultBucketName(ctx)

  if err != nil {

    return "", err

  }

  fullPath = fmt.Sprintf("/gs/%s/%s", dbn, filename)

  key, err := blobstore.BlobKeyForFile(ctx, fullPath)

  if err != nil {

    return "", err

  }

  opts := &image.ServingURLOptions{

    Secure: true,

    Size: size,

    Crop: crop,

  }

  url, err := image.ServingURL(ctx, key, opts)

  if err != nil {

    return "", err

  }

  return url.String(), nil

}

// Call it

imageURL, _ := servingURL(ctx, "path/to/image/in/gcs/L2Fwc...", 120, true)

I hope that helps. If you are interested in my upload code please let me know.

Kyle

[nunol...@gmail.com]

Hi Kyle,

The implementation I end up doing is very similar to yours.

I just have two additional steps. One in the beginning that is fetching GS filename from get request url and another in the end that is fetching image from served url and loading it in same request.

I'm not sure if this is the best solution. It's surely not the fastest because I'm doing additional request.

I'm also wondering if I should store served url in database in the moment I upload the image to GS to use it later to load images in my website.

What kind of setup do you have to serve images from served url?

[Kyle Finley]

> I'm also wondering if I should store served url in database in the moment I upload the image to GS to use it later to load images in my website.

Yes, I would recommend saving the serving url after you make the request to appengine/image.ServingURL. Because there is an overhead for creating the url. Once you have the base url you can modify the suffix to serve alternate widths and crop. My actual func includes this:

  if servingURL != "" {

    url := fmt.Sprintf("/gs/%ss%d", servingURL, size)

    if crop {

      url = url + "-c"

    }

    return url, nil

  

To get the serving base url you can do something like this, where you use a known width, and then strip it off:

  dbn, err := file.DefaultBucketName(ctx)

  if err != nil {

    return "", err

  }

  completeFilePath := fmt.Sprintf("/gs/%s/%s", dbn, filePath)

  key, err := blobstore.BlobKeyForFile(ctx, completeFilePath)

  if err != nil {

    return "", err

  }

  opts := &image.ServingURLOptions{

    Secure: true,

    Size: 100,

    Crop: false,

  }

  url, err := image.ServingURL(ctx, key, opts)

  if err != nil {

    return "", err

  }

  baseURL = url.String()

  // Strip the size and crop info

  baseURL = strings.Split(baseURL, "=s100")[0]

  

> What kind of setup do you have to serve images from served url?

The major benefit of using the image api service is that it offloads the actual serving of the image to Google. This saves you processing and instance time. So, I don't serve the image from my Go instance. I just send the url to the client, something like (https://lh3.googleusercontent.com/l0...7t=s1500-c) and let them make the request.

There are times that it makes sense to serve directly from GCS, though, e.g. the image api is too limited or you don't want to use the googleusercontent.com url. Here is the code I use to serve directly from GCS:

func ServeUserContent(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {

  ctx := appengine.NewContext(r)

  dbn, _ := aefile.DefaultBucketName(ctx)

  // XXX I'm not sure why the slash has to be removed here.

  // It must have something to do with httprouter

  path := fmt.Sprintf("/gs/%s/users%s", dbn, ps.ByName("path"))

  blobKey, err := blobstore.BlobKeyForFile(ctx, path)

  if err != nil {

    log.Infof(ctx, `/// ServeUserContent /// err: %#v `, err)

    common.SendError(w, r, http.StatusNotFound, string(http.StatusNotFound))

  }

  seconds := 84000

  w.Header().Add("Cache-Control", fmt.Sprintf("public, max-age=%d", seconds))

  blobstore.Send(w, appengine.BlobKey(blobKey))

}

The func should probably be update to include the "s-maxage" header (https://cloud.google.com/cdn/docs/caching)

If you are serving large files taking advantage of Googles edge cache will save you on instance time.

Please let me know if you have any other questions.

Kyle

[nunol...@gmail.com]

I did couple of tests and serving url was taking up to 1 second. After I have the blobstore url, load it in same request was really fast, just milliseconds.

I end up implanting a simple cache solution using memcache.

Thanks for sharing your approach.

[bertrand...@gmail.com]

It's a really interesting thread, we're using Golang to provide serving URLs as well and we've noticed that when asking for the serving URL right after sending the file to Google Cloud Storage (using a PUT request from the browser directly), there are some cases where the API (Golang API in App Engine) would simply return an API error 8 OBJECT_NOT_FOUND even though the file has been successfully written (as confirmed by the status code of the PUT request).We're suspecting that replication of data across zones might have something to do with that (i.e. a successful write doesn't mean the data will be available where you query it next).

Have you experienced that issue before? If so, any workaround you'd like to share, serving URLs from GCS are great, especially the flexibility they provide for manipulating images on the fly (and are a great companion to the new <picture> tag), but I'd like them to work 100% of the time instead of 99%.

Thanks a lot.

[Kyle Finley]

Hi,

No I didn't run into that error, but I was using the blobstore (https://godoc.org/google.golang.org/appengine/blobstore) for the file upload, where it appears that you are uploading directly to GCS. Maybe that makes a difference? When using the blobstore you can still store the files in GCS, the only downside is you don't have control over last part of the filename.

Kyle

[si...@intesoft.net]

Although it isn't in the appengine Go lib, you can add some pieces to use the image resize and cropping service that is part of AppEngine standard (that you can use from python).

This kind of stuff: https://cloud.google.com/appengine/docs/standard/python/images/#transforming-images