Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file

[Kaan Soral]

Hi

I've recently switched from windows to mac, mainly for the hardware/software stability, just managed to start working again with the dev_appserver after 2 days, my old blobstore images are working in the new setup, however the newer cloud storage images aren't working
I haven't yet dived deep enough into the inner workings to decipher and solve the issue, I would really like some help, if anyone around is fluent in the blobstore/gcs/dev_appserver inner workings

Basically all encoded_gs_file: urls seem to redirect to the /n/ncoded_gs_file folder on the storage path
the storage path has a lot of folders, like a b C D .. 1 2 .. _-.. etc. and only "n" seems to have the ncoded_gs_file empty file, so it's good up to this point, I'm not sure how things are supposed to work
I verify that the url/key is correct, because when you change the url, the error changes too, it just doesn't seem to find clodstorage images

Here is the full traceback

Traceback (most recent call last):
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/tools/devappserver2/admin/admin_request_handler.py", line 96, in dispatch
    super(AdminRequestHandler, self).dispatch()
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/Users/kaan/Desktop/APPENGINE/sdk/lib/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/tools/devappserver2/admin/blobstore_viewer.py", line 93, in get
    self._display_blob(blob_info, display)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/tools/devappserver2/admin/blobstore_viewer.py", line 125, in _display_blob
    self.response.write(reader.read())
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/ext/blobstore/blobstore.py", line 861, in read
    self.__fill_buffer(size)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/ext/blobstore/blobstore.py", line 833, in __fill_buffer
    self.__position + read_size - 1)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/ext/blobstore/blobstore.py", line 681, in fetch_data
    return rpc.get_result()
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/blobstore/blobstore.py", line 234, in _get_result_hook
    rpc.check_success()
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
    self.__rpc.CheckSuccess()
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
    self.request, self.response)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/apiproxy_stub.py", line 129, in MakeSyncCall
    method(request, response, request_id)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/blobstore/blobstore_stub.py", line 374, in _Dynamic_FetchData
    blob_file = self.__storage.OpenBlob(blobkey)
  File "/Users/kaan/Desktop/APPENGINE/sdk/google/appengine/api/blobstore/file_blob_storage.py", line 149, in OpenBlob
    return _local_open(self._FileForBlob(blob_key), 'rb')
IOError: [Errno 2] No such file or directory: '/Users/kaan/project/blobstore/dev~theapp/n/ncoded_gs_file:Z2VvYmlyZGVsZW1lbnRzL2NvbW1lbnRfdXBsb2Fkcy9mYWtlLVJfOG5hbGdyc3VoQkppakhick5jVmc9PQ=='

And the request paths look like this: /_ah/img/encoded_gs_file:Z2VvYmlyZGVsZW1lbnRzL3NoYXJlX3VwbG9hZHMvZmFrZS1kNk9qLW45dzh1ZVhFMkxTQlQzZUlnPT0==s293

dev_appserver argument paths are correct, I'm using the latest .zip python sdk

the datastore entity for the gcs file is getting fetched, I assume, since wrong keys produce different errors, but there is an issue after that operation

[Kaan Soral]

Off topic, if there is anyone around thinking of making the switch, the sqlite datastore seems to be compatible with mac os, the blobstore is compatible, the only issue seems to be this, for a direct switch
macports > py27-lxml + py27-pil + using the python2.7 of macports solves the dev_appserver requirements
Generally speaking, I wouldn't suggest switching to mac for no reason, I switched because I wasn't able to easily move my win7 setup to a new hardware, and my laptop was glitching for years, mac os's time machine / easy restore features sounded great for future hardware issues
So far I've been spending my time trying to select a new IDE / trying to solve font/svn issues - retina screens are great, however regular external lcd's render fonts really bad, so a 4K screen halved into retina resolution might be a must for those who are obsessed with IDE fonts, long story short, it's cost after cost after disappointment after cost ... but I'm hoping it's probably going to pay off in the near future

[Kaan Soral]

Well, it turns out the gcs files were never written to the blobstore directory, I'm hunting where they are, I have a hunch they are in the db.rdbms, hoping they are in a separate temp directory instead (my last blobstore update was from 05/2013)
However on the mac dev_appserver, they are actually physical files such as "ncoded_gs_file:Z2VvYmlyZGVsZW1lbnRzL2ZpbGVzL2Zha2UtaWU2VjNaVHhjWUg0TXpwUjNsUnoyQT09" in the "n" directory
so the mac sdk is looking for them in the right place (mac sdk-wise) however they are not there

Any ideas?

[PK]

Hi Kaan,

do you have the latest app engine GCS SDK? The docs say that the SDK is in googlecode but it seems it has moved in github here. 

This past week I switched my development environment from native mac (and native linux on another machine) to native docker. I need to write up about the experience but one of the obstacles was that tests and functionality around GCS stopped working. Then I updated my GCS client from the latest git repository and I am back in business on that front.

I am also not sure from your write-up, are you trying to move some  data already in app engine SDK during the transition?

PK
http://www.gae123.com

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[Kaan Soral]

Hi PK

That is exactly what I'm trying to do, thanks for the reply

The SDK/GCS serving part seems to be separate from the upload/library logic, If I'm not mistaken, so my issue is currently the serving part (offtopic, I seem to be manually modifying the cloudstorage library for the serving url part, don't remember why, I probably need to re-visit that logic when I restore old images because I can upload and verify new images, however their serving urls seem to be faulty for me, concatenating port and hostname without a ":" etc ... )

I've just uploaded a 2MB photo to my old setup, checking whether it's loaded as a blob into the .rdbms, however that wasn't the case, yet the photo is missing, I suspect I need to find the location of that photo, move all photos, and continue from there (my current theory)

Yet I'm unable to find that path, I always had --storage_path / --blobstore_path's set -- yet the photo is missing

I'm puzzled, it doesn't seem to be in the AppData / Temp folder either, although I couldn't verify it 100%, the old directories seem to be untouched

TLDR, I'm missing my GCS images, can't find them, can't transport them to my new setup

[PK]

Sorry I cannot help there, I hope there is a way that is not too much trouble. 

I have never trusted compatibility of the SDK format on the disk, I assume it can break in a moment’s notice between releases even in the same platform. So I have an import/export tool that I use for "data portability” purposes. I use it both with the SDK and production. I highly recommend you move to a similar approach.

Furthermore, I really do not use development to store data I care about. I have a tool that generates data for testing purposes and once in a while I blow up all my SDK data and generate a new system with the tool.

PK
http://www.gae123.com

[Kaan Soral]

Well checking the blobstore's "n" folder now, I see that the 0byte/empty "ncoded_gs_file" updated when I uploaded the photo

So I'm guessing there is a file access issue, the images are likely on another dimension :)

I've checked the folder with cygwin too, no trace of the images, I'm guessing it's the "/" in the images names

They seem to be named like "ncoded_gs_file/Z2VvYmlyZGVsZW1lbnRzL2ZpbGVzL2Zha2UtenFTLWhEcHFUclhKZ1N2RFROLS1Ydz09" etc...

Going to look for a way to look for them / access them now, haven't verified their existence yet

[PK]

wrt “off topic” As I have stated before, when I start working on a new project I always try to understand what is the developers' primary development environment and use the same or similar. This avoids too much bleeding especially for bleeding edge technologies. So I never used GAE on windows.

My first apengine environment was on a windows notebook but the notebook was hosting a VM with the GAE SDK/editor etc. So it was really Linux. Having two windows systems on one machine is a pain so I started thinking what to do. Native development on bare metal Linux was not an option for a number of reasons but I had started seeing a lot of people with MacBooks.

So, three years ago I made the decision and switched to a Mac. I liked the GUI, the multiple spaces---that allowed to visually contain different projects and personal work—etc. I also liked that there is UNIX underneath, I can always open a terminal window and ssh anywhere etc. At that time I figured out how to set up the SDK to run natively on the mac and have been living happily since then. I wrote about that here: http://www.gae123.com/articles/dpwf/#idp140641236359312 Using virtualenv was a key insight at the time.

For the past week I spent a lot of time—definitely much more than I had allotted to— to figure out how to run the SDK and my app engine dev environment on docker. My primary motivation has been that I want to upgrade to the latest MacOS 10.10 doing a clean install. However, I have been concerned that the native maces dev. environment will stop working and the last thing I want is to have to weather a production crisis with an unstable dev. environment.

I think I have accomplished my goal but I will need one more week to make sure I have not missed anything. I will do a write-up when I feel comfortable with the new approach.

PK
http://www.gae123.com

[Kaan Soral]

Thanks anyway

That is actually a great approach, I considered that a couple of times myself, I use the same approach with other platforms, but compared to them, GAE/SDK is extremely stable, the GCS issue is a shame, whoever implemented it should be ashamed, other than that, things seem pretty persistent across operating systems and versions

I've been maintaining the SDK setup I'm currently working on for more than 2 years, so It kind of has a lot of sentimental value, a lot of memories, that's why I'm very keen on restoring it

Worst case scenario, I might ping/upload each file manually/programmatically, from old to new, however I hope it doesn't come to that

It seems like this issue affects all windows sdk users, so please pitch in with your experience

[Kaan Soral]

Really looking forward to that article :)

I guess you could also backup, upgrade and restore a mac, if things doesn't go your way with an upgrade, however downgrading the os itself would be a challenge I guess (haven't tried/researched the process myself, I hope the installation restores the os too, instead of restoring and upgrading, in a case like that, things would be really easy in that case)

I was looking forward to having a firewall to play with, on unix/macos, however I really hate the pf firewall, there is also no simple articles online that accomplishes simple tasks with pf etc, so I'm guessing 10.10 update might be irritating for previous version users

[Kaan Soral]

Here is the final solution to the issue, I suggest the file_blob_storage modification and renaming the existing files for everyone, it's extremely clumsy to utilize special characters in file names (macos seems to handle "/" in the file name, don't know why "/" is used instead of ":" as windows, the modification renames the file name, existing urls and references keep working, yet a refactoring is needed for the files themselves)

file_blob_storage.py modification

  def _FileForBlob(self, blob_key):

    blob_key = self._BlobKey(blob_key)

    return os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:].replace("/",";").replace(":",";"))

This ensures that files have ";" instead of "/" or ":"

To pull out the special files from the windows SDK

1) Go to blob_image.py's serve_image function and pass/comment out the datastore.Get(key) check (because we're going to ping files from the serving url scheme, this ensures every file is served from that url scheme)

2) Modify file_blob_storage.py to copy the accessed files to a new location (from the old SDK)

def _FileForBlob(self, blob_key):
    blob_key = self._BlobKey(blob_key); handle=False
    path=os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])
    if path.find("ncoded")!=-1: handle=True
    if handle and not kaan_marked.get(blob_key):
        logging.info("YEY %s"%blob_key)
        kaan_marked[blob_key]=True
        blob_file = self._FileForBlob(blob_key)
        input = _local_open(blob_file, 'rb')
        output= _local_open("C:/GCS/"+str(blob_key)[1:].replace(":",";"),'wb')
        try:
          while True:
            block = input.read(1 << 20)
            if not block:
                break
            output.write(block)
        finally:
          output.close()
    logging.info(path)
    return os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])

3) Ping each image, I decided to ping them by browsing the "__GsFileInfo__"'s from the Datastore VIewer, couldn't access them easily programatically

gg=document.getElementsByTagName("td"); for(var i=0;i<gg.length;i++) if(gg[i].innerHTML.indexOf("ncoded")!=-1) { var image = document.createElement("img"); image.src = "http://localhost/_ah/img/"+gg[i].innerHTML.trim(); document.body.appendChild(image); }

Then put the images back to the old or new "n" folder

I think all windows SDK users should do this, I don't know how macos handles "/" in the filenames, yet I would suggest everyone to fix the file access manually

I've added this to my list of manual sdk modifications, currently I have 9 manual modifications, sad ... :)

[Pravin Singh]