[Plone-Users] File object migration - how to change id of new object programatically? (2.5.5)
John Schinnerer
Plone 2.5.5/zope 2.9.8 on linux.
I am writing a script to migrate ExFile objects to ATFile objects.
I have everything essential working except changing the id of the newly
created ATFile from a temporary one used for creation to a final one
(the final id for the ATFile needs to be the same as the id of the
ExFile it is replacing).
When I try to change the id of the ATFile to be that of the ExFile it is
replacing, I get either an error that doesn't make sense or a file with
an object URL and 'get file' link URL that don't match (which causes
strange behavior in plone UI). Code with extra logging statements etc.
stripped out and comments added is below. This code will get me almost
there - I end up with an ATFile that is "the same" as the ExFile it
replaces, except that the id has the "ATFile-" prefix and that needs to
be changed to be the id of the original ExFile. I am stumped on how to
get that to work right.
thanks,
John S.
---------------
def convert_files(self):
"""
Migrate ExFile to ATFile, replacing former with latter in same
location
"""
catalog = getToolByName(self, 'portal_catalog')
portal_url=getToolByName(self,'portal_url')
portal_object=portal_url.getPortalObject()
results =
catalog.searchResults(portal_type='PloneExFile',sort_on='modified',
sort_order='reverse')
for fObj in results:
o_path=fObj.getPath()
# For now just look in a subsection while testing
if o_path.startswith('/mydomain.com/migration-test/'):
# get the actual object
o=fObj.getObject()
# get essential stuff to migrate
original_id=o.id
title=o.title
description=o.Description()
primary_field=o.getPrimaryField()
file_content=o.getFile()
owner=o.getOwner()
author=o.Creator()
creation_date=o.CreationDate()
effective_date=o.getEffectiveDate()
last_modified=o.ModificationDate()
### create an ATFile object with same properties as EXFile
# put it in the same location - get container of ExFile object
container = o.aq_inner.aq_parent
# create a temp id for new object to avoid id conflict with
existing ExFile
new_obj_id="ATFile-"+original_id
if container:
container.invokeFactory(type_name="File", id=new_obj_id)
new_file = container[new_obj_id]
new_file.setTitle(title)
new_file.setDescription(description)
new_file.setCreators(author)
new_file.setEffectiveDate(effective_date)
new_file.setFile(file_content)
# finalize new file creation...
new_file.processForm()
# ...publish it
container.portal_workflow.doActionFor(new_file, 'publish')
# delete existing ExFile
container.manage_delObjects([original_id])
# PROBLEM - trying to use this to set id...
new_file.setId(original_id)
# gives following error traceback when I run the script:
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.ExternalMethod.ExternalMethod, line 225, in __call__
- __traceback_info__: ((<PloneSite at /hcrc.info>,), {}, None)
Module
/opt/Plone-2.5.5/zeocluster/client1/Extensions/convert_files.py, line
91, in convert_files
Module Products.Archetypes.BaseObject, line 235, in setId
Module Products.ATContentTypes.lib.autosort, line 149, in
manage_renameObject
Module Products.Archetypes.OrderedBaseFolder, line 209, in
manage_renameObject
Module OFS.CopySupport, line 343, in manage_renameObject
Copy Error:
The action against the
ATFile-Trans-consultant-SC-presentation-10-4-07.pdf object could not be
carried out. One of the following constraints caused the problem: The
object does not support this operation OR The currently logged-in user
does not have the Copy or Move permission respective to the object.
# I don't understand why perms would be an issue since I
just created the object I am trying to change id on; and the API docs
appear to say that ATFile does support setId().
# trying this...
new_file.id = original_id
# or this...
setattr(new_file, 'id', original_id)
# results in an ATFile where the object URL has the
"ATFile-" prefix but the link to the file itself ('get file' link in
index_html) does NOT have the "ATFile-" prefix, and so gives a 'not
found', and the object gives errors when trying to delete via plone UI
and other strange behavior.
----------------
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
jo...@eco-living.net
http://eco-living.net
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Plone-Users mailing list
Plone...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-users
ajung
Use manage_renameObject() API
-aj
--
View this message in context: http://n2.nabble.com/File-object-migration-how-to-change-id-of-new-object-programatically-2-5-5-tp4174543p4174569.html
Sent from the General Questions mailing list archive at Nabble.com.
John Schinnerer
I will try that...
I had looked into some of the "manage_" API but that one was for some
reason in my blind spot. I appreciate the illumination.
cheers,
John S.
John Schinnerer
Working on this again...see original code posting (below).
Unfortunately using manage_renameObject:
container.manage_renameObject(new_obj_id, original_id)
gives me the same error (in original code posting, below) as trying to use:
new_file.setId(original_id)
setId() apparently just calls manage_renameObject() on the container -
The traceback is identical except that the line:
Module Products.Archetypes.BaseObject, line 235, in setId
is not present (since the call is directly on the container, not through
the file object).
So I'm still stumped by this error message, and how to do something as
seemingly simple as rename a file object programmatically.
I just created the object in the same external method - how could I not
have copy/move permissions?
Or, how could the object not support the operation?
thanks,
John S.
....
ajung
You check the permissions yourself by looking yourself within the ZMI (acces
tab)
or you take the debugger and check where and why the expection is raised
within
the CopyManager code.
-aj
John Schinnerer wrote:
>
>
> I just created the object in the same external method - how could I not
> have copy/move permissions?
> Or, how could the object not support the operation?
>
>
>
--
View this message in context: http://n2.nabble.com/File-object-migration-how-to-change-id-of-new-object-programatically-2-5-5-tp4174543p4207582.html
Sent from the General Questions mailing list archive at Nabble.com.
------------------------------------------------------------------------------
John Schinnerer
ajung wrote:
> You check the permissions yourself by looking yourself within the ZMI (acces
> tab)
Error occurs on first iteration, so no ATFile is actually created on
which perms can be checked in ZMI.
> or you take the debugger and check where and why the expection is raised
> within the CopyManager code.
Where is in the traceback - line 343 of OFS.copySupport (raise...):
if not ob.cb_isMoveable():
raise CopyError, eNotSupported % escape(id)
...that is, these checks:
def cb_isMoveable(self):
# Is object moveable? Returns 0 or 1
if not (hasattr(self, '_canCopy') and self._canCopy(1)):
return 0
if hasattr(self, '_p_jar') and self._p_jar is None:
return 0
try: n=aq_parent(aq_inner(self))._reserved_names
except: n=()
if absattr(self.id) in n:
return 0
if not self.cb_userHasCopyOrMovePermission():
return 0
return 1
def cb_userHasCopyOrMovePermission(self):
if getSecurityManager().checkPermission('Copy or Move', self):
return 1
So presumably one of the above conditions fails. I will find out which
one and report back for additional help - regardless of which one fails,
I don't understand why or how to fix it in my own code.
I have been following programmatic object creation tutorials/howtos for
creating the ATFile object. None say anything about needing to
explicitly set things like _canCopy for the object or copy/move perms
for the object's creator. It seems like this would be default/automatic,
but perhaps not.
thanks,
John S.
> -aj
>
>
> John Schinnerer wrote:
>>
>> I just created the object in the same external method - how could I not
>> have copy/move permissions?
>> Or, how could the object not support the operation?
>>
>>
>>
>
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
jo...@eco-living.net
http://eco-living.net
------------------------------------------------------------------------------
John Schinnerer
To recap - I am writing a script for migrating file object content from
ExFile to ATFile objects. Problem is in changing the id of the ATFile to
the same id as the old ExFile (after deleting the ExFile so id does not
conflict).
My script code and full traceback are further below.
The traceback shows the error on attempt to rename is thrown in line 343
of OFS.copySupport. That is on a call to cb_isMoveable(), which looks
like this:
> def cb_isMoveable(self):
> # Is object moveable? Returns 0 or 1
> if not (hasattr(self, '_canCopy') and self._canCopy(1)):
> return 0
> if hasattr(self, '_p_jar') and self._p_jar is None:
> return 0
> try: n=aq_parent(aq_inner(self))._reserved_names
> except: n=()
> if absattr(self.id) in n:
> return 0
> if not self.cb_userHasCopyOrMovePermission():
> return 0
> return 1
>
> def cb_userHasCopyOrMovePermission(self):
> if getSecurityManager().checkPermission('Copy or Move', self):
> return 1
I tested all these conditions. The one that fails is this one:
if hasattr(self, '_p_jar') and self._p_jar is None:
return 0
I have no idea what this _p_jar test is for or about, or why it would
fail in this situation, or how to fix this problem and be able to change
the id of the ATFile. Any help appreciated!
thanks,
John S.
--------------
def convert_files(self):
"""
Migrate ExFile to ATFile, replacing former with latter in same
location
"""
catalog = getToolByName(self, 'portal_catalog')
portal_url=getToolByName(self,'portal_url')
portal_object=portal_url.getPortalObject()
results =
catalog.searchResults(portal_type='PloneExFile',sort_on='modified',
sort_order='reverse')
for fObj in results:
o_path=fObj.getPath()
# For now just look in a subsection while testing
if o_path.startswith('/mydomain.com/migration-test/'):
# get the actual object
o=fObj.getObject()
# get essential stuff to migrate
original_id=o.id
title=o.title
description=o.Description()
file_content=o.getFile()
author=o.Creator()
effective_date=o.getEffectiveDate()
### create an ATFile object with same properties as EXFile
# put it in the same location - get container of ExFile object
container = o.aq_inner.aq_parent
# create a temp id for new object to avoid id conflict with
existing ExFile
new_obj_id="ATFile-"+original_id
if container:
container.invokeFactory(type_name="File", id=new_obj_id)
new_file = container[new_obj_id]
new_file.setTitle(title)
new_file.setDescription(description)
new_file.setCreators(author)
new_file.setEffectiveDate(effective_date)
new_file.setFile(file_content)
# finalize new file creation...
new_file.processForm()
# ...publish it
container.portal_workflow.doActionFor(new_file, 'publish')
# delete existing ExFile
container.manage_delObjects([original_id])
# PROBLEM - trying to change id to original ExFile id:
new_file.setId(original_id)
# gives following error traceback:
------------------
------------------
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.ExternalMethod.ExternalMethod, line 225, in __call__
- __traceback_info__: ((<PloneSite at /hcrc.info>,), {}, None)
Module
/opt/Plone-2.5.5/zeocluster/client1/Extensions/convert_files.py, line
91, in convert_files
Module Products.Archetypes.BaseObject, line 235, in setId
Module Products.ATContentTypes.lib.autosort, line 149, in
manage_renameObject
Module Products.Archetypes.OrderedBaseFolder, line 209, in
manage_renameObject
Module OFS.CopySupport, line 343, in manage_renameObject
Copy Error:
The action against the
ATFile-Trans-consultant-SC-presentation-10-4-07.pdf object could not be
carried out. One of the following constraints caused the problem: The
object does not support this operation OR The currently logged-in user
does not have the Copy or Move permission respective to the object.
--------------------------
--------------------------
ajung
You might try using
transaction.savepoint(1)
before calling the CopyManager API.
Andreas
--
View this message in context: http://n2.nabble.com/File-object-migration-how-to-change-id-of-new-object-programatically-2-5-5-tp4174543p4222889.html
Sent from the General Questions mailing list archive at Nabble.com.
------------------------------------------------------------------------------
John Schinnerer
This works! The _p_jar test now passes, so the rename works. Thanks!
I would like to understand what this is about, if anyone is willing to
give a brief explanation of _p_jar and transaction.savepoint() or point
me to where I can learn about this. I have googled around but everything
I find assumes I know other things I don't about some other code
somewhere in between...
thanks again,
John S.
...
John Schinnerer
Now that the rename is working, I have hit the next (hopefully final)
issue on this script for ExFile to ATFile migration.
Before I delete the ExFile, the newly created ATFile and preview appear
to work correctly.
After I delete the ExFile, the ATFile 'get file' link returns the below
error. It doesn't matter if I delete the ExFile programmatically or via
the Plone UI - same result.
So it seems that getFile() and setFile() do not work as I think - that
is, get the actual file content and set the actual file content. Instead
(with ExFiles anyhow?) there seems to be some kind of reference to the
(ExFile) file on the filesystem that is gotten and set...?
The traceback shows that file.py index_html calls base.py index_html
which then calls FileSystemStorage.py index_html, which throws error on
final call to some iterator function which does not find file.
I don't understand why these calls end up going to FileSystemStorage.py
when the object is an ATFile. I think it means that the actual file
content somehow did not get copied, and the ATFile is somehow
referencing(?) where it used to be.
More importantly, I don't know how to copy the actual file content from
ExFile to ATFile objects, given that getFile() and setFile() don't work
the way I tried to use them. I don't see anything in the APIs that seems
more relevant than getFile() and setFile(). I have been looking into the
code, which has mostly confused me more.
Any assistance appreciated!
thanks,
John S.
----------
Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.ATContentTypes.content.file, line 118, in index_html
Module Products.ATContentTypes.content.base, line 404, in index_html
Module Products.FileSystemStorage.FileSystemStorage, line 262, in
index_html
Module ZPublisher.Iterators, line 39, in __init__
IOError: [Errno 2] No such file or directory:
'/path-to-fss-directory/fss/more-path-to/migration-test/Trans-consultant-SC-presentation-10-4-07.pdf/Trans-consultant-SC-presentation-10-4-07.pdf'
--------------
>> -------------- current script code with rename working -----
>> transaction.savepoint(1)
>> container.manage_renameObject(new_obj_id, original_id)
ajung
A ZODB transaction savepoint basically ensures that the "recorded" changes
within the current transaction are stored on disk (instead of RAM - also
reducing RAM usage). As a side-effect all affected objects (derived from
Persistent?) not having a ZODB connection attribute (_p_jar) will get one.
In general you usually never have to deal with _p_jar. I have myself only
one application where I am dealing myself with ZODB connections in the
context of linked PData chains (as used in OFS.File/Image).
This explanation is more kind of high-level - I am not personally very much
aware of the ZODB internals when it comes to savepoints and connections.
Andreas
--
View this message in context: http://n2.nabble.com/File-object-migration-how-to-change-id-of-new-object-programatically-2-5-5-tp4174543p4231207.html
Sent from the General Questions mailing list archive at Nabble.com.
------------------------------------------------------------------------------
John Schinnerer
OK, I've found a solution for this. Posting for the record in case
anyone else not already savvy with these manipulations has a similar need.
Thanks again to Ian Duplisse who offered code many months ago to
reattach "lost" FSS files to ExFile objects that use FSS.
In that same code was most of what I needed for getting the file itself
migrated.
Starts the same - find the FSS file on the file system.
Then, upload it to the new ATFile object (instead of reattaching it to
the ExFile object as Ian's original code did).
Essential code snippet:
--------------
# look for ExFile fss.cfg file first
# RESTORE is set to FSS data path on filesystem
filepath=RESTORE+fObj.getPath()
fss=filepath + '/fss.cfg'
if not os.path.exists(fss):
logger.log(logging.ERROR,'FSS file %s not found'%(fss,))
continue
# now get the actual filename from the fss file
config=ConfigParser.RawConfigParser()
config.read(fss)
filename=config.get('FILENAME','file')
filepath=filepath + '/' + filename
# if file not found, log error
if not os.path.exists(filepath):
logger.log(logging.ERROR,'file %s not found'%(filepath,))
# file was found - upload it to new ATFile object
else:
stat = os.stat(filepath)
# if not a file, log error
if not S_ISREG(stat[ST_MODE]):
logger.log(logging.ERROR,'file %s not a file'%(filepath,))
# file found and is a file
else:
#set up all the upload parameters
fp = open(filepath,"r")
env = {'REQUEST_METHOD':'PUT'}
headers = {'content-disposition':'attachment;
filename=%s'%(filename)}
fs = FieldStorage(fp=fp, environ=env, headers=headers)
upload=FileUpload(fs)
# it's going to the new ATFile
field=new_file.getField('file')
kwargs={'field':'file'}
# here it goes
field.set(new_file,upload,**kwargs)
fp.close()
# index it
catalog.indexObject(new_file)
--------------
happy new year!
John S.
--