Migration 1.9.5.1 -> 2.5.6: Assets file structure

[Simone Adami]

Hi,
I started e manual site migration of a website written with dotCMS 1.9.5.1 Community and I have to upgrade it to 2.5.6 Community.

All 1.9.5.1 assets follow this structure:

dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/cocorita.jpg

 

in 2.5.6 it's:

dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/fileAsset/cocorita.jpg
dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/metaData/content

This "dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/metaData/content" binary file what does it do? I mean, it's a binary file that seems to be useless, I tried to remove it and after save/publish, cache clean, reindex and dotCMS restart the image is still showing in a frontend test page. It seems that its presence is not necessary: but is there something I miss?

I'm thinking at this strategy:
1. manually migrate all the DB, understanting all changes (it's a 36GB DB..)
2. copy and paste 1.9.5.1 assets to 2.5.6 assets folder.
3. write a bash script (since we're on linux) that makes for every file asset for example:

mkdir dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/fileAsset/ &&
mkdir dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/metaData/ &&
mv dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/cocorita.jpg dotCMS/assets/a/2/a26cd27b-c6f9-488a-8cef-d9cf2089b130/fileAsset/;

4. enjoy the new migrated site.

What problems I could face if I don't create the right content binary file?

Thanks,

Simone Adami

[Jason Tesser]

The binary images are only stored there you may have new versions of that image or it may be archived but that binary is needed to pull the actual image

--
You received this message because you are subscribed to the Google Groups "Dotcms User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.
To post to this group, send email to dot...@googlegroups.com.
Visit this group at http://groups.google.com/group/dotcms.
For more options, visit https://groups.google.com/d/optout.

[Brent Griffin]

Simone,

Keep in mind that in addition to the content cache in memory, that there may also be a persistent cache stored on the local hard drive.  This can make it appear that certain things are unnecessary until the next time you manually flush the cache.

Brent Griffin

Sr. Java Architect

Director of Quality Assurance

dotCMS

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+unsubscribe@googlegroups.com.

[Simone Adami]

Hi,
in the upgrade guide to 2.0 it's written to only copy the assets folder: is this an obsolete statement or is it still valid?
Because my problem is to port 100+ GB 1.9.5.1 assets to 2.5.6: if dotCMS 2.5.6 understands also the 1.9.5.1's directory structure it's ok, if not I need a strategy.

Simone

To unsubscribe from this group and stop receiving emails from it, send an email to dotcms+un...@googlegroups.com.

[Falzone, Chris]

If you are performing an upgrade, I would follow the steps outlined in that doc.  There is no need to modify the folder structure of the assets folder for 1.9.5.1 to 2.5.6.

The new file structure is for files as content, but when you upgrade your files are still legacy files.  After the upgrade, if you want to convert them, you can use webdav to save them to a folder on your computer, delete them from dotCMS, and then re-upload them (webdav can help make that process a tad easier).  

--

Christopher Falzone

Interactive Developer

A Q U E N T

Digital, Creative, and Marketing Talent

aquent.com

cfal...@aquent.com

[Simone Adami]

Ok Chris, thanks for the help, but what about the DB?
I see that a lot of tables has different columns, for example contentlet, folder, template in 1.9.5.1 have no identifier, but in 2.5.6 there they are.

Some weeks ago I followed the step by step guide you mentioned for a 1.9.5.1 demo test site to upgrade it to 2.5.4, didn't go well, I'm sorry I can't tell what the error message was.

[Falzone, Chris]

The startup upgrade scripts "should" handle converting the DB.  However, as in most cases, the data is not in a good place to begin with and the upgrade does no go smoothly.

You have to tail the logs while the upgrade classes are running, determine where and why it failed.  Once you know that, you can build what I like to call a pre-upgrade script that run on the database before you run the upgrade.  So the process roughly goes like:

1.  Make a copy of the db

2.  Copy the assets

3.  Startup, tail the log and watch for errors

4.  Stop, restore the db

5.  Figure out a fix for the error and add it to the 

[Falzone, Chris]

Sorry last message hit send too soon ..

1. Make a copy of the DB
2. Copy assets
3. Run the pre-upgrade sql script 
4. Startup run upgrade, tail log
5. stop, restore db
6. add a fix to the pre-upgrade script
7. repeat until a clean upgrade.

It can be quite a daunting task.  I would recommend contacting dotCMS support as they are really experienced in running these kinds of upgrades.

To put this in contrast, we decided not to upgrade but to completely rebuild the site on the new version.  I believe this is a cleaner and easier path.  

I wish you luck!

[Simone Adami]

Ok Chris,
thanks again for the support.

Simone

[yasirk...@gmail.com]

i wish to migrate this site. guide me how??

http://www.ehealthy.co/

[Simone Adami]

Hi all,
I'm debugging a problem in my "import" of from an old 1.9.5.1 to a new 2.5.6 installation. Actually my problem remain in making the old assets work in the 2.5.6 "Files (Legacy)" portlet.
The point is: I'm most probably doing something wrong and since this version I'm working with is the zipped release, I'd like to download a 2.5.6 git source version of dotcms to be able to temporally make some additional logs and breakpoints, but from github it doesn't seem present any tag anymore.

From dotcms.log, there's the following exception:

[24/09/14 11:58:41:873 CEST] ERROR action.ViewFilesAction: Exception e =Versionable is null
com.dotmarketing.business.DotStateException: Versionable is null
    at com.dotmarketing.business.IdentifierAPIImpl.find(IdentifierAPIImpl.java:86)
    at com.dotmarketing.factories.WebAssetFactory.getAssetsAndPermissions(WebAssetFactory.java:2090)
    at com.dotmarketing.portal.struts.DotPortletAction._viewWebAssets(DotPortletAction.java:676)
    at com.dotmarketing.portal.struts.DotPortletAction._viewWebAssets(DotPortletAction.java:577)
    at com.dotmarketing.portlets.files.action.ViewFilesAction.render(ViewFilesAction.java:42)
    at com.liferay.portal.struts.PortletAction.execute(PortletAction.java:66)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
    at com.liferay.portal.struts.PortletRequestProcessor.processActionPerform(PortletRequestProcessor.java:192)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
    at com.liferay.portal.struts.PortletRequestProcessor.process(PortletRequestProcessor.java:91)
    at com.liferay.portlet.StrutsPortlet.include(StrutsPortlet.java:148)
    at com.liferay.portlet.StrutsPortlet.doView(StrutsPortlet.java:100)
    at javax.portlet.GenericPortlet.doDispatch(GenericPortlet.java:235)
    at javax.portlet.GenericPortlet.render(GenericPortlet.java:163)
    at com.liferay.portlet.CachePortlet.render(CachePortlet.java:159)
    at org.apache.jsp.html.portal.layout_jsp._jspService(layout_jsp.java:450)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
    at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
    at org.apache.jsp.html.portal.layout_005fportal_jsp._jspService(layout_005fportal_jsp.java:2594)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
    at com.liferay.portal.struts.StrutsUtil.forward(StrutsUtil.java:72)
    at com.liferay.portal.struts.PortalRequestProcessor.doForward(PortalRequestProcessor.java:172)
    at com.dotmarketing.struts.PortalRequestProcessor.doForward(PortalRequestProcessor.java:55)
    at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:237)
    at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:300)
    at com.oroad.stxx.plugin.StxxTilesRequestProcessor.processForwardConfig(StxxTilesRequestProcessor.java:134)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231)
    at com.liferay.portal.struts.PortalRequestProcessor.process(PortalRequestProcessor.java:158)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
    at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:397)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at com.liferay.portal.servlet.MainServlet.callParentService(MainServlet.java:275)
    at com.liferay.portal.servlet.MainServlet.service(MainServlet.java:501)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.liferay.filters.secure.SecureFilter.doFilter(SecureFilter.java:135)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:404)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.dotmarketing.filters.CMSFilter.doFilter(CMSFilter.java:126)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.dotmarketing.filters.AutoLoginFilter.doFilter(AutoLoginFilter.java:61)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.dotmarketing.cms.urlmap.filters.URLMapFilter.doFilter(URLMapFilter.java:85)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.dotmarketing.filters.TimeMachineFilter.doFilter(TimeMachineFilter.java:162)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.dotmarketing.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:146)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:745)
[24/09/14 11:58:41:903 CEST] ERROR servlet.MainServlet: Versionable is null

From mysql log, the query to retrieve the data just before the java exception is:

select  asset.inode as inode,
        asset.file_name as file_name2_,
        asset.file_size as file_size2_,
        asset.width as width2_,
        asset.height as height2_,
        asset.mime_type as mime_type2_,
        asset.author as author2_,
        asset.publish_date as publish_8_2_,
        asset.show_on_menu as show_on_9_2_,
        asset.title as title2_,
        asset.friendly_name as friendl11_2_,
        asset.mod_date as mod_date2_,
        asset.mod_user as mod_user2_,
        asset.sort_order as sort_order2_,
        asset.identifier as identifier2_,
        asset_1_.owner as owner0_,
        asset_1_.idate as idate0_,
        asset_1_.type as type0_
from file_asset asset
inner join inode asset_1_ on asset.inode=asset_1_.inode,
inode inode,
identifier identifier,
fileasset_version_info versioninfo
where   (asset.inode=inode.inode ) and
        (asset.identifier=identifier.id ) and
        (versioninfo.identifier=identifier.id ) and
        (identifier.host_inode='48190c8c-42c4-46af-8d1a-0cd5db894797' ) and
        (asset_1_.type='file_asset' ) and
        (asset.inode=versioninfo.working_inode ) and
        (versioninfo.deleted=0 )
order by  asset.mod_date desc limit 500;

This query finds the right results, the problem is at java-level.

How can I proceed?

 Simone

[Brent Griffin]

If you are looking for 2.5.6 source code, you can find it here:  https://github.com/dotCMS/core-2.x  This repo should have the tag you are looking for.

Brent Griffin

Sr. Java Architect

Director of Quality Assurance

dotCMS

[Simone Adami]

I didn't know the repo has been moved, I kept looking at https://github.com/dotCMS/core.
Thank you Brent.

Simone