transit bundle generation error
Robert Marianski
data. I'm following along the instructions on the wiki page:
http://code.google.com/p/onebusaway/wiki/ImportingTransitData
I'm using the gtfs that the mta recently put up, June 18, 2010, which
you can currently find here:
http://mta.info/developers/data/nyct/subway/google_transit.zip
It's a database error: Data too long for column 'description' at row 1. The mta
route descriptions in the routes.txt gtfs file are verbose. Any pointers on
what to do next? Should I dig into trying to make the column bigger or change
the type?
I'm using mysql, configured like on the wiki page
(org.hibernate.dialect.MySQL5Dialect)
I've pasted the error below. I'm more than happy to provide the bundle.xml
config file I'm using and/or the entire output if you think that'll help.
Thanks,
Robert
== ROUTE_COLLECTIONS =====>
2010-06-24 16:18:04,332 WARN [JDBCExceptionReporter.java:100] : SQL Error: 0, SQLState: 22001
2010-06-24 16:18:04,333 ERROR [JDBCExceptionReporter.java:101] : Data truncation: Data too long for column 'description' at row 1
2010-06-24 16:18:04,336 ERROR [AbstractFlushingEventListener.java:324] : Could not synchronize database state with session
org.hibernate.exception.DataException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:100)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:655)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy21.run(Unknown Source)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreator.run(FederatedTransitDataBundleCreator.java:77)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreatorMain.run(FederatedTransitDataBundleCreatorMain.java:59)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreatorMain.main(FederatedTransitDataBundleCreatorMain.java:30)
Caused by: java.sql.BatchUpdateException: Data truncation: Data too long for column 'description' at row 1
at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:652)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 18 more
org.springframework.dao.DataIntegrityViolationException: Could not execute JDBC batch update; nested exception is org.hibernate.exception.DataException: Could not execute JDBC batch update
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:639)
at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:789)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:663)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy21.run(Unknown Source)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreator.run(FederatedTransitDataBundleCreator.java:77)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreatorMain.run(FederatedTransitDataBundleCreatorMain.java:59)
at org.onebusaway.transit_data_federation.bundle.FederatedTransitDataBundleCreatorMain.main(FederatedTransitDataBundleCreatorMain.java:30)
Caused by: org.hibernate.exception.DataException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:100)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:266)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:655)
... 10 more
Caused by: java.sql.BatchUpdateException: Data truncation: Data too long for column 'description' at row 1
at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:652)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 18 more
2010-06-24 16:18:04,362 INFO [AbstractApplicationContext.java:815] : Closing org.springframework.context.support.GenericApplicationContext@8d80be3: display name [org.springframework.context.support.GenericApplicationContext@8d80be3]; startup date [Thu Jun 24 16:09:50 EDT 2010]; root of context hierarchy
2010-06-24 16:18:04,363 INFO [DefaultSingletonBeanRegistry.java:421] : Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1f8166e5: defining beans [org.springframework.context.annotation.internalPersistenceAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.aop.config.internalAutoProxyCreator,defaultPropertyPlaceholderConfigurer,ehCacheConfiguration,cacheManager,cacheableKeyFactories,cacheableInterceptor,org.springframework.beans.factory.config.MethodInvokingFactoryBean#0,hibernateProperties,hibernateAnnotatedClasses,hibernateMappingLocations,sessionFactory,transactionManager,org.springframework.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,mbeanServer,jmxAttributeSource,jmxNamingStategy,jmxInfoAssembler,mbeanExporter,ehCacheMBeanRegistration,springHibernateGtfsRelationalDaoImpl,gtfsHibernateMappingLocations,gtfsHibernateCacheableKeyFactories,gtfsBundle,bundle,org.onebusaway.container.spring.PropertyOverrideConfigurer#0,transitDataFederationHibernateAnnotatedClasses,transitDataFederationHibernateMappingLocations,transitDataFederationDaoImpl,transitDataFederationMutableDaoImpl,extendedGtfsRelationalDaoImpl,dataSource,bundleCreatorHibernateProperties,generateRouteCollectionsTask,generateRouteSearchIndexTask,generateStopSearchIndexTask,generateNarrativesTask,calendarServiceDataTask,osmProvider,generateWalkPlannerGraphTask,generateTripPlannerGraphTask,generateStopTransfersTripPlannerGraphTask,modifications,gtfs-bundles,gtfs_nyc,bundleCreatorOverrideDiffHibernateProperties]; root of factory hierarchy
2010-06-24 16:18:04,371 INFO [MBeanExporter.java:444] : Unregistering JMX-exposed beans on shutdown
2010-06-24 16:18:04,372 INFO [AbstractSessionFactoryBean.java:246] : Closing Hibernate SessionFactory
2010-06-24 16:18:04,372 INFO [SessionFactoryImpl.java:805] : closing
2010-06-24 16:18:04,375 INFO [EhCacheManagerFactoryBean.java:89] : Shutting down EHCache CacheManager
Brian Ferris
Robert Marianski
> I'm travelling out of the county at the moment, so it may be a week or so
> before I can take a look at this in detail.
>
> Brian
Sure, no worries. Enjoy your travels!
In the mean time, I'll poke around a little to see what I can turn up.
Robert
Nicholas Bergson-Shilcock
<rmari...@openplans.org> wrote:
> On Thu, Jun 24, 2010 at 10:00:21PM +0000, Brian Ferris wrote:
>> I'm travelling out of the county at the moment, so it may be a week or so
>> before I can take a look at this in detail.
>>
>> Brian
>
> Sure, no worries. Enjoy your travels!
>
> In the mean time, I'll poke around a little to see what I can turn up.
I don't know if it's any help, but I ran into a similar issue with the
route description being greater than 255 chars. Brian fixed it in
onebusaway-gtfs-hibernate-1.1.0.
See: http://code.google.com/p/onebusaway/issues/detail?id=246&can=1&q=reporter:nicholasbs
Even if it's not the same issue, looking at his patch might give an
idea of how to fix the problem you're facing.
-N
Robert Marianski
> On Thu, Jun 24, 2010 at 3:17 PM, Robert Marianski
> <rmari...@openplans.org> wrote:
> > On Thu, Jun 24, 2010 at 10:00:21PM +0000, Brian Ferris wrote:
> >> I'm travelling out of the county at the moment, so it may be a week or so
> >> before I can take a look at this in detail.
> >>
> >> Brian
> >
> > Sure, no worries. Enjoy your travels!
> >
> > In the mean time, I'll poke around a little to see what I can turn up.
>
> I don't know if it's any help, but I ran into a similar issue with the
> route description being greater than 255 chars. Brian fixed it in
> onebusaway-gtfs-hibernate-1.1.0.
>
> See: http://code.google.com/p/onebusaway/issues/detail?id=246&can=1&q=reporter:nicholasbs
>
> Even if it's not the same issue, looking at his patch might give an
> idea of how to fix the problem you're facing.
>
> -N
Thanks for the reference. This looks like the commit with that fix:
http://code.google.com/p/onebusaway/source/detail?r=1441
My issue turned out to be similar. There's a RouteCollection object
(where_route_collections is the table name) and it looks like it was
using the default mapping for a String, a varchar(255). In this case, it
looks like the mapping was being obtained from annotations and not from
xml. I added an explicit lob column type annotation, which fixed the
error for me. I'm not sure if that's the best way to do it or what all
the implications are here.
I've attached a patch with the changes I made. I added a separate test
for the change, but it's got a bit of duplication in there to test the
description as a separate test. I'm not too familiar with
hibernate/spring/testing so apologies if that's not the best way to do
it. Maybe it would have been better to fold it into one test too.