Hi everyone,
We upgraded our firewall yesterday and since then, OBA will run for about an hour and then stop responding. Are there tasks that run once an hour after the server starts? I know it looks for a new bundle, but are there other URLs or other things it is trying to reach and is possibly blocked by?
I tried to build a new bundle to rule out the scheduling data/bundle in case it was corrupted, but I cannot build a bundle either. It definitely times out at the end with the messages below.
Thanks for any help you can provide.
-Devin
2493480 [pool-2-thread-1] DEBUG org.hibernate.jdbc.AbstractBatcher - Executing batch size: 10
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.AbstractBatcher - about to close PreparedStatement (open PreparedStatements: 1, globally: 1)
2493495 [pool-2-thread-1] DEBUG org.springframework.orm.hibernate3.SessionFactoryUtils - Closing Hibernate Session
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.ConnectionManager - releasing JDBC connection [ (open PreparedStatements: 0, globally: 0) (open ResultSets: 0, globally: 0)]
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.ConnectionManager - transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!
2497214 [pool-5-thread-1] INFO org.springframework.beans.factory.support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@b300440: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.internalPersistenceAnnotationProcessor,org.springframework.aop.config.internalAutoProxyCreator,defaultPropertyPlaceholderConfigurer,ehCacheConfiguration,cacheManager,cacheableKeyFactories,cacheableMethodKeyFactoryManager,cacheableMethodManager,cacheableAnnotationInterceptor,org.springframework.beans.factory.config.MethodInvokingFactoryBean#0,hibernateProperties,hibernateAnnotatedClasses,hibernateMappingLocations,sessionFactory,transactionManager,org.springframework.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,mbeanServer,jmxAttributeSource,jmxNamingStategy,jmxInfoAssembler,mbeanExporter,ehCacheMBeanRegistration,bundle,dataSource,org.onebusaway.container.spring.PropertyOverrideConfigurer#0,shapePointsLibrary,agencyBeanServiceImpl,arrivalsAndDeparturesBeanServiceImpl,blockBeanServiceImpl,blockStatusBeanServiceImpl,nearbyStopsBeanServiceImpl,routeBeanServiceImpl,routesBeanServiceImpl,serviceAlertsBeanServiceImpl,shapeBeanServiceImpl,stopBeanServiceImpl,stopScheduleBeanServiceImpl,stopTimeBeanServiceImpl,stopWithArrivalsAndDeparturesBeanServiceImpl,stopsBeanServiceImpl,tripBeanServiceImpl,tripStatusBeanServiceImpl,tripStopTimesBeanServiceImpl,vehicleStatusBeanServiceImpl,blockCalendarServiceImpl,blockGeospatialServiceImpl,blockIndexFactoryServiceImpl,blockIndexServiceImpl,blockRunServiceImpl,blockStatusServiceImpl,scheduledBlockLocationServiceImpl,bundleSearchServiceImpl,realtimeSourceServiceImpl,bundleConfigDao,transitDataServiceExceptionInterceptor,transitDataServiceImpl,narrativeServiceImpl,gtfsRealtimeNegativeArrivalsImpl,gtfsRealtimeServiceImpl,blockLocationHistoryServiceImpl,realTimeHistoryServiceImpl,scheduleDeviationHistoryDaoImpl,siriService,blockLocationRecordCacheImpl,blockLocationRecordDaoImpl,blockLocationServiceImpl,currentVehicleEstimationServiceImpl,vehicleLocationRecordCacheImpl,vehicleStatusServiceImpl,userReportingDaoImpl,userReportingServiceImpl,scheduledServiceServiceImpl,serviceAlertsCacheInMemoryImpl,serviceAlertsPersistenceDB,serviceAlertsServiceImpl,projectedShapePointServiceImpl,shapePointServiceImpl,transitGraphDaoImpl,agencyServiceImpl,arrivalAndDepartureAlarmServiceImpl,arrivalAndDepartureServiceImpl,consolidatedStopsServiceImpl,extendedCalendarServiceImpl,refreshableCalendarServiceImpl,routeCollectionSearchServiceImpl,routeServiceImpl,stopScheduleServiceImpl,stopSearchServiceImpl,stopSequenceCollectionServiceImpl,stopSequencesServiceImpl,stopTimeServiceImpl,trivialPredictionHelperService,whereGeospatialServiceImpl,transitDataFederationHibernateAnnotatedClasses,transitDataFederationHibernateMappingLocations,taskScheduler,refreshServiceImpl,transitDataServiceTemplate,extendedCalendarServiceServiceDateRangeCache,scheduledExecutorService,org.onebusaway.container.spring.PropertyOverrideConfigurer#1,modifications,bundleManagementService,agencyEntriesFactory,blockConfigurationEntriesFactory,blockEntriesFactory,distanceAlongShapeLibrary,frequencyEntriesFactory,routeCollectionEntriesFactory,routeEntriesFactory,serviceIdOverlapCache,stopEntriesFactory,stopTimeEntriesFactory,tripEntriesFactory,calendarServiceDataTask,generateRouteCollectionSearchIndexTask,generateStopSearchIndexTask,shapeGeospatialIndexTask,shapePointHelper,uniqueServiceImpl,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#0,startTask,clearCSV,clearCSVTask,change_log,changeLogTask,task_hastus_translate,hastusTranslateTask,task_missing_trips_for_block,missingTripsForBlock,task_missing_block_for_trips,missingBlockForTrips,task_multi_agency,multiAgencyModTask,task_load_gtfs_no_stop_consolidation,stopVerificationDistanceTask,task_stop_verification_distance,stopConsolidationSuggestionsTask,task_stop_consolidation_suggest,task_clear_gtfs_pre_modifications,task_load_gtfs_modifications,loadGtfsTaskNoConsolidation,loadGtfsTask,task_validation_pre_transform,gtfsPreValidationTask,task_mod,modTask,task_merge,mergeTask,task_diff_validation,diffValidationTask,diffService,task_clear_gtfs,clearGtfsTask,task_load_gtfs_post_modifications,task_validation_post_transform,gtfsPostValidationTask,task_gtfs_full_validation,gtfsFullValidationTask,bundleValidationService,fileService,task_check_shapes,checkShapes,task_stats,gtfsStatisticsTask,task_fixed_route_data_validation,fixedRouteDataValidationTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#1,modifyStartDateTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#2,calendarServiceTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#3,tripGraphTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#4,blockIndicesTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#5,narrativesTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#6,routeCollectionsSearchIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#7,stopSearchIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#8,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#9,blockLocationHistoryTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#10,blockRunIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#11,preCacheTask,task_metatdata,bundleMetadata,stopVerificationTask,task_stop_verification,gtfsArchiveTask,task_gtfs_archive,stopConsolidationFileTask,task_stop_consolidation_file,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#12,summaryTask,configurationServiceClient,entityReplacementStrategyFactory,entityReplacementLogger,bundleRequestResponse,gtfsRelationalDaoImpl,gtfs-bundles,entityReplacementStrategy,myCustomPropertyOverrides,multiCSVLogger]; root of factory hierarchy
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'scheduledExecutorService'
2497230 [pool-5-thread-1] INFO org.springframework.scheduling.concurrent.ScheduledExecutorFactoryBean - Shutting down ExecutorService 'scheduledExecutorService'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'taskScheduler'
2497230 [pool-5-thread-1] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Shutting down ExecutorService 'taskScheduler'
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor - Invoking destroy method on bean 'blockLocationRecordCacheImpl': public void org.onebusaway.transit_data_federation.impl.realtime.BlockLocationRecordCacheImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'bundleSearchServiceImpl': [transitDataServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'transitDataServiceImpl': [trivialPredictionHelperService, bundleSearchServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'trivialPredictionHelperService': [transitDataServiceTemplate]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'transitDataServiceTemplate': [transitDataServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'mbeanExporter'
2497230 [pool-5-thread-1] INFO org.springframework.jmx.export.MBeanExporter - Unregistering JMX-exposed beans on shutdown
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor - Invoking destroy method on bean 'serviceAlertsServiceImpl': public void org.onebusaway.transit_data_federation.impl.service_alerts.ServiceAlertsServiceImpl.stop()
2497230 [pool-5-thread-1] INFO org.onebusaway.transit_data_federation.impl.service_alerts.ServiceAlertsServiceImpl - Stopping ServiceAlertsService
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'arrivalAndDepartureServiceImpl': [arrivalAndDepartureAlarmServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'arrivalAndDepartureAlarmServiceImpl': [blockLocationServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'blockLocationServiceImpl': [blockStatusServiceImpl, arrivalAndDepartureServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'blockStatusServiceImpl': [arrivalAndDepartureServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor - Invoking destroy method on bean 'blockLocationServiceImpl': public void org.onebusaway.transit_data_federation.impl.realtime.BlockLocationServiceImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor - Invoking destroy method on bean 'arrivalAndDepartureAlarmServiceImpl': public void org.onebusaway.transit_data_federation.impl.ArrivalAndDepartureAlarmServiceImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor - Invoking destroy method on bean 'vehicleLocationRecordCacheImpl': public void org.onebusaway.transit_data_federation.impl.realtime.VehicleLocationRecordCacheImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'routeBeanServiceImpl': [stopBeanServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Retrieved dependent beans for bean 'stopBeanServiceImpl': [routeBeanServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'mbeanServer'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'sessionFactory'
2497230 [pool-5-thread-1] INFO org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean - Closing Hibernate SessionFactory
2497230 [pool-5-thread-1] INFO org.hibernate.impl.SessionFactoryImpl - closing
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'hibernateMappingLocations'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'hibernateAnnotatedClasses'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'cacheableKeyFactories'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter - Invoking destroy() on bean with name 'cacheManager'
2497230 [pool-5-thread-1] INFO org.onebusaway.container.spring.ehcache.EhCacheManagerFactoryBean - Shutting down EHCache CacheManager
2497230 [pool-5-thread-1] ERROR org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl - org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityReplacementStrategy': Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityReplacementStrategy': Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:983)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:879)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.onebusaway.container.ContainerLibrary.createContext(ContainerLibrary.java:108)
at org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl.build(BundleBuildingServiceImpl.java:515)
at org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl.doBuild(BundleBuildingServiceImpl.java:147)
at org.onebusaway.admin.service.bundle.api.remote.BuildRemoteResource$BuildThread.run(BuildRemoteResource.java:167)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:157)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:570)
... 18 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at java.net.URL.openStream(URL.java:1045)
at org.onebusaway.utility.IOLibrary.getPathAsInputStream(IOLibrary.java:48)
at org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create(EntityReplacementStrategyFactory.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:145)
... 19 more
2497230 [pool-5-thread-1] INFO org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl - deconfiguring logging