OBA Hangs After an Hour

27 views
Skip to first unread message

Devin Braun

unread,
Feb 26, 2023, 6:32:04 PM2/26/23
to onebusaway-developers
Hi everyone,

We upgraded our firewall yesterday and since then, OBA will run for about an hour and then stop responding.  Are there tasks that run once an hour after the server starts?  I know it looks for a new bundle, but are there other URLs or other things it is trying to reach and is possibly blocked by?

I tried to build a new bundle to rule out the scheduling data/bundle in case it was corrupted, but I cannot build a bundle either.  It definitely times out at the end with the messages below.

Thanks for any help you can provide.

-Devin

2493480 [pool-2-thread-1] DEBUG org.hibernate.jdbc.AbstractBatcher  - Executing batch size: 10
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.AbstractBatcher  - about to close PreparedStatement (open PreparedStatements: 1, globally: 1)
2493495 [pool-2-thread-1] DEBUG org.springframework.orm.hibernate3.SessionFactoryUtils  - Closing Hibernate Session
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.ConnectionManager  - releasing JDBC connection [ (open PreparedStatements: 0, globally: 0) (open ResultSets: 0, globally: 0)]
2493495 [pool-2-thread-1] DEBUG org.hibernate.jdbc.ConnectionManager  - transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!
2497214 [pool-5-thread-1] INFO org.springframework.beans.factory.support.DefaultListableBeanFactory  - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@b300440: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.internalPersistenceAnnotationProcessor,org.springframework.aop.config.internalAutoProxyCreator,defaultPropertyPlaceholderConfigurer,ehCacheConfiguration,cacheManager,cacheableKeyFactories,cacheableMethodKeyFactoryManager,cacheableMethodManager,cacheableAnnotationInterceptor,org.springframework.beans.factory.config.MethodInvokingFactoryBean#0,hibernateProperties,hibernateAnnotatedClasses,hibernateMappingLocations,sessionFactory,transactionManager,org.springframework.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,mbeanServer,jmxAttributeSource,jmxNamingStategy,jmxInfoAssembler,mbeanExporter,ehCacheMBeanRegistration,bundle,dataSource,org.onebusaway.container.spring.PropertyOverrideConfigurer#0,shapePointsLibrary,agencyBeanServiceImpl,arrivalsAndDeparturesBeanServiceImpl,blockBeanServiceImpl,blockStatusBeanServiceImpl,nearbyStopsBeanServiceImpl,routeBeanServiceImpl,routesBeanServiceImpl,serviceAlertsBeanServiceImpl,shapeBeanServiceImpl,stopBeanServiceImpl,stopScheduleBeanServiceImpl,stopTimeBeanServiceImpl,stopWithArrivalsAndDeparturesBeanServiceImpl,stopsBeanServiceImpl,tripBeanServiceImpl,tripStatusBeanServiceImpl,tripStopTimesBeanServiceImpl,vehicleStatusBeanServiceImpl,blockCalendarServiceImpl,blockGeospatialServiceImpl,blockIndexFactoryServiceImpl,blockIndexServiceImpl,blockRunServiceImpl,blockStatusServiceImpl,scheduledBlockLocationServiceImpl,bundleSearchServiceImpl,realtimeSourceServiceImpl,bundleConfigDao,transitDataServiceExceptionInterceptor,transitDataServiceImpl,narrativeServiceImpl,gtfsRealtimeNegativeArrivalsImpl,gtfsRealtimeServiceImpl,blockLocationHistoryServiceImpl,realTimeHistoryServiceImpl,scheduleDeviationHistoryDaoImpl,siriService,blockLocationRecordCacheImpl,blockLocationRecordDaoImpl,blockLocationServiceImpl,currentVehicleEstimationServiceImpl,vehicleLocationRecordCacheImpl,vehicleStatusServiceImpl,userReportingDaoImpl,userReportingServiceImpl,scheduledServiceServiceImpl,serviceAlertsCacheInMemoryImpl,serviceAlertsPersistenceDB,serviceAlertsServiceImpl,projectedShapePointServiceImpl,shapePointServiceImpl,transitGraphDaoImpl,agencyServiceImpl,arrivalAndDepartureAlarmServiceImpl,arrivalAndDepartureServiceImpl,consolidatedStopsServiceImpl,extendedCalendarServiceImpl,refreshableCalendarServiceImpl,routeCollectionSearchServiceImpl,routeServiceImpl,stopScheduleServiceImpl,stopSearchServiceImpl,stopSequenceCollectionServiceImpl,stopSequencesServiceImpl,stopTimeServiceImpl,trivialPredictionHelperService,whereGeospatialServiceImpl,transitDataFederationHibernateAnnotatedClasses,transitDataFederationHibernateMappingLocations,taskScheduler,refreshServiceImpl,transitDataServiceTemplate,extendedCalendarServiceServiceDateRangeCache,scheduledExecutorService,org.onebusaway.container.spring.PropertyOverrideConfigurer#1,modifications,bundleManagementService,agencyEntriesFactory,blockConfigurationEntriesFactory,blockEntriesFactory,distanceAlongShapeLibrary,frequencyEntriesFactory,routeCollectionEntriesFactory,routeEntriesFactory,serviceIdOverlapCache,stopEntriesFactory,stopTimeEntriesFactory,tripEntriesFactory,calendarServiceDataTask,generateRouteCollectionSearchIndexTask,generateStopSearchIndexTask,shapeGeospatialIndexTask,shapePointHelper,uniqueServiceImpl,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#0,startTask,clearCSV,clearCSVTask,change_log,changeLogTask,task_hastus_translate,hastusTranslateTask,task_missing_trips_for_block,missingTripsForBlock,task_missing_block_for_trips,missingBlockForTrips,task_multi_agency,multiAgencyModTask,task_load_gtfs_no_stop_consolidation,stopVerificationDistanceTask,task_stop_verification_distance,stopConsolidationSuggestionsTask,task_stop_consolidation_suggest,task_clear_gtfs_pre_modifications,task_load_gtfs_modifications,loadGtfsTaskNoConsolidation,loadGtfsTask,task_validation_pre_transform,gtfsPreValidationTask,task_mod,modTask,task_merge,mergeTask,task_diff_validation,diffValidationTask,diffService,task_clear_gtfs,clearGtfsTask,task_load_gtfs_post_modifications,task_validation_post_transform,gtfsPostValidationTask,task_gtfs_full_validation,gtfsFullValidationTask,bundleValidationService,fileService,task_check_shapes,checkShapes,task_stats,gtfsStatisticsTask,task_fixed_route_data_validation,fixedRouteDataValidationTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#1,modifyStartDateTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#2,calendarServiceTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#3,tripGraphTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#4,blockIndicesTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#5,narrativesTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#6,routeCollectionsSearchIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#7,stopSearchIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#8,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#9,blockLocationHistoryTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#10,blockRunIndexTask,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#11,preCacheTask,task_metatdata,bundleMetadata,stopVerificationTask,task_stop_verification,gtfsArchiveTask,task_gtfs_archive,stopConsolidationFileTask,task_stop_consolidation_file,org.onebusaway.transit_data_federation.bundle.model.TaskDefinition#12,summaryTask,configurationServiceClient,entityReplacementStrategyFactory,entityReplacementLogger,bundleRequestResponse,gtfsRelationalDaoImpl,gtfs-bundles,entityReplacementStrategy,myCustomPropertyOverrides,multiCSVLogger]; root of factory hierarchy
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'scheduledExecutorService'
2497230 [pool-5-thread-1] INFO org.springframework.scheduling.concurrent.ScheduledExecutorFactoryBean  - Shutting down ExecutorService 'scheduledExecutorService'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'taskScheduler'
2497230 [pool-5-thread-1] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler  - Shutting down ExecutorService 'taskScheduler'
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor  - Invoking destroy method on bean 'blockLocationRecordCacheImpl': public void org.onebusaway.transit_data_federation.impl.realtime.BlockLocationRecordCacheImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'bundleSearchServiceImpl': [transitDataServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'transitDataServiceImpl': [trivialPredictionHelperService, bundleSearchServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'trivialPredictionHelperService': [transitDataServiceTemplate]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'transitDataServiceTemplate': [transitDataServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'mbeanExporter'
2497230 [pool-5-thread-1] INFO org.springframework.jmx.export.MBeanExporter  - Unregistering JMX-exposed beans on shutdown
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor  - Invoking destroy method on bean 'serviceAlertsServiceImpl': public void org.onebusaway.transit_data_federation.impl.service_alerts.ServiceAlertsServiceImpl.stop()
2497230 [pool-5-thread-1] INFO org.onebusaway.transit_data_federation.impl.service_alerts.ServiceAlertsServiceImpl  - Stopping ServiceAlertsService
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'arrivalAndDepartureServiceImpl': [arrivalAndDepartureAlarmServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'arrivalAndDepartureAlarmServiceImpl': [blockLocationServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'blockLocationServiceImpl': [blockStatusServiceImpl, arrivalAndDepartureServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'blockStatusServiceImpl': [arrivalAndDepartureServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor  - Invoking destroy method on bean 'blockLocationServiceImpl': public void org.onebusaway.transit_data_federation.impl.realtime.BlockLocationServiceImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor  - Invoking destroy method on bean 'arrivalAndDepartureAlarmServiceImpl': public void org.onebusaway.transit_data_federation.impl.ArrivalAndDepartureAlarmServiceImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.context.annotation.CommonAnnotationBeanPostProcessor  - Invoking destroy method on bean 'vehicleLocationRecordCacheImpl': public void org.onebusaway.transit_data_federation.impl.realtime.VehicleLocationRecordCacheImpl.stop()
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'routeBeanServiceImpl': [stopBeanServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory  - Retrieved dependent beans for bean 'stopBeanServiceImpl': [routeBeanServiceImpl]
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'mbeanServer'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'sessionFactory'
2497230 [pool-5-thread-1] INFO org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean  - Closing Hibernate SessionFactory
2497230 [pool-5-thread-1] INFO org.hibernate.impl.SessionFactoryImpl  - closing
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'hibernateMappingLocations'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'hibernateAnnotatedClasses'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'cacheableKeyFactories'
2497230 [pool-5-thread-1] DEBUG org.springframework.beans.factory.support.DisposableBeanAdapter  - Invoking destroy() on bean with name 'cacheManager'
2497230 [pool-5-thread-1] INFO org.onebusaway.container.spring.ehcache.EhCacheManagerFactoryBean  - Shutting down EHCache CacheManager
2497230 [pool-5-thread-1] ERROR org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl  - org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityReplacementStrategy': Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityReplacementStrategy': Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:983)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:879)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.onebusaway.container.ContainerLibrary.createContext(ContainerLibrary.java:108)
at org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl.build(BundleBuildingServiceImpl.java:515)
at org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl.doBuild(BundleBuildingServiceImpl.java:147)
at org.onebusaway.admin.service.bundle.api.remote.BuildRemoteResource$BuildThread.run(BuildRemoteResource.java:167)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public org.onebusaway.transit_data_federation.bundle.services.EntityReplacementStrategy org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create() throws java.io.IOException] threw exception; nested exception is java.net.ConnectException: Connection timed out: connect
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:157)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:570)
... 18 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at java.net.URL.openStream(URL.java:1045)
at org.onebusaway.utility.IOLibrary.getPathAsInputStream(IOLibrary.java:48)
at org.onebusaway.transit_data_federation.bundle.tasks.EntityReplacementStrategyFactory.create(EntityReplacementStrategyFactory.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:145)
... 19 more
2497230 [pool-5-thread-1] INFO org.onebusaway.admin.service.bundle.impl.BundleBuildingServiceImpl  - deconfiguring logging

Devin Braun

unread,
Mar 3, 2023, 4:54:23 PM3/3/23
to onebusaway-developers
If anyone experiences this issue, the server didn't have outgoing internet access and this was preventing a process from running and also from building new bundles.  Giving it internet access solved the issue.
Reply all
Reply to author
Forward
0 new messages