Projects Stuck between 2 tasks

chitraranjan urs V

unread,

Mar 19, 2019, 4:05:26 AM3/19/19

to ATG_Tech

Hello All,

Recently I have been facing issue while pushing the projects to staging. The projects get stuck between Approve for staging Deployment and verify staging deployment.

Initially I had assumed that was because of more number of assets in the project, but this happens with single asset as well.

The gets stuck while applying the changes to an agent and the status remain 'Applying' and does not change to 'Applied'.

As a work around to resolve the issue I do

1. Restart of merchandiser and the agents configured and revert the project to author, delete it and recreate the project and push it to staging.

2. If the above steps does not work I go for full deployment. Can anyone please suggest a permanent fix for this issue?

Any suggestion would be appreciated!!

**** Warning Mon Mar 18 16:18:12 GMT 2019 1552925892331 /atg/epub/PublishingRepository Using default JDBC type for: project:tar222 could not find this column in the table's meta data

**** info Mon Mar 18 16:20:35 GMT 2019 1552926035956 /atg/dynamo/servlet/adminpipeline/Authenticator Authenticator allowing access for administrators-group.

**** Warning Mon Mar 18 16:20:36 GMT 2019 1552926036888 /atg/commerce/catalog/ProductCatalog The GSACacheClientManager is not enabled. Service: /atg/dynamo/service/GSACacheClientManager

**** Warning Mon Mar 18 16:20:37 GMT 2019 1552926037388 /atg/commerce/catalog/ProductCatalog The GSAInvalidatorService is not enabled for jms cache invalidation events. Service: /atg/dynamo/service/GSAInvalidatorService

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097834 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent1a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097841 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXXXXAgent4a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097843 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent6a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097845 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgentLock' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097847 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent10a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097849 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent8a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097851 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgentScenario' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097853 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent2a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097859 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXXXXAgent2a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097861 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent5a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097863 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent4a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097868 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXXXXAgent3a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097871 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent3a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097873 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent7a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097876 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXAgent9a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097881 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXXXXAgent5a' : IDLE : null

**** Error Mon Mar 18 16:21:37 GMT 2019 1552926097887 /atg/epub/DeploymentServer error stopping deployment on target:agent 'Staging:XXXXXXXXAgent1a' : IDLE : null

**** info Mon Mar 18 16:21:43 GMT 2019 1552926103275 /atg/deployment/DeploymentManager Purging deployment data for deployment 88800002

**** info Mon Mar 18 16:21:43 GMT 2019 1552926103404 /atg/dynamo/service/InternalCurrentDate Resolving reference to /atg/dynamo/service/CurrentDate

**** Warning Mon Mar 18 16:21:43 GMT 2019 1552926103849 /atg/epub/PublishingRepository Using default JDBC type for: project:tar222 could not find this column in the table's meta data

**** Warning Mon Mar 18 16:22:29 GMT 2019 1552926149219 /atg/epub/PublishingRepository Using default JDBC type for: project:tar222 could not find this column in the table's meta data

**** debug Mon Mar 18 16:22:29 GMT 2019 1552926149776 /atg/epub/DeploymentServer Forward deploying following active workspaces: [workspace:76302]

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153509 /atg/epub/DeploymentServer Run first apply phase: true

**** info Mon Mar 18 16:22:33 GMT 2019 1552926153552 /com/uk/vodafone/fraud/repository/FraudConfigurationRepository_staging SQL Repository startup complete

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153553 /atg/epub/DeploymentServer Switch switchable CA datasources:

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153553 /atg/epub/DeploymentServer isDeploymentRollback(): false

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153589 /atg/epub/DeploymentServer Current Target live datasource name: DataSourceA

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153590 /atg/epub/DeploymentServer CA switching datasource: /atg/commerce/jdbc/ProductCatalogSwitchingDataSource_staging

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153590 /atg/epub/DeploymentServer Current live datasource name: DataSourceB

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153590 /atg/epub/DeploymentServer Current offline datasource name: DataSourceA

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153590 /atg/epub/DeploymentServer The current CA live datasource is pointing at the Target offline datasource. No CA switch necessary.

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153590 /atg/epub/DeploymentServer Switchable CA datasources switched.

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153644 /atg/epub/DeploymentServer DistributedDeploymentAdapter:Creating standard DeploymentData

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153650 /atg/epub/DeploymentServer Forward deploying following active workspaces: [workspace:76302]

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153651 /atg/epub/DeploymentServer DistributedDeploymentAdapter:Deploy Data

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153651 /atg/epub/DeploymentServer DistributedDeploymentAdapter:Starting DAF deployment with ID: 88900002

**** info Mon Mar 18 16:22:33 GMT 2019 1552926153652 /atg/deployment/DeploymentManager Database: Oracle

**** debug Mon Mar 18 16:22:33 GMT 2019 1552926153732 /atg/epub/DeploymentServer DistributedDeploymentAdapter:Polling Until Data Appplied

**** info Mon Mar 18 16:22:33 GMT 2019 1552926153734 /atg/deployment/DeploymentManager Starting deployment with id: 88900002

**** info Mon Mar 18 16:22:33 GMT 2019 1552926153762 /atg/deployment/DeploymentManager Executing deployment in LOCAL mode

**** Warning Mon Mar 18 16:22:34 GMT 2019 1552926154786 /atg/epub/PublishingRepository Using default JDBC type for: project:tar222 could not find this column in the table's meta data

**** debug Mon Mar 18 16:22:47 GMT 2019 1552926167717 /atg/epub/DeploymentServer DistributedDeploymentAdapter:DeploymentListener.deploymentCompleted called for deployment with ID: 88900002

**** debug Mon Mar 18 16:22:47 GMT 2019 1552926167718 /atg/epub/DeploymentServer DistributedDeploymentAdapter:DAFDeploymentID: 88900002 matches DeploymmentID: 88900002

**** debug Mon Mar 18 16:22:47 GMT 2019 1552926167718 /atg/epub/DeploymentServer DistributedDeploymentAdapter:DeploymentListener.deploymentCompleted

**** info Mon Mar 18 16:22:47 GMT 2019 1552926167718 /atg/deployment/DeploymentManager Purging deployment data for deployment 88900002

**** info Mon Mar 18 16:22:49 GMT 2019 1552926169271 /atg/deployment/DeploymentManager Deployment 88900002 finished in 0:15.518

**** info Mon Mar 18 16:22:49 GMT 2019 1552926169271 /atg/deployment/DeploymentManager Averaged 0 assets per second

**** Warning Mon Mar 18 16:24:29 GMT 2019 1552926269235 /atg/epub/PublishingRepository Using default JDBC type for: project:tar222 could not find this column in the table's meta data

Regards,

Chitraranjan Urs V

Naga

unread,

Mar 19, 2019, 5:38:51 AM3/19/19

to atg_...@googlegroups.com

Is it because CMS is running on all agents ?? did you tail the agent logs ? Any activity therE?

Full deploy isnt a good solution to go with if you have a lot of assets managed via BCC. Remember it will wipe out your catalog and deploy the assets, so can be really crazy if it gets stuck in between or has some property specific errors.

-Naga

--
--
You received this message because you are subscribed to the Google Groups "ATG_Tech" group.
To post to this group, send email to atg_...@googlegroups.com
To unsubscribe from this group, send email to atg_tech-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/atg_tech?hl=en
---
You received this message because you are subscribed to the Google Groups "ATG_Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atg_tech+u...@googlegroups.com.
To post to this group, send email to atg_...@googlegroups.com.
Visit this group at https://groups.google.com/group/atg_tech.
To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/870859b1-1e21-4444-8fa8-0caecd30d1b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

chitraranjan urs V

unread,

Mar 19, 2019, 9:38:19 AM3/19/19

to atg_...@googlegroups.com

Thanks for the quick update Naga,

CMS is configured in only one agent and I did not tail the agents logs. Will do it on the next occurrence and update.

Regards,

Chitraranjan Urs V

To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CAG1Kp32yiEeba9AAOYTfzgnPNeDqpcj%2BsGSwasNS%3DqfpgA48Kg%40mail.gmail.com.

KdM

unread,

Mar 19, 2019, 9:52:58 AM3/19/19

to atg_...@googlegroups.com

1. Check your BCC start up logs and fix all errors related to these(if any). it should be clean startup.

2. Check in CA console whether any of those agents are in failed state.(re-add add them(all together), check the RMI port whether it is open)

/ KdM

To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CA%2BMHF2ApQSOxWSjcoNRKBDXRcxf4BChtafHWdSkti58s9oYQmQ%40mail.gmail.com.

Scott Stirling

unread,

Mar 20, 2019, 12:47:14 PM3/20/19

to atg_...@googlegroups.com

Hi Chitraranjan - Two suggestions to troubleshoot and a couple ideas
of potential causes:

1. get a thread dump (aka stack trace: kill -3 <pid> or jstack <pid> >
flie) when the BCC is in this state between workflow steps. Examine
for active threads involved in CA activities.
2. get a thread dump from the last updated preview/staging instance
when this hang appears. Examine for active threads involved in CA
activities.
3, use dyn/admin to verify atg/epub/DeploymentAgent status in each
staging instance for expected state (deployment time, id, status),
also confirm agent states in BCC's atg/epub/DeploymentManager. If any
are in questionable status, get thread dumps there too.
4. Confirm agent state in BCC Admin Console switching datasource view.
Are any out of sync like A&B rather than AA or BB or are set to
opposite of all the others? This may help identify instance(s) to get
thread dumps from too.
5, examine thread dumps for activity in process and identify the
source of the hang.
6. examine logs from BCC instance and from staging instance(s) for
correlating exceptions. Look for transaction timeout / rollback
exceptions and database related exceptions in the app logs.

question: Why are there so many staging instances in the environment?
Just curious. The log shows close to 20 agents in the staging
topology. That's different from what I'm used to seeing where the
staging topology is minimal for previewing content changes, one
instance, using itself as lock manager and itself for CMS as well.
This staging topology looks like an entire replication of the prod
topo, which is interesting but potentially costly, which is why I'm
interested.

potential causes of this kind of workflow hang in ATG/OC in my experience:
1. XA transaction timeout - if too short may cause long running
transactions such as CMS and BCC deployment actions to abort at the
app server level, but the workflow doesn't rollback or forward on the
BCC. Set WLS XA timeout high (such as 6 hours) for both BCC instance
and the CMS instance in the staging env.

2. other processes invoked in the BCC deployment workflow are hanging,
such as an Endeca incremental index launched at end of BCC deployment
or some custom job. This can be confirmed / ruled out by examining
thread dumps from the BCC while it's in this state between workflow
steps.

Scott Stirling
Pivotree

> To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CAMD9EGxvRjZRHEkHBu%3Dfz752GkYa0jpkzTMLijZZPv2to3iMNw%40mail.gmail.com.

Scott Stirling

unread,

Mar 20, 2019, 12:47:39 PM3/20/19

to atg_...@googlegroups.com

OK more than 2. - SS

Scott Stirling

unread,

Mar 20, 2019, 12:51:32 PM3/20/19

to atg_...@googlegroups.com

Also - one more suggestion, check your ATG build kit patch level and
login to Oracle support and check for related bug fixes if you're not
up to date.

Your problem does not sound that unusual for the BCC. By some
fortunate chance there may be a patch available for it - always worth
a check.

Kind regards,
Scott Stirling
Pivotree

chitraranjan urs V

unread,

Mar 21, 2019, 5:28:39 AM3/21/19

to atg_...@googlegroups.com

Thanks for all the suggestions Scott,

I will follow the mentioned suggestions on the next occurrence of the issue.

And to answer the question: Why are there so many staging instances in the environment?

This is a replica of Prod environment hence, there are many instances.

Regards,

Chitraranjan Urs V

To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CAEdD_Csnj90C2Kh2X6Jv1t%3DaWiFqRocDPrOcFuCyVbnxcFnB2A%40mail.gmail.com.

Rajkumar Gundu

unread,

Mar 21, 2019, 8:46:38 PM3/21/19

to atg_...@googlegroups.com

Chitraranjan,

This oracle doc has some information:

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=505435491117958&id=1580617.1&_afrWindowMode=0&_adf.ctrl-state=zpo6baqkd_4

We had similar issues in project running on 10.2 about 4 years ago. Resolved with patch.

As Scott said, it is likely a thread lock issue at agent side.

Regards

Raj

To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CA%2BMHF2CO-3Unur5_CW7rSG56OheJUti6gQM5Z_E02D6JQTLJmA%40mail.gmail.com.

Reply all

Reply to author

Forward