Projects Stuck between 2 tasks

306 views
Skip to first unread message

chitraranjan urs V

unread,
Mar 19, 2019, 4:05:26 AM3/19/19
to ATG_Tech
Hello All,

Recently I have been facing issue while pushing the projects to staging. The projects get stuck between Approve for staging Deployment and verify staging deployment.
Initially I had assumed that was because of more number of assets in the project, but this happens with single asset as well.

The gets stuck while applying the changes to an agent and the status remain 'Applying' and does not change to 'Applied'.

As a work around to resolve the issue I do 

1. Restart of merchandiser and the agents configured and revert the project to author, delete it and recreate the project and push it to staging. 
2. If the above steps does not work I go for full deployment. Can anyone please suggest a permanent fix for this issue?

Any suggestion would be appreciated!!

<Mar 18, 2019 4:17:36 PM GMT> <Notice> <Log Management> <BEA-170027> <The Server has established connection with the Domain level Diagnostic Service successfully.>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on XXX.XXX.XXX.XXX:8006 for protocols iiop, t3, ldap, snmp, http.>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <WebLogicServer> <BEA-000330> <Started WebLogic Managed Server "XXXXXX_XX_XXX" for domain "XX_XX_XXX" running in Production Mode>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Mar 18, 2019 4:17:36 PM GMT> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>
**** Warning    Mon Mar 18 16:18:12 GMT 2019    1552925892331   /atg/epub/PublishingRepository  Using default JDBC type for: project:tar222 could not find this column in the table's meta data
**** info       Mon Mar 18 16:20:35 GMT 2019    1552926035956   /atg/dynamo/servlet/adminpipeline/Authenticator Authenticator allowing access for administrators-group.
**** Warning    Mon Mar 18 16:20:36 GMT 2019    1552926036888   /atg/commerce/catalog/ProductCatalog    The GSACacheClientManager is not enabled.  Service: /atg/dynamo/service/GSACacheClientManager
**** Warning    Mon Mar 18 16:20:37 GMT 2019    1552926037388   /atg/commerce/catalog/ProductCatalog    The GSAInvalidatorService is not enabled for jms cache invalidation events.  Service: /atg/dynamo/service/GSAInvalidatorService
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097834   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent1a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097841   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXXXXAgent4a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097843   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent6a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097845   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgentLock' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097847   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent10a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097849   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent8a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097851   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgentScenario' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097853   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent2a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097859   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXXXXAgent2a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097861   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent5a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097863   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent4a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097868   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXXXXAgent3a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097871   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent3a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097873   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent7a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097876   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXAgent9a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097881   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXXXXAgent5a' : IDLE : null
**** Error      Mon Mar 18 16:21:37 GMT 2019    1552926097887   /atg/epub/DeploymentServer      error stopping deployment on target:agent 'Staging:XXXXXXXXAgent1a' : IDLE : null
**** info       Mon Mar 18 16:21:43 GMT 2019    1552926103275   /atg/deployment/DeploymentManager       Purging deployment data for deployment 88800002
**** info       Mon Mar 18 16:21:43 GMT 2019    1552926103404   /atg/dynamo/service/InternalCurrentDate Resolving reference to /atg/dynamo/service/CurrentDate
**** Warning    Mon Mar 18 16:21:43 GMT 2019    1552926103849   /atg/epub/PublishingRepository  Using default JDBC type for: project:tar222 could not find this column in the table's meta data
**** Warning    Mon Mar 18 16:22:29 GMT 2019    1552926149219   /atg/epub/PublishingRepository  Using default JDBC type for: project:tar222 could not find this column in the table's meta data
**** debug      Mon Mar 18 16:22:29 GMT 2019    1552926149776   /atg/epub/DeploymentServer      Forward deploying following active workspaces: [workspace:76302]
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153509   /atg/epub/DeploymentServer      Run first apply phase: true
**** info       Mon Mar 18 16:22:33 GMT 2019    1552926153552   /com/uk/vodafone/fraud/repository/FraudConfigurationRepository_staging  SQL Repository startup complete
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153553   /atg/epub/DeploymentServer      Switch switchable CA datasources:
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153553   /atg/epub/DeploymentServer        isDeploymentRollback(): false
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153589   /atg/epub/DeploymentServer        Current Target live datasource name: DataSourceA
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153590   /atg/epub/DeploymentServer        CA switching datasource: /atg/commerce/jdbc/ProductCatalogSwitchingDataSource_staging
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153590   /atg/epub/DeploymentServer          Current live datasource name: DataSourceB
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153590   /atg/epub/DeploymentServer          Current offline datasource name: DataSourceA
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153590   /atg/epub/DeploymentServer          The current CA live datasource is pointing at the Target offline datasource. No CA switch necessary.
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153590   /atg/epub/DeploymentServer      Switchable CA datasources switched.
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153644   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:Creating standard DeploymentData
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153650   /atg/epub/DeploymentServer      Forward deploying following active workspaces: [workspace:76302]
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153651   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:Deploy Data
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153651   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:Starting DAF deployment with ID: 88900002
**** info       Mon Mar 18 16:22:33 GMT 2019    1552926153652   /atg/deployment/DeploymentManager       Database: Oracle
**** debug      Mon Mar 18 16:22:33 GMT 2019    1552926153732   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:Polling Until Data Appplied
**** info       Mon Mar 18 16:22:33 GMT 2019    1552926153734   /atg/deployment/DeploymentManager       Starting deployment with id: 88900002
**** info       Mon Mar 18 16:22:33 GMT 2019    1552926153762   /atg/deployment/DeploymentManager       Executing deployment in LOCAL mode
**** Warning    Mon Mar 18 16:22:34 GMT 2019    1552926154786   /atg/epub/PublishingRepository  Using default JDBC type for: project:tar222 could not find this column in the table's meta data
**** debug      Mon Mar 18 16:22:47 GMT 2019    1552926167717   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:DeploymentListener.deploymentCompleted called for deployment with ID: 88900002
**** debug      Mon Mar 18 16:22:47 GMT 2019    1552926167718   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:DAFDeploymentID: 88900002 matches DeploymmentID: 88900002
**** debug      Mon Mar 18 16:22:47 GMT 2019    1552926167718   /atg/epub/DeploymentServer      DistributedDeploymentAdapter:DeploymentListener.deploymentCompleted
**** info       Mon Mar 18 16:22:47 GMT 2019    1552926167718   /atg/deployment/DeploymentManager       Purging deployment data for deployment 88900002
**** info       Mon Mar 18 16:22:49 GMT 2019    1552926169271   /atg/deployment/DeploymentManager       Deployment 88900002 finished in 0:15.518
**** info       Mon Mar 18 16:22:49 GMT 2019    1552926169271   /atg/deployment/DeploymentManager       Averaged 0 assets per second
**** Warning    Mon Mar 18 16:24:29 GMT 2019    1552926269235   /atg/epub/PublishingRepository  Using default JDBC type for: project:tar222 could not find this column in the table's meta data


Regards,
Chitraranjan Urs V

Naga

unread,
Mar 19, 2019, 5:38:51 AM3/19/19
to atg_...@googlegroups.com
Is it because CMS is running on all agents ??  did you tail the agent logs ? Any activity therE? 

Full deploy isnt a good solution to go with if you have a lot of assets managed via BCC. Remember it will wipe out your catalog and deploy the assets, so can be really crazy if it gets stuck in between or has some property specific errors.

-Naga

--
--
You received this message because you are subscribed to the Google Groups "ATG_Tech" group.
To post to this group, send email to atg_...@googlegroups.com
To unsubscribe from this group, send email to atg_tech-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/atg_tech?hl=en
---
You received this message because you are subscribed to the Google Groups "ATG_Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atg_tech+u...@googlegroups.com.
To post to this group, send email to atg_...@googlegroups.com.
Visit this group at https://groups.google.com/group/atg_tech.
To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/870859b1-1e21-4444-8fa8-0caecd30d1b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

chitraranjan urs V

unread,
Mar 19, 2019, 9:38:19 AM3/19/19
to atg_...@googlegroups.com
Thanks for the quick update Naga,

CMS is configured in only one agent and I did not tail the agents logs. Will do it on the next occurrence and update.

Regards,
Chitraranjan Urs V

KdM

unread,
Mar 19, 2019, 9:52:58 AM3/19/19
to atg_...@googlegroups.com
1. Check your BCC start up logs and fix all errors related to these(if any). it should be clean startup.
2. Check in CA console whether any of those agents are in failed state.(re-add add them(all together), check the RMI port whether it is open)

/ KdM


Scott Stirling

unread,
Mar 20, 2019, 12:47:14 PM3/20/19
to atg_...@googlegroups.com
Hi Chitraranjan - Two suggestions to troubleshoot and a couple ideas
of potential causes:

1. get a thread dump (aka stack trace: kill -3 <pid> or jstack <pid> >
flie) when the BCC is in this state between workflow steps. Examine
for active threads involved in CA activities.
2. get a thread dump from the last updated preview/staging instance
when this hang appears. Examine for active threads involved in CA
activities.
3, use dyn/admin to verify atg/epub/DeploymentAgent status in each
staging instance for expected state (deployment time, id, status),
also confirm agent states in BCC's atg/epub/DeploymentManager. If any
are in questionable status, get thread dumps there too.
4. Confirm agent state in BCC Admin Console switching datasource view.
Are any out of sync like A&B rather than AA or BB or are set to
opposite of all the others? This may help identify instance(s) to get
thread dumps from too.
5, examine thread dumps for activity in process and identify the
source of the hang.
6. examine logs from BCC instance and from staging instance(s) for
correlating exceptions. Look for transaction timeout / rollback
exceptions and database related exceptions in the app logs.

question: Why are there so many staging instances in the environment?
Just curious. The log shows close to 20 agents in the staging
topology. That's different from what I'm used to seeing where the
staging topology is minimal for previewing content changes, one
instance, using itself as lock manager and itself for CMS as well.
This staging topology looks like an entire replication of the prod
topo, which is interesting but potentially costly, which is why I'm
interested.

potential causes of this kind of workflow hang in ATG/OC in my experience:
1. XA transaction timeout - if too short may cause long running
transactions such as CMS and BCC deployment actions to abort at the
app server level, but the workflow doesn't rollback or forward on the
BCC. Set WLS XA timeout high (such as 6 hours) for both BCC instance
and the CMS instance in the staging env.

2. other processes invoked in the BCC deployment workflow are hanging,
such as an Endeca incremental index launched at end of BCC deployment
or some custom job. This can be confirmed / ruled out by examining
thread dumps from the BCC while it's in this state between workflow
steps.

Scott Stirling
Pivotree
> To view this discussion on the web visit https://groups.google.com/d/msgid/atg_tech/CAMD9EGxvRjZRHEkHBu%3Dfz752GkYa0jpkzTMLijZZPv2to3iMNw%40mail.gmail.com.

Scott Stirling

unread,
Mar 20, 2019, 12:47:39 PM3/20/19
to atg_...@googlegroups.com
OK more than 2. - SS

Scott Stirling

unread,
Mar 20, 2019, 12:51:32 PM3/20/19
to atg_...@googlegroups.com
Also - one more suggestion, check your ATG build kit patch level and
login to Oracle support and check for related bug fixes if you're not
up to date.

Your problem does not sound that unusual for the BCC. By some
fortunate chance there may be a patch available for it - always worth
a check.

Kind regards,
Scott Stirling
Pivotree

chitraranjan urs V

unread,
Mar 21, 2019, 5:28:39 AM3/21/19
to atg_...@googlegroups.com
Thanks for all the suggestions Scott,

I will follow the mentioned suggestions on the next occurrence of the issue.

And to answer the question: Why are there so many staging instances in the environment?

This is a replica of Prod environment hence, there are many instances.

Regards,
Chitraranjan Urs V

Rajkumar Gundu

unread,
Mar 21, 2019, 8:46:38 PM3/21/19
to atg_...@googlegroups.com
Chitraranjan,
This oracle doc has some information:
We had similar issues in project running on 10.2 about 4 years ago. Resolved with patch.
As Scott said, it is likely a thread lock issue at agent side. 

Regards
Raj


Reply all
Reply to author
Forward
0 new messages