So wanted to get it up.
oracle@devdb[AGENT]$ emctl start agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Starting agent ............................................... failed.
EMAgent is Thrashing. Exiting watchdog
Consult the log files in: /oracle/product/10.2.0/agent10g/sysman/log
oracle@devdb[AGENT]$
On the first look I thought, what happened !!! , it runs fine on all the other nodes and only on this it is thrashing.
So looked into the logfiles and found that the agent when starting was trying to discover the targets.
At some point it could not do it and so it failed.
So had a quick look into the mysupport.oracle.com for relevant materials. Read about 20 articles and came across 365252.1
As per the note I did the following.
- cd $ORACLE_HOME/sysman/emd (where $ORACLE_HOME is the Agent's home)
- backed up targets.xml
- Removed all the targets information except the one for agent
- Ran agentca -d to force a new discovery of the targets on the host
oracle@devdb[AGENT]$ agentca -d
Stopping the agent using /oracle/product/10.2.0/agent10g/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.
Running agentca using /oracle/product/10.2.0/agent10g/oui/bin/runConfig.sh ORACLE_HOME=/oracle/product/10.2.0/agent10g ACTION=Configure MODE=Perform RESPONSE_FILE=/oracle/product/10.2.0/agent10g/response_file RERUN=TRUE INV_PTR_LOC=/oracle/product/10.2.0/agent10g/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform - mode is starting for action: Configure
Perform - mode finished for action: Configure
You can see the log file: /oracle/product/10.2.0/agent10g/cfgtoollogs/oui/configActions2010-12-17_08-02-57-AM.log
oracle@devdb[AGENT]$
oracle@devdb[AGENT]$ cat /oracle/product/10.2.0/agent10g/cfgtoollogs/oui/configActions2010-12-17_08-02-57-AM.log
###################################################
The action configuration is performing
------------------------------------------------------
The plug-in Agent Configuration Assistant is running
Performing free port detection on host=testserver.testdomain.com
Performing targets discovery and agent configuration
Starting the agent
AgentPlugIn:agent configuration finished with status = true
The plug-in Agent Configuration Assistant has successfully been performed
------------------------------------------------------
The action configuration has successfully completed
###################################################
oracle@devdb[AGENT]$
Now I started the agent
oracle@devdb[AGENT]$ emctl start agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Starting agent ....... started.
oracle@devdb[AGENT]$
Looks good the agent came up and the status was also good.
oracle@devdb[AGENT]$ emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /oracle/product/10.2.0/agent10g
Agent binaries : /oracle/product/10.2.0/agent10g
Agent Process ID : 27849
Parent Process ID : 27836
Agent URL : https://testserver.testdomain.com:3872/emd/main/
Repository URL : https://testserver.testdomain.com:4898/em/upload
Started at : 2010-12-17 08:03:57
Started by user : oracle
Last Reload : 2010-12-17 08:03:57
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 1124
Size of XML files pending upload(MB) : 59.90
Available disk space on upload filesystem : 23.16%
Collection Status : Disabled by Upload Manager
Last attempted heartbeat to OMS : 2010-12-17 08:05:08
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready
oracle@devdb[AGENT]$
But the collection status was disabled by upload manager. That means the xml files are not transported.
You can see that the number of XML files are 1124 which is too high for the agent to process them.
Came across 396238.1 , and did the following to recover from the issue.
- Stop the agent
- Delete any pending upload files from the agent home
rm -r /sysman/emd/upload/*
rm - Start the agent.
oracle@devdb[AGENT]$ emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /oracle/product/10.2.0/agent10g
Agent binaries : /oracle/product/10.2.0/agent10g
Agent Process ID : 5953
Parent Process ID : 5943
Agent URL : https://testserver.testdomain.com:3872/emd/main/
Repository URL : https://testserver.testdomain.com:4898/em/upload
Started at : 2010-12-17 08:26:20
Started by user : oracle
Last Reload : 2010-12-17 08:26:20
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 16
Size of XML files pending upload(MB) : 8.43
Available disk space on upload filesystem : 23.21%
Last attempted heartbeat to OMS : 2010-12-17 08:27:00
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready
oracle@devdb[AGENT]$
Looks good but the agent is not actually uploading the xml files. You will see 16 files are not uploaded
Tried to manually force an upload.
oracle@devdb[AGENT]$ emctl upload
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..
oracle@devdb[AGENT]$
From the logfile found the following;
2010-12-17 08:28:01,116 Thread-71 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"
Then came across 799618.1 , and did the following as per the instructions;
- Login to 10.2.0.5 Grid Console as sysman user
- Navigate to Setup -> Agents -> Click on problematic Agent name
- In Agent home page click on "Resycnhronize" Button on right hand top of the page.
Choose the 'Unblock Agent' option and click Continue
Ensure that the Agent is up and running when attempting the Resynchronization.
Once the re-sycnchronization is completed successfully, the Agent should be able to communicate with the OMS
oracle@devdb[AGENT]$ emctl upload
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
oracle@devdb[${ORACLE_SID}]$ emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /oracle/product/10.2.0/agent10g
Agent binaries : /oracle/product/10.2.0/agent10g
Agent Process ID : 5953
Parent Process ID : 5943
Agent URL : https://testserver.testdomain.com:3872/emd/main/
Repository URL : https://testserver.testdomain.com:4898/em/upload
Started at : 2010-12-17 08:26:20
Started by user : oracle
Last Reload : 2010-12-17 08:26:20
Last successful upload : 2010-12-17 08:31:35
Total Megabytes of XML files uploaded so far : 10.42
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 23.21%
Last successful heartbeat to OMS : 2010-12-17 08:31:02
---------------------------------------------------------------
Agent is Running and Ready
oracle@devdb[AGENT]$
You will see that all the xml files are uploaded.
All done and the agent was up and running.
Thanks to documents from mysupport.oracle.com
Posts
No comments:
Post a Comment