Troubleshooting the "Pending" status for a database target in OEM GC 12c


Right after getting OEM 12c management agent installed on target database servers, a problem has arisen. One of database instance was showing as “Status Pending” even it’s running normally.

Quick research on that guided me to one helpful Oracle Support Document “Troubleshooting The ‘Pending’ status for a Database Instance Target in Enterprise Manager 12c Cloud Control (Doc ID 1418873.1)”. This article is very informative one about the troubleshooting for this specific problem. Next, I just included the steps I followed to solve the problem on database target server.

Step 1: Confirm that the agent is started and uploading correctly

$ cd $AGENT_HOME
$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /prog/oracle11/agent12c/agent_inst
Agent Binaries    : /prog/oracle11/agent12c/core/12.1.0.3.0
Agent Process ID  : 25922
Parent Process ID : 25892
Agent URL         : https://db-host:1830/emd/main/   -- Agent use Https
Repository URL    : https://oms-host:4903/empbs/upload
Started at        : 2013-07-18 10:00:53
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2013-07-29 14:52:28  -- check if the time is current
Last attempted upload                        : 2013-07-29 14:52:28
Total Megabytes of XML files uploaded so far : 167.72               -- check agent is uploading
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0                    -- check that no xml files are pending
Available disk space on upload filesystem    : 33.91%               -- check that not running out of disk space
Collection Status                            : Collections enabled  -- check that collections are OK
Heartbeat Status                             : Ok                   -- check that it's not blocked
Last attempted heartbeat to OMS              : 2013-07-29 14:56:15  -- check that the timestamps are current
Last successful heartbeat to OMS             : 2013-07-29 14:56:15
Next scheduled heartbeat to OMS              : 2013-07-29 14:57:15
Target Manager Current Activity              : Compute Dynamic Properties (total operations: 1, active: 1, finished: 0)

---------------------------------------------------------------
Agent is Running and Ready

Result: agent is running properly.

Step 2: Check whether the agent can evaluate the response metric correctly.

$ ./emctl getmetric agent database_name_as_shown_in_console,oracle_database,Response
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Status,State,oraerr,Archiver,DatabaseStatus,ActiveState
1,OPEN,,STOPPED,ACTIVE,NORMAL

1= database is up

Result: agent can evaluate the response metric correctly.

Step 3: Check whether the target exists

$ ./emctl config agent listtargets | grep oracle_database
[db1, oracle_database]
[db2, oracle_database]
[db3, oracle_database]
[db4, oracle_database]
[db5, oracle_database]
[db6, oracle_database]
[db7, oracle_database]

Result: target database exists.

Step 4: Check if the target is broken via metric browser

In the output of Step 1, the agent URL is

https://db-host:1830/emd/main

To view metrics via browser, we need to paste work “browser” between ‘emd’ and ‘main’. So, the URL of metric browser is

https://db-host:1830/emd/browser/main

image

The user name Oracle here is unix user with which I install and set up agent on target database server.

Once logged in, the target database instance was showing without hyperlink. That tell us the database instance target is marked as broken.

image

Result: target database is marked as broken.

Step 5: Reload the dynamic properties using a temporary higher value if target database instance is marked as broken

$ ./emctl reload agent dynamicproperties -upload_timeout 240 TAPPS:oracle_database
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully
$
$ ./emctl control agent runCollection TAPPS:oracle_database Response
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

Result: dynamic properties of target database instance is reloaded successfully.

In above oracle support article, the reason we did this step is stated as below.

“If the target has been marked as ‘broken’ because the dynamic properties take too long to compute, it’s useful to try re-loading the dynamic properties with a higher timeout interval set.

Reload the dynamic properties for that particular target using a temporary higher value for “upload_timeout”. It is useful to increase the upload_timeout interval in case the target is not able to complete computing the dynamic properties in the default amount of time.

It may be necessary to increase the upload_timeout value even higher (but this may be indicative of a different underlying problem -eg database performance problem/agent performance problem or general machine performance problems.) “

Once dynamic properties of target database instance is reloaded, I refreshed the agent metric browser and the target database instance is showing with hyperlink as below.

image

And then, this database is showing ‘Up” in OEM console.

THE PROBLEM IS SOLVED!


This entry was posted in My Reference, Oracle Case Study, Oracle Point and tagged , , . Bookmark the permalink.

One Response to Troubleshooting the "Pending" status for a database target in OEM GC 12c

  1. Kristofer says:

    Thanks for the great article!
    The only difference I had was on step 2, it could not find the oracle_database target. I assume because of the collection issue.
    Thanks again!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>