Shivananda Rao | Oracle Database issues solutions

Time 2020-11-04 18:03:33

Web Name: Shivananda Rao | Oracle Database issues solutions

WebSite: http://shivanandarao-oracle.com

ID:104921

Keywords:

Oracle,Rao,Shivananda,

Description:

This article demonstrates implementing TDE in a RAC environment with standby database configured. The method used is same as what was done in my previous post, but uses a new feature of 12c called administer key management .Environment:2 node primary RAC: 12cnode1, 12cnode22 node standby RAC: 1212drnode1, 1212drnode2Primary database name: srprimStandby database name: srpstbJust ensuring before implementing TDE that the standby database is in sync with the primary database.DGMGRL show configurationConfiguration - dgconfig Protection Mode: MaxPerformance Members: srprim - Primary database srpstb - Physical standby database Fast-Start Failover: DISABLEDConfiguration Status:SUCCESS (status updated 6 seconds ago)Create the directory structure where you need to place the wallet files and add the following entry in the SQLNET.ORA file on all the nodes of the primary database. This entry is needed to determine the wallet location.In my case, I have decided to place the wallet files in the location /u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet and so have I created the same.[oracle@12cnode1 ~]$ mkdir -p /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet[oracle@12cnode2 ~]$ mkdir -p /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/walletENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet)The instances show the default location of the wallet when queried the GV$ENCRYPTION_WALLET view. So, in order to get the instances reflect the right wallet location as per the SQLNET.ORA file, we need to bounce the instances. SYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ------------------------------------------------------------ --------------- -------------------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0[oracle@12cnode1 ~]$ srvctl stop instance -instance srprim1 -database srprim[oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ srvctl start instance -instance srprim1 -database srprimSYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYP WALLET_OR FULLY_BAC CON_ID---------- ---------- -------------------------------------------------------------------- --------------- ---------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/admin/srprim/wallet NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0Similarly, restart the second instance as well.[oracle@12cnode1 ~]$ srvctl stop instance -instance srprim2 -database srprim[oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ srvctl start instance -instance srprim2 -database srprimSYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0Now since the WRL_PARAMETER is reflecting the right directory where the wallet needs to be stored, lets move in creating the wallet keystore. Use the administer key management clause to create the keystore in the wallet directory location. This needs to be done only on one instance. (In my case, I m running it on the first instance).SYS@srprim1 administer key management create keystore '/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/' identified by oracle123 keystore altered.Now when you query the gv$encryption_wallet view, you should see that status as CLOSED for the first instance where as the second instance should reflect it as NOT_AVAILABLE .SYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ CLOSED UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0Now, open the keystore that was created previously on the instance where it was created.SYS@srprim1 administer key management set keystore open identified by oracle123 keystore altered.SYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ OPEN_NO_MASTER_ PASSWORD SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0As we have opened the keystore, the status should reflect as OPEN, but in 12c, you need to have a MASTER KEY as well which is why the STATUS column reflects as OPEN_NO_MASTER_KEY where gv$encryption_wallet view is queried.So, create a MASTER KEY using the administer key management clause as stated below. The MASTER KEY would be stored in the same location where the wallet file location was specified in the sqlnet.ora file. SYS@srprim1 administer key management set key identified by oracle123 with backup;keystore altered.SYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ------------ --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ OPEN PASSWORD SINGLE NO 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0Once the MASTER KEY is created, the status for the corresponding instance turns out to be OPEN when gv$encryption_wallet is queried.As seen, the wallet file ewallet.p12 does also contains the master key. This needs to be copied on to the remaining nodes of the cluster. [oracle@12cnode1 ~]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/total 8-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12[oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$ [oracle@12cnode1 ~]$Copying the wallet file to node 12cnode2. [oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/[oracle@12cnode1 wallet]$ scp ewallet.p12 oracle@12cnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ewallet.p12 100% 3848 3.8KB/s 00:00 [oracle@12cnode1 wallet]$ Now query gv$encryption_wallet to check the status with respect to each instance. We can see that the status now reflects as OPEN for the second instance on 12cnode2 as well.SYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- ---------- --------------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ OPEN PASSWORD SINGLE NO 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ OPEN PASSWORD SINGLE NO 0Please note that at this point of time, we have setup only manual wallet management which means that when the database/instance is started, we need to open the wallet manually. [oracle@12cnode1 wallet]$ srvctl stop database -database srprim[oracle@12cnode1 wallet]$ [oracle@12cnode1 wallet]$ [oracle@12cnode1 wallet]$ srvctl start database -database srprimSYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYP WALLET_OR FULLY_BAC CON_ID---------- --------------- ---------------------------------------------------------------------- ---------- ---------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ CLOSED UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ CLOSED UNKNOWN SINGLE UNDEFINED 0SYS@srprim1 Now, let me create auto login setup which automatically opens the wallet upon the instance/database retart. This creates a file with name cwallet.sso under the same location that is specified in the sqlnet.ora file.SYS@srprim1 administer key management create auto_login keystore from keystore '/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/' identified by oracle123 keystore altered.SYS@srprim1 host[oracle@12cnode1 wallet]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/total 12-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12-rw-r--r--. 1 oracle oinstall 3893 Aug 29 12:29 cwallet.sso[oracle@12cnode1 wallet]$ srvctl stop database -d srprimCopy this auto login wallet file (cwallet.sso) on to the second node 12cnode2 as well and then start the database. Upon this, the wallets should be opened automatically for both the instances.[oracle@12cnode1 wallet]$ scp cwallet.sso oracle@12cnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/cwallet.sso 100% 3893 3.8KB/s 00:00 [oracle@12cnode1 wallet]$ srvctl start database -d srprimSYS@srprim1 select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- ---------- --------------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/ OPEN AUTOLOGIN SINGLE NO 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim2/wallet/ OPEN AUTOLOGIN SINGLE NO 0With the primary setup, the standby nodes should not be a problem. Perform the same initial steps as did on primary nodes of adding the below entry (speaking about the wallet location) in the sqlnet.ora file on all the nodes of the standby cluster.Please note that if the directory structure is different, act accordingly in specifiying the appropriate available location.ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/$ORACLE_SID/wallet) [oracle@1212drnode1 admin]$ pwd/u01/app/oracle/product/12.1.0.2/db_1/network/admin[oracle@1212drnode1 admin]$ mkdir -p srpstb1/wallet[oracle@1212drnode2 admin]$ pwd/u01/app/oracle/product/12.1.0.2/db_1/network/admin[oracle@1212drnode2 admin]$ mkdir -p srpstb2/walletSince we haven t bounced the standby instances after adding the above entry in sqlnet.ora file, the wrl_parameter yet points to the inital location. Let s copy the wallet files from the primary node and then bounce the standby instances at one go.SQL select * from gv$encryption_wallet order by 1; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYP WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ---------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/admin/srpstb/wallet NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/admin/srpstb/wallet NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED 0Scp all the wallet files (master key file and autologin file) from any one of the primary node to the first standby node.[oracle@12cnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srprim1/wallet/[oracle@12cnode1 wallet]$ ls -lrttotal 12-rw-r--r--. 1 oracle oinstall 2400 Aug 29 11:46 ewallet_2018082906160156.p12-rw-r--r--. 1 oracle oinstall 3848 Aug 29 11:46 ewallet.p12-rw-r--r--. 1 oracle oinstall 3893 Aug 29 12:29 cwallet.sso[oracle@12cnode1 wallet]$ scp *wallet.* oracle@1212drnode1:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb1/wallet/Warning: the RSA host key for '1212drnode1' differs from the key for the IP address '192.168.0.120'Offending key for IP in /home/oracle/.ssh/known_hosts:14Matching host key in /home/oracle/.ssh/known_hosts:16Are you sure you want to continue connecting (yes/no)? yesoracle@1212drnode1's password: cwallet.sso 100% 3893 3.8KB/s 00:00 ewallet.p12 100% 3848 3.8KB/s 00:00 [oracle@12cnode1 wallet]$ Similarly scp the same wallet files from the primary node to the second node of the standby cluster as well.[oracle@12cnode1 wallet]$ scp *wallet.* oracle@1212drnode2:/u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb2/wallet/The authenticity of host '1212drnode2 (192.168.0.127)' can't be established.RSA key fingerprint is 0a:f2:b7:22:3b:2b:fa:32:ea:dc:17:c1:22:7c:4e:e3.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '1212drnode2,192.168.0.127' (RSA) to the list of known hosts.oracle@1212drnode2's password: cwallet.sso 100% 3893 3.8KB/s 00:00 ewallet.p12 100% 3848 3.8KB/s 00:00 [oracle@12cnode1 wallet]$ [oracle@1212drnode1 ~]$ srvctl status database -d srpstb -v -fInstance srpstb1 is running on node 1212drnode1. Instance status: Mounted (Closed).Instance srpstb2 is running on node 1212drnode2. Instance status: Mounted (Closed).[oracle@1212drnode1 ~]$ [oracle@1212drnode1 ~]$ Now its time to bounce the standby database and check the wallet status.[oracle@1212drnode1 ~]$ srvctl stop database -d srpstb[oracle@1212drnode1 ~]$ [oracle@1212drnode1 ~]$ [oracle@1212drnode1 ~]$ srvctl start database -d srpstb[oracle@1212drnode1 ~]$ srvctl status database -d srpstb -v -fInstance srpstb1 is running on node 1212drnode1. Instance status: Mounted (Closed).Instance srpstb2 is running on node 1212drnode2. Instance status: Mounted (Closed).[oracle@1212drnode1 ~]$ Glad to see that the wallets are opened on both the standby nodes and that the wallet type represents as AUTOLOGIN .[oracle@1212drnode1 ~]$ sqlSQL*Plus: Release 12.1.0.2.0 Production on Sat Sep 8 12:14:18 2018Copyright (c) 1982, 2014, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Advanced Analytics and Real Application Testing optionsSQL set linesize 300SQL col wrl_type for a10SQL col wallet_type for a10SQL col status for a15SQL col wrl_parameter for a70SQL select * from gv$encryption_wallet order by 1; INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYP WALLET_OR FULLY_BAC CON_ID---------- ---------- ---------------------------------------------------------------------- --------------- ---------- --------- --------- ---------- 1 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb1/wallet/ OPEN AUTOLOGIN SINGLE NO 0 2 FILE /u01/app/oracle/product/12.1.0.2/db_1/network/admin/srpstb2/wallet/ OPEN AUTOLOGIN SINGLE NO 0Finally, ensure that the standby database is in sync with the primary database.[oracle@12cnode1 u02]$ dgmgrl /DGMGRL for Linux: Version 12.1.0.2.0 - 64bit ProductionCopyright (c) 2000, 2013, Oracle. All rights reserved.Welcome to DGMGRL, type help for information.Connected as SYSDG.DGMGRL show configurationConfiguration - dgconfig Protection Mode: MaxPerformance Members: srprim - Primary database srpstb - Physical standby database Fast-Start Failover: DISABLEDConfiguration Status:SUCCESSCOPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine In this post, I m writing on how to implement TDE in a 11g RAC environment with DataguardEnvironment:Oracle RDBMS version: 11.2.0.3Primary nodes: node1, node2Standby nodes: drnode1, drnode2Primary Databaase: crmsdbStandby Database: crmdbFirstly, let me check if the standby database is in sync with the primary database.[oracle@node1]$ dgmgrl /DGMGRL for Linux: Version 11.2.0.3.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type help for information.Connected.DGMGRL show configurationConfiguration - crmdb_dg Protection Mode: MaxPerformance Databases: crmsdb - Primary database crmdb - Physical standby databaseFast-Start Failover: DISABLEDConfiguration Status:SUCCESSNow, add the below entry in the SQLNET.ORA file on all the nodes of the Primary database. The SQLNET.ORA file needs to be present in the location where the TNS_ADMIN points.The value for the DIRECTORY parameter in the below entry specifies the location where the wallet needs to be stored. In my case, I m placing it ata $ORACLE_HOME/network/admin//wallet . If the specified directory structure does not exist, then you need to create one.ENCRYPTION_WALLET_LOCATION=(SOURCE= (METHOD=FILE) (METHOD_DATA=(DIRECTORY=/u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet))Please note that in a RAC environment, the wallet can be placed in a shared location where all the nodes of the cluster database can access it (Ideally, it can be placed on ASM storage which I shall expalin in my coming posts). In this environment, I m having a separate wallet for each node.Once you add the entry in SQLNET.ORA file, you need to bounce the instances so that wallet location is upated correctly when you query the gv$encryption_wallet view.As of now, I have not bounced the 2nd instance and due to which the wallet location is pointing to wrong location when I query the gv$encryption_wallet view.Hence a restart of the 2nd instance is needed.SQL select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- --------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED 2 file /u01/app/oracle/product/11.2.0.3/db_1/admin/crmsdb/wallet CLOSED[oracle@node1 admin]$ srvctl stop instance -i crmsdb2 -d crmsdb[oracle@node1 admin]$ [oracle@node1 admin]$ srvctl start instance -i crmsdb2 -d crmsdb[oracle@node1 admin]$ Now, querying the gv$encryption_wallet provides the expected result.SQL select * from gv$encryption_wallet order by inst_id; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- --------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED 2 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSEDOn any one instance of the primary database, create the wallet. Here, I m running it on the first node of the primary database.SQL alter system set encryption key identified by oracle123 System altered.Just crosscheck the wallet file presence under the wallet location.[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet[oracle@node1 wallet]$ ls -lrttotal 4-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12Now, SCP the wallet file [oracle@node1 wallet]$ scp ewallet.p12 oracle@node2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb2/wallet/ewallet.p12 100% 1573 1.5KB/s 00:00 [oracle@node1 wallet]$ You now manually need to open the walletSQL alter system set encryption wallet open identified by oracle123 System altered.SQL select * from gv$encryption_wallet order by 1; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- -------------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN 2 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPENIn order to close the wallet manually, you can use the below command.SQL alter system set encryption wallet close identified by oracle123 System altered.It is recommended to enable the wallet for auto login. This will ensure that there is no need to open or close the wallet manually. Oracle will automatically open the wallet when an encrypted object is accessed or created. This can be done by using the orapki utility.[oracle@node1 wallet]$ orapki wallet create -wallet /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet -auto_loginOracle PKI Tool : Version 11.2.0.3.0 - ProductionCopyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.Enter wallet password: [oracle@node1 wallet]$When prompted for the wallet password, provide the password that was used previously to create the key.[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet/[oracle@node1 wallet]$ ls -lrttotal 8-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12-rw-------. 1 oracle oinstall 1651 Aug 25 13:47 cwallet.sso[oracle@node1 wallet]$ scp cwallet.sso oracle@node2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb2/wallet/cwallet.sso 100% 1651 1.6KB/s 00:00 Now, bouncing the database will not stop you for opening the wallet as the above autologin option opens it automatically.[oracle@node1 ~]$ srvctl stop database -d crmsdb[oracle@node1 ~]$ [oracle@node1 ~]$ [oracle@node1 ~]$ srvctl start database -d crmsdb[oracle@node1 ~]$ sqlSQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 8 10:55:31 2018Copyright (c) 1982, 2011, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL set linesize 300SQL col wrl_parameter for a70SQL select * from gv$encryption_wallet order by 1; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- -------------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN 2 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPENNow on the DR nodes, follow the same steps as done on the primary nodes.Add same entry as above in sqlnet.ora file on all nodes of the standby and then restart the instances to get the wallet_location updated.[oracle@drnode1 admin]$ srvctl stop instance -i crmdb1 -d crmdb[oracle@drnode1 admin]$ srvctl start instance -i crmdb1 -d crmdb[oracle@drnode1 admin]$ srvctl stop instance -i crmdb2 -d crmdb[oracle@drnode1 admin]$ srvctl start instance -i crmdb2 -d crmdb[oracle@drnode1 admin]$ Once restarted, query the gv$encryption_wallet view to verify if WRL_PARAMETER value corresponds to that mentioned in the WALLET_LOCATION value in the SQLNET.ORA file.SQL select * from gv$encryption_wallet; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- --------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSED 2 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet CLOSEDOn the standby, MRP may terminateErrors with log +FRA/crmdb/archivelog/2018_08_25/thread_1_seq_64.376.985096395MRP0: Background Media Recovery terminated with error 28365Errors in file /u01/app/oracle/diag/rdbms/crmdb/crmdb1/trace/crmdb1_mrp0_12241.trc:ORA-28365: wallet is not openManaged Standby Recovery not using Real Time ApplyRecovery interrupted!Recovered data files to a consistent state at change 1361738MRP0: Background Media Recovery process shutdown (crmdb1) Now, do not create any key on the standby database. [oracle@node1 wallet]$ scp ewallet.p12 oracle@drnode1:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb1/wallet/oracle@drnode1's password: ewallet.p12 100% 1573 1.5KB/s 00:00 [oracle@node1 wallet]$ [oracle@node1 wallet]$ [oracle@node1 wallet]$ scp ewallet.p12 oracle@drnode2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb2/wallet/The authenticity of host 'drnode2 (192.168.0.123)' can't be established.RSA key fingerprint is 48:83:c6:32:28:0d:25:a8:5a:25:03:c7:21:96:6e:64.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'drnode2,192.168.0.123' (RSA) to the list of known hosts.oracle@drnode2's password: ewallet.p12 100% 1573 1.5KB/s 00:00 [oracle@node1 wallet]$ You can now open the wallet manually on the standby database using the below command:SQL alter system set encryption wallet open identified by oracle123 System altered.But since I have enabled autologin on the primary database, I d like the same to be enabled on the standby as well. So shall copy the cwallet.sso file from primary to both the nodes of the standby.[oracle@node1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmsdb1/wallet/[oracle@node1 wallet]$ ls -lrttotal 8-rw-r--r--. 1 oracle oinstall 1573 Aug 25 13:41 ewallet.p12-rw-------. 1 oracle oinstall 1651 Aug 25 13:47 cwallet.sso[oracle@node1 wallet]$ [oracle@node1 wallet]$ scp cwallet.sso oracle@drnode1:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb1/wallet/oracle@drnode1's password: cwallet.sso 100% 1651 1.6KB/s 00:00 [oracle@node1 wallet]$ [oracle@node1 wallet]$ scp cwallet.sso oracle@drnode2:/u01/app/oracle/product/11.2.0.3/db_1/network/admin/crmdb2/wallet/oracle@drnode2's password: cwallet.sso 100% 1651 1.6KB/s 00:00 [oracle@node1 wallet]$ [oracle@drnode1 ~]$ srvctl stop database -d crmdb[oracle@drnode1 ~]$ [oracle@drnode1 ~]$ srvctl start database -d crmdb[oracle@drnode1 ~]$ sqlSQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 8 11:17:50 2018Copyright (c) 1982, 2011, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL col wrl_parameter for a70SQL set linesize 300SQL select * from gv$encryption_wallet order by 1; INST_ID WRL_TYPE WRL_PARAMETER STATUS---------- -------------------- ---------------------------------------------------------------------- ------------------ 1 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN 2 file /u01/app/oracle/product/11.2.0.3/db_1/network/admin/$ORACLE_SID/wallet OPEN SQL select status,instance_name,database_role from v$database,v$instance;STATUS INSTANCE_NAME DATABASE_ROLE------------ ---------------- ----------------MOUNTED crmdb1 PHYSICAL STANDBYSQL select process,status,sequence#,inst_id,thread# from gV$managed_standby;PROCESS STATUS SEQUENCE# INST_ID THREAD#--------- ------------ ---------- ---------- ----------ARCH CONNECTED 0 1 0ARCH CONNECTED 0 1 0ARCH CONNECTED 0 1 0ARCH CONNECTED 0 1 0MRP0 APPLYING_LOG 71 1 2ARCH CLOSING 87 2 1ARCH CLOSING 70 2 2ARCH CONNECTED 0 2 0ARCH CONNECTED 0 2 0RFS IDLE 0 2 0RFS IDLE 0 2 0PROCESS STATUS SEQUENCE# INST_ID THREAD#--------- ------------ ---------- ---------- ----------RFS IDLE 88 2 1RFS IDLE 0 2 0RFS IDLE 71 2 2RFS IDLE 0 2 015 rows selected.[oracle@drnode1 ~]$ dgmgrl /DGMGRL for Linux: Version 11.2.0.3.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type help for information.Connected.DGMGRL show configurationConfiguration - crmdb_dg Protection Mode: MaxPerformance Databases: crmsdb - Primary database crmdb - Physical standby databaseFast-Start Failover: DISABLEDConfiguration Status:SUCCESSCOPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine In this article, I m demonstrating on how or what impact would be seen on the Physical standby database when you clone a Pluggable Database on the Primary.Here is the environment detail.Primary database name: targetdbStandby database name: tarstdbPDB name already plugged into CDB: targetpdbPDB name to be created: targetpdb1Please note that both TARGETPDB and TARGETPDB1 would be under the same container.Before I start, let me give a brief outcome of the setup.I have DG broker configured for the setup and I could see below that the standby database is in sync with the primary database.[oracle@ora1-1 ~]$ dgmgrl /DGMGRL for Linux: Version 12.1.0.2.0 - 64bit ProductionCopyright (c) 2000, 2013, Oracle. All rights reserved.Welcome to DGMGRL, type help for information.Connected as SYSDG.DGMGRL show configurationConfiguration - dgtest Protection Mode: MaxPerformance Members: targetdb - Primary database tarstdb - Physical standby databaseFast-Start Failover: DISABLEDConfiguration Status:SUCCESS (status updated 49 seconds ago)TARGETPDB1 will be created from TARGETPDB and would be plugged into the same ROOT container as TARGETPDBBelow shows the list of datafiles that TARGETPDB has.SYS@targetdb select status,instance_Name,database_role from v$database,v$Instance;STATUS INSTANCE_NAME DATABASE_ROLE------------ ---------------- ----------------OPEN targetdb PRIMARYSYS@targetdb show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 TARGETPDB READ WRITE NOFILE_NAME--------------------------------------------------------------------------------+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/system.272.906204413+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/sysaux.290.906204413+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/test.271.907334287+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/myobjects.291.930751499+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/migtbs1.279.930751523+DATA/TARGETDB/2DBEA8AC28E81AACE0536538A8C0F5B7/DATAFILE/users.273.9130391916 rows selected.Let me clone TARGETPDB to create TARGETPDB1. In order to clone, I need the TARGETPDB to be opened in READ ONLY mode which is done in the below steps.Logging into the CDB$ROOT container, I OPEN TARGETPDB in READ ONLY mode followed by which I issue the CREATE PLUGGABLE DATABASE command. Please note that while creating the PDB, I m also using the clause STANDBY=NONE .In case if I was cloning from PDB$SEED, then the datafiles of the newly created PDB would automatically get replicated on the physical standby side too without using the STANDBY=NONE clause. But, when a PDB is cloned from another PDB under the same container in a dataguard environment, then, the datafiles may not get replicated to the Physical standby side if the physical standby database is in MOUNT mode. In such a case, use the STANDBYS=NONE clause which would defer the recover of the newly created PDB on the physical standby database. The datafiles for the new PDB would be marked as OFFLINE/RECOVER in physical standby database and any additional redo for that PDB will be ignored. This is interpreted as at some point in the future, the PDBs files will be copied to the physical standby database and recovery for the PDB will be enabled. Here are the cloning and creation steps:On Primary, place the targetpdb in read only mode.SYS@targetdb alter pluggable database targetpdb close immediate;Pluggable database altered.SYS@targetdb alter pluggable database targetpdb open read only;Pluggable database altered.SYS@targetdb Now, create pluggable database targetpdb1 with the clause standbys=none .SYS@targetdb create pluggable database targetpdb1 from targetpdb standbys=none;Pluggable database created.SYS@targetdb SYS@targetdb show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 TARGETPDB READ ONLY NO 4 TARGETPDB1 MOUNTEDSYS@targetdb As seen above, since targetpdb is still in READ ONLY mode, we need to revert it back to READ WRITE mode. Also, we need to open the newly created pdb targetpdb1 .SYS@targetdb alter pluggable database targetpdb close immediate;Pluggable database altered.SYS@targetdb alter pluggable database targetpdb open;Pluggable database altered.SYS@targetdb alter pluggable database targetpdb1 open;Pluggable database altered.SYS@targetdb show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 TARGETPDB READ WRITE NO 4 TARGETPDB1 READ WRITE NOSYS@targetdb Moving on to the physical standby database side, let s check the status of the PDBs.SYS@tarstdb show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED MOUNTED 3 TARGETPDB MOUNTED 4 TARGETPDB1 MOUNTEDSYS@tarstdb On checking the recovery status of each of the PDBs that are plugged into the physical standby container database, we see that the recovery of the newly created PDB targetpdb1 is DISABLED and this is because we had used the clause standbys=none while creating the pdb.SYS@tarstdb select name, recovery_status from v$pdbs;NAME RECOVERY------------------------------ --------PDB$SEED ENABLEDTARGETPDB ENABLEDTARGETPDB1 DISABLEDSYS@tarstdb alter session set container=targetpdb1;Session altered.SYS@tarstdb select name, status from v$datafile;NAME STATUS------------------------------------------------------- -------/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021 SYSOFF/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026 RECOVER6 rows selected.When I query the v$recovery_file, I see that all the files of the newly created PDB seem to be errored out as Missing on the standby.SYS@tarstdb select * from v$recover_file; FILE# ONLINE ONLINE_ ERROR CHANGE# TIME CON_ID---------- ------- ------- ------------------------------ ---------- --------- ---------- 21 OFFLINE OFFLINE FILE MISSING 0 4 22 OFFLINE OFFLINE FILE MISSING 0 4 23 OFFLINE OFFLINE FILE MISSING 0 4 24 OFFLINE OFFLINE FILE MISSING 0 4 25 OFFLINE OFFLINE FILE MISSING 0 4 26 OFFLINE OFFLINE FILE MISSING 0 46 rows selected.Ok, so how do we get these files on to the standby ? Simple option would be to restore them from the Primary PDB (Targetpdb) through RMAN using the net service .I have DG Broker configured for my setup, so just making sure that the standby database is in sync with the primary database.[oracle@ora1-1 ~]$ dgmgrl /DGMGRL for Linux: Version 12.1.0.2.0 - 64bit ProductionCopyright (c) 2000, 2013, Oracle. All rights reserved.Welcome to DGMGRL, type help for information.Connected as SYSDG.DGMGRL show configurationConfiguration - dgtest Protection Mode: MaxPerformance Members: targetdb - Primary database tarstdb - Physical standby databaseFast-Start Failover: DISABLEDConfiguration Status:SUCCESS (status updated 9 seconds ago)Now, it s time for me to restore the files of the newly created PDB (targetpdb1) from primary to the standby site. I restore it using the service clause connecting to the primary.I use the set newname for pluggable database targetpdb1 to new to get the OMF format for the datafiles to be restored which would be done to the value specified in the db_create_file_dest parameter. Followed by this, is the restore pluggable database targetpdb1 from service targetdb suggesting to restore that particular PDB using the NET SERVICE to connect to primary. And finally switching the datafiles restored.[oracle@ora1-4 dbs]$ rman target /Recovery Manager: Release 12.1.0.2.0 - Production on Sat Mar 17 12:22:59 2018Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.connected to target database: TARGETDB (DBID=1297732342, not open)RMAN run3 set newname for pluggable database targetpdb1 to new;4 restore pluggable database targetpdb1 from service targetdb;5 switch datafile all;executing command: SET NEWNAMEStarting restore at 17-MAR-18Starting implicit crosscheck backup at 17-MAR-18using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=51 device type=DISKCrosschecked 14 objectsFinished implicit crosscheck backup at 17-MAR-18Starting implicit crosscheck copy at 17-MAR-18using channel ORA_DISK_1Crosschecked 11 objectsFinished implicit crosscheck copy at 17-MAR-18searching for all files in the recovery areacataloging files...no files catalogedusing channel ORA_DISK_1channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00021 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:26channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00022 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:35channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00023 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:04channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00024 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:01channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00025 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:04channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: using network backup set from service targetdbchannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00026 to +DATA_NEWchannel ORA_DISK_1: restore complete, elapsed time: 00:00:03Finished restore at 17-MAR-18RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of switch command at 03/17/2018 12:26:38ORA-19563: datafile copy header validation failed for file +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125RMAN Quite strange !! The datafiles of targetpdb1 are restored but errors out while performing the switch operation. This datafile was plugged in as part of transportable tablespace migration. Suspecting that could be the case but unsure why it failed during the execution of the switch command. I shall investigate on that and post in the upcoming days.As a workaround, I thought to rename those files manually. In order to do that, I set the StandbyFileManagement parameter to Manual on standby database and stop the RECOVERY on the standby database (tarstdb).DGMGRL edit database tarstdb set property StandbyFileManagement =Manual;Property standbyfilemanagement updatedDGMGRL edit database tarstdb set state='apply-off';Succeeded.The datafile status and it s name for the newly created PDB (targetpdb1) on standby site needs to be captured.SYS@tarstdb select name,status from v$datafile;NAME STATUS-------------------------------------------------------------------------------- -------/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021 SYSOFF/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025 RECOVER/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026 RECOVER6 rows selected.From the alert log of the standby database, I get the OMF format of the files restored in the above step for the new PDB (targetpdb1).Sat Mar 17 13:34:38 2018Full restore complete of datafile 21 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063. Elapsed time: 0:00:13 checkpoint is 10105360 last deallocation scn is 8421271 Undo Optimization current scn is 10090649Sat Mar 17 13:35:19 2018Full restore complete of datafile 22 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089. Elapsed time: 0:00:30 checkpoint is 10105405 last deallocation scn is 4478083Full restore complete of datafile 23 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125. Elapsed time: 0:00:00 checkpoint is 10105471 last deallocation scn is 1594145Full restore complete of datafile 24 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125. Elapsed time: 0:00:01 checkpoint is 10105481 last deallocation scn is 907776Full restore complete of datafile 25 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127. Elapsed time: 0:00:00 checkpoint is 10105484 last deallocation scn is 907776Full restore complete of datafile 26 to datafile copy +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129. Elapsed time: 0:00:01 checkpoint is 10105489 last deallocation scn is 3Sat Mar 17 13:37:45 2018Checker run found 13 new persistent data failuresI then rename each of these datafiles manually by connecting to the CDB$ROOT of the standby database. I understand that this is a tedious process, but for time being I had to go with this.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00021' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063';Database altered.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00022' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089';Database altered.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00023' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125';Database altered.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00024' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125';Database altered.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00025' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127';Database altered.SYS@tarstdb alter database rename file '/u01/app/oracle/product/12.1.0.2/db_1/dbs/UNNAMED00026' to '+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129';Database altered.Now, let me check the datafile names after renaming them. All looks good as below.SYS@tarstdb select name,status from v$datafile;NAME STATUS------------------------------------------------------------------------------------- -------+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/system.268.971012063 SYSOFF+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/sysaux.270.971012089 RECOVER+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/test.281.971012125 RECOVER+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/myobjects.282.971012125 RECOVER +DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/migtbs1.283.971012127 RECOVER+DATA_NEW/TARSTDB/6796FDF7366926AEE0536538A8C0B057/DATAFILE/users.263.971012129 RECOVER6 rows selected.We now need to enable the recovery for the newly created PDB targetpdb1 on standby database. For this, connect to the targetpdb1 container and enable the recovery.SYS@tarstdb show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED MOUNTED 3 TARGETPDB MOUNTED 4 TARGETPDB1 MOUNTEDSYS@tarstdb SYS@tarstdb alter session set container=targetpdb1;Session altered.SYS@tarstdb alter pluggable database enable recovery;Pluggable database altered.The changes made above to the parameter StandbyFileManagement needs to be reverted back to the value AUTO and start MRP on the standby database.DGMGRL edit database tarstdb set property StandbyFileManagement =AUTO;Property standbyfilemanagement updatedDGMGRL edit database tarstdb set state='apply-on';Succeeded.Crosscheck if all looks fine.SYS@tarstdb select name, recovery_status from v$pdbs;NAME RECOVERY------------------------------ --------PDB$SEED ENABLEDTARGETPDB ENABLEDTARGETPDB1 ENABLEDSYS@tarstdb alter session set container=targetpdb1;Session altered.SYS@tarstdb col name for a90SYS@tarstdb SYS@tarstdb select * from v$recovery_status;no rows selectedVerfiy if the standby database is in sync with the primary database.On Primary:SYS@targetdb select max(sequence#) from v$archived_log;MAX(SEQUENCE#)--------------SYS@targetdb alter system switch logfile;System altered.SYS@targetdb /System altered.SYS@targetdb /System altered.SYS@targetdb select max(sequence#) from v$archived_log;MAX(SEQUENCE#)--------------On standby:SYS@tarstdb select process,status,sequence# from v$managed_standby;PROCESS STATUS SEQUENCE#--------- ------------ ----------ARCH CLOSING 337ARCH CONNECTED 0ARCH CLOSING 336ARCH CLOSING 338RFS IDLE 0RFS IDLE 339MRP0 APPLYING_LOG 3397 rows selected.SYS@tarstdb select max(sequence#) from v$archived_log where applied='YES';MAX(SEQUENCE#)--------------All looks good !!COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine In this post, I shall be discussing on the steps involved in removing a node from RAC cluster in oracle 12c.Environment:Cluster Nodes: 12cnode1, 12cnode2, 12cnode3Database instances on the above nodes:srprim1 on 12cnode1srprim2 on 12cnode2srprim3 on 12cnode3Grid Infrastructure version: 12.1.0.2RDBMS version: 12.1.0.2Node to be removed: 12cnode3Here is a brief detail on the environment. The list of nodes captured using olsnodes .[oracle@12cnode3 ~]$ olsnodes12cnode112cnode212cnode3[oracle@12cnode3 ~]$Also this environment has Flex ASM enabled.[oracle@12cnode3 ~]$ asmcmd showclustermodeASM cluster : Flex mode enabled[oracle@12cnode3 ~]$[oracle@12cnode3 ~]$ asmcmd showclusterstateNormal[oracle@12cnode3 ~]$Below is the list of the resources managed by this cluster.[oracle@12cnode3 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details--------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE ONLINE ONLINE 12cnode3 STABLEora.DATA.dg ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE OFFLINE OFFLINE 12cnode3 STABLEora.FRA.dg ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE OFFLINE OFFLINE 12cnode3 STABLEora.LISTENER.lsnr ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE OFFLINE OFFLINE 12cnode3 STABLEora.net1.network ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE ONLINE ONLINE 12cnode3 STABLEora.ons ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE ONLINE ONLINE 12cnode3 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12cnode1.vip 1 ONLINE ONLINE 12cnode1 STABLEora.12cnode2.vip 1 ONLINE ONLINE 12cnode2 STABLEora.12cnode3.vip 1 ONLINE ONLINE 12cnode3 STABLEora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12cnode1 STABLEora.MGMTLSNR 1 ONLINE ONLINE 12cnode1 169.254.3.189 192.16 8.1.107,STABLEora.asm 1 ONLINE ONLINE 12cnode1 Started,STABLE 3 ONLINE ONLINE 12cnode2 Started,STABLEora.cvu 1 ONLINE ONLINE 12cnode1 STABLEora.mgmtdb 1 ONLINE ONLINE 12cnode1 Open,STABLEora.oc4j 1 ONLINE ONLINE 12cnode1 STABLEora.scan1.vip 1 ONLINE ONLINE 12cnode1 STABLEora.srprim.db 1 ONLINE ONLINE 12cnode1 Open,STABLE 2 ONLINE ONLINE 12cnode2 Open,STABLE 3 ONLINE ONLINE 12cnode3 Open,STABLEora.srprim.srprim_i03.svc 1 ONLINE ONLINE 12cnode1 STABLE--------------------------------------------------------------------------------Since Flex ASM is enabled, I see that all my oracle database instances are running on all the nodes even though ASM is not currently running on node 12cnode3. But that should not be a show stopper for us.[oracle@12cnode3 ~]$ srvctl status asmASM is running on 12cnode2,12cnode1[oracle@12cnode3 ~]$ crsctl get node role config -allNode '12cnode1' configured role is 'hub'Node '12cnode2' configured role is 'hub'Node '12cnode3' configured role is 'hub'[oracle@12cnode3 ~]$ crsctl get node role status -allNode '12cnode1' active role is 'hub'Node '12cnode2' active role is 'hub'Node '12cnode3' active role is 'hub'Below is the detail of the votedisk and OCR files.[oracle@12cnode3 ~]$ crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE eba1a2f1e10b4f60bfbe59bd50be0ae6 (/dev/DSK1) [DATA]Located 1 voting disk(s).[oracle@12cnode3 ~]$ ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 4 Total space (kbytes) : 409568 Used space (kbytes) : 1740 Available space (kbytes) : 407828 ID : 553047649 Device/File Name : +DATA Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user[oracle@12cnode3 ~]$Status of the database instances and the services currently running on the nodes.[oracle@12cnode3 ~]$ srvctl status database -db srprim -v -fInstance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.Instance srprim2 is running on node 12cnode2. Instance status: Open.Instance srprim3 is running on node 12cnode3. Instance status: Open.[oracle@12cnode3 ~]$Configuration of the database that is currently running on the cluster nodes.[oracle@12cnode3 ~]$ srvctl config database -db srprimDatabase unique name: srprimDatabase name: srprimOracle home: /u01/app/oracle/product/12.1.0.2/db_1Oracle user: oracleSpfile: +DATA/SRPRIM/PARAMETERFILE/spfile.291.904396381Password file: +DATA/SRPRIM/PASSWORD/pwdsrprim.276.904395317Domain:Start options: openStop options: immediateDatabase role: PRIMARYManagement policy: AUTOMATICServer pools:Disk Groups: FRA,DATAMount point paths:Services: srprim_i03Type: RACStart concurrency:Stop concurrency:OSDBA group: oinstallOSOPER group: oinstallDatabase instances: srprim1,srprim2,srprim3Configured nodes: 12cnode1,12cnode2,12cnode3Database is administrator managedBelow is the configuration of the service configured for the srprim database.[oracle@12cnode3 ~]$ srvctl config service -s srprim_i03 -db srprimService name: srprim_i03Server pool:Cardinality: 1Disconnect: falseService role: PRIMARYManagement policy: AUTOMATICDTP transaction: falseAQ HA notifications: falseGlobal: falseCommit Outcome: falseFailover type:Failover method:TAF failover retries:TAF failover delay:Connection Load Balancing Goal: LONGRuntime Load Balancing Goal: NONETAF policy specification: NONEEdition:Pluggable database name: srpdb1Maximum lag time: ANYSQL Translation Profile:Retention: 86400 secondsReplay Initiation Time: 300 secondsSession State Consistency:GSM Flags: 0Service is enabledPreferred instances: srprim3Available instances: srprim1,srprim2If you notice above, the service srprim_i03 is configured to run preferably on srprim3 instance (node 12cnode3). Hence we need to modify the preferred instances for service srprim_i03 else we would face error with message stating that Cannot delete selected instance since it belongs to following services srprim_i03 as only preferred instance. Modify the services and try again. So, let s modify the preferrend instances list for srprim_i03 service as srprim1 , and srprim2 .[oracle@12cnode3 ~]$ srvctl modify service -service srprim_i03 -db srprim -n -i srprim1,srprim2[oracle@12cnode3 ~]$ srvctl status service -s srprim_i03 -db srprimService srprim_i03 is running on instance(s) srprim1Stop and start the service to get the new configuration in place.[oracle@12cnode3 ~]$ srvctl stop service -s srprim_i03 -db srprim[oracle@12cnode3 ~]$ srvctl start service -s srprim_i03 -db srprim[oracle@12cnode3 ~]$ srvctl status service -s srprim_i03 -db srprimService srprim_i03 is running on instance(s) srprim1,srprim2[oracle@12cnode1 ~]$ srvctl status database -db srprim -v -fInstance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.Instance srprim2 is running on node 12cnode2 with online services srprim_i03. Instance status: Open.Instance srprim3 is running on node 12cnode3. Instance status: Open.Now all looks good, so let s move on with the node deletion. Removing node from a cluster involves 2 steps:1. Removing any database instance residing on the node that needs to be deleted.2. Removing the node from the cluster.Now, first, let s remove the database instance srprim3 residing on the node to be removed 12cnode3.Run dbca from different node of the cluster and not on the node which you are trying to remove. I m running dbca in silent mode here to delete the instance on the node which will be removed (srprim3).[oracle@12cnode1 ~]$ dbca -silent -deleteInstance -gdbName srprim -instanceName srprim3 -nodeList 12cnode3 -sysDBAUserName sys -sysDBAPassword oracleDeleting instance1% complete2% complete6% complete13% complete20% complete26% complete33% complete40% complete46% complete53% complete60% complete66% completeCompleting instance management.100% completeLook at the log file /u01/app/oracle/cfgtoollogs/dbca/srprim.log for further details.[oracle@12cnode1 ~]$Let s check the database and it s instances s status. Instance srprim3 has now been removed from node 12cnode3.[oracle@12cnode1 ~]$ srvctl status database -db srprim -v -fInstance srprim1 is running on node 12cnode1 with online services srprim_i03. Instance status: Open.Instance srprim2 is running on node 12cnode2 with online services srprim_i03. Instance status: Open.[oracle@12cnode1 ~]$From the database level, it s clear below that the thread information and the redo logs associated with srprim3 too have been removed.[oracle@12cnode1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 12.1.0.2.0 Production on Sun Sep 3 19:19:40 2017Copyright (c) 1982, 2014, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Advanced Analytics and Real Application Testing optionsSYS@srprim1 select group#,thread#,bytes/1024/1024 from v$log; GROUP# THREAD# BYTES/1024/1024---------- ---------- --------------- 1 1 50 2 1 50 3 2 50 4 2 50SYS@srprim1 select group#,member from v$logfile order by group#; GROUP# MEMBER---------- ------------------------------------------------------------ 1 +FRA/SRPRIM/ONLINELOG/group_1.257.904395571 1 +DATA/SRPRIM/ONLINELOG/group_1.282.904395571 2 +FRA/SRPRIM/ONLINELOG/group_2.258.904395577 2 +DATA/SRPRIM/ONLINELOG/group_2.283.904395575 3 +DATA/SRPRIM/ONLINELOG/group_3.289.904396373 3 +FRA/SRPRIM/ONLINELOG/group_3.259.904396375 4 +DATA/SRPRIM/ONLINELOG/group_4.290.904396377 4 +FRA/SRPRIM/ONLINELOG/group_4.260.9043963798 rows selected.SYS@srprim1 select name,value,inst_id from gv$parameter where name='undo_tablespace';NAME VALUE INST_ID------------------------------ -------------------------------------------------- ----------undo_tablespace UNDOTBS2 2undo_tablespace UNDOTBS1 1Now, on the node to be removed, we need to update the inventory file. So, on 12cnode3, we update the inventory file with the database home listing only the node to be removed (12cnode3). This needs to be done only on the node to be removed and is done by running oui utility.Current content of the inventory file on 12cnode3:[oracle@12cnode3 ContentsXML]$ cat inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME HOME NAME= OraDB12Home1 LOC= /u01/app/oracle/product/12.1.0.2/db_1 TYPE= O IDX= 2 NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode3 ContentsXML]$Updating the database home node list on 12cnode3. Please note that we need to specify the -local option to oui in order to update the inventory locally.[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/oui/bin/[oracle@12cnode3 bin]$[oracle@12cnode3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES={12cnode3} -localStarting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 10237 MB PassedThe inventory pointer is located at /etc/oraInst.loc'UpdateNodeList' was successful.[oracle@12cnode3 bin]$Below is the latest inventory file on 12cnode3.[oracle@12cnode3 ContentsXML]$ cat inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME HOME NAME= OraDB12Home1 LOC= /u01/app/oracle/product/12.1.0.2/db_1 TYPE= O IDX= 2 NODE_LIST NODE NAME= 12cnode3 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode3 ContentsXML]$Once the inventory is updated with the database home nodelist being only the node to be removed on 12cnode3, let s deinstall RDBMS home on 12cnode3.Again please note that we need to specify the -local option to the deinstall utility to remove the RDBMS binary on 12cnode3.[oracle@12cnode3 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/deinstall/[oracle@12cnode3 deinstall]$ ./deinstall -localChecking for required files and bootstrapping ...Please wait ...Location of logs /u01/app/oraInventory/logs/############ ORACLE DECONFIG TOOL START ##################################### DECONFIG CHECK OPERATION START ########################### [START] Install check configuration ##Checking for existence of the Oracle home location /u01/app/oracle/product/12.1.0.2/db_1Oracle Home type selected for deinstall is: Oracle Real Application Cluster DatabaseOracle Base selected for deinstall is: /u01/app/oracleChecking for existence of central inventory location /u01/app/oraInventoryChecking for existence of the Oracle Grid Infrastructure home /u01/app/12.1.0.2/gridThe following nodes are part of this cluster: 12cnode3,12cnode2,12cnode1Checking for sufficient temp space availability on node(s) : '12cnode3'## [END] Install check configuration ##Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2017-09-03_07-37-06-PM.logNetwork Configuration check config ENDDatabase Check Configuration STARTDatabase de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2017-09-03_07-37-16-PM.logUse comma as separator when specifying list of values as inputSpecify the list of database names that are configured locally on this node for this Oracle home. Local configurations of the discovered databases will be removed [srprim3,srprim3]:Database Check Configuration ENDOracle Configuration Manager check STARTOCM check log file location : /u01/app/oraInventory/logs//ocm_check936.logOracle Configuration Manager check END######################### DECONFIG CHECK OPERATION END ################################################ DECONFIG CHECK OPERATION SUMMARY #######################Oracle Grid Infrastructure Home is: /u01/app/12.1.0.2/gridThe following nodes are part of this cluster: 12cnode3,12cnode2,12cnode1The cluster node(s) on which the Oracle home deinstallation will be performed are:12cnode3Oracle Home selected for deinstall is: /u01/app/oracle/product/12.1.0.2/db_1Inventory Location where the Oracle home registered is: /u01/app/oraInventoryThe option -local will not modify any database configuration for this Oracle home.Checking the config status for CCROracle Home exists with CCR directory, but CCR is not configuredCCR check is finishedDo you want to continue (y - yes, n - no)? [n]: yA log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.out'Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.err'######################## DECONFIG CLEAN OPERATION START ########################Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2017-09-03_07-38-17-PM.logNetwork Configuration clean config STARTNetwork de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2017-09-03_07-38-17-PM.logNetwork Configuration clean config ENDOracle Configuration Manager clean STARTOCM clean log file location : /u01/app/oraInventory/logs//ocm_clean936.logOracle Configuration Manager clean END######################### DECONFIG CLEAN OPERATION END ################################################ DECONFIG CLEAN OPERATION SUMMARY #######################Cleaning the config for CCRAs CCR is not configured, so skipping the cleaning of CCR configurationCCR clean is finished#######################################################################Using properties file /tmp/deinstall2017-09-03_07-32-35PM/response/deinstall_2017-09-03_07-36-41-PM.rspLocation of logs /u01/app/oraInventory/logs/############ ORACLE DEINSTALL TOOL START ################################### DEINSTALL CHECK OPERATION SUMMARY #######################A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.out'Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-09-03_07-36-41-PM.err'######################## DEINSTALL CLEAN OPERATION START ########################## [START] Preparing for Deinstall ##Setting LOCAL_NODE to 12cnode3Setting CLUSTER_NODES to 12cnode3Setting CRS_HOME to falseSetting oracle.installer.invPtrLoc to /tmp/deinstall2017-09-03_07-32-35PM/oraInst.locSetting oracle.installer.local to true## [END] Preparing for Deinstall ##Setting the force flag to falseSetting the force flag to cleanup the Oracle BaseOracle Universal Installer clean STARTDetach Oracle home '/u01/app/oracle/product/12.1.0.2/db_1' from the central inventory on the local node : DoneDelete directory '/u01/app/oracle/product/12.1.0.2/db_1' on the local node : DoneThe Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is in use by Oracle Home '/u01/app/12.1.0.2/grid'.Oracle Universal Installer cleanup was successful.Oracle Universal Installer clean ENDClean install operation removing temporary directory '/tmp/deinstall2017-09-03_07-32-35PM' on node '12cnode3'## [END] Oracle install clean ######################### DEINSTALL CLEAN OPERATION SUMMARY #######################Successfully detached Oracle home '/u01/app/oracle/product/12.1.0.2/db_1' from the central inventory on the local node.Successfully deleted directory '/u01/app/oracle/product/12.1.0.2/db_1' on the local node.Oracle Universal Installer cleanup was successful.Oracle deinstall tool successfully cleaned up temporary directories.#######################################################################On the remaining nodes of the cluster, if we would check the inventory file, the database home would still list all the nodes. So, we need to update the inventory file on the remaining nodes with the node list under the Database home listing only the remaining nodes.On 12cnode1, let me update the node list under the Database Home using OUI utility.Current content of the inventory file on 12cnode1.[oracle@12cnode1 ~]$ cd /u01/app/oraInventory/ContentsXML/[oracle@12cnode1 ContentsXML]$ ls -lrttotal 12-rw-rw----. 1 oracle oinstall 817 Mar 24 2016 inventory.xml-rw-rw----. 1 oracle oinstall 292 Mar 24 2016 libs.xml-rw-rw----. 1 oracle oinstall 329 Mar 24 2016 comps.xml[oracle@12cnode1 ContentsXML]$ cat inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME HOME NAME= OraDB12Home1 LOC= /u01/app/oracle/product/12.1.0.2/db_1 TYPE= O IDX= 2 NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode1 ContentsXML]$Update the Database Home nodelist using oui with the cluster_nodes option listing the remaining nodes of the cluster.[oracle@12cnode1 ~]$ . oraenvORACLE_SID = [srprim1] ?The Oracle base remains unchanged with value /u01/app/oracle[oracle@12cnode1 ~]$ cd $ORACLE_HOME/oui/bin[oracle@12cnode1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES={12cnode1,12cnode2} Starting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 8125 MB PassedThe inventory pointer is located at /etc/oraInst.loc'UpdateNodeList' was successful.[oracle@12cnode1 bin]$Once the inventory is updated, let s verify the contents of the inventory file.[oracle@12cnode1 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / NODE NAME= 12cnode3 / /NODE_LIST /HOME HOME NAME= OraDB12Home1 LOC= /u01/app/oracle/product/12.1.0.2/db_1 TYPE= O IDX= 2 NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode1 bin]$So, we have now successfully removed the database instance and the RDBMS home from 12cnode3. Let me now proceed with removing the node from the cluster.Just briefing up, the olsnodes is listing out node 12cnode3. Further actions involve deconfiguring the cluster resources on 12cnode3 and then removing the GI home.[oracle@12cnode1 ~]$ . oraenvORACLE_SID = [srprim1] ? +ASM1The Oracle base remains unchanged with value /u01/app/oracle[oracle@12cnode1 ~]$[oracle@12cnode1 ~]$ olsnodes -s12cnode1 Active12cnode2 Active12cnode3 Active[oracle@12cnode1 ~]$[oracle@12cnode3 ~]$ olsnodes -s12cnode1 Active12cnode2 Active12cnode3 Active[oracle@12cnode3 ~]$On 12cnode3, deconfigure the cluster resources by running rootcrs.pl script with deconfig option from the GI home.[root@12cnode3 ~]# . oraenvORACLE_SID = [root] ? +ASM3The Oracle base has been set to /u01/app/oracle[root@12cnode3 ~]#[root@12cnode3 ~]# cd /u01/app/12.1.0.2/grid/crs/install[root@12cnode3 install]# ./rootcrs.pl -deconfig -forceUsing configuration parameter file: ./crsconfig_paramsNetwork 1 existsSubnet IPv4: 192.168.0.0/255.255.255.0/eth0, staticSubnet IPv6:Ping Targets:Network is enabledNetwork is individually enabled on nodes:Network is individually disabled on nodes:VIP exists: network number 1, hosting node 12cnode1VIP Name: 12cnode1-vip.mydomainVIP IPv4 Address: 192.168.0.117VIP IPv6 Address:VIP is enabled.VIP is individually enabled on nodes:VIP is individually disabled on nodes:VIP exists: network number 1, hosting node 12cnode2VIP Name: 12cnode2-vip.mydomainVIP IPv4 Address: 192.168.0.118VIP IPv6 Address:VIP is enabled.VIP is individually enabled on nodes:VIP is individually disabled on nodes:VIP exists: network number 1, hosting node 12cnode3VIP Name: 12cnode3-vipVIP IPv4 Address: 192.168.0.121VIP IPv6 Address:VIP is enabled.VIP is individually enabled on nodes:VIP is individually disabled on nodes:ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL falseONS is enabledONS is individually enabled on nodes:ONS is individually disabled on nodes:CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12cnode3'CRS-2673: Attempting to stop 'ora.crsd' on '12cnode3'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode3'CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode3'CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode3' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode3' has completedCRS-2677: Stop of 'ora.crsd' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.evmd' on '12cnode3'CRS-2673: Attempting to stop 'ora.storage' on '12cnode3'CRS-2673: Attempting to stop 'ora.ctssd' on '12cnode3'CRS-2673: Attempting to stop 'ora.mdnsd' on '12cnode3'CRS-2673: Attempting to stop 'ora.gpnpd' on '12cnode3'CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12cnode3'CRS-2677: Stop of 'ora.storage' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.asm' on '12cnode3'CRS-2677: Stop of 'ora.drivers.acfs' on '12cnode3' succeededCRS-2677: Stop of 'ora.asm' on '12cnode3' succeededCRS-2677: Stop of 'ora.ctssd' on '12cnode3' succeededCRS-2677: Stop of 'ora.evmd' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode3'CRS-2677: Stop of 'ora.mdnsd' on '12cnode3' succeededCRS-2677: Stop of 'ora.gpnpd' on '12cnode3' succeededCRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.cssd' on '12cnode3'CRS-2677: Stop of 'ora.cssd' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.crf' on '12cnode3'CRS-2677: Stop of 'ora.crf' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.gipcd' on '12cnode3'CRS-2677: Stop of 'ora.gipcd' on '12cnode3' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12cnode3' has completedCRS-4133: Oracle High Availability Services has been stopped.2017/09/03 20:05:03 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.2017/09/03 20:05:37 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.2017/09/03 20:05:40 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node[root@12cnode3 install]#Now, remove the node from the cluster using the crsctl utility.[root@12cnode1 ~]# . oraenvORACLE_SID = [root] ? +ASM1The Oracle base has been set to /u01/app/oracle[root@12cnode1 ~]#[root@12cnode1 ~]# crsctl delete node -n 12cnode3CRS-4661: Node 12cnode3 successfully deleted.[root@12cnode1 ~]#The following steps involve removing the GI home which is similar to what we followed while removing the RDBMS home.Firstly, update the inventory file on 12cnode3 with the information that this node is lonely and not part of the cluster. This is done by running the OUI utility from GI home.Options passed to OUI are as below:ORACLE_HOME: GI homeCLUSTER_NODES: 12cnode3 (The node to be removed)CRS=TRUE: as we are dealing with GI home and not RDBMS homelocal: which defines that the update needs to be done only on this node.[oracle@12cnode3 ~]$ iduid=501(oracle) gid=502(oinstall) groups=502(oinstall),501(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023[oracle@12cnode3 ~]$[oracle@12cnode3 ~]$ . oraenvORACLE_SID = [+ASM3] ?The Oracle base remains unchanged with value /u01/app/oracle[oracle@12cnode3 ~]$[oracle@12cnode3 ~]$ cd $ORACLE_HOME/oui/bin[oracle@12cnode3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES={12cnode3} CRS=TRUE -localStarting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 10237 MB PassedThe inventory pointer is located at /etc/oraInst.loc'UpdateNodeList' was successful.Verfiy the contents of the inventory file on 12cnode3.[oracle@12cnode3 bin]$[oracle@12cnode3 bin]$[oracle@12cnode3 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode3 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode3 bin]$Now, I can deinstall the GI home on 12cnode3 by running the deinstall script from the GI home.[oracle@12cnode3 ~]$ echo $ORACLE_SID+ASM3[oracle@12cnode3 ~]$ cd /u01/app/12.1.0.2/grid/deinstall/[oracle@12cnode3 deinstall]$ ./deinstall -localChecking for required files and bootstrapping ...Please wait ...Location of logs /tmp/deinstall2017-09-03_08-13-07PM/logs/############ ORACLE DECONFIG TOOL START ##################################### DECONFIG CHECK OPERATION START ########################### [START] Install check configuration ##Checking for existence of the Oracle home location /u01/app/12.1.0.2/gridOracle Home type selected for deinstall is: Oracle Grid Infrastructure for a ClusterOracle Base selected for deinstall is: /u01/app/oracleChecking for existence of central inventory location /u01/app/oraInventoryChecking for existence of the Oracle Grid Infrastructure home /u01/app/12.1.0.2/gridThe following nodes are part of this cluster: 12cnode3Checking for sufficient temp space availability on node(s) : '12cnode3'## [END] Install check configuration ##Traces log file: /tmp/deinstall2017-09-03_08-13-07PM/logs//crsdc_2017-09-03_08-17-00PM.logNetwork Configuration check config STARTNetwork de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/netdc_check2017-09-03_08-17-02-PM.logNetwork Configuration check config ENDAsm Check Configuration STARTASM de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/asmcadc_check2017-09-03_08-17-02-PM.logDatabase Check Configuration STARTDatabase de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/databasedc_check2017-09-03_08-17-02-PM.logDatabase Check Configuration END######################### DECONFIG CHECK OPERATION END ################################################ DECONFIG CHECK OPERATION SUMMARY #######################Oracle Grid Infrastructure Home is: /u01/app/12.1.0.2/gridThe following nodes are part of this cluster: 12cnode3The cluster node(s) on which the Oracle home deinstallation will be performed are:12cnode3Oracle Home selected for deinstall is: /u01/app/12.1.0.2/gridInventory Location where the Oracle home registered is: /u01/app/oraInventoryOption -local will not modify any ASM configuration.Do you want to continue (y - yes, n - no)? [n]: yA log of this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.out'Any error messages from this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.err'######################## DECONFIG CLEAN OPERATION START ########################Database de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/databasedc_clean2017-09-03_08-17-55-PM.logASM de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/asmcadc_clean2017-09-03_08-17-55-PM.logASM Clean Configuration ENDNetwork Configuration clean config STARTNetwork de-configuration trace file location: /tmp/deinstall2017-09-03_08-13-07PM/logs/netdc_clean2017-09-03_08-17-55-PM.logNetwork Configuration clean config END####################### DECONFIG CLEAN OPERATION SUMMARY #######################Oracle Clusterware was already stopped and de-configured on node 12cnode3 Oracle Clusterware is stopped and de-configured successfully.#######################################################################Using properties file /tmp/deinstall2017-09-03_08-13-07PM/response/deinstall_2017-09-03_08-16-54-PM.rspLocation of logs /tmp/deinstall2017-09-03_08-13-07PM/logs/############ ORACLE DEINSTALL TOOL START ################################### DEINSTALL CHECK OPERATION SUMMARY #######################A log of this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.out'Any error messages from this session will be written to: '/tmp/deinstall2017-09-03_08-13-07PM/logs/deinstall_deconfig2017-09-03_08-16-54-PM.err'######################## DEINSTALL CLEAN OPERATION START ########################## [START] Preparing for Deinstall ##Setting LOCAL_NODE to 12cnode3Setting CLUSTER_NODES to 12cnode3Setting CRS_HOME to trueSetting oracle.installer.invPtrLoc to /tmp/deinstall2017-09-03_08-13-07PM/oraInst.locSetting oracle.installer.local to true## [END] Preparing for Deinstall ##Setting the force flag to falseSetting the force flag to cleanup the Oracle BaseOracle Universal Installer clean STARTDetach Oracle home '/u01/app/12.1.0.2/grid' from the central inventory on the local node : DoneFailed to delete the directory '/u01/app/12.1.0.2/grid/dc_ocm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/has'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/ohasd/12cnode3'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/ohasd'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/crs/12cnode3'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/crs'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/css/12cnode3'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/css'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/evm/12cnode3'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth/evm'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/auth'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/wwg'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/diagnostics'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/sqlplus'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/demo'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/jdk'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/jlib'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ord'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/ologgerd/init/12cnode3.pid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/ologgerd/init/12cnode3'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ologgerd/init'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/ologgerd'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/cdata'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_start.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclsvmon'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scrctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sysresv'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orald'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dg4pwdO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/patchgen'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_register.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraenv'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsecho'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsallociO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmsort'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gsd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmcmdcore'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adrci'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osysmond.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapbind'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjob'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmodify'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdtool'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rmanO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clscfg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrdump'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/bndlchk'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scriptagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appagent.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsdepxaO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssdrim'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjobO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appvipcfg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlplus'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrconfig.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crstmpl.scr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rman'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tkprofO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgid0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/skgxpinfo'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssvfupgd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbgeu_run_action.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrcheck.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcroute'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ghappctl.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netmgr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diskmon'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfod'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbshut'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkpatch'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcasst'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aqxmlctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/renamedg0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emcrsp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsdiag.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadpsp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluutil'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gnsd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dg4pwd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapki'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oprocd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmshow.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sysresv0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrcheck'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsctl.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrpatch'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/nidO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologgerd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_relocate'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sclsspawn'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmoddn'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbstart'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlldrO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdmonitor'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgwrap'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clscfg.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agapacheas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sbttestO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okadriverstate'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnsping'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsdriverstate'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologdbg.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tkprof'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/octssd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmwatch'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfssinglefsmount'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osdbagrp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgvip'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgidwrap'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnrctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okinit0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmshow'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnptool.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsid.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/nid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evt.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmlogger'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cursize'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkstore'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impdpO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_preapply'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_start'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ojvmtc'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/jssu'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclsomon'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_apply'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssvfupgd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgeut'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmklib.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsalloci'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapcompare'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gipcd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crswrapexece.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agsiebsrvras'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtomcatas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradnssd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oracle'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osysmond'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ologdbg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdroot'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oranetmonitor.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/olsnodes'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdagent.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agsiebgtwyas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutls'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/impdp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnodes'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mapsga'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrap'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmpost.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dsml2ldif'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tstshm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadjava'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnsping0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/amdu'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lbuilder'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_monitor'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oerr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsotsO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/uidrvciO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcldr0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emdwgrd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extprocO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oc4jctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dropjava'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmkbin.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ghappctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/symfind'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/setasmgid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/maxmem'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsdb.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adapters'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ndfnceca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odnsd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gensyslib'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okdstry0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evminfo'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapwdO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmkbin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/fmputlhp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ohasd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cursizeO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/unzip'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extproc'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/schema'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genorasdksh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odnsd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kgmgrO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tstshmO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/usrvip'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sbttest'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genoccish'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oc4jctl.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpsappas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapki.bat'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_apply.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclumon.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnpd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aggoldengateas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genclntsh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orapwd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluvfy'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mapsga0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/onsctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkstore.bat'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmproxy'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oifcfg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapdelete'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsregistrymount'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldifmigrator'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnnfg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genclntst'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbua'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagsetup'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutlo.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_profile'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_getperm.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mdnsd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mkpatchO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oerr.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ncomp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stat'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraagent.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/schemasync'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rawutl0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/coraenv'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expdpO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/srvctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oifcfg.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfod.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extusrupgrade'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbhome'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfsizeO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/uidrvci'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutlo'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_unregister.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okinit'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsdb'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapaddmt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmwatch.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orabase'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/linkshlib'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okdstry'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/owm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agtctlO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crfsetenv'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_unregister'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpsbatchas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odig.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/agpspiaas'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/statusnc'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxchknlb'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/expdp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/olsnodes.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orarootagent.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_getperm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orion'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clsecho.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/qosctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsots'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjoboO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emdwgrd.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/xmlwf'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_relocate.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mdnsd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnpd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradaemonagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/umu'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gnsd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kgmgr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagcollection.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradism'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gennttab'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dumpsga'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/loadpspO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genagtsh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraxsl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/octssd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clssproxy.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/deploync'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/appvipcfg.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcroute0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/aqxmlctl.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_profile.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genezi'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rconfig'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dgmgrlO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/renamedg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrcO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclskd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmsort.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxegen'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odisrvreg'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cssdmonitor.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evminfo.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cemutls.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrconfig'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfed'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrc'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orionO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/eusm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ohasd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcsess'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdtool.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rhpctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_register'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/sqlldr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rdtool'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/echodo'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/hsdepxa'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ojvmjava'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/vipca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/relink'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapsearch'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/exp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/adrciO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/amduO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/asmcmd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmpost'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oidca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfs_client'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdload'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmlogger.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oracleO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/genksms'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oranetmonitor'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/cluvfyrac.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbfsize'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnodes.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/trcldr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnslsnr0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diskmon.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stop.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/plshprofO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/wrapO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/emcrsp.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oclskd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsreplcrs'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraping'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lmsgen'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/netca_deinst.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gipcd.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oklist0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orajaxb'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/scriptagent.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dumpsga0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oidprovtool'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_setperm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/clssproxy'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgevtf'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/imp'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gennfgt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/racgmain'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crsctl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oklist'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afddriverstate'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/tnslsnr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocssdrim.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/maxmemO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lcsscan'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/orarootagent'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsroot'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/osdbagrp0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ocrdump.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfedO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oradnssd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dgmgrl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/extjobo'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/oraxml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_initializer'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/diagcollection.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/kfodO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stat.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/zip'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsrepl_transport'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/acfsload'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/platform_common'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapadd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/plshprof'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbv'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/evmmklib'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/gpnptool'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/ldapmodifymt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lxinst'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okaload'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/fmputl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/odig'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/srvconfig'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/skgxpinfoO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_setperm.bin'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/geneziO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/rawutl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/mgmtca'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/dbvO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/crs_stop'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/okaroot'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/afdboot'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/bin/lsnrctl0'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/bin'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/javavm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/xdk'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/oracore'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ucp'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/ctss/init/12cnode3.pid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/ctss/init/12cnode3'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ctss/init'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/ctss'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/osysmond/init/12cnode3.pid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/osysmond/init/12cnode3'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/osysmond/init'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/osysmond'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/install'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/assistants'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/profiles/peer'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/profiles'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/pa'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/root'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/prdr'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets/peer'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1/wallets'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/new__12cnode1'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/profiles/peer'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/profiles'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/pa'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/root'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/prdr'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets/peer'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3/wallets'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507/stg__12cnode3'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp/gpnp_bcp__2016_3_23_18507'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/gpnp'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/tfa'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/racg'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/deinstall'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/cfgtoollogs'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/inventory'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ohasd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/sqlpatch'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/dbs'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/root.sh'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/OPatch'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/clone'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gipc'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/utl'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/admin/crf12cnode3.cfg'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run/crflogd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run/crfmond'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin/run'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/admin/crf12cnode3.ora'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/admin'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.l01'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.l02'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/localdump.hdr'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc/procdump.log'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db/12cnode3/proc'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db/12cnode3'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf/db'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crf'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/srvm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/cv'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/owm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/addnode'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/oc4j'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gnsd/init'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/gnsd'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/nls'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/scheduler'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/ldap'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/usm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/jdbc'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/eons'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmcred12.so'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/lib/stubs'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libba12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcorejava.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmddisk.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdlib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxn2.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/clntshcore.map'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntshcore.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfstoolsdriver.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/s0main.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.la'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcore12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnls12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboramysql12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcommon12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libuini12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpatchgensh12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsroot.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordimt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgx12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naeet.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nnfgt.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvm12.so0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libosbws12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcell12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfssinglefsmount.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.11.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagtsh.so.1.0'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippdcemerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgns12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_afdlib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskvol12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpksse4212.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdambr.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpd.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdroot.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/sscoreed.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsnls12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnoname12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdaudit.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcp12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcpaio12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplc12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsqlplus.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/librdjni12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_std.a.dbl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocijdbc12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnsgr12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntns12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcommand.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodm12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx212.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnetd12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libkubsagt12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpg.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsvml.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libacfs12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libwwg.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libonsx.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_base_non_fips.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgnsjni12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okaroot.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/activation.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmm.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcryptocme.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnz12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdlib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_asym.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libons.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippdcmerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipp_bz2.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpcompat.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxns.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libxml12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okalib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxp12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.12.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocrutl12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_accel_non_fips.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_base.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdtoolsdriver.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci.so.12.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmddiag.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnid.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcell12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/facility.lis'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libowm2.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdload.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcrf12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpsa12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libheteroxa12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdvol.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnzjs12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboraz.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfslib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnbeq12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_accel_fips.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2_sans_jaxp_services.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodm12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclnt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libeonsserver.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/oc4jctl_lib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdparser.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so.10.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdamdu.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmperl12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcryptocme.sig'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlparserv2_jaxp_services.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdbase.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afddriverstate.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okadriverstate.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplp12_pic.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdanlz.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordim12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libirc.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libn12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntshcore.so.12.1'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libgeneric12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsr12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nigcon.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okadriverstate.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsload.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapclnt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsregistrymount.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpls12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/sysliblist'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xschema.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okatoolsdriver.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libzx12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afdroot.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libons.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_cee.a.dbl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntst12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nautab.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ldflagsO'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsqlplus.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/nigtab.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippsemerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libncrypt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplp12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocr12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippsmerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvm12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liborion12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libzt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskgxpr.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnus12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclntsh12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsreplcrs.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntmq12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libntcps12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libeons.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnz12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libdbcfg12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ntcontab.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagfw12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/ldflags'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnro12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdexceptions.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libqsmashr.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_okalib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnque12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcxaguard.so.5'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcore.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdtmpl.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclib_jiio.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsregistrymount.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordim12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmhas12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdglobal.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libimf.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclient12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libztkg12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpls12_pic.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libcrf_mdb12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/oc4jctl_common.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libvsn12_cse.a.dbl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlmesg.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libxdb.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/afddriverstate.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnhost12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libccme_ecc_non_fips.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsroot.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okaload.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmql1.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapjclnt12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libhasgen12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naedhs.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsdriverstate.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipp_z.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libldapjclnt12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsql12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libplc12_pic.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnl12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libserver12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libctxc12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osntabst.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/scorept.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/http_client.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcpemerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagent12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdattr.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmclntsh12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnjni12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libctx12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libpatchgensh12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclntsh.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdpasswd.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libodmd12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboramysql12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdug.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnzst12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libskjcx12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xmlcomp2.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/okaroot.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libslax12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocrb12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_afdload.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libintlc.so.5'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsce12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okalib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdshare.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xsu12.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnfsodm12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libavstub12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libippcpmerged.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfslib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/clntsh.map'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfssinglefsmount.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liblxled.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpkavx212.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liboevm.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libocci.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libshpksse4212.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnnet12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_acfsdriverstate.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libagtsh.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipc1.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/acfsload.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liborabz2.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libordsdo12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/naect.o'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libexpat.so.1.5.2'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdsys.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_unix_linux_acfslib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libasmperl12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libafd12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libsrvmocr12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libclsra12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/xml.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libunls12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/mail.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmdxmlexceptions.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libnldap12.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libusmcrs12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libopc12.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libmql1.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/asmcmd_disk_header_format'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/libipc1.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/osds_okaload.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/liblzopro.a'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/lib/s_oc4jctl_lib.pm'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/lib'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/plsql'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/acfs'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/diskmon'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/admin'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/log/12cnode3/alert12cnode3.log'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/ctssd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/client'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gipcd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crsd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/racg'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/ohasd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gpnpd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/evmd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crflogd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/mdnsd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/srvm'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/gnsd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/cssd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/afd'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/crfmond'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3/xag'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/log/12cnode3'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/log'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/auth'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/demo'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/script'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/config'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/install'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/public'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/utl/12cnode3/crsconfig_fileperms'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/utl/12cnode3/crsconfig_dirs'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/utl/12cnode3'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/utl'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/trace'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/ohasd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/oka'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/init.ohasd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/12cnode3.pid'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/afd.sles'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/ohasd.sles'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/afd'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/init/12cnode3'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/init'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/lib'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/log'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/profile'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/mesg'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/diskmon.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/gpnp.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/cssd.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/asm.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/mdns.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/evm.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/drivers.acfs.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/daemon.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/application.tdf'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/ctss.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/storage.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/ohasdbase.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/generic.tdf'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/appvipx.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/TYPE_application.cap'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/crf.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/TYPE_generic.cap'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/driver.afd.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/oka.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/haip.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/gipc.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/registry.acfs.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/cssdmonitor.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/appvip.type'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/crs/template/crs.type'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/template'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs/sbs'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/crs'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/precomp'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/network'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/hs'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/dmu'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/opmn'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/instantclient'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/css'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/md'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/QOpatch'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/rest'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/evm'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/usableports.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/runstatus.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal/.buildid'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/internal'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jewt4.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/je-4.1.27.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/ojdbc6.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/share.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jdev-rt.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/Symlink.so'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/jsch.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/RATFA.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/je-5.0.84.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib/commons-io-2.2.jar'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/jlib'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/public.jks'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/getppid.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfasetup.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable/Wrap.pm'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable.pm'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/Text'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/discover_ora_stack.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/profiling.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl_lib.pm'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/patchtfa.sh'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/collectfiles.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/metric_iorm.pl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/bin'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.sunos'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.conf'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.aix'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.service'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.hpux'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/init.tfa.tmpl'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install/inittab.linux'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/install'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa.jks'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa_directories.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/bin/tnt'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/bin'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/conf/tnt.prop.tmpl'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/conf'. The directory is not empty.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-cli-1.0.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-lang-2.1.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xmlparserv2.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xternal.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-logging-1.1.jar'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/tnt.jar'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt/lib'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext/tnt'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/ext'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/input'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa_setup.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/tfa.md5'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/problemset.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/tasks.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/ignore_extensions.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/searchStrings.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/collect_all_directories.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/date_patterns.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns_internal.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/ignorefiles.txt'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/scanFileList.xml'. The file is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources/mask_strings.xml'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home/resources'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release/tfa_home'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa/release'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/tfa'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools/orachk'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/suptools'. The directory is not empty.Failed to delete the directory '/u01/app/12.1.0.2/grid/xag'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/perl'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/wlm'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/rootupgrade.sh'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/relnotes'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/mdns'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/rdbms'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/slax'. The directory is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid/oui'. The directory is in use.Failed to delete the file '/u01/app/12.1.0.2/grid/oraInst.loc'. The file is in use.Failed to delete the directory '/u01/app/12.1.0.2/grid'. The directory is not empty.Delete directory '/u01/app/12.1.0.2/grid' on the local node : Failed Delete directory '/u01/app/oraInventory' on the local node : DoneThe Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is not empty.Oracle Universal Installer cleanup was successful.Oracle Universal Installer clean ENDClean install operation removing temporary directory '/tmp/deinstall2017-09-03_08-13-07PM' on node '12cnode3'## [END] Oracle install clean ######################### DEINSTALL CLEAN OPERATION SUMMARY #######################Successfully detached Oracle home '/u01/app/12.1.0.2/grid' from the central inventory on the local node.Failed to delete directory '/u01/app/12.1.0.2/grid' on the local node.Successfully deleted directory '/u01/app/oraInventory' on the local node.Oracle Universal Installer cleanup was successful.Run 'rm -r /opt/ORCLfmap' as root on node(s) '12cnode3' at the end of the session.Run 'rm -r /etc/oratab' as root on node(s) '12cnode3' at the end of the session.Oracle deinstall tool successfully cleaned up temporary directories.#######################################################################The above error message can be ignored to remove the GI home.Now, update the inventory file on the remaining nodes that the node list does not contain 12cnode3.[oracle@12cnode1 ~]$ . oraenvORACLE_SID = [+ASM1] ?The Oracle base remains unchanged with value /u01/app/oracle[oracle@12cnode1 ~]$[oracle@12cnode1 ~]$ cd $ORACLE_HOME/oui/bin[oracle@12cnode1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES={12cnode1,12cnode2} CRS=TRUE -silentStarting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 8126 MB PassedThe inventory pointer is located at /etc/oraInst.loc'UpdateNodeList' was successful.[oracle@12cnode1 bin]$Just verifying the inventory file post update.[oracle@12cnode1 bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml ?xml version= 1.0 standalone= yes ? !-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.All rights reserved. -- !-- Do not modify the contents of this file by hand. -- INVENTORY VERSION_INFO SAVED_WITH 12.1.0.2.0 /SAVED_WITH MINIMUM_VER 2.1.0.6.0 /MINIMUM_VER /VERSION_INFO HOME_LIST HOME NAME= OraGI12Home1 LOC= /u01/app/12.1.0.2/grid TYPE= O IDX= 1 CRS= true NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / /NODE_LIST /HOME HOME NAME= OraDB12Home1 LOC= /u01/app/oracle/product/12.1.0.2/db_1 TYPE= O IDX= 2 NODE_LIST NODE NAME= 12cnode1 / NODE NAME= 12cnode2 / /NODE_LIST /HOME /HOME_LIST COMPOSITEHOME_LIST /COMPOSITEHOME_LIST /INVENTORY [oracle@12cnode1 bin]$ olsnodes too now just shows 12cnode1 and 12cnode2 as the nodes of the cluster and not 12cnode3.[oracle@12cnode1 bin]$ olsnodes -s12cnode1 Active12cnode2 Active[oracle@12cnode1 bin]$Verify the cluster resources status from any one of the existing nodes.[oracle@12cnode1 bin]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details--------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLEora.DATA.dg ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLEora.FRA.dg ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLEora.LISTENER.lsnr ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLEora.net1.network ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLEora.ons ONLINE ONLINE 12cnode1 STABLE ONLINE ONLINE 12cnode2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12cnode1.vip 1 ONLINE ONLINE 12cnode1 STABLEora.12cnode2.vip 1 ONLINE ONLINE 12cnode2 STABLEora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12cnode1 STABLEora.MGMTLSNR 1 ONLINE ONLINE 12cnode1 169.254.3.189 192.16 8.1.107,STABLEora.asm 1 ONLINE ONLINE 12cnode1 Started,STABLE 3 ONLINE ONLINE 12cnode2 Started,STABLEora.cvu 1 ONLINE ONLINE 12cnode1 STABLEora.mgmtdb 1 ONLINE ONLINE 12cnode1 Open,STABLEora.oc4j 1 ONLINE ONLINE 12cnode1 STABLEora.scan1.vip 1 ONLINE ONLINE 12cnode1 STABLEora.srprim.db 1 ONLINE ONLINE 12cnode1 Open,STABLE 2 ONLINE ONLINE 12cnode2 Open,STABLEora.srprim.srprim_i03.svc 1 ONLINE ONLINE 12cnode1 STABLE 2 ONLINE ONLINE 12cnode2 STABLE--------------------------------------------------------------------------------[oracle@12cnode1 bin]$We can run the cluvfy utility to see if any messages are reported by post node deletion.[oracle@12cnode1 ~]$ cluvfy stage -post nodedel -n 12cnode3 -verbosePerforming post-checks for node removalChecking CRS integrity...The Oracle Clusterware is healthy on node 12cnode1 CRS integrity check passedClusterware version consistency passed.Result:Node removal check passedPost-check for node removal was successful.[oracle@12cnode1 ~]$All looks good now and the node is removed from the cluster.COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine This article speaks of how Flex ASM is configured and the advantage of having it. Flex ASM feature was introduced in 12c which allows the ASM instance to run on different node and not intended to run on each of the nodes in the cluster. This is configured based on the cardinality of the ASM. In versions prior to 12c, the ASM instance needed to be run on each of the nodes of the cluster. If the ASM instance was unable to start on any of the nodes, then the associated database instances too could not be started. But, from 12c, this is no more a concern.Let s see how Flex ASM can be configured and how it works.Environment:RAC nodes : 12cnode1, 12cnode2, 12cnode3Cluster version: 12.1.0.2Database Name: srprimInstance Name: srprim1 on 12cnode1, srprim2 on 12cnode2, srprim3 on 12cnode3OS Platform: OEL 6Below shows the list of nodes within the cluster.[oracle@12cnode1 ~]$ olsnodes12cnode112cnode212cnode3[oracle@12cnode1 ~]$Check the Cluster mode through ASMCMD. It s obvious that Flex mode would be disabled.[oracle@12cnode1 ~]$ asmcmdASMCMD showclustermodeASM cluster : Flex mode disabledFlex ASM requires a separate listener called ASMLISTENER to be configured on a different port number which is not being used by any other Listener. The other important thing is that FLEX ASM requires a separate network with which the ASM instances and it s clients communicate. You can also make use of the private network ethernet (used for inter node communication) as the network for the ASM instances and it s clients to communicate. In my case, I m using the private network itself as ASM network too.We now need to convert the Normal ASM to FlexASM. This can be done via ASMCA utility. Here I m doing this using ASMCA in silent mode and not through GUI.The options passed to ASMCA are:-silent: to run in silent mode-convertToFlexASM : to convert to FlexASM-asmNetworks: ASM Network to be used in the form of interface_name/Subnet -asmListenerPort: ASM Listener Port number to be usedRun this command on one node of the cluster.[oracle@12cnode1 ~]$ asmca -silent -convertToFlexASM -asmNetworks eth1/192.168.1.0 -asmListenerPort 1526To complete ASM conversion, run the following script as privileged user in local node./u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.shOnce done, as directed we need to run the converttoFlexASM.sh script as ROOT user on the node where ASMCA was run.This script configures ASM NET LISTENER on all the nodes of the cluster as well as stops and starts all cluster resources on all the nodes of the cluster sequentially.[root@12cnode1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.shCRS-2673: Attempting to stop 'ora.crsd' on '12cnode1'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode1'CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on '12cnode1'CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '12cnode1'CRS-2673: Attempting to stop 'ora.FRA.dg' on '12cnode1'CRS-2673: Attempting to stop 'ora.DATA.dg' on '12cnode1'CRS-2673: Attempting to stop 'ora.mgmtdb' on '12cnode1'CRS-2673: Attempting to stop 'ora.srprim.db' on '12cnode1'CRS-2677: Stop of 'ora.FRA.dg' on '12cnode1' succeededCRS-2677: Stop of 'ora.DATA.dg' on '12cnode1' succeededCRS-2677: Stop of 'ora.LISTENER.lsnr' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.12cnode1.vip' on '12cnode1'CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.scan1.vip' on '12cnode1'CRS-2677: Stop of 'ora.12cnode1.vip' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.12cnode1.vip' on '12cnode3'CRS-2677: Stop of 'ora.mgmtdb' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.MGMTLSNR' on '12cnode1'CRS-2677: Stop of 'ora.scan1.vip' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.scan1.vip' on '12cnode3'CRS-2677: Stop of 'ora.MGMTLSNR' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.MGMTLSNR' on '12cnode3'CRS-2677: Stop of 'ora.srprim.db' on '12cnode1' succeededCRS-2676: Start of 'ora.12cnode1.vip' on '12cnode3' succeededCRS-2676: Start of 'ora.scan1.vip' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on '12cnode3'CRS-2676: Start of 'ora.MGMTLSNR' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.mgmtdb' on '12cnode3'CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on '12cnode3' succeededCRS-2676: Start of 'ora.mgmtdb' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.ons' on '12cnode1'CRS-2677: Stop of 'ora.ons' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.net1.network' on '12cnode1'CRS-2677: Stop of 'ora.net1.network' on '12cnode1' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode1' has completedCRS-2677: Stop of 'ora.crsd' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.ctssd' on '12cnode1'CRS-2673: Attempting to stop 'ora.evmd' on '12cnode1'CRS-2673: Attempting to stop 'ora.storage' on '12cnode1'CRS-2677: Stop of 'ora.storage' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.asm' on '12cnode1'CRS-2677: Stop of 'ora.ctssd' on '12cnode1' succeededCRS-2677: Stop of 'ora.evmd' on '12cnode1' succeededCRS-2677: Stop of 'ora.asm' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode1'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.cssd' on '12cnode1'CRS-2677: Stop of 'ora.cssd' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.evmd' on '12cnode1'CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode1'CRS-2676: Start of 'ora.evmd' on '12cnode1' succeededCRS-2676: Start of 'ora.cssdmonitor' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.cssd' on '12cnode1'CRS-2672: Attempting to start 'ora.diskmon' on '12cnode1'CRS-2676: Start of 'ora.diskmon' on '12cnode1' succeededCRS-2676: Start of 'ora.cssd' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.ctssd' on '12cnode1'CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode1'CRS-2676: Start of 'ora.ctssd' on '12cnode1' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.asm' on '12cnode1'CRS-2676: Start of 'ora.asm' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.storage' on '12cnode1'CRS-2676: Start of 'ora.storage' on '12cnode1' succeededCRS-2672: Attempting to start 'ora.crsd' on '12cnode1'CRS-2676: Start of 'ora.crsd' on '12cnode1' succeededOracle Grid Infrastructure restarted in node 12cnode1PRCC-1014 : ASMNET1LSNR_ASM was already runningPRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already runningPRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnrCRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode2'CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode1'CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on '12cnode3'ASM listener ASMNET1LSNR_ASM running alreadyCRS-2673: Attempting to stop 'ora.crsd' on '12cnode2'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on '12cnode2'CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on '12cnode2'CRS-2673: Attempting to stop 'ora.cvu' on '12cnode2'CRS-2673: Attempting to stop 'ora.oc4j' on '12cnode2'CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode2'CRS-2673: Attempting to stop 'ora.srprim.db' on '12cnode2'CRS-2677: Stop of 'ora.cvu' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.cvu' on '12cnode1'CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on '12cnode2' succeededCRS-2677: Stop of 'ora.LISTENER.lsnr' on '12cnode2' succeededCRS-2677: Stop of 'ora.srprim.db' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.12cnode2.vip' on '12cnode2'CRS-2677: Stop of 'ora.12cnode2.vip' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.12cnode2.vip' on '12cnode3'CRS-2676: Start of 'ora.12cnode2.vip' on '12cnode3' succeededCRS-2676: Start of 'ora.cvu' on '12cnode1' succeededCRS-2677: Stop of 'ora.oc4j' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.oc4j' on '12cnode1'CRS-2673: Attempting to stop 'ora.DATA.dg' on '12cnode2'CRS-2673: Attempting to stop 'ora.FRA.dg' on '12cnode2'CRS-2677: Stop of 'ora.DATA.dg' on '12cnode2' succeededCRS-2677: Stop of 'ora.FRA.dg' on '12cnode2' succeededCRS-2676: Start of 'ora.oc4j' on '12cnode1' succeededCRS-2673: Attempting to stop 'ora.ons' on '12cnode2'CRS-2677: Stop of 'ora.ons' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.net1.network' on '12cnode2'CRS-2677: Stop of 'ora.net1.network' on '12cnode2' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on '12cnode2' has completedCRS-2677: Stop of 'ora.crsd' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.ctssd' on '12cnode2'CRS-2673: Attempting to stop 'ora.evmd' on '12cnode2'CRS-2673: Attempting to stop 'ora.storage' on '12cnode2'CRS-2677: Stop of 'ora.storage' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.asm' on '12cnode2'CRS-2677: Stop of 'ora.evmd' on '12cnode2' succeededCRS-2677: Stop of 'ora.ctssd' on '12cnode2' succeededCRS-2677: Stop of 'ora.asm' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode2'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode2' succeededCRS-2673: Attempting to stop 'ora.cssd' on '12cnode2'CRS-2677: Stop of 'ora.cssd' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode2'CRS-2672: Attempting to start 'ora.evmd' on '12cnode2'CRS-2676: Start of 'ora.evmd' on '12cnode2' succeededCRS-2676: Start of 'ora.cssdmonitor' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.cssd' on '12cnode2'CRS-2672: Attempting to start 'ora.diskmon' on '12cnode2'CRS-2676: Start of 'ora.diskmon' on '12cnode2' succeededCRS-2676: Start of 'ora.cssd' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.ctssd' on '12cnode2'CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode2'CRS-2676: Start of 'ora.ctssd' on '12cnode2' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.asm' on '12cnode2'CRS-2676: Start of 'ora.asm' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.storage' on '12cnode2'CRS-2676: Start of 'ora.storage' on '12cnode2' succeededCRS-2672: Attempting to start 'ora.crsd' on '12cnode2'CRS-2676: Start of 'ora.crsd' on '12cnode2' succeededOracle Grid Infrastructure restarted in node 12cnode2CRS-2673: Attempting to stop 'ora.crsd' on '12cnode3'CRS-2677: Stop of 'ora.crsd' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.ctssd' on '12cnode3'CRS-2673: Attempting to stop 'ora.evmd' on '12cnode3'CRS-2673: Attempting to stop 'ora.storage' on '12cnode3'CRS-2677: Stop of 'ora.storage' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.asm' on '12cnode3'CRS-2677: Stop of 'ora.evmd' on '12cnode3' succeededCRS-2677: Stop of 'ora.ctssd' on '12cnode3' succeededCRS-2675: Stop of 'ora.asm' on '12cnode3' failedCRS-2679: Attempting to clean 'ora.asm' on '12cnode3'CRS-2681: Clean of 'ora.asm' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12cnode3'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12cnode3' succeededCRS-2673: Attempting to stop 'ora.cssd' on '12cnode3'CRS-2677: Stop of 'ora.cssd' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.evmd' on '12cnode3'CRS-2672: Attempting to start 'ora.cssdmonitor' on '12cnode3'CRS-2676: Start of 'ora.evmd' on '12cnode3' succeededCRS-2676: Start of 'ora.cssdmonitor' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.cssd' on '12cnode3'CRS-2672: Attempting to start 'ora.diskmon' on '12cnode3'CRS-2676: Start of 'ora.diskmon' on '12cnode3' succeededCRS-2676: Start of 'ora.cssd' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.ctssd' on '12cnode3'CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12cnode3'CRS-2676: Start of 'ora.ctssd' on '12cnode3' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.asm' on '12cnode3'CRS-2676: Start of 'ora.asm' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.storage' on '12cnode3'CRS-2676: Start of 'ora.storage' on '12cnode3' succeededCRS-2672: Attempting to start 'ora.crsd' on '12cnode3'CRS-2676: Start of 'ora.crsd' on '12cnode3' succeededOracle Grid Infrastructure restarted in node 12cnode3[root@12cnode1 ~]#Now that the execution of the converttoFlexASM.sh script is completed, let s check the cluster mode.We should see as Flex mode enabled .[oracle@12cnode1 ~]$ asmcmdASMCMD showclustermodeASM cluster : Flex mode enabledLet s check the configuration of the ASM listener. It s clear below that the ASM cluster listener is configured on port 1526 and subnet that was specified while running ASMCA.[oracle@12cnode1 ~]$ srvctl config listener -l ASMNET1LSNR_ASMName: ASMNET1LSNR_ASMType: ASM ListenerOwner: oracleSubnet: 192.168.1.0Home: CRS home End points: TCP:1526Listener is enabled.Listener is individually enabled on nodes:Listener is individually disabled on nodes:[oracle@12cnode1 ~]$Let s check the status of this listener. It needs to be running on all the nodes of the cluster.[oracle@12cnode1 ~]$[oracle@12cnode1 ~]$ srvctl status listener -l ASMNET1LSNR_ASMListener ASMNET1LSNR_ASM is enabledListener ASMNET1LSNR_ASM is running on node(s): 12cnode3,12cnode2,12cnode1Also, we shall check the configuration of the ASM to check the cardinality and see if the cluster ASM listener is being listed.[oracle@12cnode1 ~]$ srvctl config asmASM home: CRS home Password file: +DATA/orapwASMASM listener: LISTENERASM instance count: 3Cluster ASM listener: ASMNET1LSNR_ASM[oracle@12cnode1 ~]$We see that ASM cardinality has been set to 3. Since this is a 3 node cluster and in order to check the functionality of FlexASM, I would change the cardinality of ASM to 2. By doing so, only 2 ASM instances will have to run.[oracle@12cnode1 ~]$ srvctl modify asm -count 2[oracle@12cnode1 ~]$[oracle@12cnode1 ~]$ srvctl config asmASM home: CRS home Password file: +DATA/orapwASMASM listener: LISTENERASM instance count: 2Cluster ASM listener: ASMNET1LSNR_ASMWe can see below that ASM is running only on 2 nodes (12cnode1 and 12cnode3) and no ASM instance is running on 12cnode2.[oracle@12cnode1 ~]$ srvctl status asmASM is running on 12cnode3,12cnode1Since ASM is not running on node 12cnode2, let s check if the associated database instance is running on node 12cnode2 or not.[oracle@12cnode3 ~]$ srvctl status database -db srprim -v -fInstance srprim1 is running on node 12cnode1. Instance status: Open.Instance srprim2 is running on node 12cnode2. Instance status: Open.Instance srprim3 is running on node 12cnode3. Instance status: Open.We can notice above that all the instances of database srprim are running on all the 3 nodes of the cluster which means that there is no impact seen on the associated database instance even if ASM is not running on a particular node.So now, that ASM is not running on 12cnode2 but database instance srprim2 is running, we need to figure out which ASM instance is used by database instance srprim2 .Let s check this from ASM3 instance: (anyways I m using gv$ views-so can be run on any ASM instance that s running)SQL select adg.name,adg.state,ac.instance_name,ac.db_name,adg.inst_id,ac.status,ac.cluster_name from gv$asm_diskgroup adg,gv$asm_client ac where adg.GROUP_NUMBER=ac.GROUP_NUMBER and adg.inst_id=ac.inst_id and ac.db_name='srprim' and adg.name='DATA' order by adg.inst_id;NAME STATE INSTANCE_NAME DB_NAME INST_ID STATUS CLUSTER_NAME--------------- ----------- -------------------- -------- ---------- ------------ -------------------------------DATA MOUNTED srprim1 srprim 1 CONNECTED node12c-scanDATA MOUNTED srprim3 srprim 3 CONNECTED node12c-scanDATA MOUNTED srprim2 srprim 3 CONNECTED node12c-scanIt s very clear from above that srprim2 instance is being served by ASM instance 3 on node 12cnode3 . In other words, ASM instance 3 has 2 clients 1. srprim2 from 12cnode2 and srprim3 from 12cnode3.The same can be viewed from the alert log file of ASM3 instance:NOTE: Flex client id 0x0 [srprim3:srprim:node12c-scan] attempting to connectNOTE: registered owner id 0x10002 for srprim3:srprim:node12c-scanNOTE: Flex client srprim3:srprim:node12c-scan registered, osid 23500, mbr 0x0, asmb 23305 (reg:4024785090)NOTE: client srprim3:srprim:node12c-scan mounted group 1 (DATA)NOTE: client srprim3:srprim:node12c-scan mounted group 2 (FRA)Sat Feb 25 12:28:03 2017NOTE: Flex client id 0x0 [srprim2:srprim:node12c-scan] attempting to connectNOTE: registered owner id 0x10003 for srprim2:srprim:node12c-scanNOTE: Flex client srprim2:srprim:node12c-scan registered, osid 23757, mbr 0x0, asmb 24702 (reg:1349866226)NOTE: client srprim2:srprim:node12c-scan mounted group 1 (DATA)NOTE: client srprim2:srprim:node12c-scan mounted group 2 (FRA)COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine In the previous post, we have seen on upgrading the Grid Infrastructure from 11.2.0.3 to 12.1.0.2. In this, we shall see the steps involved in the database upgrade from 11.2.0.3 to 12.1.0.2Environment:RAC nodes: drnode1, drnode2DB Name: srprimDB Instances: srprim1, srprim2Current DB version: 11.2.0.3.0DB to be upgraded to version: 12.1.0.2.0Cluster Storage used: ASMPlatform: OEL 6Current DB HOME: /u01/app/oracle/product/11.2.0.3/db_1New 12c DB HOME: /u01/app/oracle/product/12.1.0.2/db_1Since this is an out-of-place upgrade, firstly install Oracle 12.1.0.2 database software. In this environment, the software here has been unzipped to location /u03 and then the database software is installed using a response file in silent mode.Here is the response file used to install the 12.1.0.2 database software.[oracle@drnode1 ~]$ cd /u03/database[oracle@drnode1 database]$ ./runInstaller -silent -responseFile /u02/db_install.rsp -ignoreSysPrereqsStarting Oracle Universal Installer...Checking Temp space: must be greater than 500 MB. Actual 32711 MB PassedChecking swap space: must be greater than 150 MB. Actual 10229 MB PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2017-02-21_07-55-11PM. Please wait ...[oracle@drnode1 database]$ You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2017-02-21_07-55-11PM.logThe installation of Oracle Database 12c was successful.Please check '/u01/app/oraInventory/logs/silentInstall2017-02-21_07-55-11PM.log' for more details.As a root user, execute the following script(s): 1. /u01/app/oracle/product/12.1.0.2/db_1/root.shExecute /u01/app/oracle/product/12.1.0.2/db_1/root.sh on the following nodes:[drnode1, drnode2]Once the 12.1.0.2 database software has been installed, it s time for us to upgrade our database. Run the preupgrd.sql script available from the newly installed oracle 12c home on the 11.2 database. This script performs pre-requisite checks on the database to be upgraded and generates 2 scripts:1. preupgrade_fixups.sql which needs to be run on the database to fix any issues reported while performing the pre-checks.2. postupgrade_fixups.sql which needs to be run on the database once upgraded to 12.1.0.2 version.Let s run the preupgrd.sql script from the newly installed oracle 12.1.0.2 home (/u01/app/oracle/product/12.1.0.2/db_1) on the database.[oracle@drnode1 ~]$ ls -lrt /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql-rw-r--r--. 1 oracle oinstall 14083 May 15 2014 /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql[oracle@drnode1 ~]$[oracle@drnode1 ~]$ sqlplus sys/oracle@srprim as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 21:22:35 2017Copyright (c) 1982, 2011, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL @/u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/preupgrd.sql***************************************************************************Executing Pre-Upgrade Checks in SRPRIM...*************************************************************************** The following are *** ERROR LEVEL CONDITIONS *** that must be addressed prior to attempting your upgrade. Failure to do so will result in a failed upgrade. You MUST resolve the above errors prior to upgrade ************************************************************ ************************************************************ ==== PRE-UPGRADE RESULTS for SRPRIM ====ACTIONS REQUIRED:1. Review results of the pre-upgrade checks: /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade.log2. Execute in the SOURCE environment BEFORE upgrade: /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade_fixups.sql3. Execute in the NEW environment AFTER upgrade: /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/postupgrade_fixups.sql ***************************************************************************************************************************************Pre-Upgrade Checks in SRPRIM Completed.*********************************************************************************************************************************************************************************************************************************We see above that the preupgrd.sql script generated preupgrade_fixups.sql ( to fix any issues reported during pre-check phase) and postupgrade_fixups.sql (to be run post the upgrade) scripts that needs to be run on the database. Review the preupgrade.log and take necessary actions on the recommendations made.================================================================================================Preupgrade warning log:[oracle@drnode1 ~]$ cat /u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade.logOracle Database Pre-Upgrade Information Tool 02-21-2017 21:24:39Script Version: 12.1.0.2.0 Build: 006********************************************************************** Database Name: SRPRIM Container Name: Not Applicable in Pre-12.1 database Container ID: Not Applicable in Pre-12.1 database Version: 11.2.0.3.0 Compatible: 11.2.0.0.0 Blocksize: 8192 Platform: Linux x86 64-bit Timezone file: V14********************************************************************** [Update parameters] [Update Oracle Database 11.2.0.3.0 init.ora or spfile]-- If Target Oracle is 32-bit, refer here for Update Parameters:WARNING: -- processes needs to be increased to at least 300-- If Target Oracle is 64-bit, refer here for Update Parameters:WARNING: -- processes needs to be increased to at least 300******************************************************************************************************************************************** [Renamed Parameters] [No Renamed Parameters in use]******************************************************************************************************************************************** [Obsolete/Deprecated Parameters] [No Obsolete or Desupported Parameters in use]********************************************************************** [Component List]**********************************************************************-- Oracle Catalog Views [upgrade] VALID-- Oracle Packages and Types [upgrade] VALID-- JServer JAVA Virtual Machine [upgrade] VALID-- Oracle XDK for Java [upgrade] VALID-- Real Application Clusters [upgrade] VALID-- Oracle Workspace Manager [upgrade] VALID-- OLAP Analytic Workspace [upgrade] VALID-- Oracle Enterprise Manager Repository [upgrade] VALID-- Oracle Text [upgrade] VALID-- Oracle XML Database [upgrade] VALID-- Oracle Java Packages [upgrade] VALID-- Oracle Multimedia [upgrade] VALID-- Oracle Spatial [upgrade] VALID-- Expression Filter [upgrade] VALID-- Rule Manager [upgrade] VALID-- Oracle Application Express [upgrade] VALID-- Oracle OLAP API [upgrade] VALID********************************************************************** [Tablespaces]**********************************************************************-- SYSTEM tablespace is adequate for the upgrade. minimum required size: 1225 MB-- SYSAUX tablespace is adequate for the upgrade. minimum required size: 1509 MB-- UNDOTBS1 tablespace is adequate for the upgrade. minimum required size: 400 MB-- TEMP tablespace is adequate for the upgrade. minimum required size: 60 MB-- EXAMPLE tablespace is adequate for the upgrade. minimum required size: 310 MB [No adjustments recommended]******************************************************************************************************************************************** [Pre-Upgrade Checks]**********************************************************************WARNING: -- Process Count may be too low Database has a maximum process count of 150 which is lower than the default value of 300 for this release. You should update your processes value prior to the upgrade to a value of at least 300. For example: ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE or update your init.ora file.WARNING: -- Enterprise Manager Database Control repository found in the database In Oracle Database 12c, Database Control is removed during the upgrade. To save time during the Upgrade, this action can be done prior to upgrading using the following steps after copying rdbms/admin/emremove.sql from the new Oracle home - Stop EM Database Control: $ emctl stop dbconsole - Connect to the Database using the SYS account AS SYSDBA: SET ECHO ON; SET SERVEROUTPUT ON; @emremove.sql Without the set echo and serveroutput commands you will not be able to follow the progress of the script.INFORMATION: -- OLAP Catalog(AMD) exists in database Starting with Oracle Database 12c, OLAP Catalog component is desupported. If you are not using the OLAP Catalog component and want to remove it, then execute the ORACLE_HOME/olap/admin/catnoamd.sql script before or after the upgrade.INFORMATION: -- Older Timezone in use Database is using a time zone file older than version 18. After the upgrade, it is recommended that DBMS_DST package be used to upgrade the 11.2.0.3.0 database time zone version to the latest version which comes with the new release. Please refer to My Oracle Support note number 977512.1 for details.INFORMATION: -- There are existing Oracle components that will NOT be upgraded by the database upgrade script. Typically, such components have their own upgrade scripts, are deprecated, or obsolete. Those components are: OLAP Catalog,OWBINFORMATION: -- Oracle Application Express (APEX) can be manually upgraded prior to database upgrade APEX is currently at version 3.2.1.00.12 and will need to be upgraded to APEX version 4.2.5 in the new release. Note 1: To reduce database upgrade time, APEX can be manually upgraded outside of and prior to database upgrade. Note 2: See MOS Note 1088970.1 for information on APEX installation upgrades.********************************************************************** [Pre-Upgrade Recommendations]********************************************************************** ***************************************** ********* Dictionary Statistics ********* *****************************************Please gather dictionary statistics 24 hours prior toupgrading the database.To gather dictionary statistics execute the following commandwhile connected as SYSDBA: EXECUTE dbms_stats.gather_dictionary_stats;^^^ MANUAL ACTION SUGGESTED ^^^********************************************************************** [Post-Upgrade Recommendations]********************************************************************** ***************************************** ******** Fixed Object Statistics ******** *****************************************Please create stats on fixed objects two weeksafter the upgrade using the command: EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;^^^ MANUAL ACTION SUGGESTED ^^^********************************************************************** ************ Summary ************ 0 ERRORS exist in your database. 2 WARNINGS that Oracle suggests are addressed to improve database performance. 4 INFORMATIONAL messages that should be reviewed prior to your upgrade. After your database is upgraded and open in normal mode you must run rdbms/admin/catuppst.sql which executes several required tasks and completes the upgrade process. You should follow that with the execution of rdbms/admin/utlrp.sql, and a comparison of invalid objects before and after the upgrade using rdbms/admin/utluiobj.sql If needed you may want to upgrade your timezone data using the process described in My Oracle Support note 1509653.1 ***********************************Oracle suggested a few recommendations as seen above that needs to be carried out before performing the upgrade. Let s try running fixup.sql script that got generated previously. Fixup sql:[oracle@drnode1 ~]$ sqlplus sys/oracle@srprim as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 21:50:00 2017Copyright (c) 1982, 2011, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL @/u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/preupgrade_fixups.sqlIf the fixup.sql script is unable to fix the recommendations made, then fix them manually.Upgrade:Now that the recommendations have been fixed, let s move with the upgrade.Create a pfile from the spfile on first instance of 11.2.0.3 database to a temporary location and comment out the following parameters.1. instance_number2. thread3. 2nd instance's undo tablespace.4. cluster_database[oracle@drnode1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 21 22:10:00 2017Copyright (c) 1982, 2011, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL create pfile='/u02/srprim.ora' from spfile;File created.[oracle@drnode1 u02]$ cat /u02/srprim.ora | grep ^##*.cluster_database=true#srprim1.instance_number=1#srprim2.instance_number=2#srprim2.thread=2#srprim1.thread=1#srprim2.undo_tablespace='UNDOTBS2'Stop the database on 11.2.0.3 home [oracle@drnode1 u02]$ srvctl stop database -d srprimNow on the first node, set the environment variables pointing to the newly oracle 12c Home[oracle@drnode1 ~]$ export ORACLE_SID=srprim1[oracle@drnode1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/db_1[oracle@drnode1 ~]$ export PATH=$ORACLE_HOME/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin[oracle@drnode1 ~]$ which sqlplus/u01/app/oracle/product/12.1.0.2/db_1/bin/sqlplusStart the first instance in upgrade mode from oracle 12c home using the pfile that was created previously in the temporary location.[oracle@drnode1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 12.1.0.2.0 Production on Tue Feb 21 22:18:19 2017Copyright (c) 1982, 2014, Oracle. All rights reserved.Connected to an idle instance.SQL startup upgrade pfile='/u02/srprim.ora';ORACLE instance started.Total System Global Area 1073741824 bytesFixed Size 2932632 bytesVariable Size 729809000 bytesDatabase Buffers 335544320 bytesRedo Buffers 5455872 bytesDatabase mounted.Database opened.Now it s time to run the catupgrd.sql script available at 12c ORACLE HOME. Here, I m running this using the parallel utility catctl.pl with 5 number of parallel processes.[oracle@drnode1 ~]$ cd /u01/app/oracle/product/12.1.0.2/db_1/rdbms/admin/[oracle@drnode1 admin]$[oracle@drnode1 admin]$ nohup /u01/app/oracle/product/12.1.0.2/db_1/perl/bin/perl catctl.pl -n 5 catupgrd.sql /u02/upgrade.log Here is the sample of the upgrade process log. Review the log that was spooled for the execution of catupgrd.sql and once the execution is completed successfully, copy the pfile and password file of the instances from 11.2.0.3 home to 12.1.0.2 oracle home.Make sure that the pfile consists of the path to spfile and nothing else.On first node:[oracle@drnode1 admin]$ cat /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim1.oraSPFILE='+DATA/srprim/spfilesrprim.ora' # line added by Agent[oracle@drnode1 admin]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/orapwsrprim1 /u01/app/oracle/product/12.1.0.2/db_1/dbs/[oracle@drnode1 admin]$[oracle@drnode1 admin]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim1.ora /u01/app/oracle/product/12.1.0.2/db_1/dbs/[oracle@drnode1 admin]$On second node:[oracle@drnode2 ~]$ cat /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim2.oraSPFILE='+DATA/srprim/spfilesrprim.ora' # line added by Agent[oracle@drnode2 ~]$[oracle@drnode2 ~]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/orapwsrprim2 /u01/app/oracle/product/12.1.0.2/db_1/dbs/[oracle@drnode2 ~]$ cp /u01/app/oracle/product/11.2.0.3/db_1/dbs/initsrprim2.ora /u01/app/oracle/product/12.1.0.2/db_1/dbs/[oracle@drnode2 ~]$Now, start the first instance from new 12c Home and review the parameters. Post that, run the postupgrade_fixups.sql script that was generated earlier.[oracle@drnode1 admin]$ sqlplus / as sysdbaSQL*Plus: Release 12.1.0.2.0 Production on Wed Feb 22 01:43:55 2017Copyright (c) 1982, 2014, Oracle. All rights reserved.Connected to an idle instance.SQL startupORACLE instance started.Total System Global Area 1073741824 bytesFixed Size 2932632 bytesVariable Size 775946344 bytesDatabase Buffers 289406976 bytesRedo Buffers 5455872 bytesDatabase mounted.Database opened.SQL show parameter clusterNAME TYPE VALUE------------------------------------ ----------- ------------------------------cluster_database boolean TRUEcluster_database_instances integer 2cluster_interconnects stringExecution of postupgrade_fixups.sql:SQL @/u01/app/oracle/product/11.2.0.3/db_1/cfgtoollogs/srprim/preupgrade/postupgrade_fixups.sqlCATCTL REPORT = /u01/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/srprim/upgrade/upg_summary.logPL/SQL procedure successfully completed.Oracle Server VALID 12.1.0.2.0 00:45:29JServer JAVA Virtual Machine VALID 12.1.0.2.0 00:13:46Oracle Real Application Clusters VALID 12.1.0.2.0 00:00:06Oracle Workspace Manager VALID 12.1.0.2.0 00:02:58OLAP Analytic Workspace VALID 12.1.0.2.0 00:01:12OLAP Catalog OPTION OFF 11.2.0.3.0 00:00:00Oracle OLAP API VALID 12.1.0.2.0 00:02:09Oracle XDK VALID 12.1.0.2.0 00:02:17Oracle Text VALID 12.1.0.2.0 00:03:09Oracle XML Database VALID 12.1.0.2.0 00:06:44Oracle Database Java Packages VALID 12.1.0.2.0 00:00:54Oracle Multimedia VALID 12.1.0.2.0 00:07:37Spatial VALID 12.1.0.2.0 00:19:07Oracle Application Express VALID 4.2.5.00.08 01:02:58Final Actions 00:05:38Post Upgrade 00:00:11Total Upgrade Time: 02:55:35PL/SQL procedure successfully completed.SQL --SQL -- Update Summary Table with con_name and endtime.SQL --SQL UPDATE sys.registry$upg_summary SET reportname = :ReportName, 2 con_name = SYS_CONTEXT('USERENV','CON_NAME'), 3 endtime = SYSDATE 4 WHERE con_id = -1;1 row updated.SQL commit;Commit complete.SQL shut immediateDatabase closed.Database dismounted.ORACLE instance shut down.From 11.2 home, crosscheck if the database is down and then remove its 11.2 configuration from the clusterware managing. (In this post, I m removing the 11.2 database configuration from clusterware management and adding it back with 12c database configuration settings. You can avoid removing the old configuration and adding the new one by just running srvctl upgrade database -db -oraclehome from the 12c ORACLE HOME.[oracle@drnode1 ~]$ cd /u01/app/oracle/product/11.2.0.3/db_1/bin/[oracle@drnode1 bin]$ ./srvctl status database -d srprimInstance srprim1 is not running on node drnode1Instance srprim2 is not running on node drnode2[oracle@drnode1 bin]$ ./srvctl remove database -d srprimRemove the database srprim? (y/[n]) y[oracle@drnode1 bin]$Now, add this database srprim with 12c home configuration to the clusterware managing.[oracle@drnode1 bin]$ cd $ORACLE_HOME/bin[oracle@drnode1 bin]$ pwd/u01/app/oracle/product/12.1.0.2/db_1/bin[oracle@drnode1 bin]$[oracle@drnode1 bin]$ ./srvctl add database -d srprim -o /u01/app/oracle/product/12.1.0.2/db_1[oracle@drnode1 bin]$[oracle@drnode1 bin]$[oracle@drnode1 bin]$ ./srvctl add instance -i srprim1 -d srprim -n drnode1[oracle@drnode1 bin]$[oracle@drnode1 bin]$[oracle@drnode1 bin]$ ./srvctl add instance -i srprim2 -d srprim -n drnode2[oracle@drnode1 bin]$[oracle@drnode1 bin]$ ./srvctl status database -d srprim -v -fInstance srprim1 is not running on node drnode1Instance srprim2 is not running on node drnode2If there were any services configured previously, then create them as such from 12c ORACLE HOME.Start the database now using srvctl and also it s services.[oracle@drnode1 bin]$ ./srvctl add service -s srprim_any -d srprim -r srprim1,srprim2[oracle@drnode1 bin]$ ./srvctl start database -d srprim[oracle@drnode1 bin]$ ./srvctl start service -s srprim_any -d srprim[oracle@drnode1 bin]$ ./srvctl status database -d srprim -v -fInstance srprim1 is running on node drnode1 with online services srprim_any. Instance status: Open.Instance srprim2 is running on node drnode2 with online services srprim_any. Instance status: Open.COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine In this article, we shall see the steps involved in upgrading the Grid Infrastructure (CRS) from version 11.2.0.3 to 12.1.0.2 in silent mode.Environment:RAC nodes: drnode1, drnode2Current GI (CRS) version: 11.2.0.3.0GI to be upgraded to version: 12.1.0.2.0Cluster Storage used: ASMPlatform: OEL 6Current CRS HOME: /u01/app/11.2.0.3/gridNew 12c CRS HOME: /u01/app/12.1.0.2/gridBelow shows the current version of the CRS. The output is shown only for the first node.[oracle@drnode1 ~]$ crsctl query crs softwareversion drnode1Oracle Clusterware version on node [drnode1] is [11.2.0.3.0][oracle@drnode1 ~]$ crsctl query crs softwareversion drnode2Oracle Clusterware version on node [drnode2] is [11.2.0.3.0][oracle@drnode1 ~]$ crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.2.0.3.0]Download oracle 12.1.0.2 Grid Infrastructure and unzip. Here, the software is unzipped under /u03 .Let s run the cluvfy utility to perform pre-requisite checks before upgrading. This is achieved by running the runcluvfy.sh script from the unzipped 12c Grid Infrastructure software location.The parameters passed to this script are:-pre crsinst: To perform pre-checks before the CRS installation-upgrade: To perform upgrade pre-checks-rolling: To perform rolling upgrade-src_crshome: Location of the Source GI home-dest_crshome: Location of the Destination GI Home-dest_version: The version to which GI will be upgraded[oracle@drnode1 ~]$ cd /u03/grid[oracle@drnode1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/11.2.0.3/grid -dest_crshome /u01/app/12.1.0.2/grid -dest_version 12.1.0.2.0Performing pre-checks for cluster services setupChecking node reachability...Node reachability check passed from node drnode1 User equivalence check passed for user oracle Package existence check passed for cvuqdisk Check: Grid Infrastructure home writeability of path /u01/app/12.1.0.2/gridPRVG-11932 : Path /u01/app/12.1.0.2/grid cannot be created on node drnode1 .PRVG-11932 : Path /u01/app/12.1.0.2/grid cannot be created on node drnode2 .Grid Infrastructure home check failedChecking CRS user consistencyCRS user consistency check successfulChecking network configuration consistency.Check for network configuration consistency passed.Checking ASM disk size consistencyAll ASM disks are correctly sizedChecking if default discovery string is being used by ASMASM discovery string /dev/DSK* is not the default discovery stringChecking if ASM parameter file is in use by an ASM instance on the local nodeASM instance is using parameter file +DATA/drnode-scan/asmparameterfile/registry.253.936445691 on node drnode1 on which upgrade is requested.Checking OLR integrity...Check of existence of OLR configuration file /etc/oracle/olr.loc passedCheck of attributes of OLR configuration file /etc/oracle/olr.loc passedWARNING:This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.OLR integrity check passedChecking node connectivity...Checking hosts config file...Verification of the hosts config file successfulCheck: Node connectivity using interfaces on subnet 192.168.1.0 Node connectivity passed for subnet 192.168.1.0 with node(s) drnode1,drnode2TCP connectivity check passed for subnet 192.168.1.0 Check: Node connectivity using interfaces on subnet 192.168.0.0 Node connectivity passed for subnet 192.168.0.0 with node(s) drnode1,drnode2TCP connectivity check passed for subnet 192.168.0.0 Checking subnet mask consistency...Subnet mask consistency check passed for subnet 192.168.0.0 .Subnet mask consistency check passed for subnet 192.168.1.0 .Subnet mask consistency check passed.Node connectivity check passedChecking multicast communication...Checking subnet 192.168.1.0 for multicast communication with multicast group 224.0.0.251 ...Check of subnet 192.168.1.0 for multicast communication with multicast group 224.0.0.251 passed.Check of multicast communication passed.Task ASM Integrity check started...ASM Running check passed. ASM is running on all specified nodesDisk Group Check passed. At least one Disk Group configuredTask ASM Integrity check passed...Checking OCR integrity...Disks +DATA are managed by ASM.OCR integrity check passedChecking ASMLib configuration.Check for ASMLib configuration passed.Total memory check failedCheck failed on nodes: drnode2,drnode1Available memory check passedSwap space check passedFree disk space check passed for drnode2:/usr,drnode2:/var,drnode2:/etc,drnode2:/sbin Free disk space check passed for drnode1:/usr,drnode1:/var,drnode1:/etc,drnode1:/sbin Free disk space check passed for drnode2:/u01/app/11.2.0.3/grid Free disk space check passed for drnode1:/u01/app/11.2.0.3/grid Free disk space check passed for drnode2:/tmp Free disk space check passed for drnode1:/tmp Check for multiple users with UID value 501 passedUser existence check passed for oracle Group existence check passed for oinstall Group existence check passed for dba Membership check for user oracle in group oinstall [as Primary] passedMembership check for user oracle in group dba passedRun level check passedHard limits check passed for maximum open file descriptors Soft limits check passed for maximum open file descriptors Hard limits check passed for maximum user processes Soft limits check passed for maximum user processes There are no oracle patches required for home /u01/app/11.2.0.3/grid .There are no oracle patches required for home /u01/app/11.2.0.3/grid .Source home /u01/app/11.2.0.3/grid is suitable for upgrading to version 12.1.0.2.0 .System architecture check passedKernel version check failedCheck failed on nodes: drnode2,drnode1Kernel parameter check passed for semmsl Kernel parameter check passed for semmns Kernel parameter check passed for semopm Kernel parameter check passed for semmni Kernel parameter check passed for shmmax Kernel parameter check passed for shmmni Kernel parameter check passed for shmall Kernel parameter check passed for file-max Kernel parameter check passed for ip_local_port_range Kernel parameter check passed for rmem_default Kernel parameter check passed for rmem_max Kernel parameter check passed for wmem_default Kernel parameter check passed for wmem_max Kernel parameter check passed for aio-max-nr Kernel parameter check passed for panic_on_oops Package existence check passed for binutils Package existence check passed for compat-libcap1 Package existence check passed for compat-libstdc++-33(x86_64) Package existence check passed for libgcc(x86_64) Package existence check passed for libstdc++(x86_64) Package existence check passed for libstdc++-devel(x86_64) Package existence check passed for sysstat Package existence check passed for gcc Package existence check passed for gcc-c++ Package existence check passed for ksh Package existence check passed for make Package existence check passed for glibc(x86_64) Package existence check passed for glibc-devel(x86_64) Package existence check passed for libaio(x86_64) Package existence check passed for libaio-devel(x86_64) Package existence check passed for nfs-utils Check for multiple users with UID value 0 passedCurrent group ID check passedStarting check for consistency of primary group of root userCheck for consistency of root user's primary group passedStarting Clock synchronization checks using Network Time Protocol(NTP)...NTP configuration file /etc/ntp.conf existence check passedNo NTP Daemons or Services were found to be runningPRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):drnode2,drnode1Clock synchronization check using Network Time Protocol(NTP) failedCore file name pattern consistency check passed.User oracle is not part of root group. Check passedDefault user file creation mask check passedChecking integrity of file /etc/resolv.conf across nodes domain and search entries do not coexist in any /etc/resolv.conf fileAll nodes have same search order defined in file /etc/resolv.conf PRVF-5636 : The DNS response time for an unreachable node exceeded 15000 ms on following nodes: drnode1,drnode2Check for integrity of file /etc/resolv.conf failedUDev attributes check for OCR locations started...UDev attributes check passed for OCR locationsUDev attributes check for Voting Disk locations started...UDev attributes check passed for Voting Disk locationsTime zone consistency check passedChecking VIP configuration.Checking VIP Subnet configuration.Check for VIP Subnet configuration passed.Checking VIP reachabilityCheck for VIP reachability passed.Checking Oracle Cluster Voting Disk configuration...Oracle Cluster Voting Disk configuration check passedClusterware version consistency passed.Checking integrity of name service switch configuration file /etc/nsswitch.conf ...All nodes have same hosts entry defined in file /etc/nsswitch.conf Check for integrity of name service switch configuration file /etc/nsswitch.conf passedChecking daemon avahi-daemon is not configured and runningDaemon not configured check passed for process avahi-daemon Daemon not running check passed for process avahi-daemon Starting check for Reverse path filter setting ...Check for Reverse path filter setting passedStarting check for Network interface bonding status of private interconnect network interfaces ...Check for Network interface bonding status of private interconnect network interfaces passedStarting check for /dev/shm mounted as temporary file system ...Check for /dev/shm mounted as temporary file system passedStarting check for /boot mount ...Check for /boot mount passedStarting check for zeroconf check ...Check for zeroconf check passedPre-check for cluster services setup was unsuccessful on all the nodes.[oracle@drnode1 grid]$Some of the checks failed such as Clock synchronization check using Network Time Protocol(NTP) where NTP daemon or service was not running, DNS response check failed . I safely ignore these checks as both NTP and DNS are not configured on these nodes.Now, update the grid_install.rsp file available under the unzipped location of the 12c GI software. The response file needs to be updated appropriately based on the environment. Here is the updated grid_install.rsp file used for this upgrade.Before proceeding with the upgrade, unset the CRS HOME if set any on the environment and run the runInstaller from the unzipped location of the 12c GI software. Also, pass the parameters -silent to run the upgrade in SILENT mode and -responsefile specifying the location of the above updated response file.[oracle@drnode1 u02]$ cd /u03/grid[oracle@drnode1 grid]$[oracle@drnode1 grid]$ unset ORA_CRS_HOME[oracle@drnode1 grid]$ echo $ORA_CRS_HOME[oracle@drnode1 grid]$ nohup ./runInstaller -silent -ignorePrereq -responsefile /u02/grid_install.rsp [1] 16082[oracle@drnode1 grid]$ nohup: ignoring input and appending output to 'nohup.out'Let s view the output file to see what s happening.[oracle@drnode1 grid]$ cat nohup.outStarting Oracle Universal Installer...Checking Temp space: must be greater than 415 MB. Actual 32905 MB PassedChecking swap space: must be greater than 150 MB. Actual 10237 MB PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2017-02-21_10-10-33AM. Please wait ...[WARNING] [INS-41813] OSDBA for ASM, OSOPER for ASM, and OSASM are the same OS group. CAUSE: The group you selected for granting the OSDBA for ASM group for database access, and the OSOPER for ASM group for startup and shutdown of Oracle ASM, is the same group as the OSASM group, whose members have SYSASM privileges on Oracle ASM. ACTION: Choose different groups as the OSASM, OSDBA for ASM, and OSOPER for ASM groups.[WARNING] [INS-41874] Oracle ASM Administrator (OSASM) Group specified is same as the inventory group. CAUSE: Operating system group oinstall specified for OSASM Group is same as the inventory group. ACTION: It is not recommended to have OSASM group same as inventory group. Select any of the group other than the inventory group to avoid incorrect configuration.You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2017-02-21_10-10-33AM.logThe installation of Oracle Grid Infrastructure 12c was successful.Please check '/u01/app/oraInventory/logs/silentInstall2017-02-21_10-10-33AM.log' for more details.As a root user, execute the following script(s): 1. /u01/app/12.1.0.2/grid/rootupgrade.shExecute /u01/app/12.1.0.2/grid/rootupgrade.sh on the following nodes:[drnode1, drnode2]Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.Successfully Setup Software.As install user, execute the following script to complete the configuration. 1. /u01/app/12.1.0.2/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE= response_file Note: 1. This script must be run on the same host from where installer was run. 2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation).We now need to run the rootupgrade.sh script from the new 12c GI home as ROOT user on all the nodes. Post that, the runInstaller demands to run configToolAllCommands on the first node drnode1 where runInstaller was run.Below is the output of the execution of the rootupgrade.sh script on first node drnode1.[root@drnode1]# /u01/app/12.1.0.2/grid/rootupgrade.shCheck /u01/app/12.1.0.2/grid/install/root_drnode1.mydomain_2017-02-21_11-08-09.log for the output of root script[root@drnode1]#=====================================================================================================================[oracle@drnode1 grid]$ cat /u01/app/12.1.0.2/grid/install/root_drnode1.mydomain_2017-02-21_11-08-09.logPerforming root user operation.The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/12.1.0.2/grid Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root script.Now product-specific root actions will be performed.Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params2017/02/21 11:08:21 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.2017/02/21 11:09:12 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.2017/02/21 11:09:19 CLSRSC-464: Starting retrieval of the cluster configuration data2017/02/21 11:09:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.2017/02/21 11:09:38 CLSRSC-363: User ignored prerequisites during installation2017/02/21 11:09:58 CLSRSC-515: Starting OCR manual backup.2017/02/21 11:10:03 CLSRSC-516: OCR manual backup successful.2017/02/21 11:10:10 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode2017/02/21 11:10:10 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.3/grid -oldCRSVersion 11.2.0.3.0 -nodeNumber 1 -firstNode true -startRolling true'2017/02/21 11:10:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode2017/02/21 11:10:27 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack2017/02/21 11:12:46 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.OLR initialization - successful2017/02/21 11:20:36 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'CRS-4133: Oracle High Availability Services has been stopped.CRS-4123: Oracle High Availability Services has been started.2017/02/21 11:27:18 CLSRSC-472: Attempting to export the OCR2017/02/21 11:27:18 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'2017/02/21 11:27:32 CLSRSC-473: Successfully exported the OCR2017/02/21 11:27:39 CLSRSC-486: At this stage of upgrade, the OCR has changed. Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.2017/02/21 11:27:39 CLSRSC-541: To downgrade the cluster: 1. All nodes that have been upgraded must be downgraded.2017/02/21 11:27:39 CLSRSC-542: 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.2017/02/21 11:27:39 CLSRSC-543: 3. The downgrade command must be run on the node drnode1 with the '-lastnode' option to restore global configuration data.2017/02/21 11:28:05 CLSRSC-343: Successfully started Oracle Clusterware stackclscfg: EXISTING configuration version 5 detected.clscfg: version 5 is 11g Release 2.Successfully taken the backup of node specific configuration in OCR.Successfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.2017/02/21 11:28:37 CLSRSC-474: Initiating upgrade of resource types2017/02/21 11:29:48 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p first'2017/02/21 11:29:48 CLSRSC-475: Upgrade of resource types successfully initiated.2017/02/21 11:30:23 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded=====================================================================================================================Below is the output of the execution of the rootupgrade.sh script on the second (last) node drnode2.[root@drnode2 u01]# /u01/app/12.1.0.2/grid/rootupgrade.shCheck /u01/app/12.1.0.2/grid/install/root_drnode2.mydomain_2017-02-21_12-22-07.log for the output of root script[root@drnode2 u01]#==============================================================================================================[root@drnode2 u01]# cat /u01/app/12.1.0.2/grid/install/root_drnode2.mydomain_2017-02-21_12-22-07.logPerforming root user operation.The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/12.1.0.2/grid Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root script.Now product-specific root actions will be performed.Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params2017/02/21 12:22:13 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.2017/02/21 12:22:13 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.2017/02/21 12:22:22 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.2017/02/21 12:22:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.2017/02/21 12:22:38 CLSRSC-464: Starting retrieval of the cluster configuration data2017/02/21 12:22:47 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.2017/02/21 12:22:47 CLSRSC-363: User ignored prerequisites during installation2017/02/21 12:23:11 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack2017/02/21 12:25:21 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.OLR initialization - successful2017/02/21 12:26:47 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'CRS-4133: Oracle High Availability Services has been stopped.CRS-4123: Oracle High Availability Services has been started.2017/02/21 12:34:02 CLSRSC-343: Successfully started Oracle Clusterware stackclscfg: EXISTING configuration version 5 detected.clscfg: version 5 is 12c Release 1.Successfully taken the backup of node specific configuration in OCR.Successfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.Start upgrade invoked..2017/02/21 12:34:31 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded2017/02/21 12:34:31 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl set crs activeversion'Started to upgrade the Oracle Clusterware. This operation may take a few minutes.Started to upgrade the OCR.Started to upgrade the CSS.The CSS was successfully upgraded.Started to upgrade the CRS.The CRS was successfully upgraded.Successfully upgraded the Oracle Clusterware.Oracle Clusterware operating version was successfully set to 12.1.0.2.02017/02/21 12:37:03 CLSRSC-479: Successfully set Oracle Clusterware active version2017/02/21 12:37:14 CLSRSC-476: Finishing upgrade of resource types2017/02/21 12:37:27 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p last'2017/02/21 12:37:27 CLSRSC-477: Successfully completed upgrade of resource types2017/02/21 12:38:31 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded=======================================================================================================Now let s run the configToolAllCommands on drnode1. This script perform PostInstallation tasks such as creation of the Grid Infrastructure Management Repository Database with the Container Database Name as MGMTDB and it s Pluggabe Database Name as .[oracle@drnode1 bin]$ /u01/app/12.1.0.2/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/u02/grid_install.rspSetting the invPtrLoc to /u01/app/12.1.0.2/grid/oraInst.locperform - mode is starting for action: configureFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: UpdateNodelist data:Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.oui_loc:/u01/app/12.1.0.2/grid/ouiFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.jre_loc:/u01/app/12.1.0.2/grid/jdk/jreFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.doNotUpdateNodeList:trueFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.rootOwnedHome:trueFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: OracleHomeToUpdate:/u01/app/11.2.0.3/grid;isCRS:false;isCFS:false;isLocal:falseFeb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: From map: Hosts:[drnode1, drnode2] = Nodelist:[drnode1, drnode2]Feb 21, 2017 1:39:07 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: Before calling api: Hosts:[drnode1, drnode2] = Nodelist:[drnode1, drnode2], update localnode? trueFeb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: UpdateNodelist data:Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.oui_loc:/u01/app/12.1.0.2/grid/ouiFeb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.jre_loc:/u01/app/12.1.0.2/grid/jdk/jreFeb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.doNotUpdateNodeList:trueFeb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: oracle.installer.rootOwnedHome:Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: OracleHomeToUpdate:/u01/app/12.1.0.2/grid;isCRS:true;isCFS:false;isLocal:falseFeb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: From map: Hosts:[drnode1, drnode2] = Nodelist:[drnode1, drnode2]Feb 21, 2017 1:40:47 PM oracle.install.driver.oui.UpdateNodelistJob callINFO: Before calling api: Hosts:[drnode1, drnode2] = Nodelist:[drnode1, drnode2], update localnode? trueFeb 21, 2017 1:44:03 PM oracle.install.config.crs.MgmtDBCDBInternalPlugIn invokeINFO: MgmtDBCDBInternalPlugin: ... adding /oui_internal Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn invokeINFO: Executing MGMTDBCDBFeb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn invokeINFO: Command /u01/app/12.1.0.2/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName DATA -datafileJarLocation /u01/app/12.1.0.2/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internalFeb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn.handleProcess() entered.Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: getting configAssistantParmas.Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: checking secretArguments.Feb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: No arguments to pass to stdinFeb 21, 2017 1:44:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: starting read loop.Feb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Registering database with Oracle Grid InfrastructureFeb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Registering database with Oracle Grid InfrastructureFeb 21, 2017 1:44:45 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: End of argument passing to stdinFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 5% completeFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 5% completeFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Copying database filesFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Copying database filesFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 7% completeFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 7% completeFeb 21, 2017 1:44:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 9% completeFeb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 9% completeFeb 21, 2017 1:46:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 16% completeFeb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 16% completeFeb 21, 2017 1:46:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 23% completeFeb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 23% completeFeb 21, 2017 1:46:47 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 30% completeFeb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 30% completeFeb 21, 2017 1:47:37 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 37% completeFeb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 37% completeFeb 21, 2017 1:48:57 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 41% completeFeb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 41% completeFeb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Creating and starting Oracle instanceFeb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Creating and starting Oracle instanceFeb 21, 2017 1:49:03 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 43% completeFeb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 43% completeFeb 21, 2017 1:49:40 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 48% completeFeb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 48% completeFeb 21, 2017 1:50:22 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 49% completeFeb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 49% completeFeb 21, 2017 1:50:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 50% completeFeb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 50% completeFeb 21, 2017 1:51:43 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 55% completeFeb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 55% completeFeb 21, 2017 1:53:07 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 60% completeFeb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 60% completeFeb 21, 2017 1:53:08 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 61% completeFeb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 61% completeFeb 21, 2017 1:54:25 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 64% completeFeb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 64% completeFeb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Completing Database CreationFeb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Completing Database CreationFeb 21, 2017 1:54:26 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 68% completeFeb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 68% completeFeb 21, 2017 1:54:27 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 79% completeFeb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 79% completeFeb 21, 2017 1:54:34 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 89% completeFeb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 89% completeFeb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 100% completeFeb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 100% completeFeb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Look at the log file /u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb.log for further details.Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Look at the log file /u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb.log for further details.Feb 21, 2017 2:00:17 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:00:18 PM oracle.install.config.crs.MgmtDBPDBInternalPlugIn invokeINFO: MgmtDBPDBInternalPlugin: ... adding /oui_internal Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn invokeINFO: Executing MGMTDBPDBFeb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke strong INFO: Command /u01/app/12.1.0.2/grid/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName drnode_scan -createPDBFrom RMANBACKUP -PDBBackUpfile /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile /u01/app/12.1.0.2/grid/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck -oui_internal /strong Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn.handleProcess() entered.Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: getting configAssistantParmas.Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: checking secretArguments.Feb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: No arguments to pass to stdinFeb 21, 2017 2:00:18 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: ... GenericInternalPlugIn: starting read loop.Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Creating Pluggable DatabaseFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Creating Pluggable DatabaseFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: End of argument passing to stdinFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 4% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 4% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 12% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 12% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 21% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 21% completeFeb 21, 2017 2:01:21 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 38% completeFeb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 38% completeFeb 21, 2017 2:04:28 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 55% completeFeb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 55% completeFeb 21, 2017 2:04:38 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 85% completeFeb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 85% completeFeb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Completing Pluggable Database CreationFeb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Completing Pluggable Database CreationFeb 21, 2017 2:05:10 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: 100% completeFeb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: 100% completeFeb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Read: Look at the log file /u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/drnode_scan/_mgmtdb.log for further details.Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessWARNING: Skipping line: Look at the log file /u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/drnode_scan/_mgmtdb.log for further details.Feb 21, 2017 2:07:31 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcessINFO: Exceeded the number of arguments passed to stdin. CurrentCount:1 Total args:0perform - mode finished for action: configureYou can see the log file: /u01/app/12.1.0.2/grid/cfgtoollogs/oui/configActions2017-02-21_01-39-03-PM.logPost execution of all the demanded scripts, let s check the latest CRS version on both the nodes.On drnode1:[oracle@drnode1 ~]$ crsctl query crs activeversion -fOracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].[oracle@drnode1 ~]$[oracle@drnode1 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs releaseversionOracle High Availability Services release version on the local node is [12.1.0.2.0][oracle@drnode1 ~]$[oracle@drnode1 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs softwareversionOracle Clusterware version on node [drnode1] is [12.1.0.2.0]On drnode2:[oracle@drnode2 ~]$ crsctl query crs activeversion -fOracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].[oracle@drnode2 ~]$[oracle@drnode2 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs softwareversionOracle Clusterware version on node [drnode2] is [12.1.0.2.0][oracle@drnode2 ~]$ /u01/app/12.1.0.2/grid/bin/crsctl query crs releaseversionOracle High Availability Services release version on the local node is [12.1.0.2.0][oracle@drnode2 ~]$Verify that all ASM instances and all the diskgroups are MOUNTED on both the nodes.Finally, you can detach the Old GI home (11.2.0.3) from the ORAINVENTORY file.[oracle@drnode1 ~]$ cd /u01/app/11.2.0.3/grid/oui/bin/[oracle@drnode1 bin]$ ./runInstaller -detachHome -silent ORACLE_HOME=/u01/app/11.2.0.3/gridStarting Oracle Universal Installer...Checking swap space: must be greater than 500 MB. Actual 10236 MB PassedThe inventory pointer is located at /etc/oraInst.locThe inventory is located at /u01/app/oraInventory'DetachHome' was successful.COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine We all know that each instance of a RAC database has it s own buffer cache in its SGA and all these caches put together forms the cache fusion. The blocks within the RAC instances are transferred using a high speed private interconnect. In order to make sure that no two or more instances are updating the same block at the same time and also to track what each instance is performing with the block, cache coherency and consistency needs to be maintained. Oracle RAC maitains this cache coherency and consistency using two main services GCS (Global Cache Service)and GES (Global Enqueue Service).GCS with the help of LMS processes, co-ordinates within the RAC instances and records the status of the cached data blocks in the GRD (Global Resource Directory). The GRD is distributed across all the active instances and contains the information of the cached block such as the block number, which instance is owning the current version of the block, mode of the block, role of the block.GES maintains the coherency of enqueues on the dictionary and library cache. It keeps track of all global enqueues of the resources in the RAC environment.Based on what a resource holder is requesting to perform, a data block can be of any of the 3 modes.1. NULL (N): A null mode indicates just a resource holder and has no access rights.2. Exclusive (X): An exclusive mode signifies exclusive access of the block. This means that the resource holder needs to perform a write operation on the block and no other resource can write over it. However, other resources can perform read operation on the block.3. Shared (S): A shared mode indicates that the resource holder has a shared lock on the block and is performing a read operation. As the mode name, since the lock is shared, any other resource can also read the block.In addition to the modes, GCS also has roles for the resources.1. Local role: When a data block is first read from the disk into the cache, it s role is said to be LOCAL. This also means that no other modified/dirty copy of the block exists in the cache.2. Global Role: When a data block is acquired from a remote instance and if this data block is already modified in the remote instance, then the role of the block is said to be GLOBAL. If multiple copies of modified/dirtied data block exists across multiple instances, then the role of this data block is considered to be GLOBAL.Another important concept that we need to know of is the Past Image (PI). As the name says, an image copy of the modified/updated data block is saved by the modifying instance before passing on this data block to the other requesting instance that would like to perform either read or write operation. A PI block is considered to be the current version of the block and in case of a node failure, GCS would start the recovery from this PI block thereby reducing the recovery time. Once the latest version of the data block is written to the disk (checkpoint), GCS informs all the instances having PI to discard those images.All the above details speak about the data blocks, but when a block is read into the memory, it would be stored in the buffer. The state of a buffer is determined based on the 3 characters.1. Lock mode N (NULL), S (Shared), X (Exclusive)2. Role L(Local), G(Global)3. Past Image(PI): Number indicating the number of past imagesGV$BH view can be used to check the status of the block against an instance of the database. The values represent:1. cr : This represents a NULL lock mode on the block.2. SCUR : This represents that a SHARED lock is held on the block by that particular instance.3. XCUR : This represents an EXCLUSIVE lock is held on the block by that particular instance.Lets consider a 3 node RAC as an example to see how the block transfer works with user A connected to instance 1, user B connected to instance 2 and user C connected to instance 3.Assuming that the the table EMP under schema BTTEST is created freshly and no connections from any of the 3 instances have accessed this table, the following scenarios have been defined. But before moving on to the scenarios, let s capture the block relevant information for a row of this EMP table. This can be done with the help of dbms_rowid.SQL select * from bttest.emp;CODE NAME---------- --------------------------------------------------100 JAMES200 SCOTT300 SMITH400 JOHNSQL select owner,object_name,data_object_id from dba_objects where object_name='EMP';OWNER OBJECT_NAME DATA_OBJECT_ID--------- ------------ ----------------BTTEST EMP 91791SQL select dbms_rowid.rowid_relative_fno(rowid) FILE#,dbms_rowid.rowid_block_number(rowid) BLOCK# from bttest.emp where code=300;FILE# BLOCK#---------- ----------11 135Considering the block 135 which holds a row of the table EMP whose value for column CODE is 300, we are explaining the following scenarios.Scenario 1:User B on instance 2 performs a SELECT operation on this table which access the contents of the block 135.SQL statement run: select * from bttest.emp where code=300;1. Since no other connections from other instances have accessed this table previously, the data block 135 needs to be read from the disk and written on to the buffer.2. User B will now hold a Shared (S) lock on this block as it s performing only a READ and not WRITE operation. Since the block is being read for the first time from the disk and that there exists no dirty copy of this block in the buffer cache, the role of the block is LOCAL (L). The third consideration is that has been no modification done to this block by USER B and hence there exists no PAST IMAGES (0). With this, the mode of the block on instance 2 would be SL0 and nothing with respect to instance 1 and 3.Querying the GV$BH view to capture the details of the block, we see that the status of the block for instance 2 is SCUR which indicates that the block is held in SHARED lock by instance 2. There is no information for instance 1 and 3 as they haven t yet accessed this block.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 scur N 91791 2===========================================================================Scenario 2:Now user C on instance 3 performs an UPDATE operation on the row in the block 135. The SQL statement run is:update bttest.emp set name= UPDATE3 where code=300;Since the block is already available in the cache of instance B, there would be no requirement to read the block from the disk and thereby avoiding physical read.1. Instance 3 would send the request to GCS which knows who currently owns the block (using the information from GRD).2. GCS forwards the request to instance B who is currently holding the SHARED lock.3. Instance 2 would downgrade it s lock on the block from SHARED to NULL and since there is no modification done on this block by instance 2, the role of the block remains LOCAL and Past Image (PI) count is 0. Thereby, the mode of the block on instance 2 would be NL0.4. Instance 2 would then send the requested block to instance 3 and updates this to the GCS.5. Instance 3 will now go for an EXCLUSIVE lock (X) on the block and the role of the block still remains LOCAL as the instance previously holding this block (instance 2) hasn t made any modification to this block and so would the PI count be 0. The mode of the block on instance 3 would be XL0.6. Instance 1 has not come into picture yet with respect to this block.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 cr N 91791 211 135 xcur Y 91791 3From the above result, we can see that the buffer status on instance 2 is CR (NULL) and no modifications being done to the block. While on instance 3, the buffer status is XCUR (EXCLUSIVE) and instance 3 has dirtied (modified) (column DIRTY Value Y ) the block.===========================================================================Scenario 3:User A on instance 1 runs a select statement to access the row in block 135.1. Instance 1 requests GCS for the data block. GCS knows that instance 3 owns the block and has an EXCLUSIVE lock on it.2. GCS forwards the request to instance 3.3. Instance 3 would now bring down it s lock from Exclusive to Shared on the requested block. The lock would have been lowered to NULL if instance 1 would have requested the block to be modified (WRITE operation), instead, it has requested for a READ operation. Since instance 3 has modified the block in the previous scenario, the role of the block would be GLOBAL and GCS be informed that the requesting instance needs to have this block in the GLBOAL role.4. Since instance 3 has modified the block in scenario 2, it retains a copy (PI) of the modified block and sends the requested block. Instance 3 will now have SG1 mode on the block.5. Instance 1 receives the block and will hold a Shared (S) lock with Global Role (as informed previously) with 0 Past Images (there exists no past images for this block on instance 1). So, instance 1 will now have SG0 mode on the block.6. The mode of the block on instance 2 will be NG0. The role of the block will be changed to GLOBAL as instance 3 has modified the block in the previous scenario.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 cr N 91791 211 135 scur N 91791 111 135 scur Y 91791 3SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 scur N 91791 111 135 scur Y 91791 3When I query the GV$BH view, I could see that both instance 1 and 3 have SHARED lock on the block 135 whereas instance 2 with just CR mode (NULL). Since instance 3 has dirtied this block previously, the value for column DIRTY against instance 3 remains as Y (YES).===========================================================================Scenario 4:Now, I run an Update statement on the same block through User B from instance 2.1. Instance 2 requests GCS for the data block. The block was last accessed by instance 1 and GCS would forward the request to instance 1.2. Instance 1 would send the block to the requesting instance 2 through GCS. But, before passing on the block, instance 1 would downgrade it s lock on the block from Shared to NULL with the role of the block being GLOBAL and not holding any Past Images. The mode of the block on instance 1 would now be NG0.3. Since instance 2 has requested for WRITE operation, instance 3 will also downgrade it s lock on the block from SHARED to NULL. The role of the block remains GLOBAL and since it holds 1 PAST IMAGE (as per Scenario 3), the mode of the block on instance 3 will be NG1.4. Instance 2 will acquire an EXCLUSIVE (X) lock on the block with the role of the block retained as GLOBAL and the PI count remains 0. The PI count remains 0 for instance 2 is because this instance has never performed any modifications to this block and thereby not retaining any copies of the modifications. The mode of the block on instance 2 would be XG0.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 xcur Y 91791 211 135 cr N 91791 211 135 cr N 91791 211 135 pi Y 91791 311 135 cr N 91791 3SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 311 135 xcur Y 91791 2Querying GV$BH, one can see that the status of the buffer on instance 2 is XCUR with the value for the column DIRTY being Y . Also, it shows a clear image of the status of the buffer being shown as PI with the value for the column DIRTY being Y against instance 3. This is related to the scenario 2 where instance 3 performed an UPDATE operation on the block 135. The values CR for the column STATUS against instance 2 or instance 3 represent the status of the block in the previous scenarios and not to the current one.===========================================================================Scenario 5:User C on instance 3 runs a SELECT statement to access row in block 135.1. Instance 3 requests GCS for the block 135. GCS then forwards the request to instance 2 which is holding the block with an EXCLUSIVE lock.2. Instance 2 downgrades its lock from EXCLUSIVE to SHARED , adds a flag that it modified the block as stated in scenario 4 and thereby declaring that it has 1 PAST IMAGE. However the role of the block still remains GLBOAL and the mode of this block would be SG1 on instance 2.3. The block is then received by instance 3 and since it requested for a READ operation, it would acquire a SHARED lock on the block. The role of the block would be GLBOAL (as multiple changes have occurred to this block by remoted instances too). The PAST IMAGE count for this bock on instance 3 would be 1 (Reason there has been only 1 change made to this block by instance 3 (as explained in scenario 2)). Finally, the mode of the block on instance 3 will be SG1.4. Since instance 1 hasn t come into picture in this scenario, it would retain it s mode as it had in the previous scenario i.e, NG0.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 scur Y 91791 211 135 cr N 91791 211 135 cr N 91791 211 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 pi Y 91791 311 135 scur N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 rows selected.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 311 135 scur N 91791 311 135 scur Y 91791 2From the above result, we can see that the status of the block against instance 2 and instance 3 is SCUR (SHARED lock) and that NULL (cr) for instance 1.===========================================================================Scenario 6:User A on instance 1 performs an UPDATE operation on the same block 135.SQL statement run: update bttest.emp set name= UPDATE2 where code=300; 1. Instance 1 requests for the block with GCS. GCS knows that the block was last modified by instance 2 and is the latest one. Hence, it would request instance 2 to transfer the block to instance 1.2. Since instance 1 wants to perform a WRITE operation on the block, it would acquire an EXCLUSIVE lock. This in turn means that all other instances should downgrade their locks on this block to NULL. The role of the block would be GLBOAL and the PAST IMAGE count of this block on instance 1 would be 0 as this is the first time that this instance has requested for an UPDATE operation on this block. So no PAST IMAGES exist. As a result, the mode of the block on instance 1 would be XG0.3. As explained in previous step, instance 2 would have NULL lock on the block with GLBOAL role. The PAST IMAGE count would still be 1 because of the update operation it did as per scenario 4. The mode of the block 135 with respect to instance 2 would be NG1.4. Similarly the mode of the block on instance 3 would be NG1.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 211 135 cr N 91791 211 135 cr N 91791 211 135 pi Y 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 xcur Y 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 113 rows selected.SYS@srprim1 select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 211 135 pi Y 91791 311 135 xcur Y 91791 1We now have the status as PI which is in dirty state for instance 2 (scenario 4) and instance 3 (scenario 2) each and that for instance 1, the block status is XCUR (EXCLUSIVE).===========================================================================Scenario 7:User A on instance 1 performs another UPDATE operation on the same block.SQL statement run: update bttest.emp set name= UPDATE11 where code=300; 1. GCS knows that the last update operation performed on this block was instance 1 itself. Now, again instance 1 has requested for an EXCLUSIVE lock.2. Mode of the block on instance 1 would be XG1 (The Past image count is 1 for instance 1 as it previously performed as UPDATE operation as per scenario 6).3. The mode of the block with respect to instance 2 and 3 would remain as in the previous stage (NG1).SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 211 135 cr N 91791 211 135 cr N 91791 211 135 xcur Y 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 pi Y 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 314 rows selected.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 pi Y 91791 2 11 135 xcur Y 91791 111 135 pi Y 91791 3From the above result, instance 1 holds an XCUR status on the block 135 which indicates an EXCLUSIVE lock is held. While the other 2 instances (2 and 3) hold NULL lock which is why the value is cr .===========================================================================Scenario 8:User B on instance 2 performs a CHECKPOINT. (This signifies that the dirty blocks in the buffer needs to be written to the disk.)1. Instance 2 requests GCS for a checkpoint.2. GCS forwards the request to instance 1 who held the block in EXCLUSIVE mode in scenario 7, to write the block to disk.3. Instance 1 does the write operation of the block to disk and informs GCS of the completion of the operation, but would still retain it s lock as EXCLUSIVE on the block.4. GCS then informs all the instances holding PIs to discard or flush those PIs and requests the instances to change the role of the block from GLOBAL to LOCAL.5. Thus, instance 1 will now have the block with mode XL0, instance 2 and instance 3 with NULL.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135;FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 cr N 91791 211 135 cr N 91791 211 135 cr N 91791 211 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 cr N 91791 311 135 xcur N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 111 135 cr N 91791 114 rows selected.SQL select file#,block#,status,dirty,objd,inst_id from gv$bh where objd=91791 and block#=135 and status 'cr';FILE# BLOCK# STATUS D OBJD INST_ID---------- ---------- ---------- - ---------- ----------11 135 xcur N 91791 1Here, one can see that the block is currently held by instance 1 in EXCLUSIVE mode while all other instances have NULL lock on this block. All the past images that instance 2 and 3 had have been flushed.COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine This article should give you an overview of the new feature in Oracle 12c Global Services. We are aware of database services in oracle. These services provide a workload management by ensuring that clients connect to the optimal instance which is offering the service. It also serves the purpose of high availability by failing over the client connections to the other survival instances that are offering the same service. This is referred to as local data services.Oracle 12c came up with a new feature of Global Data Services which extends the above feature with automated and centralized workload management across the set of replicated databases like dataguard, golden gate. In addition to this, GDS provides replication lag based workload routing, role based global services and region based global services. Note that the global services can be configured only for databases of version 12c. For databases prior 12c, only local data services can be configured. One of the major question that would be raised is the difference between local and global data services A global data service is created across a set of multiple databases, whereas local data services is created only across a set of instances of a single database. With local data services, there is more of manual intervention needed such as a local data service for a standby needs to be first created on the primary database and then on the standby database. Whereas, with GDS, it s managed globally with a single service.A global data service configuration consists of some of the major components.GDS Pool: A GDS Pool is a set of databases containing replicated data in the GDS configuration that provide unique Global Data Services and are administered by different administrator.A database can belong to only one GDS pool. All databases in a GDS pool need not provide the same global service, but all databases that provide the same global service must belong to the same GDS pool.GDS Catalog: Just as we have a catalog database in RMAN which acts as a repository for the backup configuration of the registered database, likewise, a GDS Catalog is a repository that is used to store the GDS configuration. This catalog must reside in an oracle database of version 12c.This database may reside inside or outside the GDS configuration. Please note that a catalog is associated with only one GDS configuration.GSM (Global Service Manager): As the name says, GSM serves the role of a manager in managing the Global services, failover and load balancing of global services.It acts as a middle layer between the clients and the databases, just as a remote listener does in RAC databases. In addition to this, it measures the network latency between regions of the configuration by collecting the performance metrics from the databases in the configuration, monitors the databases and global services in the GDS configuration and notifies the clients when they fail.Client first connects to the GSM and requests for a connection to the global service. The GSM, then, forwards the connection request to the database instance in the GDS configuration that is offering the requested Global Service.GDS Region: A GDS region is a set of databases within the GDS configuration and the clients that share a very little network latency. A region can contain multiple GDS pools.You can configure, modify, start or stop global services using GDSCTL utility. To use this, you need to download and install the Oracle Global Service Manager software from the Oracle site.In the coming posts, we shall see on the installation of GSM software, configuring global services for the databases, how Global Services work with Dataguard and how effective it serves when there is a replication lag. COPYRIGHT© Shivananda Rao P, 2012 to 2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Shivananda Rao and http://www.shivanandarao-oracle.com with appropriate and specific direction to the original content.DISCLAIMERThe views expressed here are my own and do not necessarily reflect the views of any other individual, business entity, or organisation. The views expressed by visitors on this blog are theirs solely and may not reflect mine Here, in this article, I m demonstrating an upgrade from cloud control 12c to 13c on a linux machine. It is required to take a look at the https://docs.oracle.com/cd/E63000_01/EMUPG/preface.htm#EMUPG101 upgrade document before proceeding further and make sure that all the pre-requisites are met.Few things to consider before you proceed with the upgrade.1. OMS 13c repository database needs to be on 12.1.0.2 version. So, if the repository database is on a pre-12c release, then you need to upgrade it before upgrading the Cloud control.2. Cloud control 13c is not supported on OEL/RHEL 5 and so are the 13c agents. You cannot upgrade cloud control 12c on linux machine less than version 6 to 13c.Environment details is as below:The environment used here has the Cloud control 12c installed on OEL 6 with the repository database of version 12.1.0.2 which is a non-CDB and has the 12c agents deployed on linux machine of release 6 (OEL 6).Cloud Control 12c OMS hostname: ora1-2Repository database version: 12.1.0.2Make sure that the COMPATIBLE parameter on the OMS respository database is set to 12.1.0.2.0.SYS@omsdb show parameter compatibleNAME TYPE VALUE------------------------------------ ----------- ------------------------------compatible string 12.1.0.2.0noncdb_compatible boolean FALSESYS@omsdb SYS@omsdb show parameter adaptiveIf the adaptive optimizer feature (optimizer_adaptive_features) is enabled, then it needs to be disabled on the OMS repository database before upgrading it.NAME TYPE VALUE------------------------------------ ----------- ------------------------------optimizer_adaptive_features boolean TRUEoptimizer_adaptive_reporting_only boolean FALSEparallel_adaptive_multi_user boolean TRUESYS@omsdb SYS@omsdb alter system set optimizer_adaptive_features=false;System altered.SYS@omsdb show parameter adaptiveNAME TYPE VALUE------------------------------------ ----------- ------------------------------optimizer_adaptive_features boolean FALSEoptimizer_adaptive_reporting_only boolean FALSEparallel_adaptive_multi_user boolean TRUEIf there are any invalid objects in the OMS Repository database, then those need to be validated.Take a backup of the current OMS. Refer the documentation http://docs.oracle.com/cd/E24628_01/install.121/e24089/ha_backup_recover.htm#EMADV9639 #8221; on how to backup the OMS.Copy the EMKEY from the exsting OMS to the existing management repository. EMKEY is the encryption key which is used by Enterprise Manager to encrypt/decrypt sensitive data such as passwords, preferred credentials.[oracle@ora1-2 ~]$ echo $OMS_HOME/u02/oms12c/oms[oracle@ora1-2 ~]$ $OMS_HOME/bin/emctl config emkey -copy_to_reposOracle Enterprise Manager Cloud Control 12c Release 4Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.After the required operation has been completed, secure the EMKey by running emctl config emkey -remove_from_repos .[oracle@ora1-2 ~]$ $OMS_HOME/bin/emctl status emkeyOracle Enterprise Manager Cloud Control 12c Release 4Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :The EMKey is configured properly, but is not secure. Secure the EMKey by running emctl config emkey -remove_from_repos .[oracle@ora1-2 ~]$Stop all the components of OMS which includes WebTier, OMS and AdminServer.[oracle@ora1-2 u03]$ /u02/oms12c/oms/bin/emctl stop oms -allOracle Enterprise Manager Cloud Control 12c Release 4Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.Stopping WebTier...WebTier Successfully StoppedStopping Oracle Management Server...Oracle Management Server Successfully StoppedAdminServer Successfully StoppedOracle Management Server is Down[oracle@ora1-2 u03]$If there is an agent configured for the OMS server, then stop that too.[oracle@ora1-2 ~]$ /u02/12cagent/core/12.1.0.4.0/bin/emctl stop agentOracle Enterprise Manager Cloud Control 12c Release 4Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.Stopping agent ... stopped.Copy all the EM13c software parts into a staging directory. I have them copied into directory /u03/em13csoftware .[oracle@ora1-2 u03]$ ls -lrt /u03/em13csoftware | grep -i em -rw-r--r--. 1 oracle oinstall 1315250699 Jun 11 15:54 em13100_linux64-2.zip-rw-r--r--. 1 oracle oinstall 2145473190 Jun 11 16:05 em13100_linux64-3.zip-rw-r--r--. 1 oracle oinstall 2141357330 Jun 11 16:15 em13100_linux64-4.zip-rw-r--r--. 1 oracle oinstall 331590923 Jun 11 16:16 em13100_linux64-5.zip-rw-r--r--. 1 oracle oinstall 841114290 Jun 11 16:20 em13100_linux64.binSet the executable permission on the file em13100_linux64.bin which is the executable binary file.Execute the em13100_linux64.bin file to invoke the 13c GUI component.[oracle@ora1-2 em13csoftware]$ chmod +x em13100_linux64.bin[oracle@ora1-2 em13csoftware]$ ls -lrt em13100_linux64.bin-rwxr-xr-x. 1 oracle oinstall 841114290 Jun 11 16:20 em13100_linux64.bin[oracle@ora1-2 em13csoftware]$ ./em13100_linux64.bin0%...............................................................100%Launcher log file is /tmp/OraInstall2016-06-12_11-28-00AM/launcher2016-06-12_11-28-00AM.log.Starting Oracle Universal InstallerChecking if CPU speed is above 300 MHz. Actual 3180.718 MHz PassedChecking monitor: must be configured to display at least 256 colors. Actual 16777216 PassedChecking swap space: must be greater than 512 MB. Actual 10236 MB PassedChecking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)Checking temp space: must be greater than 300 MB. Actual 22122 MB Passed

TAGS:Oracle Rao Shivananda 

<<< Thank you for your visit >>>

Oracle Database issues & solutions

Hot Websites