ORA-15260: permission denied on ASM disk group

Problem

Try to set ASM diskgroup attribute, then get below errors:

$ sqlplus / as sysdba
...
..
.
SQL> alter diskgroup OCR_VOTE SET ATTRIBUTE 'compatible.asm' ='11.2.0.2';
alter diskgroup OCR_VOTE SET ATTRIBUTE 'compatible.asm' ='11.2.0.2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15260: permission denied on ASM disk group

Solution

Logon ASM instance with right privilege

$ sqlplus / as sysasm

...
..
.
SQL> alter diskgroup OCR_VOTE SET ATTRIBUTE 'compatible.asm' ='11.2.0.2';

Diskgroup altered.
Posted in ASM, GI and RAC, ORA- ERRORS, User Privileges and Roles | Tagged , , | Leave a comment

CRS-4995: The command ‘Delete resource’ is invalid in crsctl. Use srvctl for this command.

Situation

Trying to delete the database resource “ora.RACTEST.db” by using CRSCTL, because ORACLE_HOME has been uninsulated. But below error occurred.

$crsctl delete resource ora.RACTEST.db
CRS-4995: The command 'Delete resource' is invalid in crsctl. Use srvctl for this command.

Solution

Add “-unsupported” option to the command.  It should apply to “crsctl modify resource” as well.

$crsctl delete resource ora.RACTEST.db -unsupported

Resource which starts with “ora.” should be managed by “srvctl” of ORACLE_HOME. But in this case, the ORACLE_HOME has been unavailable.

Posted in GI and RAC | Tagged , , , , | Leave a comment

SP2-0027: Input is too long (> 2499 characters) – line ignored

Sometimes execute SQL scripts from SQL*PLUS, then  get below error:

SP2-0027: Input is too long (> 2499 characters) - line ignored

Workaround:

  1. To make one line sql script into different lines, which have less than 2500 characters fir each line.
  2. Use tools like SQL Developer, which does not have 2500 characters limitation.
Posted in Client, SQL Net, SQL Scripts For DBA | Tagged , | Leave a comment

How to Monitor DataPump

1) Get DataPump job details from database:

SQL>select JOB_NAME,OPERATION,JOB_MODE,DEGREE,ATTACHED_SESSIONS , STATE from dba_datapump_jobs

JOB_NAME                   OPERATION  JOB_MODE   DEGREE     ATTACHED_SESSIONS STATE
-------------------------- ---------- ---------- ---------- ----------------- ------------------------------
SYS_EXPORT_SCHEMA_01       EXPORT     SCHEMA     4          1                 EXECUTING

2) Also the job name can be found from the DataPump log file:

...
..
.
Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
Starting "SYS"."SYS_EXPORT_SCHEMA_01": directory= ..... PARALLEL=4

3) Query V$SESSION_LONGOPS:

SQL> select inst_id, username,target_desc,sofar,totalwork,TIME_REMAINING from gV$SESSION_LONGOPS where TIME_REMAINING>1000;

INST_ID    USERNAME  TARGET_DESC  SOFAR     TOTALWORK        TIME_REMAINING
---------- --------- ------------ --------- --------------- ----------------- 
 4         SYS                    248779    4294967296       28121732
 3         SYS                    467021    4294967296       15108237
 3         SYS             EXPORT   8824         21597           2335
 2         SYS                    313184    4294967296       22351992

4)Attach to the running DataPump job:

$ expdp ATTACH=SYS_EXPORT_SCHEMA_01

Export> help
...
..
.
 ------------------------------------------------------------------------------
HELP
 Summarize interactive commands.

KILL_JOB
 Detach and delete job.

PARALLEL
 Change the number of active workers for current job.

REUSE_DUMPFILES
 Overwrite destination dump file if it exists [NO].

START_JOB
 Start or resume current job.
 Valid keyword values are: SKIP_CURRENT.

STATUS
 Frequency (secs) job status is to be monitored where
 the default [0] will show new status when available.

STOP_JOB
 Orderly shutdown of job execution and exits the client.
 Valid keyword values are: IMMEDIATE.

$status
...
..
.

5) Kill the running DataPump job:

Export> KILL_JOB
Are you sure you wish to stop this job ([yes]/no): yes
$
Posted in DataPump | Tagged , , , , , , | Leave a comment

Unable to load Sfix.ini file: java.io.FileNotFoundException: 22644310\sfix.ini (The system cannot find the path specified)

Situation

Trying to apply two below patches onto ORMB database, then got the below errors:

08/08/2017 11:52 AM <DIR> 20986229
02/04/2016 04:11 AM <DIR> 22329775

Directory of D:\PATCHES\V4.3.0.1.0-25963443_MultiPlatform\FW.V4.3.0.1.0-25963443\prereqpatches\RollupPack\FW\Database

08/08/2017 11:52 AM <DIR> .
08/08/2017 11:52 AM <DIR> ..
08/08/2017 11:52 AM <DIR> 20986229
02/04/2016 04:11 AM <DIR> 22329775
08/08/2017 11:52 AM 63 CDXPatch.ini
17/05/2016 08:33 PM 1,117 IgDbErr
17/05/2016 08:33 PM 5,115 ouafDatabasePatch.cmd
17/05/2016 08:33 PM 3,846 ouafDatabasePatch.sh
 4 File(s) 10,141 bytes
 4 Dir(s) 26,984,898,560 bytes free

D:\PATCHES\V4.3.0.1.0-25963443_MultiPlatform\FW.V4.3.0.1.0-25963443\prereqpatches\RollupPack\FW\Database>ouafDatabasePatch.cmd
"CMDLINE::: d:\java\bin\java.exe -cp D:\dbpatch_tools\lib\*;D:\dbpatch_tools\config com.oracle.ouaf.database.patch.OUAFPatch "

Folder for 22644310 is missing
File 22644310\sfix.ini is missing
File 22644310\CDXPatch.lang is missing
Folder for 21133947 is missing
File 21133947\sfix.ini is missing
File 21133947\CDXPatch.lang is missing
Folder for 22244788 is missing
File 22244788\sfix.ini is missing
File 22244788\CDXPatch.lang is missing
Folder for 23201172 is missing
File 23201172\sfix.ini is missing
File 23201172\CDXPatch.lang is missing
Folder for 23194604 is missing
File 23194604\sfix.ini is missing
File 23194604\CDXPatch.lang is missing
Unable to load Sfix.ini file: java.io.FileNotFoundException: 22644310\sfix.ini (The system cannot find the path specified)

Investigation

Configuration file CDXPatch.ini includes other patch ids:

20986229
22329775
21133947
22244788
22644310
23194604
23201172

Solution

Edit configuration file CDXPatch.ini to include the right patch ids which will be applied onto ORMB database.

20986229
22329775

...
..
.

Applying 22329775 ...

Writing to log file: ORMB_DEV001\log22329775.log

-----------------------------------------------------------

--Applying patch 22329775 at 08-09-2017 14:29:10using $LastChangedRevision: 42479 $

---------------------------------------------------------------

--Copying language information

-----------------------------------------------------------

--Patch 22329775 applied successfully at 08-09-2017 14:29:11

Patch applied successfully..
Posted in ORMB | Tagged , , , , , , , , , , , , | Leave a comment

The agent is overloaded [current requests: 128]

Situation

Below alerts are received from racnode1 -“The agent is overloaded [current requests: 128]”

From: oracle 
Sent: Friday, 4 August 2017 7:07 PM
Cc: 
Subject: EM Event: Warning: racnode1 - Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable.

...
..
.
Categories=Availability 
Message=Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable. 
Severity=Warning 
Event reported time=Aug 4, 2017 7:06:27 PM AEST
...
..
.

Investigations

1)   check agent status.

Agent is running
Agent upload is not working
Agent reload is not working
OMS heartbeat is not working

$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.5.0
OMS Version : 13.2.0.0.0
Protocol Version : 12.1.0.1.0
..
.
Last Reload : 2017-08-04 11:28:59
Last successful upload : 2017-08-04 14:51:03  <--- 5 hours ago
Last attempted upload : 2017-08-04 14:51:03
..
.
Last attempted heartbeat to OMS : 2017-08-04 14:50:23
Last successful heartbeat to OMS : 2017-08-04 14:50:23
Next scheduled heartbeat to OMS : 2017-08-04 14:51:23

2) Upload agent

$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:The agent is overloaded [current requests: 128]

3) Reload agent

$ emctl reload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD reload error:The agent is overloaded [current requests: 128]

4) “emagent_perl.trc” file has no information updated since agent restarted.

5) check “gcagent.log”

Java layer deadlock —“Dead Lock detected!!”

2017-08-04 19:28:59,071 [43:GCThread-13] ERROR -
Dead Lock detected!!
Participating threads:Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
 > Accumulated wait time (msec): 1372208 (1 times)

"HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" tid=2141 BLOCKED
 > Accumulated wait time (msec): 11036289 (76 times)
 > Accumulated blocked time (msec): 16506994 (4 times)

"oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Fri Aug 04 14:51:06 EST 2017)" tid=3088 BLOCKED
 > Accumulated blocked time (msec): 16672145 (7 times)

"HTTP Listener-1022 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150181021899001])" tid=1022 WAITING
 > Accumulated wait time (msec): 28746227 (37 times)
 > Accumulated blocked time (msec): 133 (12 times)

"HTTP Listener-1078 - /emd/main/ (DispatchRequests OMS.console@16398@omsnode=>[150181015881006])" tid=1078 WAITING
 > Accumulated wait time (msec): 28719225 (44 times)

=================
Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
 sun.misc.Unsafe.park(Native Method)
 - waiting on <0x149717ec> (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is owned by "HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" (tid=2141)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
 java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
...
..
.

Solution

1) stop agent

$ emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Stopping agent ...
 stopped.

2) start agent

$ emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Starting agent ............................................ started.

3) upload agent successfully

$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

4) reload agent successfully

$ emctl reload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully

5) check agent status successfully

$ emctl status agent
...
..
Last attempted heartbeat to  OMS : 2017-08-04 19:53:31
Last successful heartbeat to OMS : 2017-08-04 19:53:31
Next scheduled heartbeat to  OMS : 2017-08-04 19:54:32

---------------------------------------------------------------
Agent is Running and Ready

 

Posted in OEM12c, OEM13c | Tagged , , , , , , , | Leave a comment

High Swap Usage On Oracle Database Server

Situation:

When investigating one client’s Oracle database server high swap memory usage, we find the swap space usage is constantly very high on this Linux server, which causes Oracle database a big performance issue.

OS: RHEL 7.3
DB: Oracle 12.2.0.1

Findings:

1)top

Tasks: 352 total, 2 running, 350 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.4%us, 4.1%sy, 0.0%ni, 79.3%id, 2.2%wa, 0.3%hi, 0.8%si, 0.0%st
Mem: 32172820k total, 32015956k used, 156864k free, 14528k buffers
Swap: 16777208k total, 7435428k used, 9341780k free, 11129844k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
137049 oracle 15 0 16.2g 5.3g 5.3g S 20.6 17.4 10:14.17 oracle
 72457 oracle 15 0 16.2g 4.7g 4.7g S 15.3 15.3 10:50.15 oracle
...
..
.

2) pmap

$ pmap -x 137049
137049: oracleRACTEST1 (LOCAL=NO)
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 11704 0 r-x-- oracle
0000000006419000 444 140 4 rwx-- oracle
0000000006488000 148 100 80 rwx-- [ anon ]
000000001966e000 532 176 92 rwx-- [ anon ]
0000000060000000 16779264 5444888 1485768 rwxs- [ shmid=0x670005 ]
00000032b6a00000 112 108 0 r-x-- ld-2.5.so
00000032b6c1c000 4 0 0 r-x-- ld-2.5.so
...
..
.
00007fff4c504000 160 136 132 rwx-- [ stack ]
00007fff4c5d2000 12 4 0 r-x-- [ anon ]
ffffffffff600000 8192 0 0 ----- [ anon ]
---------------- ------ ------ ------
total kB 16954204 5477952 1490448

3) swappiness

$ cat /proc/sys/vm/swappiness
10

4) /proc/meminfo

$ cat /proc/meminfo
MemTotal: 32172820 kB
MemFree: 158096 kB
Buffers: 12512 kB
Cached: 11124148 kB
SwapCached: 878556 kB
Active: 11885576 kB
Inactive: 376796 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32172820 kB
LowFree: 158096 kB
SwapTotal: 16777208 kB
SwapFree: 9336848 kB
Dirty: 136 kB
Writeback: 0 kB
AnonPages: 487212 kB
Mapped: 11089784 kB
Slab: 104984 kB
PageTables: 2779136 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 24475008 kB
Committed_AS: 20456368 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8192
HugePages_Free: 8192
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Causes:

The huge pages are not used by Oracle database server, because the number of hugepages set-up is too small.

Solution:

1)Increase the number of  hugepages.

$ cat /etc/sysctl.conf|grep -i hugepages
vm.nr_hugepages = 8200

2) Reboot the server.

3) Check the results.

$ cat /proc/meminfo
...
..
.
SwapTotal: 16777208 kB
SwapFree:  16777208 kB
Dirty:          116 kB
Writeback:        8 kB
AnonPages:  1076372 kB
Mapped:      108400 kB
Slab:        132980 kB
PageTables:   37540 kB
NFS_Unstable:     0 kB
Bounce:           0 kB
CommitLimit: 24466816 kB
Committed_AS: 2249980 kB
VmallocTotal: 34359738367 kB
VmallocUsed:       271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8200
HugePages_Free: 26
HugePages_Rsvd: 19
Hugepagesize:        2048 kB


$ ipcs -ma
------ Shared Memory Segments --------
key          shmid owner perms bytes nattch status
0x00000000 5275649 root    644    80 2
0x00000000 5308419 root    644 16384 2
0x00000000 5341188 root    644   280 2
0xbb94f3d4 5373957 oracle 640 17181966336 87

------ Semaphore Arrays --------
key semid owner perms nsems
0x17a558e8 131073 oracle 640 250
...
..
.


--
-- top 
--

top - 15:15:40 up 20:13, 7 users, load average: 1.13, 0.64, 0.50
Tasks: 289 total, 3 running, 286 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.9%us, 1.4%sy, 0.0%ni, 63.0%id, 23.3%wa, 0.4%hi, 1.0%si, 0.0%st
Mem: 32172820k  total, 19801684k used, 12371136k free, 825948k buffers
Swap: 16777208k total,        0k used, 16777208k free, 931716k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 97575 oracle 16 0 16.1g 31m 25m R 16.6 0.1 0:11.88 oracle
 97973 oracle 17 0 16.1g 28m 21m D 8.6 0.1 0:01.61 oracle
...
..
..


$ pmap -x 97575
97575: oracleRACTEST1 (LOCAL=NO)
Address          Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 17880    0 r-x-- oracle
0000000006419000   444   216   56 rwx-- oracle
0000000006488000   148   116  116 rwx-- [ anon ]
000000000a80a000   532   428  428 rwx-- [ anon ]
0000000060000000 16779264  0    0 rwxs- 1 (deleted)
00000032b6a00000   112   108    0 r-x-- ld-2.5.so
00000032b6c1c000    4      4    4 r-x-- ld-2.5.so
...
..
.
ffffffffff600000 8192 0 0 ----- [ anon ]
---------------- ------ ------ ------
total kB        16948952 42024 17284


$ sar 1 10
Linux 2.6.18-408.el5 (racnode1) 08/02/2017

03:18:48 PM CPU %user %nice %system %iowait %steal %idle
03:18:49 PM all 5.81 0.00 1.52 1.26 0.00 91.41
03:18:50 PM all 4.77 0.00 1.76 2.01 0.00 91.46
03:18:51 PM all 3.54 0.00 1.26 1.01 0.00 94.19
03:18:52 PM all 2.27 0.00 0.51 1.52 0.00 95.71
03:18:53 PM all 0.25 0.00 0.50 0.50 0.00 98.74
03:18:54 PM all 5.54 0.00 1.01 1.01 0.00 92.44
03:18:55 PM all 2.78 0.00 0.51 0.00 0.00 96.72
03:18:56 PM all 1.01 0.00 0.25 0.25 0.00 98.49
03:18:57 PM all 1.01 0.00 0.25 0.51 0.00 98.23
03:18:58 PM all 2.76 0.00 1.26 0.50 0.00 95.48
Average: all 2.97 0.00 0.88 0.86 0.00 95.29

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r b swpd free       buff cache     si so bi   bo in cs us sy id wa st
 0 0    0 12377776 827836 932156     0 0 1259 387  86 73 5 1 90 5 0
 0 0    0 12377776 827856 932140     0 0   15 398 3028 2395 3 2 95 1 0
 3 0    0 12373368 827856 932160     0 0  231 270 2065 2786 21 16 63 0 0
 0 0    0 12369324 827864 932140     0 0    0 168 1124 676 17 1 82 1 0
 0 0    0 12369324 827868 932160     0 0   40 112 1311 754 1 1 98 1 0
Posted in Linux, Performance Tunning | Tagged , , , , , , , , , , , , , , | Leave a comment

ohasd failed to start

Situation :

1) Two 11.2.0.4 RAC nodes( racnode1/2). Deleted node racnode2 . Upgraded OS from RHEL 4  to RHEL 7 for racnode2, then tried to add node racnode2 back into cluster from racnode1.

2) export IGNORE_PREADDNODE_CHECKS=Y, Ran addnode.sh ->  orainstRoot.sh->root.sh, then got the below errors:

...
..
.
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2017-07-24 16:11:23.098:
[client(11401)]CRS-2101:The OLR was formatted using version 3.
2017-07-24 16:11:24.001:
[client(11424)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /u01/app/11.2.0.4/grid/crs/install/roothas.pl line 377, line 4.

Solution :

1)Create ohasd-oracle.service before running root.sh:

[root@racnode1 system]# cat /etc/systemd/system/oracle-ohasd.service

# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.

#

# Oracle OHASD startup



[Unit]

Description=Oracle High Availability Services

After=syslog.target network.target remote-fs.target



[Service]

ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

Type=simple

Restart=always



[Install]

WantedBy=multi-user.target graphical.target
[root@racnode2]#systemctl start oracle-ohasd
[root@racnode2]#systemctl status oracle-ohasd
 oracle-ohasd.service - Oracle High Availability Services
 Loaded: loaded (/etc/systemd/system/oracle-ohasd.service; enabled; vendor preset: disabled)
 Active: active (running) since Wed 2017-07-26 14:55:19 AEST; 21h ago
 Main PID: 3073 (init.ohasd)
 CGroup: /system.slice/oracle-ohasd.service
Jul 26 14:55:19 racnode2 systemd[1]: Started Oracle High Availability Services.
Jul 26 14:55:19 racnode2 systemd[1]: Starting Oracle High Availability Services...
Jul 26 14:55:29 racnode2 init.ohasd[3073]: /etc/init.d/init.ohasd: Writing 4769 to oracle-ohasd tasks
ith operating system process ID 10136
...
..
.

OR

2) Download and apply patch  18370031 for 11.2.0.4.0 before running root.sh.

OR

3) Download and apply 11.2.0.4 GIPSU05 or later one before running root.sh. For 11.2.0.4, fter GIPSU05,  patch  18370031 is included in the PSU already.

Posted in GI and RAC, Upgrading and Patching | Tagged , , , , , , , , , , | Leave a comment

Copy Oracle Home Binary from One RAC Node to Another RAC Node

For some reason on RAC node 2 ( racnode2) , RAC oracle home binary needs to be copied from another node racnode1.

1)As RAC Oracle home owner, go to ORACLE_HOME/oui/bin  on racnode1:

oracle@racnode1:/u01/app/oracle/product/11.2.0/dbhome_1/oui/bin$ ./addNode.sh "CLUSTER_NEW_NODES={racnode2}"

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "racnode1"

Checking user equivalence...
User equivalence check passed for user "oracle"

WARNING:
Node "racnode2" already appears to be part of cluster

Pre-check for node addition was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 15152 MB Passed
Oracle Universal Installer, Version 11.2.0.4.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

Performing tests to see whether nodes racnode2 are available
............................................................... 100% Done.

SEVERE:The new nodes 'racnode2' are already part of the cluster.

2) check oraInventory and find racnode2 has been part of the cluster for ORACLE_HOME. so we manually detach it for this node – racnode2:

$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 "CLUSTER_NODES={racnode1}"

3) Copy the ORACLE_HOME binary again from racnode1 to racnode2:

oracle@racnode1:/u01/app/oracle/product/11.2.0/dbhome_1/oui/bin$ ./addNode.sh "CLUSTER_NEW_NODES={racnode2}"

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "racnode1"

Checking user equivalence...
User equivalence check passed for user "oracle"

WARNING:
Node "racnode2" already appears to be part of cluster

Pre-check for node addition was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 14856 MB Passed
Oracle Universal Installer, Version 11.
Copyright (C) 1999, 2013, Oracle. All rights reserved.


Performing tests to see whether nodes racnode2 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
 Source: /u01/app/oracle/product/11.2.0/dbhome_1
 New Nodes
Space Requirements
 New Nodes
 racnode2
 /u01: Required 6.10GB : Available 90.89GB
Installed Products
 Product Names
 Oracle Database 11g 11.2.0.4.0
 Java Development Kit 1.5.0.51.10
 Installer SDK Component 11.2.0.4.0
 Oracle One-Off Patch Installer 11.2.0.3.4
 Oracle Universal Installer 11.2.0.4.0
 Oracle USM Deconfiguration 11.2.0.4.0
 Oracle Configuration Manager Deconfiguration 10.3.1.0.0
 Oracle DBCA Deconfiguration 11.2.0.4.0
 Oracle RAC Deconfiguration 11.2.0.4.0
 Oracle Database Deconfiguration 11.2.0.4.0
 Oracle Configuration Manager Client 10.3.2.1.0
 Oracle Configuration Manager 10.3.8.1.0
 Oracle ODBC Driverfor Instant Client 11.2.0.4.0
 LDAP Required Support Files 11.2.0.4.0
 SSL Required Support Files for InstantClient 11.2.0.4.0
 Bali Share 1.1.18.0.0
 Oracle Extended Windowing Toolkit 3.4.47.0.0
 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
 Oracle Real Application Testing 11.2.0.4.0
 Oracle Database Vault J2EE Application 11.2.0.4.0
 Oracle Label Security 11.2.0.4.0
 Oracle Data Mining RDBMS Files 11.2.0.4.0
 Oracle OLAP RDBMS Files 11.2.0.4.0
 Oracle OLAP API 11.2.0.4.0
 Platform Required Support Files 11.2.0.4.0
 Oracle Database Vault option 11.2.0.4.0
 Oracle RAC Required Support Files-HAS 11.2.0.4.0
 SQL*Plus Required Support Files 11.2.0.4.0
 Oracle Display Fonts 9.0.2.0.0
 Oracle Ice Browser 5.2.3.6.0
 Oracle JDBC Server Support Package 11.2.0.4.0
 Oracle SQL Developer 11.2.0.4.0
 Oracle Application Express 11.2.0.4.0
 XDK Required Support Files 11.2.0.4.0
 RDBMS Required Support Files for Instant Client 11.2.0.4.0
 SQLJ Runtime 11.2.0.4.0
 Database Workspace Manager 11.2.0.4.0
 RDBMS Required Support Files Runtime 11.2.0.4.0
 Oracle Globalization Support 11.2.0.4.0
 Exadata Storage Server 11.2.0.1.0
 Provisioning Advisor Framework 10.2.0.4.3
 Enterprise Manager Database Plugin -- Repository Support 11.2.0.4.0
 Enterprise Manager Repository Core Files 10.2.0.4.5
 Enterprise Manager Database Plugin -- Agent Support 11.2.0.4.0
 Enterprise Manager Grid Control Core Files 10.2.0.4.5
 Enterprise Manager Common Core Files 10.2.0.4.5
 Enterprise Manager Agent Core Files 10.2.0.4.5
 RDBMS Required Support Files 11.2.0.4.0
 regexp 2.1.9.0.0
 Agent Required Support Files 10.2.0.4.5
 Oracle 11g Warehouse Builder Required Files 11.2.0.4.0
 Oracle Notification Service (eONS) 11.2.0.4.0
 Oracle Text Required Support Files 11.2.0.4.0
 Parser Generator Required Support Files 11.2.0.4.0
 Oracle Database 11g Multimedia Files 11.2.0.4.0
 Oracle Multimedia Java Advanced Imaging 11.2.0.4.0
 Oracle Multimedia Annotator 11.2.0.4.0
 Oracle JDBC/OCI Instant Client 11.2.0.4.0
 Oracle Multimedia Locator RDBMS Files 11.2.0.4.0
 Precompiler Required Support Files 11.2.0.4.0
 Oracle Core Required Support Files 11.2.0.4.0
 Sample Schema Data 11.2.0.4.0
 Oracle Starter Database 11.2.0.4.0
 Oracle Message Gateway Common Files 11.2.0.4.0
 Oracle XML Query 11.2.0.4.0
 XML Parser for Oracle JVM 11.2.0.4.0
 Oracle Help For Java 4.2.9.0.0
 Installation Plugin Files 11.2.0.4.0
 Enterprise Manager Common Files 10.2.0.4.5
 Expat libraries 2.0.1.0.1
 Deinstallation Tool 11.2.0.4.0
 Oracle Quality of Service Management (Client) 11.2.0.4.0
 Perl Modules 5.10.0.0.1
 JAccelerator (COMPANION) 11.2.0.4.0
 Oracle Containers for Java 11.2.0.4.0
 Perl Interpreter 5.10.0.0.2
 Oracle Net Required Support Files 11.2.0.4.0
 Secure Socket Layer 11.2.0.4.0
 Oracle Universal Connection Pool 11.2.0.4.0
 Oracle JDBC/THIN Interfaces 11.2.0.4.0
 Oracle Multimedia Client Option 11.2.0.4.0
 Oracle Java Client 11.2.0.4.0
 Character Set Migration Utility 11.2.0.4.0
 Oracle Code Editor 1.2.1.0.0I
 PL/SQL Embedded Gateway 11.2.0.4.0
 OLAP SQL Scripts 11.2.0.4.0
 Database SQL Scripts 11.2.0.4.0
 Oracle Locale Builder 11.2.0.4.0
 Oracle Globalization Support 11.2.0.4.0
 SQL*Plus Files for Instant Client 11.2.0.4.0
 Required Support Files 11.2.0.4.0
 Oracle Database User Interface 2.2.13.0.0
 Oracle ODBC Driver 11.2.0.4.0
 Oracle Notification Service 11.2.0.3.0
 XML Parser for Java 11.2.0.4.0
 Oracle Security Developer Tools 11.2.0.4.0
 Oracle Wallet Manager 11.2.0.4.0
 Cluster Verification Utility Common Files 11.2.0.4.0
 Oracle Clusterware RDBMS Files 11.2.0.4.0
 Oracle UIX 2.2.24.6.0
 Enterprise Manager plugin Common Files 11.2.0.4.0
 HAS Common Files 11.2.0.4.0
 Precompiler Common Files 11.2.0.4.0
 Installation Common Files 11.2.0.4.0
 Oracle Help for the Web 2.0.14.0.0
 Oracle LDAP administration 11.2.0.4.0
 Buildtools Common Files 11.2.0.4.0
 Assistant Common Files 11.2.0.4.0
 Oracle Recovery Manager 11.2.0.4.0
 PL/SQL 11.2.0.4.0
 Generic Connectivity Common Files 11.2.0.4.0
 Oracle Database Gateway for ODBC 11.2.0.4.0
 Oracle Programmer 11.2.0.4.0
 Oracle Database Utilities 11.2.0.4.0
 Enterprise Manager Agent 10.2.0.4.5
 SQL*Plus 11.2.0.4.0
 Oracle Netca Client 11.2.0.4.0
 Oracle Multimedia Locator 11.2.0.4.0
 Oracle Call Interface (OCI) 11.2.0.4.0
 Oracle Multimedia 11.2.0.4.0
 Oracle Net 11.2.0.4.0
 Oracle XML Development Kit 11.2.0.4.0
 Oracle Internet Directory Client 11.2.0.4.0
 Database Configuration and Upgrade Assistants 11.2.0.4.0
 Oracle JVM 11.2.0.4.0
 Oracle Advanced Security 11.2.0.4.0
 Oracle Net Listener 11.2.0.4.0
 Oracle Enterprise Manager Console DB 11.2.0.4.0
 HAS Files for DB 11.2.0.4.0
 Oracle Text 11.2.0.4.0
 Oracle Net Services 11.2.0.4.0
 Oracle Database 11g 11.2.0.4.0
 Oracle OLAP 11.2.0.4.0
 Oracle Spatial 11.2.0.4.0
 Oracle Partitioning 11.2.0.4.0
 Enterprise Edition Options 11.2.0.4.0
-----------------------------------------------------------------------------

Instantiating scripts for add node (Friday, July 21, 2017 11:14:20 AM EST)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Friday, July 21, 2017 11:14:24 AM EST)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Friday, July 21, 2017 11:19:34 AM EST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh #On nodes racnode2
To execute the configuration scripts:
 1. Open a terminal window
 2. Log in as "root"
 3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1 was success
Please check '/tmp/silentInstall.log' for more details.

4) Run “/u01/app/oracle/product/11.2.0/dbhome_1/root.sh” on node racnode2.

Posted in GI and RAC, Install and Upgrade | Tagged , , , , | Leave a comment

Move OCR, Voting Disk File, ASM SPILE to a New Diskgroup ( 11.2.0.4 )

Below exercise is to move everything from old OCR/VOTING diskgroup OCR_VOTE  to new diskgroup OCR_VOTE2.

1)Create New CRS Diskgroup:

SQL> create diskgroup OCR_VOTE2 external redundancy disk 'ORCL:OCR_VOTE2';

Diskgroup created.

SQL> alter diskgroup OCR_VOTE2 set attribute 'compatible.asm'='11.2.0.0';

Diskgroup altered.

2) Mount the Diskgroup if Not Yet.

SQL> alter diskgroup OCR_VOTE2 mount;
alter diskgroup OCR_VOTE2 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "OCR_VOTE2" cannot be mounted
ORA-15013: diskgroup "OCR_VOTE2" is already mounted

3) Move OCR and Vote file from Old Diskgroup to New Diskgroup.

# export ORACLE_HOME=/u01/app/11.2.0.4/grid
# $ORACLE_HOME/bin/ocrconfig -add +OCR_VOTE2

# $ORACLE_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
 1. ONLINE 2acf2e1fd3524ffebf39abf0235da8bb (ORCL:OCR_VOTE1) [OCR_VOTE]
Located 1 voting disk(s).

# $ORACLE_HOME/bin/ocrconfig -delete +OCR_VOTE

# $ORACLE_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
 1. ONLINE 2acf2e1fd3524ffebf39abf0235da8bb (ORCL:OCR_VOTE1) [OCR_VOTE]
Located 1 voting disk(s).

# $ORACLE_HOME/bin/crsctl replace votedisk +OCR_VOTE2
Successful addition of voting disk aed2371729704f38bff6cb2716ba90f5.
Successful deletion of voting disk 2acf2e1fd3524ffebf39abf0235da8bb.
Successfully replaced voting disk group with +OCR_VOTE2.
CRS-4266: Voting file(s) successfully replaced

# $ORACLE_HOME/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
 1. ONLINE aed2371729704f38bff6cb2716ba90f5 (ORCL:OCR_VOTE2) [OCR_VOTE2]
Located 1 voting disk(s).

# $ORACLE_HOME/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version : 3
 Total space (kbytes) : 262120
 Used space (kbytes) : 3736
 Available space (kbytes) : 258384
 ID : 2091720096
 Device/File Name : +OCR_VOTE2
 Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

4) Change ASM spfile location from Old DiskGroup to New DiskGroup

SQL> show parameter spfile

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +OCR_VOTE/blc5/asmparameterfil e/registry.253.767457217

SQL> create pfile='/tmp/tmp.ora' from spfile;

File created.

SQL> ! cat /tmp/tmp.ora
+ASM1.__large_pool_size=29360128
+ASM1.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM1.__shared_pool_size=230686720
+ASM2.asm_diskgroups='FRA','DG1'#Manual Mount
+ASM1.asm_diskgroups='DG1','FRA','OCR_VOTE2'#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest='/u01/app/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.memory_target=1073741824
*.remote_login_passwordfile='EXCLUSIVE'

SQL> create spfile='+OCR_VOTE2' from pfile='/tmp/tmp.ora';

File created.

SQL> ! $ORACLE_HOME/bin/gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
 /u01/app/11.2.0.4/grid/bin/gpnptool.bin get -o-

<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="11" ClusterUId="7fd95a772df4df5ebf03f302598ae4f1" ClusterName="BLC5" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="10.4.15.0" Adapter="bond0" Use="public"/><gpnp:Network id="net2" IP="10.4.14.0" Adapter="bond1" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+OCR_VOTE2/blc5/asmparameterfile/registry.253.950184589"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>7qFsk4u0KgpO52RoVbHjlk2dAzk=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>T2vB/oL0FAq1Qvnk8A2bXutSzlxGHpHvPXElTHAj+ctYSoAuVuDa9arni3AOYjz3NjhHWjjAI9zCaQAJ+Rue0vtom4TFFpGHsw+ogxPurgZc74plFGd0Fv7wmztHBOwveQ4rG4AJ4PViLxM0ECwymfD08494jPEihb29wVki6js=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
Success.

5) Stop and Start CRS, then verify.

a: $ORACLE_HOME/bin/crsctl stop crs
b: $ORACLE_HOME/bin/crsctl start crs

SQL> select name,state,usable_file_mb,total_mb,free_mb,required_mirror_free_mb from v$asm_diskgroup;

NAME STATE  USABLE_FILE_MB  TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB
----- ----- --------------- -------- -------- ----------------------
DG1 MOUNTED 266999          1536213    266999 0
FRA MOUNTED 671329          1024142    671329 0
OCR_VOTE2 MOUNTED 20126       20479     20126 0
OCR_VOTE DISMOUNTED 0             0         0 0

$ $ORACLE_HOME/bin/crsctl stat res -init -t

$ $ORACLE_HOME/bin/ocrcheck
 Status of Oracle Cluster Registry is as follows :
 Version : 3
 Total space (kbytes) : 262120
 Used space (kbytes) : 3736
 Available space (kbytes) : 258384
 ID : 2091720096
 Device/File Name : +OCR_VOTE2
 Device/File integrity check succeeded

Device/File not configured  

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

$ $ORACLE_HOME/bin/crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
 1. ONLINE aed2371729704f38bff6cb2716ba90f5 (ORCL:OCR_VOTE2) [OCR_VOTE2]

SQL> show parameter pfile

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +OCR_VOTE2/blc5/asmparameterfile/registry.253.950184589
Posted in ASM, GI and RAC | Tagged , , , , , , , , , , , | Leave a comment