How to modify an Oracle RAC Service to startup in a specific DB Mode (For Data Guard)?

Below, I have mentioned commands to modify an Oracle RAC Cluster Managed DB Service to startup in a specific DB Mode (for Data Guard).




Modify the Service to startup in a Primary Mode:
srvctl modify service -d RACDB1 -s RACDB1SERV1 -l primary


Modify the Service to startup in Physical Standby Mode:
srvctl modify service -d RACDB1 -s RACDB1SERV1 -l physical_standby
Modify the Service to o startup in a Snapshot Standby Mode:
srvctl modify service -d RACDB1 -s RACDB1SERV1 -l snapshot_standby


How to Check the configuration/status of the Cluster Managed DB Service:srvctl config service -d RACDB1 -s RACDB1SERV1




Note: Please remember to change the name of the DB Names/Services with your own.

Creating and Changing Encryption Wallets/Passwords in Oracle

This post covers the commands to create and then change Wallet Files/Passwords for Oracle databases using the ORAPKI utility.

$ orapki help
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
orapki [crl|wallet|cert|help] <-nologo>
Syntax :
[-option [value]]     : mandatory, for example [-wallet [wallet]]
[-option <value>]     : optional, but when option is used its value is mandatory.
<option>              : optional, for example <-summary>, <-complete>
[option1] | [option2] : option1 'or' option2
In this example the -auto_login switch enables the Oracle database to automatically startup with the Wallet file.
$ orapki wallet create -wallet /u01/wallet/DBNAME -pwd "insert_pwd_here" -auto_login
This command shows how to change the existing Wallet Password utilizing the ORAPKI utility.
orapki wallet change_pwd -wallet /u01/DBNAME/wallet -oldpwd insert_old_password -newpwd insert_new_password
The following SQL commands show how to open, close, authenticate and query Encryption Wallet Passwords and status.
alter system set wallet open identified by "xxxxxx";
alter system set wallet close identified by "xxxxxxxx";
alter system set encryption key authenticated by "xxxxxxx";
select * from v$encryption_wallet;

RAC Installs: root.sh fails after installation on the first node

This post covers RAC Install failures: root.sh fails after installation on the first node


Excerpt from Problem Logs:
Disk Group GRID01 creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15017: diskgroup "GRID01" cannot be mounted
ORA-15003: diskgroup "GRID01" already mounted in another lock name space
Configuration of ASM ... failed

see asmca logs at /u01/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6912.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed


Cause: "This particular issue arises when root.sh is run concurrently on the first and other nodes on the RAC cluster. The correct approach is to run root.sh on the first node followed by the other nodes of the RAC cluster.\

Solution:
root.sh Fails on the First Node for 11gR2 Grid Infrastructure Installation (Doc ID 1191783.1)
Run the following as the root OS user on Node 1.
"$GRID_HOME/crs/install/rootcrs.pl -deconfig -force -verbose"
Followed by this command on the other nodes:
"$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode"


How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation (Doc ID 942166.1)
Here is the master note for troubleshooting/fixing Grid Infrastructure Startup issues.
Troubleshoot Grid Infrastructure Startup Issues (Doc ID 1050908.1)

Useful OS Commands/Utilities for Oracle Databases on IBM AIX environments

In this post, I have compiled some useful Commands/Utilities for running Oracle Databases on IBM AIX environments.


Gives useful resource/info about the LPAR/Virtual Server:
$ lparstat -I


The following utilities (topas and nmon) Gives useful resource consumption about the LPAR/Virtual Server (:
$ nmon
       -> h
┌─HELP─────────most-keys-toggle-on/off───────────────────────────────────────────────────────────────────┐
│h = Help information     q = Quit nmon             0 = reset peak counts                                │
│+ = double refresh time  - = half refresh          r = ResourcesCPU/HW/MHz/AIX                          │
│c = CPU by processor     C=upto 1024 CPUs          p = LPAR Stats (if LPAR)                             │
│l = CPU avg longer term  k = Kernel Internal       # = PhysicalCPU if SPLPAR                            │
│m = Memory & Paging      M = Multiple Page Sizes  P = Paging Space                                      │
│d = DiskI/O Graphs       D = DiskIO +Service times o = Disks %Busy Map                                  │
│a = Disk Adapter         e = ESS vpath stats       V = Volume Group stats                               │
│^ = FC Adapter (fcstat)  O = VIOS SEA (entstat)    v = Verbose=OK/Warn/Danger                           │
│n = Network stats        N=NFS stats (NN for v4)   j = JFS Usage stats                                  │
│A = Async I/O Servers    w = see AIX wait procs   "="= Net/Disk KB<-->MB                                │
│b = black&white mode     g = User-Defined-Disk-Groups (see cmdline -g)                                  │
│t = Top-Process --->     1=basic 2=CPU-Use 3=CPU(default) 4=Size 5=Disk-I/O                             │
│u = Top+cmd arguments    U = Top+WLM Classes       . = only busy disks & procs                          │
│W = WLM Section          S = WLM SubClasses        @=Workload Partition(WPAR)                           │
│[ = Start ODR            ] = Stop ODR              i = Top-Thread                                       │
│~ = Switch to topas screen 

$ topas
       -> h
One-character commands:
  @ - Pressing the '@' key repeatedly toggles to wpar and normal mode
  a - Show all the variable subsections being monitored. Pressing the
      the 'a' key always returns topas to the main initial display.
  c - Pressing the 'c' key repeatedly toggles the CPU subsection
      between the cumulative report, off, and a list of busiest CPUs.
  d - Pressing the 'd' key repeatedly toggles the disk subsection between
      total disk, off, and busiest disks list activity for the system.
  t - Pressing the 't' key repeatedly toggles the tape subsection between
      total tape, off, and busiest tape list activity for the system.
  f - Pressing the 'f' key repeatedly toggles the file system subsection
      between total file system, off, and busiest file system list activity
      for the system.Also Moving the cursor over a WLM class and pressing 'f'
      shows the list of top processes in the class on the bottom of the
      screen(WLM Display Only).Similarly moving the cursor over a WPAR name and
      pressing'f' shows the list of top file system belonging to that wpar
      on the bottom of the screen(FS Display with @ option only)
  e - Pressing the 'e' key repeatedly toggles between the AME and NFS
      subsections. The key has no effect if AME(Active memory expansion)
      is not enabled in the machine.
  n - Pressing the 'n' key repeatedly toggles the network interfaces subsection
      between total network, off, and busiest interfaces list activity.
  p - Pressing the 'p' key toggles the hot processes subsection on and off.
  P - Toggle to the Full Screen Process Display
  q - Quit the program                                                           r - Refresh the screenGives useful resource consumption about the System Memory info (Including Large Page Consumption/Information):
$ svmon
Gives Disk Usage Information on/attached-to the LPAR:
$ df -g

PeopleSoft Tables: Deadlocking and Locking causing performance issues

PeopleSoft has some inherent Deadlocking and Locking issues in some of it's tables. These issues can translate into some serious performance problems. I have compiled a list of MOS Notes outlining the more well-known issues and potential solutions in this post. If you come across any other, please add them in the comments section and I can incorporate them as well:

• Deadlocking in PSIBQUEUEINST (Doc ID 656090.1)
• Integration Broker Performance Downgraded because of Locks On Table PSIBQUEUEINST and PSAPMSGPUBSYNC (Doc ID 1367618.1)
• Dead Lock on PSIBQUEUEINST and GetNextNumberWithGapsCommit() Function (Doc ID 1476687.1)
•  CRM: Deadlocks at Database Level on PSLOCK and/or PSVERSION Tables(Doc ID 653099.1)
• E-SEC: Deadlocking on PSVERSION and PSLOCK Tables(Doc ID 1064647.1)
• E-PT: Locking at Database Level on PSLOCK and/or PSVERSION Tables(Doc ID 1951231.1)
• E-SEC/DB2 Issue With Deadlocking On PSLOCK Table When Saving Roles in PT 8.5x(Doc ID 1372612.1
• E-IB: Deadlocks at PSAPMSGPUBINST Table Updates in PeopleTools 8.40-8.47(Doc ID 1291577.1)
• EC: EOPMTOCI Failling With Deadlock Error(Doc ID 2188015.1)

Valid Node Checking for Registration (VNCR) Implementation on Oracle

Valid Node Checking and Registration (VNCR) is a new feature introduced in versions 11.2.0.4 and 12c of Oracle. VNCR allows registrations to the Oracle Listener more secure such that, they are only allowed from known servers/nodes; hence the namesake. The main advantage of VNCR is to get by without implementing Class of Secure Transport (COST) configurations that, tend to be much more complicated and expensive. VNCR is relatively simple to implement - Following MOS Notes contain the directions and other useful resources.

• Valid Node Checking For Registration (VNCR) (Doc ID 1600630.1)
• How to Enable VNCR on RAC Database to Register only Local Instances (Doc ID 1914282.1)

The following example shows how to implement VNCR on RAC to register the local RAC instances:
VALID_NODE_CHECKING_REGISTRATION_LISTENER=1
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=1
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=(<enter the list of public ip's of all nodes separated by commas>)
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=1
REGISTRATION_INVITED_NODES_LISTENER_SCAN2=(<enter the list of public ip's of all nodes separated by commas>)
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=1
REGISTRATION_INVITED_NODES_LISTENER_SCAN3=(<enter the list of public ip's of all nodes separated by commas>)

Active Memory Expansion in IBM AIX 7.x - Useful Resources

This post covers some of the Q/A and MOS Notes and IBM Docs related to Active Memory Expansion (Fancy Name for Memory Compression) for Oracle Databases on IBM AIX 7.x.


1. Is IBM Active Memory Expansion (AME) - AIX 7.1 certified/supported for Oracle DB 11.2.0.4 (RAC and Non-RAC)?
Is IBM AIX Active Memory Expansion (AME) Certified or supported for Oracle Databases? ( Doc ID 1524569.1 )


2. Is IBM AME (AIX 7.x) certified for the Oracle Database?
Certification Information for Oracle Database on IBM AIX on Power systems ( Doc ID 1307544.1 )


3. What are the things to watch out for implementing AME on AIX 7.x - Any performance degradations to watch out for?
IBM POWER7 AIX and Oracle Database performance considerations -- 10g & 11g [ Note 1507249.1 ]


Other MOS Notes/Resources:

How to Tune Parameters Available in AIX to AvoNote Memory Failures Under Heavy Load [ Note 457271.1 ]
AIX: Top Things to DO NOW to Stabilize 11gR2 GI/RAC Cluster [ Note 1427855.1 ]




IBM Memory Compression - Active Memory Expansion (AME):
http://www.ibm.com/developerworks/aix/library/au-aix7memoryoptimize1/