RAC NOTES
==================================
Oracle Clusterware Startup Process :
===================================
1.File Read Order:
- OLR (Oracle Local Registry)
- VD (Voting Disk)
- OCR (Oracle Cluster Registry)
2.Process Start Order:
- OHASD (Oracle High Availability Services Daemon) - Main Wrapper script (etc/inittab) , OLR
- CSSD (Cluster Synchronization Services Daemon) - Main Voting disk - GPNP profile - KFED/KFOD
- CRSD (Cluster Ready Services Daemon) - Main look for OCR and GPNP profile
Since Oracle 11g Release 2, the ASM spfile (Automatic Storage Management Server Parameter File) is automatically stored in the first disk group created during Grid Infrastructure installation.
However, because the Voting Disk and OCR are stored in ASM, ASM must be started for Clusterware to function. This creates a challenge: how do you start ASM if its spfile is also stored on an ASM disk group?
Clusterware Startup Sequence and Resolution:
============================================
1) When a node restarts, OHASD is the first process to start, initiated by the system. OHASD reads the OLR (Oracle Local Registry), stored on the local file system, to gather the necessary data to continue initialization.
2) OHASD then starts GPNPD (Grid Plug and Play Daemon) and CSSD (Cluster Synchronization Services Daemon). CSSD reads the GPNP profile from the local file system, which contains essential information, including:
- The ASM_DISKSTRING parameter, which tells it where to find ASM disks.
- The name of the disk group containing the ASM spfile.
- The location of the Voting Disk.
3) CSSD then scans the headers of the ASM disks, as indicated in the ASM_DISKSTRING, to find the disk that holds the Voting Disk. With this information, CSSD can access the Voting Disk and complete its initialization, allowing it to start or join an existing cluster.
4) To start the ASM instance, it's not necessary to open the entire disk group. OHASD reads the header of the ASM disk containing the ASM spfile (based on the GPNP profile) and uses the information in the disk header to access and load the ASM spfile. The ASM instance is then started.
5) With ASM running and its disk groups mounted, CRSD is started by OHASD. CRSD now has access to the OCR (Oracle Cluster Registry) stored on the ASM disk group.
6) Finally, Oracle Clusterware completes its initialization and brings up other cluster services.
This startup process ensures that Oracle Clusterware can access the necessary components, even though some of them, like the Voting Disk and ASM spfile, are stored on ASM disks.
What are local registry and cluster registry? (OLR)
-------------------------
Local Registry:
The local registry, also known as the local node registry, is specific to each node in the Oracle RAC cluster.
It stores configuration information relevant to the local instance, such as instance-specific parameters, initialization parameters, and certain cluster-related settings.
Each RAC node maintains its own local registry to manage its local instance’s configuration independently.
Cluster Registry:
-----------------
The cluster registry is a shared repository that contains configuration information relevant to the entire RAC cluster.
It stores cluster-wide configuration settings, such as SCAN (Single Client Access Name) configurations, service configurations, and global resources like Oracle Clusterware resources.
The cluster registry is typically maintained in a shared location accessible by all nodes in the RAC cluster.
It facilitates centralized management of cluster-wide settings and resources, ensuring consistency and coherence across all nodes.
It contains :
Node list
Node membership mapping
Database instance, node and other mapping information
Characteristics of any third-party applications controlled by CRS
The file location is specified during the installation, the file pointer indicating the OCR device location is the ocr.loc, this can be in either of the following
linux - /etc/oracle
solaris - /var/opt/oracle
The file contents look something like below, this was taken from my installation
------------------
OCR Recovery :
-----------------
OLR OCR and Votedisk
OLR - node specific (Manual bkp)
OCR - cluster specific (auto bkp + includes votedisk ) Every 4hr +1day +1week auto bkp
Votedisk - cluster specific (No bkp)
./ocrconfig -manualbackup Manual backup of OCR
ocrconfig -local -restore /u01/app/oracle/crsdata/oranode1/olr/backup_20230212_101854.olr
+RECO/oranode-clu/OCRBACKUP/backup00.ocr.379.1128582589
ocrconfig -copy +RECO:/oranode-clu/OCRBACKUP/backup00.ocr.379.1128582589
ocrconfig -restore +DATA
ocrconfig -restore +RECO:/oranode-clu/OCRBACKUP/backup00.ocr.379.1128582589
ocrconfig -restore +RECO/oranode-clu/OCRBACKUP/backup00.ocr.379.1128582589
amdu method
/u01/app/19.0.0.0/grid/bin/amdu -diskstring '/dev/oracleasm/disks/DATA1' -extract DATA.255
/u01/app/19.0.0.0/grid/bin/amdu -diskstring '/dev/oracleasm/disks/DATA2' -extract DATA.255
/u01/app/19.0.0.0/grid/bin/amdu -diskstring '/dev/oracleasm/disks/DATA3' -extract DATA.255
ocrconfig -restore /tmp/amdu_2023_02_12_11_05_05/DATA_255.f
dd if=/dev/zero of=/dev/oracleasm/disks/ACFSDG bs=4096 count=10
crsctl start crs -excl -nocrs
crsctl query css votedisk
crsctl replace votedisk +VOTE1
1. Restoring OCR from Backup
crsctl stop crs
ocrconfig -restore /path_to_backup/ocrbackup
ocrcheck
crsctl start crs
If an OCR backup is not available, restoring the Oracle Cluster Registry (OCR) becomes more challenging :
------------------------------------------------------------
1. Recreate the OCR from Scratch
Stop Oracle Clusterware: crsctl stop crs
Reinitialize the OCR - ocrconfig -create +DATA/ocrfile
2. Restore OCR Using Redundant Copies
ocrconfig -restore /path_to_redundant_copy
3. Use Voting Disk to Rebuild OCR
crsctl replace ocr
crsctl start crs
ocrcheck
4. Reconfigure the Cluster
$GRID_HOME/deinstall/deinstall
What is the use of OLR? :
-----------------------
Ans: Oracle Local repository contains information that allows the cluster processes to be started up with the OCR being in the ASM storage system. Since the ASM file system is unavailable until the Grid processes are started up a local copy of the contents of the OCR is required which is stored in the OLR.
./ocrconfig -local -manualbackup To take manualbackup of OLR file
ocrconfig -local -restore /u01/app/oracle/crsdata/oranode1/olr/backup_20230212_101854.olr
orc.loc ocrconfig_loc=/u02/oradata/racdb/OCRFile
ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror
local_only=FALSE
3. Steps to Recreate OLR (If no backup of available)
The OLR is node-specific and stored in a local file (default location: $GRID_HOME/cdata/<hostname>/olr.ora).
Recreate OLR using the ocrconfig command:
$GRID_HOME/bin/ocrconfig -local -restore
If ocrconfig -local -restore doesn't work, use ocrcheck to verify and recreate the OLR:
$GRID_HOME/bin/ocrconfig -local -create
Restart the node and verify:
crsctl start crs
crsctl check crs
5. Restore Configuration
If OCR and OLR are recreated, you may need to reconfigure your cluster resources:
Reconfigure VIPs, SCAN listeners, and services using srvctl.
Re-add databases, listeners, and ASM instances:
srvctl add database -db <db_name> -oraclehome <db_home>
srvctl add asm -asmhome <asm_home>
srvctl add listener
6. Perform a Cluster Reboot
Restart the cluster to ensure everything is functioning correctly:
crsctl stop crs
Voting Disk :
-------------
The voting disk in the architecture is used to resolve membership issues in the event of a partitioned cluster, the voting disk protects data integrity.
It containes Network heartbeat + disk heartbeat details
1. Network Heartbeat:
Oracle RAC nodes use a private interconnect to send network heartbeats to each other.
These heartbeats are sent every second to confirm that the nodes are active and healthy.
If a node misses heartbeats for a predefined duration (default is 30 seconds), it is considered to have failed, and Oracle Clusterware may evict the node.
2. Disk Heartbeat (Hardware Heartbeat):
Oracle Clusterware nodes also write heartbeats to the voting disks to ensure that the cluster is functioning correctly.
Disk heartbeats are updated every second as well.
This ensures that the cluster knows which nodes are actively participating in the cluster.
Timeout and Failure:
The 30-second duration you mentioned is related to the failure detection timeout:
If a node stops sending heartbeats (both network and disk) for 30 seconds, Oracle Clusterware will treat that node as failed and initiate eviction.
Simple Majority rule : truncate = (n/2) + 1 : n = no of nodes
3=nodes , 3/2+1 = 2.5 Which nodes written less than 2.5 will be evicted from cluster .
querying - crsctl query css votedisk
What is CSSD in Oracle RAC?
-----------------------------
CSSD (Cluster Synchronization Services Daemon) is a critical background process in Oracle Real Application Clusters (RAC) that is part of the Oracle Clusterware stack. It is responsible for maintaining synchronization and communication between the nodes in a cluster.
Key Roles of CSSD:
Node Membership Management:
CSSD ensures that all nodes in the cluster are aware of each other's presence.
It maintains the membership of nodes and detects when a node joins or leaves the cluster.
Heartbeat Mechanism:
CSSD uses heartbeats (via network and shared disk) to monitor the health of nodes.
If a node fails to send a heartbeat within a defined timeout period, CSSD considers the node unhealthy and evicts it from the cluster.
Cluster Synchronization:
It ensures that the nodes in the cluster are synchronized and can work together seamlessly.
CSSD helps coordinate operations that require cluster-wide consistency.
Voting Disk Management:
CSSD interacts with the voting disks, which are shared disks used to determine cluster membership.
If there are split-brain scenarios, CSSD uses the voting disks to decide which nodes remain in the cluster and which ones are evicted.
Reconfiguration Events:
CSSD triggers reconfiguration events when nodes join or leave the cluster.
This ensures that the cluster configuration is always up-to-date.
Process Name: ocssd.bin
$GRID_HOME/log/<node_name>/cssd/
----------------------
Explain how client connection is established in RAC database ? :
------------------------
LREG process on each instances registers the database service of the node with default local listener and scan listener. The listeners store the workload information of each node.
So when client tries to connect using scan_name and port,
scan_name will be resolved through DNS , which will redirect to 1st scan ip ( out of 3).
The client will connected to the respective scan listener.
The scan listener compares the work load of both the instances and if scan determines that node1 has least load ,
then scan listener send the vip address and port details of that particular nodes local listener to client.
Now client connects to that local listeners and a dedicated server process is created.
Client connection becomes successful and it starts accessing the database.
======================
what is scan, vip?
==================
1. SCAN (Single Client Access Name)
Key Features:
Simplified Client Connection: Clients use a single name (the SCAN) to connect, and the connection is routed to the appropriate node automatically.
Load Balancing: SCAN listeners distribute client connections across all active instances in the cluster.
High Availability: If a node becomes unavailable, SCAN seamlessly redirects connections to surviving nodes without client reconfiguration.
Advantages:
No need to update client TNS files when nodes are added or removed from the cluster.
Simplifies management and enhances flexibility for scaling the cluster.
Example:
In a 3-node RAC cluster:
SCAN hostname: rac-scan.example.com
SCAN IPs: 192.168.1.11, 192.168.1.12, 192.168.1.13
The SCAN hostname resolves to these IPs in round-robin fashion.
2. VIP (Virtual IP Address) :
Definition:
A VIP is a virtual IP address assigned to each node in the RAC cluster.
It is used for client connections and is bound to the public network interface of the respective node.
Key Features:
Fast Failover: If a node fails, its VIP is automatically moved to another surviving node. This allows clients connected to the failed node to quickly receive a connection reset and reconnect to a different node.
Node-Specific IP: Each node has its own unique VIP that clients use for direct connections.
Advantages:
Improves client failover performance.
Provides a consistent network identity for each node, independent of its physical network interface.
Example:
In a 3-node RAC cluster:
Node 1: Public IP = 192.168.1.101, VIP = 192.168.1.201
Node 2: Public IP = 192.168.1.102, VIP = 192.168.1.202
Node 3: Public IP = 192.168.1.103, VIP = 192.168.1.203
================
what is dynamic and static listener listener :
Feature | Dynamic Listener | Static Listener/local
----------------------------|-------------------------------------------------------|----------------------------------------------------
Service Registration | Automatically registers services at runtime. | Services must be manually registered in listener.ora.
Configuration | No need to manually configure listener for new instances. | Must configure each instance/service in listener.ora.
Usage | Typically used for Oracle RAC and environments with multiple dynamic changes. | Typically used for single-instance setups or environments with minimal changes.
Configuration File | Does not require constant edits to the listener.ora file. | Requires manual edits to listener.ora to add/remove instances.
Listener Mode | Operates in dynamic registration mode (via LOCAL_LISTENER). | Operates in static mode (entries are predefined in listener.ora).
Best For | Environments where instances/services are frequently added or removed (e.g., Oracle RAC). | Environments where the setup is static, with little to no change in instances.
Key Differences Local VS SCAN
Feature Local Listener SCAN Listener
Level Node-specific Cluster-wide
Connection Handling Handles connections for its own node Balances connections across all RAC nodes
IP Dependency Relies on the node's specific IP Uses virtual IPs configured for SCAN
Load Balancing No inherent load balancing Provides load balancing and failover
Configuration Set in the listener.ora for each node Configured at the cluster level
Simplified Access Requires node-specific client TNS entry Single SCAN entry for the entire cluster
1)listener.ora :
Defines the network listener configuration for the database.
Found on the server side where the database listener runs.
Helps manage incoming connection requests to the database.
Contains information about the listener, including protocol, host, port, and SIDs it serves.
- Listener name
- Address (host, port, protocol)
- SID or Service details
Used by the server to accept and route connection requests.
SID_LIST_LISTENER defines database SIDs for a listener.
Required for the database listener to function.
Can dynamically register services if configured in the database.
# listener.ora Network Configuration File
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = orcl) # Database SID
(ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1) # Oracle Home
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname)(PORT = 1521)) # Protocol, Host, and Port
)
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521)) # IPC Protocol for External Procedures
)
)
# Optional for enabling dynamic registration
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER = ON
2) tnsnames.ora :
Specifies client-side connection details to the database. DR setup and cloning purpose
Found on the client side (and also on the server if needed for local connections).
Helps clients locate and connect to the database.
Contains mappings of database service names (aliases) to connection details.
- Alias (e.g., ORCL)
- Address (host, port, protocol)
- Service name or SID
Used by clients (or applications) to initiate connections to the database.
SERVICE_NAME or SID for connecting to a database.
Required for clients to connect to the database.
Does not dynamically change—static client configuration.
PROD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = db1.db.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = prod)
)
)
==================
An Oracle RAC (Real Application Clusters) database consists of multiple interconnected components that enable it to function as a unified database across multiple nodes. Here's a breakdown of the key components:
---
### **1. Cluster Components**
These components facilitate communication and coordination among nodes in the cluster:
#### **a. Cluster Interconnect**
- A private network that enables communication between RAC nodes.
- Used for Global Cache Fusion (for transferring data blocks between nodes).
- Typically consists of high-speed, low-latency network connections (e.g., InfiniBand or Ethernet).
#### **b. Clusterware**
- Manages the cluster and ensures high availability.
- Components include:
- **Cluster Ready Services (CRS)**: Manages node membership and resources.
- **Cluster Synchronization Services (CSS)**: Synchronizes the cluster.
- **Event Management (EVM)**: Handles event notifications.
- Oracle Grid Infrastructure software provides clusterware functionality.
---
### **2. Database Components**
These components make up the RAC database:
#### **a. Database Instances**
- Each node in the RAC cluster runs a separate Oracle instance.
- Each instance has its own memory structures and background processes.
- Instances share access to a single set of database files.
#### **b. Global Database View**
- All instances provide a unified view of the database.
- Clients can connect to any instance and access the same database.
#### **c. Database Files**
- Shared storage contains:
- **Datafiles**: Store user and system data.
- **Control Files**: Maintain database structure information.
- **Redo Log Files**: Log transaction information.
- **Temp Files**: Store temporary data.
---
### **3. Automatic Storage Management (ASM)**
- Oracle's file system and volume manager for managing database files.
- Manages disk groups for optimal performance and availability.
- Handles striping and mirroring of data automatically.
---
### **4. Cache Fusion**
- The mechanism that synchronizes the data buffer cache across all RAC instances.
- Ensures that each instance has a consistent view of the data.
- Uses the Global Cache Service (GCS) and Global Enqueue Service (GES) to manage locks and access.
---
### **5. Oracle Networking Components**
- Ensure connectivity between clients and the RAC database:
- **VIP (Virtual IP)**: Each node has a virtual IP address for client connections.
- **SCAN (Single Client Access Name)**: Provides a single hostname for the entire RAC cluster.
- **Listeners**: Each node runs a local listener for connection requests.
---
### **6. Background Processes**
RAC introduces additional background processes to manage cluster-specific tasks:
#### **a. Global Cache Service (GCS) Processes**
- Manage data block transfers between instances.
#### **b. Global Enqueue Service (GES) Processes**
- Handle global resource management, including locks.
#### **c. LMON (Lock Manager)**
- Monitors and manages global locks and cluster membership.
#### **d. LMD (Lock Manager Daemon)**
- Manages global enqueue requests and deadlock detection.
#### **e. LMS (Lock Manager Server)**
- Handles cache-to-cache block transfers for Cache Fusion.
#### **f. LCK (Lock Process)**
- Manages instance-level locks.
---
### **7. Shared Storage**
Shared storage is crucial for RAC, enabling all nodes to access the same database files:
#### **a. Disk Groups**
- Managed by ASM or other shared storage systems.
#### **b. Storage Types**
- SAN (Storage Area Network)
- NAS (Network Attached Storage)
- Shared disks (e.g., via Oracle ASM)
---
### **8. Management Tools**
Tools for managing and monitoring RAC environments:
#### **a. Oracle Enterprise Manager (OEM)**
- Provides a graphical interface for monitoring and managing RAC databases.
#### **b. Command-Line Tools**
- `srvctl`: Manages cluster database resources.
- `crsctl`: Manages clusterware processes.
- `asmcmd`: Manages ASM storage.
---
### **9. High Availability Features**
Features that enhance RAC's reliability and fault tolerance:
#### **a. Failover**
- Automatically redirects connections to surviving nodes in case of failure.
#### **b. Load Balancing**
- Distributes client connections across all nodes for optimal performance.
#### **c. Transparent Application Failover (TAF)**
- Ensures session continuity during node failures.
---
### **10. Configuration Files**
#### **a. Oracle Cluster Registry (OCR)**
- Stores cluster configuration information.
#### **b. Voting Disk**
- Records which nodes are active in the cluster.
#### **c. Parameter Files (SPFILE)**
- Stores instance-specific configuration parameters.
===========================
Here is a **step-by-step guide** for installing Oracle RAC on a new Linux OS for a 2-node cluster.
=========================
---
## **Step 1: Prerequisites**
### **1. Prepare the Operating System**
1. **Install Required Packages**:
```bash
yum install -y binutils compat-libcap1 gcc gcc-c++ glibc glibc-devel libaio libaio-devel libgcc libstdc++ libstdc++-devel ksh make sysstat smartmontools
```
2. **Configure Kernel Parameters**:
Edit `/etc/sysctl.conf` and add the following:
```bash
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4398046511104 # Set to half of physical memory
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
```
Apply the settings:
```bash
sysctl -p
```
3. **Set Limits for Users**:
Edit `/etc/security/limits.conf`:
```bash
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
```
4. **Disable SELinux**:
Edit `/etc/selinux/config`:
```bash
SELINUX=permissive
```
Apply the change:
```bash
setenforce 0
```
5. **Disable Firewall**:
```bash
systemctl stop firewalld
systemctl disable firewalld
```
6. **Enable NTP/Chrony**:
Install and configure `chronyd` for time synchronization:
```bash
yum install -y chrony
systemctl enable chronyd
systemctl start chronyd
chronyc sources
```
---
### **2. Configure Shared Storage**
- Provision shared disks using ASM (Automatic Storage Management).
- Use tools like `multipath`, `ASMlib`, or `udev` to configure and map shared storage.
Example for udev rules:
```bash
vi /etc/udev/rules.d/99-oracle-asm.rules
KERNEL=="sd*", NAME="oracleasm/%k", OWNER="grid", GROUP="asmadmin", MODE="0660"
```
Reload udev:
```bash
udevadm control --reload-rules && udevadm trigger
```
---
### **3. Configure Network**
1. **Set Public, Private, and Virtual IPs in `/etc/hosts`**:
```bash
192.168.1.101 rac1-public
192.168.1.102 rac2-public
10.0.0.1 rac1-private
10.0.0.2 rac2-private
192.168.1.111 rac1-vip
192.168.1.112 rac2-vip
192.168.1.120 rac-scan
```
2. **Test Network Connectivity**:
```bash
ping rac2-public
ping rac1-private
```
---
### **4. Setup SSH Connectivity**
1. Generate SSH keys on both nodes for the `grid` user:
```bash
ssh-keygen -t rsa
```
2. Copy keys to each node:
```bash
ssh-copy-id grid@rac2
```
3. Verify passwordless SSH:
```bash
ssh rac2 hostname
```
---
## **Step 2: Install Grid Infrastructure**
1. **Download and Extract Oracle Grid Software**:
- Place the Grid Infrastructure installation files on a shared directory or local directory on each node.
- Example: `/u01/software/grid/`.
2. **Run the Installer**:
On Node 1:
```bash
/u01/software/grid/runInstaller
```
3. **Follow Installer Steps**:
- Choose **"Install and Configure Oracle Grid Infrastructure for a Cluster"**.
- Select **Cluster Name** and provide SCAN name and SCAN listener.
- Specify the **Cluster Node Information** (e.g., Node 1 and Node 2 details).
- Assign **Public**, **Private**, and **Virtual IPs** for each node.
- Provide the shared disk details for ASM configuration.
- Select the disk group redundancy (e.g., Normal/High/External).
4. **Run Configuration Scripts**:
- The installer will prompt you to run scripts like `root.sh` on both nodes.
---
## **Step 3: Install Oracle RAC Database**
1. **Download Oracle Database Software**:
- Place the database installation files on a shared directory or local directory on each node.
- Example: `/u01/software/db/`.
2. **Run the Installer**:
On Node 1:
```bash
/u01/software/db/runInstaller
```
3. **Follow Installer Steps**:
- Choose **"Install Database Software Only"**.
- Select **"Oracle Real Application Clusters"**.
- Add both nodes to the cluster configuration.
4. **Run Configuration Scripts**:
- The installer will prompt you to run scripts like `root.sh` on both nodes.
5. **Create RAC Database**:
After software installation, use `DBCA` (Database Configuration Assistant) to create the RAC database:
```bash
dbca
```
- Choose **"Oracle RAC Database"**.
- Select both nodes.
- Specify database name, SID prefix, and ASM disk group for storage.
---
## **Step 4: Verify the RAC Installation**
1. **Check Cluster Status**:
```bash
crsctl check cluster
```
2. **Verify ASM Disk Groups**:
```bash
asmcmd lsdg
```
3. **Verify Database Status**:
```bash
srvctl status database -d <db_unique_name>
```
4. **Test Failover**:
- Stop the database instance on one node and check if the other node continues to function:
```bash
srvctl stop instance -d <db_unique_name> -i <instance_name>
srvctl status database -d <db_unique_name>
```
---
===========================
Here are the high-level steps for installing Oracle RAC 19c on Linux:
Pre-Installation
1)System Requirements:
Ensure the hardware meets Oracle's requirements.
Verify the Linux version is supported (e.g., RHEL 7.x or 8.x).
2)Configure Nodes:
Set up multiple nodes for the cluster with appropriate hostname and IP configuration.
Configure public, private (interconnect), and virtual IPs for each node.
3)Shared Storage:
Provision shared storage for ASM (Automatic Storage Management) using SAN, NFS, or iSCSI.
Label and map ASM disks correctly on all nodes.
4)OS Configuration:
Configure kernel parameters (/etc/sysctl.conf) as per Oracle's recommendations.
Set up required OS groups (e.g., oinstall, dba).
Install prerequisite packages using the runcluvfy.sh tool or Oracle-provided RPM.
5)Network Setup:
Configure name resolution in /etc/hosts or DNS.
Ensure SSH passwordless connectivity between all nodes for the grid and oracle users.
Grid Infrastructure Installation
Install Oracle Grid Infrastructure :
====================================
1)Run the Oracle Grid Infrastructure installer on one node.
Select the "Configure Oracle Grid Infrastructure for a Cluster" option.
Specify cluster configuration (nodes, public/private/virtual IPs).
Choose ASM for cluster storage and configure ASM disk groups.
2)Run Root Scripts:
Execute root.sh and orainstRoot.sh as instructed during the installation process.
3)Verify Cluster Setup:
Use crsctl and srvctl commands to verify that the cluster is running.
Oracle RAC Database Installation
4)Install Oracle Database Software:
Run the Oracle Database installer on one node.
Select "Install database software only" and choose RAC installation.
Ensure the installation is applied to all nodes in the cluster.
5)Create a RAC Database:
Use DBCA (Database Configuration Assistant) to create a RAC database.
Select the ASM disk groups for storage.
Configure services, memory, and other parameters.
Post-Installation
6)Verify Database and Cluster:
Check the database and RAC configuration with srvctl and crsctl.
Ensure all instances are running across nodes.
7)Configure Backups:
Set up RMAN backups and validate recovery strategies.
8)Performance Tuning:
Configure load balancing and failover policies.
Test interconnect and storage performance.
9)Monitoring and Maintenance:
Use OEM or third-party tools to monitor the RAC environment.
Regularly apply patches and updates.
===========================
Here are the high-level steps for creating an Oracle database on Linux:
===========================
Pre-Creation Steps
Verify System Requirements:
Ensure the system meets Oracle's hardware and software requirements.
Verify prerequisites such as sufficient memory, storage, and OS settings.
1)Install Oracle Database Software:
Install the Oracle Database software if not already installed.
Use the Oracle Universal Installer (OUI) and choose "Install database software only" if planning to create the database later.
2)Set Environment Variables:
Configure environment variables (ORACLE_HOME, ORACLE_SID, PATH, etc.) in the Oracle user’s profile file (e.g., .bash_profile).
3)Prepare Storage:
Set up file system or ASM for database storage.
Ensure permissions are correctly configured for the Oracle user.
Database Creation
4)Run DBCA (Database Configuration Assistant):
Start dbca from the Oracle Home:
dbca
5)Choose one of the following options:
Create a Database: Use this for creating a new database interactively.
Template-Based Creation: Use pre-configured templates for quick setup.
6)Database Configuration Options:
Global Database Name: Specify the database name and SID.
Storage Type: Choose between ASM and file system.
Character Set: Select the appropriate character set (e.g., AL32UTF8).
Memory Settings: Configure SGA/PGA based on server resources.
Options & Features: Select features like Enterprise Manager, sample schemas, etc.
7)Specify Listener and Networking:
Configure listener details or use an existing one.
8)Create Database Files:
Specify the location of database files (e.g., control files, data files, redo logs).
If using ASM, select the appropriate disk group.
9)Database Initialization:
DBCA will generate initialization parameter files (spfile) and start the database.
Post-Creation Steps
10)Validate Database:
Log in to the database to verify it is running:
sqlplus / as sysdba
SELECT name FROM v$database;
11)Configure Backup:
Set up RMAN for database backups.
Tune Database:
Adjust initialization parameters based on workload requirements.
12)Monitor and Maintain:
Use Oracle Enterprise Manager (OEM) or SQL scripts to monitor the database.
Regularly check logs (alert.log) for issues.
===============
Oracle ASM :
==============
(Automatic Storage Management) is an integrated storage management solution that simplifies the management of database storage, provides high availability, and optimizes performance. Here are key ASM concepts and their uses:
ASM Instance :
--------------
An ASM instance is a background process in Oracle that manages the storage for Oracle databases. It runs separately from the database instance and handles the physical storage layout and allocation.
Use: Manages database files (data files, redo logs, etc.) in an Oracle environment, providing high availability, redundancy, and performance optimizations.
Uses of ASM in Oracle Databases :
High Availability: ASM's mirroring and fault tolerance capabilities ensure that database files remain available in case of hardware failures.
Performance: ASM optimizes I/O by distributing data evenly across disks and supporting striping for faster access.
Simplified Storage Management: ASM abstracts the underlying storage hardware, making it easier to manage database files without worrying about individual disks and partitions.
Scalability: ASM allows you to easily add disks to disk groups, enabling easy expansion of storage as needed.
Cost Savings: By managing different types of storage efficiently, such as using SSDs and HDDs together, ASM can optimize costs while ensuring good performance.
In summary, ASM is a powerful storage management solution that enhances the performance, availability, and scalability of Oracle databases by managing storage resources and ensuring high fault tolerance.
Example:
Assume you are adding a disk /dev/oracleasm/disks/DISK5 to the disk group DATA.
ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/DISK5';
ALTER DISKGROUP DATA REBALANCE POWER 5;
Example: create tbs in asm
CREATE TABLESPACE NEW_TBS
DATAFILE '+DATA' SIZE 500M AUTOEXTEND ON NEXT 100M MAXSIZE 5G ;
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO;
Add a Datafile to the USERS Tablespace in the DATA Disk Group:
ALTER TABLESPACE USERS ADD DATAFILE '+DATA' SIZE 100M AUTOEXTEND ON NEXT 50M MAXSIZE 5G;
Example: Resize
ALTER DATABASE DATAFILE '+DATA/ORCL/DATAFILE/users.257.123456789' RESIZE 1G;
ASM QUERY -
To check diskgroups from asm instance :
select group_number,name,state,type from v$asm_diskgroup;
To check asm instance is connected and running:
select group_number,db_name,status from v$asm_client;
To check the asm disks and their status :
select group_number,mount_status,state ,header_status from v$asm_disk;
Instances status
select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME from gv$instance;
CREATING DISKGROUP :
--------------------
EXTERNAL REDUNDANCY - 0 copy
CREATE DISKGROUP DATA1 EXTERNAL REDUNDANCY DISK
'/dev/oracleasm/disks/DATA1';
NORMAL REDUNDANCY - 1main and 1copy of data
CREATE DISKGROUP DATA2 NORMAL REDUNDANCY
FAILGROUP FG1 DISK
'/dev/oracleasm/disks/D1' NAME DISK2
FAILGROUP FG2 DISK
'/dev/oracleasm/disks/D2' NAME DISK2;
HIGH REDUNDANCY : 1 Main and 2copy of data
CREATE DISKGROUP DATA3 HIGH REDUNDANCY DISK
'/dev/oracleasm/disks/D3','/dev/oracleasm/disks/D4','/dev/oracleasm/disks/D5';
Add disks into diskgroup :
ALTER DISKGROUP DG1 ADD DISK
'/dev/oracleasm/disks/D1',
'/dev/oracleasm/disks/D2';
DROP DISK - First we need to dismount the disk on all nodes and mount on 1 node then drop disk.
ALTER DISKGROUP DATA1 DISMOUNT;
DROP DISKGROUP DATA1 including contents;
ALTER DISKGROUP DATA1 DROP DISK D1;
ALTER DISKGROUP DATA1 DISK A*;
ALTER DISKGROUP DATA1 UNDROP DISKS; (We can undrop the disks using this cmd)
CHANGE REDUNDANCY NOT POSSIBLE AND CHANGE ALLOCATION UNIT NOT POSSIBLE (If we want to do then create new diskgroup move datafile to that dg and drop old DG)
Alter diskgrup :
ALTER DISKGROUP DATA1 MOUNT;
MOVE DATAFILE TO ANOTHER DISKGROUPS :
ALTER DATABASE MOVE DATAFILE '+DATA/DEVDB/DATAFILE/system.dbf' TO '+DGROUB';
compatibility check in ASM disks :
COL name FOR A10;
COL compatibility FOR A15;
SELECT name, compatibility FROM v$asm_diskgroup;
ALTER DISKGROUP ASMDATA1 SET ATTRIBUTE 'compatible.asm' = '19.0';
ASM spfile location
show parameter spfile;
/u01/app/19c/grid/dbs/spfile+ASM2.ora
Password file ASM
srvctl config asm
Password file: +OCR1/orapwASM
ASMCMD - DATAFILE location in asm
+ASMDATA1/ORA19C/DATAFILE
lsdsk -k (To check the asmdisks)
ASM DYNAMIC PERFORMANCE VIEWS
V$ASM_ALIAS V$ASM_ATTRIBUTE V$ASM_CLIENT
V$ASM_DISK V$ASM_DISK_IOSTAT V$ASM_DISK_STAT
V$ASM_DISKGROUP V$ASM_DISKGROUP_STAT V$ASM_FILE
V$ASM_OPERATION V$ASM_TEMPLATE
--------------------------------------
ASM Background Process in DB Instance :
--------------------------------------
Oracle database that uses ASM disks, two new background processes exists
ASMB : ASMB performs communication between the database and the ASM instance.
RBAL: RBAL performs the opening and closing of the disks in the disk groups on behalf of Oracle database.The RBAL performs rebalancing when a disk is added or removed.
ARBn: The ARBn performs actual extent movement between the disks in the diskgroup.
This RBAL is the same process as in ASM instance but it performs a different function.
To find asm background process in oracle db instance, connect to Oracle database and issue below query
SQL> select sid, serial#, process, name, description
from v$session join v$bgprocess using(paddr);
LMON global enqueue service monitor
GMON diskgroup monitor
MMAN Memory Manager
RBAL ASM Rebalance master
MMNL Manageability Monitor Process 2
ASMB ASM Background process database and asm communication
LMS0 global cache service process
What is the ASM POWER_LIMIT? :
----------------------------
Ans: This is the parameter that controls the number of Allocation units the ASM instance will try to rebalance at any given time. In ASM versions less than 11.2.0.3 the default value is 11 however it has been changed to unlimited in later versions.
asm_power_limit integer 1
How do you back up ASM Metadata? :
Ans: You can use md_backup to restore the ASM disk group configuration in case of ASM disk group storage loss.
20. What files can be stored in the ASM disk group?
Ans: In 11g the following files can be stored in ASM disk groups.
Datafiles
Redo logfiles
Spfiles
In 12c the files below can also now be stored in the ASM Diskgroup
Password file
What are Diskgroup’s and Failuregroups?
---------------------------------------
Diskgroup is a terminology used for logical structure which holds the database files. Each Diskgroup consists of Disks/Raw devices where the files are actually stored. Any ASM file is completely contained within a single disk group. However, a disk group might contain files belonging to several databases and a single database can use files from multiple disk groups.
Failuregroups are used when using Normal/High Redundancy. They contain the mirrored ASM extents and must be containing different disks and preferably on separate disk controller.
How does ASM provides Redundancy?
---------------------------------
When you create a disk group, you specify an ASM disk group type based on one of the following three redundancy levels:
Normal for 2-way mirroring – When ASM allocates an extent for a normal redundancy file, ASM allocates a primary copy and a secondary copy. ASM chooses the disk on which to store the secondary copy in a different failure group other than the primary copy.
High for 3-way mirroring. In this case the extent is mirrored across 3 disks.
External to not use ASM mirroring. This is used if you are using Third party Redundancy mechanism like RAID, Storage arrays.
Can we change the Redundancy for Diskgroup after its creation. :
------------------------------------
No, we cannot modify the redundancy for Diskgroup once it has been created. To alter it we will be required to create a new Diskgroup and move the files to it. This can also be done by restoring full backup on the new Diskgroup. Following metalink note describes the steps
Note.438580.1 – How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy)
I am unable to open the ASM instance. What is the reason?
------------------------------
ASM instance does not have open stage. It has got only two options
Nomount- This starts the ASM instance
Mount- At this stage, Diskgroup defined in ASM_DISKGROUPS parameter are mounted
When you try to open the ASM instance , you get following error
alter database open
*
ERROR at line 1:
ORA-15000: command disallowed by current instance type
What kind of files can be stored on ASM Diskgroup:
-------------------------------------------
You can store the following file types in ASM disk groups:
Control files
Datafiles, temporary datafiles, and datafile copies
SPFILEs
Online redo logs, archive logs, and Flashback logs
RMAN backups
Disaster recovery configurations
Change tracking bitmaps
Data Pump dumpsets
Does ASM instance automatically rebalances and takes care of hot spots?
No. This is a myth and ASM does not do it. It will initiate automatic rebalance only when a new disk is added to Diskgroup or we drop a disk from existing Diskgroup.
How can we copy the files from/to ASM?
-----------------------------------
You can use RMAN or DBMS_FILE_TRANSFER.COPY_FILE procedure to copy the files to/from ASM from/to Filesystem. Starting from Oracle 11g, you can use cp command in asmcmd to perform the same between ASM Diskgroups and also to OS Filesystem.
Whats is Kfed?
----------------
kfed is a utility which can be used to view the ASM Disk information. Syntax for using it is
KFED (Kernel Filesystem Editor):
Purpose:
Used to inspect and repair ASM metadata structures.
Operates at the ASM disk level and can read or edit the ASM disk header and other metadata.
Usage:
Primarily used for advanced troubleshooting and support when there are issues with ASM disks or disk groups.
Commands:
View ASM disk headers:
kfed read /dev/<asm_disk>
kfed repair /dev/<asm_disk> [parameters]
kfed read datadisk
* KFOD (Kernel Filesystem Oracle Disk Discovery):
Purpose:
Used for ASM disk discovery.
Lists ASM disks and their attributes, such as disk names, sizes, headers, and states.
Usage:
Commonly used during ASM configuration and troubleshooting to verify the visibility of ASM disks.
Command Example:
kfod asm_diskstring='/dev/oracleasm/disks/*'
Output:
Shows details about discovered ASM disks, including:
Disk name
Disk group
Status
Size
Failure group
What is the maximum size of Disk supported by ASM?
-------------------------------------------------
ASM supports disks upto 2Tb, so you need to ensure that lun size should be less then 2Tb. 10.2.0.4 and 11g database will give error if you try to create a diskgroup with ASM disks having disk size >2Tb.
Maximum Number of Disks in a Disk Group:
Oracle 11g and Later:
Up to 10,000 disks per disk group.
Verify Disk Shareability with ASM Tools :
--------------------------------------
If the disk is used for ASM, confirm it is marked as shareable:
Using oracleasm Utility:
Check the disk configuration:
oracleasm listdisks
Check if the shared disk is visible on all nodes:
lsblk
Flex ASM (Flexible ASM):
-------------------------
It is a feature introduced in Oracle 12c (12.1) that enhances the traditional ASM (Automatic Storage Management) by providing increased flexibility, scalability, and high availability in Oracle RAC (Real Application Clusters) environments.
How Flex ASM Works
Flex ASM Cluster Mode:
Flex ASM is enabled at the cluster level during the Oracle Grid Infrastructure installation or configuration.
ASM Listeners:
ASM clients use the ASM network listeners to connect to ASM instances. This may include remote connections if the local ASM instance is unavailable.
ASM Network:
Flex ASM uses a private ASM network for communication between ASM instances and database clients.
ASM Client Access:
Each database client is connected to one or more ASM instances. The connection is managed dynamically, with failover capabilities.
Benefits of Flex ASM :
-----------------------
Scalability:
In large clusters, you can have a smaller number of ASM instances managing storage for a large number of database instances.
High Availability:
Automatic client failover ensures that database operations continue seamlessly if an ASM instance fails.
Resource Optimization:
Reduces the resource overhead of running ASM instances on every node in the cluster.
Dynamic Load Balancing:
Connections between database clients and ASM instances are dynamically load-balanced for better performance.
Verify :
ASMCMD> showclustermode
ASM cluster : Flex mode enabled - Direct Storage Access
ASMCMD> exit
[grid@dr ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwASM
Backup of Password file: +DATA/orapwASM_backup
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
-------------------
Example relatime :
---
### **Problem: ASM Disk Group Not Mounting on One Node**
**Scenario:**
- A 2-node Oracle RAC cluster is configured with ASM for storage.
- After a node reboot, the ASM disk group fails to mount on Node 2, while it is mounted and operational on Node 1.
- This causes database services running on Node 2 to crash, and the instance cannot start due to missing storage.
---
### **Investigation Steps:**
1. **Check ASM Disk Group Status:**
- On Node 1:
```sql
SQL> SELECT name, state FROM v$asm_diskgroup;
```
- The disk group is in the `MOUNTED` state.
- On Node 2:
```sql
SQL> SELECT name, state FROM v$asm_diskgroup;
```
- The disk group shows as `DISMOUNTED`.
2. **Check ASM Disk Visibility:**
- Verify if the ASM disks are visible on both nodes:
```bash
oracleasm listdisks
```
- Some disks were not listed on Node 2.
3. **Check OS-Level Disk Accessibility:**
- Use the following command to check if the disks are accessible at the OS level:
```bash
ls -l /dev/oracleasm/disks/
```
- Missing or incorrect permissions were identified for some disks on Node 2.
4. **Verify Disk Discovery String:**
- Check the ASM `diskstring` parameter:
```sql
SQL> SHOW PARAMETER asm_diskstring;
```
- The discovery path was correct, but the missing disks indicated an underlying issue.
5. **Check Logs:**
- Review the ASM alert log and system messages:
```bash
tail -f /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM2.log
tail -f /var/log/messages
```
- Errors such as `ORA-15032: not all alterations performed` and `ORA-15040: diskgroup is incomplete` were found.
---
### **Root Cause:**
- After the reboot, the device paths for ASM disks were not properly reconfigured or recognized by Node 2. This could be due to:
- Incorrect udev rules for persistent device naming.
- Disk multipathing (e.g., `DM-Multipath`) misconfiguration.
---
### **Solution:**
1. **Reconfigure Disk Permissions:**
- Ensure correct ownership and permissions for ASM disks:
```bash
chown -R grid:asmadmin /dev/oracleasm/disks/*
chmod 660 /dev/oracleasm/disks/*
```
2. **Verify and Correct Multipathing Configuration:**
- Check the multipath configuration:
```bash
multipath -ll
```
- Update `/etc/multipath.conf` if necessary, and reload multipath:
```bash
service multipathd reload
```
3. **Reload ASM Disk Discovery:**
- Rescan ASM disks on Node 2:
```bash
oracleasm scandisks
```
4. **Mount the Disk Group:**
- Mount the disk group on Node 2:
```sql
SQL> ALTER DISKGROUP <diskgroup_name> MOUNT;
```
5. **Check RAC Instance Status:**
- Restart the database instance on Node 2 if required:
```bash
srvctl start instance -d <db_name> -n <node_name>
```
6. **Validate RAC Cluster:**
- Verify cluster and database status:
```bash
crsctl stat res -t
srvctl status database -d <db_name>
```
---
### **Preventive Actions:**
- Ensure persistent device paths using correct **udev rules** or **ASMLib**.
- Test disk visibility on all nodes after any storage changes or reboots.
- Implement regular checks for multipath configurations.
---
### **Outcome:**
- After resolving the disk visibility issue and mounting the disk group, the RAC database was fully operational on both nodes. Services were restored without data loss.
=======================================
Here's an easier-to-understand version of the Oracle RAC (Real Application Clusters) working process:
==========================
Oracle RAC Working Process:
===========================
Step 1: Starting Clusterware Services
OHASD starts the Oracle Clusterware by initiating background processes like crsd.bin for Cluster Ready Services and cssd.bin for the Cluster Synchronization Service.
CSSD checks the status of each node using heartbeats stored in the Voting Disk. It verifies connectivity and cluster membership before proceeding with node startup.
CRS is initiated, which brings up essential resources and manages cluster components.
Step 2: ASM and Storage Management Initialization :
=================================================
ASM initializes, providing a volume manager and file system for database storage. It is managed by the Oracle Clusterware and ensures consistent data access across nodes.
ASM starts its own set of background processes (ASM instance), managing disk groups and rebalancing workloads across disks to prevent bottlenecks.
Step 3: Database Instance Startup :
=================================
Each node starts its own instance of the Oracle database, connecting to the shared ASM storage. Each instance has dedicated processes for memory and process management (e.g., DBWR, LGWR).
Cache Fusion, facilitated by the interconnect, allows instances to share data in real-time, synchronizing blocks among nodes to ensure data consistency.
Step 4: SCAN and VIP Addressing :
===============================
The SCAN service provides a single point of entry for clients. It allows connections to be routed across nodes, depending on load and node availability.
VIPs facilitate quick redirection of connections in case a node goes down. If a node fails, its VIP will move to another node, minimizing disruption for the clients.
Failover and Recovery in RAC:
=============================
In the event of a node failure, Oracle Clusterware identifies the failed node and performs eviction using the Voting Disk to prevent split-brain.
CRS automatically restarts the failed instance on another node, and VIP migrates, ensuring high availability.
ASM rebalance may occur if disk groups are affected, ensuring continuous access to data without impact on database operations.
Oracle RAC's combination of Clusterware, ASM, Grid Infrastructure, and network components ensures a high-performing, resilient, and scalable database environment. By balancing workloads and providing failover capabilities, Oracle RAC supports businesses with demanding, always-available applications.
======================================
PFILE NEW PARAM IN RAC :
cluster_database - options are true or false, mounts the control file in either share (cluster) or exclusive mode, use false in the below cases
Converting from no archive log mode to archive log mode and vice versa
Enabling the flashback database feature
Performing a media recovery on a system table
Maintenance of a node
Alternatively, stop it on the local node:
crsctl stop crs
Start the Clusterware in Exclusive Mode:
Use the crsctl start cluster command with the -excl flag to start the cluster in exclusive mode:
crsctl start cluster -excl
Verify Cluster State:
Check the cluster status to confirm it is running in exclusive mode:
crsctl check cluster
active_instance_count - used for primary/secondary RAC environments
cluster_database_instances - specifies the number of instances that will be accessing the database (set to maximum # of nodes)
dml_locks - specifies the number of DML locks for a particular instance (only change if you get ORA-00055 errors)
gc_files_to_locks - specify the number of global locks to a data file, changing this disables the Cache Fusion.
max_commit_propagation_delay - influences the mechanism Oracle uses to synchronize the SCN among all instances
instance_groups - specify multiple parallel query execution groups and assigns the current instance to those groups
parallel_instance_group - specifies the group of instances to be used for parallel query execution
gcs_server_processes - specify the number of lock manager server (LMS) background processes used by the instance for Cache Fusion
remote_listener - register the instance with listeners on remote nodes.
local_listener -
==================
RAC QUESTIONS :
===================
What is OCLUMON used for in a cluster environment? :
---------------------------
Ans: The Cluster Health Monitor (CHM) stores operating system metrics in the CHM repository for all nodes in a RAC cluster. It stores information on CPU, memory, process, network, and other OS data, This information can later be retrieved and used to troubleshoot and identify any cluster-related issues.
It is a default component of the 11gr2 grid install. The data is stored in the master repository and replicated to a standby repository on a different node.
how to change the VIP in rac :
-------------------------
Backup Configuration:
Backup your RAC configuration files (e.g., /etc/hosts, network, grid_home, etc.).
srvctl config vip -node <node_name>
Stop the RAC Services on the Node
stop VIP :
srvctl stop vip -node <node_name>
Stop Related Services: Stop other services dependent on the VIP (e.g., database, listener) on the node:
srvctl stop database -db <db_name>
srvctl stop listener -node <node_name>
Change the VIP Address :
Update the VIP Configuration: Use srvctl to modify the VIP:
srvctl modify vip -node <node_name> -address <new_vip_address>
Verify the Change: Check the updated VIP configuration:
srvctl config vip -node <node_name>
What are some of the RAC-specific parameters?
Ans: Some of the RAC parameters are:
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
INSTANCE_TYPE (RDBMS or ASM)
ACTIVE_INSTANCE_COUNT
UNDO_MANAGEMENT
undo_tablespaces
cluster_interconnects
remote_listener
thread
What current block and CR block and PI in RAC? :
-------------------------------------------
Data block requests from global cache are of two types.
current block(cur) – When we want to update a data, oracle must locate the most recent version of the block in cache, it is known as current block
consistent read(CR) – When we want to read a data, then only committed data will be provided( with help of undo). that is known as consistent read.
There can be multiple CR blocks, But there will be always one Current block.
There can multiple scur(shared current) , But only xcur( exclusive current).
Past image(PI) – When node A wants to updates the block, which is present on node B and node B has also updated the block , then node B will send the current copy of the block to Node A, it will keep a past image( PI) of the block , until it is written to the disk. Once commit happens on node B for that transaction or when checkpoint happens , the PI images will be flushed to disk.
What is gc buffer busy wait?
---------------------------
Mean a session is trying to access a buffer in buffer cache, But that particular buffer is currently busy with global cache operation.So during that time gc buffer busy wait will happen.
Example –
Let’s say session A want to access block id 100 , But currently that block is in buffer cache of session B.
So session A requested session B LMS process to transfer the block.
While transfer is going on , session B also tried to access that block. But as already that block/buffer is already busy in global cache operation. Session B has to wait with wait event, gc buffer busy wait.
Reasons – Concurrency related, Right hand index growth.
Why RAC has separate redo thread for each node?
---------------------------
In RAC, each instance have their own lgwr process , So there has to be separate online redolog for each instance ( called as thread), So that lgwr will write to the respective redo log.
Why RAC has separate undo tablespace for each node?
------------------------------------
In Oracle Real Application Clusters (RAC), each node has a separate undo tablespace to facilitate the management of undo data in a distributed and parallel processing environment. The primary reasons for having separate undo tablespaces for each node in a RAC setup include:
Isolation of Undo Segments:
Parallel Processing and Scalability:
Reduced Global Resource Contention:
Enhanced Concurrency:
Improved High Availability:
Master node in RAC :
--------------------
In an Oracle RAC (Real Application Clusters) environment, the concept of a "master node" plays an essential role in managing specific cluster-related tasks and coordination.
1. Cluster Resource Coordination
The master node helps coordinate access and management of cluster resources such as voting disks, OCR (Oracle Cluster Registry), and database resources.
It ensures orderly communication and arbitration in the cluster.
2. Cluster Synchronization
The master node maintains synchronization across nodes, ensuring consistency in activities like cache coherency, lock management, and global resource allocation.
3. Cluster Health Monitoring
The master node manages certain health checks and ensures the integrity of the cluster. For example, it may be responsible for initiating reconfiguration in case of node failures or adding nodes.
4. Node Arbitration
In case of a network split or other failure scenarios, the master node participates in quorum-based decisions and manages the voting process.
Check master node :
* crsctl query css votedisk
Disk group: DATA
/dev/asm_disk1
/dev/asm_disk2
/dev/asm_disk3
Voting disk file(s):
Disk 1: /dev/asm_disk1
Disk 2: /dev/asm_disk2
Disk 3: /dev/asm_disk3
Master node: rac1
* ocrcheck
Status of Oracle Cluster Registry is as follows:
Version: 3
Total space (kbytes): 262120
Used space (kbytes): 4000
Available space (kbytes): 258120
ID: 1536440416
Device/File Name: +DATA
Cluster registry integrity check succeeded
OCR master node: rac2
[grid@dr ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE dr STABLE
ora.crf
1 ONLINE ONLINE dr STABLE
ora.crsd
---------
1 ONLINE ONLINE dr STABLE
SELECT inst_id, master_node, name FROM gv$resource WHERE type = 'GC';
CRF - CLuster Resource File :
------------------------------
In Oracle RAC, CRF stands for Cluster Health Monitor Repository File or Cluster Resource File depending on the context. It is primarily associated with Cluster Health Monitor (CHM), which is a tool used to collect and store detailed diagnostic data about the cluster's operating environment.
Purpose of CRF (Cluster Health Monitor Repository File):
Health Monitoring:
CHM collects data about CPU, memory, processes, network, and disk usage from all nodes in the cluster.
This data is stored in the CRF repository and can be used for troubleshooting and performance tuning.
Failure Diagnosis:
Helps Oracle's Diagnostic Framework (ADR) to identify and analyze issues like node evictions, cluster hangs, or performance degradation.
Integration with ORAchk and TFA:
The data stored in CRF is utilized by tools like ORAchk and TFA (Trace File Analyzer) to generate health reports and identify potential issues.
CRF Repository Location
The CRF data repository is managed under the Cluster Health Monitor and typically resides in:
$GRID_HOME/crf
What is Flex Cluster introduced in oracle 12c? :
-------------------------------------------------
Oracle Flex Cluster, introduced in Oracle Database 12c, is a clustering architecture offering two types of server pools: Hub-and-Spoke and Uniform. It provides flexible resource management, simplified administration, and enhanced high availability for Oracle RAC environments.
Hub-and-Spoke Architecture
Uniform Cluster Architecture
Dynamic Resource Management
Enhanced High Availability
Simplified Administration
Key Features of Flex Cluster
Separation of Roles:
Hub Nodes focus on critical database operations.
Leaf Nodes handle application workloads, reducing the load on Hub Nodes.
Shared and Non-Shared Storage:
Hub Nodes access shared storage for RAC database operations.
Leaf Nodes do not need access to shared storage, simplifying infrastructure requirements.
Improved Interconnect Usage:
Leaf Nodes do not need direct interconnect communication, reducing the inter-node communication overhead.
Integration with Oracle Flex ASM:
Flex Cluster works seamlessly with Oracle Flex ASM, enabling efficient storage management.
Node Limits in Oracle Clusters :
--------------------------------
1. Standard Cluster:
Maximum Nodes:
Up to 100 nodes can be configured in a Standard Cluster.
Details:
All nodes are tightly coupled.
Every node communicates directly with all other nodes via the private interconnect.
2. Flex Cluster:
Maximum Nodes:
64 Hub Nodes and thousands of Leaf Nodes.
Details:
Hub Nodes:
Full members of the cluster with access to shared storage.
Can run Oracle RAC database instances.
Leaf Nodes:
Lightweight nodes used for application or middleware workloads.
Communicate only with Hub Nodes and not directly with other Leaf Nodes.
What is TAF? :
--------------
Automatic Failover: TAF allows client connections to be automatically redirected to another available RAC node if the currently connected node or instance fails.
Transparent to Applications: TAF is transparent to the application layer, meaning that applications do not need to handle connection failover logic explicitly. The failover process is managed by Oracle client libraries and Oracle Net Services.
Session Persistence: TAF maintains session state during failover, ensuring that in-flight transactions and session data are preserved when switching to a new node.
Fast Recovery: TAF facilitates fast recovery by quickly redirecting client connections to an available node, minimizing downtime and disruption to application users.
Supported Failover Modes: TAF supports various failover modes, including Failover, Restart, and Session State. These modes determine the behavior of client connections during failover events.
What is TAF in oracle RAC?
----------------------------
BASIC
PRECONNECT
SELECT FAILOVER
SESSION FAILOVER
Configuration of TAF :
1. Ensure the Database is Configured for High Availability
Cluster Configuration: TAF is typically used in a Real Application Clusters (RAC) environment, but it can also be used with Data Guard.
Ensure the database services are registered with both instances in the cluster
2. Create a Service for TAF
Use DBMS_SERVICE to create a service that includes TAF attributes. For example:
BEGIN
DBMS_SERVICE.CREATE_SERVICE(
service_name => 'my_taf_service',
network_name => 'my_taf_service'
);
DBMS_SERVICE.START_SERVICE('my_taf_service');
END;
/
Alternatively, use srvctl for RAC:
srvctl add service -db mydb -service my_taf_service -preferred inst1 -available inst2
srvctl start service -db mydb -service my_taf_service
3. Configure TAF Parameters
Specify the TAF configuration during service creation or later using SQL. Key attributes:
Failover Type:
SESSION: Failover to a new session.
SELECT: Failover and resume a query (if supported).
Failover Method:
BASIC: Failover occurs when the primary connection is lost.
PRECONNECT: Pre-establishes the backup connection.
Example:
EXEC DBMS_SERVICE.MODIFY_SERVICE(
service_name => 'my_taf_service',
failover_type => DBMS_SERVICE.FAILOVER_TYPE_SESSION,
failover_method => DBMS_SERVICE.FAILOVER_METHOD_BASIC
);
4. Configure the Client-Side TNS Entry
Update the tnsnames.ora file to include TAF settings. Example:
MY_TAF_SERVICE =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1.example.com)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = node2.example.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = my_taf_service)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 5)
(DELAY = 2)
)
)
)
TYPE: Failover type (SESSION or SELECT).
METHOD: Failover method (BASIC or PRECONNECT).
RETRIES: Number of retry attempts.
DELAY: Seconds to wait between retries.
5. Test the Configuration
Connect to the database using the service:
sqlplus user/password@MY_TAF_SERVICE
Simulate failure (e.g., stop one node or service) and verify that the session fails over to the other node.
6. Monitoring and Validation
Query the service configuration:
SELECT name, failover_type, failover_method FROM dba_services WHERE name = 'MY_TAF_SERVICE';
Use srvctl for RAC:
srvctl status service -db mydb -service my_taf_service
---------------------------------------------------------
Can we start crs in exclusive mode? and its purpose?
-------------------------------
Yes, Oracle Clusterware (CRS) can be started in exclusive mode. It is used for maintenance tasks, troubleshooting, and ensuring isolation of a single node from the cluster to perform critical operations without affecting other nodes’ services.
If crs is not coming up , then what are things you will start looking into?
---------------------
If CRS is not coming up, I would start by checking logs, reviewing alert logs, verifying dependencies, checking disk space, reviewing configuration files, verifying permissions, checking cluster interconnect, reviewing OS logs, checking cluster health, and investigating hardware failures.
Explain about local_listener and remote_listener parameter in RAC?
-------------------------
Local_listener:
Definition: This parameter specifies the network address that an Oracle RAC instance uses to listen for local connection requests.
Purpose: The local listener is responsible for managing connection requests originating from the same node where the instance is running.
Configuration: The local_listener parameter is set to a TNS entry or a net service name that resolves to a network address (hostname and port) where the local listener is running.
ALTER SYSTEM SET LOCAL_LISTENER = ‘mydb_listener’ SCOPE=both;
Here, ‘mydb_listener’ is a TNS entry pointing to the local listener’s address.
Remote_listener:
------------------
Definition: This parameter specifies one or more network addresses that an Oracle RAC instance uses to listen for remote connection requests from other instances in the cluster.
Purpose: The remote listener facilitates communication between instances on different nodes in the RAC cluster.
Configuration: The remote_listener parameter is set to a list of TNS entries or net service names that resolve to network addresses (hostnames and ports) where the remote listeners are running on other nodes in the cluster.
ALTER SYSTEM SET REMOTE_LISTENER = ‘node1_listener, node2_listener’ SCOPE=both;
Here, ‘node1_listener’ and ‘node2_listener’ are TNS entries pointing to the remote listeners on different nodes.
What is client side load balancing and server side load balancing? :
-------------------------------------------------------------
Client-side load balancing: Clients distribute requests across multiple servers, deciding which server to connect to based on various algorithms such as round-robin or least connections.
Server-side load balancing: Load balancing is handled by a dedicated device or software component, such as a load balancer or proxy server, which distributes incoming requests among multiple servers based on predefined algorithms and server health checks.
What are the RAC related background processes? :
--------------------
LMSn (Lock Manager Server): Manages global resource coordination and distributed locks across RAC instances.
LMD (Lock Manager Daemon): Coordinates lock-related messaging among RAC instances and manages lock conversion requests.
LMON (Global Enqueue Service Monitor): Monitors global resources and manages global enqueue services, such as space allocation for global resources.
LMSn (Global Cache Service Process): Handles cache fusion operations by transferring data blocks between RAC instances to maintain cache coherence.
GCR (Global Cache Service Resources): Coordinates global cache resources and manages cache fusion and resource requests.
GCS (Global Cache Service): Manages the global cache and facilitates cache fusion operations between RAC instances.
RMSn (Resource Manager Server): Controls resource allocation and prioritization for database sessions based on resource management policies.
RBAL (Rebalance): Coordinates instance and resource rebalancing operations to maintain workload distribution across RAC instances.
MMON (Manageability Monitor): Monitors database and instance health, collecting statistics and managing advisory components.
MMNL (Manageability Monitor Light): Light version of MMON for low-priority monitoring tasks.
DIA0 (Diagnosability Process): Handles diagnostic data collection and monitoring for cluster-related issues.
LREG (Local Enqueue Service Registration): Registers local enqueue services with the Global Cache Service (GCS).
Give details on RSMN:
--------------------
This process is called as Remote Slave Monitor.This process manages background slave process creation and
communication on remote instances. This is a background slave process.This process performs tasks on behalf of a
co-ordinating process running in another instance.
Give details on LCK0:
---------------------
This process is called as Instance enqueue process.This process manages non-cache fusion resource requests such as
libry and row cache requests.
Give details on GTX0-j :
----------------------
The process provides transparent support for XA global transactions in a RAC environment.The database autotunes the number of these processes based on the workload of XA global transactions.
What is GRD?
--------------
GRD stands for Global Resource Directory. The GES and GCS maintains records of the statuses of each datafile and each cahed block using global resource directory.This process is referred to as cache fusion and helps in data integrity.
What is the role of LMON background process?
---------------------
The LMON (Global Enqueue Service Monitor) background process in Oracle RAC manages and monitors global resources, including global enqueue locks and resources. It helps ensure consistency and coordination across multiple instances in the RAC cluster by managing distributed lock management and resolving lock conflicts.
How instance recovery happens in oracle RAC? :
-----------------------------------
When any one of the instance is crashed in RAC, then this node failure is detected by the surviving instances. Now the GRD resouces will be distributed across the existing instances. The instance which first detects the crash, will the start the online redo log thread of the crashed instance. The SMON of that instance.
Sequence
Normal RAC operation, all nodes are available.
One or more RAC instances fail.
Node failure is detected.
Global Cache Service (GCS) reconfigures to distribute resource management to the surviving instances.
The SMON process in the instance that first discovers the failed instance(s) reads the failed instance(s) redo logs to determine which blocks have to be recovered.
SMON issues requests for all of the blocks it needs to recover. Once all blocks are made available to the SMON process doing the recovery, all other database blocks are available for normal processing.
Oracle performs roll forward recovery against the blocks, applying all redo log recorded transactions.
Once redo transactions are applied, all undo records are applied, which eliminates non-committed transactions.
Database is now fully available to surviving nodes. will read the redo to do rollforward ( i.e to apply both committed and noncommited data). Once rollforward is done, it will rollback the uncommited transactions using UNDO tablespace of the failed instance.
What g stand for in views like gv$session , gv$sql etc.?
-----------------------
In Oracle Database, the “g” in views like gv$session, gv$sql, and similar views stands for “Global.” These views provide a global or cluster-wide perspective of database activity and resources in Oracle Real Application Clusters (RAC) environments.
What is load balancing advisory?
--------------------------------
Load Balancing Advisory in Oracle Database is a feature designed to assist in distributing client connections across instances within a Real Application Clusters (RAC) environment. It provides recommendations to help determine the optimal instance to direct a new connection request based on the current workload and resource utilization across the cluster.
The Load Balancing Advisory collects and analyzes statistics related to instance performance, such as CPU utilization, I/O rates, and other metrics. Based on these statistics, it determines which instance is the least loaded and most suitable to handle incoming connection requests.
By leveraging the Load Balancing Advisory, Oracle Connection Managers can intelligently route new connection requests to the recommended instance, thereby achieving better load distribution, maximizing resource utilization, and improving overall system performance in Oracle RAC deployments.
What is gc cr 2 way and gc cr 3 way? :
----------------------
gc cr 2-way (Global Cache Consistent Read 2-way):
This wait event occurs when an instance in the RAC environment is waiting for a consistent read block from another instance.
It indicates contention between two instances for access to the same consistent read block in the global cache.
The number “2-way” signifies that the contention is between two instances.
gc cr 3-way (Global Cache Consistent Read 3-way):
Similar to “gc cr 2-way”, this wait event occurs when an instance is waiting for a consistent read block from another instance.
However, “gc cr 3-way” indicates contention among three instances for access to the same consistent read block in the global cache.
It signifies contention between three instances
What are some RAC related wait events? :
------------------------------------
Some RAC-related wait events include:
gc buffer busy: Indicates contention for access to a global cache buffer in a RAC environment.
gc cr block busy: Denotes contention for access to a consistent read block in the global cache.
gc current block busy: Indicates contention for access to the current block in the global cache.
gc freelist: Denotes contention for access to the global cache freelist.
gc grant 2-way: Indicates contention for global cache resource grants between two RAC instances.
What is ACMS? :
------------
ACMS stands for Atomic Controlfile to Memory Service. It synchronizes the control file updates across Oracle Real Application Clusters (RAC) instances, ensuring consistency and high availability of database metadata.
In a rac cluster if one node gets down then how will you rectify the root couse ?
1)Immediate Response:
Ensure that other nodes in the cluster are still operational.
Verify if the failed node is completely unreachable or if it's just experiencing issues with certain services or components.
2)Check Cluster Logs and Alerts:
Review Oracle Clusterware logs, Cluster Health Monitor (CHM) logs, Oracle ASM alert logs, and operating system logs on the failed node to
identify any error messages or warnings leading up to the failure.
Use tools like Cluster Verification Utility (CVU) to assess the overall health of the cluster and identify potential issues.
3)Node Diagnostics:
Connect to the failed node remotely if possible and perform diagnostics to identify hardware or software issues.
Check system logs, hardware component status (using tools like Integrated Lights Out Manager - ILOM), and Oracle diagnostics
(e.g., Automatic Diagnostic Repository - ADR) for insights into the problem.
4)Attempt Node Restart :
If the node failure appears to be a transient issue, attempt to restart the node using Oracle Clusterware utilities like srvctl or crsctl.
Monitor the node's startup process for any errors or warnings.
5)Failover and Service Relocation:
If the failed node cannot be restarted immediately or if the failure persists, initiate a failover or service relocation to other
available nodes in the cluster.
Use Oracle Clusterware commands (srvctl) to relocate services or resources to healthy nodes to minimize downtime.
6)Root Cause Analysis:
Once the node is back online or services have been relocated, conduct a thorough root cause analysis to determine the underlying reason
for the node failure.
Investigate hardware failures, software issues, network problems, or configuration errors that may have contributed to the failure.
7)Resolution and Remediation:
Address the identified root cause by resolving hardware or software issues, applying patches or updates, adjusting configuration settings,
or implementing best practices for cluster management and maintenance.
Document the actions taken and any recommendations for preventing similar issues in the future.
8)Monitoring and Prevention:
Implement proactive monitoring and alerting mechanisms to detect and respond to potential node failures or performance issues in the future.
Configure Oracle Clusterware alarms, Grid Infrastructure Management Repository (GIMR), or third-party monitoring tools to provide real-time
visibility into cluster health and performance.
12) Mention what is split brain syndrome in RAC?
In Oracle RAC, all the instances/servers communicate with each other using a private network.
When the instance members in a RAC fail to ping/connect to each other via this private network and continue to
process data block independently. Then this process is referred as Split Brain Syndrome.
Simple Majority rule : truncate = (n/2) + 1 - n = no of nodes
3=nodes , 3/2+1 = 2.5 Which nodes written less than 2.5 will be evicted from cluster .
13) What happens if you keep split brain syndrome in RAC unresolved? How it can be resolved?
If you keep split brain syndrome unresolved, then there would be data integrity issue. The blocks changed in one instance
will not be locked and could be over-written by another instance. It is resolved by using the voting disk, it will decide which
node(s) will survive and which node(s) will be evicted
Voting disk -The Voting Disk. Contains information to determine which nodes are active members of the cluster at any given moment.
What is deadlock and how to resolve it
A deadlock occurs when two or more sessions are waiting for data locked by each other, resulting in all the sessions being blocked.
Oracle automatically detects and resolves deadlocks by rolling back the statement associated with the transaction that detects the deadlock.
Typically, deadlocks are caused by poorly implemented locking in application code.
Solution -
Oracle is smart and it is able to find deadlock situation within 3 seconds. Below are the options to avoid deadlocks inside the database
Ask the session getting deadlock error ORA-00060 to issue either COMMIT or ROLLBACK
Ask the waiting session to kill the SQL / transaction
alter system kill session 'sid,serial#';
Look inside alert log / trace file for the sessions involved in deadlock and inform application team to improve the code
As a DBA, whenever you get a Deadlock alert, immediately contact application team and inform them.
=====================================
RAC SETUP :
==============
192.168.1.101 - Public IP of NODE1 - (server connectivity as public)
192.168.0.101 Private IP NODE1 (node to node connectivity)
192.168.1.111 Virtual IP of NODE1(same subnet mask as Public IP,1Ping method and runs on public of each node)
192.168.1.102 - Public IP of NODE2 (server connectivity as public)
192.168.0.102 - private IP of NODE2 - (node to node connectivity)
192.168.1.112 Virtual IP of NODE2 (same subnet mask as Public IP,1Ping method and runs on public of each node)
192.168.1.121 db-scan.db.com db-scan (1static scan list)
192.168.1.122 db-scan.db.com db-scan
192.168.1.123 db-scan.db.com db-scan
We are configuring only 1static listner.becoz no DNS sever
(For 2node cluster we need total 9IPS- 2pub,2priv,2vip,3scan(dns) we are using statis scan only 1 ip)
======================================
Configure IPs
192.168.0.101 - Private IP - Make sure to turn it on automatically
192.168.1.101 - Public IP - Make sure to turn it on automatically
## as root
export PATH=$PATH:/bin:/usr/bin
vi /etc/hosts
127.0.0.1 localhost.db.com localhost
# Public
192.168.1.101 db1.db.com db1
192.168.1.102 db2.db.com db2
# Private
192.168.0.101 db1-priv.db.com db1-priv
192.168.0.102 db2-priv.db.com db2-priv
# Virtual
192.168.1.111 db1-vip.db.com db1-vip
192.168.1.112 db2-vip.db.com db2-vip
# SCAN
192.168.1.121 db-scan.db.com db-scan
192.168.1.122 db-scan.db.com db-scan
192.168.1.123 db-scan.db.com db-scan
echo "grid" | passwd --stdin grid
echo "%oinstall ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers
hostnamectl set-hostname db2.db.com
chmod 777 /u01
chmod 777 /u01/oracle/
chown oracle:oinstall /u01/app/oracle/product/19/db_2
chown oracle:oinstall /u01/app/oracle1
chown oracle:oinstall /u01/app/grid_base
chown oracle:oinstall /u01/app/19c/grid
chown oracle:oinstall /u01/app/grid_base
chown oracle:oinstall /u01/app/19c/grid
chmod 777 /u01/app/grid_base
chmod 777 /u01/app/19c/grid
chown oracle:oinstall /u01/app/oracle/product/19/db_2
chmod 777 /u01/app/oracle/product/19/db_2
----------
chown oracle:oinstall /u01/app/oracle/product/19.3/db_home1
chown oracle:oinstall /u01/app/oracle1
chown grid:oinstall /u01/app/grid_base
chown grid:oinstall /u01/app/19c/grid
/u01/app/oracle/product/19/db_2;
cd /u01/oracle; ls -l
chown oracle:asmadmin /dev/oracleasm/disks/ASMDATA1
chmod 660 /dev/oracleasm/disks/ASMDATA1
ALTER DISKGROUP ASMDATA MOUNT FORCE;
ALTER DISKGROUP ASMDATA SET ATTRIBUTE 'compatible.asm' = '21.0';
NODE 1-
## ORACLE - /home/oracle/.bashrc
PATH=$PATH:$HOME/bin ; export PATH
TMP=/tmp ; export TMP
TMPDIR=$TMP ; export TMPDIR
ORACLE_SID=ora19c1 ; export ORACLE_SID
ORACLE_UNQNAME=ora19c ; export ORACLE_UNQNAME
ORACLE_GLOBAL_NAME=ora19c1.db.com ; export ORACLE_GLOBAL_NAME
ORACLE_BASE=/u01/app/oracle ; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/19/db_2; export ORACLE_HOME
ORACLE_GRID=/u01/app/19c/grid_home ; export ORACLE_GRID
ORACLE_HOSTNAME=db1.db.com; export ORACLE_HOSTNAME
PATH=/usr/sbin:$PATH ; export PATH
PATH=$ORACLE_HOME/bin:$PATH ; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib ; export CLASSPATH
CLASSPATH=$ORACLE_HOME/JRE:$CLASSPATH ; export CLASSPATH
#export DISPLAY=192.168.1.6:7.0
---- vi grid_env
export ORACLE_HOME=/u01/app/19c/grid_home
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=+ASM1
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H
----
cat db_env
ORACLE_SID=ora19c1; export ORACLE_SID
ORACLE_HOME=/u01/app/oracle/product/19/db_2; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPAT H
You have new mail in /var/spool/mail/oracle
---
NODE 2
## ORACLE - /home/oracle/.bashrc
PATH=$PATH:$HOME/bin ; export PATH
TMP=/tmp ; export TMP
TMPDIR=$TMP ; export TMPDIR
ORACLE_SID=ora19c2 ; export ORACLE_SID
ORACLE_UNQNAME=ora19c ; export ORACLE_UNQNAME
ORACLE_GLOBAL_NAME=ora19c2.db.com ; export ORACLE_GLOBAL_NAME
ORACLE_BASE=/u01/app/oracle1 ; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/19.3/db_home1; export ORACLE_HOME
ORACLE_GRID=/u01/app/19c/grid ; export ORACLE_GRID
ORACLE_HOSTNAME=db2.db.com ; export ORACLE_HOSTNAME
PATH=/usr/sbin:$PATH ; export PATH
PATH=$ORACLE_HOME/bin:$PATH ; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib ; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib ; export CLASSPATH
CLASSPATH=$ORACLE_HOME/JRE:$CLASSPATH ; export CLASSPATH
#export DISPLAY=192.168.1.6:7.0
------
## GRID - /home/grid/.bashrc
PATH=$PATH:$HOME/bin ; export PATH
TMP=/tmp ; export TMP
TMPDIR=$TMP ; export TMPDIR
ORACLE_SID=+ASM1 ; export ORACLE_SID
ORACLE_UNQNAME=+ASM1 ; export ORACLE_UNQNAME
ORACLE_GLOBAL_NAME=ora19c1.db.com ; export ORACLE_GLOBAL_NAME
ORACLE_BASE=/u01/app/oracle1; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/19/db_2; export ORACLE_HOME
ORACLE_GRID=/u01/app/19c/grid_home ; export ORACLE_GRID
ORACLE_HOSTNAME=db1.db.com ; export ORACLE_HOSTNAME
PATH=/usr/sbin:$PATH ; export PATH
PATH=$ORACLE_HOME/bin:$PATH ; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib ; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib ; export CLASSPATH
CLASSPATH=$ORACLE_HOME/JRE:$CLASSPATH ; export CLASSPATH
#export DISPLAY=192.168.1.6:7.0
NODE 2
## GRID - /home/grid/.bashrc
PATH=$PATH:$HOME/bin ; export PATH
TMP=/tmp ; export TMP
TMPDIR=$TMP ; export TMPDIR
ORACLE_SID=+ASM2 ; export ORACLE_SID
ORACLE_UNQNAME=+ASM2 ; export ORACLE_UNQNAME
ORACLE_GLOBAL_NAME=ora19c2.db.com ; export ORACLE_GLOBAL_NAME
ORACLE_BASE=/u01/app/oracle1; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/19/db_2; export ORACLE_HOME
ORACLE_GRID=u01/app/19c/grid_home ; export ORACLE_GRID
ORACLE_HOSTNAME=db2.db.com ; export ORACLE_HOSTNAME
PATH=/usr/sbin:$PATH ; export PATH
PATH=$ORACLE_HOME/bin:$PATH ; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib ; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib ; export CLASSPATH
CLASSPATH=$ORACLE_HOME/JRE:$CLASSPATH ; export CLASSPATH
#export DISPLAY=192.168.1.6:7.0
-----------
Add entry in /etc/oratab
+ASM1: /u01/app/19c/grid:Y
sudo chown -R oracle:oinstall /u01/app/oraInventory
sudo chown -R oracle:oinstall /u01/app/19c/grid
sudo chmod -R 775 /u01/app/oraInventory
sudo chmod -R 775 /u01/app/19c/grid
ssh db1 'sudo chown -R oracle:oinstall /u01/app/oraInventory'
ssh db1 'sudo chown -R oracle:oinstall /u01/app/19c/grid'
sudo chmod -R 775 /u01/app/oraInventory
sudo chmod -R 775 /u01/app/19c/grid
export ORACLE_HOME=/u01/app/19c/grid
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=+ASM
Check if /dev/sdb1, /dev/sdc1, /dev/sdd1 had ownership as oracle user. If not then issue below commands as root user on both nodes
# chown oracle:oinstall /dev/nvme0n2p1
# chown oracle:oinstall /dev/nvme0n2p2
# chown oracle:oinstall /dev/nvme0n2p3
2. Also give full permissions on above disks
# chmod 777 /dev/nvme0n2p1
# chmod 777 /dev/nvme0n2p2
# chmod 777 /dev/nvme0n2p3
chown oracle:oinstall /dev/oracleasm/disks/DATA
chmod 660 /dev/oracleasm/disks/DATA
oracleasm scandisks
oracleasm listdisks
===================================
CRSCTL Commands :- Cluster control
Cluster Related Commands
# ps -ef|grep d.bin
Shows HA resource status (hard to read)
[grid@dr ~]$ ps -ef|grep d.bin
root 4704 1 0 05:48 ? 00:00:16 /u01/app/19.0.0.0/grid/bin/ohasd.bin reboot
root 5111 1 0 05:48 ? 00:00:06 /u01/app/19.0.0.0/grid/bin/orarootagent.bin
grid 5299 1 0 05:48 ? 00:00:10 /u01/app/19.0.0.0/grid/bin/oraagent.bin
grid 5335 1 0 05:48 ? 00:00:08 /u01/app/19.0.0.0/grid/bin/evmd.bin
grid 5336 1 0 05:48 ? 00:00:03 /u01/app/19.0.0.0/grid/bin/mdnsd.bin
grid 5425 1 0 05:48 ? 00:00:04 /u01/app/19.0.0.0/grid/bin/gpnpd.bin
grid 5521 5335 0 05:48 ? 00:00:03 /u01/app/19.0.0.0/grid/bin/evmlogger.bin -o /u01/app/19.0.0.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/19.0.0.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
grid 5596 1 0 05:48 ? 00:00:08 /u01/app/19.0.0.0/grid/bin/gipcd.bin
root 6392 1 0 05:48 ? 00:00:04 /u01/app/19.0.0.0/grid/bin/cssdmonitor
root 6394 1 0 05:48 ? 00:00:32 /u01/app/19.0.0.0/grid/bin/osysmond.bin
root 6431 1 0 05:48 ? 00:00:04 /u01/app/19.0.0.0/grid/bin/cssdagent
grid 6453 1 0 05:48 ? 00:00:13 /u01/app/19.0.0.0/grid/bin/ocssd.bin -S 1
root 7131 1 0 05:49 ? 00:00:08 /u01/app/19.0.0.0/grid/bin/octssd.bin reboot
root 8325 1 0 05:49 ? 00:00:16 /u01/app/19.0.0.0/grid/bin/crsd.bin reboot
root 8648 1 0 05:49 ? 00:00:16 /u01/app/19.0.0.0/grid/bin/orarootagent.bin
grid 8658 1 0 05:49 ? 00:00:18 /u01/app/19.0.0.0/grid/bin/oraagent.bin
grid 8963 1 0 05:49 ? 00:00:04 /u01/app/19.0.0.0/grid/bin/scriptagent.bin
grid 8981 1 0 05:49 ? 00:00:00 /u01/app/19.0.0.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid 9004 1 0 05:49 ? 00:00:00 /u01/app/19.0.0.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
root 10494 1 0 05:49 ? 00:00:06 /u01/app/19.0.0.0/grid/bin/ologgerd -M
oracle 12613 1 0 05:49 ? 00:00:15 /u01/app/19.0.0.0/grid/bin/oraagent.bin
grid 76726 53614 0 06:55 pts/0 00:00:00 grep --color=auto d.bin
crsstat Output of crs_stat t formatted nicely
# crsctl check crs CSS,CRS,EVM appears healthy
[grid@dr ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
crsctl stop crs Stop crs and all other services
crsctl disable crs Prevents CRS from starting on reboot
crsctl enable crs Enables CRS start on reboot
crs_stop all Stops all registered resources
crs_start all Starts all registered resources
crsctl stop cluster -all Stops the cluster in all nodes
crsctl stop cluster -n db2 stop specific node cluster
crsctl start cluster -all Starts the cluster in all nodes
# olsnodes -n check all nodes in cluster
[grid@dr ~]$ olsnodes -n
dr 1
# olsnodes -l Check current node
[grid@dr ~]$ olsnodes -l
dr
crsctl check cluster -all Check all nodes crs status
[grid@dr ~]$ crsctl check cluster -all
**************************************************************
dr:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
*************************************************************
crsctl check cluster -n dr Check specific node crs status
[grid@dr ~]$ crsctl check cluster -n dr
**************************************************************
dr:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
*************************************************************
# crsctl status resource -t -init All cluster resource init status
[grid@dr ~]$ crsctl status resource -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE dr STABLE
ora.crf
1 ONLINE ONLINE dr STABLE
ora.crsd
1 ONLINE ONLINE dr STABLE
ora.cssd
1 ONLINE ONLINE dr STABLE
ora.cssdmonitor
1 ONLINE ONLINE dr STABLE
ora.ctssd
1 ONLINE ONLINE dr OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE dr STABLE
ora.evmd
1 ONLINE ONLINE dr STABLE
ora.gipcd
1 ONLINE ONLINE dr STABLE
ora.gpnpd
1 ONLINE ONLINE dr STABLE
ora.mdnsd
1 ONLINE ONLINE dr STABLE
ora.storage
1 ONLINE ONLINE dr STABLE
--------------------------------------------------------------------------------
# crsctl status server -f Nodes server status
[grid@dr ~]$ crsctl status server -f
NAME=dr
MEMORY_SIZE=31804
CPU_COUNT=4
CPU_CLOCK_RATE=2445
CPU_HYPERTHREADING=1
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
SERVER_LABEL=
PHYSICAL_HOSTNAME=
CSS_CRITICAL=no
CSS_CRITICAL_TOTAL=0
RESOURCE_TOTAL=2
SITE_NAME=dbsysbjwc2nvq
STATE=ONLINE
ACTIVE_POOLS=Generic ora.DB1127_DR
STATE_DETAILS=
ACTIVE_CSS_ROLE=hub
# crsctl status res ora.dr.vip VIP status checking
[grid@dr ~]$ crsctl status res ora.dr.vip
NAME=ora.dr.vip
TYPE=ora.cluster_vip_net1.type
TARGET=ONLINE
STATE=ONLINE on dr
# crsctl status res ora.db1.vip -p VIP status checking
[grid@dr ~]$ crsctl status res ora.dr.vip -p
NAME=ora.dr.vip
TYPE=ora.cluster_vip_net1.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
ALLOW_RESTART=default
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=120
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DELETE_TIMEOUT=60
DESCRIPTION=Oracle VIP resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_USR_ORA_STATIC_VIP=
GEN_USR_ORA_VIP=
HOSTING_MEMBERS=dr
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NETWORK_COUNT=
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
OPC_NAT_ADDRESS=
PLACEMENT=favored
RELOCATE_BY_DEPENDENCY=1
RELOCATE_KIND=offline
RESOURCE_GROUP=
RESTART_ATTEMPTS=0
RESTART_DELAY=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.net1.network) weak(global:ora.gns) pullup(ora.net1.network)
START_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="network" type="Res">ora.net1.network</Arg></xml>
START_DEPENDENCIES_TEMPLATE=<xml>hard(<Res>{network}</Res>) pullup(<Res>{network}</Res>) weak(global:ora.gns)</xml>
START_TIMEOUT=120
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.net1.network)
STOP_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="network" type="Res">ora.net1.network</Arg></xml>
STOP_DEPENDENCIES_TEMPLATE=<xml>hard(<Res>intermediate:{network}</Res>)</xml>
STOP_TIMEOUT=0
TARGET_DEFAULT=default
TYPE_VERSION=2.2
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=ORA_VIP_GARP_RETRIES=30" "ORA_VIP_GARP_AFTER=10
USR_ORA_VIP=dr-vip.subnet11131831.vcn11131831.oraclevcn.com
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0
-->OCR COMMANDS
# ocrcheck OCR status
[grid@dr ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 901284
Used space (kbytes) : 84288
Available space (kbytes) : 816996
ID : 950261719
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
./ocrdump /tmp/ocrdump.bkp Every resource of cluster stored ocrdump
#[root@dr bin]# ./ocrconfig -local -showbackup - olr backup
dr 2024/11/27 18:07:39 /u01/app/grid/crsdata/dr/olr/autobackup_20241127_180739.olr 161823088
dr 2025/01/18 07:12:39 /u01/app/grid/crsdata/dr/olr/backup_20250118_071239.olr 161823088
dr 2024/11/27 17:53:51 /u01/app/grid/crsdata/dr/olr/backup_20241127_175351.olr 161823088
# [root@dr bin]# ./ocrconfig -showbackup - ocr backup
PROT-24: Auto backups for the Oracle Cluster Registry are not available
dr 2025/01/18 07:12:04 +DATA:/dbSysbjwc2nvq/OCRBACKUP/backup_20250118_071204.ocr.274.1190704325 161823088
./ocrconfig -local -restore (filename) Restore the olr backup
[root@dr bin]# ./ocrconfig -local -restore /u01/app/grid/crsdata/dr/olr/autobackup_20241127_180739.olr
./ocrconfig -restore (filename) - restore ocr backup
ocrconfig -showbackup auto OCR autobackup check (4backup every hr max 3bkp kept) (every 1day and every week 1 max 2)
# [grid@dr ~]$ ocrconfig -showbackup auto
PROT-24: Auto backups for the Oracle Cluster Registry are not available
./ocrconfig -manualbackup Manual backup of OCR
[root@dr bin]# ./ocrconfig -manualbackup
dr 2025/01/18 07:12:04 +DATA:/dbSysbjwc2nvq/OCRBACKUP/backup_20250118_071204.ocr.274.1190704325 161823088
./ocrconfig -backuploc <newloc> To change the backup location (Default loc- +ASMOCR:/db-cluster/OCRBACKUP/day.ocr.259.1178436777)
./ocrconfig -restore (filename) Restore the ocr backup
# cat /etc/oracle/olr.loc Default loc of OLR file
[grid@dr ~]$ cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/grid/crsdata/dr/olr/dr_19.olr
crs_home=/u01/app/19.0.0.0/grid
# ocrcheck -config -local To check the OLR loc
[grid@dr ~]$ ocrcheck -config -local
Oracle Local Registry configuration is :
Device/File Name : /u01/app/grid/crsdata/dr/olr/dr_19.olr
ocrdump -local /tmp/olr.bkp To see the contents of OLR
/ocrdump -local -xml /tmp/olr.bkp To contents of OLR in xml
[root@dr bin]# ./ocrconfig -manualbackup ocr manualbackup
dr 2025/01/18 08:35:00 +DATA:/dbSysbjwc2nvq/OCRBACKUP/backup_20250118_083500.ocr.275.1190709301 161823088
dr 2025/01/18 07:12:04 +DATA:/dbSysbjwc2nvq/OCRBACKUP/backup_20250118_071204.ocr.274.1190704325 161823088
# ./ocrconfig -local -manualbackup To take manualbackup of OLR file
[root@dr bin]# ./ocrconfig -local -manualbackup
dr 2025/01/18 07:12:39 /u01/app/grid/crsdata/dr/olr/backup_20250118_071239.olr 161823088
dr 2024/11/27 17:53:51 /u01/app/grid/crsdata/dr/olr/backup_20241127_175351.olr 161823088
# ./crsctl query css votedisk To check votedisk status (why css bz votedisk is adminstrated by css process)
[root@dr bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE d38fd84990a34f1dbf14ac48ccae5919 (/dev/DATADISK3) [DATA]
Located 1 voting disk(s).
/u01/app/19c/grid/log/db1 crs alert log loc
systemctl status ohasd
[root@dr bin]# systemctl status ohasd
● ohasd.service - LSB: Start and Stop Oracle High Availability Service
Loaded: loaded (/etc/rc.d/init.d/ohasd; generated)
Active: active (exited) since Sat 2025-01-18 05:48:45 UTC; 1h 25min ago
Docs: man:systemd-sysv-generator(8)
Process: 4159 ExecStart=/etc/rc.d/init.d/ohasd start (code=exited, status=0/SUCCESS)
Tasks: 657 (limit: 202995)
Memory: 5.0G
CGroup: /system.slice/ohasd.service
Jan 18 05:48:44 dr systemd[1]: Starting LSB: Start and Stop Oracle High Availability Service...
Jan 18 05:48:44 dr ohasd[4220]: Starting ohasd:
Jan 18 05:48:45 dr ohasd[4323]: /etc/init.d/init.ohasd: Found cgroupfs on /sys/fs/cgroup/systemd, legacy hierarchy
Jan 18 05:48:45 dr clsecho[4323]: /etc/init.d/init.ohasd: Found cgroupfs on /sys/fs/cgroup/systemd, legacy hierarchy
Jan 18 05:48:45 dr ohasd[4225]: Running cgroupv1 with CONFIG_RT_GROUP_SCHED enabled.
Jan 18 05:48:45 dr root[4539]: Oracle HA daemon is enabled for autostart.
Jan 18 05:48:45 dr ohasd[4553]: CRS-4123: Oracle High Availability Services has been started.
Jan 18 05:48:45 dr systemd[1]: Started LSB: Start and Stop Oracle High Availability Service.
systemctl enable ohasd
Wrapper scrpits to start ohasd
/etc - initab
/etc/init.d - ohasd
/etc/rc.d/init.d - ohasd
SRVCTL Commands :- server control
Database Related Commands
srvctl start instance -d <db_name> -i <inst_name> Starts an instance
srvctl start database -d <ora19c> -o nomount|mount|open Starts all database choose
srvctl stop database -d <db_name> Stops all instances, closes database
srvctl stop instance -d <db_name> -i <inst_name> Stops an instance
srvctl start service -d <db_name> -s <service_name> Starts a service
srvctl stop service -d <db_name> -s <service_name> Stops a service
# srvctl status service -d DB1127_DR Checks status of a service
[grid@dr ~]$ srvctl status service -d DB1127_DR
Service drdb_drpdb.paas.oracle.com is running on instance(s) DRDB
srvctl config database -d DB1127_DR All config information will show
# [grid@dr ~]$ srvctl config database -d DB1127_DR
Database unique name: DB1127_DR
Database name: DRDB
Oracle home: /u01/app/oracle/product/19.0.0.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DB1127_DR/PARAMETERFILE/spfile.261.1186164689
Password file:
Domain: subnet11131831.vcn11131831.oraclevcn.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECO,DATA
Mount point paths: /opt/oracle/dcs/commonstore
Services: DRDB_DRPDB.paas.oracle.com
Type: SINGLE
OSDBA group: dba
OSOPER group: dbaoper
Database instance: DRDB
Configured nodes: dr
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
# srvctl status instance -d DB1127_DR -i dr Checks an individual instance
Instance DB1127_DR is running on node dr
# srvctl status database -d DB1127_DR Checks status of all instances
[grid@dr ~]$ srvctl status database -d DB1127_DR
Instance DRDB is running on node dr
# srvctl status scan Status of scan listener
[grid@dr ~]$ srvctl status scan
SCAN VIP scan1 is disabled
SCAN VIP scan1 is not running
# srvctl config scan Configuration of scan listener
[grid@dr ~]$ srvctl config scan
SCAN name: dr-scan, Network: 1
Subnet IPv4: 10.0.0.0/255.255.255.0/ens3, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 10.0.0.239
SCAN VIP is disabled.
srvctl status asm Status of ASM instance
[grid@dr ~]$ srvctl status asm
ASM is running on dr
srvctl start nodeapps -n <node_name> Starts gsd, vip, listener, and ons
srvctl stop nodeapps -n <node_name> Stops gsd, vip and listener
srvctl start asm start ASM
srvctl stop asm -f Stop ASM
ASMCMD- start and stop with asmcmd
asmcmd-startup
asmcmd-shutdown --abort
asmcmd - showclustermode
# srvctl config scan_listener All scan listener status show (configured by clusterware)
[grid@dr ~]$ srvctl config scan_listener
SCAN Listeners for network 1:
Registration invited nodes:
Registration invited subnets:
Endpoints: TCP:1521
SCAN Listener LISTENER_SCAN1 exists
SCAN Listener is disabled.
srvctl status asm -n db1 -a ASM status node wise
# srvctl config network Network config
[grid@dr ~]$ srvctl config network
Network 1 exists
Subnet IPv4: 10.0.0.0/255.255.255.0/ens3, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
# srvctl config vip -n dr VIP checking on nodes
[grid@dr ~]$ srvctl config vip -n dr
VIP exists: network number 1, hosting node dr
VIP Name: dr-vip.subnet11131831.vcn11131831.oraclevcn.com
VIP IPv4 Address: 10.0.0.239
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
# srvctl config listener Listener configure
[grid@dr ~]$ srvctl config listener
Name: LISTENER
Type: Database Listener
Network: 1, Owner: grid.
Home: <CRS home>
End points: TCP:1521
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
Network IP and VIP addition in cluster :
srvctl add network -k 2 -S 192.168.1.20/255.255.255.0
srvctl add vip -n db1 -k 2 -A db1-vip2/255.255.255.0
srvctl add vip -n db2 -k 2 -A db2-vip2/255.255.255.0
crsctl stat res db1-vip2 -p
appvipcfg create
-network=1
-ip 192.168.1.66
-vipname=MyVIP
-user=root
==================
Oracle Clusterware Logs
All the clusterware logs reside under
$GRID_HOME/log/devdb
Clusterware alert log
$GRID_HOME/log/devdb/alertracn1.log
Other process log files
$GRID_HOME/log/<hostname>/crsd --> crs log
$GRID_HOME/log/<hostname>/ctssd --> cts log
$GRID_HOME/log/<hostname>/evmd --> evmd log
$GRID_HOME/log/<hostname>/ohasd --> ohasd log
$GRID_HOME/log/<hostname>/diskmon --> diskmon log
CRSD Oraagent logs
$GRID_HOME/log/<hostname>/agent/crsd/oragent_<crs admin username>
CRSD Ora root agent logs
$GRID_HOME/log/<hostname>/agent/crsd/orarootagent_<crs admin username>
=======================================
mkdir -p /u01/app/19c/grid
mkdir -p /d01/app/oracle/product/19c/db_1
chown -R oracle:oinstall /grid
chown -R oracle:oinstall /u01/software
chmod -R 775 /grid/
chmod -R 775 /u01/app/oracle1
ROOT SCRIPTS:-
/u01/app/oraInventory/orainstRoot.sh
/u01/software/root.sh
Download GRID and DB SW
unzip on the server
export CV_ASSUME_DISTID=OEL7.6
cd /u01/app/19c/grid/
./gridSetup.sh
# yum clean all
sudo chown -R oracle:oinstall /u01
sudo chmod -R 775 /u01
sudo chown -R oracle:oinstall /u01/app/19c/grid
=======================
PARTITION CRATION
[root@test ~]# fdisk /dev/nvme0n2
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x7955b83e.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-104857599, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-104857599, default 104857599): +10G
Created a new partition 1 of type 'Linux' and of size 10 GiB.
Command (m for help): n
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2):
First sector (20973568-104857599, default 20973568):
Last sector, +sectors or +size{K,M,G,T,P} (20973568-104857599, default 104857599): +20G
Created a new partition 2 of type 'Linux' and of size 20 GiB.
Command (m for help): n
Partition type
p primary (2 primary, 0 extended, 2 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3):
First sector (62916608-104857599, default 62916608):
Last sector, +sectors or +size{K,M,G,T,P} (62916608-104857599, default 104857599):
Created a new partition 3 of type 'Linux' and of size 20 GiB.
Command (m for help): p
Disk /dev/nvme0n2: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7955b83e
Device Boot Start End Sectors Size Id Type
/dev/nvme0n2p1 2048 20973567 20971520 10G 83 Linux
/dev/nvme0n2p2 20973568 62916607 41943040 20G 83 Linux
/dev/nvme0n2p3 62916608 104857599 41940992 20G 83 Linux
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
[root@test ~]# fdisk -l
Disk /dev/nvme0n1: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd3d3fe05
Device Boot Start End Sectors Size Id Type
/dev/nvme0n1p1 * 2048 616447 614400 300M 83 Linux
/dev/nvme0n1p2 616448 8892415 8275968 4G 82 Linux swap / Solaris
/dev/nvme0n1p3 8892416 209715199 200822784 95.8G 83 Linux
Disk /dev/nvme0n2: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7955b83e
Device Boot Start End Sectors Size Id Type
/dev/nvme0n2p1 2048 20973567 20971520 10G 83 Linux
/dev/nvme0n2p2 20973568 62916607 41943040 20G 83 Linux
/dev/nvme0n2p3 62916608 104857599 41940992 20G 83 Linux
/dev/oracleasm/disks
sudo rpm -ivh oracleasm-support-2.1.11-2.el7.x86_64.rpm
ls /usr/sbin/oracleasm
sudo /usr/sbin/oracleasm configure -i
sudo /usr/sbin/oracleasm init
[root@RAC1 ~]# oracleasm createdisk CRS1 /dev/nvme0n2p1
Writing disk header: done
Instantiating disk: done
[root@RAC1 ~]# oracleasm createdisk DATA1 /dev/nvme0n2p2
Writing disk header: done
Instantiating disk: done
[root@RAC1 ~]# oracleasm createdisk FRA1 /dev/nvme0n2p3
Writing disk header: done
Instantiating disk: done
[root@RAC1 ~]#
oracleasm scandisks
oracleasm listdisks
sudo yum install chrony
sudo systemctl enable chronyd
sudo systemctl start chronyd
======================
ssh oracle@RAC1
source /u01/app/19c/grid/crs/install/crsenv
./cluvfyrac.sh stage -pre crsinst -n oraracn1,oraracn2 -fixup -verbose
$GRID_HOME/bin/cluvfyrac.sh stage -pre crsinst -n oraracn1,oraracn2 -fixup -verbose
Display Config and Start Grid Installation
su –
export DISPLAY=:0.0
xhost +
su – oracle
[oracle@dbwr1 ~]$ DISPLAY=dbwr1.localdomain:0.0; export DISPLAY
Install and Configure Oracle 19c Grid Infrastructure for a Cluster
[oracle@dbwr1 ~]$ cd /u01/app/19c/grid/
[oracle@dbwr1 grid]$ ./gridSetup.sh
Launching Oracle Grid Infrastructure Setup Wizard…
================================================
ssh connectivity using grid:
ssh-keygen
then copy cd .ssh/id_rsa.pub public key to home/oracle/.ssh authorized_keys all nodes
/u01/app/19c/grid/oui/prov/resources/scripts/sshUserSetup.sh -user oracle -hosts "db1.db.com db2.db.com" -advanced -noPromptPassphrase
================================================
[root@oraracn2 ~]# systemctl enable oracleasm.service
[root@oraracn2 ~]# systemctl start oracleasm.service
[root@oraracn2 ~]# systemctl enable chronyd
[root@oraracn2 ~]# systemctl restart chronyd
[root@oraracn2 ~]# chronyc -a 'burst 4/4'
200 OK
[root@oraracn2 ~]# chronyc -a makestep
200 OK
[root@oraracn2 ~]#
=========================
Cluster deconfig :-
[oracle@oraracn1 ~]$ /u01/app/19c/grid/crs/install/rootcrs.sh -force -deconfig -paramfile "/tmp/deinstall2024-08-19_12-10-14AM/response/deinstall_OraGI21Home1.rsp" -lastnode
for deconfig -
-> grid_home./crs/install/rootcrs.sh -deconfig -force
<GI_HOME>/crs/install/crsgenconfig_params
sudo /u01/app/19c/grid/crs/install/rootcrs.sh -force -deconfig -paramfile "/correct/path/to/deinstall_OraGI21Home1.rsp" -lastnode
NOde removal -
sudo /u01/app/19c/grid/crs/install/rootcrs.sh -deconfig -force -lastnode
sudo /u01/app/19c/grid/crs/install/rootcrs.sh -deconfig -force
sudo /u01/app/19c/grid/crs/install/rootcrs.sh
delete ocr.loc from /etc/oracle for node deletion
===============================
node addition steps in rac
===========================
cluvfy stage -pre nodeadd -n db1
.oraenv
ORACLE_SID =+ASM2
olsnodes -n -s
cd $ORACLE_HOME/addnode
pwd ; ls -lastnode
./addnode.sh -silent "CLUSTER_NEW_NODES=db1" "CLUSTER_NEW_VIRTUAL_HOSTNAMES=db1-vip" -ignorePrereq
As a root on new node
/u01/app/oraInventory/orainsRoot.sh
/u01/app/19c/grid/root.sh
Update Inventory on node 3
/u01/app/oracle/product/19/db_2/oui/bin/runInstaller -updateNodeList -noCluster ORACLE_HOME=/u01/app/oracle/product/19/db_2 CLUSTER_NODES=db1,db2,db3 CRS=false "INVENTORY_LOCATION=/u01/app/oraInventory" LOCAL_NODE=3
OR We can use ./gridSetup.sh
node addition then select node add it
==================================
Adding Oracle_home to new instance
cd $ORACLE_HOME/addnode
./addnode.sh -silent "CLUSTER_NEW_NODES=db1" "CLUSTER_NEW_VIRTUAL_HOSTNAMES=db1-vip" -ignorePrereq
/u01/app/oracle/product/19/db_2/root.sh
OR GUI
./dbca >>> select add instance
srvctl add instance -d DEVDB -i DEVDB3 -n db3
Deleting node from cluster
===========================
1)./dbca >> Instance management -- delete instance
srvctl status database -d DEVDB
srvctl config database -d DEVDB
or
srvctl remove instance -d DEVDB -i DEVDB3 -n db3
2)Disable and stop listner
srvctl disable listener -l LISTENER_DB3
srvctl stop listener -l LISTENER_DB3
3)Update Inventory on delete node
$grid_home/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/19/db_2 "CLUSTER_NODES={db3}" -local
4)Deinstall oracle_home from node3
$ORACLE_HOME/deinstall -local
$ORACLE_HOME/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/19/db_2 "CLUSTER_NODES={db1,db2}"
5)olsnodes -s -t
6)Disable the clusterware daemon process from node 3
root user - /u01/app/19c/grid/crs/install/root.sh -deconfig -force -verbose
7)delete clusterware configuration from other nodes
root user - /u01/app/19c/grid/bin/crsctl delete node -n db3
8)check crs status:
crsctl check crs -all
crsctl stat res -t
9)Update Inventory
cd $grid_home/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={db3} CRS=TRUE -local
10)Deinstall GRID_HOME
cd /u01/app/19c/grid/deinstall -./deinstall -local
cd $grid_home/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/19c/grid "CLUSTER_NODES={db1,db2}" CRS=TRUE
11)verify the integrity of the cluster after nodes removed -
cluvfy stage -post db1 -n db3
==============================
SQL> select instance_name,status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
ora19c1 OPEN
ora19c2 OPEN
select instance_name,status from v$instance;
INSTANCE_NAME STATUS
---------------- ------------
ora19c2 OPEN
Alter system switch logfile; -->Affects only the current instance (Everything will be run in background)
Alter system archive log current; --> To force a global log switch (Stops untill the archiving has been done then it will return to sql prompt
wait for complete work then return the sql prompt )
-rwxrwxr-x 1 oracle oinstall 11904 Sep 10 01:51 cvuqdisk-1.0.10-1.rpm
[root@db1 rpm]# rpm -i cvuqdisk-1.0.10-1.rpm
package cvuqdisk-1.0.10-1.x86_64 is already installed
[root@db1 rpm]#
===========================
Linux performance scrpits -
#!/bin/bash
#
white="\033[1;23m"
cyan="\033[1;36m"
reset="\033[0m"
echo " "
echo -e "${white}------------------------------------------------------------------System Information--------------------------------------------------------------${reset}"
echo -e "${cyan}Hostname:${reset}\t\t"hostname -f
echo -e "${cyan}IP Address:${reset}\t\t"hostname -I
echo -e "${cyan}Uptime:${reset}\t\t"uptime | awk '{print $3,$4}' | sed 's/,//'
echo -e "${cyan}Machine Type:${reset}\t\t"vserver=$(lscpu | grep Hypervisor | wc -l); if [ $vserver -gt 0 ]; then echo "VM"; else echo "Physical"; fi
echo -e "${cyan}Product Name:${reset}\t\t"cat /sys/class/dmi/id/product_name
echo -e "${cyan}Operating System:${reset}\t\t"cat /etc/redhat-release
echo -e "${cyan}Kernel:${reset}\t\t"uname -r
echo -e "${cyan}Architecture:${reset}\t\t"arch
echo -e "${cyan}Processor Name:${reset}\t\t"awk -F':' '/^model name/ {print $2}' /proc/cpuinfo | uniq | sed -e 's/^[ \t]*//'
echo -e "${cyan}CPU Cores:${reset}\t\t"grep -c ^processor /proc/cpuinfo
echo -e "${cyan}Total RAM:${reset}\t\t"free -h | awk '/^Mem:/{print $2}'
echo -e "${cyan}Disk Usage:${reset}\t\t"
df -h | grep '^/dev/'
echo -e "${cyan}Load Average:${reset}\t\t"uptime | awk -F'load average:' '{ print $2 }'
echo " "
echo -e "${white}------------------------------------------------------------------Resource Usage--------------------------------------------------------------${reset}"
echo -e "${cyan}CPU Usage:${reset}\t\t"awk '{u=$2+$4; t=$2+$4+$5; if (NR==1){u1=u; t1=t;} else print ($2+$4-u1) * 100 / (t-t1) "%";}' <(grep 'cpu ' /proc/stat) < (sleep 1;grep 'cpu ' /proc/stat)
echo -e "${cyan}Memory Usage:${reset}\t\t"free | awk '/Mem/{printf("%.2f%"), $3/$2*100}'
echo -e "${cyan}Swap Usage:${reset}\t\t"free | awk '/Swap/{printf("%.2f%"), $3/$2*100}'
echo -e "${cyan}Top 5 Processes by CPU Usage:${reset}"
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -6
echo -e "${cyan}Top 5 Processes by Memory Usage:${reset}"
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head -6
echo -e "${cyan}Disk I/O:${reset}\t\t"
iostat -x | head -10
echo -e "${cyan}Network Usage:${reset}\t\t"
netstat -i | grep -vE '^Kernel|Iface|lo'
echo -e "${cyan}Active Network Connections:${reset}"
netstat -ant | grep 'ESTABLISHED'
echo -e "${cyan}Swap Activity:${reset}"
vmstat 1 5
echo -e "${cyan}Zombie Processes:${reset}\t\t"
ps aux | awk '{ if ($8 == "Z") print $0; }'
echo -e "${cyan}System Temperature:${reset}\t\t"
sensors | grep 'Core'
echo " "
===================
CREATE PFILE = '/u01/app/init.ora' FROM SPFILE;
ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 500M SCOPE=SPFILE;
ALTER SYSTEM SET SGA_TARGET = 1563M SCOPE=SPFILE;
ALTER SYSTEM SET SGA_MAX_SIZE = 2.5G SCOPE=SPFILE;
--------------------
CREATE CONTROLFILE REUSE DATABASE "ORCL" RESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 1
MAXLOGHISTORY 292
DATAFILE
'+asmdata/ora19c/datafile/SYSTEM.257.1180244259',
'+asmdata/ora19c/datafile/SYSAUX.258.1180244387',
'+asmdata/ora19c/datafile/UNDOTBS1.259.1180244453',
'+asmdata/ora19c/datafile/UNDOTBS2.269.1180250085',
'+asmdata/ora19c/datafile/USERS.260.1180244461'
LOGFILE
GROUP 1 ('+asmdata/ora19c/onlinelog/group_1.262.1180244835') SIZE 50M,
GROUP 2 ('+asmdata/ora19c/onlinelog/group_2.263.1180244835') SIZE 50M,
GROUP 3 ('+asmdata/ora19c/onlinelog/group_3.270.1180250547') SIZE 50M,
GROUP 4 ('+asmdata/ora19c/onlinelog/group_4.271.1180250549') SIZE 50M
CHARACTER SET AL32UTF8;
Restore:
========
==========================
ACFS
crsctl stat res ora.ACFS_DG.dg -t
asmcmd volcreate -G ACFS_DG -s 10G acfs_test
asmcmd volinfo -G ACFS_DG acfs_test
crsctl stat res -t | grep -i "advm"
crsctl stat res ora.ACFS_DG.ACFS_TEST.advm -t
mkfs -t acfs /dev/asm/acfs_test-278
(mkfs -t ext3/ext4/xfs /dev/sdb)
mkdir -p /acfs_test
chown oracle:oinstall /acfs_test
/sbin/acfsutil registry -a /dev/asm/acfs_test-278 /acfs_test -u oracle
crsctl stat res -t | grep -i "acfsdg"
crsctl stat res ora.acfs_dg.acfs_test.acfs -t
srvctl status filesystem -d /dev/asm/acfs_test-278
srvctl start filesystem -d /dev/asm/acfs_test-278
srvctl stop filesystem -d /dev/asm/acfs_test-278
asmcmd volinfo -G ACFS_DG acfs_test
srvctl config filesystem
srvctl stop filesystem -d /dev/asm/acfs_test-278
/sbin/acfsutil rmfs /dev/asm/acfs_test-278
asmcmd voldisable -G ACFS_DG acfs_test
asmcmd voldelete -G ACFS_DG acfs_test
****************
Comments
Post a Comment