Leveraging Oracle ASM Cluster File System for Private Cloud Storage
Kai Yu Oracle Solutions Engineering Dell Inc
About Author Kai Yu, Senior Principal Architect, Dell Database Engineering 21 years Oracle DBA/Apps DBAS and Solutions Engineering Specializing in Oracle RAC, Virtualization and Database Infrastructure Oracle ACE Director, authored 120 + presentations and 30 + whitepapers 2011 OAUG Innovator of Year 2012 Oracle Excellence Award: Technologist of the Year: Cloud Architect by Oracle Magazine IOUG Spotlight for March 2016 Co-founder and VP of IOUG Cloud Computing SIG My Oracle Blog: http://kyuoracleblog.wordpress.com/ Co-author Apress Book “Expert Oracle RAC 12c”
Agenda Oracle ACFS for Cloud Applications Configuration of Oracle ACFS Leverage ACFS Snapshot Feature Oracle ACFS or ASM for Database
Oracle ACFS for Cloud Applications Oracle ASM Cluster File System (ACFS): •
Initially introduced in Oracle 11gR2 to store non-Oracle Database files.
•
ACFS mounted in OS and accessible through OS commands
•
A cluster file system shared and accessible by single/cluster nodes with unified namespace.
•
ACFS files stored in the Oracle ASM diskgroup with benefits from ASM •
IO load balancing through data stripping among the ASM disks
•
Data reliability through mirroring within the ASM diskgroup
Oracle ACFS for Cloud Applications Oracle ASM Cluster File System (ACFS): •
ACFS doesn’t support Oracle Grid Infrastructure, which should be stored in local disks
•
All databases stored on Oracle ACFS can leverage Advanced Database services: snapshots, tagging, security, encrypting, and auditing
•
Starting with Oracle 12cR1, it supports general purpose files as well as database files across multiple OS and planforms, except for data files and redologs for Oracle Restart
Oracle ACFS for cloud applications Oracle Automatic Storage Management Stack • Oracle ASM provide the basic functionality of Automatic Storage Management • Oracle ASM files for Oracle Database supports diskgroups for database files • Oracle ASM Dynamic Volume Manager (ADVM) : provides the logical volume management and standard device interface to files systems such as ACFS, OCFS2 and regular ext4 file systems •
Oracle RAC database can be stored in Oracle ASM files as well as in Oracle ACFS.
Oracle ACFS for Cloud Applications
Oracle ACFS Integration with Oracle ASM
• Oracle ACFS is always configured with Oracle ASM storage. • Traditional Storage Devices presented to Oracle ASM • ASM diskgroup is built with the storage devices, presented to Oracle ADVM • Oracle ADVM volumes are created with the ADVM in the ASM diskgroup and presented to OS as an OS logical block device. • Oracle ACFS file system is bound to the Oracle ADVM volumes through the file system creation ASM Cluster File System(ACFS)
/acfs1 RAC DB1
RAC DB2
/acfs2
/acfs3
Cluser ware
ASM Diskgroup DG
ADVM
ASM Volume
ASM Volume ASM Volume
ASM Volume ASM Volume
ASM Diskgroup FG Oracle ASM
ASM Disks
ASM Disks
Oracle ACFS for Cloud Applications Oracle ACFS and Oracle Clusterware •
Oracle ACFS is based on Oracle ASM, and is tightly coupled with Oracle Clusterware , directly participate Clusterware cluster membership state transaction and resource based high availability (HA) management
•
Oracle ACFS is a cluster enabled file system that can shared by multiple cluster nodes.
•
Oracle 12c RAC database files can stored in Oracle ASM diskgroups as well as Oracle ACFS.
Oracle ACFS for Cloud Applications Oracle ACFS Resource-based Management • Oracle clusterware resource support ACFS for enabling and disabling volumes, loading drivers and mounting and unmounting file systems: • When Oracle ASM instance starts, It starts the Oracle ACFS driver,
Oracle ADVM driver, Oracle Kernel Service drivers (OKS) • Oracle ACFS file system can be automatically mounted or dismounted based on Oracle clusterware resource action.
Oracle ACFS for Cloud Applications Limitations of the standard ASM • Each node has an ASM instance which costs CPU/memory • Local ASM instance failure will cause DB instance failure
.
Oracle ACFS for Cloud Applications
Oracle ACFS with Flex ASM for higher flexibility, scalability and availability • Flex ASM is an option on Oracle 12c : enabled or disabled • A small # of ASM instance. (default 3, specified by admin)
• DB instances connects to any ASM instance (local/remote)
Oracle ACFS for Cloud Applications Two kinds of Oracle ASM configurations: • Local ASM clients connect to local ASM instance • Flex ASM clients connect to a remote ASM instance
ASM network • Added for Flex ASM for communication between ASM clients and ASM • On Oracle 12cR1, share the network with cluster private interconnect
Oracle ACFS for Cloud Applications
• Oracle ACFS with Flex ASM for higher flexibility, scalability and availability Database instance can connect to remote ASM instance if local ASM instance fails • ADVM and Oracle ACFS file system leverage the Flex ASM for a better HA • ADVM and ACFS utilize an Oracle ASM proxy to connect remote ASM instance
Configuring Oracle ACFS Prerequisites of creating Oracle ACFS
• Oracle Grid Infrastructure is installed/configured with Oracle ASM instance running: [grid@inmem1 bin]$ /u01/app/12.1.0/grid/bin/crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [grid@inmem1 bin]$ ps -ef | grep pmon | grep ASM grid 31863 1 0 2015 ? 00:10:36 asm_pmon_+ASM1 • Required modules are loaded: These modules should be automatically loaded in Oracle 12c RAC. [root@inmem1 ~]# lsmod | grep oracle oracleacfs 3307969 3 oracleadvm 506254 8 oracleoks 505749 2 oracleacfs,oracleadvm
Configuring Oracle ACFS Outline of the ACFS creation: • ACFS Creation Process: • Create ASM diskgroup • Create ASM volumes
• Create ACFS file system •
Tools for ACFS creation: • ASMCA GUI tool
• Oracle Enterprise Manager • ASMCMD command line tool • SQLPLUS
Configuring Oracle ACFS Example of creating a ACFS with ASMCMD command line tool: • Create ASM volume “acfs_vm1” in ASM diskgroup “ACFSDG1” ASMCMD> volcreate -G ACFSDG1 -s 1g acf_vm
• Create ACFS file system on ASM volume mkfs -t acfs /dev/asm/acf_vm1-105 mkfs.acfs: version = 12.1.0.1.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acf_vm1-105 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete.
• Mount the acfs file to an OS mount /acfs_01
#mkdir /acfs_vol1 #/sbin/acfsutil registry -a /dev/asm/acf_vm1-105 /acfs_vol1 acfsutil registry: mount point /acfs_vol1 successfully added to Oracle Registry Now you can see the new ACFS file system mounted to /acfs_vol1: # df -k | grep 'Filesystem \|asm' Filesystem 1K-blocks Used Available Use% Mounted on /dev/asm/acf_vm1-105 1048576 119112 929464 12% /u01/acfs/asm_vol1
Configuring Oracle ACFS Example of creating a ACFS with asmca GUI tool • Create ASM volume “SSD_ACFS_VM” in ASM diskgroup “SSD_ACFS”
Configuring Oracle ACFS Example of creating a ACFS with asmca GUI tool • Create a ACFS file system on the ASM volume
[
Configuring Oracle ACFS
# /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh ACFS file system /SSD_ACFS is mounted on nodes inmem1,inmem2
root@inmem1 ~]# df -k | grep 'Filesystem \|asm'
Filesystem 1K-blocks Used Available Use% Mounted on /dev/asm/ssd_acfs_vm-329 5077204992 3978716232 1098488760 79% /SSD_ACFS
Configuring Oracle ACFS Creating ACFS for Oracle Database
• Use asmca GUI tool to create ACFS for Oracle Database as well as Oracle RAC Home
Configuring Oracle ACFS Creating a ACFS for Oracle Database • Create Oracle RAC database on ACFS with dbca: • ACFS supports Oracle 2c as well as pre-12c DBs
Configuring Oracle ACFS Creating a ACFS for Oracle Database
• After dbca, you can see the 12c RAC database configuration:
Leverage ACFS Snapshot feature
Oracle ACFS snapshot
• An online, read-only or read-write , point in time copy of the Oracle ACFS file system. • Immediately available, always online, can be resized. • Snapshot is stored in the file system . ACFS/snaps directory , can be resized. • ACFS director not visible
Snapshot is in /SAN_ACFS/.ACFS/snaps/ro_dbsnap
Leverage ACFS Snapshot Feature Oracle ACFS Snapshot
• Read-write snapshot can be read and written without impacting of ACFS system as a copy of the production system for testing without impacting the original production data. • Read-write snapshot : the diskgroup compatiability attribute for ADVM set to 11.2.0.3.0 or higher. • Take a snapshot for another snapshot, or convert a snapshot between read-only and read-write • set diskgroup compatiability attribute for ADVM set to 12.1 or higher) • The commands to create a snapshot : #acfsutil snap create snapshot_name ACFS_name • Check the snapshot status: #acfsutil snap info ACFS_name
Leverage ACFS Snapshot Feature Perform database backup with Oracle ACFS snapshot
• Oracle ACFS in Oracle GI 12.1 supports all Oracle database files (11.2.0.4 or higher) • Use AFCS snapshot to do a quick backup or clone the database. Verify the database files stored in the ACFS file system: oracle@inmem1 ACFSDB]$ pwd /SAN_ACFS/db_acfs/ACFSDB [oracle@inmem1 ACFSDB]$ ls archivelog controlfile datafile onlinelog
Leverage ACFS Snapshot Feature Perform database backup with Oracle ACFS snapshot • Create an ACFS snapshot: #acfsutil snap create ro_dbsnap /SAN_ACFS #acfsutil snap create: Snapshot operation is complete.
•
Check the snapshot status # acfsutil snap info /SAN_ACFS snapshot name: ro_dbsnap snapshot location: /SAN_ACFS/.ACFS/snaps/ro_dbsnap RO snapshot or RW snapshot: RO parent name: /SAN_ACFS snapshot creation time: Wed Nan 30 14:48:43 2016
Leverage ACFS Snapshot Feature Perform database backup with Oracle ACFS snapshot • What the snapshot looks like: the database files stored in the ACFS file system:
database files in the Snapshot:
• Restore the database from the snapshot The snapshot can be used to restore the database in case we lose the original database files in the ACFS system. You just need to copy the files from the snapshot to the original directories with the cp command
Leverage ACFS Snapshot feature • Restore the database from Oracle ACFS snapshot cp -r /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/archivelog/* /SAN_ACFS/db_acfs/ACFSDB/archivelog cp -r /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/archivelog/* /SAN_ACFS/db_acfs/ACFSDB/archivelog cp /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/onlinelog/* /SAN_ACFS/db_acfs/ACFSDB/onlinelog cp /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/datafile/* /SAN_ACFS/db_acfs/ACFSDB/datafile •
Startup the database after restore.
Oracle ACFS Other Usage High Availability NFS (HANFS) • Provide un-interrupted service of NFS exports • Exports based on Highly Availability Virtual IPs(HAVIP) • Oracle Clusterware agents ensure HAVIP and NFS NFS export always online • HAFS is configured using ACFS cluster • If a cluster node fails, the HAVIP and NFS services are are automatically taken over by a surviving
Oracle Database Appliance (ODA) • Use ACFS to store database files. • Database Appliance Virtualized Platforms use ACFS for shared storage access. Oracle E-Business Suite uses OCFS2 instead of ACFS as the shared storage in the multi-node configuration.
Oracle ACFS or ASM for Database Standalone database • Use a regular OS file system such as Linux and Windows file system • If use ASM, need to install Oracle Grid infrastructure for single node: Oracle Restart and ASM • Oracle ACFS doesn’t support database files and redo files in Oracle Restart configuration Oracle Database on Oracle Database Appliance • Use ACFS to store the database files. Oracle RAC Database • Need Oracle GI 12.1 to use Oracle ACFS for database • Pre-11.2.0.4 databases are not supported by ACFS, must use ASM • Pre-12.1.0.2 Oracle databases on Windows are not supported by ACFS • Use ACFS for advanced features such as snapshot – for test and development • ASM and ACFS offer the similar performance • ACFS is built on ASM and has all the benefits of ASM management • Oracle Databases Appliance uses ACFS for all databases
References 1.
Oracle Automatic Storage Management Administrator's Guide, 12cR1, Part III Oracle Automatic Storage Management Cluster File System
2. Oracle Whitepaper : Oracle ACFS, July 2014 3. Integrating ACFS Snapshots With RMAN, by Anju Garg, All things Oracle from Redgate 4. Benefits of Oracle ACFS , Oracle Whitepaper 2Janurary 2015 5. Export Oracle RAC 12c, Apress , by Riyaj Shamsudeen, Syed Jaffar Hussain, Tariq Farooq, Kai Yu , Chapter 5, Storage and ASM Practices 6. OCFS2 Certified with EBS 12.2 Shared File System Configurations By Steven Chan - EBS-Oracle on Jun 06, 2016 7. Design and Architecture of Dell Acceleration Appliances for Database (DAAD): A Practical Approach with High Availability Guaranteed, Kai Y, Yuxiang G, Peng Z., Meikang Q (2015) In proceedings of 2015 IEEE 17th IHPCC-ICESS-CSS, New York, USA 8. Implementing Oracle Database12c's Heat Map and Automatic Data Optimization for Database Storage Performance and Cost, Kai.Yu, Yuxiang Gao, IOUG Collaborate 15
Thank You and QA
Contact me at
[email protected] or visit my Oracle Blog at http://kyuoracleblog.wordpress.com/
32
