[Title will be auto-generated] by Lucio Teixeira

Agenda • Administering ASM DiskGroups Using Exadata Cell. • Administering Grid Disks with ASM.

– 2–

Administering ASM DiskGroups with Exadata CellGrid Disks. Introduction Automatic Storage Management (ASM) ASM is the cluster volume manager and file system used to manage Exadata Cell resources. ASM provides enhanced storage management by: Striping database files evenly across all available storage cells and disks for optimal performance. Using mirroring and failuregroups to avoid any single point of failure. Enabling dynamic add and drop capability for nonintrusive cell and disk allocation, deallocation, and reallocation. Enabling multiple databases to share storage cells and disks.

– 3–

Administering ASM DiskGroups with Exadata CellGrid Disks. ASM DiskGroup is the primary storage abstraction within ASM and is composed of one or more disks. Exadata Cell exposes grid disks to ASM as individual disks available for membership in ASM disk groups. Whenever possible, grid disk names should correspond closely with ASM diskgroup names to assist in problem diagnosis between ASM and Exadata Cell. To take advantage of Exadata Cell features, such as predicate processing offload, the diskgroups must contain only Exadata Cell grid disks, and the tables must be fully inside these diskgroups.

– 4–

Administering ASM DiskGroups with Exadata CellGrid Disks. ASM FailGroup is a subset of disks in an ASM diskgroup that can fail together because they share the same hardware and guarantees data redundancy. For Exadata Cell, all griddisks (ASM disk group members and candidates) can effectively fail together if the storage cell fails. Because of this scenario, all ASM disks (griddisks) sourced from a given storage cell should be assigned to a single failuregroup representing the cell. Failure groups for Exadata Cell grid disks are set by default so that the disks on a single cell are in the same failure group, making correct failure group configuration simple for Exadata Cell.

– 5–

Administering ASM DiskGroups with Exadata CellGrid Disks. For example, if all grid disks from two storage cells (A and B) are added to a single ASM diskgroup with normal redundancy, then all grid disks on storage cell A are designated as one failure group, and all grid disks on storage cell B are designated as another failure group. This enables Exadata Cell and ASM to tolerate the failure of either storage cell.

– 6–

Administering ASM DiskGroups with Exadata CellGrid Disks. 1. Administering ASM DiskGroups Using Exadata Cell. a) Creating an Exadata Cell ASM DiskGroup:

Using the CREATE DISKGROUP SQL command. Determine the GridDisks. Choose similar names for the ASM diskgroup and its griddisks. The default ASM disk name is the grid disk name (recommended). The default failure group name is the cell name (recommended).

Note: If a cell is renamed, and a disk from that cell is added to an existing diskgroup that has disks from that cell, then ASM will add the new disk to a failure group using the new cell name. To ensure all the disks from the cell are in one failure group, add the disk to the diskgroup and specify the original failure group name.

– 7–

Administering ASM DiskGroups with Exadata CellGrid Disks. 1. Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance: SQL> SELECT PATH, header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%'; PATH

STATUS

------------------------------------------------------------ -----------o/192.168.22.129:5042/data_CD_disk01_cell1

CANDIDATE

o/192.168.22.129:5042/data_CD_disk07_cell1

CANDIDATE

o/192.168.22.129:5042/data_CD_disk09_cell1

CANDIDATE

. .

o/192.168.22.130:5042/data_CD_disk11_cell2

CANDIDATE

o/192.168.22.130:5042/data_CD_disk08_cell2

CANDIDATE

o/192.168.22.130:5042/data_CD_disk10_cell2

CANDIDATE

. .

24 rows selected.

– 8–

Administering ASM DiskGroups with Exadata CellGrid Disks. 2. Use the SQL CREATE DISKGROUP command to create the ASM diskgroup. The next diskgroup attributes must be set: SQL> CREATE DISKGROUP data NORMAL REDUNDANCY -- These grid disks are on cell1 FAILGROUP cell01 DISK 'o/*/data_CD_disk*_cell1‘ -- These grid disks are on cell2 FAILGROUP cell02 DISK 'o/*/data_CD_disk*_cell2‘ -- These disk group attributes must be set for cell access -- Note that this diskgroup is set for cell only ATTRIBUTE 'compatible.rdbms'='11.1.0.7.0', 'compatible.asm'='11.1.0.7.0', 'au_size'='4M', 'cell.smart_scan_capable'='TRUE';

– 9–

Administering ASM DiskGroups with Exadata CellGrid Disks. 3. View the ASM disk groups and associated attributes with the next SQL query on V$ASM dynamic views: SQL> SELECT dg.name AS diskgroup, SUBSTR(a.name,1,24) AS name, SUBSTR(a.value,1,24) AS value FROM V$ASM_DISKGROUP dg, V$ASM_ATTRIBUTE a WHERE dg.group_number = a.group_number; DISKGROUP

NAME

VALUE

------------------------------ ------------------------ -----------------DATA

disk_repair_time

3.6h

DATA

au_size

4194304

DATA

cell.smart_scan_capable

TRUE

DATA

compatible.rdbms

11.1.0.7.0

DATA

compatible.asm

11.1.0.7.0

– 10 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 4.

Create a tablespace in the disk group to take advantage of Exadata Cell features, such as offload processing. This tablespace should contain the tables that you want to query with offload processing: SQL> CREATE TABLESPACE SALES DATAFILE '+DATA';

Verify that the tablespace is in an Exadata Cell disk group. The PREDICATE_EVALUATION column of the DBA_TABLESPACES view indicates whether predicates are evaluated by host (HOST) or by storage (STORAGE): SQL> SELECT tablespace_name, predicate_evaluation FROM dba_tablespaces WHERE tablespace_name = 'SALES'; TABLESPACE_NAME

PREDICATE_EVALUATION

------------------------------ --------------------SALES

STORAGE

– 11 –

Administering ASM DiskGroups with Exadata CellGrid Disks. b) Adding a Disk to an ASM DiskGroup: 1. Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance. If the header status is set to CANDIDATE that means the disk is a candidate for a diskgroup: SQL> SELECT PATH, header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%' and STATUS like 'CANDIDATE' ; PATH

STATUS

------------------------------------------------------------ -----------o/192.168.22.129:5042/data_CD_disk13_cell1

CANDIDATE

o/192.168.22.130:5042/data_CD_disk13_cell2

CANDIDATE

rows selected.

Note: Do not add Exadata Cell grid disks to a non-Exadata Cell ASM diskgroup unless you are planning to migrate the diskgroup to an Exadata Cell diskgroup. © 2008 Oracle Corporation – Proprietary and Confidential

– 12 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 2. Use the SQL ALTER DISKGROUP command with the ADD DISK clause to add the disk to the ASM diskgroup: SQL> ALTER DISKGROUP DATA ADD DISK 'o/*/data_CD_disk13_cell1', 'o/*/data_CD_disk13_cell2';

When the disk is added, ASM will rebalance the diskgroup. You can query the V$ASM_OPERATION view for the status of the rebalance operation. SQL> SELECT * FROM V$ASM_OPERATION;

– 13 –

Administering ASM DiskGroups with Exadata CellGrid Disks. c) Mounting or Dismounting an ASM DiskGroup: To mount or dismount a diskgroup, use the SQL ALTER DISKGROUP statement with the MOUNT or DISMOUNT option: SQL> ALTER DISKGROUP DATA DISMOUNT;

SQL> ALTER DISKGROUP DATA MOUNT;

Note: You can use the FORCE option of the ALTER DISKGROUP statement MOUNT clause to mount diskgroups if their components are unavailable, which results in a loss of full redundancy.. SQL> ALTER DISKGROUP DATA MOUNT FORCE;

– 14 –

Administering ASM DiskGroups with Exadata CellGrid Disks. d) Changing a Disk to Offline or Online: 1. Determine which disk you want offline or online in the ASM disk group by querying the V$ASM_DISK and V$ASM_DISKGROUP views on the ASM instance: SQL> select name, path, failgroup from v$asm_disk order by 3;

2. Use the SQL ALTER DISKGROUP statement with the OFFLINE DISK or ONLINE DISK clause to change the disk in the ASM disk group to offline or online: SQL> ALTER DISKGROUP DATA OFFLINE DISK DATA_CD_DISK16_CELL1;

SQL> ALTER DISKGROUP DATA ONLINE DISK DATA_CD_DISK16_CELL1;

– 15 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Note: As soon as the disk is online, ASM will rebalance the disk group. You can query the V$ASM_OPERATION view for the status of the rebalance operation. SQL> SELECT * FROM V$ASM_OPERATION;

– 16 –

Administering ASM DiskGroups with Exadata CellGrid Disks. e) Dropping a Disk from an ASM DiskGroup: 1. Determine which disks you want to drop from the ASM diskgroup by querying the V$ASM_DISK and V$ASM_DISKGROUP views on the ASM instance: SQL> SELECT d.name as asmdisk, dg.name as diskgroup FROM V$ASM_DISK d, V$ASM_DISKGROUP dg WHERE dg.name LIKE 'DATA%' AND d.group_number = dg.group_number; ASMDISK

DISKGROUP

------------------------------ -----------------------------DATA_CD_DISK12_CELL2

DATA

DATA_CD_DISK16_CELL1

DATA

DATA_CD_DISK04_CELL2

DATA

DATA_CD_DISK07_CELL2

DATA

– 17 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Note: If you are removing an Exadata Cell grid disk, ensure that you identify the grid disks that are mapped to each ASM diskgroup. 2. Use the SQL ALTER DISKGROUP statement with the DROP DISK clause to drop the disks from the ASM diskgroup: SQL> ALTER DISKGROUP DATA DROP DISK DATA_CD_DISK16_CELL1;

Note: When the disk is dropped from the ASM diskgroup, ASM will rebalance the diskgroup. You can query the V$ASM_OPERATION view for the status of the rebalance operation. SQL> SELECT * FROM V$ASM_OPERATION;

– 18 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Note: After an Exadata Cell grid disk is dropped from the ASM diskgroup, you can drop the grid disk from a cell.

– 19 –

Administering ASM DiskGroups with Exadata CellGrid Disks. f)

Dropping an ASM DiskGroup:

1. Determine the disk group that you want to drop by querying the V$ASM_DISKGROUP view on the ASM instance: SQL> select name, state from v$asm_diskgroup; NAME

STATE

------------------------------ ----------DATA

MOUNTED

2. Use the SQL DROP DISKGROUP statement to drop the ASM disk group. SQL> DROP DISKGROUP DATA;

– 20 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Note: If you cannot mount a diskgroup but must drop it, you can use the FORCE option of the DROP DISKGROUP statement: SQL> DROP DISKGROUP DATA FORCE;

– 21 –

Administering ASM DiskGroups with Exadata CellGrid Disks. g) Migrating an Existing ASM DiskGroup to an Exadata Cell DiskGroup. If ASM normal or high redundancy is used for the legacy storage, then you can migrate the entire diskgroup with no downtime by adding the Oracle Exadata Storage to the diskgroup, removing the legacy storage from the diskgroup, and setting the diskgroup attribute cell.smart_scan_capable to true. true To optimize the operation, remove and add storage as one command in the ASM instance. The advantages and disadvantages for this strategy are as follows: Zero downtime. Requires ASM to be upgraded to release 11.1.0.7. Necessity to use the rebalance POWER setting to control the effect on applications.

– 22 –

Administering ASM DiskGroups with Exadata CellGrid Disks. To migrate an existing ASM diskgroup to an Exadata Cell-only diskgroup, perform the following steps. 1. Determine the diskgroup that you want to migrate by querying the V$ASM_DISKGROUP view on the ASM instance. The ASM diskgroup must be mounted to complete this procedure. SQL> select name, state from v$asm_diskgroup; NAME

STATE

------------------------------ ----------DATA

MOUNTED

– 23 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 2. Determine which Exadata Cell grid disks are available by querying the V$ASM_DISK view on the ASM instance. SQL> SELECT PATH, header_status STATUS FROM V$ASM_DISK WHERE path LIKE 'o/%' and STATUS like 'CANDIDATE' ; PATH

STATUS

------------------------------------------------------------ -----------o/192.168.22.129:5042/data_CD_disk01_cell3

CANDIDATE

o/192.168.22.130:5042/data_CD_disk01_cell4

CANDIDATE

o/192.168.22.129:5042/data_CD_disk02_cell3

CANDIDATE

o/192.168.22.130:5042/data_CD_disk02_cell4

CANDIDATE

o/192.168.22.129:5042/data_CD_disk03_cell3

CANDIDATE

o/192.168.22.130:5042/data_CD_disk03_cell4

CANDIDATE

rows selected.

– 24 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 3. Use the SQL ALTER DISKGROUP statement to set the following for the existing diskgroup: 'cell.smart_scan_capable' = 'FALSE' 'compatible.asm' = '11.1.0.7.0' 'compatible.rdbms' = '11.1.0.7.0' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'cell.smart_scan_capable' = 'FALSE' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.asm' 'compatible.asm' = '11.1.0.7.0' SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.rdbms' 'compatible.rdbms' = '11.1.0.7.0'

Note: When you alter the diskgroup to add Exadata Cell grid disks, the cell.smart_scan_capable attribute must be set to FALSE. FALSE

– 25 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 4. Use the SQL ALTER DISKGROUP statement with the ADD DISK clause to add all the Exadata Cell grid disks to the ASM diskgroup: SQL> ALTER DISKGROUP DATA ADD DISK 'o/*/data_CD_disk13_cell8', 'o/*/data_CD_disk13_cell8' 'o/*/data_CD_disk13_cell9'; 'o/*/data_CD_disk13_cell9' SQL> ALTER DISKGROUP DATA ADD DISK 'o/192.168.22.129:5042/data_CD_disk01_cell8', 'o/192.168.22.130:5042/data_CD_disk01_cell9', 'o/192.168.22.129:5042/data_CD_disk02_cell8', 'o/192.168.22.130:5042/data_CD_disk02_cell9', 'o/192.168.22.129:5042/data_CD_disk03_cell8', 'o/192.168.22.130:5042/data_CD_disk03_cell9'; 'o/192.168.22.130:5042/data_CD_disk03_cell9'

Note: Allow time for ASM to rebalance the diskgroup.

– 26 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 5. Use the SQL ALTER DISKGROUP statement with the DROP DISK clause to drop all the non-Exadata Cell disks from the ASM diskgroup: SQL> ALTER DISKGROUP DATA DROP DISK DATA1,DATA2,DATA3,DATA4,DATA5,DATA6 REBALANCE POWER 11;

Note: Allow time for ASM to rebalance the diskgroup.

6. After the disk drop operation is completed, use the SQL ALTER DISKGROUP statement to set the cell.smart_scan_capable disk group attribute to be TRUE: TRUE SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'cell.smart_scan_capable' = ‘TRUE TRUE‘

– 27 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 2. Administering Grid Disks with ASM. a) Naming Conventions for Exadata Cell Grid Disks: The name of the grid disk should contain the cell disk name to make it easy to determine which grid disks belong to a cell disk. To also help determine which grid disks belong to an ASM diskgroup, a subset of the grid disk name should match all or part of the name of the ASM disk group to which the grid disk will belong. For example, if a grid disk is created on the cell disk CD_disk01_cell1 and that grid disk will belong to an ASM diskgroup named DATA, then the grid disk name should contain data_CD_disk01_cell1 and data. The grid disk name should be data_CD_disk01_cell1.

– 28 –

Administering ASM DiskGroups with Exadata CellGrid Disks. CellCLI> list celldisk CD_disk01_cell1

CellCLI> list griddisk CellCLI> list celldisk data_CD_disk01_cell1 CD_disk01_cell1

SQL>

normal

active normal

select path from v$asm_disk order by 1;

PATH ---------------------------------------------------------------o/192.168.22.129:5042/data_CD_disk01_cell1

– 29 –

Administering ASM DiskGroups with Exadata CellGrid Disks. When you use the ALL PREFIX option with CREATE GRIDDISK, a unique grid disk name is automatically generated that includes the prefix and cell name. If you do not use the default generated name when creating grid disks, you must ensure that the grid disk name is unique across all cells. If the disk name is not unique, it might not be possible to add the grid disk to an ASM diskgroup. CellCLI> create griddisk all prefix=data CellCLI> list griddisk data_CD_disk01_cell1

active

data_CD_disk02_cell1

active

data_CD_disk03_cell1

active

data_CD_disk10_cell1

active

data_CD_disk11_cell1

active

data_CD_disk12_cell1

active

. .

– 30 –

Administering ASM DiskGroups with Exadata CellGrid Disks. b) Changing a Grid Disk that Belongs to an ASM DiskGroup: When you change a grid disk that belongs to an ASM diskgroup, you must be aware of how that might affect the ASM diskgroup to which the grid disk belongs. •

Changing a Grid Disk Name: Name To change attributes of a grid disk, use the CellCLI ALTER GRIDDISK command. First, use the DESCRIBE GRIDDISK command to determine which grid disk attributes can be modified. CellCLI> describe griddisk name

modifiable

availableTo

modifiable

cellDisk comment

modifiable

creationTime errorCount id offset size

modifiable

status

– 31 –

Administering ASM DiskGroups with Exadata CellGrid Disks. CellCLI> ALTER GRIDDISK data11 name='data_CD_3_cell04_11‘

Caution: Before changing the name of a grid disk that belongs to an ASM diskgroup, ensure that the corresponding ASM disk is offline.

– 32 –

Administering ASM DiskGroups with Exadata CellGrid Disks. 2.

Making a Grid Disk Active or Inactive: Inactive To activate or deactivate a grid disk, use the CellCLI ALTER GRIDDISK command. A grid disk should be made inactive before dropping the grid disk to ensure that the grid disk is not in use. CellCLI> ALTER GRIDDISK data0_CD_3_cell04_31 ACTIVE CellCLI> ALTER GRIDDISK data0_CD_3_cell04_31 INACTIVE

Note 1: When a grid disk in use by a database client is made inactive, then ASM takes the corresponding ASM disk offline when I/Os to the disk fail. To make the disk usable again, first make the grid disk active in the cell and then bring the corresponding ASM disk back online in ASM. Note 2: There is usually no need to change the grid disk state from active to inactive, or from inactive to active. ASM will discover all active disks. However, if a grid disk should not be available to ASM, then the grid disk should be changed to inactive. © 2008 Oracle Corporation – Proprietary and Confidential

– 33 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Note 3: If a diskgroup is created with external redundancy, and a grid disk in that group is made inactive, then the diskgroup cannot be mounted.

– 34 –

Administering ASM DiskGroups with Exadata CellGrid Disks. Dropping a Grid Disk: Disk To drop a grid disk, use the CellCLI DROP GRIDDISK command. Make the grid disk inactive before dropping the grid disk to ensure that the grid disk is not in use. The FORCE option can be used to force the grid disk that is in use to be dropped.

Caution 1: Before dropping a grid disk that belongs to an ASM diskgroup, ensure that the corresponding ASM disk was dropped from the diskgroup. Caution 2: Before dropping a grid disk using the FORCE option, ensure that the ASM disk was dropped from the diskgroup.

Make the grid disk inactive. CellCLI> ALTER GRIDDISK data_CD_16_cell01 INACTIVE

CellCLI> ALTER GRIDDISK ALL PREFIX=data INACTIVE

– 35 –

Administering ASM DiskGroups with Exadata CellGrid Disks. ii.

Drop the corresponding ASM disk from the diskgroup. SQL> ALTER DISKGROUP DATA DROP DISK data_CD_16_cell01; SQL> ALTER DISKGROUP DATA DROP DISK data_CD_01_cell01, . .

data_CD_15_cell01, data_CD_16_cell01;

iii.

Drop the grid disk. CellCLI> DROP GRIDDISK data_CD_16_cell01 CellCLI> DROP GRIDDISK ALL PREFIX=data CellCLI> DROP GRIDDISK data_CD_16_cell01 FORCE

– 36 –

Administering ASM DiskGroups with Exadata CellGrid Disks. c) Determining Which ASM DiskGroup Contains a Grid Disk: If a grid disk name matches the ASM disk name and the name contains the ASM diskgroup name, then you can determine the ASM diskgroup to which the grid disk belongs. You can also use SQL*Plus on the ASM instance to find the ASM diskgroup that matches part of the specific grid disk name. This can help you to determine which ASM diskgroup contains a specific grid disk. SQL> SELECT d.name as asmdisk, dg.name as diskgroup FROM V$ASM_DISK d, V$ASM_DISKGROUP dg WHERE dg.name LIKE 'DATA%' AND d.group_number = dg.group_number; ASMDISK

DISKGROUP

------------------------------ ------------------------DATA_CD_DISK12_CELL2

DATA

DATA_CD_DISK04_CELL2

DATA

DATA_CD_DISK07_CELL2

DATA

. .

– 37 –

Administering ASM DiskGroups with Exadata CellGrid Disks. d) Determining Which Grid Disks Belong to an ASM Disk Group: If a grid disk name contains the ASM diskgroup name, then you can use SQL*Plus on the ASM instance to list the ASM diskgroup names and use the CellCLI utility to search for specific grid disk names. SQL> SELECT name FROM V$ASM_DISKGROUP; NAME -----------------------------FRA DATA CellCLI> LIST GRIDDISK WHERE name LIKE '.*data.*' data_CD_disk01_cell1

active

data_CD_disk02_cell1

active

data_CD_disk03_cell1

active

data_CD_disk04_cell1

active

. .

– 38 –

– 39 –