Add a new node, Run root.sh failed with " Configuration of ASM ... failed "

problem description

Adding 3rd nodes on a 2 node cluster, execute root.sh on 3rd nodes.

$ sudo /u01/app/crs/11.2.0.3/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/crs/11.2.0.3

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/crs/11.2.0.3/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3-lnx'
CRS-2676: Start of 'ora.mdnsd' on 'rac3-lnx' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3-lnx'
CRS-2676: Start of 'ora.gpnpd' on 'rac3-lnx' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3-lnx'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac3-lnx'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac3-lnx' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac3-lnx' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3-lnx'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3-lnx'
CRS-2676: Start of 'ora.diskmon' on 'rac3-lnx' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac3-lnx' succeeded

Symptoms


found following error in ocssd.log


2013-05-20 18:28:17.654: [   SKGFD][1098938688]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2013-05-20 18:28:17.654: [   SKGFD][1098938688]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2013-05-20 18:28:17.654: [   SKGFD][1098938688]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2013-05-20 18:28:17.654: [   SKGFD][1098938688]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
)
2013-05-20 18:28:17.654: [    CSSD][1098938688]clssnmvDiskVerify: Successful discovery of 0 disks
2013-05-20 18:28:17.655: [    CSSD][1098938688]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2013-05-20 18:28:17.655: [    CSSD][1098938688]clssnmvFindInitialConfigs: No voting files found
2013-05-20 18:28:17.655: [    CSSD][1098938688](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds


Cause
By default, Oracle Database ASM instance parameter 'ASM_DISKSTRING' is unset and therefore uses a default value of '*'. This means that the ASM instance scans both/dev/raw/and/dev/oracleasm/disks/directories when searching for oracle ASM disks.

When using a combination of oracleasm disks (/dev/oracleasm/disks/*) with raw devices (/dev/raw/*), in particular where some raw devices are not owned by or accessible to the oracle user) as well as ASM instance parameter 'ASM_DISKSTRING' with an unspecified value, error ORA-15186 is likely to occur.

Solution
1. either edit the file /etc/sysconfig/oracleasm-_dev_oracleasm and change the lines:
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
to
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
or alternatively run the following command (as user root)
/usr/sbin/oracleasm configure -i -e -u user -g group -o "dm" -x "sd"

2. stop & restart ASMLIB as user root using:
/usr/sbin/oracleasm exit
/usr/sbin/oracleasm init

3. restart CRS or reboot node

The above steps need to be executed on all nodes