NFS mounts with Shared Application Tier

Here is an interesting thing that I noticed. Oracle does not say it anywhere in the metalink notes, but if you plan to use NFS mounts to share application tier between different nodes, you should mount NFS with NOLOCK option. E.g. in Red Hat ES Linux 4.0
mount -t nfs :/mnt/inst /mnt/dir -o nolock
or update fstab such as:
:/mnt/inst /mnt/dir nfs auto,nolock 0 0

Read below for complete story ……….

We were in the process of upgrading our 11.0.3 application to 11.5.10.2. All the tiers were running in Hp-UX. We had successfully completed 4 iteration of the upgrade and also moved our application tier-forms/web server from Hp-UX to Linux (Red hat 4.0) (using note Note:238276.1 , as post upgrade step)

Forms/web server on Linux (RED HAT 4.0) had been successfully running for past 2 months now. At this point we wanted to have Cisco load balancer in front of two Linux forms/web server and use shared application tier for these two Linux servers.

As part of this step, I was trying to enable “shared application tier” on first Linux node-1, NFS mount application tier from node1 to node2 and add node2 as secondary node to shared application tier (Note:233428.1). After successful completion of adding node to shared application, enable http layer hardware load balancing (Note:217368.1).

However I was having problem bringing up apache on node2 after enabling “shared application tier” and adding the secondary node. The error_log of apache shows the following:

[Tue Sep 12 23:46:39 2006] [debug] opm_tbl.c(230): OPM:Into opm_tbl_create_tbl
[Tue Sep 12 23:46:39 2006] [debug] opm_sys.c(128): OPM:Into opm_sys_init
[Tue Sep 12 23:46:39 2006] [debug] opm_sys.c(267): OPM:Into opm_sys_init_mtx
[Tue Sep 12 23:46:39 2006] [error] OPM:Fail to allocate memory for mutex in shmpool
[Tue Sep 12 23:46:39 2006] [error] OPM:Fail to call opm_sys_init
[Tue Sep 12 23:46:39 2006] [error] OPM: process manager could not create internal tables
[Tue Sep 12 23:46:39 2006] [crit] (37)No locks available: Apache JServ
encountered a fatal error; check your ApJServLogFile for details if none are
present in this file. Exiting.

Please note that I specified config_top as local mount point on node2 when adding secondary node to “shared application tier”. However I saw that apache was still trying to write opm_sys_shm and similar lock files on $IAS_ORACLE_HOME/Apache/Apache/logs which is a NFS mount instead of $IAS_CONFIG_TOP/Apache/Apache/logs, which was a local mount. The strace of the adapcctl.sh shows the:
fcntl64(10, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = -1 ENOLCK (No locks available)

Of course you will have issue writing lock file on NFS mount. The workaround I found was that you have to mount NFS with NOLOCK option.