Home » Infrastructure » Unix » Veritas Cluster, problem connecting to DB service with virtual IP
Veritas Cluster, problem connecting to DB service with virtual IP [message #226096] Thu, 22 March 2007 09:16 Go to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Dear all

I'm running Veritas Cluster Server on two Solaris 10 nodes. In service group Oracle, I have configured a database service which can be run either on node1 or node2. Basically, resources that the service relies on are:
- disk group (external storage) where the database actually resides (which needs to be attached either to node1 or node2, depending on which node currently runs the DB)
- virtual IP which is used by DB clients to connect to database

Both startup of the virtual IP interface and attaching the disk group works fine. TNS ping also works fine from both nodes.

However, the DB service itself does not start. When I try to connect to the database using sylplus user/pass@service, I get the response "shared memory realm does not exist".

I tried to play with listeners on both nodes, and I learned that if I configure Oracle listener (listener.ora, tnsnames.ora) with the physical IP address of the node, then everything works fine, the DB service can be started from Cluster GUI and clients can normally connect.

Any sugestions (even the most obvious ones) will be appreciated.
Regards
Tihomir

% tnsping database

TNS Ping Utility for Solaris: Version 10.1.0.3.0 - Production on 22-MAR-2007 15:12:41

Copyright (c) 1997, 2003, Oracle. All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oracle)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = database)))
OK (10 msec)
%
% sqlplus system/password as sysdba

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 15:01:55 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 167772160 bytes
Fixed Size 1301272 bytes
Variable Size 69477608 bytes
Database Buffers 96468992 bytes
Redo Buffers 524288 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

% sqlplus user/password@database

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 15:03:54 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory


Enter user-name:
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226113 is a reply to message #226096] Thu, 22 March 2007 10:29 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Also noticed this, when I try to startup the DB manually:

% sqlplus /nolog

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 16:23:29 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

SQL> connect / as sysdba
Connected.
SQL> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL>
SQL> startup
ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist
SQL> startup mount database
ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist

Is there any place in Oracle configuration where I could make a binding between this virtual IP and hostname for it? I have a feeling that oracle somehow doesn't link between Veritas' virtual IP for oracle, and the virtual hostname which is defined in listener.ora and tnsnames.ora, and which I added into /etc/hosts....

Or is it Veritas to blame??

Thanks
Tihomir

[Updated on: Thu, 22 March 2007 10:30]

Report message to a moderator

Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226127 is a reply to message #226096] Thu, 22 March 2007 11:09 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>Or is it Veritas to blame??
Most likely PEBKAC is to blame.
SQL> / AS SYSDBA
SQL> SHUTDOWN ABORT
SQL> STARTUP
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226139 is a reply to message #226127] Thu, 22 March 2007 11:32 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Yes, probably Smile Exactly PEBKAC I'm trying to improve by asking questions...

It's not a problem of badly closed and dismounted db, I think... This doesn't help. The DB works perfectly when listener is set up to work with physical IP... The problem happens only when I use the hostname "oracle" in listener.ora (this hostname is related to Veritas' virtual IP for Oracle service group). This hostname is pingable, even tnsping works, but the database itself cannot be accessed while the listener is configured this way....

Second opinion, maybe?

Thank you
Tihomir
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226142 is a reply to message #226096] Thu, 22 March 2007 11:47 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Just to show you that I tried:

(mounted disk, virtual IP enabled)

% tnsping database
TNS Ping Utility for Solaris: Version 10.1.0.3.0 - Production on 22-MAR-2007 17:47:07
Copyright (c) 1997, 2003, Oracle. All rights reserved.
Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oracle)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = database)))
OK (20 msec)

% echo $ORACLE_SID
database

% sqlplus /nolog
SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 17:43:52 2007
Copyright (c) 1982, 2004, Oracle. All rights reserved.
SQL> connect / as sysdba
Connected.
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 167772160 bytes
Fixed Size 1301272 bytes
Variable Size 69477608 bytes
Database Buffers 96468992 bytes
Redo Buffers 524288 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

% sqlplus user/password@database
SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 17:44:35 2007
Copyright (c) 1982, 2004, Oracle. All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory


Enter user-name:
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226144 is a reply to message #226096] Thu, 22 March 2007 11:54 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>The DB works perfectly when listener is set up to work with physical IP
Your problem statement is confusing/contradictory to me.
IMO, two seperate issues seem to have been reported.
1) STARTUP
2) SQL*NET access
First & foremost SQL*Net is NOT required for a running database.
If in fact the #1 line above reflects reality, then exactly what is the problem?
Do NOT describe the problem.
Provide CUT & PASTE that shows exactly what you are doing & how the system responds.
I suspect a problem in listener.ora, so posting its contents may provide clues.
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226149 is a reply to message #226144] Thu, 22 March 2007 12:10 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Hello

I'm sorry if I sound confusing -- I'm not experienced with using Oracle, just trying to give as much data as possible... Here the contents:

bash-3.00# cat /opt/oracle/10/network/admin/listener.ora
# listener.ora Network Configuration File: /opt/oracle/10//network/admin/listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = database)
(ORACLE_HOME = /opt/oracle/10/)
(SID_NAME = database)
)
)

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = oracle)(PORT = 1521))
)
)
)

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

bash-3.00# cat /opt/oracle/10/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File: /opt/oracle/10/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

DATABASE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = oracle)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = database)
)
)


"oracle" is on the Solaris level resolved to x.x.x.17, which is a pingable virtual IP created by Veritas for use by Oracle clients.

If I change "oracle" to real hostname of the node, I can login with sqlplus user/password@database... But if I do that, I'm loosing intended redundancy -- the idea is that virtual IP should be used...



And this is the log of what I was doing:

% ping oracle
oracle is alive
% tnsping database

TNS Ping Utility for Solaris: Version 10.1.0.3.0 - Production on 22-MAR-2007 18:17:46

Copyright (c) 1997, 2003, Oracle. All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oracle)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = database)))
OK (10 msec)
% sqlplus user/password@database

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 18:18:06 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory


Enter user-name: ^C
% sqlplus /nolog

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 18:18:19 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

SQL> connect / as sysdba
Connected.
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 167772160 bytes
Fixed Size 1301272 bytes
Variable Size 69477608 bytes
Database Buffers 96468992 bytes
Redo Buffers 524288 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
% sqlplus user/password@database

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 22 18:19:08 2007

Copyright (c) 1982, 2004, Oracle. All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory


Enter user-name: ^C^C
%

[Updated on: Thu, 22 March 2007 12:20]

Report message to a moderator

Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226157 is a reply to message #226096] Thu, 22 March 2007 12:33 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
hmmm....
I'm am unclear why you say ""oracle" is on the Solaris level resolved to x.x.x.17, which is a pingable virtual IP created by Veritas "; cuz I thought hostname to IP# translation was handled by DNS & not by some volume manager.

It is not clear from which box you were attempting the SQL*Net connection.
If it is from the database server itself what is the contents of /etc/host & what is the name resolution order from /etc/nsswitch.conf

If you are really, really getting to the desired listener on the DB server, a tail -f listener.log should show the incoming request. If this log file does not report the attempt to connect, then the attempt is going to some other host which then reports the DB is not available on what ever node gets the request.
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226172 is a reply to message #226096] Thu, 22 March 2007 13:11 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Hi

I try to run sqlplus on the same machine the DB is running on. There are all together two nodes, which are in the same network, fully visible to each other, and controlled by Veritas. DNS isn't used at all (for increased reliability), every hostname that needs to get resolved (just a couple of them) is in /etc/hosts, which looks like this:

% cat /etc/hosts
#
# Internet host table
#
127.0.0.1 localhost
10.6.132.7 emm1 emm1.ims.mncxxx.mccxxx.3gppnetwork.org loghost
10.6.132.7 emm1-ce1
10.6.132.8 emm1-ce5
10.6.132.10 emm2
10.6.132.17 oracle
10.6.132.12 bgw
10.6.132.12 emm

Listener shows the status
emm1% /opt/oracle/10/bin/lsnrctl status LISTENER

LSNRCTL for Solaris: Version 10.1.0.3.0 - Production on 22-MAR-2007 18:56:08

Copyright (c) 1991, 2004, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Solaris: Version 10.1.0.3.0 - Production
Start Date 22-MAR-2007 18:45:16
Uptime 0 days 0 hr. 10 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/10/network/admin/listener.ora
Listener Log File /opt/oracle/10/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.6.132.17)(PORT=1521)))
Services Summary...
Service "database" has 1 instance(s).
Instance "database", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
emm1%

And tail -f listener.log added up the following line when I entered sqlplus user/password@database:
22-MAR-2007 18:57:33 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=database)(CID=(PROGRAM=)(HOST=emm1)(USER=database))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.6.132.17)(PORT=37807)) * establish * database * 0

What I find strange in this line is that it uses "emm1" instead of "oracle" (although the connection gets established apparently)

I'm close to reinstalling Oracle Smile

Thanks
Tihomir

[Updated on: Thu, 22 March 2007 13:17]

Report message to a moderator

Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226175 is a reply to message #226096] Thu, 22 March 2007 13:22 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>I'm close to reinstalling Oracle
IMO, this will be a useless exercise.
I suspect that the problem is at the OS level.
>And tail -f listener.log added up the following line when I entered sqlplus user/password@database:
Did you actually get connected to the DB using sqlplus?
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226176 is a reply to message #226096] Thu, 22 March 2007 13:46 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Hi

Yes, I did connect using "sqlplus user/pass@db", but only after editing listener.ora and tnsnames.ora, replacing "oracle" with "emm1"...

Which supports your opinion that the problem is in OS, or in Veritas Cluster -- and not in Oracle itself...

Regards
Tihomir
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226179 is a reply to message #226096] Thu, 22 March 2007 14:01 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
You have TWO different IP#s in play.
If the OS is NOT configured to accept both IP#s by the same h/w interface, then I don't see how the listener can be expected to acknowledge/process any request targeted for *17
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226394 is a reply to message #226096] Fri, 23 March 2007 08:53 Go to previous messageGo to next message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Hi

Thank you for your time and effort, first of all. What did you mean by mentioning two IP addresses? I have configured network interfaces ce1 (addr. .7) and ce5 (addr. .Cool. Also, in addition to that, Veritas Cluster forms a virtual interface (addr. .17), which is intended to serve for Oracle requests and is bound to hostname "oracle" in /etc/hosts. So, except entering "oracle" in listener.ora and tnsnames.ora (instead of a localhost), is there anything else I should do?

Recently I did the following:
node2% lsnrctl services LISTENER

LSNRCTL for Solaris: Version 10.1.0.3.0 - Production on 23-MAR-2007 14:43:17

Copyright (c) 1991, 2004, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle)(PORT=1521)))
Services Summary...
Service "database" has 1 instance(s).
Instance "database", status UNKNOWN, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:20 refused:0
LOCAL SERVER
The command completed successfully
emm2% tnsping database


Is there any way to change this "LOCAL SERVER" into anything else? Could that perhaps be the problem?

I also tried to configure listener with EXTPROC part (just read about it), but then the listener reports that it doesn't know of service requested in connect descriptor...

Thanks
Tihomir
Re: Veritas Cluster, problem connecting to DB service with virtual IP [message #226418 is a reply to message #226096] Fri, 23 March 2007 10:53 Go to previous message
tcavuzic
Messages: 9
Registered: March 2007
Location: Croatia
Junior Member

Hi

The problem has been solved Smile

The problem was in the following files:
/opt/oracle/.login
/var/opt/oracle/oratab

ORACLE_HOME values in these two files must be (of course) completely the same, but interestingly, they MUST end with / -- if the last slash is left out (which still is a valid Solaris path!) in either one or in both files, there are problems.

Thank you for trying to help!!
Tihomir
Previous Topic: Junk Character in VI editor
Next Topic: Difference between init 0 and init 5 -solaris 8.
Goto Forum:
  


Current Time: Thu Mar 28 07:43:08 CDT 2024