Sun Software Product Map

The name "Sun Java Systems Communications Suite" is now history. 

Communications Suite is one of the products we are very strong in. 

I just find it weird that Sun Messaging Server has been rebranded as Oracle Communications Messaging Exchange Server.  

Do we really need that word?

PS: The full listing here.

.

Weird IPMP Output - Part II

A telco in the Philippines engaged us to troubleshoot a IPMP (IP Multipathing) issue on one of their Solaris box.

The customer is using Probe-based detection, not Link-based detection.

They always encounter the following 2 problems when IPMP is configured on this particular Solaris box:

1. The standby interface (e1000g2) on the node will always fail whenever it is configured for IPMP
2. Otherwise, when e1000g2 is the active interface, it will never fail-back to the primary interface (e1000g0)

Discovery:
1. OS version is Solaris 10 U7
2. The Patch Level is 141445-09.

3. A check on SunSolve reveals the following fact:

Solaris 10 Kernel Patches 141444-09 (SPARC) and 141445-09 (x86) cause interface failure in IPMP when configured for probe based failure detection. This issue does not occur with a IPMP link based failure detection configuration. (Read here)

4. Resolution

This issue is addressed in the following releases:

SPARC Platform:
Solaris 10 with patch 142900-02 or later

x86 Platform:
Solaris 10 with patch 142901-02 or later

Disk-Path Failure Handling

Disk-Path Failure Handling refers to the ability to automatically reboot a node if all its paths to shared disks have failed. Faster reaction in case of severe disk-path failure enables improved availability.

Note that this feature is only available on Sun Cluster 3.2 onwards.

Sun Cluster vs Veritas Cluster

Last week, my colleague and I traveled to Kuala Lumpur to conduct a TOI (Transfer of Information) session to our customer, a telco solutions provider.

During the session, we were asked the differences between Sun Cluster and Veritas Cluster.

Fairly common question, I'hv been asked a few times before.

Sun Cluster

* Kernel-based - faster in failure-detection

* Only support Sun Solaris OS

* Cheaper overall - e.g. IPMP, MPxIO, UFS, SVM all free from Solaris OS

Veritas Cluster

* Software-based

* Support multiple platforms

* More expensive overall - e.g. VxFS requires extra license

A typical scenario to utilize Veritas Cluster is a mid-sized to large-sized company having multiple platforms (Solaris, HP-UX, AIX ...) in their data center. In order to standardize the clustering software (so as to reduce training and operation cost), Veritas Cluster will be the choice to use.

Fault Monitoring of resource

I'm still in Senegal busy with the Sun Cluster for Oracle HA UAT.

(Pictures below are Novotel Dakar. Very nice cozy hotel just besides the sea.)

Besides configuring Oracle database for HA, we are also responsible for monitoring customer's applications via Sun Cluster.

There are 2 ways to configure Fault Monitoring for Generic Data Service (GDS):

1. Port monitoring (default)

2. Probe command monitoring

Port monitoring is fairly straight-forward. It assumes your application is running on a particular port. If Sun Cluster detects that this port is down, it will assume that your application is faulted. It will then attempt to restart the resource automatically.

The application for this teleco here is pretty complicated. There are times when the port is still alive, but the application has hung. This is exactly what happened here!!

So Port monitoring is not reliable in this case, at least for this application per se.

We need to use Probe command monitoring instead. Probe command will require us to write shell script that return values like 0 (successful), 100 (complete failure) and 201 (immediate failover).

Now, there is an issue - port monitoring is turned on by default. If you have probe command monitoring added, port monitoring is still running. As such, even if probe command returns 100, but if the port is still alive, Sun Cluster still treats the resource to be alive.

This is no good. We need to disable port monitoring and rely totally on probe command monitoring.

How do we achieve that?

-x Network_aware=FALSE

LUN, Device Group and Mount Points are correlated

I have flown 30 hours to reach Senegal for a Sun Cluster for Oracle HA UAT. Customer is a teleco in Senegal.

The following is what I have implemented for them due to a constraint I encountered at the Storage level:

Oracle and App are now residing on the same node at the same time. Always.

Ideally, one would like to have Oracle active on 1 node; and App active on the 2nd node. That would be best in term of performance.

However, there is only 1 LUN being created by the storage engineer before the whole setup was shipped from China. No one here in Senegal knows how to reconfigure the LUN.

What's the implication of 1 LUN?

Now, if one is a Software person like the customer, it would be very hard to appreciate the issue. I need to explain in a more layman term.

Let's look at the table below:

A software person can only understand until column 3 ("Mount Point on Solaris OS"). Beyond that, it would be fairly difficult to grab the terms like Device Group, LUN, etc..

In order to understand, we need to read from right-to-left (yes, the ancient Chinese way of reading).

If there is 1 LUN, there can only be 1 Device Group. If we only have 1 Device Group, then all mount points have to be defined under this group.

As such, as all mount points have to be always together, it implies all applications (Oracle and App) have to be always together.

The ideal architecture will be the one shown below:

Oracle on 1 node; App on the other node

In order to have such a setup, we need to define at least 2 LUN at the Storage level.

Sun Cluster 3.1 Data Service for Oracle 11gR2 HA

A teleco in Brunei required our service to install and configure Sun Cluster Data Service for Oracle HA.

Fairly simple requirement ... only if all components are of the latest, I thought. But when I took a look at the BOM list, I was surprised that they are using a fairly old version of Sun Cluster 3.1. For Oracle database, they want the latest 11gR2 release though.

There isn't enough information on the Internet. Thus, we wrote in to Sun and below is the reply:

Oracle Server

8.1.6 32&64bit 8.1.7 32&64bit 9i 32&64bit SC 3.1 Solaris 8, 9
•Note thatOracle8.1.x have been desupported by Oracle. However, when the customer has continuing support for Oracle 8.1.x from Oracle,Sun will continue supporting the Sun Cluster HA Oracle agent with it.

9i R232 & 64bit 10G R1& R2 64bit Solaris 8, 9,10
•Both Standard and Enterprise Editions are supported

11g Solaris 9, 10
•Both Standard and Enterprise Editions are supported

Sun High Performance Computing

Sun is gearing up big on High Performance Computing (HPC) from my own observation on the ground. 

It now offers Business Ready HPC:

Industry solutions optimized for maximum application performance, efficiency, and scale with reliability built-in. Shorten your product development cycles. Enable you to make better decisions, faster. Boost your ROI, giving you a major competitive advantage.

The solution is compelling. It has readily-deployable solution for the industries that require high-performance and great reliability. Read more here. 

Besides storage, the central piece of Sun HPC solution its software - Sun HPC Software, Linux Edition.

Sun HPC Software, Linux Edition is an integrated, open-source software solution for Sun HPC clusters. It simplifies the deployment of HPC clusters by providing a ready-made framework of software components to use to turn a bare-metal system into a running HPC cluster. It provides software to provision, manage, and operate large scale Linux HPC clusters and serves as a foundation for optional add-ons such as schedulers, like Sun's Sun Grid Engine, and other components not included with the solution.

As usual, the Linux Edition is open-source and thus FOC. If you do not feel comfortable without support, you can run Solaris Cluster (formerly known as Sun Cluster). 

Sun Cluster Hardening

For those in the defense industry, you might be required to harden your Sun Cluster after deployment. 

Do take note of the following:

Sun "supports" Sun Cluster hardening via the Solaris Security Toolkit (aka JASS) only. The reason is that we test it and fix bugs in either product as required. There are many subtle issues when hardening  clusters.

Of particular note, a service which is expected to be up, should be up and observable. Otherwise it is indistinguishable from being down and can lead to cluster reconfiguration.

You should also point this blueprint article out for your customer which explains some of the issues (though it is slightly aged).

Solaris Security Toolkit ( formerly known as JumpStart Architecture and Security Scripts [JASS] ) can be downloaded from here.

How to Boot a Cluster Node in Noncluster Mode for x86 platform

I was conducting a UAT for a local defense organization. They are running Sun Cluster 3.2 on Solaris 10 x86 platform.

One question the engineer asked after I demostrated how to shut down all nodes in the cluster by just issuing "shutdown -g0 -y":

Now that the nodes are shutdown and the system displays "Press any key to continue", how can we bring one of the node up while we perform maintenance activity on the other?

Hmm... good question. If it's for Sparc, I have a ready answer ("$ boot -xs") since most of our implementations are for Sparc platform. Now that they are using x86, I was kind of stumbled.

For quick turnaround, I called my colleague instead. He has the answer for me instanteously!

a. In the GRUB menu, use the arrow keys to select the appropriate Solaris entry and type e to edit its commands.

b. In the boot parameters screen, use the arrow keys to select the kernel entry and type e to edit the entry.

c. Add -x to the command to specify that the system boot into noncluster mode.

He saved my day! Thank you very much!

PS: Detailed reading here.

What happens when both heartbeat cables are unplugged from Sun Cluster nodes?

I deployed a Sun Cluster for a local defense customer a few months ago. On Friday, the engineer called to ask:

What happens when both heartbeat cables (Cluster Private Interconnect) are unplugged?

Above is a typical logical network diagram for a Sun Cluster deployment. (In fact, I use the same diagram for all cluster projects :> Just make sure the hostnames and NIC interfaces are named appropriately for that particular customer)

The answer is simple:

Whichever node that loses the vote from the Quorum Device will go into panic mode. The winning node will survive. If the Resource Group was with the losing node, then it will be failover to the winning node.

The key thing is note is:

There is 50-50% chance that a node will win. Thus we can never configure the cluster such that a particular node will always lose if the heartbeat cables are unplugged.

By the way, this scenario almost never happen before. Imagine the configuration is such that there are 2 heartbeat cables connected to 2 separate NIC cards on each node. You'll strike lottery if this scenario really happens!

Weird IPMP Output

In Solaris, there is this very nice feature called IPMP (IP Multipathing).

It is able to failover the IP address from one network card to the other when the primary fails. From the end-user point of view, it's transparent. Services continue and there's no downtime. This is nice!

IPMP is mandatory when Sun Cluster is configured.

Anyway, we were conducting UAT few days back and I observed this very weird output when both network cables are unplugged from the 2 network cards. We thought the IPMP was not configured properly. In fact, nothing was wrong.

In normal operational situation, you'll see the following output:

Now, we unplug the cable from e1000g0 interface. The output is still correct:

(Notice that the IP address 10.50.129.81 from e1000g0 has failover to bge0:1)

Let's proceed to unplug the cable from bge0 interface. Now, the output is misleading:

Why is bge0 and bge0:1 still showing UP? On the same machine, we can even perform a ping to 10.50.129.81/10.50.129.90 and they are still showing alive.

Very strange indeed. We were puzzled. 

It's only some time later then we realized we did not take a detailed look at the output:

Although the UP flag is there, there is a FAILED flag behind which implies that the interface is indeed down.

We switched our test: bring bge0 down; then bring e1000g0 down. This time round, e1000g0 is showing the UP + FAILED flag. It's always the second interface to be brought in the IPMP group that will show this UP + FAILED flag.

Misleading indeed ...

Total Fibre Cables Failure Test on Sun Cluster

We were having our UAT few days back with our National Healthcare customer here in Singapore.

They have SAP with Oracle running on Sun Cluster 3.2. There are 2 nodes which share a common storage SL500 connected via 2 pairs of Fibre Cables.

One of the test was to ensure that if the 2 fibre cables which are connected to a node are accidentially plug out, the resources running on that node should fail over to the other active node (which still has FC connection to the storage).

As this is a migration job for a hardware upgrade, they would like to use back the same UAT Test Cases from another vendor. (Actually I do not like this arrangement, really)

Nevertheless, we went ahead, but were stuck with this Total Fibre Cables Failure Test.

Instruction:

• Unplug both fibre cables from node nodeA and run “vxdctl enable” to rescan the devices, plug both cables after test.

Expected Results

• There will be error message on node nodeA's console showing link failure of both FC HBA port. After some time, the resource group, oracledb-rg, will failover to node nodeB.

We kept testing but the resource group simply refused to fail-over. We did, however, saw the link failure error message.

We debugged and we discussed. We wanted to know what was the actual expected result then. We were then told by the customer that he actually saw nodeA rebooting when the FC are taken out from nodeA. And this action actually causes the resource group to failover to nodeB.

Now, this is simple!

root@nodeA # clnode show

=== Cluster Nodes ===
Node Name: nodeA
Node ID: 1
Enabled: yes
reboot_on_path_failure: disabled

Node Name: nodeB
Node ID: 2
Enabled: yes
reboot_on_path_failure: disabled


The reboot_on_path_failure was set to "disabled" which means even if there is a FC path failure, no action is taken. (aka nodeA will not reboot; and if nodeA does not reboot, the resource group will not failover)

To solve this problem, it's simple.

root@nodeA # clnode set -p reboot_on_path_failure=enabled nodeA nodeB


In addition, we need to make sure local disk is set to unmonitored.

root@nodeA # cldev status

=== Cluster DID Devices ===

Device Instance   Node  Status
---------------   ----  ------
/dev/did/rdsk/d1  nodeA Unmonitored
/dev/did/rdsk/d10 nodeA Ok
                  nodeB Ok

local disk refers to "/dev/did/rdsk/d1".

If only we could write our own UAT document .... *sigh*

# cluster shutdown -g0 -y

Sun Cluster 3.2 can be installed on both Sparc and x86/64 platform. 

The command "cluster shutdown -g0 -y" shuts down the entire cluster in an orderly fashion.

“cluster shutdown” performs the following actions when it shuts down the cluster: 

• Changes all functioning resource groups on the cluster to an offline state
  
• Unmount all file systems
  
• Shuts down all active device services
  
• Runs /usr/sbin/init 0 on all nodes

Now, the only difference between Sparc and x86/64 platform is in the last step when /usr/sbin/init 0 is executed.

On Sparc platform, all nodes will drop to "OK" prompt. On x86/64 platform, all nodes will display "Press any key to continue".

By the way, you'll be surprised how extensively is Sun Cluster being used in the healthcare/maritime/defense industry in S'pore. 

Cannot start Java Web Console (smcwebserver)

I deployed Sun Cluster for MySQL HA two weeks ago for a defense customer. Everything runs fine and they are able to manage the cluster via CLI (command line interface).

Now, they request for Sun Cluster Manager which was introduced to them during their course. Frankly speaking, I'm trained in Sun Cluster but I have never seen the GUI version before. I did not know of its existence. :)

Upon their request, I made some research and realized it's accessible  via Java Web Console. Now, I know a lot about Java Web Console! I'hv deployed numerous Sun Java System Directory Manager 6.x. Managing directory configuration/data can be made via Java Web Console.

Well, to start Java Web Console is trivial.

1. Ensure Cacao is running

    $ cacaoadm status

2. Ensure SMC Web Server is running

    $ smcwebserver status

3. If it is not running, start it

    $ smcwebserver start

Today, I'm not lucky. I kept getting this error while starting Java Web Console.

$ smcwebserver start

Starting Sun Java(TM) Web Console Version 3.1 ...

Cannot determine if console service is running.

Check log file: /var/svc/log/system-webconsole:console.log

Run "svcs system/webconsole:console" to determine its status.

Finally, the issue was resolved.

$ /usr/share/webconsole/private/bin/wcremove -i console

$ svcadm clear system/webconsole:console

Restart of the server is required. We need to be careful as we are in cluster mode.

On node 1,

$ clnode evacuate 

$ init 6

Upon node 1 reboot, few things happen:

1. Node 1 rejoins the cluster

2. cacao and smcwebserver daemon start automatically

3. Able to access https://localhost:6789

On node 2,

$ clnode evacuate 

$ init 6

The same happens to node 2 upon reboot.

*Phew*

Framework and Data Services for Sun Cluster 3.2

As you know by now, the company (Azimuth Labs) I work for is a partner of Sun Microsystems. We carried out Professional Services on behalf of Sun.

Yesterday, I went to perform a Sun Cluster 3.2 Data Service for SAP on Solaris 10 for one of the national healthcare group in Singapore.

It was supposed to be a breeze job for me. However, when I arrived, I realized the EIS was not performed. I spent a couple of minutes to explain to the anxious customer who insisted that the SAP had to be clustered by end of the day.


To get Sun Cluster to be fully functional, 2 tasks are involved:

• Installation of Sun Cluster Framework (Cluster binary install; Volume Manager install & configuration)
• Configuration of Sun Cluster Data Service

Here's the difference between Framework and Data Services for any Sun Cluster implementation:

• EIS ("Enterprise Installation Service") is a comprehensive installation service that comes packaged with your Solaris OS, Sun Cluster and Volume Manager (Solaris or Veritas) purchase.
• Data Service is a configuration service that comes packaged with your specific Cluster Agent purchase. (in this case, customer bought Oracle HA agent, SAP agent, NFS agent, and Generic Data Service agent for her SAP systems)

Now, the next thing to understand is:

• EIS is carried out by the SS (System Service) team from Sun and is time-consuming
• Data Service is carried out by the PS (Professional Service) team from Sun - this is where I play my part

As such, we went to the extend to activate the SS team from Sun to perform the EIS for Sun Cluster framework installation. It took many hours before I could continue with my Data Service implementation.