HACMP ADAPTERS

As stated above, each network defined to HACMP should have at least two adapters per node. While it ispossible to build a cluster with fewer, the reaction to adapter failures is more severe: the resource groupmust be moved to another node. AIX provides support for Etherchannel, a facility that can used to aggregateadapters (increase bandwidth) and provide network resilience. Etherchannel is particularly useful forfast responses to adapter / switch failures. This must be set up with some care in an HACMP cluster.When done properly, this provides the highest level of availability against adapter failure. Refer to the IBMtechdocs website: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101785 for furtherdetails.Many System p TM servers contain built-in Ethernet adapters. If the nodes are physically close together, itis possible to use the built-in Ethernet adapters on two nodes and a "cross-over" Ethernet cable (sometimesreferred to as a "data transfer" cable) to build an inexpensive Ethernet network between two nodes forheart beating. Note that this is not a substitute for a non-IP network.Some adapters provide multiple ports. One port on such an adapter should not be used to back up anotherport on that adapter, since the adapter card itself is a common point of failure. The same thing is trueof the built-in Ethernet adapters in most System p servers and currently available blades: the ports have acommon adapter. When the built-in Ethernet adapter can be used, best practice is to provide an additionaladapter in the node, with the two backing up each other.Be aware of network detection settings for the cluster and consider tuning these values. In HACMP terms,these are referred to as NIM values. There are four settings per network type which can be used : slow,normal, fast and custom. With the default setting of normal for a standard Ethernet network, the networkfailure detection time would be approximately 20 seconds. With todays switched network technology thisis a large amount of time. By switching to a fast setting the detection time would be reduced by 50% (10seconds) which in most cases would be more acceptable. Be careful however, when using custom settings,as setting these values too low can cause false takeovers to occur. These settings can be viewed using a varietyof techniques including : lssrc –ls topsvcs command (from a node which is active) or odmgetHACMPnim grep –p ether and smitty hacmp.ApplicationsThe most important part of making an application run well in an HACMP cluster is understanding theapplication's requirements. This is particularly important when designing the Resource Group policy behaviorand dependencies. For high availability to be achieved, the application must have the ability tostop and start cleanly and not explicitly prompt for interactive input. Some applications tend to bond to aparticular OS characteristic such as a uname, serial number or IP address. In most situations, these problemscan be overcome. The vast majority of commercial software products which run under AIX are wellsuited to be clustered with HACMP.Application Data LocationWhere should application binaries and configuration data reside? There are many arguments to this discussion.Generally, keep all the application binaries and data were possible on the shared disk, as it is easyto forget to update it on all cluster nodes when it changes. This can prevent the application from starting orworking correctly, when it is run on a backup node. However, the correct answer is not fixed. Many applicationvendors have suggestions on how to set up the applications in a cluster, but these are recommendations.Just when it seems to be clear cut as to how to implement an application, someone thinks of a newset of circumstances. Here are some rules of thumb:If the application is packaged in LPP format, it is usually installed on the local file systems in rootvg. Thisbehavior can be overcome, by bffcreate’ing the packages to disk and restoring them with the preview option.This action will show the install paths, then symbolic links can be created prior to install which pointto the shared storage area. If the application is to be used on multiple nodes with different data or configuration,then the application and configuration data would probably be on local disks and the data sets onshared disk with application scripts altering the configuration files during fallover. Also, remember theHACMP File Collections facility can be used to keep the relevant configuration files in sync across the cluster.This is particularly useful for applications which are installed locally.Start/Stop ScriptsApplication start scripts should not assume the status of the environment. Intelligent programming shouldcorrect any irregular conditions that may occur. The cluster manager spawns theses scripts off in a separatejob in the background and carries on processing. Some things a start script should do are: First, check that the application is not currently running! This is especially crucial for v5.4 users asresource groups can be placed into an unmanaged state (forced down action, in previous versions).Using the default startup options, HACMP will rerun the application start script which may causeproblems if the application is actually running. A simple and effective solution is to check the stateof the application on startup. If the application is found to be running just simply end the start scriptwith exit 0.Verify the environment. Are all the disks, file systems, and IP labels available?If different commands are to be run on different nodes, store the executing HOSTNAME to variable.Check the state of the data. Does it require recovery? Always assume the data is in an unknown statesince the conditions that occurred to cause the takeover cannot be assumed.Are there prerequisite services that must be running? Is it feasible to start all prerequisite servicesfrom within the start script? Is there an inter-resource group dependency or resource group sequencingthat can guarantee the previous resource group has started correctly? HACMP v5.2 and later hasfacilities to implement checks on resource group dependencies including collocation rules inHACMP v5.3.Finally, when the environment looks right, start the application. If the environment is not correct anderror recovery procedures cannot fix the problem, ensure there are adequate alerts (email, SMS,SMTP traps etc) sent out via the network to the appropriate support administrators.Stop scripts are different from start scripts in that most applications have a documented start-up routineand not necessarily a stop routine. The assumption is once the application is started why stop it? Relyingon a failure of a node to stop an application will be effective, but to use some of the more advanced featuresof HACMP the requirement exists to stop an application cleanly. Some of the issues to avoid are:Be sure to terminate any child or spawned processes that may be using the disk resources. Considerimplementing child resource groups.Verify that the application is stopped to the point that the file system is free to be unmounted. Thefuser command may be used to verify that the file system is free.In some cases it may be necessary to double check that the application vendor’s stop script did actuallystop all the processes, and occasionally it may be necessary to forcibly terminate some processes.Clearly the goal is to return the machine to the state it was in before the application start script was run.Failure to exit the stop script with a zero return code as this will stop cluster processing. * Note: This is not the case with start scripts!Remember, most vendor stop/starts scripts are not designed to be cluster proof! A useful tip is to have stopand start script verbosely output using the same format to the /tmp/hacmp.out file. This can be achievedby including the following line in the header of the script: set -x && PS4="${0##*/}"'[$LINENO]