Friday, September 30, 2011

Application server load balancing and failover

In a previous blog’s entry, I explained about Multiple application messaging servers on Peopletools 8.51 and facilities for load balance and failover on messaging domain server.
In a very similar way, the application server can also be used in load balance and failover configuration.
First of all, you need at least two applications servers, please note that if you are on the same server, then domain name and JSL (jolt) port should be different from each other. Of course, there are not many advantages to have this configuration if working on one and only one server, but rather good to have different server, if one crash, the other one takes place.
Here we go.
First application server :
Apps_LoadB_001 
And the second one :
Apps_LoadB_002
Please, note that whether I name one “first” and the other one “second”, there is no precedence of one compared to the other by default (except if you define a weight in configuration.properties, but let’s leave it for later explanation).
Then starting each application server
Apps_LoadB_003 
Note that the number of listened ports regarding Jolt is defined is the psappsrv.cfg of the domain as following :
[JOLT Listener]
;=========================================================================
; Settings for JOLT Listener
;=========================================================================
Address=%PS_MACH%
Port=9000
Encryption=0
Min Handlers=5
Max Handlers=7
In the current case, the ports between 9000 and 9005 will be immediately in use after started :
[psadm2@psovmfscmfp1 APPDOM]$ netstat -an|grep 900
tcp        0      0 192.168.1.150:9000          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9001          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9002          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9003          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9004          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9005          0.0.0.0:*                   LISTEN
[psadm2@psovmfscmfp1 APPDOM]$
It means the second domain needs to be configured with a port above 9005.
Apps_LoadB_004 

Here’s the output from the database side, since my two apps are on the same machine, there’s nothing really useful so far, cannot distinguished which domain a database session is coming from :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
SQL> /

USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSQRYSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSQRYSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,

21 rows selected.

Now, webserver configuration to be done in configuration.properties file ($PS_HOME/webserv/peoplesoft//applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps).
This is a easy configuration, put the application servers separated by simple comma as following :
[psadm2@psovmfscmfp1 peoplesoft]$ more ./applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps/configuration.properties
<<...snipped...>>
# psserver must point to your application server machine name (recommended) or IP address
# and JOLT port.  The PeopleTools release of the application server must match the PeopleTools
# release of the PeopleSoft files on the web server.
#
# To enable jolt failover and load balancing, provide a list of application server
# domains in the format of;  psserver=AppSrvr:JSLport,...
# For example:  psserver=SERVER1:9000,SERVER2:9010,SERVER3:9020
## sd="Application Server Connect String", ld="This string embodies the failover and load balancing string for PIA to connect to Application Server domains.  Consult the PeopleB
ooks recommendations on load balancing and failover for more information.  The format is psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name.   Multiple d
omains should be comma separated", tip="Format: psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name, e.g  psserver=SERVER1.oracle.com:9000,SERVER2.oracle.
com:9010"
psserver=psovmfscmfp1:9000,psovmfscmfp1:9010
## sd="Jolt Pooling", dt="c", ld="Enable Jolt Pooling"
joltPooling=true
<<...snipped...>>
Configured as above, all the application servers have the same weight (the same priority) in the round-robin algorithm. This algorithm affect ingoing transaction randomly to one or the other configured application server, there’s no priority, no load checkup, no network checkup, it’s just random (find out more in the My Oracle Support note E-WS: Load Balancing Options for Web Server and Application Server [ID 618367.1]).
But we could also introduced weight (from 1 to 10) on application server to make one precedence on the other as following :
psserver=psovmfscmfp1:9000#10,psovmfscmfp1:9010#1
In that case, the first server can takes extra work for an heavy load compared to the second one. 

Not only the load balance will be set, but the failover as well.

After the first connection to the front end application, we can check the client status of both application servers :
Apps_LoadB_007 
We can see a first pool as been instantiated.
Since round-robin algorithm is randomly given connection to the application server, it’s not always easy to show when alone, furthermore, the pool connection are making it even less simple.
But you probably can check it from the back end :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
  2* order by 2
SQL> /

USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,

Here, from the same client machine, different users are in connected through the 2 applications servers (APPDOM and APPDOM2).

Now, let’s simulate a application server crash :
Apps_LoadB_010
The online users won’t be disconnected, and the opened connections are automatically redirected to the second application server (failover) :
Apps_LoadB_011
A second pool is now created, but that one on APPDOM2. Everything will be transparent to the front end users.

Find out more in the OTN hosted Peoplebooks : Working with Jolt Configuration Options

Nicolas.

PS: whilst I was writing this blog entry, I was reading an interesting note as well recommending the use of IP address instead of the server name for security reason in the configuration.properties (worth reading it Health Check Alert: Consider setting PSSERVER for web server JOLT connections to refer to an IP-address:port combination [ID 957484.1]).

Monday, September 19, 2011

3 months update later, finally

I did not maintained this blog for quite a while during this last quarter, I should apology to the readers.
What happened during that time ? “Finally” is probably the word to keep in mind for that period.

CentOS 6.0
In July, CentOS finally released the version 6.0. It has been a very long waiting and many people moved away for an other (equivalent) distribution, I named Scientific Linux (see below). I do not know if the CentOS dev. team made the rights choice to go for the version 6.0 before the update 5.7, but truly enough, no update at all for several months is quite unusual and rather disappointed even though that’s a free distribution.
I did not tested this version yet, but according to my reading across a couple of article on Internet, it could have been done much better especially after that long (long) waiting.
Now, we may have to wait for the update 6.1 which has been promised very soon, still have to define “soon”.

CentOS 5.7
Few days ago, earlier in Sept., we also get that update I was looking for several months too. Much appreciated, the upgrade from 5.6 went smoothly, as usual. The only thing for that one was the delay since the RHEL 5.7 date release. Again, a lot of debates on that…

Oracle Linux 6.1
Of course, this is also free (the use, not the support), but developed by a company, a big company, so the time to get the new version is quite different compared to CentOS.
Oracle Linux is available since the 1st of June, 5.7 since August.

Scientific Linux 6.1 and 5.7
An other free distribution of a RHEL recut, Scientific Linux interest has considerably grown since CentOS took too much time for the newest release delivery.
Whether the update 5.7 has just been released, their update 6.1 is already out for a month. Some can argue the developers behind are also paid… still, this distribution has a good chance to see new customers.

Oracle VM 3.0 (August, the 23rd)
After several years of promises, Oracle has finally released a new version of their virtualization software, Oracle VM 3.0. A lot of new things has to be discovered, according to Oracle, it is much better than ever. But have they not too late compared the the concurrent like VMWare vSphere Hypervisor ?

Open World
On Oracle database and Peoplesoft sides, no major thingy, nothing unusual or non expected, the summer was rather quiet. So, no worries.
But the Oracle Open World is just in the corner by now. Probably we have to count on new Peopletools version 8.52 around that event, maybe new Peoplesoft OVM templates as well (?). Saying about Oracle Open World, this year I won’t attend. I hope attendees will report back on their blog about the exciting sessions.

Other
Above all, and most importantly my son is born also the 23rd of August, which explain the delay and absence of blog update a bit before and since then… but I’ll be back.
The only “problem” I have now, the room which was used to be my server room is occupied. I have to find new space for my noisy server and NAS. But that’s the part of the fun to have children !

Nicolas.

Multiple application messaging servers

In a standard mode, when dealing within messaging, one domain is configured as followed, and most of the time it is enough :
MultipleDomains_001
However, if you have to deal with a high number of messages, to process a maximum number of messages, you may want to create multiple domains to handle those messages.
Let’s create a new one as below : 
MultipleDomains_003 MultipleDomains_004
Then, coming back on the domain status’ page, we see that new domain :
MultipleDomains_005
We have to activate it to make this new domain processing messages. Change the status, and click “update” button : 
MultipleDomains_006
MultipleDomains_007
Here we go. An error is raised, we cannot have more than one master domain. And being master is set by default for a given domain.
MultipleDomains_008
What is that ?
A master domain can be used to balance ingoing/outgoing messages between master and slave(s) domains. Particularly useful when dealing a very high number of messages in a short time.

We can handle it in two ways.

1. Static slave
In the configuration file appsrv.cfg, we can change a parameter value – Domain Slave Mode – from 0 to 1 to make the domain a slave domain.
Change from default value
MultipleDomains_009
To 
MultipleDomains_010
Reload the application server domain, and coming back to the domain status’ page to make it active :
MultipleDomains_011
MultipleDomains_012 
After updating the status, we can see the domain APPDOM2 activated and also that’s a slave domain. It is fixed, there’s no way to change but modify the configuration file manually and reload again the domain.
Without any failover set, it is probably the easiest way to achieve the goal : handle a high number of messages, you could make multiple slaves domains all managed by one and only master domain.
Using multiple slaves domains can also be useful, you could set a load balance in between all of them (click on “Master/Slave LoadBalance”) to configure all the priorities, such as following :
MultipleDomains_013

2. Dynamic slave
An other way for handling a high number of messages and multiple domains would be to make slave domain dynamic. It is maybe a little bit more tricky, but also probably more reliable and scalable.
A dynamic slave domain can become a master domain in case of failure of the current master domain. Really nice features if the messages have to be processed asap or with a minimum of down-time.
So, we return back to the original domain status’ page, the second domain APPDOM2 is inactive (it is not set as static slave in the configuration file):
MultipleDomains_023
Click on “SetUp Failover”, then check “enable Failover”, define a failover time (when the slave domain has to become master after original master failure, here 5 minutes) and most importantly check “Dynamic Slave Option”. Finally set a failover priority (1, 2, 3…) and save :
MultipleDomains_024
It makes that second domain defined as dynamic slave as shown above in the failover setting as well as below in the domain status page :
MultipleDomains_025
From now, we have two domains which proceed with the messages, given the fact that if the first one fails, the second will become the master.
Just to show it, here I stopped the first domain APPDOM, once the failover time is off (as shown earlier, 5 minutes), we can see the second domain APPDOM2 becoming master (it is not “dynamic slave” anymore), and the first obviously becoming inactive :
MultipleDomains_028MultipleDomains_029 
When the first domain APPDOM is restarted, it becomes the slave domain whereas the second domain APPDOM2 remains the master.
MultipleDomains_030


Making choice between static and dynamic slave domain may not be an easy task, it depends of your all architecture and requirements you have in messages processing. The first might be useful on not so big environment whereas the second is more than recommended on big environments (you don’t want to make dynamic domain on non-messages dedicated domain).

Enjoy,

Nicolas.

Find out more on OTN hosted Peoplebooks : http://download.oracle.com/docs/cd/E24150_01/pt851h2/eng/psbooks/tiba/book.htm?File=tiba/htm/tiba13.htm