Thursday, April 14, 2011

Application messaging troubleshooting : endless ping node

It’s really common to use a small configuration for environments such as DEMO, SYS or even for development databases to avoid the server to be overloaded within useless processes. But doing so, you should be sure it is not set too small otherwise you could have some issues when setting the messaging up.
Here’s an example of a common issue a Peoplesoft DBA might be face to.
I’m working here on the PSOVM CRM 9.1 based on Peopletools 8.50.09. Of course, it is assuming the Pub/Sub service has been set to “Yes” as shown below:
AppMsg_Endlessping_027

Checking the local node for proper configuration (user and password must be set) :
 AppMsg_Endlessping_003
So far, so good.
Now, let’s configure the gateway. For initial configuration, load the connector first :
AppMsg_Endlessping_010 
AppMsg_Endlessping_011
And try to ping the gateway to confirm it works :
AppMsg_Endlessping_012
AppMsg_Endlessping_013
Now, let’s configure the node in the gateway :
AppMsg_Endlessping_014
Default user id is “administrator”, default password is “password”.
AppMsg_Endlessping_015
AppMsg_Endlessping_016
Now try to ping that new added node :
AppMsg_Endlessping_017
AppMsg_Endlessping_018
It will process endlessly until reaching the timeout.
AppMsg_Endlessping_020
Note that there’s no error messages in any log files, nothing to help.
Having a look into the configuration file, psappsrv.cfg, PSAPPSRV has only 1 min. instance :
AppMsg_Endlessping_001
Change it to a minimum of 2 :
 AppMsg_Endlessping_019
Reconfigure and restart the application server :
AppMsg_Endlessping_021 
AppMsg_Endlessping_022
Note there’s now 2 processes PSAPPSRV.

Coming back in the front end application, trying again to ping the local node within the gateway configuration as previously failed :
AppMsg_Endlessping_023
Few seconds later, it is successful :
AppMsg_Endlessping_024

So, when you want to work within the messaging, always set at least 2 as a minimum instances for PSAPPSRV process in the configuration file.

As a side note, if you fill a wrong Peopletools release as shown below, it will fail but that time with a clear explanation (Integration Gateway: PeopleSoftTargetConnector - invalid PeopleTools release). So, be sure about the correctness  of the release until the patch level :
AppMsg_Endlessping_025
AppMsg_Endlessping_026

Nicolas.

Sunday, April 10, 2011

Peoplesoft PIA silent deployment

Whether the silent Weblogic installation is documented in the Peopletools Installation Guide (Task 2-1-5: Installing Oracle WebLogic on Linux or UNIX in Silent Mode), the silent PIA deployment is not. Is that because it does not exists ? Is that because there’s no use of that ? Well, it exists and in few cases it can help. Especially if you are applying Peopletools patch on regular basis. As you may know, it is most often required to redeploy the domain.

Last month I replied to a resurrected thread on Peoplesoft OTN Forum regarding a silent deployment of the PIA. I was initially – last year - thinking that was not possible. But working a lot with the PSOVM since then, I saw the PIA deployment on “fly” during the OVM template installation.

Going through the PSOVM App/Batch/PIA server once deployed, there’s a script which is called to deploy the PIA : /opt/oracle/psft/vm/installpia.sh
The script is relatively easy to understand :
PT_HOME=/opt/oracle/psft/pt
TOOLS_HOME=$PT_HOME/tools
BEA_HOME=$PT_HOME/bea
SILENT_INSTALL_FILE=install.properties
 
echo "Doing PIA silent install"
if [ ! -d $TOOLS_HOME/tmp ]; then
    mkdir $TOOLS_HOME/tmp
fi
 
cd $TOOLS_HOME/tmp
 
 
# If there is a pre-existing silent response file we
# remove it
if [ -f $SILENT_INSTALL_FILE ]; then
    rm -f $SILENT_INSTALL_FILE
fi
 
echo PS_HOME=$TOOLS_HOME >> $SILENT_INSTALL_FILE
echo DOMAIN_NAME=peoplesoft >> $SILENT_INSTALL_FILE
echo SERVER_TYPE=weblogic >> $SILENT_INSTALL_FILE
echo BEA_HOME=$BEA_HOME >> $SILENT_INSTALL_FILE
echo USER_ID=system >> $SILENT_INSTALL_FILE
echo USER_PWD=password >> $SILENT_INSTALL_FILE
echo INSTALL_ACTION=CREATE_NEW_DOMAIN >> $SILENT_INSTALL_FILE
echo DOMAIN_TYPE=NEW_DOMAIN >> $SILENT_INSTALL_FILE
echo INSTALL_TYPE=SINGLE_SERVER_INSTALLATION >> $SILENT_INSTALL_FILE
echo WEBSITE_NAME=$PIA_SITENAME >> $SILENT_INSTALL_FILE
echo APPSERVER_NAME=localhost >> $SILENT_INSTALL_FILE
echo JSL_PORT=$JOLT_PORT >> $SILENT_INSTALL_FILE
echo HTTP_PORT=$PIA_HTTP_PORT >> $SILENT_INSTALL_FILE
echo HTTPS_PORT=$PIA_HTTPS_PORT >> $SILENT_INSTALL_FILE
echo AUTH_DOMAIN= >> $SILENT_INSTALL_FILE
echo WEB_PROF_NAME=DEV >> $SILENT_INSTALL_FILE
echo WEB_PROF_USERID=PTWEBSERVER >> $SILENT_INSTALL_FILE
echo WEB_PROF_PWD=PTWEBSERVER >> $SILENT_INSTALL_FILE
echo REPORTS_DIR=/var/tmp/psreports >> $SILENT_INSTALL_FILE
 
cd $TOOLS_HOME/setup/PsMpPIAInstall
sh setup.sh -javahome $TOOLS_HOME/jre -i silent \
-DRES_FILE_PATH=$TOOLS_HOME/tmp/$SILENT_INSTALL_FILE -tempdir $TOOLS_HOME/tmp

So, first build your own response file with all the required fields according to your environment and action you are doing (deployment, re-deployment), and call it within DRES_FILE_PATH. That parameter was unknown for me. It is rather a nice learn. Of course, it might require a lot of testing, but it could be helpful if you have a high number of environments to re-build time to time, especially useful after a Peopletools (patch) upgrade which can be done on regular basis.

Save the response file in a secure place, and eventually you’ll see the advantages of silent PIA deployment. Run the installer with the silent option avoids some painful time to remember what ports are used, what webprofile, what reports directory…
I was initially not convinced about the usefulness of that, but try it. Use it.

Nicolas.

Saturday, April 09, 2011

Manage Peoplesoft with OEM Grid Control (3/3)

Last part of this series about Peoplesoft managed in OEM Grid Control, once the OEM Grid Control is installed with the Peoplesoft plug-in, once the Agent is installed on the targets with the Peoplesoft plug-in, it’s time for configuring the all thing in OEM GUI.

First, add the Peoplesoft target : OEM_InstPsPlugin_030
The user is the Peoplesoft domain owner (not necessarily the owner of the PS_HOME), for the PSOVM servers, that’s psadm2.
OEM_InstPsPlugin_048
The discovery path is the PS_CFG_HOME. If it is missing from the environment variable, it is going to the user’s home directory. For the PSOVM servers, it is going to the psadm2’s home :
OEM_InstPsPlugin_056
Choose the type you want to manage :
OEM_InstPsPlugin_057 
OEM_InstPsPlugin_058
We can link the Oracle database when Peoplesoft is discovered (here the db server was not started) :
OEM_InstPsPlugin_059 OEM_InstPsPlugin_060
And finally, everything selected earlier is now recognized :
OEM_InstPsPlugin_061

From now, we can start/stop (all in one shot if required) and configure the all Peoplesoft domain.

Let’s see the Process Scheduler possibilities :OEM_MgntPsPlugin_004 OEM_MgntPsPlugin_005
Here, we can do whatever we are doing within the psadmin tool from the target host:
OEM_MgntPsPlugin_006 
OEM_MgntPsPlugin_008 

Let’s see the Application Server possibilities :OEM_MgntPsPlugin_010 OEM_MgntPsPlugin_011  OEM_MgntPsPlugin_013  OEM_MgntPsPlugin_015

OEM_MgntPsPlugin_016

And the WebServer :OEM_MgntPsPlugin_017 OEM_MgntPsPlugin_018 OEM_MgntPsPlugin_019  
I won’t go through every single sub-menus over there, there are too many. But as you can see above, everything we – as Peoplesoft DBA - usually do from the backend, directly connected onto a server, can be done through OEM Grid Control. Rather nice, especially when managing a lot of environments. That’s maybe a bit of work for configuring everything properly, according to all the specificities of the environments, but that makes the life so easier once it is done that I think it is worth to do and to have on board.

Enjoy it.

Nicolas.

Friday, April 08, 2011

Manage Peoplesoft with OEM Grid Control (2/3)

After installing the OEM Grid Control 11gR1 – the OVM template, including the Peoplesoft plug-in for OMS as shown here, it’s time for the target system to be prepared. All the client servers have to have the OM Agent. By client, it means client for the OMS, say so the clients are the databases servers and Apps/Batch/PIA servers which I created based on PSOVM as shown in previous posts. But the Peoplesoft plug-in is crawling a PS_HOME, it has to be installed only on Apps/Batch/PIA servers.

1. The OEM Agent
Whether there’s space enough in $ORACLE_HOME (/u01/app/oracle/product) on the PSOVM database servers (user oracle), there’s no space left next to the Oracle client (/opt/oracle/psft/pt) on the PSOVM App/Batch/PIA servers (user psadm1).
Before installing the Agent on App/Batch/PIA, let’s make a dedicated folder where’s space allows us to install it, as following :
[root@psovmhcm /]# cd /
[root@psovmhcm /]# df -k .
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2             3876120    724696   3111424  19% /
[root@psovmhcm /]# mkdir oracle_agent
[root@psovmhcm /]# chown -R psadm1:oracle oracle_agent
[root@psovmhcm /]# cd /opt/oracle/psft/pt
[root@psovmhcm pt]# ln -s /oracle_agent .
[root@psovmhcm pt]# ls -lrt
total 56
drwx------  2 root   root   16384 Dec 20 19:44 lost+found
drwxr-xr-x  9 root   root    4096 Dec 20 20:08 bea
drwxr-xr-x  7 root   root    4096 Dec 20 20:12 oracle-client
drwxr-xr-x  5 root   root    4096 Jan 23 09:37 cobol
drwxr-xr-x 28 psadm1 oracle  4096 Jan 23 09:43 tools
lrwxrwxrwx  1 root   root      13 Mar 30 11:16 oracle_agent -> /oracle_agent
[root@psovmhcm pt]#

Remember, the PSOVMs are 64bit word sized (furthermore Peopletools 8.5x is only supported on 64bit OS), the Agent to be downloaded is from Linux x86-64, 11.1.0.1.0 :
http://www.oracle.com/technetwork/oem/grid-control/downloads/agentsoft-090381.html
Once downloaded, upload the file to the PSOVM OEM Grid Control previously installed, into the folder /u01/OracleHomes/Middleware/oms11g/sysman/agent_download/11.1.0.1.0 and unzip it.

Then, everything can be done through the OEM GUI. Much easier than ever, especially when deploying Agent through multiple target. However, screenshots below are coming from a deployment on one target only :
OEM_InstAgent_001OEM_InstAgent_002
Choose the platform Linux-x86-64, that’s the one we’ve just upload to the OMS server :
OEM_InstAgent_003
Here you mention the owner of the Agent home (OMA_HOME),
=> for the PSOVM database servers it will be oracle,
=> for the PSOVM App/Batch/PIA servers it will be psadm1
Then, here you also mention the parent folder hosting the Agent home (OMA_HOME),
=> for the PSOVM database servers it will be /u01/app/oracle/product,
=> for the PSOVM App/Batch/PIA servers it will be /opt/oracle/psft/pt/oracle_agent 
Uncheck the run of root.sh, it has to be done manually later on (somehow it failed in my case when it was checked). OEM_InstAgent_009
Give the sysman password :
OEM_InstAgent_005OEM_InstAgent_006
OEM_InstAgent_007
Be patient during these phases, depending of the number of targets you are deploying to and of the network bandwidth, it can take a quite while :
OEM_InstAgent_008
OEM_InstAgent_012OEM_InstAgent_014
Now, the new host has been added in the host’s list :
OEM_InstAgent_015
Do not forget to run root.sh (in $OMA_HOME). We can eventually check the agent status from the target host :
OEM_InstAgent_021
And the database as well in the database’s list :
OEM_InstAgent_022
Just to show it, I also deployed the Agent through multiple targets at once :
OEM_InstAgent_023
In the end, we got all the hosts targets
OEM_InstAgent_032
         
Warning :
1. Do not forget to run root.sh from the target Agent home, otherwise you may encountered the following error (RemoteOperationException: ERROR: NMO not setuid-root (Unix-only)) later on when managing the environment :
OEM_MgntPsPlugin_052
2. Do not forget to maintain the /etc/hosts according to the all environment. The /etc/hosts from OMS server should contains its own IP address, hostname and domain name, but also all the target servers’ references (IP, hostname and domain). Here’s the one I’m using on the OMS host server :
[oracle@ovmoem111 ~]$ more /etc/hosts
127.0.0.1       localhost.localdomain   localhost
192.168.1.160 ovmoem111.phoenix.nga ovmoem111
192.168.1.133   psovmdb.phoenix.nga psovmdb
192.168.1.143   psovmcrm.phoenix.nga psovmcrm
192.168.1.142   psovmcrmdb.phoenix.nga psovmcrmdb
192.168.1.145   psovmfscm.phoenix.nga psovmfscm
192.168.1.144   psovmfscmdb.phoenix.nga psovmfscmdb
192.168.1.147   psovmhcm.phoenix.nga psovmhcm
192.168.1.146   psovmhcmdb.phoenix.nga psovmhcmdb
192.168.1.148   psovmhcmdb2.phoenix.nga psovmhcmdb2
[oracle@ovmoem111 ~]$

The /etc/hosts file from target hosts must contains their own IP address, hostname and domain name as well as the ones of the OMS server. Here’s an example on one of the target host :
[root@psovmcrm ~]# more /etc/hosts
127.0.0.1       localhost.localdomain   localhost
::1     localhost6.localdomain6 localhost6
192.168.1.143   psovmcrm.phoenix.nga psovmcrm
192.168.1.160 ovmoem111.phoenix.nga ovmoem111
[root@psovmcrm ~]#

If not doing so, you can have the following error (not possible to configure target database for instance) :
OEM_InstAgent_020

2. The Peoplesoft plug-in for the Agent 
To be done only on App/Batch/PIA PSOVM servers.
To be able to manage the Peoplesoft environment from OEM Grid Control, the Agent must have the Peoplesoft plug-in. This is the same package used for the plug-in previously installed on OMS.
To be downloaded from https://edelivery.oracle.com (free registration), Product “Peoplesoft Enterprise”, Platform “Linux x86-64” (but the results do not differ, it does not matter for Peoplesoft packages), then choose the pack “Oracle Application Management Pack 1.0.0.2.0 for PeopleSoft Enterprise Media Pack v2” part number B60821-02.
Download the part number V22794-01 named “PeopleSoft Enterprise Environment Management Plug-in for Oracle Enterprise Manager (8.51.0.1.0)”. Despite the name, it is compatible on Peopletools 8.48 and above, moreover it is also compatible with OEM Grid Control 11gR1.

The Peoplesoft plug-in installation on Agent will be the easiest part, using the silent installation option as following.
Warning : the Peoplesoft plug-in for OEM Agent must be done whilst the psconfig.sh has not been previously invoked in the session (the PS environment must not be set), make it not happening.

Here I rename the psconfig.sh to avoid all confusion.
login as: root
root@192.168.1.143's password:
Last login: Wed Mar 30 11:19:04 2011 from 192.168.1.202
[root@psovmcrm ~]# su - psadm1
[psadm1@psovmcrm ~]$ cd $PS_HOME
[psadm1@psovmcrm tools]$ pwd
/opt/oracle/psft/pt/tools
[psadm1@psovmcrm tools]$ mv psconfig.sh _psconfig.sh
[psadm1@psovmcrm tools]$ exit
logout
[root@psovmcrm ~]# su - psadm1
-bash: /opt/oracle/psft/pt/tools/psconfig.sh: No such file or directory

From the Peoplesoft plug-in, unzip the file 5016553.zip (present under the downloaded Peoplesoft plug-in package) and copy it on your local server for installation (the target OMA host).
[psadm1@psovmcrm ~]$ cd /tmp
[psadm1@psovmcrm tmp]$ scp -r root@ovmoem111:/nfs/software/OracleCD/OEM/PeoplesoftPlugin/8.51/Disk1/5016553 .
root@ovmoem111's password:
<…snipped…>
[psadm1@psovmcrm tmp]$

You must set the ORACLE_HOME variable to the OMA_HOME and stop the Agent.
[psadm1@psovmcrm tmp]$ export ORACLE_HOME=/opt/oracle/psft/pt/oracle_agent/agent11g
[psadm1@psovmcrm tmp]$ cd $ORACLE_HOME/oui/bin
[psadm1@psovmcrm bin]$ cd ../../bin
[psadm1@psovmcrm bin]$ ./emctl stop agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent is Not Running
[psadm1@psovmcrm bin]$


Then, from the OUI Installer of Agent, run the silent installation.
[psadm1@psovmcrm bin]$ cd -
/opt/oracle/psft/pt/oracle_agent/agent11g/oui/bin
[psadm1@psovmcrm bin]$ ./runInstaller.sh -silent -ignoreSysPrereqs -waitForCompletion -responseFile /tmp/5016553/Disk1/response/psemagent.rsp -sourceLoc /tmp/5016553/Disk1/stage/products.xml -invPtrLoc $ORACLE_HOME/oraInst.loc ORACLE_HOME=$ORACLE_HOME
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 2055 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-04-01_07-56-41AM. Please wait ...Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

You can find the log of this install session at:
/home/psadm1/oraInventory/logs/installActions2011-04-01_07-56-41AM.log
.................................................................................................... 100% Done.

Loading Product Information
............................................................... 100% Done.

Analyzing dependencies
............................................................... 100% Done.

-----------------------------------------------------------------------------
Summary
Global Settings
    Source: /tmp/5016553/Disk1/stage/products.xml
    Oracle Home: /opt/oracle/psft/pt/oracle_agent/agent11g (agent11g1)
    Installation Type: Typical
Product Languages
   English
Space Requirements
   / Required 23MB (includes 7MB temporary) : Available 1.82GB
New Installations (1 products)
   PeopleSoft Environment Management Plug-in (Agent) 8.51.0.1.0
-----------------------------------------------------------------------------

Installation in progress (Friday, April 1, 2011 7:58:52 AM EDT)
...............................................................................................                                 95% Done.
Install successful

Linking in progress (Friday, April 1, 2011 7:59:03 AM EDT)
Link successful

Setup in progress (Friday, April 1, 2011 7:59:03 AM EDT)
Setup successful

End of install phases.(Friday, April 1, 2011 7:59:11 AM EDT)
Starting to execute configuration assistants
Configuration assistant "PeopleSoft Environment Management Agent Plug-in" succeeded
The installation of PeopleSoft Environment Management Plug-in (Agent) was successful.
Please check '/home/psadm1/oraInventory/logs/silentInstall2011-04-01_07-56-41AM.log' for more details.
[psadm1@psovmcrm bin]$

The Agent is restarted automatically, we can check it.
[psadm1@psovmcrm bin]$ cd -
/opt/oracle/psft/pt/oracle_agent/agent11g/bin
[psadm1@psovmcrm bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/oracle/psft/pt/oracle_agent/agent11g
Agent binaries    : /opt/oracle/psft/pt/oracle_agent/agent11g
Agent Process ID  : 1731
Parent Process ID : 1708
Agent URL         :
https://psovmcrm.phoenix.nga:3872/emd/main/
Repository URL    : https://ovmoem111.phoenix.nga:4900/em/upload
Started at        : 2011-04-01 07:59:47
Started by user   : psadm1
Last Reload       : 2011-04-01 07:59:47
Last successful upload                       : 2011-04-01 08:01:25
Total Megabytes of XML files uploaded so far :     0.43
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    52.60%
Last successful heartbeat to OMS             : 2011-04-01 08:05:27
---------------------------------------------------------------
Agent is Running and Ready
[psadm1@psovmcrm bin]$


Rename back to the original name the psconfig.sh for further connection.
[psadm1@psovmcrm bin]$ cd /opt/oracle/psft/pt/tools
[psadm1@psovmcrm tools]$ mv _psconfig.sh psconfig.sh
[psadm1@psovmcrm tools]$


From now, we should be able to add this Peoplesoft environment in the OEM Grid Control.
So far we have this :
OEM_InstPsPlugin_030
Nest step is then to add Peoplesoft target. To be continued here.