I recently tried to set up SiteScope 9.5 to monitor Weblogic Portal. It was not a pleasant experience. There are posts in various blogs about setting up JMX based monitoring of Weblogic. Some of the information is older, inaccurate, or didn’t apply to my configuration. So I’m consolidating the relevant information to my situation together here.
Note: HP does not support monitoring Weblogic 10.3 with SiteScope 9.5. They recommend upgrading to SiteScope 10.10. I was able to monitor with the steps below, but it may not work in all cases. I had no problem running these JMX monitors for Weblogic 10.3 and Linux memory and CPU monitors together. But it is possible that the wlfullclient.jar will cause a problem with other monitors.
Here is my configuration.
Monitoring Server: Window Server 2003 SP2, SiteScope 9.5, Sun JDK 1.5.0_08-b03
Weblogic Server: Red Had 5 – 64 bit Kernel 2.6.18, Weblogic 10.3, JRockit 1.60_05 Build R27 – 32 bit, running 6 instances on 2 different servers
These are the steps I needed. Links to the posts that helped me out are at the bottom.
Create a User for Monitoring
This step isn’t required. You can use the sytem user if you like, but I prefer to set up a separate user for monitoring so the system user’s password can be modified without impacting monitoring. It is a better security practice, and can be helpful in troubleshooting issues as well. To set up the user, go to the Security Realms/myrealm/Users and Groups/Users page and create a new user. You need to also put the user in the “Monitors” Weblogic group so the user has access to read the MBeans.
Enable IIOP and set the Default IIOP Username / Password
The key point here is that if you don’t set the Default IIOP Username and Password you will get the error message “org.omg.CORBA.NO_PERMISSION” when you try to connect. You do not have to actually connect as this user when your monitoring session authenticates. Your monitoring user can be different, but this entry needs to be there, and it must be a valid username/password. I used the monitoring user that I set up in the first step. This is done by going to the Weblogic server configuration/Protocols/IIOP. Select to Enable IIOP, click Advanced, and set the “Default IIOP Username” and “Default IIOP Password”.
Upgrade the Java version to 1.6
In order to monitor Weblogic 10.3 that is running in a Java VM version 1.6, you must upgrade the JVM that is running Sitescope to 1.6. Sitescope 9.53 seems to run fine on a Sun 1.6 version of the JVM. After you download and install a 1.6 JVM, of course you have to update your JAVA_HOME and registry settings for Sitescope to get it to use the new JVM.
Put the wlfullclient.jar on the SiteScope’s Classpath
This was the biggest issue I was having. Weblogic 10.x uses additional JMX capabilities that are not provided by weblogic.jar. You MUST use wlfullclient.jar to monitor Weblogic 10.x as discussed here http://download-llnw.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html#wp1077742. If you don’t have the wlfullclient.jar, SiteScope will still connect to Weblogic, and it appears to be working. But, you won’t get the list of monitors, and the log will show the error: “java.rmi.MarshalException: CORBA MARSHAL 1398079745 Maybe; nested exception is: org.omg.CORBA.MARSHAL: vmcid: SUN minor code: 257 completed: Maybe”. You can modify the SiteScope classpath and put the jar file on it, or you can just put the wlfullclient.jar in the C:SiteScopejavalibext directory. Either approach should work.
Modify SiteScope Startup Parameters
If you are having CORBA timeout errors in the log, you can edit the registry entry “HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/SiteScope/ServiceParam”. That is the SiteScope startup command line. You can increase the CORBA timeout settings by putting “-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:60000:500:10″ in the command line. Connection timeout errors can be misleading however. I was seeing them in the log even though the real issue was that I didn’t have the wlfullclient.jar file in the classpath. Weblogic eventually closed the connection because it didn’t understand the request, and SiteScope logged it as a connection timeout. If you see timeout errors, make the change above. If you are still seeing them, you probably have some other issue.
You can also increase the JVM heap size in the registry entry if you are seeing out of memory errors when pulling back the list of counters.
Create a JMX Monitor
Weblogic 9 and up use JMX for monitoring. You cannot use the Weblogic SiteScope monitor. You need to set up a JMX SiteScope monitor.
Managed Instance JMX URL: service:jmx:rmi:///jndi/iiop://:/weblogic.management.mbeanservers.runtime
Admin Instance JMX URL: service:jmx:rmi:///jndi/iiop://:/weblogic.management.mbeanservers.domainruntime
The can be a DNS name or the ip address. One suggestion on the web is that you have to use the Machine Name that is configured in weblogic. I didn’t find that to be the case. Just using the ip address or the host name worked fine. The port can be just the regular port number that the instance is listening on. Some suggestions of using command line options to have weblogic listen for JMX on a separate port are not needed. You can just use the normal listen port.
The Domain Runtime MBean is only available on the Admin instance. Oracle recommends monitoring the managed instances by connecting only to the Admin instance and using the Domain Runtime MBean. It gives you a single place to go to configure all of your monitors. I had trouble with the CORBA connection timing out when it was trying to pull back all of the MBean data for the Domain MBean, but was eventually able to get over that error by increasing the second value in the CORBA timeout string above from 60000 to 180000 and reducing the last value from 10 to 1. After solving that issue, I then got “ERROR – Failed to get class property details for property: availableCountersHierarchical in monitor type: JMXMonitor” in the SiteScope error log. Doing a little research on-line, it sounds like some of the data coming back from the Domain MBean isn’t being understood by SiteScope, so it just quits. Rather than continue trying to get monitoring from the Domain MBean working, I just set up a monitor for each managed instance separately. I was able to get JConsole to query the data from the Domain MBean on the Admin instance, so connectivity and permissions were fine. SiteScope just wasn’t able to work with the data coming back.
Select Your Counters
When you click on the counter button, the list of counters should return fairly quickly. I had some out of memory conditions at times using the previous max memory setting on SiteScope when clicking on “Get Counters”. The instance seemed to recover and display them after clicking again, but I increased the JVM memory to alleviate that issue. Some of the counters we are using are listed below. Of course would be the name of your managed server.
java.lang/Memory/HeapMemoryUsage/used
java.lang/Threading/ThreadCount
com.bea//portalDataSource/JDBCConnectionPoolRuntime/ActiveConnectionsCurrentCount
com.bea//portalDataSource/JDBCConnectionPoolRuntime/ActiveConnectionsHighCount
com.bea//portalDataSource/JDBCConnectionPoolRuntime/WaitingForConnectionHighCount
com.bea//portalDataSource/JDBCConnectionPoolRuntime/WaitSecondsHighCount
com.bea//ServerRuntime/OpenSocketsCurrentCount
com.bea//_/bea_wls_internal/bea_wls_internal/WebAppComponentRuntime/OpenSessionsCurrentCount
You’re Done
The monitors should work at this point
Other Monitoring Tools
These same basic steps can be used to monitor Weblogic 10.x with any JMX monitoring tool. JConsole is much better than sitescope or the Weblogic Admin console for real time monitoring during performance tests. SiteScope is good to monitor a few key health statistics on an on-going basis, but it isn’t granular enough or as useful for detailed monitoring during tuning or troubleshooting efforts.
These are the posts that helped me get things working.
General steps – http://www.performanceengineer.com/blog/monitoring-weblogic-using-jmx/
Key information about WL 10.3 –http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1268846849907+28353475&threadId=1315741
JConsole info – http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
wlfullclient.jar info – http://download-llnw.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html#wp1077742
The comment from Kai Ellinger on this post was the most useful. A proxy server shouldn’t be needed –http://blogs.oracle.com/jamesbayer/2009/03/workaround_for_using_jconsole.html
Bug report on IIOP problems between jdk version. http://forums.sun.com/thread.jspa?threadID=5259072
Troubleshooting
We recently had problems when using Sun jdk 1.6.0 communicating with jrockit 1.5.0 running Weblogic 10.2. We were getting the CORBA error below. This appears to be a bug in IIOP with Sun JDKs as discussed in this bug report http://forums.sun.com/thread.jspa?threadID=5259072. Using Sun jdk 1.5 to communicate with the jrockit jdk 1.5 worked fine.Jun 22, 2010 2:13:03 PM com.sun.corba.se.impl.io.InputStreamHook throwOptionalDataIncompatibleException WARNING: "IOP00800008: (MARSHAL) Not enough space left in current chunk" org.omg.CORBA.MARSHAL: vmcid: OMG minor code: 8 completed: No at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2709) at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2731) at com.sun.corba.se.impl.io.InputStreamHook.throwOptionalDataIncompatibleException(InputStreamHook.java:348) at com.sun.corba.se.impl.io.InputStreamHook$InReadObjectPastDefaultsRemoteDidNotUseWOState.readData(InputStreamHook.java:342) at com.sun.corba.se.impl.io.IIOPInputStream.read(IIOPInputStream.java:655) at javax.management.MBeanInfo.readObject(MBeanInfo.java:671) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Some Tips
Some of the things I ran into are listed below.
Enabling Anonymous Admin Lookup wasn’t needed. Turning this on didn’t hurt anything, but it wasn’t required.
Creating a separate port for JMX to communicate on. This would probably work, but you don’t need a separate port. You can just use the normal port that weblogic is listening on.
Using the Machine Name in the JMX url string. This didn’t seem to matter. Using the ip address worked fine as well.
When using JConsole, if you are using Windows and your JDK is installed under c:program filesJava…, when you set up your classpath you need to put it in double quotes (“). The space between the words program and files causes a problem and you will get a class not found error.
I did not need to set “Use Authorization Providers to Protect JMX Access”. This is because I added the monitoring user I created to the “Monitors” Weblogic security group. If you connect as a user that isn’t in one of the default Weblogic groups that has MBean access (Admin, Deployer, Operator, and Monitor) then you would need to enable this and set up rules to allow your user to access the MBeans
Note: HP does not support monitoring Weblogic 10.3 with SiteScope 9.5. They recommend upgrading to SiteScope 10.10. I was able to monitor with the steps below, but it may not work in all cases. I had no problem running these JMX monitors for Weblogic 10.3 and Linux memory and CPU monitors together. But it is possible that the wlfullclient.jar will cause a problem with other monitors.
Here is my configuration.
Monitoring Server: Window Server 2003 SP2, SiteScope 9.5, Sun JDK 1.5.0_08-b03
Weblogic Server: Red Had 5 – 64 bit Kernel 2.6.18, Weblogic 10.3, JRockit 1.60_05 Build R27 – 32 bit, running 6 instances on 2 different servers
These are the steps I needed. Links to the posts that helped me out are at the bottom.
Create a User for Monitoring
This step isn’t required. You can use the sytem user if you like, but I prefer to set up a separate user for monitoring so the system user’s password can be modified without impacting monitoring. It is a better security practice, and can be helpful in troubleshooting issues as well. To set up the user, go to the Security Realms/myrealm/Users and Groups/Users page and create a new user. You need to also put the user in the “Monitors” Weblogic group so the user has access to read the MBeans.
Enable IIOP and set the Default IIOP Username / Password
The key point here is that if you don’t set the Default IIOP Username and Password you will get the error message “org.omg.CORBA.NO_PERMISSION” when you try to connect. You do not have to actually connect as this user when your monitoring session authenticates. Your monitoring user can be different, but this entry needs to be there, and it must be a valid username/password. I used the monitoring user that I set up in the first step. This is done by going to the Weblogic server configuration/Protocols/IIOP. Select to Enable IIOP, click Advanced, and set the “Default IIOP Username” and “Default IIOP Password”.
Upgrade the Java version to 1.6
In order to monitor Weblogic 10.3 that is running in a Java VM version 1.6, you must upgrade the JVM that is running Sitescope to 1.6. Sitescope 9.53 seems to run fine on a Sun 1.6 version of the JVM. After you download and install a 1.6 JVM, of course you have to update your JAVA_HOME and registry settings for Sitescope to get it to use the new JVM.
Put the wlfullclient.jar on the SiteScope’s Classpath
This was the biggest issue I was having. Weblogic 10.x uses additional JMX capabilities that are not provided by weblogic.jar. You MUST use wlfullclient.jar to monitor Weblogic 10.x as discussed here http://download-llnw.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html#wp1077742. If you don’t have the wlfullclient.jar, SiteScope will still connect to Weblogic, and it appears to be working. But, you won’t get the list of monitors, and the log will show the error: “java.rmi.MarshalException: CORBA MARSHAL 1398079745 Maybe; nested exception is: org.omg.CORBA.MARSHAL: vmcid: SUN minor code: 257 completed: Maybe”. You can modify the SiteScope classpath and put the jar file on it, or you can just put the wlfullclient.jar in the C:SiteScopejavalibext directory. Either approach should work.
Modify SiteScope Startup Parameters
If you are having CORBA timeout errors in the log, you can edit the registry entry “HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/SiteScope/ServiceParam”. That is the SiteScope startup command line. You can increase the CORBA timeout settings by putting “-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:60000:500:10″ in the command line. Connection timeout errors can be misleading however. I was seeing them in the log even though the real issue was that I didn’t have the wlfullclient.jar file in the classpath. Weblogic eventually closed the connection because it didn’t understand the request, and SiteScope logged it as a connection timeout. If you see timeout errors, make the change above. If you are still seeing them, you probably have some other issue.
You can also increase the JVM heap size in the registry entry if you are seeing out of memory errors when pulling back the list of counters.
Create a JMX Monitor
Weblogic 9 and up use JMX for monitoring. You cannot use the Weblogic SiteScope monitor. You need to set up a JMX SiteScope monitor.
Managed Instance JMX URL: service:jmx:rmi:///jndi/iiop://
Admin Instance JMX URL: service:jmx:rmi:///jndi/iiop://
The
The Domain Runtime MBean is only available on the Admin instance. Oracle recommends monitoring the managed instances by connecting only to the Admin instance and using the Domain Runtime MBean. It gives you a single place to go to configure all of your monitors. I had trouble with the CORBA connection timing out when it was trying to pull back all of the MBean data for the Domain MBean, but was eventually able to get over that error by increasing the second value in the CORBA timeout string above from 60000 to 180000 and reducing the last value from 10 to 1. After solving that issue, I then got “ERROR – Failed to get class property details for property: availableCountersHierarchical in monitor type: JMXMonitor” in the SiteScope error log. Doing a little research on-line, it sounds like some of the data coming back from the Domain MBean isn’t being understood by SiteScope, so it just quits. Rather than continue trying to get monitoring from the Domain MBean working, I just set up a monitor for each managed instance separately. I was able to get JConsole to query the data from the Domain MBean on the Admin instance, so connectivity and permissions were fine. SiteScope just wasn’t able to work with the data coming back.
Select Your Counters
When you click on the counter button, the list of counters should return fairly quickly. I had some out of memory conditions at times using the previous max memory setting on SiteScope when clicking on “Get Counters”. The instance seemed to recover and display them after clicking again, but I increased the JVM memory to alleviate that issue. Some of the counters we are using are listed below. Of course
java.lang/Memory/HeapMemoryUsage/used
java.lang/Threading/ThreadCount
com.bea/
com.bea/
com.bea/
com.bea/
com.bea/
com.bea/
You’re Done
The monitors should work at this point
Other Monitoring Tools
These same basic steps can be used to monitor Weblogic 10.x with any JMX monitoring tool. JConsole is much better than sitescope or the Weblogic Admin console for real time monitoring during performance tests. SiteScope is good to monitor a few key health statistics on an on-going basis, but it isn’t granular enough or as useful for detailed monitoring during tuning or troubleshooting efforts.
These are the posts that helped me get things working.
General steps – http://www.performanceengineer.com/blog/monitoring-weblogic-using-jmx/
Key information about WL 10.3 –http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1268846849907+28353475&threadId=1315741
JConsole info – http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
wlfullclient.jar info – http://download-llnw.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html#wp1077742
The comment from Kai Ellinger on this post was the most useful. A proxy server shouldn’t be needed –http://blogs.oracle.com/jamesbayer/2009/03/workaround_for_using_jconsole.html
Bug report on IIOP problems between jdk version. http://forums.sun.com/thread.jspa?threadID=5259072
Troubleshooting
We recently had problems when using Sun jdk 1.6.0 communicating with jrockit 1.5.0 running Weblogic 10.2. We were getting the CORBA error below. This appears to be a bug in IIOP with Sun JDKs as discussed in this bug report http://forums.sun.com/thread.jspa?threadID=5259072. Using Sun jdk 1.5 to communicate with the jrockit jdk 1.5 worked fine.Jun 22, 2010 2:13:03 PM com.sun.corba.se.impl.io.InputStreamHook throwOptionalDataIncompatibleException WARNING: "IOP00800008: (MARSHAL) Not enough space left in current chunk" org.omg.CORBA.MARSHAL: vmcid: OMG minor code: 8 completed: No at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2709) at com.sun.corba.se.impl.logging.OMGSystemException.rmiiiopOptionalDataIncompatible2(OMGSystemException.java:2731) at com.sun.corba.se.impl.io.InputStreamHook.throwOptionalDataIncompatibleException(InputStreamHook.java:348) at com.sun.corba.se.impl.io.InputStreamHook$InReadObjectPastDefaultsRemoteDidNotUseWOState.readData(InputStreamHook.java:342) at com.sun.corba.se.impl.io.IIOPInputStream.read(IIOPInputStream.java:655) at javax.management.MBeanInfo.readObject(MBeanInfo.java:671) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Some Tips
Some of the things I ran into are listed below.
Enabling Anonymous Admin Lookup wasn’t needed. Turning this on didn’t hurt anything, but it wasn’t required.
Creating a separate port for JMX to communicate on. This would probably work, but you don’t need a separate port. You can just use the normal port that weblogic is listening on.
Using the Machine Name in the JMX url string. This didn’t seem to matter. Using the ip address worked fine as well.
When using JConsole, if you are using Windows and your JDK is installed under c:program filesJava…, when you set up your classpath you need to put it in double quotes (“). The space between the words program and files causes a problem and you will get a class not found error.
I did not need to set “Use Authorization Providers to Protect JMX Access”. This is because I added the monitoring user I created to the “Monitors” Weblogic security group. If you connect as a user that isn’t in one of the default Weblogic groups that has MBean access (Admin, Deployer, Operator, and Monitor) then you would need to enable this and set up rules to allow your user to access the MBeans
No comments:
Post a Comment