Page 1

Performance Tuning and Troubleshooting for Oracle OC4J

Jurijs Velikanovs 7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2


Topics: z z z z z z

Introduction Usage considerations OC4J Internals OC4J Troubleshooting OC4J Performance tuning BC4J


Introduction


Who am I? And what are my responsibilities? z

Who am I? I

am a DBA team leader  I’m neither a Java fan nor a Java Developer nor a Java Guru  My team has spent a lot of time researching and struggling to stabilize the production environments based on OC4J z

Responsibilities  To

support production and development environments  To ensure production availability  To troubleshoot production system  To provide all the necessary information to the developers or support organizations to solve a problem


How were we involved? z

z

z

Local Oracle Marketing team has pushed customers to demand the use of OC4J in their tenders’ conditions In 2002 the company management decided to switch the whole Development from Oracle Forms to OC4J/BC4J/JDeveloper technology We had  100

forms developers without any Java experience  A few Java fans  Oracle 9iAS R2 (9.0.2), later 9.0.3.1  In 2003 the first phase of the first application was deployed


Example of the Applications we are working with

z

Healthcare financial system  Enterprise

application  2+ years in the production and still under development  Oracle 9iAS 9.0.3.1 OC4J, Oracle 9.2.0.5 DB  JSP = 1448 files, CLASS = 3283 files  965 tables, 593 views, 251 types, 220 packages,  DB Size 50 GB, REDO 2-8 GB Daily  800 heavy users per day z

10x5 hour availability


Why is it so important? z

OC4J is the present strategic direction for Oracle  JDeveloper,

z

Forms, Reports, Discoverer, OEM, etc.

Difficult to troubleshoot  Complex

technology  Many components  Java on Java


Why is it so important? z

OC4J is a strategic direction for Oracle  JDeveloper,

z

Forms, Reports, Discoverer, OEM

Difficult to troubleshoot  Complex

technology  Many components  Java on Java z z z

There is not much information about this product Poor vendor support Many Oracle customers use it and have troubles


I will not tell you how to develop in Java …  I’m not going to tell you about  Web and EJB application technology  VMC  etc.

 The

technical layer is of great interest


I’m going to speak about OC4J internals … ONS

PORT: 3001

Thread

AJP Connection Listener

OC4JMonitor Thread

new req ApplicationServerThread

Thread

ApplicationServerThread

ApplicationServerThread ApplicationServerThread

Java native

ApplicationServerThread ...

AJP 1.3

ONet

ApplicationServerThread ApplicationServerThread

Memory Structures Application objects

3101

RMI RMIServer Server

3201

Http Session List

JMSServer

PORT: 3101

PORT: 3201

...

TaskManager

TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread

JDBC Connection pool

Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread

VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler


Usage considerations or how to avoid troubles


Oracle AS Architecture Considerations z

You may make the configuration complicated, with many components highly dependent on each other


Oracle AS Architecture Considerations BUT: z Keep it as simple as possible z At first you should be sure that the system works properly in a simple configuration z Make components as independent as possible z Don’t use the infrastructure if there is no strong need for it (i.e. Single-Signon, Discoverer, Clustering)


Oracle AS Typical Architecture DCM-daemon

OPMN

DMS-metrics DMS.JAR (java) statistics reflecting

PM, ONS (c) strat, stop, ping, notification

DCM (java) configuration management

DCM repository

opmn.xml

httpd.conf

HTTP

PORT: 80, 7780

WAN

OHS (c) Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server Child Server Child Server Child Server Child Server Child Server Child Server Child Child Server Server

JVM JVM(java) JMSServer JMSServer JMSServer Thread JMSServer Thread Thread Thread

AJP 1.3

PORT: 3001

Apache

server.xml

OC4J

AJP AJP Connection Connection Listener Listener

ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread

OC4JMonitor OC4JMonitor Thread Thread (dms)

TaskManager TaskManager TaskManager TaskManager

Java native Java native VM Thread VM Thread RMI RMIServer Server PORT: 3101

JMSServer PORT: 3201

Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread

VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler

DB


Installation Considerations z z

z z

Use the latest version of Oracle AS depending on your circumstances Use the latest versions of Oracle Forms, Reports, Discoverer, Portal. Install them in the OH separated from your custom development OC4J OH Install the latest patches for OS, JDK, OC4J, JDBC components Install the version of BC4J which was used by the Development. Don’t use the default version of BC4J/ADF/JHeadstart (See. JDeveloper install.html)


Operational Considerations z z

Keep running only the components you need Don’t startup in the production environment:  Oracle

Enterprise Manager 10g Application Server Control  OC4J_Demos component  “home” component z z z

Don’t deploy your applications to the “home” component Create your own component and use it to deploy and run your applications Use the opmnctl, dcmctl, dmstool utilities to maintain your AS environment


Deployment Considerations z

Get a commitment from the Development to use industry standards for the Application development 

Use EAR files to deploy any application changes  Use Ant tool (http://ant.apache.org) to make EAR/WAR files  In emergency cases use WAR files to deploy small changes to application z z z

Use undeployApplication/deployApplication combination rather than redeployApplication dcmctl command Use dcmctl saveInstance before deploying to backup current configuration Deploy BC4J.EAR to your OC4J component to keep “home” down


Versions Considerations Recommendations a dedicated test environment and verify all the changes there before implementing in production  If you have the automatic stresstesting scenarios, run them after each change in the test environment  Monitor the impact of changes and if the situation becomes worse rollback them

Application

 Have

APP Frame Work BC4J 9.0.3.10.7 OC4J

JDBC

c.c.c.x 9.0.3.1

c.c.c.x 9.2.0.5

JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise


Versions Considerations z

OS 

 

z

It is important to install the latest patches from an OS vendor (Kernel, Network, etc.) Usually the patches can be rollbacked easily The change of OS main release requires more testing and efforts than patching

JDK   



Install the latest versions of JDK within c.c. boundaries Use the JDK from Sun ww.JavaSoft.com To install it replace the OH/jdk directory. These changes can be rollbacked easily The change of c.c. version requires the Development involvement and more testing

Application APP Frame Work BC4J 9.0.3.10.7 OC4J

JDBC

c.c.c.x 9.0.3.1

c.c.c.x 9.2.0.5

JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise


Versions Considerations z

JDBC  

 z

Install the latest version of JDBC within c.c.c. version To install it replace OH/jdbc directory. These changes can be rollbacked easily It requires the minimal functionality testing

Application APP Frame Work BC4J 9.0.3.10.7

OC4J     

Can be easily upgraded within a c.c.c. version boundaries It requires the minimal functionality testing Follow the patch installation instructions (9iAS 9.0.3.1) There are 50+ one-off patches (*.class) The change of c.c.c. version requires the Development involvement and more testing

OC4J

JDBC

c.c.c.x 9.0.3.1

c.c.c.x 9.2.0.5

JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise


Versions Considerations z

BC4J  A BC4J version sets the requirements for OC4J version  It can’t be changed without Development involvement even within x.x.x.x. version boundaries  Often there are needs for application changes  It requires a lot of testing  New version can have negative impact to your application  The change of c.c.c. version requires more application code changes, testing, and other efforts

Application APP Frame Work BC4J 9.0.3.10.7 OC4J

JDBC

c.c.c.x 9.0.3.1

c.c.c.x 9.2.0.5

JDK c.c.x_x, 1.3.1_15 OS 2.4.9-e.38enterprise


Versions Considerations z

How to Determine the Versions 

OS: uname -r

$ uname -r 2.4.9-e.38enterprise



JRE: java –version

$ $ORACLE_HOME/jdk/bin/java -version java version "1.3.1_13" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03) Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)



OC4J: java -jar oc4j.jar –version

$ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/j2ee/home/oc4j.jar -version Oracle9iAS (9.0.3.1.0) Containers for J2EE (build 031214.BP1.2075)



JDBC: a program with a call to meta.getDriverVersion (location $OH/jdbc/lib/classes12dms.jar, Notes:73629.1/244074.1)

$ $ORACLE_HOME/jdk/bin/java JDBCVersion Oracle JDBC driver 9.2.0.5.0



BC4J: a program with a call to meta.getDriverVersion (location $BC4J/lib/bc4jct.jar)

$ $ORACLE_HOME/jdk/bin/java BC4JVersion 9.0.3.10.7


Other Considerations z

z z z

z

z

Have at least one senior level developer, who understands used technology well, is capable to make technical decisions and to lead the framework development, maintenance and troubleshooting Synchronize OC4J, BC4J versions with Development Use an Automatic testing tool for the application stresstesting A commitment from Oracle is desirable. If you can manage a connection to the Oracle JDeveloper development team to support your project, it will be an invaluable help during troubleshooting and decision making Have a skilled DBA for environments maintenance, monitoring and troubleshooting. The DBA provides all possible information to the development for application troubleshooting Have a monitoring framework


Before going to life

!!! Stresstest your application !!! (or ask for results)


OC4J Internals


OC4J Internals ONS

PORT: 3001

Thread

AJP Connection Listener

OC4JMonitor Thread

new req ApplicationServerThread

Thread

ApplicationServerThread

ApplicationServerThread ApplicationServerThread

Java native

ApplicationServerThread ...

AJP 1.3

ONet

ApplicationServerThread ApplicationServerThread

Memory Structures Application objects

3101

RMI RMIServer Server

3201

Http Session List

JMSServer

PORT: 3101

PORT: 3201

...

TaskManager

TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread

JDBC Connection pool

Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread

VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler


Processes (threads) within OC4J z

Java native threads 

VM Thread, VM Periodic Task Thread, Signal Dispatcher, Suspend Checker Thread, Finalizer, Reference Handler  Exist in any Java based application z

OC4J Listener - threads 

AJP/HttpConnectionListener, RMIServer (2), JMSServer  Accepting clients requests, assigning them to the worker threads  The configuration of listened ports is in opmn.xml file z

Set of processes for communicating with ONS 

OC4JMonitorThread, Thread(2)  These processes make connection to the OPMN immediately after the startup of OC4J  They send event messages to the OPMN to be broadcasted to subscribers


Processes (threads) within OC4J z

TaskManager 

Responsible for a periodic tasks executing (housekeeping)    



server.http.SessionTimeoutTask (drops expired session) server.ApplicationServerTask (clear expired threads) sql.DriverManagerXADataSource …

You can set an executing frequency in server.xml file taskmanager-granularity=1000 (default 1 sec)



Use the dmstool to monitor the statistics of the process dmstool -table oc4j_task

z

ApplicationServerThreads (Global Thread Pool) 

Execute users requests, run an application code, communicate with the Apache processes and the database  Initial count of the processes can be set in server.xml file global-thread-pool min=“200" max="200" queue="20“


OC4J Troubleshooting


Classification of Problem Areas OC4J JVM(java) PORT: 3001

4.

AJP JMSServer JMSServer Connection Listener

Apache

New Request as sig n

ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread TaskManager

1.

ApplicationServerThread

2.

3.

DB

ApplicationServerThread ApplicationServerThread ApplicationServerThread

JDK OS

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks) 2. Interface to the Apache processes 3. Interface to the database 4. 500 Internal Error


Possible Approaches & Information Sources z z

Java Full thread dump Other  Log

files  OS Monitoring commands  OS Tracing commands  Apache side information  Java code decompilation  Documentation, External searches and Metalink z

Troubleshooting information gathering (example)


Java “Full thread dump” z z

How to get a Full thread dump? The Full thread dump interpretation  Different

types of the threads within OC4J container  Different states of the threads  Some threads examples z z

Useful java runtime parameter Java DEADLOCK problem


How to get a “Full thread dump”? z Get

pid of the OC4J process

Use : ps, pstree or opmnadmin $ opmnadmin debug comp=pm HTTP/1.1 200 OK Content-Length: 677 Content-Type: text/html Response:

======== PM ======== PM requests processed: 5 PROCESS TABLE UID ---------456236 390700 521772 259628 194092 128556

PID ----13512 13511 15874 13415 13338 13281

FLAGS TYPE STATUS REF HTTP AJP RMI JMS -------- ------- ------- ------ ----- ----- ----- ----00000040 OC4J Alive 1 0 3003 3104 3204 00000040 OC4J Alive 1 0 3002 3103 3203 00000040 OC4J Alive 1 0 3001 3102 3202 00000040 OC4J Alive 1 0 3000 3101 3201 00000040 Apache Alive 1 7780 0 0 0 00000040 Generic Alive 1 0 0 0 0


How to get a “Full thread dump”? z

Find out the output file name (OH/opmn/logs/*) $ ps -fw --width 4000 -p 16114 UID PID PPID C STIME TTY TIME CMD oracle 15874 15902 0 23:40 ? 00:00:00 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java -Xms512M Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.8000 -Doracle.ons.indexid=PANDA.default_island.1 -DOPMN=true -jar oc4j.jar config /u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml -properties instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.8000#J2EE01.ap1.voava.lv#521772#PANDA#PANDA#default_island -ajp 3001 -rmi 3102 jms 3202 $ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}' PANDA.default_island.1

z

Send SIGQUIT (Quit from keyboard) signal $ kill -3 15874


How to get a “Full thread dump”? z

Check the output file

OH/opmn/logs/PANDA.default_island.1 Full thread dump: "Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor [0xa51ff000..0xa51ff89c] at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:427) at java.util.TimerThread.run(Timer.java:380) "Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c] "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479) "Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor [0xa55ff000..0xa55ff89c] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:415) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:253) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:213) at oracle.ons.SenderThread.run(SenderThread.java:81)


The “Full thread dump” Interpretation ===================================================== "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479)

===================================================== [OC4JMonitorThread] – name (type) of the thread OC4JMonitorThread.java Thread.currentThread().setName("OC4JMonitorThread");

[nid=0x3ef2] – Thread ID (in decimal = 16114) [waiting on monitor] – State of the thread at the snap time [oracle.ons.NotificationQueue.internalDequeue] – java stack


TYPES of the Threads OC4J Specific AJPConnectionListener [0.0.0.0/0.0.0.0:3001] 1 RMIServer [0.0.0.0/0.0.0.0:3102] count:1 1 RMIServer [0.0.0.0/0.0.0.0:3102] count:2 1 JMSServer 1 OC4JMonitorThread 1 TaskManager 1 Thread 5 ApplicationServerThread 195 JAVA GENERIC VM Thread 1 VM Periodic Task Thread 1 Reference Handler 1 Signal Dispatcher 1 Suspend Checker Thread 1 Finalizer 1 1


STATES of the Threads z

z

z

Waiting on monitor 

In most of cases it is an idle state of the thread. The thread is waiting for a task to execute or voluntary sleeps



All the available ApplicationServerThreads are in this state

Runnable 

The thread is executing application code or reading from the socket



All listeners must be in this state

Waiting for monitor entry 

Waiting to lock an object (other thread is holding the lock). This happens if two or more threads are trying to execute synchronized java code  None of threads must be in this state for a long time  If the system thread stays in this state for a long time you are in trouble  If many threads are in this state you have the situation of serialization


Threads Examples 1. OC4JMonitorThread is waiting for a work. Usual state for most of the threads "OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c] at java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224) at oracle.ons.Subscriber.receive(Subscriber.java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419) at java.lang.Thread.run(Thread.java:479)

2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task "ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c] at java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.poll(BoundedBuffer.java:170) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:768) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:802) at java.lang.Thread.run(Thread.java:479)

3. "Thread-9" is reading from the socket (connected to the ONS process, port 7100). This thread most of time is waiting for a data to be read from the socket "Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:267) at oracle.ons.InputBuffer.getNextString(InputBuffer.java:222) at oracle.ons.ReceiverThread.run(ReceiverThread.java:225)


Threads Examples 4. "AJPConnectionListener� is waiting for new connections from the Apache processes "AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable [0xb19ff000..0xb19ff89c] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463) at java.net.ServerSocket.implAccept(ServerSocket.java:238) at java.net.ServerSocket.accept(ServerSocket.java:217) at com.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:58) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)

5. "ApplicationServerThread-5" is reading the request from the Apache process "ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at com.evermind.io.SingleReadBufferInputStream.readChunk(SingleReadBufferInputStream.java:116) at com.evermind.io.SingleReadBufferInputStream.read(SingleReadBufferInputStream.java:35) at com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:345) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:131) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)

Apache


Threads Examples 6. "ApplicationServerThread-71" is executing database query, waiting for the results to be retrieved "ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.DataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986) at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2484) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2850) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:537) at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:1522) at oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:1530) at oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:1482) at oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:4596) at _common._comp._CMN__DataScrollerComponent._jspService(_CMN__DataScrollerComponent.java:115) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:119) at com.evermind.server.http.GetParametersRequestDispatcher.include(GetParametersRequestDispatcher.java:95) at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:280) at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.java:70) at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.java:249) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)

DB


Useful Java Runtime Parameter z

-XX:+JavaMonitorsInStackTrace (opmn.xml) <java-option value="-server -Xms512M -Xmx512M -verbose:gc -XX:+JavaMonitorsInStackTrace"/>

z

The parameter wasnâ&#x20AC;&#x2122;t set

"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626) at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189) at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.java:150) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) ...

z

The parameter was set

"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376) - waiting to lock <460e0338> (a oracle.jbo.JboSyncLock) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626) at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189) - locked <45f722d0> (a lv.vovaa.panda.common.datatags.DataSourceCmn) at _nsk._nsk001._AP3220F__Tickets__browse._jspService(_AP3220F__Tickets__browse.java:105) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) ...

OC4J


Useful Java Runtime Parameter z

It is possible to determine the blocking thread easily

"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) ... at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:549) at oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:2600) - locked <460e0338> (a oracle.jbo.JboSyncLock) at lv.vovaa.panda.common.pbo.PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.java:324) at lv.vovaa.panda.common.pbo.PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.java:309) at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:521) ...

z

Then we can locate the source code for troubleshooting purposes

ViewObjectImpl.java

protected void executeQueryForCollection(Object obj, Object aobj[], int i) { synchronized(getSyncLock()) { int j = 0; if(InstrumentedEvent.isActive) j = InstrumentedEvent.startEvent(EventGroup.EXECUTE_QUERY, "ViewObject executeQueryForCollection " + getName()); ((QueryCollection)obj).executeQuery(aobj, i); if(InstrumentedEvent.isActive) InstrumentedEvent.endEvent(j); } }


Java DEADLOCK z

Information from the OC4J process “Full thread dump”

Full thread dump: "ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor entry [0xa19ff000..0xa19ff89c] ... ... FOUND A JAVA LEVEL DEADLOCK: ---------------------------"ApplicationServerThread-58": waiting to lock monitor 0x80b9804 (object 0x4595b408, a java.lang.Object), which is locked by "ApplicationServerThread-51" "ApplicationServerThread-51": waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession), which is locked by "ApplicationServerThread-58" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: -----------------------------------------------Java Stack for "ApplicationServerThread-58": ========== at oracle.jbo.http.HttpContainer.setValue(HttpContainer.java:352) - waiting to lock <4595b408> (a java.lang.Object)

z z z

JDK/JRE (before 1.5.X) is unable to solve DEADLOCK situations OPMN doesn’t detect such kind of situations You need to take action from outside 

Gather troubleshooting information and restart OC4J process


Classification of Problem Areas OC4J JVM(java) PORT: 3001

4.

AJP JMSServer JMSServer Connection Listener

Apache

New Request as sig n

ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread ApplicationServerThread TaskManager

1.

ApplicationServerThread

2.

3.

DB

ApplicationServerThread ApplicationServerThread ApplicationServerThread

JDK OS

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks) 2. Interface to the Apache processes 3. Interface to the database 4. 500 Internal Error


Java “Full thread dump” (conclusion) z

It is an extremely useful source of information for understanding OC4J internals and troubleshooting

z

Take a few snaps to see dynamics

z

You need to analyze whole OC4J process “Full thread dump” to make conclusions

z

Java on Java. The ideal person for such analysis is the person who has a good knowledge of java, used technologies, application framework as well as application running on the container

z

DBA/Administrator should ensure all necessary information to be available to application troubleshooting person


Log Files z

OC4J output file  Located

at OH/opmn/log/<comp_name>.default_island.<proc_nu m> (example PANDA.default_island.1)  Store standard output of OC4J process (JAVA DUMP, task manager info, debug info -Dajp.io.debug=true)  It can be useful to add time stamp information * * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1

z

OC4J default-web-access.log  Located

at

OH/ j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log

 It

is an analogue of access_log for Apache


OS Monitoring Commands z

System wide: vmstat, iostat, uptime, pstree

z

Top resources consumers: top, ps -eo pcpu,pid,args | sort -k1n | tail -10

z

Particular process information

$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3 25.3 $ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ; done 1095372 258244 1095372 258244 1095372 258244 $ ps -wo args --width 4000 -p 6018


OS Tracing Commands z

strace - trace system calls and signals for particular process

$ strace â&#x20AC;&#x201C;s1000 -f -p 7495 read(19, $ strace â&#x20AC;&#x201C;s1000 â&#x20AC;&#x201C;f -p 27021 --- SIGSTOP (Stopped (signal)) --read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) send(104, "A", 1, 0) send(104, "B", 1, 0) send(104, "\0", 1, 0) send(104, "\3", 1, 0) send(104, "\6", 1, 0) send(104, "\37", 1, 0) send(104, "\372", 1, 0) read(104, "\0224\0\2\0\0", 4096) ...

z

= = = = = = = = = = = = = = = = =

6 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 6

pstack - print a stack trace of running processes


Apache Side Information z

Apache Server Status page

http://ap1.voava.lv:7780/server-status?refresh=15

z

log files  Â

Apache log files $OH/Apache/Apache/logs Sometimes information can mislead

access_log 192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET /PANDA/NSK/NSK003/NS0291F_Browse.jsp HTTP/1.1" 500 0 erorr_log [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling ajp13 response message, got an invalid value: -2. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDA_15 and it is not recoverable. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDA's service() to service the request.


Java Code Decompilation z

Extremely useful for troubleshooting purposes  

z

OC4J 

z



 

OH/dcm/lib/dcm.jar OH/dcm/lib/oc4j_deploy_tools.jar OH/dcm/lib/oc4j_remote_deploy.jar

DMS 

z

OH/jdbc/lib/classes12.zip OH/jdbc/lib/classes12dms.jar

DCM 

z

OH/j2ee/home/oc4j.jar

JDBC 

z

Unzip archive (jar/zip) Decompile *.class files (http://www.kpdus.com/jad.html)

OH/lib/dms.jar

BC4J   

OH/BC4J/redist/bc4j.ear OH/BC4J/jlib/* OH/BC4J/lib/*


Java Code Decompilation Usage Example z

Decompile all oc4j.jar classes z Grep through *.java files for “System.getProperty” 9.0.4 oc4j.jar … opmnPingInterval opmn.query.debug oracle.arraylist.deepCopy oracle.aurora.jem.scheduling.threads oracle.dms.console.DMSConsole oracle.dms.gate oracle.dms.sensors oracle.dms.transtrace.ecidenabled oracle.home oracle.ias.jcache


Debugging Switches z

Debug switches are documented in

Containers for J2EE User's Guide 10g Release 2 (10.1.2) Part No. B14011-01 B Additional Information B.8 OC4J Command-Line Options and System Properties z

Debug information example (-Dajp.io.debug=true)

PANDA.default_island.1 1107387907360:ThrP[50] ThrQ[0] 1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0] AJP: Input request length is: 1 AJP: AJPOutputStream::write() of 1 AJP: AJPOutputStream::writeChunk(): 1 1107387908370:ThrP[50] ThrQ[0] 1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]

z

Some of available categories are : ajp, http, jms, rmi, jdbc


Documentation and Internet z z z

The information in 9.0.X documentation is very limited Oracle AS Documentation Library 10g Release 2 (10.1.2) Highly Recommended books 

Containers for J2EE User's Guide (144)  Containers for J2EE Standalone User's Guide (134)  Server Administrator's Guide (542)  OPMN Administrator's Guide (158)  DCM Administrator's Guide (116)  Server Performance Guide (234) z z z z z

10g Best Practices 10g (9.0.4) Part No. B12223-01 BC4J JDeveloper Online Help system. www.google.com http://otn.oracle.com http://metalink.oracle.com


Metalink Patches z

Look through available patches.

………………………………………………………………..


Metalink Patches z

Strongly recommended in case of 9.0.3 OC4J


Metalink Patches z

Monitor for new patches, which can be relative to your environment


Troubleshooting information gathering (example)


Performance tuning


Our Findings (Java Workers) z

Preset java workers threads for OC4J process

OH/j2ee/<comp_name>/config/server.xml <global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/>



Set min to high value. It allows OC4J not to create the additional threads on the fly  



You don’t have to take care on max, keepAlive and queue parameters then OC4J doesn’t need to manage queue

debug="true" – Allows to see statistics in the OC4J process standard output file 

It is useful as troubleshooting information as well. Record appears in the output file at the end of each Task manager cycle.

<PANDA.default_island.1> 1107390315893:ThrP[100] ThrQ[0] 1107390315893:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1107390316903:ThrP[100] ThrQ[0] 1107390316903:CxP[100] CxQ[0] - WebC[0] RmiC[0] ...


Our Findings (OC4J Processes) z

Use many OC4J processes without enabled state replication

OH/opmn/conf/opmn.xml <oc4j maxRetry="3" instanceName="PANDA" gid="PANDA" numProcs="1"> <config-file path="/u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml"/> <java-option value="-server -Xms512M -Xmx512M"/> <oc4j-option value="-properties "/> <port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/> <island id="default_island" numProcs="3"/> </oc4j>

z z z z z

Initially it was recommended by Oracle support while TAR resolution OC4J container is more stable on a lower users load mod_oc4j distributes requests through 3 OC4J identical containers 1/3 of users load is handled by each OC4J process Do not use state replication   

z

It can add significant load on CPU It complicates configuration It can lead to opposite effect

Be careful  

You have to test this configuration for your application. It can lead to unpredictable DB connection behavior Memory/threads configuration is for each process of the OC4J instance


Our Findings (Memory) OH/opmn/conf/opmn.xml <oc4j … <java-option value="-server -Xms512M -Xmx512M"/> </oc4j> z Set max=min  

z

JVM doesn’t need to manage memory allocation dynamically You have reserved memory on the server

Java Garbage collector 

There is a lot of information around  



z

http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/ http://www.cons.org/cmucl/doc/gc-tuning.html

-verbose:gc

Java Profilers  

Add significant overhead to be used in production environments Extremely useful in test/dev environments for troubleshooting


Our Findings (Memory) z

Monitor memory usage in production   

grep log files for “out of memory” error use OS utilities use dmstool

$ dmstool -table JVM | grep 3001 | grep Memory /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue ap1.voava.lv*J2EE01-/u01/app/oracle/product $

495796 kbytes 495796.0 kbytes 522468.0 kbytes 524032 kbytes 524032.0 kbytes 524032.0 kbytes


Our Findings (Stresstesting) z z

The best way to tune performance is a stresstesting According to our experience     

After Development had performed the performance testing, a lot of problems were solved An automatic testing tool has been used (Rational Robot) Different modules were tested Different load profiles were tested Testing was repeated several times until acceptable results were achieved


Response Time Monitoring by DMS

Apache OHS [c]

Send

Request B

OC4J mod_plsql handle Handle B

JVM [java]

ApplicationServerThread Read Resolve

Process B JSP Processing Servlet Processing

mod_oc4j handle

Process E Handle E Handle B Handle E

Request E

Recive

mod_perl mod_php ...

DMS Sensors

DMS Sensors

DMS-client DMS.JAR (java) statistics reflecting

DB


Response Time Breakdown Example $ dmstool -table ohs_server | grep time .../Apache/connection.time 48123608767 usecs .../Apache/request.time 15916131942 usecs .../Apache/handle.time 15913988395 usecs rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf $ $ dmstool -table ohs_module | grep time | sort -k2n | tail -2 .../Apache/Modules/mod_dms.c/handle.time 8081655 usecs .../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs $ $ dmstool -table oc4j_jspExec | grep time .../oc4j/PANDA/WEBs/PANDA/JSPs/processRequest.time 17 563 793 msecs $ $ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5 .../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0285F_searchForm.jsp/service.time 1 072 570 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0291F_Browse.jsp/service.time 2 942 655 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_Edit.jsp/service.time 3 151 184 msecs .../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2260F_Browse.jsp/service.time 4 847 408 msecs rhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf $


DMS Sensors Classification z

Informative sensors – reflect a configuration information and some system wide statistics   

z

Average performance sensors – reflect response time totals by modules, applications, etc.  

z

ohs_child, ohs_responses, JVM, oc4j_ear, oc4j_opmn, oc4j_task, JDBC_Driver , JDBC_Connection

ohs_server , ohs_module, ohs_virtualHost, oc4j_context, oc4j_web_module, oc4j_jspExec

Units performance sensors – reflect response time statistics for each unit  

oc4j_servlet oc4j_jsp(threadsafe=true)


Collecting and Reflecting DMS Statistics z z z

DMS Sensors statistics is incremental Make periodic snaps of statistics for further analysis Develop interface to access statistics easily Â

Graphical interface for system wide statistics

Â

Textual interface for units (JSP/Servlet) response time analysis


Collecting and Reflecting DMS Statistics SYS:PROD> @sp_jsp Completed Snapshots Snap Id ------530 531 532 549

Snap Started ------------------2005-02-03 12:33:26 2005-02-03 13:03:24 2005-02-03 13:33:27 2005-02-03 14:03:32

Specify the Begin and End Snapshot Ids ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enter value for begin_snap: 532 Enter value for end_snap: 549 Saved in c:\temp\532_549.log Snap Id Snap Time ------- -----------------Begin Snap: 532 2005-02-03 13:33:27 End Snap: 549 2005-02-03 14:03:32 Elapsed: 30.08 (mins) JSP ordered by Response time JSP Name Response Execution Average (sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ---------../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22 ../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99 ../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21 ../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11 ../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19 ../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56 ../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82 ../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77 ../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82 ../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16 ../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08 ../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41 ../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67 ../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41 ../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5 ../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16 ../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75 ../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09 ../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08 SYS:PROD>


Monitor All the Tiers of the System z

Do not forget to monitor  OS

and HW resources utilization  Network  Database


Instrument Your Code z

AskTom.oracle.com

http://asktom.oracle.com/pls/ask/f?p=4950:8:16675873436862412624::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12836314571537

â&#x20AC;Ś Another example is asktom.oracle.com. Just go there and click on any article from the home page. You'll see a URL similar to: http://asktom.oracle.com/pls/ask/f?p=...::NO:: If you simply type over that URL and replace the word NO with YES - you'll see the same page but with lots and lots of state/timing information dumped into it. â&#x20AC;Ś


BC4J


OC4J Internals & BC4J ONS

PORT: 3001

Thread

AJP Connection Listener

OC4JMonitor Thread

new req ApplicationServerThread

Thread

ApplicationServerThread

ApplicationServerThread ApplicationServerThread

Java native

ApplicationServerThread ...

AJP 1.3

ONet

ApplicationServerThread ApplicationServerThread

Memory Structures JBO Application Module Pool JBO Application Module Pool JBO Application Module Pool JBO Application Module Pool

JBO Connection pool Application objects JBO View Objects

3101

RMI RMIServer Server

3201

Http Session List

JMSServer

PORT: 3101

PORT: 3201

...

TaskManager

TaskManager JMSServer JMSServer JMSServer JMSServer Thread Thread

...

Java native VM Thread VM Thread Signal Dispatcher Signal Dispatcher Suspend Checker Suspend ThreadChecker Thread

VM Periodic Task VMThread Periodic Task Thread Finalizer Finalizer Reference Reference Handler Handler


Usage Considerations z z z z

Most information is available under JDeveloper cover Use development version of BC4J Deploy BC4J.EAR to your OC4J component (do not use home component) There is the BC4J monitoring page


Useful Information Sources 0. 1. 2. 3. 4.

Metalink Note:273350.1 BC4J configuration tuning tips Metalink Note:283476.1 BC4J (ADF BC) Configuration Tuning Tips Metalink Note:238634.1 JBO / BC4J Parameters And Defaults Metalink Note:231141.1 BC4J Row Spillover and tuning How to Performance Tune an ADF Business Components (ADF BC) Application September 23, 2004 Steve Anderson and Michael Gantman http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html 5. Understanding Application Module Pooling Concepts and Configuration Parameters Author: Steve Muench, BC4J Development Team Date: August 5, 2004 http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html 6. Getting Application Module Pool Statistics to Aid with Size Tuning By Steve Muench http://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html 7. Overview of Temporary Tables Created By BC4J An Oracle technical white paper April 2002 http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html 8. How To Support Dynamic JDBC Credentials An Oracle BC4J technote Version 3: April 23rd 2002 http://www.oracle.com/technology/products/jdev/howtos/bc4j/howto_dynamic_jdbc.html 9. BC4J/JClient Performance Study Written by Steve Muench, Oracle Corporation January, 2004 http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html 10. Dive into BC4J and ADF Tips and tricks from Steve Muench on using Oracle's J2EE frameworks, JDeveloper Java IDE, and XML technologies... http://radio.weblogs.com/0118231/


Performance Tuning Parameters jbo.pers.max.active.nodes=1000 jbo.pers.max.rows.per.node=1000 jbo.doconnectionpooling=true jbo.poolminavailablesize=0 jbo.poolmaxavailablesize=600 jbo.poolmonitorsleepinterval=600000 jbo.poolmaxinactiveage=600000 jbo.ampool.monitorsleepinterval=300000 jbo.ampool.maxinactiveage=600000

JBO Parameter lookup order: Set programmatically or configuration in bc4j.xcfg file) Applet tags -D flags (or oc4j.properties file) BC4J.properties file in current working directory of Java VM /oracle/jbo/BC4J.properties resource in the runtime class path /oracle/jbo/server/jboserver.properties resource (by default found inside bc4jmt.jar) /oracle/jbo/common/Diagnostic.properties resource (by default found inside bc4jmt.jar) System defined default as a fallback


Summary


Summary z z z z z z

Introduction Usage considerations OC4J Internals OC4J Troubleshooting OC4J Performance tuning BC4J


Q&A

Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (slides - long version)  
Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (slides - long version)  

Jurijs Velikanovs 7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2 Introduction Usage considerations OC4J Internals OC4J Troubleshooting...

Advertisement