TIP 80 : AS10g maintenance/upgrade experience

If you are using Oracle application server 10gR2, you may notice that oracle does not provide any CPU patch for the default 10gR2 release until AS environment is patched up to higher version.
In other words, no CPU patch is realesed for 10.1.2.0.2 application server (default AS 10g R2 release). Regarding to metalink note 420061.1 , all application server should be patched up to 10.1.2.2 which all new CPU patches are based on.
As the result, I went throght upgrading application server from 10.1.2.0.2 to 10.1.2.2 for a client.

Here is my experience in this rocky road.

Client AS topology
=====================
A client is using 10gR2 application server on AIX 64bit.
Metarepository database is on 10gR2 database (created by Repca) and is on the same box of infrastructure tier.
Mid tier is on separate box with BI-Form full install and SSL-Enabled.

Patches
===============
Regarding to metalink note 415222.1, the following is the roadmap of upgrade to new 10.1.2.2 which would be more or less valid for similar AS environment



1. Apply 4960210 to Mid tier. (Software update)
2. Apply 4960210 to Infra tier. (Software update)
3. Apply 4960210 from mid tier. (Metarepository update)
4. Apply 5861907 to mid tier. (First thing right after upgrade)
5. Apply 5861907 to infra tier. (First thing right after upgrade)
6. Apply 5983475 to mid tier. (Web cache fix) It might fail with OPatch detects your platform as 23 while this patch <5983475> supports platforms: 212 (AIX-Based Systems (64-bit) 5L)Solution : Note => 427295.1
7. Apply 2617419 to mid tier. (Update Opatch)
8. Apply 2617419 to mid tier. (Update Opatch)
9. Apply 5488476 to mid tier (Form fix) <=========== Does not require.Form is higher version
10. Apply 5901894 to infra tier (April CPU Patch)
11. Apply 5922121 to mid tier (April CPU Patch)
12.Apply 5955554 to discoverer in mid tier (April CPU Patch)
13. Make sure JDK version after upgrade is compliant with TZ on both infra and mid tier.
14. Apply patch 4700543 ???????



Problems/resolutions
=======================

------------------------------ 1 ------------------------------------

JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
..JVMDG318: Heap dump file written to /o004/home/oracle/tmp/heapdump966718.1182525925.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to /o004/home/oracle/tmp/javacore966718.1182525930.txt
JVMDG274: Dump Handler has Processed OutOfMemory


Memory requirement should be the same as base install requirement for 10.1.2.X.
Each AS should have 1.5GB memory space available and 1.5GB swap space.
Also as a resolution, OC4J heap size can be increased.

--------------------------------- 2 ------------------------------------------
No progress in OC4J Instance Configuration Assistant.

This stpes metarepository database and DCM tablespace are being updated.
If DCM tablespace has not enough free space and metarepository database is in resumable mode, the following error appeared in metarepository alert log

ORA-1691: unable to extend lobsegment DCM.SYS_LOB0000051867C00007$$ by 1024 in tablespace DCM
statement in resumable session 'User DCM(58), Session 493, Instance 1' was suspended due to ORA-01691: unable to extend lob segment DCM.SYS_LOB0000051867C00007$$ by 1024 in tablespace DCM

------------------------------- 3 -----------------------------------
Default Portal page is not accessible.

- Put metarepository tablespaces in autoextend. (DCM tablespace will grow during upgrade).
- Make sure to have at least 1GB free memory and 4GB swap space.
- Default portal page : web.xml.