TIP 83#: Download patch using wget

If you are using updates.oracle.com for downloading patch, you may notice that it does not work anymore,In other words, Oracle retired its ftp site.

Connected to updates.oraclegha.com.
421-*********************** Downtime Notice ************************
421-This service was retired as of November 06, 2009.

So what would be the best option ?

If you environment was setup to run any browser(mozilla,firefox,IE for Win), start it and directly login to metalink and down load the patch.
If you are in Non-Windows environment, sometimes it is hard and time consuming to setup browser for the first time, so the best way would be to use wget.The following command helps you to get the patch. (You need to know URL which you can get it from metalink.

wget --no-check-certificate --http-user 'username' --http-passwd 'password' 'url' -O outputfile_name

Note : do not forget to put URL between ' not to let it misinterpreted.

TIP 82# : Browser craches when Oracle Forms is accessed

If you use internet explorer (Any version) and it crashes at time of accessing Oracle Forms, the following step could resolve the issue.

- Disable 3rd party browser extension (Tools -> Internet Options -> Advanced -> Browsing)
- Deinstall all Jinit versions. (Optional but preferred)
- Cleanup IE cache
- Close IE broswer
- Open IE and access URL

I have tested this with AS 10R2 ( and IE 6 and IE7 SP2.

TIP 81# : Oracle Forms and new Verisign certificate

After renewing Verisign certificate, you may see an issue with Oracle Forms when it is accessed. I have seen it couple weeks ago after a client renewed its Verisign certificate for Oracle AS : Client could access web server via SSL (e.g. https://servername:port/ was reachable) while accessing the Form server ended with Java exception and SSL handshake failed error message.(e.g. https//servername:port/forms/frmservlet?config=jpi)
I was noticed that Verisign has introduced a two-tier CA hierarchy for Standard SSL Certificates (Called chained cetrtificate sometimes) which changed the old way of having only a root certificate. With this method, Verisign provides Root certificate and also intermediate certiificate.It is interesting to know that Verisign has not been issued any ceritificate since Oct2008 in the old fashion.

Unfortunately, the latest Oracle Jinitiator (despite metalink 456658.1) can not handle new Verisign fashion and if Forms server uses Jinitiator, you may see Java exception and Handshake failure when Forms is accessed. Jinitiator and later (at time of writing this blog, the latest is can not handle the latest intermediate since Verisign keeps changing the intermediate certificate and as Jinitiator support is ended by Jan 31th,2010 (https://support.oracle.com/CSP/main/article?cmd=show&id=761159.1&type=NOT), it does not seem Oracle tries to catch up with the Verisign change.

Based on the environment and diversity of clients, I do recommend the following options :

Option 1 :
Migrate from Jinitiator to Java Plug-in (1.5)
Option 2 :
Migrate to at least Jinitiator
Copy intermediate file to cretdb.txt on each client box

(File is located on {Jinit install folder}\security\lib. (Please be informed that only upgrading jinitiator to the latest version may not work).

TIP 80 : AS10g maintenance/upgrade experience

If you are using Oracle application server 10gR2, you may notice that oracle does not provide any CPU patch for the default 10gR2 release until AS environment is patched up to higher version.
In other words, no CPU patch is realesed for application server (default AS 10g R2 release). Regarding to metalink note 420061.1 , all application server should be patched up to which all new CPU patches are based on.
As the result, I went throght upgrading application server from to for a client.

Here is my experience in this rocky road.

Client AS topology
A client is using 10gR2 application server on AIX 64bit.
Metarepository database is on 10gR2 database (created by Repca) and is on the same box of infrastructure tier.
Mid tier is on separate box with BI-Form full install and SSL-Enabled.

Regarding to metalink note 415222.1, the following is the roadmap of upgrade to new which would be more or less valid for similar AS environment

1. Apply 4960210 to Mid tier. (Software update)
2. Apply 4960210 to Infra tier. (Software update)
3. Apply 4960210 from mid tier. (Metarepository update)
4. Apply 5861907 to mid tier. (First thing right after upgrade)
5. Apply 5861907 to infra tier. (First thing right after upgrade)
6. Apply 5983475 to mid tier. (Web cache fix) It might fail with OPatch detects your platform as 23 while this patch <5983475> supports platforms: 212 (AIX-Based Systems (64-bit) 5L)Solution : Note => 427295.1
7. Apply 2617419 to mid tier. (Update Opatch)
8. Apply 2617419 to mid tier. (Update Opatch)
9. Apply 5488476 to mid tier (Form fix) <=========== Does not require.Form is higher version
10. Apply 5901894 to infra tier (April CPU Patch)
11. Apply 5922121 to mid tier (April CPU Patch)
12.Apply 5955554 to discoverer in mid tier (April CPU Patch)
13. Make sure JDK version after upgrade is compliant with TZ on both infra and mid tier.
14. Apply patch 4700543 ???????


------------------------------ 1 ------------------------------------

JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
..JVMDG318: Heap dump file written to /o004/home/oracle/tmp/heapdump966718.1182525925.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to /o004/home/oracle/tmp/javacore966718.1182525930.txt
JVMDG274: Dump Handler has Processed OutOfMemory

Memory requirement should be the same as base install requirement for 10.1.2.X.
Each AS should have 1.5GB memory space available and 1.5GB swap space.
Also as a resolution, OC4J heap size can be increased.

--------------------------------- 2 ------------------------------------------
No progress in OC4J Instance Configuration Assistant.

This stpes metarepository database and DCM tablespace are being updated.
If DCM tablespace has not enough free space and metarepository database is in resumable mode, the following error appeared in metarepository alert log

ORA-1691: unable to extend lobsegment DCM.SYS_LOB0000051867C00007$$ by 1024 in tablespace DCM
statement in resumable session 'User DCM(58), Session 493, Instance 1' was suspended due to ORA-01691: unable to extend lob segment DCM.SYS_LOB0000051867C00007$$ by 1024 in tablespace DCM

------------------------------- 3 -----------------------------------
Default Portal page is not accessible.

- Put metarepository tablespaces in autoextend. (DCM tablespace will grow during upgrade).
- Make sure to have at least 1GB free memory and 4GB swap space.
- Default portal page : web.xml.

TIP 79 : Simple and effective backup script.

How many times you have seen different backup script with different commands ? Have you ever asked if all consider Oracle best practices and what is pros and cons of each ?
Here, I am trying to focus on some Oracle best recommendations in terms of backup/recovery and then at the end represent a sample RMAN backup script which takes into account all those recommendations

Recommendations :

- Check logical corruption to make sure backup is good.
- Put full database backup as incremental level 0 so it is considered in incremental backup/recovery scenario.
- Make sure that backup pieces have time-stamp to overcome look-up performance issue when recovery catalog is backup.
- Make sure to have each datafile in a single backup piece so for partial recovery RMAN goes through only one piece.
- Make sure to have unique name for each backup piece so RMAN does not overwrite backup pieces if for any reason backup is taken more often.
- Tag backups properly to make searching them and restoring them easier.
- Keep DBID in controlfile backup piece in order to save time to find it when it is required.
- Take current controlfile backup although autobackup is ON,that way it guarantees that always controlfile backup is there.

Based on the above, the following is the standard template that covers all features.

allocate channel t1 type disk;
BACKUP AS COMPRESSED BACKUPSET check logical INCREMENTAL LEVEL 0 DATABASE filesperset 1 plus archivelog format '/mnt/u05/backuptest/backup_%d_set%s_piece%p_copy%c_%T_%U' TAG = DB_BACK_FULL_DAILY ;
backup current controlfile format '/mnt/u05/backuptest/backup_controlfile_%d_DBID%I_%T_%U.ctl' TAG=CTL_BACK_FULL_DAILY;
backup spfile format '/mnt/u05/backuptest/backup_parameter_%d_DBID%I_%T_%U.ctl' TAG=PARAM_BACK_FULL_DAILY ;
delete obsolete;
release channel t1;

TIP #78:Database server gets freezed after increasing memory foot print.

I have a client which went through a switchover practice for the first time.
After switching over to a standby box, I realized a performance issue in terms of IO write.As the result, it was decided to bump up database memory footprint for SGA and PGA since the box has enough memory.
Changes were made on spfile and new SGA was bumped up to 24GB and PGA to 4GB from total 32GB. After bouncing database, database hung in nomount and after a minute,the box was totally freezed which did not allow connection anymore.
The following was reported in Alert log :

ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

According to Oracle (Metalink note 560309.1), This could be lack of memory or improper setting of swap. Since in my case, physical memory was enough, it turned out that the issue was because of improper setting of swap. Swap needs to be set at least 0.75 times of physical RAM when memory>8GB

In my case, swap was 16GB while physical memory was 32GB which explained the case.
This issue could happen on any platform, my case was on Sun Solaris 64bit.