TIP 119 - Why some ODA commands hang?

If you are an ODA customer and you see that "oakcli show version -detail" hangs or some patch command does run forever with no output, I think you might need to make sure .bash_profile is not customized for user input...

Please see below :

Oakcli Commands All Hang, Do Not Complete (Doc ID 1981186.1)

Lesson learned - Do not change .bash_profile in ODA

TIP 118# : Virtualized Exadata-Adding new local disk to DomU

I have seen more Oracle customer adopting virtualized platform in Oracle Exadata. At the same time, I hear concerns regarding virtualization on Exa would be new and it could be buggy, could not be well documented and well tested compare with Bate-Metal install. All these provide me a motivation  to publish some posts regarding real experience in this platform.

Here is the first post :

If you are one of the Oracle customer with a virtualized Exadata, you may notice that the available local space in each DomU may not be sufficient to for various things which you may do such as Grid software, multiple Oracle software, 3rd party monitoring, backup agents and etc.
Also you may notice that there is no full step by step document on how to add more space to local space in DomU.

This post is to share my experience with performing this task and walk you through step by step :

Step 1 :  List existing disks and find out available disk

lsblk -id

NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda 202:0    0   25G  0 disk
xvdb 202:16   0   50G  0 disk /u01/app/12.1.0.2/grid
xvdc 202:32   0   50G  0 disk /u01/app/oracle/product/12.1.0.2/dbhome_1
xvdd 202:48   0   58G  0 disk
sda    8:0    0  128M  0 disk
sdb    8:16   0  128M  0 disk
sdc    8:32   0  128M  0 disk

I will use xvde (the next available) as new disk

Step 2 : Prepare new 20GB file

All VM guest physical files are in Dom0 under /EXAVMIMAGES/GuestImages. The new file is named db12.1.0.2.160719-3-2ndHome.img

qemu-img create /EXAVMIMAGES/GuestImages/exavmadm01vm01.newco.local/db12.1.0.2.160719-3-2ndHome.img 20G

Step 3 : Attach the newly generated file to VM

Use the xm block-attach command to dynamically attach the new generated file to the existing VM.

xm block-attach exavmadm01vm01.newco.local  file:/EXAVMIMAGES/GuestImages/exavmadm01vm01.newco.local/db12.1.0.2.160719-3-2ndHome.img /dev/xvde w

exavmadm01vm01.newco.local  is the VM and the file which is created in step3 is attached as xvde. 
The new disk should be visible in VM now.

lsblk -id
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda 202:0    0   25G  0 disk
xvdb 202:16   0   50G  0 disk /u01/app/12.1.0.2/grid
xvdc 202:32   0   50G  0 disk /u01/app/oracle/product/12.1.0.2/dbhome_1
xvdd 202:48   0   58G  0 disk
sda    8:0    0  128M  0 disk
sdb    8:16   0  128M  0 disk
sdc    8:32   0  128M  0 disk
xvde 202:64   0   20G  0 disk
 

Step 4 : Find VM uuid and generate a new uuid for the new disk

VM uuid is set in vm.cfg and a new uuid has to be generated for the new disk.

grep -i uuid vm.cfg
uuid = 'b0c2dd2ee4b14f6cabe0b546b930b6e8'  

uuidgen | tr -d '-'
666a1ab9604e4e3d95bbdfce30d0b739

b0c2dd2ee4b14f6cabe0b546b930b6e8 is uuid for VM and 666a1ab9604e4e3d95bbdfce30d0b739 is uuid for the new disk.
These two uuids establish the new disk location. In my example, it would be such as below :

/OVS/Repositories/b0c2dd2ee4b14f6cabe0b546b930b6e8/VirtualDisks/666a1ab9604e4e3d95bbdfce30d0b739.img 

ln -s /EXAVMIMAGES/GuestImages/exavmadm01vm01.newco.local/db12.1.0.2.160719-3-2ndHome.img /OVS/Repositories/b0c2dd2ee4b14f6cabe0b546b930b6e8/VirtualDisks/666a1ab9604e4e3d95bbdfce30d0b739.img

Step 5 - Add the new disk to vm configuration file

vm configuration file (vm.cfg) has disk setting which the new disk needs to be added

disk = [...,
'file:/OVS/Repositories/b0c2dd2ee4b14f6cabe0b546b930b6e8/VirtualDisks/666a1ab9604e4e3d95bbdfce30d0b739.img,xvde,w']

Step6 - Bounce VM and format the new disk

This is mainly the last step - It would be safe to bounce VM and when it backs up, the new disk should be visible.

xm reboot exavmadm01vm01.newco.local

The new disk needs to be formatted and then it is mounted (in this case, it is mounted as /u01/app/oracle/product/12.1.0.2/dbhome_2) and the mount point needs to be added to /etc/fstab

mkfs -t ext4 /dev/xvde
mkdir -p /u01/app/oracle/product/12.1.0.2/dbhome_2
mount -t ext4 /dev/xvde /u01/app/oracle/product/12.1.0.2/dbhome_2


/etc/fstab
.
.
.
/dev/xvde               /u01/app/oracle/product/12.1.0.2/dbhome_2       ext4    defaults        1 1

Now the new disk is ready for use.
Hope it helps.

TIP 117#: Access to ODA ILOM is challenging ?

New ODA is configured with ILOM IPs and network cable is plugged to Netmgt port. Everything is ready for connecting to the server via ILOM and bummer!, you may see the whole connection or at least the connection to remote console does not work as it is expected.

The following item could come handy to troubleshoot and fix the issue:

ILOM client requirements:

Per the following link, the client access should meet certain requirement.

http://docs.oracle.com/cd/E24707_01/html/E24528/z40000191582334.html

I found that having right Java and registering it in browser are the key.
For new ODA (X5-2), per 888888.1, consider upgrading Java to 7 to avoid bug 20420055

After you make all necessary changes per above requirement, you may still have some problem connecting to remote console.
The following two issues are reported on several customers which we have :

Issue : ILOM remote console is blocked by rule set

Solution : The simplest solution is to remove rule set file - The file is C:\Windows\Sun\Java\Deployment\DeploymentRuleSet.jar and it is not dependent on where Java is installed.
If the issue still stays, consider setting Java security level to medium in Java control panel (in windows in Windows control panel)

Issue : ILOM remote console errors out with "Video redirection error - no appropriate protocol found"

Solution: Comment out jdk.tls.disabledAlgorithms=SSLv3 in JAVA_folder/lib/security/java.security and restart the browser
 

TIP 116# : Oracle Database Appliance - 3 Observations and 3 Resolutions

During last couple weeks, I have been involved in various ODA activities from patching of an existing ODA to latest version to setting up new environments on ODA X5-2
I thought to share some lesson learnt :
 
1)  Patch failure
 
If you encounter in a situation which ODA patch fails and reports the following error
Unable to get the nodenumber in the chassis
The resolution is to create /opt/oracle/oak/conf/node_num.conf on each compute node and add NODENUM=0 or NODENUM=1.
 
2) Network issue
 
If you setup new ODA and in the first boot, it should ask for fiber or copper connection and after providing the right option, ODA will keep the appropriate driver and will remove unnecessary drivers. If you do not see it in any new ODA x3-2 onwards when it is booted for the first time, you may see issue with setting up network later.
To fix the issue, before further progress with deployment and setting up environment, run /opt/oracle/oak/lib/SetupX4Network.pl

3)  Some oakcli commands fail
You may notice that some oakcli commands fail with the following error:


OAKERR:6000 Exception encountered in Module getMasterNode
OAKERR:6002 No master node was found

This cause usually could be that the process cannot connect to oakd daemon on ODA. 
You could confirm this with running
oakcli show ismaster
If this command fails, it could be fixed by bouncing oakd using oakcli. As long as oakcli show ismaster does not work well, you may experience failure of some oakcli commands. 
 
 
 

TIP 111# : ODA and CPU license

Last week, I was working on a customer with ODA x4-2 which we found 5 CPU licenses are sold for EE deployment. My first reaction was on how to put 5 CPU license (10 cores) in ODA x4-2. As you may know, the multiplier of 4 cores can be activated in X3-2 and above and multiplier of 2 in ODA v1. In this post, I am trying to explain what is the minimum core requirement for various ODA server with virtualized and non-virtualized deployment.

We all know that "pay as grow" is one of the great feature of Oracle Database Appliance. In other words, you could start with as minimum core as possible and then scale up.
However, due to CPU specification on various ODA version, there is some limitation in regards to minimum number of cores which could be setup as it is shown in below :

Min license
	V1	X3-2,3,4
EE	2x0.5=1	4x0.5=2	10 day rule
RACOne	2x0.5=1	4x0.5=2	10 day rule
RAC	2x2x0.5=2	2x4x0.5=4
EE -VM	2x0.5=1	2x0.5=1	10 day rule
RACOne-VM	2x0.5=1	2x0.5=1	10 day rule
RAC - VM	2x2x0.5=2	2x2x0.5=2

---

Couple notes :

• 10 day rule is to have all instances in EE or RAC1 only on one ODA node and without paying license to be allowed to flip over to the second node in case the first node is not available only for 10 days per calendar year.
• Since ODA platform is x86, the contributing factor is 0.5. This is where you can see all multiplier of 0.5 in the table.
• On ODA x3-2 and above odd number of CPU can not be setup unless ODA is setup as virtualized platform.
• Virtualization provides more flexibility in terms of lowering the number of the required CPU specially for EE or RAC1. If you are dealing with small customer, you may end up to go virtualized even ODA is purposed only for database deployment.

Hope this post clarifies some confusion.

TIP 107# : Virtualization on Oracle Database Appliance (ODA)

 As you know, Early this year Oracle started supporting virtualization on Oracle Database Appliance which enhances the functionality of ODA. One of the main usage of this virtualization is to utilize ODA for Weblogic as well as database.

I had a chance to test various Weblogic offerings on ODA and I thought to share my findings with readers, hopefully it helps readers with planning and architecting the environment using ODA and virtualization. Basically, the two main versions which can be run on ODA V1 and ODA X3-2 and support virtualization are ODA Version 2.5 and ODA version 2.6. The following shows my findings on Weblogic install for each version:

ODA 2.5

• Oracle Weblogic template on ODA is available on e-delivery for Weblogic 10.3.6 and 12c
• Oracle Weblogic template includes Weblogic configurator which is a new tool (specific to ODA) to automate installation of Weblogic on ODA similar to what OAM does for database.
• Weblogic configurator only installs Weblogic in cluster mode, having non-cluster weblogic is not supported with the configurator.
• Weblogic configurator installs OTD (Oracle Traffic Director) and as of now, there is no way to skip it during installation. However after installation, it can be shutdown if it is not needed.
• Minimum installation with Weblogic configurator includes two Weblogic cluster nodes which installs 6VMs in total (1 VM for Weblogic admin,2 VMs (one per compute node) for Weblogic managed server, 1 VM for OTD admin, 2 VMs (one per compute node) for OTD)
• Weblogic configurator also creates a database for internal work on ODA_BASE.
• Weblogic configurator installs all components as root due to a bug.
• Since 6VMs are the possible minimum configuration and each VM should have at least 2VCPU, there will be total of 12VCPU (6Cores) for minimum installation 

ODA 2.6

• No Weblogic template on ODA is released for 2.6
• No Weblogic configurator for automation of Weblogic is released
• As of now, creation readhat Linux and installing Weblogic manually would be the only approach in this release

Bottom line: 

Although it is recommended to utilize automation on ODA to ensure ODA stays with Oracle standard template and to have less impact on future changes such as patching and upgrade, however Oracle Weblogic configurator looks not to be mature enough to be utilized. First of all, to use it, you end up to stay with 2.5 (not with the latest ODA release ) plus the configurator in 2.5 is buggy (installation of weblogic as root) and it has several shortcomings which are listed above as well as not much flexible. I recommend to wait (if you can) as Weblogic configurator should be out soon or use manual process for Weblogic with the risk that in long run, you may not be able to utilize patch bundle (if Oracle includes weblogic in ODA bundle) or any further automation which may be planned for Weblogic. Also consider that installation of OTD does add complexity to weblogic licensing on ODA.

At the end, I found FAQ pdf file on otn website helpful.

http://www.oracle.com/technetwork/middleware/weblogic-oda/overview/faq-weblogicserver-on-oda-1927929.pdf

 If you have any questions or you want to see more on this topic, please drop me a line.