Oracle Solaris Logical Domain – How to deploy a redundant LDOM Setup part 1

I had this great opportunity to deploy the new infrastructure for a project. From storage to the database level. I will be sharing stuff that I have learnt during the implementation as I believe through explaining, I will get a better understanding of the concept too!

This will be a 3 part posts as the topic is just too huge to be covered in just a single post!

Oracle’s answer to IBM LPAR (Logical Partitioning) is LDOMs, which is also known as logical domains. (I believe Oracle has rename LDOMs to Oracle VM but I still like calling it LDOMs!) What I like about Oracle LDOMs is that it uses command line for management. I do not need to think where to click or repetitively click 200 times just to create 20 guest domains.

It’s features is no doubt more limited as compared to IBM LPARs. This is expected since IBM got into virtualization business much much earlier than SUN/Oracle.

Please note that the stuffs I am talking about later on are mainly for Oracle VM for SPARC, not the x86 version.

For basic LDOMs concept, please read the documentation below.

How to configure an additional IO domain. 

Initially, the control domain will own all the resources in the box. You need to reconfigure the control domain so that it can assign the resources out to the guest domains.

primary#ldm list
primary active -n-c-- UART 256 130560M 0.4% 16m
primary# ldm start-reconf primary
primary# ldm set-vcpu -c 2 primary
primary# ldm set-memory 8G primary
primary# ldm add-config initial
primary# init 6
primary# svcadm -v disable -s volfs
 svcadm: Pattern 'volfs' doesn't match any instances
primary# svcs -a | grep volfs
primary# ldm list
primary active -n-cv- UART 16 8G 5.2% 4m
IO domain is in fact just another guest domain that directly owns a PCIe device or the entire PCI bus. You will need to first configure the resources for the IO domain.

IMPORTANT!! Do not assign the PCI bus that is currently “holding” your internal disk which your OS sits on! Identify clearly all the devices on the PCI buses before assigning them out!

This is how you can identify the PCI bus which your OS sits on.
primary# zpool status rpool
  pool: rpool
  state: ONLINE
  scan: resilvered 112M in 0h0m with 0 errors on Mon Dec 3 19:41:32 2012
  rpool ONLINE 0 0 0
  c0t5000C50048680AABd0s0 ONLINE 0 0 0
errors: No known data errors
 primary# ls -l /dev/dsk/c0t5000C50048680AABd0s0
 lrwxrwxrwx 1 root root 48 Jun 19 2012 /dev/dsk/c0t5000C50048680AABd0s0 -> ../../devices/scsi_vhci/disk@g5000c50048680aab:a
primary# prtconf -v | more
location: /dev/chassis//SYS/MB/HDD0/disk
  Paths from multipath bus adapters:
  Path 1: /pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000c50048680aa9,0
The internal disk is on pci400 bus!

Assigning the resources for the secondary IO domain.

primary# ldm add-domain secondary
primary# ldm set-vcpu -c 2 secondary
primary# ldm add-memory 8G secondary
primary# ldm set-var auto-boot\?=false secondary

primary# ldm list-io

 -- --------- ------
pci@400 pci_0 primary
 pci@500 pci_1 primary
 pci@600 pci_2 primary
 pci@700 pci_3 primary
 ---- --------- ------ ------
 pci@400/pci@2/pci@0/pci@1 /SYS/PCI-EM0 EMP -
 pci@400/pci@2/pci@0/pci@2 /SYS/PCI-EM1 EMP -
 pci@400/pci@1/pci@0/pci@4 /SYS/PCI-EM2 EMP -
 pci@400/pci@2/pci@0/pci@3 /SYS/PCI-EM3 EMP -
 pci@400/pci@1/pci@0/pci@0 /SYS/MB/REM0 OCC primary
 pci@400/pci@1/pci@0/pci@2 /SYS/RIO/NET0 OCC primary
 pci@500/pci@1/pci@0/pci@1 /SYS/PCI-EM8 OCC primary
 pci@500/pci@2/pci@0/pci@2 /SYS/PCI-EM9 EMP -
 pci@500/pci@1/pci@0/pci@2 /SYS/PCI-EM10 EMP -
 pci@500/pci@2/pci@0/pci@3 /SYS/PCI-EM11 EMP -
 pci@600/pci@1/pci@0/pci@4 /SYS/PCI-EM4 EMP -
 pci@600/pci@2/pci@0/pci@5 /SYS/PCI-EM5 EMP -
 pci@600/pci@1/pci@0/pci@5 /SYS/PCI-EM6 EMP -
 pci@600/pci@2/pci@0/pci@0 /SYS/PCI-EM7 EMP -
 pci@600/pci@2/pci@0/pci@3 /SYS/RIO/NET2 OCC primary
 pci@700/pci@2/pci@0/pci@3 /SYS/PCI-EM12 OCC primary
 pci@700/pci@2/pci@0/pci@4 /SYS/PCI-EM13 EMP -
 pci@700/pci@1/pci@0/pci@4 /SYS/PCI-EM14 EMP -
 pci@700/pci@2/pci@0/pci@5 /SYS/PCI-EM15 EMP -
 pci@700/pci@1/pci@0/pci@0 /SYS/MB/REM1 OCC primary

primary# ldm remove-io pci_1 primary
primary# ldm remove-io pci_2 primary
primary# ldm add-config io-domain
primary#init 6
primary# ldm add-io pci_1 secondary
primary# ldm add-io pci_3 secondary

primary# ldm list-io
 -- --------- ------
 pci@400 pci_0 primary
 pci@500 pci_1 secondary
 pci@600 pci_2 primary
 pci@700 pci_3 secondary
After the IO domain is setup, you may proceed to install the OS on the IO domain.

Something very important;

When I assigned the entire PCI bus to the secondary IO domain, it owns the entire bus. This means if the network card is slotted on that bus, the secondary IO domains owns the entire card and is able to see it. With this concept, all the fibre channel ports and ethernet ports on pci500 and pci700 is own by secondary domain and therefore is able see the LUNs, internal disks and network devices.

Please keep this in mind as it is a very critical point for managing Oracle LDOMs.

  1. #1 by Khalid Awad on June 15, 2013 - 19:30

    Perfect … Thank you …

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: