I had this great opportunity to deploy the new infrastructure for a project. From storage to the database level. I will be sharing stuff that I have learnt during the implementation as I believe through explaining, I will get a better understanding of the concept too!
This will be a 3 part posts as the topic is just too huge to be covered in just a single post!
Oracle’s answer to IBM LPAR (Logical Partitioning) is LDOMs, which is also known as logical domains. (I believe Oracle has rename LDOMs to Oracle VM but I still like calling it LDOMs!) What I like about Oracle LDOMs is that it uses command line for management. I do not need to think where to click or repetitively click 200 times just to create 20 guest domains.
It’s features is no doubt more limited as compared to IBM LPARs. This is expected since IBM got into virtualization business much much earlier than SUN/Oracle.
Please note that the stuffs I am talking about later on are mainly for Oracle VM for SPARC, not the x86 version.
For basic LDOMs concept, please read the documentation below. http://www.oracle.com/technetwork/documentation/vm-sparc-194287.html
How to configure an additional IO domain.
Initially, the control domain will own all the resources in the box. You need to reconfigure the control domain so that it can assign the resources out to the guest domains.
primary#ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-c-- UART 256 130560M 0.4% 16m primary# ldm start-reconf primary primary# ldm set-vcpu -c 2 primary primary# ldm set-memory 8G primary primary# ldm add-config initial primary# init 6 primary# svcadm -v disable -s volfs svcadm: Pattern 'volfs' doesn't match any instances primary# svcs -a | grep volfs primary# ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- UART 16 8G 5.2% 4m
IO domain is in fact just another guest domain that directly owns a PCIe device or the entire PCI bus. You will need to first configure the resources for the IO domain.
IMPORTANT!! Do not assign the PCI bus that is currently “holding” your internal disk which your OS sits on! Identify clearly all the devices on the PCI buses before assigning them out!
This is how you can identify the PCI bus which your OS sits on.
primary# zpool status rpool pool: rpool state: ONLINE scan: resilvered 112M in 0h0m with 0 errors on Mon Dec 3 19:41:32 2012 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c0t5000C50048680AABd0s0 ONLINE 0 0 0 errors: No known data errors primary# ls -l /dev/dsk/c0t5000C50048680AABd0s0 lrwxrwxrwx 1 root root 48 Jun 19 2012 /dev/dsk/c0t5000C50048680AABd0s0 -> ../../devices/scsi_vhci/disk@g5000c50048680aab:a primary# prtconf -v | more location: /dev/chassis//SYS/MB/HDD0/disk Paths from multipath bus adapters: Path 1: /pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000c50048680aa9,0 The internal disk is on pci400 bus!
Assigning the resources for the secondary IO domain.
primary# ldm add-domain secondary primary# ldm set-vcpu -c 2 secondary primary# ldm add-memory 8G secondary primary# ldm set-var auto-boot\?=false secondary
primary# ldm list-io
IO PSEUDONYM DOMAIN -- --------- ------ pci@400 pci_0 primary pci@500 pci_1 primary pci@600 pci_2 primary pci@700 pci_3 primary PCIE PSEUDONYM STATUS DOMAIN ---- --------- ------ ------ pci@400/pci@2/pci@0/pci@1 /SYS/PCI-EM0 EMP - pci@400/pci@2/pci@0/pci@2 /SYS/PCI-EM1 EMP - pci@400/pci@1/pci@0/pci@4 /SYS/PCI-EM2 EMP - pci@400/pci@2/pci@0/pci@3 /SYS/PCI-EM3 EMP - pci@400/pci@1/pci@0/pci@0 /SYS/MB/REM0 OCC primary pci@400/pci@1/pci@0/pci@2 /SYS/RIO/NET0 OCC primary pci@500/pci@1/pci@0/pci@1 /SYS/PCI-EM8 OCC primary pci@500/pci@2/pci@0/pci@2 /SYS/PCI-EM9 EMP - pci@500/pci@1/pci@0/pci@2 /SYS/PCI-EM10 EMP - pci@500/pci@2/pci@0/pci@3 /SYS/PCI-EM11 EMP - pci@600/pci@1/pci@0/pci@4 /SYS/PCI-EM4 EMP - pci@600/pci@2/pci@0/pci@5 /SYS/PCI-EM5 EMP - pci@600/pci@1/pci@0/pci@5 /SYS/PCI-EM6 EMP - pci@600/pci@2/pci@0/pci@0 /SYS/PCI-EM7 EMP - pci@600/pci@2/pci@0/pci@3 /SYS/RIO/NET2 OCC primary pci@700/pci@2/pci@0/pci@3 /SYS/PCI-EM12 OCC primary pci@700/pci@2/pci@0/pci@4 /SYS/PCI-EM13 EMP - pci@700/pci@1/pci@0/pci@4 /SYS/PCI-EM14 EMP - pci@700/pci@2/pci@0/pci@5 /SYS/PCI-EM15 EMP - pci@700/pci@1/pci@0/pci@0 /SYS/MB/REM1 OCC primary primary# ldm remove-io pci_1 primary primary# ldm remove-io pci_2 primary primary# ldm add-config io-domain primary#init 6 primary# ldm add-io pci_1 secondary primary# ldm add-io pci_3 secondary primary# ldm list-io IO PSEUDONYM DOMAIN -- --------- ------ pci@400 pci_0 primary pci@500 pci_1 secondary pci@600 pci_2 primary pci@700 pci_3 secondary
After the IO domain is setup, you may proceed to install the OS on the IO domain.
Something very important;
When I assigned the entire PCI bus to the secondary IO domain, it owns the entire bus. This means if the network card is slotted on that bus, the secondary IO domains owns the entire card and is able to see it. With this concept, all the fibre channel ports and ethernet ports on pci500 and pci700 is own by secondary domain and therefore is able see the LUNs, internal disks and network devices.
Please keep this in mind as it is a very critical point for managing Oracle LDOMs.