Oracle Solaris Logical Domain – How to deploy a redundant LDOM Setup part 2 (Network Redundancy)
To continue to what was left off in part 1, once the IO domain is set up, when you should see the following.
# ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- UART 16 8G 5.3% 34d 6h 59m secondary active -n--v- 5000 16 8G 0.0% 34d 6h 59m
Now, we need to create the basic services that are required by the control domain to manage the guest domains.
Create the virtual console # ldm add-vcc port-range=5000-5100 primary-vcc0 primary Enable Virtual Network Terminal Server Daemon # svcadm enable vntsd # svcs -a | grep vntsd online 16:46:35 svc:/ldoms/vntsd:default
Typically, all the high end systems like T4 or p750 comes with multiple PCIe slots for you to add in additional ethernet card or HBA card for FC connectivity. In this implementation, I had 2 additional ethernet card. And I slot them on EM0 and EM8, on PCI bus 400 and 500 (Refer to part 1).
# ldm list-io IO PSEUDONYM DOMAIN -- --------- ------ pci@400 pci_0 primary pci@500 pci_1 primary pci@600 pci_2 primary pci@700 pci_3 primary PCIE PSEUDONYM STATUS DOMAIN ---- --------- ------ ------ pci@400/pci@2/pci@0/pci@1 /SYS/PCI-EM0 EMP - pci@400/pci@2/pci@0/pci@2 /SYS/PCI-EM1 EMP - pci@400/pci@1/pci@0/pci@4 /SYS/PCI-EM2 EMP - pci@400/pci@2/pci@0/pci@3 /SYS/PCI-EM3 EMP - pci@400/pci@1/pci@0/pci@0 /SYS/MB/REM0 OCC primary pci@400/pci@1/pci@0/pci@2 /SYS/RIO/NET0 OCC primary pci@500/pci@1/pci@0/pci@1 /SYS/PCI-EM8 OCC primary pci@500/pci@2/pci@0/pci@2 /SYS/PCI-EM9 EMP - pci@500/pci@1/pci@0/pci@2 /SYS/PCI-EM10 EMP - pci@500/pci@2/pci@0/pci@3 /SYS/PCI-EM11 EMP - pci@600/pci@1/pci@0/pci@4 /SYS/PCI-EM4 EMP - pci@600/pci@2/pci@0/pci@5 /SYS/PCI-EM5 EMP - pci@600/pci@1/pci@0/pci@5 /SYS/PCI-EM6 EMP - pci@600/pci@2/pci@0/pci@0 /SYS/PCI-EM7 EMP - pci@600/pci@2/pci@0/pci@3 /SYS/RIO/NET2 OCC primary pci@700/pci@2/pci@0/pci@3 /SYS/PCI-EM12 OCC primary pci@700/pci@2/pci@0/pci@4 /SYS/PCI-EM13 EMP - pci@700/pci@1/pci@0/pci@4 /SYS/PCI-EM14 EMP - pci@700/pci@2/pci@0/pci@5 /SYS/PCI-EM15 EMP - pci@700/pci@1/pci@0/pci@0 /SYS/MB/REM1 OCC primary
In this way, my control domain will own 02 x onboard ethernet port and 02 x extra ethernet port on a additional PCI card. That applies to my IO domain too.
This is where the planning of the slotting of the devices comes into play!
I recommend to do a link aggregation on both the control domain (1 from onboard, 1 from PCI card). Then do the same on IO domain. You may refer to the Oracle docs to do it. Once done, you may continue the configuration in the control domain.
Create the Virtual Switch Server Checking the link aggregation configuration on both the domains. primary # dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS aggr0 L4 auto passive short ----- secondary$ dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS aggr1 L4 auto off short ----- # ldm add-vsw net-dev=aggr0 primary-vsw0 primary # ldm add-vsw net-dev=aggr1 secondary-vsw0 secondary # ldm list-services VCC NAME LDOM PORT-RANGE primary-vcc0 primary 5000-5100 VSW NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK primary-vsw0 primary 00:14:4f:fb:d4:ea aggr0 0 switch@0 1 1500 on VSW NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK secondary-vsw0 secondary 00:14:4f:f9:2c:b7 aggr1 0 switch@0 1 1500 on
Well, some of you may be wondering, assuming if I already assign out the PCI bus from the control domain to the IO domain, this means that the control domain shouldn’t be able to see what physical devices IO domain owns. Furthermore, the control domain shouldn’t be able to know the link aggregation configuration of the IO domain since IO domain is another Solaris 11 domain and there should be some level of data isolation here!
Honestly, I am not sure about this. Apparently, the control domain is able to “export” the Virtual Server Switch from the IO domain as proved from the output above. With this, you are done with the network redundancy! The part 3 will talk about disk redundancies, which is a critical factor in the implementation.
P.S: I am terribly sorry about the delay post for this series. I was totally tied down with work and studies.