Oracle Solaris Logical Domain – How to deploy a redundant LDOM Setup part 2

Oracle Solaris Logical Domain – How to deploy a redundant LDOM Setup part 2 (Network Redundancy)

To continue to what was left off in part 1, once the IO domain is set up, when you should see the following.

# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 8G 5.3% 34d 6h 59m
secondary active -n--v- 5000 16 8G 0.0% 34d 6h 59m

Now, we need to create the basic services that are required by the control domain to manage the guest domains.

Create the virtual console
# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
Enable Virtual Network Terminal Server Daemon
# svcadm enable vntsd
# svcs -a | grep vntsd
online 16:46:35 svc:/ldoms/vntsd:default

Typically, all the high end systems like T4 or p750 comes with multiple PCIe slots for you to add in additional ethernet card or HBA card for FC connectivity. In this implementation, I had 2 additional ethernet card. And I slot them on EM0 and EM8, on PCI bus 400 and 500 (Refer to part 1).

# ldm list-io
IO PSEUDONYM DOMAIN
 -- --------- ------
pci@400 pci_0 primary
 pci@500 pci_1 primary
 pci@600 pci_2 primary
 pci@700 pci_3 primary
PCIE PSEUDONYM STATUS DOMAIN
 ---- --------- ------ ------
 pci@400/pci@2/pci@0/pci@1 /SYS/PCI-EM0 EMP -
 pci@400/pci@2/pci@0/pci@2 /SYS/PCI-EM1 EMP -
 pci@400/pci@1/pci@0/pci@4 /SYS/PCI-EM2 EMP -
 pci@400/pci@2/pci@0/pci@3 /SYS/PCI-EM3 EMP -
 pci@400/pci@1/pci@0/pci@0 /SYS/MB/REM0 OCC primary
 pci@400/pci@1/pci@0/pci@2 /SYS/RIO/NET0 OCC primary
 pci@500/pci@1/pci@0/pci@1 /SYS/PCI-EM8 OCC primary
 pci@500/pci@2/pci@0/pci@2 /SYS/PCI-EM9 EMP -
 pci@500/pci@1/pci@0/pci@2 /SYS/PCI-EM10 EMP -
 pci@500/pci@2/pci@0/pci@3 /SYS/PCI-EM11 EMP -
 pci@600/pci@1/pci@0/pci@4 /SYS/PCI-EM4 EMP -
 pci@600/pci@2/pci@0/pci@5 /SYS/PCI-EM5 EMP -
 pci@600/pci@1/pci@0/pci@5 /SYS/PCI-EM6 EMP -
 pci@600/pci@2/pci@0/pci@0 /SYS/PCI-EM7 EMP -
 pci@600/pci@2/pci@0/pci@3 /SYS/RIO/NET2 OCC primary
 pci@700/pci@2/pci@0/pci@3 /SYS/PCI-EM12 OCC primary
 pci@700/pci@2/pci@0/pci@4 /SYS/PCI-EM13 EMP -
 pci@700/pci@1/pci@0/pci@4 /SYS/PCI-EM14 EMP -
 pci@700/pci@2/pci@0/pci@5 /SYS/PCI-EM15 EMP -
 pci@700/pci@1/pci@0/pci@0 /SYS/MB/REM1 OCC primary

In this way, my control domain will own 02 x onboard ethernet port and 02 x extra ethernet port on a additional PCI card. That applies to my IO domain too.

This is where the planning of the slotting of the devices comes into play!

I recommend to do a link aggregation on both the control domain (1 from onboard, 1 from PCI card). Then do the same on IO domain. You may refer to the Oracle docs to do it.  Once done, you may continue the configuration in the control domain.

Create the Virtual Switch Server
Checking the link aggregation configuration on both the domains.
primary # dladm show-aggr
LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS
aggr0 L4 auto passive short -----
secondary$ dladm show-aggr
LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS
aggr1 L4 auto off short -----
# ldm add-vsw net-dev=aggr0 primary-vsw0 primary
# ldm add-vsw net-dev=aggr1 secondary-vsw0 secondary
# ldm list-services
VCC
 NAME LDOM PORT-RANGE
 primary-vcc0 primary 5000-5100
VSW
 NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK
 primary-vsw0 primary 00:14:4f:fb:d4:ea aggr0 0 switch@0 1 1500 on
VSW
 NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK
 secondary-vsw0 secondary 00:14:4f:f9:2c:b7 aggr1 0 switch@0 1 1500 on

Well, some of you may be wondering, assuming if I already assign out the PCI bus from the control domain to the IO domain, this means that the control domain shouldn’t be able to see what physical devices IO domain owns. Furthermore, the control domain shouldn’t be able to know the link aggregation configuration of the IO domain since IO domain is another Solaris 11 domain and there should be some level of data isolation here!

Honestly, I am not sure about this. Apparently, the control domain is able to “export” the Virtual Server Switch from the IO domain as proved from the output above. With this, you are done with the network redundancy! The part 3 will talk about disk redundancies, which is a critical factor in the implementation.

P.S: I am terribly sorry about the delay post for this series. I was totally tied down with work and studies.

Advertisements
  1. #1 by John Green on June 24, 2014 - 19:57

    I configured the link aggregation as outlined above. I have assigned pci_2 and pci_3 to my IO domain (secondary). When I reboot the primary domain, I lose the ability to enter new commands on my guest domain and cannot login. (I get the password prompt but the prompt does not return until the primary domain is up.) Is this expected or do I have something configured incorrectly?

    VSW
    NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK
    primary-vsw0 primary 00:14:4f:fa:56:be aggr0 0 switch@0 1 1 1500 on

    VSW
    NAME LDOM MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK
    secondary-vsw0 secondary 00:14:4f:f9:e0:b8 aggr1 0 switch@0 1 1 1500 on

  2. #2 by John Green on June 26, 2014 - 13:59

    I was able to figure out my networking issue. There is no need to respond to my previous comment. Thank you for this blog. It has been a lot of help to me in configuring my T5.

  3. #3 by Wei Shan on June 26, 2014 - 14:59

    Hello John,

    Sorry. I did not notice your comments until earlier on. Do you mind sharing what was the issue?

    • #4 by papajgreen on July 2, 2014 - 18:32

      I apologize for the delay but I couldn’t get to my data center until today. I was mistaken. When the primary domain is shutdown, I lose network connectivity to both guest domains and to the secondary domain even though I have aggr1 setup in the secondary IO domain. I can ping primary and secondary interfaces in both guest domains and the aggr1 interface in the secondary IO domain. However, ssh connections to the guest domains hang after the password is entered.

      • #5 by Wei Shan on July 4, 2014 - 16:44

        Hi John,

        Is this a T5-4 machine? I believe you assigned the PCIe bus slots wrongly.

        Do you have support with Oracle?

      • #6 by papajgreen on July 4, 2014 - 19:04

        It’s a T5-2. It’s weird because I see the interfaces on the secondary IO domain. I will open a ticket with Oracle support.

  4. #7 by Samar Ju on February 8, 2015 - 18:52

    Hi guys .. i have such question .. Is it possible to create link aggregation in guest ldom ? I dont think the virtual switch is clever enough but anyway is it possible ??
    Thanks
    Samar Ju

  1. Oracle Solaris Logical Domain – How to deploy a redundant LDOM Setup part 3 | Weishan's Oracle Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: