Recently I had to add an additional interface to a VM cluster post deployment. For the most part it’s pretty well documented but can get confusing depending on the configuration. Unfortunately mistakes were made however a lot was learnt.
The Exadata in question was an X6-2 Quarter rack, with 2x 10Gbps up links configured in active-backup bonds. The VMs were pretty standard configured with 1x Management interface on the 1Gbps Copper, 1x Bonded Client interface on the 10Gbps opitcals and the 2x Active-Active Private interfaces on the InfiniBand.
We needed to add an additional VLAN interface for backups which shared the underlying 10Gbps bond.
The first step is log into the management domain (Dom0) to create the VLAN interface on the existing bond “bondeth0” which makes use of “eth4” and “eth5”. Unfortunately I muddled this up and ended up creating a new bond and caused a network outage inside the VM layer because eth4 and eth5’s bondmaster then became bondeth1. I also forgot to declare the vlan tag.
/opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 vmbondeth1 eth4 eth5
[root@exd1dbadm01 ~]# /opt/oracle.cellos/ipconf.pl -check-consistency
[Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl -check-consistency
Logging started to /var/log/cellos/ipconf.log
[Info]: Verify that the configured values in the Exadata configuration file /opt/oracle.cellos/cell.conf agree with the actual values in use on this system
[Error]: merge_cell_bridge: Slave interface already exists in the cell and bridge configurations, but has different masters: Slave interface; Master in cell configurations; Master in bridge configuration: eth4; bondeth0; bondeth1
Uh Oh!
So to solve this, I had to first change the bondmaster in /etc/sysconfig/network-scripts/ifcfg-eth4 and /etc/sysconfig/network-scripts/ifcfg-eth5 back to bondeth0. Then delete /etc/sysconfig/network-scripts/ifcfg-bondeth1. Finally I had to delete /etc/exadata/ovm/bridge.conf.d/bridge.vmbondeth1.vmbondeth0.bondeth0.eth4.eth5.xml . Then restarted the networking on the server with service network restart.
This brought network connectivity back to the VMs and the ipconf consistency check was clean.
I then created VLAN 114 on the correct bond interface. Note that this is a one time command so even if you need to add VLAN 114 to other VM’s you don’t run it again
[root@exd1dbadm01 ~]# /opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 vmbondeth0 eth4 eth5 114 [root@exd1dbadm01 ~]# /opt/oracle.cellos/ipconf.pl -check-consistency [Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl -check-consistency [Info]: Verify that the configured values in the Exadata configuration file /opt/oracle.cellos/cell.conf agree with the actual values in use on this system Loading basic configuration settings from ILOM ... [Info]: Consistency check PASSED
Once that was done I went ahead and allocated a virtual interface to the VM.
[root@exd1dbadm01 ~]# /opt/exadata_ovm/exadata.img.domu_maker allocate-bridge-domu vmbondeth0.114 exd1db01vm01.devzero.co.za [INFO ] Add bridge in the configuration file: Configuration; Bridge: /EXAVMIMAGES/GuestImages/exd1db01vm01.devzero.co.za/vm.cfg; vmbondeth0.114
[INFO ] Save original configuration file: Original configuration; Backup configuration: /EXAVMIMAGES/GuestImages/exd1db01vm01.devzero.co.za/vm.cfg; /EXAVMIMAGES/GuestImages/exd1db01vm01.devzero.co.za/vm.cfg.backup.by.domu_maker
[INFO ]
[INFO ] -------- MANUAL STEPS TO BE COMPLETED --------
[INFO ] 1. Determine the unique network interface names for use in the target DOMU: exd1db01vm01.devzero.co.za
[INFO ] Bonded interfaces are named bondeth<number> such as bondeth1.
[INFO ] Non bonded interfaces are named eth<number> such as eth2.
[INFO ] These names must be unique within DOMU.
[INFO ] 2. Add the following line for each unique interface to the file /etc/udev/rules.d/70-persistent-net.rules in the DOMU.
[INFO ] Line uses example names ethX or bondethX. Use the actual name from step 1.
[INFO ] SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:16:3e:80:0c:c0", KERNEL=="eth*", NAME="bondethX"
[INFO ] 3. Reboot DOMU to apply the changes.
[INFO ]
In the previous step the utility is nice enough to tell you what work needs to be completed inside the VM. It also tells you to reboot the VM. What I found is that simply rebooting the VM doesn’t make Dom0 reread the configuration file and actually pass a new network interface into the vm. instead shutdown the VM and start it again
[root@exd1dbadm01 ~]# xm shutdown exd1db01vm01.devzero.co.za -w [root@exd1dbadm01 ~]# xm create /EXAVMIMAGES/GuestImages/exd1db01vm01.devzero.co.za/vm.cfg
Next I completed the bits inside the VM to finish the setup. First we need to get the udev rules to name the newly added interface correctly at boot. /etc/udev/rules.d/70-persistent-net.rules should look something like this:
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:00:00:c0:ff:ee:", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:00:de:ad:b3:3f", KERNEL=="eth*", NAME="bondeth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:16:3e:80:0c:c0", KERNEL=="eth*", NAME="bondeth1"
Reboot the VM. Once it’s back up verify that the link names are consistent
ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:00:00:c0:ff:ee brd ff:ff:ff:ff:ff:ff 3: bondeth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:00:de:ad:b3:3f brd ff:ff:ff:ff:ff:ff 3: bondeth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:16:3e:80:0c:c0 brd ff:ff:ff:ff:ff:ff 5: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 7000 qdisc pfifo_fast state UP qlen 1024 link/infiniband 80:00:05:5a:fe:80:00:00:00:00:00:00:00:02:22:c2:b5:8e:93:78 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 6: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 7000 qdisc pfifo_fast state UP qlen 1024 link/infiniband 80:00:05:5b:fe:80:00:00:00:00:00:00:00:02:e0:ab:bb:0b:45:7f brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
Next I created a backup of the current config and run /opt/oracle.cellos/ipconf.pl -nocodes and complete the interactive questions to complete the setup.
cp /opt/oracle.cellos/cell.conf /opt/oracle.cellos/cell.conf.bak /opt/oracle.cellos/ipconf.pl -nocodes [Info]: ipconf command line: /opt/oracle.cellos/ipconf.pl -nocodes Logging started to /var/log/cellos/ipconf.log Interface ib0 is Linked. hca: mlx4_0 Interface ib1 is Linked. hca: mlx4_0 Interface bondeth0 is Linked. driver/mac: vif/00:00:de:ad:b3:3f Interface bondeth1 is ... Linked. driver/mac: vif/00:16:3e:80:0c:c0 The current nameserver(s): 8.8.8.8 8.8.4.4 192.168.123.123 Do you want to change it (y/n) [n]: The current timezone: Africa/Johannesburg Do you want to change it (y/n) [n]: The current NTP server(s):127.0.0.1 192.168.123.123 192.168.123.124 Do you want to change it (y/n) [n]: Network interfaces Name State Status IP address Netmask Gateway Net type Hostname ib0 Linked UP 192.168.100.100 255.255.255.0 Private exd1db01vm01-priv.devzero.co.za ib1 Linked UP 192.168.100.102 255.255.255.0 Private exd1db01vm01-priv2.devzero.co.za bondeth0 eth4,eth5 UP 10.0.0.100 255.255.255.0 10.0.0.1 SCAN exd1db01vm01.devzero.co.za bondeth1 None UNCONF Select interface name to configure or press Enter to continue: bondeth1 Use of uninitialized value in string eq at /opt/oracle.cellos/ipconf.pl line 4007. Selected interface. bondeth1 For guest DOMU slave interfaces define physical interfaces in DOM0 Slaves separated by a comma or none: eth4,eth5 Select connection mode for bondeth1 interface from the list below 1: active-backup 2: lacp Connection mode: 1 IP address or none: 1.2.3.4 Netmask: 255.255.255.0 Gateway (IP address or none) or none: 1.2.3.1 Select network type for interface from the list below 1: Management 2: SCAN 3: Other Network type: 3 Fully qualified hostname or none: exd1db01vm01-bu.devzero.co.za Continue configuring or re-configuring interfaces? (y/n) [y]: Network interfaces Name State Status IP address Netmask Gateway Net type Hostname ib0 Linked UP 192.168.100.100 255.255.255.0 Private exd1db01vm01-priv.devzero.co.za ib1 Linked UP 192.168.100.102 255.255.255.0 Private exd1db01vm01-priv2.devzero.co.za bondeth0 eth4,eth5 UP 10.0.0.100 255.255.255.0 10.0.0.1 SCAN exd1db01vm01.devzero.co.za bondeth1 eth4,eth5 UP 1.2.3.4 255.255.255.0 1.2.3.1 Other exd1db01vm01-bu.devzero.co.za Select interface name to configure or press Enter to continue: Select canonical hostname from the list below 1: exd1db01vm01-priv.devzero.co.za 2: exd1db01vm01-priv2.devzero.co.za 3: exd1db01vm01.devzero.co.za 4: exd1db01vm01-bu.devzero.co.za Canonical fully qualified domain name [3]: Select default gateway interface from the list below 1: bondeth0 2: bondeth1 Default gateway interface [1]: Canonical hostname: exd1db01vm01.devzero.co.za Nameservers: 8.8.8.8 8.8.4.4 192.168.123.123 Timezone: Africa/Johannesburg NTP servers:127.0.0.1 192.168.123.123 192.168.123.124 Default gateway device: bondeth0 Network interfaces Name State Status IP address Netmask Gateway Net type Hostname ib0 Linked UP 192.168.100.100 255.255.255.0 Private exd1db01vm01-priv.devzero.co.za ib1 Linked UP 192.168.100.102 255.255.255.0 Private exd1db01vm01-priv2.devzero.co.za bondeth0 eth4,eth5 UP 10.0.0.100 255.255.255.0 10.0.0.1 SCAN exd1db01vm01.devzero.co.za bondeth1 eth2,eth3 UP 1.2.3.4 255.255.255.0 1.2.3.1 Other exd1db01vm01-bu.devzero.co.za Is this correct (y/n) [y]:y
Once completed on the one node. Complete the same steps on the second node of the cluster.
