Today I learned an inelegant but very effective way to access iLO2 remotely.
From the safety and comfort of my own bed I was able to log in to the iLO2 function, access the full remote physical- and virtual-console for the server, and carry out the usual range of console-related management/admin functions.
Brilliant.
Meanwhile, back in the datacentre…
One of the early problems I encountered with the first DL380 out of the box (server c1, obv), was an inability to configure iLO2.
I’d racked the server up, plugged the three NICs in to the router (eth0, eth1, iLO2), installed the base CentOS operating system, and configured eth0 for static IP.
Then I spent ages trying to get to the bottom of why iLO2 wouldn’t work.
And I mean four-five weeks, off and on.
A couple of nights ago (while watching a YouTube video of someone configuring their iLO2 with annoying ease), I read a revelation.
iLO2 won’t work outside the host network (LAN).
So even if I got iLO2 working, I couldn’t use it remotely (in the pure sense of the word – remotely), unless I built and stretched a VLAN to wherever in the world I was working remotely from.
This is a massive pain in the backside.
Obv.
Armed with this new information I dismissed not having iLO2 as an inconvenience – a non operational inconvenience – as not having iLO2 wasn’t actually stopping anything from proceeding,
So I set this slight disappointment aside and carried on rolling out the datacentre.
Last weekend I fitted the remaining disks to the second server in the rollout, server c2.
I configured the disks to RAID 6, installed CentOS, virtualised the physical environment, configured a static ip on the network on eth0, and brought the server online.
From server c2 I successfully pinged the internal ip address for server c1, to confirm everything was working on the LAN, and return pinged server c2 from server c1 but left c2 unconfigured for WAN access for now, then left the datacentre feeling a bit pleased.
Over the last couple of weeks I’ve been reading and watching a lot of tutorials.
One or two have gone in to detail about how both eth0 and eth1 should be configured.
I had only configured eth0 on both of the servers I have brought up so far.
Having eth0 and eth1 configured is more to do with redundancy and resilience rather than enhancing speed (because it won’t).
So that’s a new task on my list of things to do.
I’ve also spent a lot of time reading up on iLO2 problems, because I don’t like an unanswered query.
I kept these things in my mental pot, and mulled them over when I had lots of time (commuting!).
Last Friday afternoon I got a notification that one of the sshd components on server c1 – the public-facing physical host – had stopped working.
c1 was still ‘there’ and pinging away to my queries, but sshd wasn’t behaving exactly as it should.
Unfortunately I couldn’t access server c1 remotely, because whatever it was that was adversely affecting sshd, was blocking my attempt to remote on to the server.
Saturday morning I rocked in to the datacentre and accessed server c1 on console.
I queried sshd (service sshd status) and got a ‘not found’ response.
Hmm, sshd stopped working and shut itself down?
What could cause that?
I checked eth0 status (ifconfig -a) and got the responses I expected (found: eth0 configured to 192.168.1.4, found eth1 unconfigured state, found lo unconfigured state).
So, as I already knew from my ping responses, the server was actually still online, just not 100% there.
I then checked the devices attached to the LAN via the router admin panel.
The attached devices query found ILOGB8724JETY on 192.168.1.4 and (MAC address of c1 eth0 NIC) on 192.168.1.5 and – surprisingly – ILOGB87505WE7 on 192.168.1.6 and (MAC address of c2 eth0 NIC) on 192.168.1.7
This puzzled me.
All ports were forwarded to 192.168.1.4, but the router was telling me that the attached device on that ip address wasn’t (MAC address for eth0), but was the iLO2 address.
I checked the config on eth0 and it was definitely set to pick up a static address of 192.168.1.4
Puzzling!
Where had ILOGB8724JETY on 192.168.1.4 come from?
And why hadn’t the static address config on eth0 over-ruled it?
I decided to leave the iLO2 address alone for now and go for simplicity.
I powered down server c2 and unplugged it from the mains, to remove all traces it of from the network.
On server c1 I configured eth1 to a static ip of 192.168.1.6 and reconfigured eth0 to a higher static ip of 192.168.1.5
Rebooted the server.
Checked the attached devices in the router.
Sure enough, I suddenly – and for the first time – had a full house:
ILOGB8724JETY on 192.168.1.4
(MAC address for c1 eth0 NIC) on 192.168.1.5
(MAC address for c1 eth1 NIC) on 192.168.1.6
On a hunch I attempted to access and login to the iLO2 console.
Success!
And then I changed the port-forwarding rules to pick up 192.168.1.5, and saw that the sshd service was fully working.
Of course, it might have been the reboot that brought the sshd back online, but I thought there was more to it than that.
I wanted to test a hunch that was forming, so I reconfigured eth0 to 192.168.1.7 and eth1 to 192.168.1.8 and rebooted the server.
When I checked the attached devices in the router I still had a full house, but the addressing was updated:
ILOGB8724JETY on 192.168.1.6
(MAC address for c1 eth0 NIC) on 192.168.1.7
(MAC address for c1 eth1 NIC) on 192.168.1.8
Aha!
So iLO2 attaches to the LAN with an ip address of eth0 -1
Well that was a revelation (and thank you, HP, for probably burying this information in the reams of words about iLO2 and for not making it plain and obvious).
This discovery meant that all of the I/O traffic that server c1 had been processing for the last couple of weeks on 192.168.1.4 was actually being forced to the iLO2 NIC by virtue of the port-forwarding rules, and not being passed via the eth0 NIC.
I hadn’t noticed any speed issues, despite this misrouting, but I resolved to fix this.
I reset the ip addresses, rebooted the server and checked the attached devices in the router. and saw the (now expected) full house of:
ILOGB8724JETY on 192.168.1.4
(MAC address for c1 eth0 NIC) on 192.168.1.5
(MAC address for c1 eth1 NIC) on 192.168.1.6
Then I reset the router’s port-forwarding rules to pick up the eth0 NIC on 192.18.1.5
I ran some WAN tests, just to be sure, and was pleasantly surprised by the speed responses.
The bottom line here is that it looks as though an internal IP address conflict between the iLO2 and the static 192.168.1.4 for eth0 is what stopped sshd from running.
So this is a good result. I now have iLO2 running and I have detected and resolved the internal IP address conflict – and sshd is running normally.
