Monthly Archives: March 2015

Server c1 is done.

The OS is installed.

The environment has been virtualised.

MySQL and Postfix installed.

The environment has been pen/security tested.

Four (client-facing) VMs have been built and are being used by various people, trying to break them.

Three layers of firewall have been implemented (2x physical, 1x software).

As far as hosting BaaS data goes, the environment feels very close to being absolutely right.

And I have to say that the environment is very fast.

I would like to put some time and effort in to practicing building VMs for FQDN hosting.

I guess that’s what I’ll be doing this week.

Somewhere around Wednesday evening, about 72 hours after I fixed the remote SSH problem by changing the Plusnet-supplied Sagecom router for a Netgear router, all port 80 and all port 22 calls to the server c1 started being dropped.

There was nothing I could do, because I was down in Bristol and server C1 needed an onsite visit back at the Nottingham datacentre.

Frustrating!

Eventually the weekend rolled around and I tottered off my sickbed in to the datacentre to begin explorations.

Server c1 is an HP DL380/G5.

It had just one (500Gb) disk, which contained all the CentOS 6.6 goodies that had been rolled out so far.

Which wasn’t much cop, because server c1 wouldn’t stay alive.

When I walked up to the cabinet, c1 was definitely receiving power, but was switched off.

I pushed the button and it whirred and whined, noisily, to life.

The console showed me the usual boot sequence.

Then server c1 just powered itself down.

I tried again; it booted up. This time it got as far as the CentOS login prompt.

And then powered down again.

Long story short, I removed the PSU from server c1, cleaned all the PSU and serverside contacts, and replaced it.

The server booted up and stayed up.

I logged in as root and performed some basic functions.

Everything looked fine.

Rather than leave things like that for the week, I decided I’d like to add some extra resilience to the situation.

I removed the PSU from server c2 (another HP DL380/G5), and slotted that in to the spare PSU bay in server c1 (the HP DL380/G5 servers have the capability for two independent PSUs running at the same time).

So server c1 is now running two PSUs, and I’ll keep an eye on the server logs to see if the original PSU drops out, or if there any more powerdown problems.

I spent last weekend working through a real pain in the arse problem: couldn’t get remote SSH access configured on to server C1.

Local access via console worked brilliantly.

And I could attach another device to the internal network and run SSH sessions to the internal IP that server C1 had been configured with.

But I couldn’t get SSH consistently working, in a stable, always-up, kind of way, via remote.

The best I could get was for remote SSH to stay up and running for around 20 minutes.

In the end, frustrated beyond belief, I binned the router that Plusnet had supplied (a neat-looking Sagecom device).

Then I looked out a spare Netgear router that I had at home.

I copied the config details from my home-hosted NAS in to the spare Netgear router, and installed that in the datacentre.

Changed the account credentials to match those of the datacentre, obv.

And lo and behold, I had remote SSH.

But for how long?

An hour later it was still working.

This was a new record.

Three hours later it was still up and running.

Twenty-one hours later, we were still golden.

So it seems like, to me at least, that the Plusnet router had a ‘go to sleep’ rule set in the firmware.

If it hadn’t seen any port 22 traffic for around 20 minutes, it shut down port 22.

And wouldn’t wake up when port 22 traffic came knocking on the door.

Marvellous.

Except not, obv.

But the Netgear router fixed that and I had remote SSH and public port 80 access to the server C1.

For 72 hours.

It’s been a mixed bag on the datacentre project, this weekend.

I feel that I’m about half a dozen steps further forward, and have only taken one or two steps back.

But it has been a weekend of problems.

The biggest obstacle to making progress actually took me a while to realise exactly what the problem was.

I had downloaded CentOS 7, as this was to be my operating system and virtualisation agent of choice for the hosting servers.

I’d set aside Saturday as the main day of installation.

I inserted the DVD media containing CentOS 7, and booted up the first server.

The system went through its normal start-up/boot sequences, and I took this opportunity to set the iLO2 config.

Then I set the system language, keyboard language, timezone and country settings.

And then the OS wouldn’t let me go any further because it said I had no storage space.

Except the server had half a terabyte of storage, live and flashing a green light at me.

I ran through the boot cycle five times, and each time the OS said I had no storage, and the flashing green light continued to contradict.

I stepped through the server boot sequence, and sure enough the array controller said there was plenty of storage space.

So I did a google, and you know what?

It turns out that CentOS 7 has a compatibility problem with the HP DL360/G5 array controller.

So I downloaded CentOS 6, and burned that to DVD as an iso.

Some hours later I put the CentOS 6 media in the server DVD drive and booted up.

Success!

After the installation, I ran the yum update command except it wouldn’t run.

I tried several commands for online activity and none of them worked.

A bit more googling told me that by default CentOS 6.5 produces a closed server – unlike CentOS 7, which is the product I’ve been doing all my reading on.

CentOS 6.5 needs to have the eth0 and eth1 ports opened by the root administrator.

I did this, and then ran yum update, and downloaded and installed 79Mb of update packages.

I rebooted the system and then successfully pinged a FQDN or two.

Then I shutdown the server and called it a day.

I had intended to get as far as enabling remote access via SSH, but I haven’t even got in to Firewall rules and Securitisation.

And I know that’s another solid half-day of effort.

I’m guessing another 10 hours to bring just the first server in the cluster, to where I want it.

So that’s next weekend then.

I’m looking for a very experienced, remotely-located freelancer for some ad hoc work on a small datacentre.

The skills required are:

• CentOS
• DNS Server
• LAMP admin
• PostFix
• MariaDB/MySQL
• php
• Perl
• Virtualisation
• VPN

The work is in two areas:

1. Project delivery (to assist with consultancy and advice, and, if things go wrong, to take a hands-on role in installation, setup, config)
2. Ad Hoc support (on an ‘as required’ basis)

The salary will be an agreed hourly rate, paid by whatever means you prefer (PayPal, etc).

I’m not too fussy where in the world you’re based – timezone parity isn’t a big deal for me.

If you’re interested in the role, please drop me a line in the comments box, and I’ll email you back, and we can take the conversation forward.