We live, eat, breathe and love monitoring here at Power Admin. So we’re always looking for ways to improve. One way is to scale our software to handle ever increasing loads.
Some of our larger customers (the largest anti-virus and computer security software company in America which starts with ‘Sym’ and ends with ‘antec’, and a very well known operating system company frequently known as MS) have networks far, far larger than our own. So we needed a way to at least simulate their needs in our software. One way we did that was by setting up our LAB_BEAST server 🙂
Multiple NICs
LAB_BEAST is a Windows 2012 R2 VM running on an ESX 5.5 host. Despite the name, we didn’t assign it huge resources, although it did get 3 NIC cards.
In Windows we see that we have the three network connections showing up nicely.
We assigned them 192.168.8.1, 192.168.9.1 and 192.168.10.1 respectively. Since we want them to be on the network with the rest of the servers we had to change our typical netmask from 255.255.255.0 to 255.255.240.0 which will give us a nice little range to add additional IP addresses.
Multiple IP Addresses
Now for the secret sauce – each network connection can have more than one IP address assigned to it! The steps below show how that is done.
So the secret is to add many IP addresses to each network connection. We found by doing a little research that Windows will apparently allow up to 255 addresses per connection. Just to make it simple, we assigned xxx.xxx.xxx.1 to xxx.xxx.xxx.250 to each. That means our lab server now has 750 IP addresses!
When we started we setup the first 100 IP addresses manually as shown above. That’s very tedious. But then we found this great post over at How-To Geek: How to Quickly Add Multiple IP Addresses to Windows Servers
So in our case we just needed to run these three commands:
FOR /L %A IN (0,1,250) DO netsh interface ipv4 add address “Ethernet” 192.168.8.%A 255.255.240.0
FOR /L %A IN (0,1,250) DO netsh interface ipv4 add address “Ethernet 2” 192.168.9.%A 255.255.240.0
FOR /L %A IN (0,1,250) DO netsh interface ipv4 add address “Ethernet 3” 192.168.10.%A 255.255.240.0
So now LAB_BEAST handles requests to these three IP address ranges:
192.168.8.1 – 192.168.8.250,
192.168.9.1 – 192.168.9.250
192.168.10.1 – 192.168.10.250
Essentially, this one server acts and responds as though it is 750 different Windows servers on the network.
Monitoring Load
You might ask how bogged down it gets. So far we’ve setup monitors to check for: Ping, free disk space and a variety of performance counters on all of those IP addresses. As you can see in the PA Server Monitor charts below, LAB_BEAST is not even struggling:
What’s even more surprising, is we have an external Satellite monitoring some of these IP addresses as well, so the server is actually getting hit as though it was 1000 separate devices. Amazing.
Now, we chose those three particular monitors for a reason — they are very light weight. Monitoring services and the Event Log is a little heavier. Monitoring services and the Event Log for one server is no big deal — the server doesn’t even feel it. I suspect multiplying it by 1000 might have some impact – that’s a test yet to come.
What’s next?
Monitoring 1000 server/devices is nice. But our customers monitor MUCH more than that. And all from a single PA Server Monitor installation. So we’ve setup a few other servers like LAB_BEAST to push the limits.
We know from speaking with customers that a single PA Server Monitor installation can easily monitor 2000+ real servers/devices from a single installation. We’ll see how much further we can push it 🙂
