The IT world has been in shock as we’ve all learned about the Solarwinds hack
(UPDATE: And now also the Kaseya hack). Network and server monitoring software inhabits a special niche where it often has full access to the servers and devices on the network in order to monitor all internal resources. Because of this it needs to be as robust, safe and trustworthy as possible. Given the fall out of the Solarwinds hack, customers have been asking us about our product safety. We’d like to assure you our software is safe and explain the steps we take to keep it that way.
When we talk about our products, it’s useful to split the files that make these products into two types of interest – those that are executable (.EXE, .DLL and .SYS files) that we produce, and executables from other vendors.
Executables from Other Vendors
We have recently completed an audit where we have checked every single executable file that we get from vendors to ensure they haven’t been tampered with. Some of them were already signed by the vendor so we know they are OK. Others we have tracked back to their source and compared to ensure they haven’t been altered. We keep all binary files in our source control solution (something not all companies do) so we can audit and see if any of them ever change. For the 3rd party executable files that were not already signed, we have done a one-time signing of them now that we have audited them and found them matching the originals.
Our Executables
Our executable files are those that we compile from our source code. There are a very few that are not compiled at the time of each build – those were signed the last time they were built, and then not touched again. For the files that are compiled each time, the .EXE and .SYS files have been signed for a number of years. Starting now, we are also signing all DLLs that we compile.
Source Code
According to an analysis from Microsoft, Solarwinds was actually hacked at the source file level. Hacks such as this require obfuscating certain bits of the code to hide critical details, such as external hostnames that will be contacted. Sources files written in .NET languages have a large library of possibilities for obfuscation such as compression and encoding/decoding data. We don’t use very much .NET in our products, so we were able to look at every one of our .NET source files manually to check for correctness, and nothing unexpected was found. Note that in the Solarwinds case 4000 lines of new source code had to be introduced for the hack, so it would be easy to spot manually in our smaller .NET scenario. For C/C++ code, extensive built-in libraries don’t exist and have to be added manually. We’ve ensured no additional libraries have been added, and then checked all code that does compression or encoding/decoding of data and found nothing unusual.
All source code is in our source control system. One nice side effect of being a smaller company is we all know each other and know what each other are working on. Any unusual file changes would stand out and attract attention because each time source files are committed to source control, a notification goes out to the Slack channel letting all developers stay aware of changes.
We have the compiler turned up to the highest warning level, and never allow warnings or errors to go unfixed in order to help catch bugs. In addition, we use static code analysis tools to further audit the source code for any potential bugs – we want the code to be as robust as possible.
In addition to the above, we have automated monitoring watching project configuration files, resources files, source code, and other inputs into the build that sends out an email every few hours reporting files that have changed. It just takes a few seconds to scan those emails so we are disciplined and actually read each one.
Build Checks
Now that all 3rd party executables are signed (either originally by the 3rd party or one-time by us), and all of our executables are signed at build time, our build system verifies all executables are properly signed as part of the normal build process.
Our Build System
Our build server is only accessed by one person. That server has our AD Login Monitor on it so we can see who accesses it.
In addition, we have our File & Directory Change monitor watching files that shouldn’t be changing so we’ll receive alerts if they do change.
Nothing in our process automatically downloads or updates components. We’ve been concerned about supply chain security much longer than it has been fashionable, so we’ve always taken the methodical and manual approach to checking updates as they come out.
Network Security
To help prevent attackers from getting into our network in the first place we have a number of protections. Anti-virus applications are used and pattern files are kept up to date. Windows Updates are regularly installed on all workstations and servers. Windows Firewall is on and running on all computers. We use two-factor authentication for remote access, and each user can only access their own workstation. Of course, being a monitoring company means our software is on multiple servers (test and production) which provides overlapping monitoring and alerting of all systems. We are taking this opportunity to further enhance our internal monitoring and have setup additional alerts watching for suspicious changes.
Our production web, support and mail servers are hosted remotely and not on the same network as our development and office systems.
We have always taken security seriously, and take recent events as a chance to look at our systems again to see how we might improve further.
