What If...?

It must've been around December - the time of year surrounding personnel atrophy due to pending holidays, when my team lead suggested I use the rare slowdown to look into building a locally-cached software repository to upgrade our aging systems by bypassing WAN cloud servers to greatly speed migration and reduce outage duration. It was a rousing success. Wasn't long of course until someone asked, "What if..?"

I don't remember the exact question, but its nature is ever-present in the tangible viscosity of production operations and almost always surrounds some aspect of increased productivity, greater uptime, diminishing outages, or the verbalization of any number of vague nuisances which could eventually lead to the kind of re-framing which reverses that elusive malaise which occasionally clouds inspiration. Pattern-recognition reveals I rarely need further prodding.

As a global corporate entity we have a robust operating arena replete with sandboxes, quality-assurance boxes, development servers, and of course production. What we lack however, is a lab. Surely it could be argued that such a rich development infrastructure negates its necessity, especially given the diverse talent our highly-integrated professionals demonstrate daily in breathtaking concerts of mutually-supportive teamwork. If diversity is our strength, encouraging the intrinsic motivation of the individual with the freedom of experimentation - and failure - absolutely has the potential to synthesize collaborative innovation. It is one of the reasons I actively choose to continue my position; supportive management which fosters this unique culture.

To better answer the, "What if..." question, I procured a small Enterprise-level server and installed a hypervisor along with my own locally-cached repository. As the list of virtual machines grew, so did my need for DNS, and storage. I procured another server and built an open-source NAS. Then added SAN emulation and installed some fiber-optic network interface cards to more closely replicate our boot-from-SAN environment. As these things often happen, at some point I realized losing everything would be quite devastating, so one more server and an open-source Enterprise backup solution. This required another tier of storage, one geared more toward volume and retention over speed and throughput. Another server, another open-source application. This lead to a desire for monitoring which then necessitated a mail server. The list just grows and grows. It's exciting.

It also gives those of us who have been forced to follow the industry best-practice of moving away from holistic datacenter maintenance into more specialized fields the opportunity to keep our ancillary skills honed; I was able to leverage a newfound skillset during our latest disaster recovery exercise, greatly increasing its success - a wonderful feeling when professional growth positively manifests as a benefit to the entire team. I've since added my old UltraSPARC, Itanium, and MIPS boxes into the mix (if anyone has a spare pizzabox PA-RISC or SPARC sun4c/m hit me up!) as these systems are often still supported in datacenters as legacy systems.

There is an unstated satisfaction which comes from utilizing the fruits of one's labors in hopes of illustrating value to management, and while having the speed and flexibility to quickly answer those, "What if..." questions no matter the form they may take - right at your fingertips - there is always room for improvement, and I'm excited to see where each iteration leads.