The Empirical Manifesto

Back in the day we had just about every fantasy, comic book, and sci-fi fandom covered in our server naming conventions; more often than not clashing when they let us - the system administrators - name the data center servers and the workstations on our desks. Daily, we'd log into wookie from elrond to check the backups from batman the night before. Long gone are those days not only in corporate, but even in my own home lab. I have 150+ entries in my DNS server and all hostnames are utilitarian in nature because there's just too much to keep up with. While it might not be as much fun to log into smt, rmt, susemgr, sles12sp4, dnsmasq, bkpsrv, and hpvm4 its a hell of a lot easier than when I was younger and had fewer boxes.

There is one notable exception: belanna.

My first unix server was a comically heavy HP 9000/755 workstation - with those enormous full height SCSI drives which was reminiscent of turbofans found on large, military transports during initial power-on. I don't know why I settled on B'Elanna as my first server name, but soon a slew of Sun Solaris boxes surrounded her; kes, sesksa, janeway, and 7of9 (yes, even after all this time I remember them all fondly; SPARCstation 5, SPARCstation 10, ULTRASparc 60, and an EnterpriseSPARC 250). doctor was my first IRIX box, an Indigo 2 I no longer have, and the Indy sitting adjacent me as I author this, has inherited its namesake.

At some point I grew weary of lugging around that ridiculously heavy 755/125 with its special HIL-keyboard and mouse and awkward sync-on-green coax-connect monitor - and traded it in on a beautifully light and functional HP 9000 712/80. It was impressively quick given its age when it was running 10.20, much less so when running 11.0. I was like a madman with the upgrades stuffing a 10k RPM drive in its minuscule chassis and maxing out its RAM. The fact that it could use a standard PS/2 keyboard and mouse and regular VGA monitor gave a whole new meaning to the word, "portable." This was belanna for many more years.

I wish I had kept all my pizza box form-factors, I do. I have so few regrets in my life, and that is one of them. I had amassed so many 64-bit boxes, we'd decided to ditch every 32-bit machine we had. Later (and I don't remember the exact circumstances in which Mr. Ernest Cody suggested I take home a couple of rx2600's he'd acquired), belanna evolved to Itanium and has been running on and off ever since, and what a workhorse she's been! I used her to teach myself IVM (Itanium Virtual Machine - later HP Virtual Machine), Service Guard clustering, and have pressed her into service every time we've had an immediate need at work for an HP/UX machine (you may be surprised how often that happens, even now). And this is why every login, and every hpvm carries the notation, "HEC Foundation."

She's down a bank of RAM these days, and HPVMs disallow her from unlocking her hyper-threaded cores, but she's currently running four 11iv3 hpvms for training purposes, and I couldn't be more proud of her.

SUSE Linux Enterprise Server (SLES) and OpenSUSE have recently been including, free-of-charge, the entire IANA registered port list in /etc/services! Long gone are the days of old when we either had to commit ports to memory, or look up the IANA registered ports, and manually add them to the /etc/services file. Because really, who wants to do all that time-consuming labor? In short, me! Why?

Because its more work to comment out the 999,999 services I don't need, than to add the couple hundred I do need. Especially when the couple hundred conflict with the entire IANA registered port list pre-populated into the /etc/services file. Now what I could see as being awesome is including the entire IANA registered port list already commented out because that would allow a quick-and-easy lookup, and a quick-and-easy port entry with a single keystroke. But that's not what is happening here.

So I have a list of 57 non-production hosts (not even going to count the production numbers yet), with 57 discrete /etc/services files, all requiring 397 blocks of conflicting ports disabled. Some suggested awk, some suggested sed, but each file being unique made attempting this arduous.

Until one enterprising young man introduced us to vi substitution using absolute line numbers.

:set number

:6015,6030s/^/#

This essentially replaces (s) the first of the line (^) with a comment (#) for all ports listed between lines numbers 6015 and 6030.

( example )

Things have calmed down now that the builds are complete, the server is patched, and all the user accounts have been created. I still need to get with (as of yesterday) a former co-worker on the best way to patch the HPUX HPVM - I'm convinced it should be easier than it is, and I'll have the added benefit of a best-practices outside an ignite sever landscape. And speaking of former co-workers I finally touched base with yabanjin62 after far, far too long.

You know what else has been a long time? Seeing a CDE login screen! The Itanium server accepts Radeon 7500 series framebuffers and I just happened to have one laying around. I immediately modified the login screen (below) but while fun and wistful, I have zero use for CDE - I can launch X from my SUSE VirtualBox VM, and redirecting output to the screen disabled the MP console!!! At some point I'll have to find the EFI path to redirect it back, but right now, its kinda nostalgic.

Downloaded and unziped depots from the HP/UX Porting and Archiving Centre (because hpux still doesn't have an online package management/repository)...

[/] root@belanna# cd /var/tmp/packages

[/var/tmp/packages] root@belanna# for x in `ls`; 
do swcopy -vv -x autoselect_dependencies=false 
-x enforce_dependencies=false 
-s /var/tmp/packages/$x \* @/var/spool/sw/PORT_ARCH; 
done

[/var/tmp/packages] root@belanna# swinstall -s /var/spool/sw/PORT_ARCH \*

[/var/tmp/packages] root@belanna# swinstall -s /var/tmp/packages/depothelper-2.00-ia64-11.31.depot \*

FREAKING KORN SHELL...

/etc/.profile

stty erase "^?"
export PS1="[\$PWD] $(whoami)@`hostname`: "

...to look like BASH (which is also installed)


PS1="[\w] \u@\h: "

SMH = Port:2301

   .-------------..--.                   _.--------.___
  |  ==========> ||   )             _.--'              `--._
   `-------------'`--'           .-'   \                    `-._
      /--.     /               .'       \.--------.__           `.
     /---'    |      _____.--.'`-.__     \  ___      `-._         `.
  .-'   ____.-------'      .'     __`-._.--'__ `-.       `.         \
.'  |.-'    ___________.--'------'  _ |  .-/[_],  `.   |   `.        \
|---' | .--'  _____________[|___   | || |]]]/ \|   |   | .-----.  .--.|
|---. | `--.___________    [|      |_|| |]]]\_/|   |   | `-----'  `--'|
`.  |`-.____           `--.------.__  |_ `-\[_]`  .'   |   .'        /
  `-.       `-------._____ `.     __.-' `--.___.-'      _.'         /
     \---.    |           `--`..-'       /         __.-'          .'
      \--'     \               `.       /`--------'            _.'
   .-------------..--.           `-._  /                   _.-'
  |  ==========> ||   )              `--._          ___.--'
   `-------------'`--'                    `--------'

HP-UX belanna B.11.31 U ia64 1645094063 unlimited-user license

Wherever does the time go?

I've started reading The Hobbit to my children. Again. There's not many of you left here who remember when I attempted this back in 07. They weren't ready then - but despite my youngest's protestations, she managed to giggle quite a bit at the trolls in Chapter Two. Or at least the Cockney accent I gave them - either way, its a start.

My next door neighbor, who is a gadget-freak and bought the iPhone 4 the day it came out, sold us his 3G. The timing was perfect, as my wife's phone was getting a little long in the tooth and she'd mentioned looking to replace it. I simply put her SIM car in it and she's up and running - no data plan, but we have wifi at the house so she's thrilled. I'm only a little jealous, because its way cooler than the blackberry.

I dreamed last night two of my long-time coworkers, both with engineering backgrounds (Yo, SJ! & Ernest), had discovered a new unix operating system and had written a program for it. They were eager to try it out but had no hardware. I'd come home late to find them standing in my house arguing, so dusted off my rx2600 to present to them. It loaded like IRIX but looked a lot like BeOS. Weird.

I was late because I was in high school again! God I hated that. I was all like, "Now what." Anyway, so there I was, much as I am today, 40-years-old, and I'm in high school. I found I had zero tolerance for the drama, so just cynically made my way from class to class without speaking to anyone. My only friend - Chris Richardson - another real life coworker - was a band geek, and I didn't want to be caught dead anywhere near the band room. Or whatever that place is called where they practice their instruments. By the time I found him, most everyone else has left - the parking lot was mostly empty. We both lived in Rhome (which is where I *actually* lived when I was attending high school) which is why he was riding with me, but the high school was MASSIVE and had seven inter-connected parking lots...My 1968 Dodge Coronet 440 was in the last one.

I'm burning the covers of my spooky albums onto the surface of their respective discs with lightscribe, but as I inserted Interlunation: Lullabies of the Damned it was identified in Gracenote as "Four-Day Workweeks: Legal And Practical Guidance For Employers."

I've been working odd hours and weekends, evenings, overnights and whatnot this week and last, not to mention a summer cold which knocked me out of work two days last week. Once again I find myself behind the curve where my livejournal FL is concerned. Thankfully a lot of you post also to Facebook and Twitter which both go directly to my blackberry, but I do plan to settle in this weekend a little bit and catch up.

And maybe drink a little beer.

So not only have I managed to unofficially create a 'linux group' with the Wintel guy here at work (and for a unix shop, that's big news) but I'm now also (apparently) the Integrity VM dude - that is, HP/UX's proprietary virtual machine solution. Something we've never used, let alone deployed before. This came about rather abruptly when one of the projects found they were a host short during the discovery of a flaw in their design. They've asked us to completely engineer a VM environment while trying to maintain our current project dates. Yeah, that second part - not gonna happen. Anyway, after an entire day of setbacks...

Historically, the filesystem on HP/UX which houses the kernel configuration and binary files was required to be HFS - HP's legacy Hi Performance FileSystem. That has changed in 11iv3 (11.31) but by default, the system is still created with an HFS /stand partition. Unfortunately Integrity VM won't install on any box which has even a single HFS filesystem.

So this week I'm deploying an Integrity VM host and two guests on an IA64 Blade chassis.

Firstly, there are three ways to convert a filesystem from HFS to VXFS with varying degrees of downtime, length, and accuracy. Here's the FAST AND DANGEROUS way which worked like a chiz-amp!

# grep -i hfs /etc/fstab
# umount /stand
# fsck -F hfs /dev/vg00/lvol1
# fbackup -f /opt/stand_fbackup.090915.egh -i /stand
# vxfsconvert -y /dev/vg00/lvol1
# vi /etc/fstab
# fsck -F vxfs -y -o full /dev/vg00/lvol1
# mount -o rw,suid,delaylog -F vxfs /dev/vg00/lvol1 /stand
# fsadm -ed /stand
# frecover -r -y -f /opt/stand_fbackup.090915.egh

** IMPORTANT **
Don't forget to run the restore before you reboot, else you've lost your kernel and there's no easy way to mount up your volumes like in Solaris. Yeah, I had to re-ignite.

Secondly, 11iv3 "unlocked" the hyperthreading on the Itanium chips, but Integrity VM will not run with it enabled. Another kernel parameter change, another reboot. And for those of you not in the know - HP/UX servers do not reboot quickly.

Now for the nuts & bolts. We have one of those cutsey point-and-click SAS array SANs; I previously created two mirrored volumes - one for each of the two VMs (raw SAN devices can be presented). I have to first create the first virtual switch, start it (-b) and create the first VM:

# hpvmnet -c -S switch1 -n 2
# hpvmnet -b -S swtich1
# hpvmnet -v
Version B.04.10.00

Name     Number State   Mode      NamePPA  MAC Address    IPv4 Address
======== ====== ======= ========= ======== ============== ===============
localnet      1 Up      Shared             N/A            N/A               
switch1       2 Up      Shared    lan2                    [IP ADDRESS] 

# hpvmcreate -P hostname -a disk:scsi::disk:/dev/rdisk/disk9 -a network:lan::vswitch:switch1 -B auto -O HPUX:11.31 -c 4 -r 10G
# hpvmstatus -v
Version B.04.10.00

Virtual Machine Name VM #  OS Type State     #VCPUs #Devs #Nets Memory  Runsysid
==================== ===== ======= ========= ====== ===== ===== ======= ========
hostname                1 HPUX    Off            4     1     0   10 GB        0

Now all we have to do is fire it up, and gain access to it through a virtual management processor (hpvmconsole -P hostname) in order to tell it where to boot from to install an operating system. In this example, we're going to boot it from the same ignite image from which we installed the host O/S. But first we need to create a dbprofile followed by the boot command. Sadly, in the virtual console you can't turn off xchar :(

dbprofile -dn ignite -sip ignite_server_ip -m netmask -gip gateway -cip client_ip -b /opt/ignite/boot/nbp.efi

lanboot select -dn ignite

Let me try to explain to you just how freaking fast this performs as compared to a standard Ignite. I was downright giddy. I've never seen anything install this quickly. My Itanium box at home? THREE HOURS from DVD. A standard Ignite? ONE HOUR. This took half that.

So it took me 12-hours to grasp the technology and deploy it, but I can now clone this one as many times as I need, anywhere, configuring CPU/memory on-the-fly (hpvmclone). I simply boot it, run a set_parms against it, and its its own machine.

Best part?

I don't need a GUI which only runs on Windows.

Boy was suffering from a variety of infections due to a decreased immune system brought on by the flu, which he apparently contracted without us being aware of it. The doctor put him on bedrest for the entire week and I worked from home - for a week. A very long week. I've been hopped up on MONSTER and scooby-snack, and last night I was dancing the Numa Numa Dance along with that chunky kid from NJ for hours on end after bsdcat sent me the URL.

Usually, I don't like working from home unless I have something to do, so I begged for work, and it was given to me tenfold. Perhaps more than I wanted to do, nonetheless, I managed to accomplish everything by Friday afternoon. I had to get a environment ready for production. Fortunately the machines had already been built, so all I was doing was readying them. It was pretty basic sysadmin stuff, but the proof, as they say, is always in the pudding. No, the manually intensive part boiled down to access:

• One terminal concentrator


• Seven ports


• Two domains per port


• Half a frame on each port


• Six servers on each frame

Each requiring a root passwd reset (I was provided a list of 50 possible previous root passwords to work from) and thus began my access hurdle. I knew I wanted tabbed terminal windows for this activity (opening each frame's six servers in a single window) and PuTTy is a touch behind the times on this, and my work laptop which is a dual-monitor setup running ubuntu doesn't do well on my awkwardly aspected 16:10 widescreen home monitor. So...I turned to OSX which handled the job flawlessly. That is, once I found a VPN client for OSX, imported the settings, and killed all the CPU-sapping processes (telnet [to my HP/UX box's (hereafter referred to as belanna) Management Processor] had apparently hung driving utilization to 100%). But then that's why I have a KVM, and my stalwart XP box - for the heavy loading. I really would like one of those new Psystar boxes. Oh the wonderful things ehowton could do!



New 320MB/sec SCSI drives in the previously empty drive slots


Ever since harddrives went over 9GB, I stopped breaking out discrete filesystems, a practice which has served me well in Solaris. Not so much in HP/UX. My 11iv2 (11.23) installation filled up, just prior to that feudal hardware-excreting lord Ernest asking me to perform an 11iv2 to 11iv3 (11.31) upgrade out of curiosity to see if the full-version VxVM which was included on the "Mission Critical Operating Environment" (MCOE) would be retained. Because this coincided with work's decision to start rolling out and testing 11iv3, rather than repair the existing filesystem problem prior to performing the upgrade, I just installed 11iv3 standalone, and was surprised to discover in the "Data Center Operating Environment" (DCOE) a plethora of installations options ranging from a full-scale High-Availability Suite, to a workstation-level install ("Technical Computing Operating Environment" (TCOE)) - which not only includes the full VxVM, but now also bundles their Auto Port Aggregate (APA) NIC-load balancing/failover software. There are probably many other packages included that I'm not yet aware of which were previously unbundled, but for me, the sweet spot was the inclusion of the VSEOE, or "Virtualization Server Operating Environment" which not only fully supports virtual partitions (vPars) and virtual machines, but 11iv3 now takes advantage of a previously unknown feature of the Itanium 2 (ia64) architecture: Hyerpthreading! In theory (and I'm a long way from practical application) I would be able to carve out a linux instance, running within HP/UX, and assign it a portion of a processor.

Here's where things get dicey. In Solaris, any machine which can run Solaris 10 is capable becoming a host for other Solaris "zones" or virtual servers. With HP/UX, only specific hardware is supported. Now I'm no fool and learned a long time ago that 'supported' is merely a term used by companies when they no longer want to technically sustain an application. In other words, just because its not supported doesn't mean it won't work. HP/UX vPars however, while different from nPars (electrically-separated servers in the same machine) are not "zones" as in Solaris. They require a specific 'frame' in which to reside, and apparently the rx2600 *actually* won't run them. But I'm still running that down. Also, the VSEOE's "virtual machine" being mentioned outside of vPar leads me to believe that a Solaris zone-like build is now possible.

Lastly, SAM has been almost entirely deprecated, having been replaced with SMH - System Management Homepage. As I'm running this headless, I've only seen its cli interface. If you can believe it, its actually slower than SAM, but does seem to have a more intuitive interface.

---

One last story about Ernest. Today I rec'd an invitation to "connect" with this attractive lady on Yahoo:





Yet she has no profile and I don't know who she is. How did she find me? What does she want with me? I'm all for expanding my friend base and interacting with new people - I actively seek it out - but with no note or clue as to who she is, or why, frankly I'm baffled.

Which brings me to Ernest. Awhile back his wife found my Facebook page and invited me to join her network. Before I could accept, however, I was directed to chose from a small set of circumstances which explained how I knew her. Unable to proceed without this data, and without an accurate choice for depicting our relationship, I chose what I thought at the time was the most ambiguous explanation, "We hooked up."

Apparently in the parlance of our times, that was the wrong box to check.

I plugged an Ethernet cable into the replacement rx2600's Management Processor and waited for my router to provide it a DHCP address. Using telnet, I supplied the default username and password and gained access into the standalone daughterboard, familiarizing myself with the menus before using the embedded ARM processor equipped MP to boot the server. It came alive from my laptop, connected to the wireless router, and I was able to bring the box to life from the comfort of my bed, and without pants.





I went through the installation process to identify any issues, discovering that the two processors ran at 1.3GHz and that all 3GB of RAM was showing up. I'd used the Technical Computing Operating Environment (TCOE) as the install medium as it contained a full Vertias Volume Manager (VXvM) install instead of HP/UX's less-robust LVM, and I'm woefully inadequate with VXvM. When the installation was complete I went through several troubleshooting steps on the other chassis and determined that not only had both power supplies failed, but the MP was also either locked out or non-functional. I could've pulled my Wyse Terminal from the garage, but I don't know where my null modem cable for it is. Probably hanging off the back of my rack now that I think about it. I got the rx2600 (now named belanna) racked and reinstalled.





Outside of this activity, which took most of the day, I've been listening to the fourth and final book of the Twilight saga, Breaking Dawn. I can read much faster than these audio books, and they really slow me down. Its soooooo frustrating. Well, that and this book is teh bore. 28 hours worth. I planned to get the last book from my mother-in-law last weekend but she hadn't read it yet, so I downloaded the audiobooks. Its nice to have them, just in case anyone else wants them.





My wife has spent the last two days in my boy's room re-imagining it. She's turned it into 'Shadow the Hedgehog' with paint; wild black points outlined with orange and obtuse orange triangles surrounded with red. All this upon his gray walls. It looks really good, but I think its wearing her out. I have next week off, she doesn't. She comes to bed after I'm already asleep, and is at it again before I awake.

I hope everyone had a nice Christmas.

Drank precisely one ounce of Lucid last night with a helping of ice cold spumante and slipped into a dream. An uneasy dream about my recent past, but surely that was just images from the disturbing storylines in the third season of Criminal Minds courtesy mr_dowg. Surely.

This morning I slept in exactly two hours, gave my undercarriage a bit of a 'how's your father' after consuming nearly a full pot of coffee my wife had left for me (Thank you sweetheart!) and finished my morning with eggs before leisurely driving into work, where my day began at 1000 hours.

In a tremendous move of glory, that magical man-beast Ernest provided me a 64-bit Itanium rx2600 as a replacement for belanna. I am thrilled beyond words. I snagged a copy of HP/UX 11iv3 (11.23) and despite having to work this Sunday, will attempt to get everything in place to begin building this out. As cameras are not authorized in the building or parking lot, I can't provide you an image of it sitting in my car, so please accept this vendor photo as my offering until I can take my own:





The day has passed far too quickly for me.

I'll be the first to admit that Blastwave's pkg-get (for Solaris) and Debian's apt-get have spoiled me rotten. With the issuance of a single command, the pre-compiled binary of your whim is automatically paired with its dependencies, checked against already installed programs, and installed, along with any other necessary packages if need be. Seconds after hitting 'Enter' the program is already auto-configured and ready to use, or in applicable cases, the daemon is already not only running but sitting in your init.d directory as well.

BRILLIANT!

Gone are the days of meticulously pouring over patchsets which affect your particular environment as prudent security practices now dictate frequent and massive patch 'bundles' be installed to thwart any suspected threat. I love it, I really do. And while I was one of the last people on earth to jump aboard the linux bandwagon, I now sing its praises aloud to the people from its platform. Timely GNU ports to Solaris and...vise-versa? Previously unheard of a decade ago, yet here we are. The winds of change have constructed very exciting times for the likes of us, and I plan to stick around awhile and see how it all plays out.

However...

There are those ancient monoliths which remain with us to this day. Large, imposing proprietary systems whose very name is spat from the lips as if a curse: SGI. (less so since they went linux, but since they have no concept of marketing...) AIX. HP/UX. While their 'unofficial' GNU binary sites were, once upon a time, on par with http://sunfreeware.com many packages have not been updated, or worse - removed. But in SGI's defense, I will say this - installing packages directly from the webpage was freaking awesome!. Mind you, all the same dependency woes were there...actually, I can see that being much easier now with tabbed browsing, but I digress.

WHY ARE THERE NO ONLINE AUTOMATED PACKAGE MANAGEMENT SYSTEMS FOR HP/UX?



Please state the nature of the medical emergency.