Saturday, January 31, 2015

Actually, it mostly is a desk job

In all honest, it really is a 95% desk job.  The vast majority of my days are spent staring at three monitors (and usually at least one laptop), often trying to make sense of multiple different bits of information coming in from different sources.

Network Administration isn't as much of a Break/Fix job as being a PC Technician or to a lesser extent, System Administrators.  PC Techs fix the computers, System Administrators fix the servers.  And I fix all the stuff that connects them together.  But if I'm lucky, the stuff I manage doesn't actually break that often.  Networking is more of a build-it-and-leave-it task, and anything that's completely broken is 99% exceedingly simple (someone unplugged a cable) or that 1% that is ridiculously convoluted (when a strange routing error was sending traffic from a specific server over the VLAN that should only be for our wireless access point management).  Most of my day is spent working on projects of various sorts, upgrading things, making tweaks and adjustments to better suit different situations, and poking at long-term projects.

So as for what that IT guy in your office who's always sitting on his butt and staring absentmindedly at his screens is actually doing, here's what's on my plate right now.

  • Statically assign some IP addresses to a bunch of dataloggers out in the field, then open specific holes in our firewall to allow science groups to access their equipment here on station from their universities back in the states.
  • Upgrade the operating systems on a handful of switches and routers that are on our separate "Monitoring and Control" network, which is used to manage our satellite connection.
  • Update the ACLs (Access Control Lists) and SNMP (Simple Network Monitoring Protocol) traps for those devices as well, to bring them in line with the rest of our network and allow us to monitor them better.
  • Write some documentation
  • Figure out how to move the MAC address filtering for some of our wireless networks off our WLC (Wireless LAN Controller) and onto our ACS (Access Control Server). This is particularly complicated as the devices that we're MAC filtering aren't on the domain, nor do the users have domain accounts.  I've got a few theories about how to accomplish this, it's a matter of testing them and figuring out which is the best for long-term manageability.
  • Rebuild most of our automated Cacti graphs that we use for monitoring bandwidth utilization at different points around our network, after a glitch last winter broke about 70% of them.
  • Write more documentation
  • Go over to our weather and aircraft control facility and wireshark the connections on the two computers that are supposed to be able to use our backup WAN link if our main connection goes down, and try and figure out the routing problem that's resulting in them sometimes using the backup connection for specific web sites even when our main connection is still up (Even though we've got the costs set high in EIGRP, for some sites it still seems to use the backup when trying to access our severs in Christchurch)
  • Replace a 24 port switch in another building with a 48 port, to allow for the eventual expansion of the office space and installation of a wireless network.
  • Figure out why our WLC isn't doing automated backups (probably an authentication issue between it an our FTP server)
  • Write even more documentation
  • Install some new WAPs (Wireless Access Points) into our bars and coffee house, for the new iPad-based Point-of-sale systems.
  • Trunk the dedicated Financial VLAN to the WLC, create a new virtual interface inside the WLC and set those Point-Of-Sale access points to send all data through that VLAN.  Also will need to configure a DHCP scope for it.
  • Configure that SSID for the POS system to be shared among all the APs, MAC filtered with a Pre-Shared Key for now.  But eventually I'd like to move authentication for it onto the ACS, because of it's vastly better security and manageability.
  • There's probably some things that still need to be documented, so I should write that.
  • Install a small switch and temporary wireless network into "Hut 10", the small house-like structure that some DVs (Distinguished Visitors, aka VIPs) will be staying in for a few days next week.
  • Figure out how to trace MAC addresses over wireless bridges, because my usual method of the show mac address-table address command doesn't work when the target is at the other end of a point-to-multipoint wireless network. (AH HA VICTORY I actually figured this one out mid-draft of this post.  Log into the wireless root bridge, use show bridge verbose | inc with the target mac address, which will give you the virtual interface that points in that direction.  Then show cdp neighbor vi# will identify the device on the other end of that virtual interface)
  • Oh, I figured out something new.  I should probably document that.

It's a lot of desk work.  An awful, awful lot of desk work, and that's one of the hardest thing so far about this job to get used to.  Most of my prior work experience in IT has been as a PC Tech of some kind, which has you frequently out in the field or at least moving stuff between different buildings. Even if you're not doing hard labor, you're still moving around.  But now I've had many days where the only time I get out of my chair is to go to lunch, or maybe go to a meeting where I'll try not to fall asleep.  For every hike up to the Beach Ball, or drive out to the Runway or Balloon facility, there's a solid week of sitting at my desk and pushing buttons.

Although now that it's warmed up a bit more, on the occasions I do heve to hitch a ride somewhere I can just jump in the bed of a pickup truck.

The thing that has surprised me most about this job, though, is how mentally exhausting it is.  Doing building maintenance at Palmer was a hard job, and there was some thinking involved, but it wasn't nearly as intensely cerebral as being a Network Admin.  As a guy-with-a-wrench I would get off work at the end of the day and my head would be buzzing with ideas and energy, sighting new photography or time-lapse opportunities or building some kind of craft project or writing up a new blog post.  But now I get to the end of the day and my brain just wants a rest; I've expended almost all of my creative energy at work and when it's done, I just want to consume information rather than create it.  I want to read, or play some games on my phone, or go bartend, just hang out in the Galley with my Stewie friends.  I stare at a screen for 9 hours a day now and when it's done, I find it difficult to conjure the energy and motivation to sit down and spend the few hours that a really good blog post actually requires (Hence, the once-a-month updates).

Let me be very clear: I love this job.  It's everything I wanted it to be, and so much more. It's fascinating to a degree that makes me a little bit embarrassed to admit; even I find it a little weird at just how excited I can get over a successful RADIUS authentication to our ACS, even if seeing that success is the culmination of four days worth of effort.

When I was a kid my friends and I were borderline obsessed with a computer game called Myst, which was an open-world exploration/puzzle solving game that was (and is still) famous for it's mind-bending difficulty.  It was a game where you might be exploring a building, and in this building you might find a machine with many levers, buttons and knobs, all of which did something but you had no idea what.  And with copious note-taking, lots of testing and a bit of luck (and the occasional "Ah-ha!" moment), you'd eventually figure out what each button and lever on the machine did, how to operate the machine, and how to get the machine to do what you wanted.  In the game, it was often something like "Manufacture a key to allow you to open this door".

And I realized a few weeks ago that my job is exactly like playing Myst.  I'm pointed at systems that while they do have a logic to them, it's often a unique logic and sorting out how it works is a long process of "Click on everything you can find, and see if you can start making connections between your inputs and the device's output".  And every so often you get those "Ah-ha!" moments and it's even more rewarding when it's something useful and practical in real life, rather than just a game.

I could see how this could drive some people insane; we've got extremely high personnel turnover year-to-year and documentation/knowledge transfer is often scant.  Quite a number of systems here were set up years ago by someone who may have been an expert at them, but they've long since left the program and no one has really touched them since.  People who are process-oriented, who thrive on order and predictability and who like knowing every step of a task before they start it would start bleeding from their ears in places like this.

But it's what I thrive on, and it's exactly how I learn.  I don't have much/any formal education; I'm not sure if I ever graduated from high school and I never went to college.  Outside of a couple certification courses that I took many years ago, everything I know I've taught myself, by stubbornly poking at something and taking it apart until I can find out how it works.

I love it.  I absolutely love it, and I can't wait to come back and continue the process next season.

(I promise I'll do a post soon that has actual pictures, rather than a random brain dump about my job that I'm a little bit too passionate about)