Man, it's been a while since I've blogged. I've been busy working, and finding a new balance between working and.. well.. not working. With everything that's been going on in my personal life, I wouldn't say 'relaxing'. And I haven't really been hacking on my days off.
I have been hacking at work a lot. And learning. A lot. Didn't realize how incredibly green I was when I took the job-- thank the gods I mellowed down my arrogance years ago. :)
Anyhoo, besides brushing up on my linux admin skills and getting back my programming chops, I've had to learn a system that used to be a mainframe OS back in the day, and is now a giant, sprawling database with its own scripting language that still sometimes thinks it's an OS. That system is D3(formerly known as Pick). Interesting stuff. I've gotten fairly decent at programming in pick/basic now (which is the lovechild of Dartmouth Basic and Fortran), and I'm really starting to get the hang of admining it.
My first project I did myself for it was to try to bring modern syslog-style system and debug logs to it. I've been fairly successful, although my needs are now outstripping the capabilities of my system of scripts. But, as my boss pointed out, the original/current production code is a proof-of-concept. It's written in a mix of pick/basic and python. The pick/basic subroutine takes in the error/info message as an argument (the string is freeform to allow as little or as much content as you want included), appends user, port number, timestamp, and a bunch of other data, then writes it out to a file in a queue in the underlying linux system. From there, a python TFTP client picks it up, hands it off to a python TFTP server on another system, which puts it in a queue for a script that writes the entry out to syslog and various log files. It needs to pass the logs off to another system because space is at a premium on the D3 server, but not so much on other servers. Plus then I can move it over to a Debian server, which is much easier to hack on than RHEL. RHEL 4 doesn't even have git, and still has python2.4.
Sounds good so far. But right now, it does everything from log debug messages from various subsystems in our D3 frontend, to collect web hits. Which means it's running up to 10 times a second at peak times. The scripts themselves are fine with this (actually, I have to keep a sleep() in the TFTP client so it doesn't start cycling too fast and maxing the CPUs when there's nothing for it to pick up), that's just a massive amount of logs. Plus the web stuff shouldn't be treated the same as debug messages. So I'm forking the logger to handle web hits separately. I'm also going to re-write the regular logger to use inotify rather than looping to read the contents of the queue to see if there's anything new to upload-- this should both cut down on overhead and keep it from freaking out when the queue is empty. The web logger will do the same thing, but instead of logging to syslog, it'll store info in an SQL database since that's more efficient for storing long-term data. Not to mention easier to analyze.
There's some other projects I'm working on, but I think this blog is long enough for now. I should be back with more follow-up on the web logger, and will probably have interesting war stories about talking to FedEx's web services with python to do on-demand tracking, and doing PDF parsing for uploading price data.
