Bit of a work rant. I'm not trying to use this as a whine-fest, although it'll sound like one at points. Take this more of a case study or illustrative, anecdotal example of why it's proper practice to write out a formal specification for your developers.
There's a project I've been working on for 9 months now; I'm not sure if I've mentioned it, but if I have it's the pricing update program. Anyway, this was originally slated to be a 6mo project. Well, obviously, it hasn't been. Part of the problem has been, I'm learning Flashbasic (Pick/Basic, whatever) as I'm writing it. So that's obviously going to make things take longer than if someone experienced in the language is working on it.
However, I'm getting ready to put it into testing for the 3rd time here in the next week or so. Why? I had no specification to work off of. All I've had to go by is a couple meetings with the heads of the involved departments, and streams of emails back and forth. Plus the input during the previous 2 demos/tests. Neither meeting was that enlightening-- the first one got bogged down in minutiae and was more confusing to me than explanatory. I took down some notes, and ended up with something that got completely rejected when I called the second meeting. However, that was all I got out of the second meeting: what you're doing won't work, it's too unlike what we're doing now and generally wrong anyway, but the current way of doing this is still terrible. I cobbled together something based on what I could glean from the second meeting and from what I was learning on my own from the way pricing currently works. Then I demoed it. It had a lot of flaws, was missing a lot of features, there were a lot of miscommunicated aspects. I took copious notes, learned that the department head I hadn't consulted with so much a) had taken on the duty of updating pricing and therefore needed the project more than the other department did, and b) he was more analytical and demanding, and therefore easier to hammer out the program with. Cue the storm of emails back and forth on what was needed where. Got a mostly workable version, demoed it again, except for a major missing feature it was great. Now I'm finishing that feature and reworking the program flow to shoehorn this whole extra step in, and hoping it'll fly (it's necessary still to edit the spreadsheet, just not to the extent it was the last time I showed them the program-- they want to do 0 editing. This is not possible).
Note all the miscommunication, poorly stated needs and goals, and tendency to leave how to write the program (and therefore alter the business processes surrounding it) to an inexperienced programmer who has never had anything to do with that aspect of the business, and never will on a daily basis. Note how much time was wasted simply trying to get information and input from the people who will be using it every day for the rest of their career with the company (theoretically). This could have been avoided with one thing-- a program specification.
Even though my classes in college were theoretically all intro classes (although you could get more out of them if you put more in, as I did), we were taught to work with specs. In Linux Administration class, Netware Administration, C Programming, Java Programming, Security, Forensics, all the electronics classes (DC/AC Circuit Analysis, Semiconductor Applications, Digital Electronics, Intro to Microprocessors), even the first semester classes like Computer Repair and Survey of Operating Systems-- they all dealt with both writing specs, and working from one. With a proper spec, there is no question of what needs to be done. Everything is laid out, from the current process, to what is broken about the current process, to what needs to remain the same with the new process, to the new process. Even if the new process is a proposal, it still gives the programmer, who does not and will not work in the department that needs the program they're tasked to write, a starting point to work from. And in the meeting about the proposed spec, there can be discussion over what is possible, what the needs are, etc. Rather than everyone just throwing up their hands and asking "well, what do you think it should do?". I was, in fact, asked by the less-analytical dept. head that exact question. That was the point I threw up my hands and called the second meeting, so her boss and my boss could provide us some guidance. Had there been a spec prior to the first meeting, much of that back-and-forth wouldn't have happened, and much of my time (and everyone else's) wouldn't have been wasted.
Unfortunately, and this is where it gets a little whiny again, I've brought up the suggestion (during the first demo) of writing a spec from now on. It was met with total resistance. So right now I'm basically... how shall I put this politely?.... whistling in the wind, as far as applying this to myself.
But maybe this can be an example to someone else, and save them the heartache I'm relegated to. However, this experience has taught me a lot about my coworkers, and working with them on designing programs. And hopefully taught them a thing or two about working with the programmer who hasn't basically built everything from the ground up, or rebuilt it for the company's needs, for the past 15 years.
I'm just really looking forward to the end of this project, and to move on to other, more interesting projects I have lined up.
7.04.2012
SolusOS
I have a new hobby. For the past couple months, I've been working on SolusOS. For those who don't know, in a nutshell it's a desktop distro based on Debian Stable, with some of our own packages thrown in, and Gnome3 Fallback mode hacked up to look like Gnome2. I'd worked with Justin and Ikey on Mint, and left shortly after they did. Around the time they left Mint, they came up with the idea for SolusOS, which is what LMDE was supposed to be but never was. I love it (obviously)-- not only do I squash bugs and maintain a couple packages for it, but it's running on pretty much all my systems.
I'm basically head of bugsquashing. It's not the most glamorous job, but I'm good at it, and want to see it done right. Right now I maintain the non-PAE i686 kernel package, and I'm getting ready to package up inxi, smxi, and sgfxi. Those scripts are already included in the distro by default, but they need to be packaged for upgrade and ISO-rolling purposes.
I'm still figuring out balancing working and hacking, but I discovered last weekend I can bring work home and not let it kill my weekend. And I've also figured out to hack while my fiance's playing video games, since that's great 'me' time. So slowly but surely, I'm taking on more stuff with SolusOS.
So yeah, if you're reading this and you haven't tried SolusOS, go try it. It's awesome, I promise.
1.29.2012
Work Hacks
Man, it's been a while since I've blogged. I've been busy working, and finding a new balance between working and.. well.. not working. With everything that's been going on in my personal life, I wouldn't say 'relaxing'. And I haven't really been hacking on my days off.
I have been hacking at work a lot. And learning. A lot. Didn't realize how incredibly green I was when I took the job-- thank the gods I mellowed down my arrogance years ago. :)
Anyhoo, besides brushing up on my linux admin skills and getting back my programming chops, I've had to learn a system that used to be a mainframe OS back in the day, and is now a giant, sprawling database with its own scripting language that still sometimes thinks it's an OS. That system is D3(formerly known as Pick). Interesting stuff. I've gotten fairly decent at programming in pick/basic now (which is the lovechild of Dartmouth Basic and Fortran), and I'm really starting to get the hang of admining it.
My first project I did myself for it was to try to bring modern syslog-style system and debug logs to it. I've been fairly successful, although my needs are now outstripping the capabilities of my system of scripts. But, as my boss pointed out, the original/current production code is a proof-of-concept. It's written in a mix of pick/basic and python. The pick/basic subroutine takes in the error/info message as an argument (the string is freeform to allow as little or as much content as you want included), appends user, port number, timestamp, and a bunch of other data, then writes it out to a file in a queue in the underlying linux system. From there, a python TFTP client picks it up, hands it off to a python TFTP server on another system, which puts it in a queue for a script that writes the entry out to syslog and various log files. It needs to pass the logs off to another system because space is at a premium on the D3 server, but not so much on other servers. Plus then I can move it over to a Debian server, which is much easier to hack on than RHEL. RHEL 4 doesn't even have git, and still has python2.4.
Sounds good so far. But right now, it does everything from log debug messages from various subsystems in our D3 frontend, to collect web hits. Which means it's running up to 10 times a second at peak times. The scripts themselves are fine with this (actually, I have to keep a sleep() in the TFTP client so it doesn't start cycling too fast and maxing the CPUs when there's nothing for it to pick up), that's just a massive amount of logs. Plus the web stuff shouldn't be treated the same as debug messages. So I'm forking the logger to handle web hits separately. I'm also going to re-write the regular logger to use inotify rather than looping to read the contents of the queue to see if there's anything new to upload-- this should both cut down on overhead and keep it from freaking out when the queue is empty. The web logger will do the same thing, but instead of logging to syslog, it'll store info in an SQL database since that's more efficient for storing long-term data. Not to mention easier to analyze.
There's some other projects I'm working on, but I think this blog is long enough for now. I should be back with more follow-up on the web logger, and will probably have interesting war stories about talking to FedEx's web services with python to do on-demand tracking, and doing PDF parsing for uploading price data.
I have been hacking at work a lot. And learning. A lot. Didn't realize how incredibly green I was when I took the job-- thank the gods I mellowed down my arrogance years ago. :)
Anyhoo, besides brushing up on my linux admin skills and getting back my programming chops, I've had to learn a system that used to be a mainframe OS back in the day, and is now a giant, sprawling database with its own scripting language that still sometimes thinks it's an OS. That system is D3(formerly known as Pick). Interesting stuff. I've gotten fairly decent at programming in pick/basic now (which is the lovechild of Dartmouth Basic and Fortran), and I'm really starting to get the hang of admining it.
My first project I did myself for it was to try to bring modern syslog-style system and debug logs to it. I've been fairly successful, although my needs are now outstripping the capabilities of my system of scripts. But, as my boss pointed out, the original/current production code is a proof-of-concept. It's written in a mix of pick/basic and python. The pick/basic subroutine takes in the error/info message as an argument (the string is freeform to allow as little or as much content as you want included), appends user, port number, timestamp, and a bunch of other data, then writes it out to a file in a queue in the underlying linux system. From there, a python TFTP client picks it up, hands it off to a python TFTP server on another system, which puts it in a queue for a script that writes the entry out to syslog and various log files. It needs to pass the logs off to another system because space is at a premium on the D3 server, but not so much on other servers. Plus then I can move it over to a Debian server, which is much easier to hack on than RHEL. RHEL 4 doesn't even have git, and still has python2.4.
Sounds good so far. But right now, it does everything from log debug messages from various subsystems in our D3 frontend, to collect web hits. Which means it's running up to 10 times a second at peak times. The scripts themselves are fine with this (actually, I have to keep a sleep() in the TFTP client so it doesn't start cycling too fast and maxing the CPUs when there's nothing for it to pick up), that's just a massive amount of logs. Plus the web stuff shouldn't be treated the same as debug messages. So I'm forking the logger to handle web hits separately. I'm also going to re-write the regular logger to use inotify rather than looping to read the contents of the queue to see if there's anything new to upload-- this should both cut down on overhead and keep it from freaking out when the queue is empty. The web logger will do the same thing, but instead of logging to syslog, it'll store info in an SQL database since that's more efficient for storing long-term data. Not to mention easier to analyze.
There's some other projects I'm working on, but I think this blog is long enough for now. I should be back with more follow-up on the web logger, and will probably have interesting war stories about talking to FedEx's web services with python to do on-demand tracking, and doing PDF parsing for uploading price data.
Subscribe to:
Posts (Atom)