Search This Blog



The Beauty and Wonder of Programming Specs

Bit of a work rant. I'm not trying to use this as a whine-fest, although it'll sound like one at points. Take this more of a case study or illustrative, anecdotal example of why it's proper practice to write out a formal specification for your developers.

There's a project I've been working on for 9 months now; I'm not sure if I've mentioned it, but if I have it's the pricing update program. Anyway, this was originally slated to be a 6mo project. Well, obviously, it hasn't been. Part of the problem has been, I'm learning Flashbasic (Pick/Basic, whatever) as I'm writing it. So that's obviously going to make things take longer than if someone experienced in the language is working on it.

 However, I'm getting ready to put it into testing for the 3rd time here in the next week or so. Why? I had no specification to work off of. All I've had to go by is a couple meetings with the heads of the involved departments, and streams of emails back and forth. Plus the input during the previous 2 demos/tests. Neither meeting was that enlightening-- the first one got bogged down in minutiae and was more confusing to me than explanatory.  I took down some notes, and ended up with something that got completely rejected when I called the second meeting.  However, that was all I got out of the second meeting: what you're doing won't work, it's too unlike what we're doing now and generally wrong anyway, but the current way of doing this is still terrible.  I cobbled together something based on what I could glean from the second meeting and from what I was learning on my own from the way pricing currently works.  Then I demoed it.  It had a lot of flaws, was missing a lot of features, there were a lot of miscommunicated aspects.  I took copious notes, learned that the department head I hadn't consulted with so much a) had taken on the duty of updating pricing and therefore needed the project more than the other department did, and b) he was more analytical and demanding, and therefore easier to hammer out the program with.  Cue the storm of emails back and forth on what was needed where.  Got a mostly workable version, demoed it again, except for a major missing feature it was great.  Now I'm finishing that feature and reworking the program flow to shoehorn this whole extra step in, and hoping it'll fly (it's necessary still to edit the spreadsheet, just not to the extent it was the last time I showed them the program-- they want to do 0 editing.  This is not possible).

Note all the miscommunication, poorly stated needs and goals, and tendency to leave how to write the program (and therefore alter the business processes surrounding it) to an inexperienced programmer who has never had anything to do with that aspect of the business, and never will on a daily basis.  Note how much time was wasted simply trying to get information and input from the people who will be using it every day for the rest of their career with the company (theoretically).  This could have been avoided with one thing-- a program specification.

Even though my classes in college were theoretically all intro classes (although you could get more out of them if you put more in, as I did), we were taught to work with specs.  In Linux Administration class,  Netware Administration, C Programming, Java Programming, Security, Forensics, all the electronics classes (DC/AC Circuit Analysis, Semiconductor Applications, Digital Electronics, Intro to Microprocessors), even the first semester classes like Computer Repair and Survey of Operating Systems-- they all dealt with both writing specs, and working from one.  With a proper spec, there is no question of what needs to be done.  Everything is laid out, from the current process, to what is broken about the current process, to what needs to remain the same with the new process, to the new process.  Even if the new process is a proposal, it still gives the programmer, who does not and will not work in the department that needs the program they're tasked to write, a starting point to work from.  And in the meeting about the proposed spec, there can be discussion over what is possible, what the needs are, etc.  Rather than everyone just throwing up their hands and asking "well, what do you think it should do?".  I was, in fact, asked by the less-analytical dept. head that exact question.  That was the point I threw up my hands and called the second meeting, so her boss and my boss could provide us some guidance.  Had there been a spec prior to the first meeting, much of that back-and-forth wouldn't have happened, and much of my time (and everyone else's) wouldn't have been wasted.

Unfortunately, and this is where it gets a little whiny again, I've brought up the suggestion (during the first demo) of writing a spec from now on.  It was met with total resistance.  So right now I'm basically... how shall I put this politely?.... whistling in the wind, as far as applying this to myself.

But maybe this can be an example to someone else, and save them the heartache I'm relegated to.  However, this experience has taught me a lot about my coworkers, and working with them on designing programs.  And hopefully taught them a thing or two about working with the programmer who hasn't basically built everything from the ground up, or rebuilt it for the company's needs, for the past 15 years.

I'm just really looking forward to the end of this project, and to move on to other, more interesting projects I have lined up.



I have a new hobby. For the past couple months, I've been working on SolusOS. For those who don't know, in a nutshell it's a desktop distro based on Debian Stable, with some of our own packages thrown in, and Gnome3 Fallback mode hacked up to look like Gnome2. I'd worked with Justin and Ikey on Mint, and left shortly after they did. Around the time they left Mint, they came up with the idea for SolusOS, which is what LMDE was supposed to be but never was. I love it (obviously)-- not only do I squash bugs and maintain a couple packages for it, but it's running on pretty much all my systems.

I'm basically head of bugsquashing. It's not the most glamorous job, but I'm good at it, and want to see it done right. Right now I maintain the non-PAE i686 kernel package, and I'm getting ready to package up inxi, smxi, and sgfxi. Those scripts are already included in the distro by default, but they need to be packaged for upgrade and ISO-rolling purposes.

I'm still figuring out balancing working and hacking, but I discovered last weekend I can bring work home and not let it kill my weekend. And I've also figured out to hack while my fiance's playing video games, since that's great 'me' time. So slowly but surely, I'm taking on more stuff with SolusOS.

So yeah, if you're reading this and you haven't tried SolusOS, go try it. It's awesome, I promise.


Work Hacks

Man, it's been a while since I've blogged. I've been busy working, and finding a new balance between working and.. well.. not working. With everything that's been going on in my personal life, I wouldn't say 'relaxing'. And I haven't really been hacking on my days off.

I have been hacking at work a lot. And learning. A lot. Didn't realize how incredibly green I was when I took the job-- thank the gods I mellowed down my arrogance years ago. :)

Anyhoo, besides brushing up on my linux admin skills and getting back my programming chops, I've had to learn a system that used to be a mainframe OS back in the day, and is now a giant, sprawling database with its own scripting language that still sometimes thinks it's an OS. That system is D3(formerly known as Pick). Interesting stuff. I've gotten fairly decent at programming in pick/basic now (which is the lovechild of Dartmouth Basic and Fortran), and I'm really starting to get the hang of admining it.

My first project I did myself for it was to try to bring modern syslog-style system and debug logs to it. I've been fairly successful, although my needs are now outstripping the capabilities of my system of scripts. But, as my boss pointed out, the original/current production code is a proof-of-concept. It's written in a mix of pick/basic and python. The pick/basic subroutine takes in the error/info message as an argument (the string is freeform to allow as little or as much content as you want included), appends user, port number, timestamp, and a bunch of other data, then writes it out to a file in a queue in the underlying linux system. From there, a python TFTP client picks it up, hands it off to a python TFTP server on another system, which puts it in a queue for a script that writes the entry out to syslog and various log files. It needs to pass the logs off to another system because space is at a premium on the D3 server, but not so much on other servers. Plus then I can move it over to a Debian server, which is much easier to hack on than RHEL. RHEL 4 doesn't even have git, and still has python2.4.

Sounds good so far. But right now, it does everything from log debug messages from various subsystems in our D3 frontend, to collect web hits. Which means it's running up to 10 times a second at peak times. The scripts themselves are fine with this (actually, I have to keep a sleep() in the TFTP client so it doesn't start cycling too fast and maxing the CPUs when there's nothing for it to pick up), that's just a massive amount of logs. Plus the web stuff shouldn't be treated the same as debug messages. So I'm forking the logger to handle web hits separately. I'm also going to re-write the regular logger to use inotify rather than looping to read the contents of the queue to see if there's anything new to upload-- this should both cut down on overhead and keep it from freaking out when the queue is empty. The web logger will do the same thing, but instead of logging to syslog, it'll store info in an SQL database since that's more efficient for storing long-term data. Not to mention easier to analyze.

There's some other projects I'm working on, but I think this blog is long enough for now. I should be back with more follow-up on the web logger, and will probably have interesting war stories about talking to FedEx's web services with python to do on-demand tracking, and doing PDF parsing for uploading price data.


New Job

Sudden random real life post... I got a job a couple weeks ago as an assistant sysadmin. :) No more retail hell. I still have 1 idi^H^H^Hluser to deal with, but hey. Better than being surrounded by hundreds. Not sure how much hacking I'm going to do, apparently it's even true for me-- once you work on computers for a living, you're not too inclined to mess with them for fun. :\ I'm stuck on whiskey, I have to learn pyqt to make any more headway. Trash80 is trucking along on rewriting coreutils in python, and I'm sitting back giving advice and feedback. And now my fiance is asking me to make a one-off distro for him.

On a job-related note, I'm compiling a list of "Sysadmin Rules to Live By". I have 3 so far.

1. Everyone is an idiot.
Seriously, everyone. I have my stupid moments too, I won't lie.
2. The user didn't do anything.
If I had a dollar for every time I've heard "I didn't do anything, it just broke!" I could buy a curry chicken lunch combo... after just 3.5 weeks on the job.
3. It's impossible, but it has to be done anyway.
Had an example of that this morning: clean out the IT detritus in the back of customer service... even though there's no room in the server room for more crap. And storing it in an empty cubicle is not an option. Oy.

But, owing to my massive masochistic streak, I do enjoy my work now. Maybe after I settle in I'll find my way back to hacking.


Whiskey is on launchpad!

Whiskey is now on launchpad, in 2 places since I can't figure out how to link habanero (trash80's and my project name) to my personal account. and

Update on the gui: since pyglet will never upgrade to python 3.0, I'm using pyjamas. PIL hasn't upgraded yet either, but it's tempting to grab the bits I need (a small subset of the whole module), fork it, and upgrade it. I haven't decided on that yet, but for right now whiskey still depends on python2.6 (arch's stable python, and the highest version I know of that PIL will work with), although I'm writing my code to be upwards compatible with 3.0.

Although whiskey is up on launchpad, it doesn't have a PPA yet simply because it's not particularly useful yet. Hence why it's still in experimental. Arch has the debian buildtools though, so once whiskey is close to what I'm aiming for (or at least close enough that I feel comfortable opening it up to a wider audience), look for a PPA.


whiskey db backend

After playing around with it and reading the specs, yaml is not the right tool for the job. It's too hard to have multiple attributes for the same key (ie, colours: grey, blue, green) and even harder to parse that back out and search it. Some sort of nosql db seems to be what I'm looking for, and mongodb seems to be the friendlier of the python-supported nosql dbs out there. So... scratch the yaml stuff (until it comes time for config files, that's still a good idea), mongodb is going to be the db backend. And I should have some working, prealpha code going up on launchpad soon. No promises though, my work hours have expanded (yay money, boo exhaustion) so it's tough to get anything done but work, chugging coffee, and sleeping. But I'm trying. :)



Now I'm working on a desktop-agnostic wallpaper manager. I jokingly called it 'whiskey' one night in a chat, and the name kinda stuck. Anyhoo, it walks through the user's home directory, looking for jpegs, gifs, pngs, and xcfs, looks at the size to determine if it's a wallpaper by its size (4:3 and 16:9 sizes are supported), and if it's not a standard size, it asks the user if they want it added to the database. The database is written in YAML. What will be stored in the database is the filename (full path), format, size, aspect ratio, colours, and design details. The colours and design details will be user-supplied. The gui will be done with pyglet. The image processing library I'm using to parse the metadata and will be displaying the preview in the gui is PIL (Python Image Library). Though, PIL's docs say the function to display the image is really inefficient, so if it turns out to suck too badly, I'll find something else to handle that.

I'm thinking to handle tiling, I'll grab the screen resolution (somehow) and use PIL to paste the image over and over into a bigger image the size of the screen, then save it as something like "foo.tiled.png" or so.

I'm still not sure exactly how I'm going to actually set the wallpaper, whether I'll shell out to fbsetbg/esetbg/etc., or write my own routine to handle it.

The way to look through the wallpapers is a slideshow. There'll be a 'set' button, and a delete button that'll trigger a menu to either simply delete the file from the database, or from the system.

So far that's all I have planned out. Which I suppose kinda makes sense, since all I have coded so far is the function to walk down the home directory, and the function to read the metadata prior to writing it to the database. And I have a sketch of the UI (not my work, my fiance is my design person).