Something's Wrong

I dropped this nugget in a comment on the blog of my company's owner. One of the last lines in the comment was bait, but no one took it (or maybe no one read it). So I've decided to post it in a place that is guaranteed no one will read it ;)
The Post:
There is something fundamentally wrong to me about the size of the failure-potential software development has always had and still carries. Maybe I'm crazy but in other professions is this also the case?

We don't follow engineering principles because we claim that they are too rigid and don't take advantage of all of the flexibility the software universe gives us, but if software's failure rate is as high as I think it is then what does the flexibility really give us? Do we need the flexibility simply because we never get it right?

I say 'we' in the all-encompassing profession-of-software-development sense. Because, well, you know 'we' NEVAR fail... ;)

The point here is that no one gets it right (except in the case of the occasional occurrence of luck or a very flexible and accepting [read: push over] client). Upon neither of these things am I willing to build a career on.

Can all we really hope for is to get lucky project after project, just long enough to get promoted into another role? Do we have to pull out the smoke and mirrors when we don't get lucky on the design in order to fool our bosses and the client into thinking the project was a success? When I switched from mechanical engineering to computer science I wasn't thinking I was going to find the moral clarity of a Peace Corp volunteer, but I also didn't think I would have to cheat and steal to be successful. When we aren’t honest about our work and we make it seem that the troubles we had didn’t exist or were instantly overcome, then we set unrealistic expectations for ourselves, our colleagues and our company.

What is the fundamental problem here (other than the resulting failure)? What is the one desire that supersedes all when an application is done? I feel it every time I release an application: "I wish I could just redesign it", either in part or in whole.

Why do I feel this way? Is it because I screwed up the implementation so bad that I want to go back and fix it? No (well maybe once or twice :). It is because only now, at the end do I understand the business of the client, their needs and the true intent of the software and what functionality the application was supposed to provide. To those who, when done with the application, don’t care when they make the job of 900 users more painful, but rather only care that their bosses and the client business owner buy the illusion that the project was a raving success: I say good luck on somebody else’s team. Because if you are not measuring your application’s success by the positive value you provided to your client’s employees and profit, then you are using the wrong metrics.

Ok so let’s think about this. Why is it that we only know this at the end? Isn't that supposed to be acquired during Requirements gathering / Spec writing etc.? A true understanding only comes from the journey one takes with the client during construction and the feedback given during UAT. You never know if you are right/wrong/close or don't have a clue until you understand thoroughly your operating environment or until the client tells you.

So maybe here is where you fault the developers for not doing their due diligence in extracting this knowledge from the client beforehand. But you'd be, in most cases, wrong. This is because it’s not until the end, after they have taken the journey with you and see the results, that the client has any real understanding either.

Whoa, wait. Doesn’t this fall to the Project Manager/ Tech Lead to set client expectation and ensure that the project’s direction and design provides the client with the software they need?
Yes.

And do the managers/leads do this and correct during development as needed?
Most do.

So what is the problem?
The problem is that the projects in which this occurs invariably run over budget.

So what? The client had to pay more but they are getting the system they need, the system that solves their problem (even though they might not know it yet).
Many people agree with this. In fact, projects going over budget is such a common occurrence that claiming you are going to hit budget is most often perceived as a joke or arrogant. When a project does hit budget and deliver on time that team is immortalized. (See where the motivation to make every project appear to be a success comes from?) While being bedazzled by the efficient delivery, what the client fails to see at the time is that the application is rarely the tool the users need or were promised. Eventually, when the client business owner realizes this… well good luck to you.

Providing the client with the system they need, doesn’t nullify the fact that you told them the system would cost $700K and are now asking them to pay $900K. Ever tried to explain that to the client? Explain that the original design was not going to provide them with the system they needed and that all the time spent in the beginning doing the end to end design and spec writing that you had to assure them wasn’t a waste of money… actually was. Now, not only are they not going to pay the $900K but they also don’t want to pay the $700K either.

So if we only know what they needed once we finish, then how can we possibly build an application that will satisfy their needs the first go round?

More to follow...