I’m writing this post as a rant. I am tired of hearing people who have never worked in Federal IT try and come forward with ideas about what was wrong about the way Healthcare.gov was developed. I have one statement for all of you who think you could have done better.

You would have ALL failed miserably.

Federal IT is broken. Hell, all of Federal contracting is broken from what I’ve seen, but I want to focus on the IT side for now.

Before I get started, a quick reminder of my background. My first Federal project was back in the late 90s as the tech lead for the Secretary of the Air Force’s correspondence tracking system. Over the years, I have worked on a multitude of projects and managed many more while I was the Director of Technology Solutions for Washington Consulting . I’ve responded to many proposals and run Federal IT projects through the wide variety of hurdles that they face.

I can tell you right now, I am impressed that Healthcare.gov even boots up.

Success is Rare

I can quickly list for you all the Federal IT projects priced at over $5 million that were delivered on-time.

Zero.

Did you know that from 2003 to 2012, only 6.4% of projects with labor costs over $10 million in Federal IT were successful? Over 40% were never even completed they were such abject failures.

Projects of smaller scale than Healthcare.gov fail all the time. CIO.com has a nice little article about the futility that is Federal IT. They sum it up nicely by saying that Healthcare.gov Didn’t have a Chance in Hell.

Let’s face it. Healthcare.gov is a very public, politically-charged, system that is large and complex by nature. When you put out an inflexible go-live date that is in the middle of a political battle to delay or cancel the program completely, failure was certain.

What Went Wrong?

Let’s forget the obvious facts surrounding a system that is interfacing with over 36 states, a multitude of insurance carriers, and several federal agencies that think they have the best IT shop in the land. I want to talk about the over-arching process.

In Federal IT, typically nobody with both knowledge and authority owns all the components of a system. Even on smaller efforts, one contractor owns the data center, another runs the database infrastructure, one is developing the actual system, and a fourth contractor is in charge of making sure all the rules are followed.

I hate that fourth contractor. From what I can tell, they believe they won’t get paid if they don’t find problems in the documentation, the project plan, or your accent. I’ve seen projects held up because the columns in a table weren’t perfectly aligned. What should have been, Looks good, just fix the alignment before the next review, becomes, Go fix that table and we’ll reevaluate the entire package again.

In retrospect, hate is too mild. I despise them. I despise them like I despise the guy in the movie theater that has to call his friend to tell him that yes, he saw that hidden logo in the film this time and he now agrees that it does foreshadow the death of the main character at the end of the film.

The Federal IT process is not just a waterfall development cycle. That is an understatement. It is the Victoria Falls of software development processes. There are multiple reviews at every stage of the development lifecycle. Each and every interface needs it’s own collection of documentation. If you are talking to another Government agency, there are separate security documents that explain how the communication channels will be authenticated and protected. That documentation has to be completed before any holes in a firewall are even considered for opening.

Remember the part about 36 states? Multiply that out.

Let’s go back to the Data Center team for a minute. Do you need 300 Terabytes of storage for Content? No problem. We’ll go buy this huge NAS array. Do you need better performance for your database system? No worries. The vendor assures us that this storage system can handle the performance needs of any database system. Maybe you need to tune your database queries.

Keep in mind, this is for a small system and isn’t even counting the subcontractors involved. After all, there are small business quotas that have to be met in every procurement and if your team can’t check every single box in the RFP then you will be disqualified.

I’ve seen requirements for 5 years of experience with software that was only 2 years old.

It all works out though because the best qualified team always gets awarded the contract. Who wants to use those other guys when they promised to do it for 30% less. That would be everyone because if their proposal checks all the boxes, the procurement officer won’t get fired for selecting them. After all, who can lodge a legitimate protest to an award that will save 30% over the competition even if it promises failure.

How does that winning firm plan to deliver on the project while saving the government money? They just hire younger staff who have never done anything more challenging than fixing bugs for a 10 year old system in another agency and mix them in with aging IT architects who are just so happy to have a job that they will gleefully overdesign the system just to prove that they know how to use this new technology called a whiteboard.

If there was competent leadership on the Federal side there might be some hope. As I alluded to in the beginning of this rant, there isn’t. The best and the brightest usually get scooped up by a Federal contractor. They gladly make the change when they realize that they can do similar work for a lot more money. There is an IT labor shortage after all.

This is capped off when that talent gets hired away and then put into the same role at the same agency. The difference is that now they are a contractor. They are paid more money for the same job, and have less authority because they aren’t a Federal employee anymore.

In this environment, find a project manager that can effectively control the purported 55 contractors that worked on this effort. Keep in mind, contractors only listen to the people that sign the checks for their contract.

Like I said. I’m impressed Healthcare.gov even turns on.

Healthcare.gov is likely salvageable. Federal IT isn’t. Federal IT needs a do-over and the freedom from politics to let that happen.