Normally, it's hard to get anyone but the geeks fired up about information infrastructure. But the Navy Marine Corps Intranet isn't your normal network. With 700,000+ seats, it's the world's second-biggest network, after the internet itself. NMCI's technical complexities and hiccups are the stuff of dark legend around the Navy. The fights it's sparked between the military and its main contractor, Electronic Data Systems (now owned by Hewlett-Packard), have been epic. And then there's that contract; after paying out $10 billion, the Navy still doesn't own a single router. So it's giving HP another $3.3 billion dollars – a "hostage" payment, as one Department of the Navy civilian put it, mostly so the military can buy the gear and look at the network blueprints sailors and marines have used for a decade.

So maybe it's not so surprising that our item on NMCI's next phase triggered so many heated – and passionately smart – responses. In the comments section, we heard from sailors, from marines, from HP employees, and interested observers, too. Here's a sample:

I still work for the NMCI and I have to say that I honestly have a hard time looking sailors in the eye when I’m out and about because I’m so ashamed of the job that’s done. I’ve never seen more disregard for the well-being of a customer than I have at HP...

We’re given sub-standard tools and people that are hired off the street that have NO computer knowledge. There is absolutely no incentive or upward advancement, and on top of that, there have been pay cuts since HP took over. They aren’t retaining any talented individuals, and it shows. We’re understaffed. We were working mandatory overtime for almost a year (10-11 months). Now that’s been eliminated since it’s no longer cost effective to care about the users for the remainder of this contract seeing as how the new contract has been signed and made official...

Stretched for resources as they are, people like me are pulled from their normal duties (dealing with customer issues, training agents, and completing order requests) to just answer basic Service Desk calls. When this happens, we worsen our situation because there’s a major loss in training and resolution time. Agents that I know that have been resourced to working e-mail backlog have said that there is a 15 day turnaround time for nothing more than a response back...

We’re digging ourselves into a hole that we can’t possibly hope to get out of ethically, and I shudder to think of what it’s going to take to even restore ourselves back to the position of a semi-effective source of troubleshooting and support for our customers.

– NMCI Monkey

NMCI is absolutely ridiculous. Most commands have a shared drive of less than 1GB for over 300 people to share! One large attachment can put your inbox over its size limit. You can only send email in plain text, because they can’t secure the network properly... As much is it sucks working on that network, the taxpayers are the ones who should be really pissed. HP is using this government contract as a huge cash cow, meanwhile the military is stuck with substandard IT support.

– the grizz51

Booz Allen Hamilton had no involvement in negotiating the CoSC contract or pricing as stated in your August 31 article. Our firm's involvement focused on working with the Navy and HP to clarify technical matters within the contract language.

– Booz Allen Hamilton

I’m an IT in the Navy. I signed up and was trained to do a job that I haven’t actually done in quite a while... Just last month, we were having a multi-national exercise of pretty intense significance. One of my new user’s couldn’t get access to a mailbox. Since I’m filling the billet of a Network Admin, but have the same level of access as any other user, I had to call the help desk. I explained the issue and got “Sorry, man. We used to be able to make permission modifications, but now everything has to go through the IA people. Unfortunately, they’ve all been converted to Govt-Service and work normal 8-4 hours now. I can still call them after-hours, but only for VIPs...”

My first shore duty was where I finally met the beast known as NMCI. My job, again as a Network Administrator, was to do nothing technical. I had no access different from a normal user, and would continually get “why are you guys even here” from pissed customers who thought we actually did IT work. Right-hand to God, the only job I did was wait for the fattest woman in the world, who was the NMCI tech, to drive to our command. I would then carry the workstation or other equipment for her to the office it was to be installed in. She would then go to the switch room, a room which only she had the key to, and then enable the respective port. All the money and credit with none of the labor!...

– WarrenPeace

While mostly accurate, your article is horribly one sided. First of all blaming HP is a bit odd. This contract came through EDS and HP acquired EDS about a 2 years ago. (Yes technically it is HP.) Secondly, obviously the Navy negotiated these contracts. Other vendors have been trying to sell to the Navy for years, but could not because the Navy essentially sole-sourced to EDS (now HP). Everyone in the IT world thought it was stupid, but the Navy did it. They’re the ones to blame. Does the writer expect HP to just hand over all the equipment, software and IP just because it is a government customer?

– redhed

Let’s set the record straight. As someone who worked for EDS, I was fortunate enough not to work on the NMCI contract, but I knew a lot of people who did, and I was friendly with enough senior management that I could usually get the “off the record” skinny on what was really going on in the company.

First, the NMCI deal nearly put EDS into bankruptcy. The deal was so one-sided in favor of the military that the CEO of the company (Michael Jordan at the time, I think) had a face-to-face with Rumsfeld and Cheney and asked for them to please let EDS renegotiate the contract. He was told no, that they could not do any favors for EDS during an election year (2004).

Second, the military grossly misrepresented the size and amount of work which would be needed to consolidate the network and secure it. EDS was left in the worst possible scenario– it could not hire enough engineers/developers to move the Navy/Marine Core code off legacy systems without going out of business, and if they didn’t hire more engineers/developers, they would never get the project finished on time. What was EDS supposed to do, sue the government for fraud?

Third, EDS had at one point a ton of really, really, really smart people working there. (I was not one of them) If those incredibly smart people could not figure out how to make all of this work, under budget and on time, do you really think anyone else could? There is a *reason* the Navy/Marine Core went to EDS- because they could not handle it themselves. Fourth, when I talk about costs being underestimated, I’m not talking by about 20%. Or 50%. Or even 100%. It was more on the order of 1000%. Think about that...

Do I love EDS? No, I left that sinking ship two years ago, because I knew the company was going to be sold, and at least half of the workforce would be let go. It was a grossly mismanaged company that, like Microsoft, totally missed the boat on how the Internet was going to impact the computer world. But I can’t sit by and read a completely crap piece of journalism like this one without defending the everyday people who did everything they could to try to make the abortion that was the NMCI contract actually work.

– thor1310

I worked at the NMCI helpdesk for about 2 years. First as a tier 1 agent for about a year, then I moved up to tier 3/4 support where we handled the tickets that the tier 1 and tier 2 couldn’t fix. Pretty much what this article says is true. It was frustrating as a Tier 1 agent trying to help the users when my own hands were tied by some stupid policy. When we were discourage from spending too much time on a call. Even though I could fix the issue, it would just take a bit of time, I had to escalate it so I could take another call. I felt the user’s pain when I saw the incoming call queue sitting at 100+ calls and the wait time was over an hour. Trying to remote into a computer in Japan that was a Pentium 3 system with 256MB of RAM made me want to bash my own head in (this was in 2007). We would have to work mandatory overtime just to cover all the calls so we could make our metrics for the month so the leadership could pat themselves on the back for keeping withing the SLA... Now I’m at a medium size Defense contractor and I’m much happier to not be working for EDS an HP Company anymore!

– TheAngryIntern

You neglected to mention that the Navy is running its own enterprise network, it is called OCONUS NAVY ENTERPRISE NETWORK (ONENET) at all the overseas locations. And it is a success compared to NMCI from what the NMCI customers who visit tell us and from my personal experience with NMCI. This network was started the same way as NMCI, grouping together many different legacy networks to make one centrally managed Navy network for Overseas locations.

So the folks in your article that indicate they are not sure if the Navy is ready to manage its own network need to look around and get out of the US mentality because it is already successfully being done. It may not be as big as NMCI [25,000 seats vs 700,000 – ed.] , but it is still a managed enterprise that costs a whole lot less that $3.3 Billion to operate and maintain.

– F

I wish I could compliment you on a well written article however I just could not bring myself to lie to you the way you have lied to the general public. You article is overflowing with inaccuracies, slander and out right lies. You have no understanding of how the contract was written, is managed, or how it operates. You have demonstrated that is possible to be a paid journalist without actually being a true journalist. Allow me to clear up a few points for you.

The Navy has always made every policy decision in regards to the NMCI network.

The Navy directs what maintenance happens and when.

The Navy directs what applications are permitted on the network and when.

The Navy directed that the contractor purchase all equipment.

The Contractor wanted to bill the Navy for each segment of the network. That is Billing for Infrastructure, Printing, Computers, Servers, E-mail, Security, etc. The Navy wanted a rolled up cost divided out per seat. A large majority of IT professionals can not even begin to describe the complexities of the Network as a whole let alone understand the cost of it in terms of hardware, software, manpower, etc. The Limitations of the Network in terms of storage size or mail capacity are not due to a choice by the contractor but by the contract itself. An attempt to renegotiate was made in 2007 and passed on by the Navy. The Navy was offered to by everything back at one time but could not afford to do so. Funny, it nearly destroyed EDS trying to build this network with little income at the time and now they are being vilified for doing for the Navy what no other company in the world can do, even today.

You question agility? What do you know of it. The only limitations are the restrictions the Navy puts on itself. There are no limitations on what HP is ready and willing to offer....

I have seen how poorly the Navy and Marine Corps communicated the program to the service members... I have seen the Government Civilians fail to do their job time and again, the government fail to meet contractual requirements, and to constantly demand more for free (a violation of numerous laws) all while wasting millions of dollars annually on frivolous end of year spending. HP has always delivered what was requested of them, delivered new and innovative technologies and worked tirelessly to meet the needs of the Navy and Marine Corps. Certainly there have been some shortcomings on the part of HP, but they are always addressed as soon as they are identified. I think it is amazing that the employees of HP have put forth as much effort as they have, many working tirelessly in support of the contract, to meet unrealistic expectation all the while constantly being berated by the end users who are simply following hearsay, rumor, innuendo, and slanderous pieces scattered about…

There is way more to this story than you or anyone else is reporting but in this era of corporate hate and government love it will never be told.

– M