A programming glitch in Apple's OS X operating system is making it hard for Mac users to tell their computers not to trust digital certificates, exacerbating an ongoing security problem with a Dutch certificate authority that was recently hacked.

Mac users began reporting problems Tuesday when they tried to revoke digital certificates issued by DigiNotar, a Dutch company whose servers were compromised last month and used to issue fraudulent digital certificates. Mac users revoked the certificates on their computers, but still saw some sites that used those certificates being marked as trustworthy.

Digital certificates are an important part of the way the Internet works, and are essential whenever two computers try to connect using the HTTPS protocol. The problem is that Apple's operating system does not allow users to revoke DigiNotar certificates properly, and marks some websites as trustworthy when it shouldn't.

Seth Bromberger noticed the issue Tuesday afternoon. After reading a news report about DigiNotar being compromised, he decided to take matters into his own hands and revoke DigiNotar's certificates on his Mac, using Apple's Keychain software. That meant that any time he tried to visit a site signed by DigiNotar or one of its intermediaries, he should have received a warning.

He didn't. A visit to DigiNotar's website soon confirmed that all kinds of HTTPS material on the page that should have been marked by his browser as untrusted looked exactly as it had before he'd revoked the certificate. "I just wanted to validate that the solution that was proposed fixed the problem. And it didn't."

Most users don't revoke digital certificates themselves; they let the browser makers handle it. Chrome, Firefox and Internet Explorer have all blocked DigiNotar certificates, but Apple hasn't said what it plans to do with its Safari browser. That means that, for now, Mac Safari users will have a hard time solving the problem.

Ryan Sleevi, a software developer who has contributed to Google's Chrome project, noticed the issue too. After poking around the Mac OS X source code, though, he uncovered the cause.

Users can revoke a certificate using Keychain, but if they happen to visit a site that uses the more-secure Extended Validation Certificates, the Mac will accept the EV certificate even if it's been issued by a certificate authority marked as untrusted in Keychain.

"When Apple thinks you're looking at an EV Cert, they check things differently," Sleevi said in an interview Wednesday. "They override some of your settings and completely disregard them."

Designed as a way to reassure Web surfers that they're not being phished, Extended Validation Certificates turn the browser address bar green. They're widely used by sites that have a lot of HTTPS traffic.

It's troubling that such a basic component of Internet security could have such an obvious flaw on the Mac, several security experts said Wednesday. "In a real-world sense, it probably won't affect a lot of people, but for me it's a little bit troubling that the security advice on what you're supposed to do plain doesn't work," said Jeremiah Grossman, chief technology officer with WhiteHat Security.

Apple, which is often tight-lipped about anything to do with computer security, did not return messages Wednesday seeking comment.

Problems with digital certificates are troubling, but they're hard for hackers to exploit. That's because even when hackers can issue a fake digital certificate -- one saying that a server set up for phishing is Gmail.com, for example -- they still need to trick their victims into visiting that server and believing it really is Gmail. For that to happen, the bad guys must take control of their victim's DNS (Domain Name System) software too, using what's known as a man-in-the-middle attack.

But someone seems interested in doing this. When DigiNotar was hacked in July, security experts say the hackers issued themselves hundreds of fake digital certificates for domains including google.com, mozilla.com, yahoo.com and torproject.org.

On Sunday, Google said just such an attack had been launched against users of its Gmail service, primarily targeting users in Iran.

Robert McMillan covers computer security and general technology breaking news for The IDG News Service. Follow Robert on Twitter at @bobmcmillan. Robert's e-mail address is robert_mcmillan@idg.com