Most of the controversy that has erupted over smartphone apps has focused on things like the sharing of personal information and the presence of malware. But the wild-west craziness of the app world apparently extends to the medical arena, where the Food and Drug Administration is still pondering if and how to regulate smartphone software that interacts with medical devices. The FCC has been forced to act, fining app makers who claimed that their software could treat acne simply by turning the smartphone screen a specific color.

That latter tidbit is cited as a cause of concern in a new article in JAMA Dermatology that looks at another area of medicine with "an app for that": melanoma diagnosis. Apparently, there are over half a dozen apps that claim to be able to tell you whether an oddly colored patch on your skin is worrisome enough that you should see a doctor. But the authors' testing of the apps suggests that relying on a real, live dermatologist remains the way to go.

Rather than slaving away in the review labs of a tech website, our intrepid app testers were all in the Department of Dermatology at the University of Pittsburgh Medical Center. As such, they had access to resources that could really put the software to the test: a set of photos taken of suspicious skin lesions, which were later removed and examined in detail by members on the hospital staff. Thus, they could link the images (60 melanomas and 128 benign growths) fed to the apps with a definitive diagnosis.

Or at least they could for four of them. Three other apps had to be excluded because they would only work with an image taken within the app itself using the smartphone camera.

The remaining four apps used several different techniques for determining whether a lesion was cause for concern. One simply ran an automated image analysis algorithm. Another interacted with users to define the lesion's edges before doing its analysis. And one required an in-app purchase to have your image forwarded to a board-certified pathologist, who would send back a diagnosis later.

The three automated systems provided a diagnosis on more than 90 percent of the images sent in (ranging from a low of 90 to a high of 98). The actual pathologists were a bit pickier, only giving an answer for 85 percent of the photos they saw.

But their caution paid off, as they handily beat the rest of the apps in terms of providing a sound diagnosis. Of the responses that suggested a lesion might be melanoma, the clinicians were right 42 percent of the time; the best application managed a 36 percent score (this figure is called a "positive predictive value," or PPV). Of course, you'd expect a bit of overdiagnosis, given these growths were all considered problematic enough that a dermatologist chose to remove them.

The bigger cause for concern is the negative predictive value (NPV), which measures the rate at which the growths were accurately diagnosed as being benign. You want this to be very high, because that would indicate that very few melanomas were getting misdiagnosed as harmless. And the trained clinicians nailed these, having an NPV of 97 percent. The apps? Well, even the best of them misdiagnosed more than a quarter of the melanomas. The worst missed more than a third. In referring to the automated systems, the authors note, "Even the best-performing among these three applications classified 18 of 60 melanomas (30 percent) in our study as benign."

That's a serious problem. If your app tells you something's OK, you're less likely to have it checked out by a doctor. And with melanoma, time is critical. Growths that are caught early have a much better chance of being completely removed.

The authors declined to name any of the applications, simply referring to them by number throughout. But a quick search of the iTunes store for "melanoma" revealed one that charged $4.99 to have your photo forwarded to a pathologist. It also revealed another that offered to evaluate skin lesions for free, claiming that it will "help diagnose if you may have melanoma or basal cell carcinoma"—before later noting its results were "for educational purposes only."

But this search revealed you shouldn't discount all medical applications just because the diagnostic ones have problems. The top search result is a free app from the University of Michigan that guides you through a skin self-exam in which you create documentation of anything notable so that you can go back and determine if there have been significant changes. It's no substitute for an exam performed by a trained physician—but it doesn't pretend to be.

JAMA Dermatology, 2013. DOI: 10.1001/jamadermatol.2013.2382 (About DOIs).