Answer by Hyrum Anderson, Technical Director of Data Science at Endgame, on Quora:

As we enter into 2018, what are some of the topics in AI/ML that are mostly hype?

Let me break this down into a few categories.

First, just a general note about how artificial intelligence (AI) and machine learning (ML) have been misapplied generally in the infosec market. While it’s not incorrect to label what some infosec companies are doing as “Artificial Intelligence”, it’s certainly imprecise, and one can’t help but wonder whether there’s some latent hope to impress by sophistication. Let’s all just agree to this: if we’re talking about an autonomous agent intended to behave and interact and reason, I’m totally fine labeling that as AI. If we’re talking about a model trained on data to make predictions, let’s stick with ML (a subfield of AI). It’s probably the case that 90+% of product features that companies market as “AI” is actually “narrow AI” and specifically “ML”, but might be intended to conjure up images of “general AI” Jarvis or Hal9000. I think this is a disservice to customers/users, who at this point are sophisticated and intelligent enough to tell the difference. (Full disclosure: at Endgame, we market our Artemis intelligent assistant chatbot an AI, because it is an “agent that behaves/reasons/interacts”. Our next-gen AV features that detect malware and evil? Machine Learning.)

Next, let me take a kinder view of “overhyped” AI/ML in the sense that there’s a lot of excitement and buzz, but the real end-user product implications have yet to emerge, especially in infosec. In my opinion, it’s important to not dismiss this as compelling research, but it might require a few more years to move from “cool research” to “useful product”. In this category of “interesting research, but hard to pull off reliably at scale”, I’m going to lump in things I’ve researched and published on: generative adversarial networks (GANs) for infosec, and reinforcement learning (RL) for infosec. These are really cool topics that are moving very quickly, but in my experience, don’t work “right out of the box” for many infosec applications. (I say this only because of the large number of hours of my life spent tweaking and fiddling trying to get them to perform as hoped for infosec problems. With some marginal success.) Generally speaking, GANs are seeing a ton of research activity with impressive results—the excitement is totally warranted. Unfortunately, there’s also been a lack of systematic and objective evaluation metrics in their development. See: [1711.10337] Are GANs Created Equal? A Large-Scale Study.

As an industry, we’re getting better at being more precise in marketing and labeling. But some of the loaded buzz-words will linger. I think it’s incumbent on industry leaders to better educate customers about how to cut through the hype. Ask the right questions. Look beyond sophisticated technology and focus on how, objectively, results stack up against competing solutions. Who cares if it’s deep learning or hand-crafted features or rules or signatures? What’s the FP rate? What’s the TP rate? As judged by a third party test, when possible, or using one’s own evaluation technique when not possible.

In general for 2018: ask careful questions about “AI” in infosec. Applaud new research applied to infosec problems, but demand results (from third-party testing, where available) no matter how it’s built.

This question originally appeared on Quora - the place to gain and share knowledge, empowering people to learn from others and better understand the world. You can follow Quora on Twitter, Facebook, and Google+. More questions: