As most techsec professionals, I’ve been asked to do more and more in application security matters, an area that I’ve usually seen specialist ninjas dominate due to the often extreme technical depth required of the realm.

I’ve written previously about how [in]competence seems to be very hard to self-identify in most humans even when not dealing with areas of high complexity, so why not have some quality guidance so that one does not sound like so many of our kind; talking out of their depth and giving bad advice.

So okay. How far should someone who wants to remain generalized, but wants to be able to offer some kind of limited competent analysis and dialogue on application security, go without going off the rails into overreaching bad advice? Appsec is pretty much the biggest deal in the information security field since everyone adopted garbage technologies as lame development platforms a few years back. Writing secure code has always been important, but things used to be a little different. The appstack is now wide and deep and, if you own the appstream, you can pretty much access/subvert/rob/blind everything that matters in most environments.

What I should really do, I thought, is ask for advice on how much confidence one who doesn’t specialize in this area should reasonably expect to have when putting together solutions to solve problems in this realm where the best experts spend most of their waking lives knee deep in it and I’m all over the place. I thought that who better to comment on the reach of a generalist in this area than a ninja who specializes in appsec and fights with the other experts on public mailing lists. To this end I asked a friend of mine who is one of those specialized appsec ninjas what he thought a generalist should know about application security, how far they should go when they talk about about subjects therein, and what were the most important areas that they should be familiar without making themselves part of the problem.

Not surprisingly, he had a lot to say on the subject.

With my background as a sysadmin, network engineer, integrator, enterprise implementer, general technology consultant, and process/workflow/control/whateverist, people have come to me to solve complicated problems on a routine basis. The most interesting of these scenarios tend to be when I’m presented with a half-solution from very competent people based on how they would approach a problem, but leave the details to me to pull together and fit those square pegs in their matching round holes.

Big developers tend to think that their skillset is the pinnacle of all technology competence and nerd achievement and also, perhaps secretly, all other technology people are aspiring to be them. If you have visited at any big dev shop, but especially Microsoft, Amazon, or Google (and I have), you will have seen this first hand.

Additionally, might also discover the same god-of-all-the-lesser-nerds mindset with many talented high end network architects and nanog-type network engineers, SAGE-type big bearded UNIX admins, and other specialized people. They, perhaps, under-appreciate that there is a lot more art and craft to managing complicated systems than just writing code, routing packets, or server uptime statistics exclusively.

To review:

As a generalist, how far can I go in being effective in this space?

What are some good authored sources of methodologies that people can read for them to begin thinking about how best to approach this iceberg situationaly.

How is it best to employ experts effectively: how to pivot from analysis of unit test results, running automated processes using assorted codebase analysis tools, and how far should a generalist go in performing the analysis themselves before punting to a specialist?

What would you have to say on the balancing act between developer time and secure coding practices; making the business case for quality and developer maturity instead of an absolute ship schedule.

This principal from a major application security SaaS vendor had this to say:

Network sysadmin types with those backgrounds and that have limited experience with any programming language including Unix shell, BASIC, LOGO, or hardware robotics/electronics and that can’t understand concepts like recursion or loops simply not need waste their time touching anything related to appsec or pen-testing. This is inclusive of pen-testing, social engineering, as well as appsec, and even physical pen-testing. Social engineering newbies have Metasploit and/or SET as the #1 tool in their toolbox, even if it means paying $15k to Rapid7 for the Pro version of Nexpose and some mid-to-senior level pen-tester a few grand to automate SET with a custom GUI. THEY STILL NEED TO KNOW BASICS! That being said, I know very few network or sysadmin or even audit people that want to do pen-testing or appsec that DO NOT have at least a little scripting experience. A little scripting experience is going to get someone reasonably far in pen-testing and appsec. A word on the tools: they suck. Even when you run all of the commercial ones, they still suck in the wrong hands. They say a fool with a tool is still a fool and it is 100 BILLION PERCENT TRUE. I do not want to cite specific people in our industry, but they exist. They have social engineered companies who want appsec and they run every commercial appsec tool (all app scanners, fuzzers, network vulnerability scanners, and static analysis tools) on everything they can get a URL, build button to drop companyname.com on, or box to “make all” with. They are bunk, but people keep buying their garbage, which in turn gives them more ability to buy more tools, which makes everyone in the appsec industry look bad. Then you’ve got other bunkos that have a SaaS offerings, who hasn’t updated their product in 5 years. It’s a perl script that fault-injects HTTP parameters and pumps the HTTP responses through a massive regex. Then, they have 19-23 year old kids with no degrees and no infosec experience, but perhaps some QA experience at A Major Game Company for three months, and verify the true positives by running through checklists. The concept has been tried in various other organizations and it is well known as a “scan factory” as if that’s somehow a good thing.

Clearly a dig at people who know who they are but no one else does. I love it when people do that.

Again, the tools blow. You have to use your brain even if you are writing all of the tools yourself or not. It’s the way that you use the tools or your own tools is what matters. Training kinda helps, but the problem is that SANS and Ethical Hacker programs just tells people the names of the tools and the slightest introduction to them. Really, they tell you nothing about how to use them. Nobody tells anyone how to use the tools. It’s simply not written down or available on YouTube. Offensive Security has some cool training for network pen-testing, but they don’t really get into appsec proper. The best training on appsec comes from Aspect Security and by reading the beejesus out of the OWASP website (and going retarded on the OWASP community, both online and IRL), and perhaps memorizing the MITRE CWE/CAPEC. Perhaps combined with a good pen-test mentality (from Offensive Security and learning how to use the tools on your own by playing with them on test grounds or real-world environments i.e. YEARS OF EXPERIENCE), this can make a ship leave the dockyards. If you want to set sail with appsec, you are in for quite a disappointment. NOBODY gets it. NOBODY. You can memorize TAOSSA and WAHH but this isn’t going to do anything for you. You can write these books and still be totally clueless when it comes to fixing root causes and root issues in our industry. This is where even the best and brightest fail. Anyone that’s done their tenure at Aspect Security is very rooted in awesomesauce, and certainly there are other people that kick all of Aspect Security’s butts, but they are one-offs and usually esoteric specialists in some arcane appsec art form.

I’ve heard examples made of people who everyone knows who specialize in XSS, SQLi, know everything there is to know about code audit in a particular favorite language, etc.

There are no appsec generalists. There are some super rockstars, but they are too busy at kicking ass and taking names to talk to anyone. So you have to totally learn this on your own. YOU ARE ON YOUR OWN. I suggest just becoming a rockstar. A search for “application security” on books24x7.com lists over 1500 books. I suggest reading all of those, and then doing the same on Safari Books Online. It should only take about two years of straight reading and doing nothing else to accomplish this, so it’s not that hard because it’s just work on your eyes and you can get a Kindle if your eyes strain too much. You can glue it to your stairmaster controls and stay in shape at the same time while reading all of them! After reading and knowing where to find the info you’ll need when you get stuck performing real world bid’ness, download every testing ground and every tool and use everything on everything until you get results that you like. This is likely going to take at least five years, and you’ll also probably want to be testing real world things during this time, too. Think of it like earning five different CCIE programs while being the only guy running the BGP traffic engineering at Level 3. You have to work sick hard and play around with things a lot until you get them right. Also, you’ll want to use things like statistics and math (at least a little bit) in order to judge if you are going in a good direction and are being successful.

Using metrics and statistics to assess success? Hooray!

Finally, you’ll need to write your own tools, tool wrappers, and books. It’s like studying for law school finals and having to outline every chapter of sixteen four thousand page reporters, which basically means you’re writing a four thousand page book that summarize the sixteen reporters and the lectures and work you’ve done, but really it’s all about getting into the psychology of the court cases and who was involved and trying to pretend you were in that court room. Legal metaphors aside, capturing the mindset is key. But there are some shortcuts and things I’ve learned to help others succeed. It actually helps quite a lot to know something about their background and what they are capable of doing. Becoming an appsec (or IMO, any infosec) expert means being a trusted adviser to an organization. Fulfilling this need requires multiple people in multiple roles on multiple teams. Yes. Teams. Building your dream team is basically what this is all about; you can’t be a lone hacker cowboy. It’s not like that. You get a tiger team like in the movie Sneakers. Robert Redford did not do it without that crazy blind guy and Dan Aykroyd as the conspiracy nutball. More over, in reality, the leaders are never as good as Hannibal or Redford. Everyone has to pull equal weight. When you work for a CISO, you need to have the same title as everyone: “Assistant to the CISO”. Flat hierarchy. Everybody can probably do some of everything, but people count on each other to get their bit done and done right the first time and all that jazz.

I usually see a random title and suffix of working for “the office of the CISO” when I run into members of these teams.

So when I say to a team guy/gal, “I am going to help you with appsec and pen-test,” it means a ton of loading on them is going to happen. I am going to assume it’s someone like me 7 years ago, or maybe someone like Ian Gorrie, or somewhere in-between but also representative of others skillsets that are somewhat like us — and probably as cool, but probably never going to be a ninja or super rockstar, even though Ian and I might be that cool someday.

I think I’m alright at some things.

On fundamentals:

He had more to say later about appsec in general and if one should make the extreme effort to be a practitioner, he provides more detail about what specific things should be done and learned.

Dood. It’s easy. #1 : Don’t be a douchebag. If you really can’t code at ALL, especially on ANY/EVERY platform in nearly EVERY language — don’t bother doing appsec. I don’t mean you have to know the differences between assemblers, but it would be nice to know who or where to go to if you need immediate answers to programming questions. Try like SafaiBooksOnline, Books24x7, Ebrary, a digital library, etc. Build a meta search engine and your own dorking libraries. DON’T sell yourself on being an appsec guy like a douche who doesn’t know that he can accomplish the same from a Python script as with a Unix or NT shell. (See also most QSAs)

Complaining about he general badness of auditors has been done before, but since so many that I run into view assessment = audit = pen test = appsec (which should pretty much deprecate any other opinion they might deliver after such a claim) it bears repeating.

Browser tools:

I suggest the Multi Links and All-in-One Gestures add-ons for Firefox, especially good when combined with the Web Developer Populate Form Fields drop-down or FireBug. I also like Fireforce with the skullsecurity lists (rockyou-75.txt is my current favorite, but I tend to mix it up with L517 and interesting related data). Fireforce is useful for when you don’t have a login, and haven’t yet stolen credentials with XSS or wherever else. You tried to login with SQLi, right? If you use any other Firefox add-ons, then you’re an idiot. Learn how to do it with a bookmarklet. Also see rule #1: Don’t write a 40 line Python script to do what you can do on-the-fly with a 14 character Unix or NT shell-fu command line. Burp will be able to handle all of that. Repeater is your new browser. You use Proxy History and the Comparer tool, too. Or Intruder. I use all of these tools. I hate Target and Spider and you will, too. Better to automate stuff faster using APIs and fast things you find around. I am guessing that O2 does this probably ok, but I’d probably just figure out how to do it at a Unix or NT shell (wherever I happen to be at the time, although as a side note, it’s nice to have some statically compiled tools for various platforms especially something like that previously mentioned Parrot VM). I really prefer to have Ruby, PHP, Python, bash, Perl, OSSP js, OpenSSL, and a few other things around whenever I’m on a command line, especially calculators like bc(1) and dc(1). It’s nice to know more than a few interpreters by heart. You might want to learn some statistics and encryption along with your shell-fu. If you can’t figure out a potentially vulnerable URI with potentially vulnerable insertion points (parameter names, parameter values, URI/REST/arbitrary parameters, known or arbitrary cookies, known or arbitrary headers, etc) using Burp Pro, then you may want to go back to x5s. I prefer attacking insertion points that I know affect the app itself, which may require a lot of workflow using your browser. Work for it! Use QA techniques such as equivalence classification and pairwise testing to provide some focus and attention to what you are doing. Get some books on exploratory testing and apply the techniques to security testing. Make sure you understand your target environment the best you can with the resources you have (also see laresconsulting.com techniques or similar). Yes, RDP hijacking is cool — especially when you’ve got Forest Admin and recreate what the current NT admins think is their Domain infrastructure. What would be better would just be to have system access to all of the target’s public-facing RDBMS servers — and an XSS Proxy or XSS Tunnel inside a few key employees’ browsers, so that you can constantly spit DriveSploitMSFbrowser_autopwn Fast-Track fu all over their target OS environment. You probably got RDBMS server access through a UDF injection or silly stored procedure, didn’t you? UDF is a cooler acronym than RDP. XSS is a lot cooler of an acronym than most people are willing to credit it. Remember, laresconsulting.com has to do a lot of pen-tests local (e.g. in the target’s Porsche while sitting in their parking space). You can do this stuff from space. –update–: Ok, as for the above, I barely do any of it anymore. I just don’t have the time while on assessments. I look for higher value AuthN/AuthZ targets most of the time. For the basic bug checks, I just slam the host with RAFT and SVNDigger lists (directory config and lists first, followed by files) using a properly configured DirBuster. While that’s running, I prop up SearchDiggity on the site if it’s an Internet one (or do similar digging myself if it’s something I have to Intranet or VPN into). Best of case scenario is that I have root access and can grab the source code, throw it into Fortify SCA (or run SecurityScope), and/or run audit tools, especially cvechecker. I also tend to look for high value vulnerabilities first, and try to fit parameters to them. For example, if I see a “/” in any parameter values, then I hit them with path traversals. If I see any “orderby/desc/asc” parameter names, then I hit their values with SQLi. Anything that includes user-controllable HTML gets some custom XSS checks by hand. Final point (since I get source code a lot) is that Fortify SCA can be made to easily do things when you specify custom analyzer views with the taint:web and diagrams showing.

–end update

A PHP example:

On working in enterprise environments:

So… um… it’s apps. XML is like 51% of data exchanged by apps in enterprises and it’s a lot like HTML. So learn that right away! I suggest if you have a Unix background, install libxml2 binaries (this is default on like Ubuntu, Cygwin, and Mac OS X I presume — or it SHOULD be) and type xmllint. Then read the –help and man page. Then start playing with it and XML and HTML files. You can even feed it a URL and drop to a shell. I recommend doing that first and using “help” inside the shell and you’ll figure it out real quick. Then learn to program more with XML — there’s a few other ways of doing that CLI with like xmlstream or some other lame commands, but you’ll probably want to learn lxml for Python or whatever language you want to learn or already know. I highly suggest the Refactoring HTML book for learning this too. Then you can go in multiple directions with that XML/HTML knowledge. You can use it for code knowledge, server configurations, as well as reading the crap in HTTP responses. HTTP is pretty big. Besides the Offensive Security network pen-test stuff, you’ll want to enhance your ability to apply this to the apps. Sure, SSL is important but ssllabs.com almost precludes getting too fancy with the openssl CLI or stunnnel. RFC 2616 and related are penultimate to understanding HTTP, but really you want to learn the web proxies, especially Fiddler2 and Burp Suite Free/Pro. You can skip this knowledge and become super elite, but why? Web proxies and MITM proxies in general are awesomesauce for speaking language with others and showing them things. The Microsoft Press Hunting Security Bugs book is great and it comes with web-based companion content. Really, you want to master something like Mallory instead, but a broad knowledge of all types of Web/MITM proxies is going to be a boon. You’ll probably want to learn uhooker, Echo Mirage, and WPE Pro as well while you’re at it. ImmDbg/Olly or what not might be great. You can play with protocols like an appdev designing and implementing the protocols in apps or better. It’s RE awesomesauce. There’s not too many good classes on RE, but I’ve seen the SANS material and it’s not horrible. Their RE and adv exploit dev classes are much better than their webapp pen-test classes. There’s a million ways to skin these cats and I recommend learning a bunch and moving on to getting closer and closer to the apps you intend to understand. Since you got HTTP under your belt, try out the different test grounds and real-world apps. See the names and types of parameters and how they work together — their associations to page and other parameters. Try to understand how the parameters might work on the backend, whether a web service, other service like LDAP, XML query structure (e.g. XPath, which you can also learn via xmllint), database, or files. Fall back to that SMTP know-how you learned from the Offensive Security stuff — email is common in app forms and is heavily service-oriented. I think it really helps to learn basic QA 101 here — knowing how to not repeat testing the same stuff over and over (removing equivalent classes, or recurring idioms — to say it another way). Know how to get to the “meat” of the problem that you are trying to solve. This is your brain at work! Combinatorial explosions are a neat concept — it’s like mashing buttons and finding awesome combos! Mash up some parameters and see how they go. Start with things like login and password fields. Sometimes the authentication is done via a single query string, therefore you can combine stuff like ‘ OR 1=1/* (for the login) and */– for the password. Understand that if you want to do security, there are no shortcuts and that everyone MUST be tested, but realize that some kinds of testing will give you quicker results (and better/faster understanding). Build some fuzzers and fault-injectors now!

What if someone doesn’t want to spend a decade learning all of that?

There are some really cheap wins in appsec that just brute-force awesomeness. Like ProxyFuzz. IMO, some of the webapp fault-injectors are classes above the rest, and the ones that are more actively developed are nice, such as arachni or w3af. However, stability can be a super differentiator, which is why many people rely on tools such as Burp Pro and techniques as seen in Gray Hat Python. The funny thing is that even these are not stable — you have to gauge their stability at handling any one particular task depending on a lot of factors. Knowing how to scale and do network/system performance stuff can really help here. I wish I used iftop more to see how fast any given brute-force, fuzz, or fault-injection tool is performing (and we all wish we could just test everything from a localhost — not a bad strategy. Often RE’ing and making your own version of something in a lab is faster than testing it in a prod, dev, or test system). Ease-of-use combined with power are also nice. It shows in all versions of Metasploit and definitely in Burp Pro and Netsparker Pro. I fight the typical mindshare/mindset stuff usually, and perhaps I’m wrong about that, but Appscan, WebInspect, and Codenomicon just don’t do it for me when the free Wapiti and skipfish + the cheaper Netsparker and Acunetix seem to do so much better with the testing grounds and real world apps. Static analysis is straightforward: one loads Fortify into Eclipse for small-to-medium sized projects that involve almost all code except corner cases or unsupported languages. Languages like SFDC Apex are best done with Checkmarx, especially when there are just snippets of code and not a buildable system (good for cloud based apps). Knowing how to build the code is important. Know your Eclipse/Ant and VisualStudio/MSBuild well by building tons of code — easy to find in the world of open-source. Especially try large or difficult projects which you can find via ohloh.net. If you can’t get Fortify, then do your best with FindBugs and CAT.NET, perhaps Pixy or others. The C language stuff is straightforward if you are using VisualStudio — tons of resources out there. But for others, it’s probably ideal to try the Hacking Exposed Linux 3E way of using either Klocwork or Coverity, depending on your industry’s code types. Embedded/aviation/etc should probably be Klocwork and Enterprise is usually Coverity with tons of rigor being applied to these. Also see the SATE analysis stuff, especially if you are hitting C/C++ hard. If you don’t understand the code parts, you’ll have to go towards it slowly over time as static analysis isn’t the best approach if you already can pen-test stuff well from a full-knowledge system style test (blackbox is a total misnomer here). I suggest learning how to do server configuration traversals matching inputs (e.g. URLs and parameters) to the XML config file, to the jar/container/etc file, to the objects/source files, to the line of code [LOC]. Then it’s a matter of understanding how to read code (try the Code Reading book) and understand a good security approach (try TAOSSA Code Audit Strategies, Chapter 4). There are good books on debugging and unit testing that may also be worthwhile in this study. Once you get to the LOC, you can either use a Fortify-in-IDE to trace the dataflow, or alternatively, just learn how to do this work using things like the OWASP code review guide, searching millions of books at once on books24x7 and safaribooksonline, etc. Maybe it’s best to work with developers. Sometimes the code is in a stored procedure in a database that makes an exec call or similar — so the code may not even be included in the buildable containers. There is a problem with static analysis tools and even manual code review that will lead to “lost sinks.”

Their practical advice for non-devs in code review: Don’t.

More and more, I think non-devs should skip over the code review, automated partially/fully or NO, and stick with understanding the framework (this assumes managed code). Managed code is awesome and the VMs themselves can be tweaked. Instead of just playing with Eclipse/Ant or VisualStudio/MSBuild warnings/errors/bug-finding-plugins — one can jump directly to the underlying framework and change the VM (i.e. harden the VM like one would harden a Cisco router or Linux box) using a tool like Reframeworker (see the Managed Code Rootkits book). So then, a security dood can type up all of the mistakes his/her developer-friends are making and tell the VM to spit out an exception, say, when the developer builds the code on a system with the hardened VM and uses, say, Statement instead of preparedStatement (and the other billion things to check for). This could potentially be a nice human-facing blacklist, much like the banned functions list that you can throw into VisualStudio. Developers can work around these little intracacies, which changes the human factor up again. They can do lots of good and bad things. It’s good to know what they are doing, so you really have to be a developer in order to do this — or at least read their code and start to understand the psychology. Maybe you just need to interview them and learn a lot about liars and their lying (like that TV show with Tim Roth). I like to take an architecture perspective because it’s totally high level and lead devs (who like to talk about this stuff) grok it well. This is again, usually only possible with OO and probably managed code languages. It has to do with patterns. Patterns are certainly something that netengs/sysadmins or whoever else can usually get, and if not, at least the basics. One can dig pretty deep here. I have tons of ideas, but I’m not sure they will pan out. I think this is a huge area for excitement and improvement. This leads me to threat-modeling discussion, which is really a horrible term. The BSI / Cigital is aligned with the old style OOAD, which I think has been ultimately replaced by DDT in the same way that CASE has been replaced by prototyping/refactoring, modern unit/component testing, and static analysis. They called it something silly like Software Architectural Risk Analysis. Which brings me to BSSIM / OpenSAMM / Microsot SDL, etc. This stuff is trash. Replace the above with actual real dev/QA improvements. I think appsec people should be adding appdev, apptest, and appperf value! It’s all about the trusted adviser role and teams, as before said over and over.

One might be surprised how often it is, in fact, not said over and over in the context where it needs to be said. Instead what is often implemented is a hamster wheel of testing, detecting a few defects, fixing those low hanging fruit, certifying the resulting code product as “good enough” and then moving on without actually improving the process that generates flaws and usually imposing an ineffective and expensive bureaucratic cost to development. I hear what he’s saying, but it’s my view that one needs some kind of process to encourage developer maturity programs.

I’ve seen a lot of manhours spent pushing secure development lifecycle efforts and development maturity frameworks to disapproving audiences. Sadly, what I have seen happen more often than not is that it becomes not a developer maturity program that leads to fewer defects and a more efficient higher quality process, but a checkbox-driven recurring third-party code-audit, threat modeling, and pen testing cycle. These can be useful, but only when performed at the right times in the development process and could easily be out performed by an approach similar to what they describe.

One conundrum that is really unsettled is the exploitation and risk management stuff. I have my own opinions, but they match Aspect Security very closely. This stuff also takes experience, but experience with knowing the business, business risk, data breach notification laws and how they play out, auditors for various compliance/regulatory crap and how that plays out, etc. It’s really whacky stuff that is so custom to any org.

I’ll be writing up a risk management thinktank churn piece sometime soon going into this (again) in greater detail.