Internet Engineering Task Force J. Livingood Internet-Draft N. Mody Intended status: Informational M. O'Reirdan Expires: February 27, 2010 Comcast August 26, 2009 Recommendations for the Remediation of Bots in Large ISP Networks draft-oreirdan-mody-bot-remediation-01 Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on February 27, 2010. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. Livingood, et al. Expires February 27, 2010 [Page 1]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract This document contains recommendations on how large Internet Service Providers (ISPs) can manage the effects of large numbers of bot infected computers used by their subscribers via various remediation techniques. At the time that this document was published, computers infected by bots and the users of those computers comprise a substantial number of users for large ISPs. Those Internet users are exposed to risks such as loss of personal data, increased susceptibility to online fraud and/or phishing, and becoming an inadvertent participant in or component of an online crime, spam, and/or phishing network. Mitigating the effects of and remediating the installations of bots affecting large numbers of Internet users will make it more difficult for botnets to operate and could reduce the level of online crime on the Internet in general and/or on a particular ISP's network. Livingood, et al. Expires February 27, 2010 [Page 2]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 1 . Requirements Language RFC2119]. 2 . Key Terminology 2.1 . Bots RFC1459], where a continual, interactive presence can be a requirement for participating in the games, interacting with a computing resource, or other purposes. However, for the purposes of this document, all mention of bots should assume that the bots involved are malicious in nature. Such malicious bots shall generally be assumed to have been deployed without the permission or conscious understanding of a particular Internet user. Thus, without a user's knowledge, bots may transform the user's computing device into a platform from which malicious activities are conducted. 2.2 . Bot Networks, or Botnets Livingood, et al. Expires February 27, 2010 [Page 4]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 command and control topologies employed by bot masters make them much more resistant to deactivation. With the introduction of P2P, HTTP and other resilient communication protocols along with the widespread adoption of encryption, bots are considerably more difficult to identify and isolate from typical network usage. As a result increased reliance is being placed on anonmaly detection and behavioral analysis, both locally and remotely, to identify bots. 2.3 . Computer 2.4 . Malware 3 . Introduction and Problem Statement Livingood, et al. Expires February 27, 2010 [Page 5]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 This can present a major problem for an ISP for a number of reasons (not to mention of course the problems created for users). First, these bots can be used to send spam, in some cases very large volumes of spam. This spam can result in extra cost for the ISPs in terms of wasted network, server, and/or personnel resources, among many other potential costs or side effects. Such spam can also negatively affect the reputation of the ISP, their customers, and the email reputation of the IP address space used by the ISP (often referred to simply as "IP reputation"). In addition, these bots can act as platforms for directing, participating in, or otherwise conducting attacks on critical Internet infrastructure. Bots are frequently used as part of concerted Distributed Denial of Service (DDoS) attacks for criminal, political or other motivations. For example, bots have been used to attack Internet resources and infrastructure ranging from web sites, to email servers and DNS servers, as well as the critical Internet infrastructure of entire countries. Motivations for such coordinated DDoS attacks can range from criminal extortion attempts through to online protesting and nationalistic fervor. While any computing device can be infected with bots, the majority of bot infections affect the personal computers used by Internet end users. As a result of the role of ISPs in providing IP connectivity, among many other services, to Internet users, these ISPs are in a unique position to be able to attempt to detect and observe bot nets operating in their networks. Furthermore, ISPs may also be in a unique position to be able to communicate to Internet users which are their customers, when customers computers may have been determined to have been or possibly have been infected with one or more bots. From an end user perspective, knowing that their computer has been infected with one or more bots is very important information. Once they know this, they can take steps to remove the bot, protect themselves in the future, and resolve any problems which may stem from the bot infection. Given that bots can drain the local computing and network resources, enable theft of personal information (including personal financial information), enable the computer to be used from criminal activities (that may result in the Internet user being legally culpable), destroy or leave the PC in an unrecoverable state via 'kill switch' bot technologies, it is important to notify the user that they may be infected with a bot. As a result, the intent of this document is to provide a guide to ISPs and other organizations for the remediation of these computers infected with bots, so as to reduce the size of bot nets and minimize the potential harm that bots can inflict upon Internet infrastructure generally, as well as on individual Internet users. Efforts by ISPs Livingood, et al. Expires February 27, 2010 [Page 6]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 and other organizations could therefore, over time, reduce the pool of computers infected with bots on the Internet, which in turn could result in smaller bot nets with less capability for disruption. 4 . Important Notice of Limitations 5 . Detection, Notification and Remediation 5.1 . Detection of Bots Livingood, et al. Expires February 27, 2010 [Page 7]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 privacy of the personally identifiable information of their customers. The ISP also should not block legitimate traffic in the course of bot detection, and should instead employ detection methods, tools, and processes which seek to be non-disruptive, as well as being transparent to Internet users. Detection methods, tools, and processes may include things such as analysis of specific network and/or application traffic flows (such as traffic to an email server), analysis of aggregate network and/or application traffic data, data feeds received from other ISPs and organizations (such as lists of the ISP's IP addresses which have been reported to have sent spam), feedback from the ISP's customers or other Internet users, as well as a wide variety of other possibilities. It is likely that a combination of multiple bot detection data points will prove to be an effective approach in order to corroborate information of varying dependability or consistency, as well as to avoid or minimize the possibility of false positive identification of computers. Detection should also, where possible and feasible, attempt to classify a bot in order to confirm that it is malicious in nature, estimate the variety and severity of threats it may pose (such as spam bot, key-logging bot, file distribution bot, etc.), and to determine potential methods for eventual remediation. However, given the dynamic nature of botnet management and the criminal incentives to seek quick financial rewards, botnets frequently update or change their core malicious capabilities. As a consequence, botnets that are initially detected and classified by the ISP need to be continuously monitored and tracked in order to correctly identify the threat they pose at any particular point in time. Detection is also time-sensitive. If complex analysis is required and multiple confirmations are needed to confirm a bot is indeed present, then it is possible that the bot will do its damage (to either the infected computer or a remotely targeted system) before it can be stopped. This may mean that an ISP may need to balance the desire or need to definitively classify and/or confirm a bot, which may take an extended period of time, with the ability to predict the strong likelihood of a bot in a very short period of time. This also means that Internet users may benefit from the deployment of client- based software protections or other software tools, which can enable rapid performance of heuristically-based detection bot activity, such as the detection of a bot as it starts to communicate a bot net and execute some type of command. Any bot detection systems should also be capable of learning and adapting, either via manual intervention or automatically, in order to cope with a rapidly evolving threat. As noted above, detection methods, tools, and processes should ensure that privacy of customers' personally identifiable information is Livingood, et al. Expires February 27, 2010 [Page 8]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 maintained. While bot detection methods, tools, and processes are similar to spam and virus defenses deployed by the ISP for the benefits of their customers (and may be directly related to those defenses), attempts to detect bots should take into account the need of an ISP to take care to ensure that such personally identifiable information is properly protected. Finally, depending upon the geographic region within which an ISP operates, certain methods relating to bot detection may need to be included in relevant terms of service documents or other documents which are available to the customers of a particular ISP. There are several bot detection methods, tools, and processes that an ISP may choose to utilize, as noted in the list below. It is important to note that the technical solutions available are relatively immature, and are likely to change over time, and to evolve rapidly in the coming years. While these items are described in relation to ISPs, they may also be applicable to organizations operating other networks, such as campus networks and enterprise networks. a. Where legally permissible or otherwise an industry accepted practice in a particular market region, an ISP may in some manner "scan" their IP space in order to detect un-patched or otherwise vulnerable hosts. This may provide the ISP with the opportunity to easily identify Internet users who appear to already be or are at great risk of being infected with a bot. ISP's should note that some types of port scanning may leave network services in a hung state or render them unusable due to common frailties, and that many modern firewall and host-based intrusion detection implementations may alert the Internet user to the scan. As a result the scan may be interpreted as a malicious attack against the computer. Vulnerability scanning has a higher probability of leaving accessible network services and applications in a damaged state and will often result in a higher probability of detection by the Internet user and subsequent interpretation as a targeted attack. Depending upon the vulnerability being scanned, some automated methods of vulnerability checking may result in data being altered or created afresh on the Internet users computer which be a problem in many legal environments. b. An ISP may also communicate and share selected data, via feedback loops or other mechanisms, with various third parties. Feedback loops are consistently formatted feeds of real-time (or nearly real-time) abuse reports offered by threat data clearinghouses, security alert organizations, other ISPs, and other organizations. The data may include, but is not limited to, lists of the IP addresses computers which have or are likely to have a bot running, domain names or fully qualified domain names Livingood, et al. Expires February 27, 2010 [Page 9]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 (FQDNs) known to host malware and/or be involved in the command and control of botnets, IP addresses know to host malware and/or be involved in the command and control of botnets, recently tested or discovered techniques or detecting or remediating bot infections, new threat vectors, and other relevant information. Good examples of this include SNDS from Microsoft, XBL and PBL from Spamhaus and DSHIELD AS tool from SANS c. An ISP may use Netflow [RFC3954] or other similar passive network monitoring to identify network anomalies that may be indicative of botnet attacks or bot communications. For example, an ISP may be able to identify compromised hosts by identifying traffic destined to IP addresses associated with the command and control of botnets. In addition, bots can be idenfied when a remote host is under distribute attack because computers participating in the attack will likely be infected by a bot. d. An ISP may use DNS-based techniques to perform detection. For example, a given classified bot may be known to query a specific list of domain names at specific times or on specific dates (in the example of the so-called "Conficker" bot), often by matching DNS queries to a well known list of domains associated with malware. In many cases such lists are distributed by or shared using third parties, such as threat data clearinghouses. Alternative dynamic DNS based techniques may look for associations of domain names with known bad actor lists and networks with poor reputations, or heuristic techniques that rank the domain name based upon previously identified botnet usage and bot characteristics. e. User complaints: Because bot infected hosts are frequently used to send spam or participate in DDoS attacks, the ISP servicing those hosts will normally receive complaints about the malicious network traffic. Those complaints may be sent to RFC2142- specified [RFC2142] role accounts, such as abuse@ or postmaster@ or to abuse or security addresses specified by the site as part of its WHOIS (or other) contact data. f. ISPs may also discover likely bot infected hosts located at other sites; when legally permissible or otherwise an industry accepted practice in a particular market region, it may be worthwhile for ISPs to share evidence relating to those compromised hosts with the relevant remote ISP, with security researchers, and with blocklist operators. g. ISP's may operate or subscribe to services that provide sinkholing or honeynet capabilities. This enable the ISP to obtain realtime lists of bot infected computers as they attempt Livingood, et al. Expires February 27, 2010 [Page 10]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 to join the larger botnet or propagate. These technologies may allow the ISP to enumerate bot infections within their customer population. 5.2 . Notification to Internet Users 5.2.1 . Email Notification Livingood, et al. Expires February 27, 2010 [Page 11]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 is that the user, their email client, and/or their email servers could determine or classify such a notification as spam, which could delete the message or otherwise file it in an email folder that the user may not check on a regular and/or timely basis. Bot masters have also been known to impersonate the ISP or trusted sender and send fradulant emails to the users. This technique of solical engineering often leads to new bot infestations. Finally if the user's email credentials are compromised, then a hacker and/or a bot could simply login to the user's email account and delete the email before it is read by the user. 5.2.2 . Telephone Call Notification 5.2.3 . Postal Mail Notification 5.2.4 . Walled Garden Notification Livingood, et al. Expires February 27, 2010 [Page 12]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 While in many cases, the user is almost guaranteed to view the notification message and take any appropriate remediation actions, this approach poses can pose other challenges. For example, it is not always the case that a user is actively using a computer that uses a web browser or which has a web browser actively running on it. In one case, a user could be playing a game online, via the use of a dedicated, Internet-connected game console. In another case, the user may not be using a computer with a web browser when they are placed in the walled garden and may instead be in the course of a telephone conversation, or may be expecting to receive a call, using a Voice Over IP (VOIP) device of some type. As a result, the ISP may feel the need to maintain a potentially lengthy white list of domains which are not subject to the typical restrictions of a walled garden, which could well prove to be an onerous task, from an operational perspective. The ISP has several options to determine when to let the user out of the walled garden. One approach may be to let the user determine when to exit. This option is suggested when the purpose of the walled garden is to notify users and provide information on remediation only, particularly since notification is not a guarantee of successful remediation. It could also be the case that, for whatever reason, the user makes the judgment that they cannot then take the time to remediate their computer and that other online activities which they would like to resume are more important. Exit from the walled garden should involve require process to verify that it is indeed the user who is requesting exit from the walled garden and not the bot. Once the user acknowledges the notification, then the user decides to either remediate and then exit the walled garden, or exit the walled garden without addressing the issue. Another approach may be to enforce a stricter policy and require the user to clean the computer prior to permitting the user to exit the walled garden, though this may not be technically feasible depending upon the type of bot, obfuscation techniques employed by a bot, and/or a range of other factors. Thus, the ISP may also need to support tools to scan the infected computer and determine whether it is still infected or rely on user judgment that the bot has been disabled or removed. One challenge with this approach is that if the user has multiple computers sharing a single IP address, such as via a common home gateway device which performs Network Address Translation (NAT), then the ISP may need to determine from user feedback or other means that all affected computers have been remediated, which may or may not be technically feasible. A list of well known addresses for both operating system vendors and security vendors should be created. This can be referenced when Livingood, et al. Expires February 27, 2010 [Page 13]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 allowing access from the walled garden by end users in search of operating system and application patches. 5.2.5 . Instant Message Notification 5.2.6 . Short Message Service (SMS) Notification Livingood, et al. Expires February 27, 2010 [Page 14]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 notify the wrong user if the intended user changes their mobile number but forgets to update it with the ISP. 5.2.7 . Web Browser Notification Section 5.2.4). 5.3 . Remediation of Bot Infected Machines Livingood, et al. Expires February 27, 2010 [Page 15]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 more slowly than usual or you notice regular disk activity even when you are not doing anything. Ignoring this problem is not really an option since your personal information is currently at risk. Thus, bots need to be removed to protect your data and your personal information." It is also important to note that it may not be immediately apparent to the Internet user precisely which device has been or which multiple devices have been infected with a particular bot. This is because of the user's home-networking configurations and the growing prevalence of IP enabled devices within a household that now connect to the public Internet and/or Private IP networks. The consequence of this for an ISP is that remediation advice may not ultimately be actionable by the Internet user. For example, the Internet user may be operating a computer, a notebook, a video console and multimedia system on their personal network. All of these device may connect to the Internet via a single gateway appliance. Any of these devices can be infected with a bot through a number of vectors. yet the remediation advice is likely to be quite different and may or may not be directly serviceable by the Internet user. Diligence needs to be taken by the ISP in understanding the specific nature of the device that has been infected with a bot, and providing appropriate remediation advice. 5.4 . Guided Remediation Process http://update.microsoft.com/microsoftupdate/v6/ default.aspx?ln=en-us as well as to Apple MacOS updates at http://support.apple.com/kb/HT1338?viewlocale=en_US. 3. Explain how to configure the computer to automatically install updates for the OS, A/V and other common Web Browsers such as Microsoft Internet Explorer, Mozilla Firefox, Apple Safari, Opera, and Google Chrome. Livingood, et al. Expires February 27, 2010 [Page 16]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 4. The flow should also have the option for users to get professional assistance if they are unable to remove the bots themselves. If purchasing third party assistance, then the user should be encouraged to pre-determine how much they are willing to pay for that help. If the computer that is being remediated is old and can easily be replaced with a new, faster, larger and more reliable system for three or four hundred dollars, the it makes no sense to spend five or six hundred dollars to fix the old computer, for example. On the other hand, if the customer has a brand new computer that cost several thousand dollars, it might make perfect sense to spend the money in attempting to remediate it. 5. To continue, regardless of whether the user or a knowledgeable technical assistant is working on remediating the computer, their first task should be to determine which of multiple potentially- infected machines may be the one that needs attention (in the common case of multiple computers in a home network). Sometimes, as in cases where there is only a single directly-attached computer, or the user has been noticing problems with one of their computers, this can be easy. Other times, it may be more difficult. If the user is behind a home gateway/router, then the first task may be to ascertain which of the machines is infected. In some cases the user may have to check all machines to identify the infected one. 6. User surveys to solicit feedback on whether the notification and remediation process is effective and what recommended changes could be made in order to improve the ease, understandability, and effectiveness the remediation process. 7. If the user is interested in reporting his or her computer's bot infection to an applicable law enforcement authority, then the computer effectively becomes a cyber "crime scene" and should not be mitigated unless or until law enforcement has collected the necessary evidence. For individuals in this situation, the ISP should refer them to local, state, federal, or other relevant computer crime offices. (Note: Some "minor" incidents, even if highly traumatic to the user, may not be sufficiently serious for law enforcement to commit some of their limited resources to an investigation.) 6 . Security Considerations Livingood, et al. Expires February 27, 2010 [Page 17]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 o -01 version published Appendix B . Open Issues Section 3 Fix the odd list spacing in Section 5.1 Consider revision of the OS update links, to simplify them. Add some point about notification to large networks may not be useful -- such as coffee shops or hotels with WiFi networks. Authors' Addresses Jason Livingood Comcast Cable Communications One Comcast Center 1701 John F. Kennedy Boulevard Philadelphia, PA 19103 US Email: jason_livingood@cable.comcast.com URI: http://www.comcast.com Nirmal Mody Comcast Cable Communications One Comcast Center 1701 John F. Kennedy Boulevard Philadelphia, PA 19103 US Email: nirmal_mody@cable.comcast.com URI: http://www.comcast.com Livingood, et al. Expires February 27, 2010 [Page 19]

Internet-Draft Remediation of Bots in Large ISP Networks August 2009 Mike O'Reirdan Comcast Cable Communications One Comcast Center 1701 John F. Kennedy Boulevard Philadelphia, PA 19103 US Email: michael_oreirdan@cable.comcast.com URI: http://www.comcast.com Livingood, et al. Expires February 27, 2010 [Page 20]