TECHNICAL SESSIONS

All sessions taking place in the San Jose Marriott are denoted by the Marriott logo icon throughout the LISA '10 Web site. All other session locations are in the San Jose McEnery Convention Center.



See the floor plans: Marriott | Convention Center

Just Up! Videos of the presentations will be posted as soon as they become available. Access is currently restricted to USENIX and SAGE members and LISA '10 conference attendees. Not a member? Join today!

Conference full papers and formal proceedings are available to conference registrants immediately and to everyone beginning Wednesday, November 10. Everyone can view the abstracts and the proceedings front matter immediately.

Proceedings Front Matter: Title Page and List of Organizers | Table of Contents | Message from the Program Chair



Complete Proceedings (PDF)

NEW! E-Book Proceedings: Read the proceedings on the go in iPad-friendly EPUB format or Kindle-friendly Mobipocket format.

Tech Sessions: Wednesday, November 10 | Thursday, November 11 | Friday, November 12 | Invited Talk Speakers

Wednesday, November 10

8:45 a.m.10:30 a.m. Wednesday

A1–3/6–8

Opening Remarks, Awards, and Keynote Address LISA '10 Program Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands

View the Video

Keynote Address

The LHC Computing Challenge: Preparation, Reality, and Future Outlook Tony Cass, CERN

View the Video | Slides

Listen to the MP3

CERN's IT department, in collaboration with particle physics computing institutes around the world, has been preparing for the arrival of Large Hadron Collider (LHC) data since early 2000. Now that the accelerator is in operation—with routine operation at record colliding energies since March—I will report on how the computing infrastructure has stood up to the test of real data, compare today's reality to both recent tests and original predictions, and then risk some predictions about the future evolution of the Worldwide LHC Computing Grid.





10:30 a.m.–11:00 a.m. Break with Beverages and Snacks, Concourse 1

11:00 a.m.–12:30 p.m. Wednesday

REFEREED PAPERS A2/7 Session Chair: Cory Lueninghoener, Los Alamos National Laboratory A Survey of System Configuration Tools

Thomas Delaet, Wouter Joosen, and Bart Vanbrabant, DistriNet, K.U. Leuven Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

High Performance Multi-Node File Copies and Checksums for Clustered File Systems

Paul Z. Kolano and Robert B. Ciotti, NASA Ames Research Center Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Fast and Secure Laptop Backups with Encrypted De-duplication

Paul Anderson and Le Zhang, University of Edinburgh Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands IPv6: No Longer Optional

Richard Jimmerson, ARIN

View the Video

Listen to the MP3

When LISA '10 takes place, there will still be IPv4 address space remaining in the free pools of the Regional Internet Registries. When LISA '11 takes place, it will likely be fully depleted—gone. In this session, Richard Jimmerson, CIO of the American Registry for Internet Numbers (ARIN), will describe the key considerations for and benefits of IPv6 adoption and the steps all system and network administrators and engineers should be taking to prepare for IPv4 depletion challenges. INVITED TALKS II A3/6 Session Chair: Doug Hughes, D. E. Shaw Research, LLC Storage Performance Management at Weta Digital

Matt Provost, Weta Digital

View the Slides

View the Video

Listen to the MP3

Weta Digital has had to deal with enormous growth in storage from The Fellowship of the Ring through to Avatar. In this talk Matt will outline some of the problems with NFS storage the company has faced over the years and the solutions they have developed. This will include the design and evolution of DSMS, the system Weta uses to provide a global namespace, unifying multiple heterogeneous file servers and file systems. He will also cover NFS performance monitoring, management, and troubleshooting in a high-performance computing cluster environment and the challenges these present at a large scale. THE GURU IS IN C1/2 Session Chair: Tom Limoncelli, Google Inc. Time Management

Tom Limoncelli, Google, Inc.

View the Slides Thomas A. Limoncelli is an internationally recognized author, speaker, and system administrator. His books include The Practice of System and Network Administration (Addison-Wesley) and Time Management for System Administrators (O'Reilly). He received the SAGE 2005 Outstanding Achievement Award. He works at Google in NYC and blogs at https://EverythingSysadmin.com/.

12:30 p.m.–2:00 p.m. Lunch (on your own; head over to the concession cart in the Vendor Exhibition)

2:00 p.m.–3:30 p.m. Wednesday

REFEREED PAPERS A2/7 Session Chair: Matt Disney, Oak Ridge National Laboratory The Margrave Tool for Firewall Analysis

Timothy Nelson, Worcester Polytechnic Institute; Christopher Barratt, Brown University; Daniel J. Dougherty and Kathi Fisler, Worcester Polytechnic Institute; Shriram Krishnamurthi, Brown University Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Towards Automatic Update of Access Control Policy

Jinwei Hu, University of Western Sydney and Huazhong University of Science and Technology; Yan Zhang, University of Western Sydney; Ruixuan Li, Huazhong University of Science and Technology Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Awarded Best

Student Paper!

First Step Towards Automatic Correction of Firewall Policy Faults

Fei Chen and Alex X. Liu, Michigan State University; JeeHyun Hwang and Tao Xie, North Carolina State University Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Amy Rich, Tufts University Storage over Ethernet: What's in It for Me?

Stephen Foskett, Gestalt IT

View the Slides

View the Video

Listen to the MP3

Convergence is coming, or so the storage, network, and server vendors want us to believe. Fibre Channel is moving to Ethernet with FCoE, and SCSI is already there, thanks to iSCSI. But why is everything heading to Ethernet and what's the value to the system administrator and the business? This session gives an overview of this new "everything over Ethernet" world, from DCB to virtual I/O, and offers practical advice on systems architecture, how long to wait, and when to buy. INVITED TALKS II A3/6 Session Chair: Duncan Hutty, Carnegie Mellon University The 10 Commandments of Release Engineering

Dinah McNutt, Google

View the Slides

View the Video

Listen to the MP3

A system administrator who is educated in the features of a well-designed build and release process can become proactive and make sure that the products being developed internally and purchased are deployable and maintainable. I will present 10 commandments that I have found to be truths in my 20 years of building commercial software. This talk will help sysadmins understand more about how decisions made during the release engineering process can positively (or negatively) impact the successful deployment of software and will arm them with the ability to develop their own commandments and communicate those truths to vendors and development groups. THE GURU IS IN C1/2 Session Chair: TBA Disaster Recovery

Joseph Kern, Google, Inc.

3:30 p.m.–4:00 p.m. Break with Beverages in the Vendor Exhibition, Exhibit Hall 1

4:00 p.m.–5:30 p.m. Wednesday

PRACTICE AND EXPERIENCE REPORTS A2/7 Session Chair: Narayan Desai, Argonne National Laboratory When Anti-virus Doesn't Cut It: Catching Malware with SIEM

Wyman Stocks, NetApp Read the Abstract |

Full paper

View the Video

Listen to the MP3

In-Flight Mechanics: A Software Package Management Conversion Project

Philip J. Hollenback, Yahoo, Inc. Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Experiences with Eucalyptus: Deploying an Open Source Cloud

Rick Bradshaw and Piotr T Zbiegiel, Argonne National Laboratory Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands Commencing Countdown: DNSSEC On!

Roland van Rijswijk, SURFnet Middleware Services

View the Slides

View the Video

Listen to the MP3

DNSSEC is taking off: The root was signed this year and many top-level domains are DNSSEC-ready. The focus is now shifting to second-level domain owners; they can now start rolling out DNSSEC. But DNSSEC is a complex technology and many organizations face challenges if they want to deploy DNSSEC. This talk will focus on the why and how of DNSSEC: Why do you need it? What is the impact on your systems? How can you roll out DNSSEC? What can you expect when you roll it out? SURFnet's own experiences with DNSSEC will serve as a basis for this presentation. INVITED TALKS II A3/6 Session Chair: Doug Hughes, D. E. Shaw Research, LLC Postfix: Past, Present, and Future

Wietse Venema, IBM T.J. Watson Research Center

View the Slides

View the Video

Listen to the MP3

In the twelve years since its initial release by IBM, the open source Postfix mail system has become part of the email infrastructure. The system has proven itself on personal systems and on ISP infrastructures with tens of millions of mailboxes. After Postfix reached completion in 2006, the focus of development moved from building new functionality toward making the system more extensible and more survivable in the face of changing requirements and threats. In this presentation Wietse will review lessons learned and current developments and will offer some speculation about the future. THE GURU IS IN C1/2 Session Chair: William LeFebvre, Digital Valence, LLC ZFS

Richard Elling, Nexenta Systems Richard Elling has been designing and building dependable, networked computer systems to solve complex problems for more than 25 years. He was an early adopter of ZFS and has developed benchmarks and analysis techniques for evaluation of data protection schemes and performability of systems. He wrote Designing Enterprise Solutions with Sun Cluster 3.0 (Prentice Hall, 2002) and has authored many white papers, Sun BluePrints, and refereed papers on dependable systems and performability. He is a regular contributor to the ZFS community and is currently the Director of Solution Engineering for Nexenta Systems.

5:30 p.m.–6:30 p.m. Wednesday

Vendor Exhibition, Exhibit Hall 1 Exhibit Hall Happy Hour Join us at the Vendor Exhibition for refreshments and take the opportunity to learn about the latest products and technologies. Don't forget to get your vendor passport stamped!

6:30 p.m.–7:30 p.m. Wednesday

Concourse 1 Poster Session Check out the cool new ideas and the latest preliminary work on display at the Poster Sessions. Take advantage of an opportunity to mingle with colleagues who may be interested in the same area. The list of accepted posters is available here.

Thursday, November 11

9:00 a.m.10:30 a.m. Thursday

REFEREED PAPERS A2/7 Session Chair: Sean Kamath, PDI/Dreamworks Using TCP/IP Traffic Shaping to Achieve iSCSI Service Predictability

J. Bjørgeengen, University of Oslo; H. Haugerud, Oslo University College Read the Abstract |

Full paper View the Slides YAF: Yet Another Flowmeter

Christopher M. Inacio, Carnegie Mellon University; Brian Trammell, ETH Zurich Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Nfsight: NetFlow-based Network Awareness Tool

Robin Berthier, University of Illinois at Urbana-Champaign; Michel Cukier, University of Maryland, College Park; Matti Hiltunen, Dave Kormann, Gregg Vesonder, and Dan Sheleheda, AT&T Labs—Research Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: William LeFebvre, Digital Valence, LLC Visualizations for Performance Analysis (and More)

Brendan Gregg, Joyent

View the Slides

View the Video

Listen to the MP3

Visualizations that include heat maps can be an effective way to present performance data: I/O latency, resource utilization, and more. Patterns can emerge that would be difficult to notice from columns of numbers or line graphs, which are revealing previously unknown behavior. These visualizations are used in a product as a replacement for traditional metrics such as %CPU and are allowing end users to identify more issues much more easily (and some issues are becoming nearly impossible to identify with tools such as vmstat(1)). This talk covers what has been learned, crazy heat map discoveries, and thoughts for future applications beyond performance analysis. INVITED TALKS II A3/6 Session Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands Rethinking Passwords

William Cheswick, AT&T Labs—Research

View the Slides

View the Video

Listen to the MP3

Passwords and PINs are used everywhere these days, but their use is often painful. Traditional password advice and rules are seldom appropriate for today's threats, yet we labor with the password rules and servers of yesteryear. Strong passwords are weakening our security, and it is time to fix that. There are numerous proposals for new password solutions. I will present a few half-baked ideas. But there are good solutions available now. We are facing much more worrisome security challenges: we ought to get this easy stuff right. THE GURU IS IN C1/2 Session Chair: Alva L. Couch, Tufts University Interviewing and Job Hunting

Adam Moskowitz Adam Moskowitz has at various times been a programmer, a system administrator, a manager of sysadmins, and a technical trainer. He has been a Guru and speaker at past LISA conferences, has taught several LISA tutorials about interviewing, has run the LISA Advanced Topics Workshop for the past 13 years, and was the Program Chair of LISA '09. Adam doesn't blog, but he does maintain a Web page of cute pictures of his dog Ancho.

10:30 a.m.–11:00 a.m. Break with Beverages and Snacks in the Vendor Exhibition, Exhibit Hall 1

11:00 a.m.–12:30 p.m. Thursday

PRACTICE AND EXPERIENCE REPORTS A2/7 Session Chair: Carolyn Rowland, National Institute of Standards and Technology (NIST) Implementing IPv6 at ARIN

Matt Ryanczak, ARIN Read the Abstract |

Full paper

View the Video

Listen to the MP3

Awarded Best Practice

and Experience Report!

Internet on the Edge

Andrew Mundy, National Institute of Standards and Technology (NIST) Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Managing Vendor Relations: A Case Study of Two HPC Network Issues

Loren Jan Wilson, Argonne National Laboratory Read the Abstract |

Full paper

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Alva L. Couch, Tufts University System Administrators in the Wild: An Outsider's View of Your World and Work

Eben M. Haber, IBM Research—Almaden

View the Slides You understand the work of system administration, but how do you explain it to others? Since 2002 a group at IBM has been studying sysadmins in the wild to better understand how they work, both to inspire improvements in tools and practices and to explain the ever-growing human costs in enterprise IT. As outsiders we were fascinated by what we learned, so we've written a book on the subject to explain your work to the rest of the world. This talk provides a summary of our most important findings, supported by real-life footage of sysadmins at work. INVITED TALKS II A3/6 Session Chair: Tom Limoncelli, Google Inc. Enterprise-scale Employee Monitoring

Mario Obejas, Raytheon

View the Slides

View the Video

Listen to the MP3

Since June 2009, I have been the program manager responsible for successfully deploying an employee monitoring system in a 12K employee business unit of a large US firm. This talk is for anyone considering implementing employee monitoring and may include the following mission-specific elements: Data leak vectors

Privacy and legal concerns

Resistance to monitoring

Carefully crafting the communications

Deployment ramp-up considerations (e.g., who goes first, second, etc.)

Management challenges

What to do when part of it doesn't work THE GURU IS IN C1/2 Session Chair: Amy Rich, Tufts University Consulting

Strata Rose Chalup, Project Management Consultant, and Adam Moskowitz Strata Rose Chalup began as a fledgling sysadmin in 1983. She has been leading and managing complex IT projects for many years, serving in roles ranging from Project Manager to Director of Network Operations. She has authored a number of articles on management and working with teams and has applied her management skills on various volunteer boards, including BayLISA and SAGE. Strata has a keen interest in network information systems and new publishing technologies and built a successful consulting practice around being an avid early adopter of new tools, starting with ncsa_httpd and C-based CGI libraries in 1993 and moving on to wikis, RSS readers, and blogging. Another MIT dropout, Strata founded VirtualNet Consulting in 1993. Adam Moskowitz has at various times been a programmer, a system administrator, a manager of sysadmins, and a technical trainer. He has been a Guru and speaker at past LISA conferences, has taught several LISA tutorials about interviewing, has run the LISA Advanced Topics Workshop for the past 13 years, and was the Program Chair of LISA '09. Adam doesn't blog, but he does maintain a Web page of cute pictures of his dog Ancho.

12:30 p.m.–2:00 p.m. Lunch (on your own; head over to the concession cart in the Vendor Exhibition)

2:00 p.m.–3:30 p.m. Thursday

REFEREED PAPERS A2/7 Session Chair: Alva L. Couch, Tufts University Using Syslog Message Sequences for Predicting Disk Failures

R. Wesley Featherstun and Errin W. Fulp, Wake Forest University Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Awarded Best Paper!

Log Analysis and Event Correlation Using Variable Temporal Event Correlator (VTEC)

Paul Krizak, Advanced Micro Devices, Inc. Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Chukwa: A System for Reliable Large-Scale Log Collection

Ariel Rabkin and Randy Katz, University of California, Berkeley Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Mario Obejas, Raytheon Flying Instruments-Only: Navigating Legal and Security Issues from the Cloud

Richard Goldberg, Attorney at Law, Washington, DC

View the Slides

View the Video

Listen to the MP3

Although cloud computing is seen as a simple, low-cost alternative for storing, protecting, and providing access to important information, the legal and privacy concerns are largely being ignored. This talk addresses the following questions, among others: What legal risks are created when data is located "elsewhere"—and users don't know more than that? Can using cloud computing violate federal, state, or international data-privacy laws? Can everyone do everything right and still create unreasonable risks? Who is responsible if—or, more likely, when—something goes wrong? What precautions can be taken to solve these problems? And will that be enough? INVITED TALKS II A3/6 Session Chair: Amy Rich, Tufts University The Path to Senior Sysadmin

Adam Moskowitz

View the Slides

View the Video

Listen to the MP3

Being a senior system administrator is about more than knowing all the options to mount(8) or that modprobe is what's used to replace that buggy kernel module with the latest version. Rather, a good senior sysadmin will have a wide knowledge of relevant technical topics, in-depth knowledge of one or more technologies, good interpersonal skills, and the ability to manage "problem users" and will be comfortable making presentations to and negotiating with mid- and upper-level management. This talk will cover the skills a senior sysadmin needs and why they are necessary and will provide some suggestions for how to acquire these skills. THE GURU IS IN C1/2 Session Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands IPv6

Owen DeLong, DeLong Consulting Owen DeLong is an IPv6 Evangelist at Hurricane Electric and a member of the ARIN Advisory Council. Owen brings more than 25 years of industry experience. He is an active member of the system administration, operations, and IP Policy communities. In the past, Owen has worked at Tellme Networks (Senior Network Engineer); Exodus Communications (Senior Backbone Engineer), where he was part of the team that took Exodus from a pre-IPO start-up with two data centers to a major global provider of hosting services; Netcom Online (Network Engineer), where he worked on a team that moved the Internet from an expensive R&E tool to a widely available public access system accessible to anyone with a computer; Sun Microsystems (Senior Systems Administrator); and more.

3:30 p.m.–4:00 p.m. Break with Beverages, Concourse 1

4:00 p.m.–5:30 p.m. Thursday

REFEREED PAPERS A2/7 Session Chair: Matthew Sacks, GlassCode Inc. How to Tame Your VMs: An Automated Control System for Virtualized Services

Akkarit Sangpetch, Andrew Turner, and Hyong Kim, Carnegie Mellon University Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Empirical Virtual Machine Models for Performance Guarantees

Andrew Turner, Akkarit Sangpetch, and Hyong S. Kim, Carnegie Mellon University Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

RC2—A Living Lab for Cloud Computing

Kyung Dong Ryu, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, Stefan Berger, Dilma M Da Silva, Jim Doran, Frank Franco, Alexei Karve, Herb Lee, James A Lindeman, Ajay Mohindra, Bob Oesterlin, Giovanni Pacifici, Dimitrios Pendarakis, Darrell Reimer, and Mariusz Sabath, IBM T.J. Watson Research Center Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Mario Obejas, Raytheon Panel: Legal and Privacy Issues in Cloud Computing

Richard Goldberg, Attorney at Law, Washington, DC; Bill Mooz, VMware

View the Video

Listen to the MP3

INVITED TALKS II A3/6 Session Chair: William LeFebvre, Digital Valence, LLC Centralized Logging in a Decentralized World

Tim Hartmann and Jim Donn, Harvard University

View the Slides

View the Video

Listen to the MP3

As environments grow and systems become more complex, building and managing a usable centralized logging infrastructure can be a daunting task. In this talk, we will walk through our real-life experiences implementing a centralized logging infrastructure for our Network, Systems, Security, and Application teams. Over the past three years, we have had to change our strategies and architecture to account for organic customer growth, changes in team requirements, and evolutions in technology. THE GURU IS IN C1/2 Session Chair: Tom Limoncelli, Google Inc. Project Management

Strata Rose Chalup, Project Management Consultant Strata Rose Chalup began as a fledgling sysadmin in 1983. She has been leading and managing complex IT projects for many years, serving in roles ranging from Project Manager to Director of Network Operations. She has authored a number of articles on management and working with teams and has applied her management skills on various volunteer boards, including BayLISA and SAGE. Strata has a keen interest in network information systems and new publishing technologies and built a successful consulting practice around being an avid early adopter of new tools, starting with ncsa_httpd and C-based CGI libraries in 1993 and moving on to wikis, RSS readers, and blogging. Another MIT dropout, Strata founded VirtualNet Consulting in 1993.

5:30 p.m.–6:30 p.m. Thursday

Concourse 1 Poster Session Check out the cool new ideas and the latest preliminary work on display at the Poster Sessions. Take advantage of an opportunity to mingle with colleagues who may be interested in the same area. The list of accepted posters is available here.

6:30 p.m.–8:30 p.m. Thursday

Ballroom LISA '10 Reception: Adventure Carnival Join us for a dinner buffet and games designed to test your inner Indy. Show off your skills in boulder bowling, mingle as you root for your favorite dromedary in the camel races, or challenge your colleagues in the monkey brains shuffle. Collect raffle tickets for winning at each booth for the chance to win fabulous prizes.

Friday, November 12

9:00 a.m.–10:30 a.m. Friday

REFEREED PAPERS A2/7 Session Chair: Chad Verbowski, eBay PeerMon: A Peer-to-Peer Network Monitoring System

Tia Newhall, Jānis Lībeks, Ross Greenwood, and Jeff Knerr, Swarthmore College Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Keeping Track of 70,000+ Servers: The Akamai Query System

Jeff Cohen, Thomas Repantis, and Sean McDermott, Akamai Technologies; Scott Smith, Formerly of Akamai Technologies; Joel Wein, Akamai Technologies Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Troubleshooting with Human-readable Automated Reasoning

Alva L. Couch, Tufts University; Mark Burgess, Oslo University College and Cfengine AS Read the Abstract |

Full paper View the Slides INVITED TALKS I A1/8 Session Chair: Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands 10,000,000,000 Files Available Anywhere: NFS at Dreamworks

Sean Kamath and Mike Cutler, PDI/Dreamworks

View the Slides "I need it all, available anywhere. Physics, shymsics! Make it so." OK, Jean Luc, you asked for it. Since Dreamworks employees collaborate globally, we represent data so that it feels like a laptop has a 2PB drive. Learn how we leverage NFS and the automounter to maximum effect. We uncover vendor bugs and implementation inconsistencies as we push the performance envelope to the limit. It's about free thinking and breaking some rules that we all cling to. Step over the brink and prepare to leave with ideas you can implement in your environments that will change how you think. INVITED TALKS II A3/6 Session Chair: Cory Lueninghoener, Los Alamos National Laboratory Data Structures from the Future: Bloom Filters, Distributed Hash Tables, and More!

Thomas A. Limoncelli, Google, Inc.

View the Slides

View the Video

Listen to the MP3

Greetings, earthlings of the year 2010! I've traveled back in time to share with you some of the technologies that system administrators operate in the future. Chances are you know what a cache is and how to tune it. In the future, there will be glorious things such as "bloom filters," "distributed hash tables," and "NoSQL databases." I will reveal what they are and (more important) how to tune them. (This will be an informal talk with a lot of hand-waving.) THE GURU IS IN C1/2 Session Chair: Matthew Sacks, GlassCode Inc. Production Documentation

Matthew Sacks, GlassCode Inc.

View the Slides | Handout Matthew Sacks is a system administrator specializing in high-traffic Web sites and their accompanying infrastructures. He founded the USENIX Blog Team and is also the Founding Editor of TheBitsource.com, a successful online technical publication and research company. He has written technical articles for various IT and computing industry magazines such as InformIT.com, Linux Pro Magazine, and Sys Admin Magazine.

10:30 a.m.–11:00 a.m. Break with Beverages and Snacks, Concourse 1

11:00 a.m.–12:30 p.m. Friday

PRACTICE AND EXPERIENCE REPORTS A2/7 Session Chair: Æleen Frisch, Exponential Consulting Configuration Management for Mac OS X: It's Just Unix, Right?

Janet Bass and David Pullman, National Institute of Standards and Technology (NIST) Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

Anycast as a Load Balancing Feature

Fernanda Weiden and Peter Frost, Google Switzerland GmbH Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

iSCSI SANs Don't Have to Suck

Derek J. Balling, Answers.com Read the Abstract |

Full paper View the Slides

View the Video

Listen to the MP3

INVITED TALKS I A1/8 Session Chair: Tom Limoncelli, Google Inc. Operations at Twitter: Scaling Beyond 100 Million Users

John Adams, Twitter

View the Slides

View the Video

Listen to the MP3

John will cover many aspects of Twitter's scaling efforts, including: Finding the weak points in Ruby on Rails and repairing them

In-house peer-to-peer: High-speed deploys across thousands of machines in no time at all

Managing thousands of machines: Why you need a central machine database, now

User management: How do you onboard many new developers and still remain fault-tolerant?

Caching methodologies and Twitter's open source efforts

Asynchronous versus synchronous processing during request lifetime

Life after syslog: What do you do when it won't work anymore? INVITED TALKS II A3/6 Session Chair: Matt Disney, Oak Ridge National Laboratory Er, What? Requirements, Specifications, and Reality: Distilling Truth from Friction

Cat Okita

View the Slides Listen to the MP3

A humorous look at requirements and specifications: what they are and why you'd want them in the first place, with some ideas about how to create good requirements and specifications, recognize bad ones, and play nicely with others to end up with results that everyone can live with. THE GURU IS IN C1/2 Session Chair: Carolyn Rowland, National Institute of Standards and Technology (NIST) SELinux

Rik Farrow, Security Consultant Rik Farrow has been teaching UNIX security classes since 1987. He wrote the second book on UNIX security, as well as hundreds of security-related articles. His experience with Linux security goes back over ten years and has led him to believe that sandboxing applications with SELinux is not just a good idea, but necessary. Rik Farrow is also Editor of ;login:.

12:30 p.m.–2:00 p.m. Lunch (on your own)

2:00 p.m.–3:30 p.m. Friday

INVITED TALK A2/7 Session Chair: Alva L. Couch, Tufts University Using Influence to Understand Complex Systems

Adam J. Oliner, Stanford University

View the Video

Listen to the MP3

When a complex production system misbehaves, the debugging technique of first resort is to examine instrumentation logs, which are normally noisy and incomplete. We define a statistical notion of "influence" between components that gracefully handles this situation. Intuitively, two components influence each other if they tend to exhibit surprising behavior around the same time. We show how to efficiently compute and answer queries about influence and how to present results in a useful form as a Structure-of-Influence Graph among components, and we give example applications of these ideas to several production systems, including autonomous vehicles, clusters, and supercomputers. INVITED TALKS I A1/8 Session Chair: Duncan Hutty, Carnegie Mellon University Scalable, Good, Cheap: Get Your Infrastructure Started Right

Avleen Vig, Patrick Carlisle, and Marc Cluet, woome.com

View the Slides

View the Video

Listen to the MP3

Traditional organic infrastructure growth enables quick start-up but increases your technical debt, while spending too long planning is resource-heavy and can result in lost opportunities. We'll discuss how to balance your infrastructure's growth with planning for the future, using the proper balance of open source tools, good process, architecture and design, monitoring, configuration management, and the data mining you will need to empower your systems. Learning how to do the right things now will enable your future success. INVITED TALKS II A3/6 Session Chair: Matthew Sacks, GlassCode Inc. Reliability at Massive Scale: Lessons Learned at Facebook

Robert Johnson, Director of Engineering, Facebook, Inc.; Sanjeev Kumar, Engineering Manager, Facebook, Inc.

View the Video

Listen to the MP3

As the Facebook Web site and platform grow to an ever larger scale, one of the most difficult challenges is running reliably while constantly changing our product. Over the years we have developed a number of principles around avoiding large failures while making frequent, small changes to our system. These principles have allowed us to run with a low rate of serious incidents, but they still do occur. I'll be walking through the details of a recent site outage to illustrate the way these principles work and how things can go wrong when they aren't followed. THE GURU IS IN C1/2 Session Chair: Chad Verbowski, eBay Security

Jamie Adams, Trusted Computer Solutions Jamie Adams is the Principal Secure Systems Engineer for Trusted Computer Solutions. He has spent over 20 years in the business of operating system security and compliance, developing security solutions for the government and commercial companies. Jamie holds considerable expertise on compliance with industry standard guidelines (DISA STIGs, CIS, PCI, etc.) and is an expert on how to provide "real world" security at the IT foundation.

3:30 p.m.–4:00 p.m. Break with Beverages, Concourse 1

4:00 p.m.–5:30 p.m. Friday