EHLO again! I had the pleasure of speaking at QCon NYC last week and I must say it was a pretty damn good conference. Unlike most of the conferences I’ve spoken at, this one was a developer conference. For anyone that likes speaking on security-related topics, I can’t recommend speaking at developer conferences strongly enough. It’s great to speak with one another in the security industry about all the problems plaguing the state of security in the world, but nine times out of 10 we are not the ones with boots on the ground responsible for fixing the myriad holes that we find. This responsibility quite often falls on the shoulders of developers, and as such we should see it as our responsibility to work closely with the software development community to equip them with the knowledge required to improve the general security posture of software.

But I digress – the talk that I gave was entitled Addressing Security Regression Through Unit Testing. This post is a write-up of the topic and a run-through of the software that I wrote to support the talk. I hope this content serves to inspire some of you to implement similar methods in your own codebases!!

The slides for the talk can be found on my Slideshare here:

The code for the talk can be found here:

https://github.com/lavalamp-/security-unit-testing

Once the talk recording is up, I’ll post a link as well.

Security Regression

Regression in codebases is a known and largely addressed subject. The problem of regression is, simply put, that there is no guarantee that the integrity of code is maintained as new code is added. In order to address this problem developers commonly rely upon unit tests – they write unit tests that check whether or not small components of code are working as intended. These tests are then run prior to deployment as new functionality is added to ensure that the new functionality has not broken the older, unit tested functionality.

Through my time in penetration testing, I’ve come to the conclusion that regression with respect to security is a similarly large (if not larger) problem. The number of times that I’ve been on an engagement, found a number of issues in software, counseled the software owners through what the problems were and how to properly fix them, verified that the proper fixes were in place, and then came back six months later to find that the problems were back is far more than I would like to admit. Sometimes the vulnerabilities were back in the same place that they had been found previously. Sometimes the same type of vulnerability was present in new functionality. In all cases though, it seemed that there was little lasting improvement to application security posture as a result of the engagement I had conducted. We can have a much longer conversation around the shortcomings of offensive security testing (which I may reserve for another blog post), but regardless of the details it appeared that even when teams were able to properly address vulnerabilities in their codebases/networks/environments there was no guarantee that those fixes would have any bearing on the continued improvement of the affected organization’s security posture.

So what can we do to start addressing this problem? Well that’s the purpose of this talk and blog post. It turns out we can use the same techniques that developers use to address integrity regression to address security regression. Furthermore, by using a technique that so much infrastructure has already been built around, we get additional improvements to the security posture of the affected codebase by leveraging that infrastructure (continuous integration/deployment infrastructure for example). Not only that but we can use introspection to dynamically generate security unit tests that will provide us with guarantees around the security posture of code that hasn’t even been written yet. Put altogether, we can use unit tests to address security regression to a significant extent.

Dynamically Generating Unit Tests

The code for this blog post is written using the Django Web Framework. The techniques discussed here will certainly work with other frameworks (and perhaps even compiled languages), but one core component that we’ll be leveraging here is the presence of an explicit mapping of URL routes to the views that handle requests to those routes. For example:

URL routes in settings.py urlpatterns = [ # Admin url(r"^admin/", admin.site.urls), # Posts url(r"^$", views.PostListView.as_view(), name="post-list"), url(r"^my-posts/?$", views.MyPostsListView.as_view(), name="my-posts"), url(r"^new-post/?$", views.CreatePostView.as_view(), name="new-post"), url(r"^view-post/(?P<pk>[-\w]+)/?", views.PostDetailView.as_view(), name="view-post"), url(r"^edit-post/(?P<pk>[-\w]+)/?", views.EditPostView.as_view(), name="edit-post"), url(r"^delete-post/(?P<pk>[-\w]+)/?", views.DeletePostView.as_view(), name="delete-post"), url(r"^post-successful/(?P<pk>[-\w]+)/?", views.SuccessfulPostDetailView.as_view(), name="post-successful"), url(r"^get-posts-by-title/?$", views.GetPostsByTitleView.as_view(), name="get-post-image"), # Authentication url(r"^login/?$", auth_views.login, {"template_name": "pages/login.html"}, name="login"), url(r"^logout/?$", auth_views.logout, {"template_name": "pages/logout.html"}, name="logout"), url(r"^register/?$", views.CreateUserView.as_view(), name="register"), url(r"^register-success/?$", views.CreateUserSuccessView.as_view(), name="register-success"), # Error Handling url(r"^error-details/?$", views.ErrorDetailsView.as_view(), name="error-info"), # Redirection url(r"^redirect/?$", views.RedirectView.as_view(), name="redirect"), ] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 urlpatterns = [ # Admin url ( r "^admin/" , admin . site . urls ) , # Posts url ( r "^$" , views . PostListView . as_view ( ) , name = "post-list" ) , url ( r "^my-posts/?$" , views . MyPostsListView . as_view ( ) , name = "my-posts" ) , url ( r "^new-post/?$" , views . CreatePostView . as_view ( ) , name = "new-post" ) , url ( r "^view-post/(?P<pk>[-\w]+)/?" , views . PostDetailView . as_view ( ) , name = "view-post" ) , url ( r "^edit-post/(?P<pk>[-\w]+)/?" , views . EditPostView . as_view ( ) , name = "edit-post" ) , url ( r "^delete-post/(?P<pk>[-\w]+)/?" , views . DeletePostView . as_view ( ) , name = "delete-post" ) , url ( r "^post-successful/(?P<pk>[-\w]+)/?" , views . SuccessfulPostDetailView . as_view ( ) , name = "post-successful" ) , url ( r "^get-posts-by-title/?$" , views . GetPostsByTitleView . as_view ( ) , name = "get-post-image" ) , # Authentication url ( r "^login/?$" , auth_views . login , { "template_name" : "pages/login.html" } , name = "login" ) , url ( r "^logout/?$" , auth_views . logout , { "template_name" : "pages/logout.html" } , name = "logout" ) , url ( r "^register/?$" , views . CreateUserView . as_view ( ) , name = "register" ) , url ( r "^register-success/?$" , views . CreateUserSuccessView . as_view ( ) , name = "register-success" ) , # Error Handling url ( r "^error-details/?$" , views . ErrorDetailsView . as_view ( ) , name = "error-info" ) , # Redirection url ( r "^redirect/?$" , views . RedirectView . as_view ( ) , name = "redirect" ) , ]

The reason that this approach requires explicit mapping is that we will use introspection to look into these URL routes and use them to dynamically generate unit tests for all of the views registered in the application.

While we can use introspection to enumerate all of these views, one thing I did not particularly want to do dynamically was figure out how to invoke all of the different HTTP verb functionality (e.g. GET, POST, PUT, DELETE, etc) for all of the registered views with valid HTTP requests. For instance, sending a POST request to the CreatePostView requires title, description, and image parameters. Dynamically figuring out how to create a valid request is certainly possible but would take a significant amount of effort. And so, the approach that I’m using requires a small amount of additional code written for every view in the application in the form of a Requestor class. The base Requestor class is shown below:

BaseRequestor class in requestors/base.py class BaseRequestor(object): """ This is a base class for all requestor classes used by the Street Art project. """ # Class Members requires_auth = False supported_verbs = [] # Instantiation # Static Methods # Class Methods # Public Methods def get_delete_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP DELETE requests to the view. :param user: A string depicting the user to get DELETE data for. :return: A dictionary containing data to submit in HTTP DELETE requests to the view. """ return None def get_get_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP GET requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP GET requests to the view. """ return None def get_patch_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP PATCH requests to the view. :param user: A string depicting the user to get PATCH data for. :return: A dictionary containing data to submit in HTTP PATCH requests to the view. """ return None def get_post_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP POST requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP POST requests to the view. """ return None def get_put_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP PUT requests to the view. :param user: A string depicting the user to get PUT data for. :return: A dictionary containing data to submit in HTTP PUT requests to the view. """ return None def get_trace_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP TRACE requests to the view. :param user: A string depicting the user to get TRACE data for. :return: A dictionary containing data to submit in HTTP TRACE requests to the view. """ return None def get_url_path(self, user="user_1"): """ Get the URL path to request through the methods found in this class. :param user: A string depicting the user that the requested URL should be generated off of. :return: A string depicting the URL path to request. """ return None def send_delete(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a DELETE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.delete. :param kwargs: Keyword arguments for client.delete. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.delete( self.get_url_path(user=user_string), data=self.get_delete_data(user=user_string), *args, **kwargs ) def send_get(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a GET request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.get. :param kwargs: Keyword arguments for client.get. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.get( self.get_url_path(user=user_string), data=self.get_get_data(user=user_string), *args, **kwargs ) def send_head(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a HEAD request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.head. :param kwargs: Keyword arguments for client.head. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.head( self.get_url_path(user=user_string), *args, **kwargs ) def send_options(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send an OPTIONS request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.options. :param kwargs: Keyword arguments for client.options. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.options( self.get_url_path(user=user_string), *args, **kwargs ) def send_patch(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a PATCH request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.patch. :param kwargs: Keyword arguments for client.patch. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.patch( self.get_url_path(user=user_string), data=self.get_patch_data(user=user_string), *args, **kwargs ) def send_post(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a POST request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.post. :param kwargs: Keyword arguments for client.post. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.post( self.get_url_path(user=user_string), data=self.get_post_data(user=user_string), *args, **kwargs ) def send_put(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a PUT request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.put. :param kwargs: Keyword arguments for client.put. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.put( self.get_url_path(user=user_string), data=self.get_put_data(user=user_string), *args, **kwargs ) def send_request_by_verb(self, verb, *args, **kwargs): """ Send a request to the configured view based on the given verb. :param verb: The verb to send the request as. :param args: Positional arguments for the send method. :param kwargs: Keyword arguments for the send method. :return: The HTTP response. """ verb = verb.lower() if verb == "get": return self.send_get(*args, **kwargs) elif verb == "post": return self.send_post(*args, **kwargs) elif verb == "options": return self.send_options(*args, **kwargs) elif verb == "delete": return self.send_delete(*args, **kwargs) elif verb == "put": return self.send_put(*args, **kwargs) elif verb == "head": return self.send_head(*args, **kwargs) elif verb == "patch": return self.send_patch(*args, **kwargs) elif verb == "trace": return self.send_trace(*args, **kwargs) else: raise ValueError( "Unsure of how to handle HTTP verb of %s." % (verb.upper(),) ) def send_trace(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a TRACE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.trace. :param kwargs: Keyword arguments for client.trace. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.trace( self.get_url_path(user=user_string), data=self.get_trace_data(user=user_string), *args, **kwargs ) # Protected Methods # Private Methods # Properties # Representation and Comparison def __repr__(self): return "<%s - %s>" % (self.__class__.__name__, self.get_url_path()) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 class BaseRequestor ( object ) : """ This is a base class for all requestor classes used by the Street Art project. """ # Class Members requires_auth = False supported_verbs = [ ] # Instantiation # Static Methods # Class Methods # Public Methods def get_delete_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP DELETE requests to the view. :param user: A string depicting the user to get DELETE data for. :return: A dictionary containing data to submit in HTTP DELETE requests to the view. """ return None def get_get_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP GET requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP GET requests to the view. """ return None def get_patch_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP PATCH requests to the view. :param user: A string depicting the user to get PATCH data for. :return: A dictionary containing data to submit in HTTP PATCH requests to the view. """ return None def get_post_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP POST requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP POST requests to the view. """ return None def get_put_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP PUT requests to the view. :param user: A string depicting the user to get PUT data for. :return: A dictionary containing data to submit in HTTP PUT requests to the view. """ return None def get_trace_data ( self , user = "user_1" ) : """ Get a dictionary containing data to submit in HTTP TRACE requests to the view. :param user: A string depicting the user to get TRACE data for. :return: A dictionary containing data to submit in HTTP TRACE requests to the view. """ return None def get_url_path ( self , user = "user_1" ) : """ Get the URL path to request through the methods found in this class. :param user: A string depicting the user that the requested URL should be generated off of. :return: A string depicting the URL path to request. """ return None def send_delete ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a DELETE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.delete. :param kwargs: Keyword arguments for client.delete. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . delete ( self . get_url_path ( user = user_string ) , data = self . get_delete_data ( user = user_string ) , * args , * * kwargs ) def send_get ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a GET request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.get. :param kwargs: Keyword arguments for client.get. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . get ( self . get_url_path ( user = user_string ) , data = self . get_get_data ( user = user_string ) , * args , * * kwargs ) def send_head ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a HEAD request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.head. :param kwargs: Keyword arguments for client.head. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . head ( self . get_url_path ( user = user_string ) , * args , * * kwargs ) def send_options ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send an OPTIONS request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.options. :param kwargs: Keyword arguments for client.options. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . options ( self . get_url_path ( user = user_string ) , * args , * * kwargs ) def send_patch ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a PATCH request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.patch. :param kwargs: Keyword arguments for client.patch. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . patch ( self . get_url_path ( user = user_string ) , data = self . get_patch_data ( user = user_string ) , * args , * * kwargs ) def send_post ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a POST request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.post. :param kwargs: Keyword arguments for client.post. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . post ( self . get_url_path ( user = user_string ) , data = self . get_post_data ( user = user_string ) , * args , * * kwargs ) def send_put ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a PUT request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.put. :param kwargs: Keyword arguments for client.put. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . put ( self . get_url_path ( user = user_string ) , data = self . get_put_data ( user = user_string ) , * args , * * kwargs ) def send_request_by_verb ( self , verb , * args , * * kwargs ) : """ Send a request to the configured view based on the given verb. :param verb: The verb to send the request as. :param args: Positional arguments for the send method. :param kwargs: Keyword arguments for the send method. :return: The HTTP response. """ verb = verb . lower ( ) if verb == "get" : return self . send_get ( * args , * * kwargs ) elif verb == "post" : return self . send_post ( * args , * * kwargs ) elif verb == "options" : return self . send_options ( * args , * * kwargs ) elif verb == "delete" : return self . send_delete ( * args , * * kwargs ) elif verb == "put" : return self . send_put ( * args , * * kwargs ) elif verb == "head" : return self . send_head ( * args , * * kwargs ) elif verb == "patch" : return self . send_patch ( * args , * * kwargs ) elif verb == "trace" : return self . send_trace ( * args , * * kwargs ) else : raise ValueError ( "Unsure of how to handle HTTP verb of %s." % ( verb . upper ( ) , ) ) def send_trace ( self , user_string = "user_1" , do_auth = True , enforce_csrf_checks = False , * args , * * kwargs ) : """ Send a TRACE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.trace. :param kwargs: Keyword arguments for client.trace. :return: The HTTP response. """ client = Client ( enforce_csrf_checks = enforce_csrf_checks ) if self . requires_auth and do_auth : user = SaFaker . get_user ( user_string ) client . force_login ( user ) return client . trace ( self . get_url_path ( user = user_string ) , data = self . get_trace_data ( user = user_string ) , * args , * * kwargs ) # Protected Methods # Private Methods # Properties # Representation and Comparison def __repr__ ( self ) : return "<%s - %s>" % ( self . __class__ . __name__ , self . get_url_path ( ) )

The Requestor class contains all of the functionality required for our unit tests to invoke all of the functionality for a view in the application. The class contains methods for sending requests for each of the supported HTTP verbs, as well as methods for retrieving the data that should be supplied alongside a given HTTP verb request in order to invoke the functionality successfully. The class also contains a list of the HTTP verbs that the view supports as well as whether or not the view requires authentication. Creating a Requestor for a particular view is quite simple. For example, the Requestor class for the CreatePostView view is shown below:

CreatePostViewRequestor in tests/requestors/pages/post.py class CreatePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the CreatePostView view. """ supported_verbs = ["HEAD", "OPTIONS", "GET", "POST", "PUT"] def get_post_data(self, user="user_1"): return SaFaker.get_create_post_kwargs() def get_put_data(self, user="user_1"): return SaFaker.get_create_post_kwargs() def get_url_path(self, user="user_1"): return "/new-post/" 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class CreatePostViewRequestor ( BaseRequestor ) : """ This is a requestor class for sending requests to the CreatePostView view. """ supported_verbs = [ "HEAD" , "OPTIONS" , "GET" , "POST" , "PUT" ] def get_post_data ( self , user = "user_1" ) : return SaFaker . get_create_post_kwargs ( ) def get_put_data ( self , user = "user_1" ) : return SaFaker . get_create_post_kwargs ( ) def get_url_path ( self , user = "user_1" ) : return "/new-post/"

So now that we have the ability to enumerate all of the views found within the application, and we have the Requestor classes for invoking the functionality for all of these views, the last thing we need is to establish a mapping from the views to the requestors written for them. To accomplish this I introduce the notion of the Registry , which contains a dictionary mapping views to the Requestor classes written to invoke the relevant view functionality:

TestRequestorRegistry in tests/registry.py @Singleton class TestRequestorRegistry(object): """ This is a class that maintains mappings from views to test cases that are configured to send HTTP requests to the view in question. """ # Class Members # Instantiation def __init__(self): self._registry = {} # Static Methods # Class Methods # Public Methods def add_mapping(self, requestor_path=None, requested_view=None): """ Add a mapping from the view to the given requestor class specified by requestor_path. :param requestor_path: The path to the requestor class configured for the given view. :param requested_view: The view that the requestor is meant to send requests for. :return: None """ try: requestor_class = self.__import_class(requestor_path) except (ImportError, AttributeError) as e: raise RequestorNotFoundException( "Unable to load requestor at %s: %s." % (requestor_path, e.message) ) if not issubclass(requestor_class, BaseRequestor): raise InvalidRequestorException( "Class of %s is not a valid requestor class." % (requestor_class.__name__,) ) self._registry[requested_view] = requestor_class def does_view_have_mapping(self, view): """ Check to see if a mapping exists between the given view and a requestor class. :param view: The view to check a mapping for. :return: Whether or not a mapping exists for the given view. """ return view in self.registry def get_requestor_for_view(self, view): """ Get the requestor configured to send requests to the given view. :param view: The view to retrieve the requestor for. :return: The requestor configured to send requests to the given view. """ return self.registry[view] def print_mappings(self): """ Print all of the mappings currently stored within the registry. :return: None """ for k, v in self.registry.iteritems(): print("%s --> %s" % (k, v)) # Protected Methods # Private Methods def __import_class(self, class_path): """ Import the class at the given class path and return it. :param class_path: The class path to the class to load. :return: The loaded class. """ components = class_path.split(".") mod = __import__(components[0]) for component in components[1:]: mod = getattr(mod, component) return mod # Properties @property def registry(self): """ Get the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. :return: the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. """ return self._registry 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 @ Singleton class TestRequestorRegistry ( object ) : """ This is a class that maintains mappings from views to test cases that are configured to send HTTP requests to the view in question. """ # Class Members # Instantiation def __init__ ( self ) : self . _registry = { } # Static Methods # Class Methods # Public Methods def add_mapping ( self , requestor_path = None , requested_view = None ) : """ Add a mapping from the view to the given requestor class specified by requestor_path. :param requestor_path: The path to the requestor class configured for the given view. :param requested_view: The view that the requestor is meant to send requests for. :return: None """ try : requestor_class = self . __import_class ( requestor_path ) except ( ImportError , AttributeError ) as e : raise RequestorNotFoundException ( "Unable to load requestor at %s: %s." % ( requestor_path , e . message ) ) if not issubclass ( requestor_class , BaseRequestor ) : raise InvalidRequestorException ( "Class of %s is not a valid requestor class." % ( requestor_class . __name__ , ) ) self . _registry [ requested_view ] = requestor_class def does_view_have_mapping ( self , view ) : """ Check to see if a mapping exists between the given view and a requestor class. :param view: The view to check a mapping for. :return: Whether or not a mapping exists for the given view. """ return view in self . registry def get_requestor_for_view ( self , view ) : """ Get the requestor configured to send requests to the given view. :param view: The view to retrieve the requestor for. :return: The requestor configured to send requests to the given view. """ return self . registry [ view ] def print_mappings ( self ) : """ Print all of the mappings currently stored within the registry. :return: None """ for k , v in self . registry . iteritems ( ) : print ( "%s --> %s" % ( k , v ) ) # Protected Methods # Private Methods def __import_class ( self , class_path ) : """ Import the class at the given class path and return it. :param class_path: The class path to the class to load. :return: The loaded class. """ components = class_path . split ( "." ) mod = __import__ ( components [ 0 ] ) for component in components [ 1 : ] : mod = getattr ( mod , component ) return mod # Properties @ property def registry ( self ) : """ Get the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. :return: the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. """ return self . _registry

In order to establish the mapping from views to their related Requestor classes, I use the following decorator:

requested_by decorator in tests/registry.py def requested_by(requestor_path): """ This is a decorator for views that maps a requestor class to the view that it is configured to submit requests to. :param requestor_path: A string depicting the local file path to the requestor class for the given view. :return: A function that maps the view class to the requestor path and returns the called function or class. """ def decorator(to_wrap): registry = TestRequestorRegistry.instance() registry.add_mapping(requestor_path=requestor_path, requested_view=to_wrap) return to_wrap return decorator 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 def requested_by ( requestor_path ) : """ This is a decorator for views that maps a requestor class to the view that it is configured to submit requests to. :param requestor_path: A string depicting the local file path to the requestor class for the given view. :return: A function that maps the view class to the requestor path and returns the called function or class. """ def decorator ( to_wrap ) : registry = TestRequestorRegistry . instance ( ) registry . add_mapping ( requestor_path = requestor_path , requested_view = to_wrap ) return to_wrap return decorator

This decorator can then be used to decorate the view classes, and the only argument to the decorator is the import path of the Requestor related to the view. For instance, the CreatePostView class is decorated as follows:

@requested_by("streetart.tests.requestors.pages.CreatePostViewRequestor") class CreatePostView(BaseFormView): """ This is a view for creating new street art posts. """ 1 2 3 4 5 @ requested_by ( "streetart.tests.requestors.pages.CreatePostViewRequestor" ) class CreatePostView ( BaseFormView ) : """ This is a view for creating new street art posts. """

With this approach, now every view will automatically be mapped to its related Requestor as soon as it is imported. To demonstrate this, run the following from the sectesting directory:

python manage.py shell -c "from sectesting import urls; from streetart.tests import TestRequestorRegistry; registry = TestRequestorRegistry.instance(); registry.print_mappings()"

The result of running this command is shown below:

With the mapping in place, we now have the ability to:

Enumerate all views in the application Retrieve a requestor for every view that has the ability to invoke all of the view’s functionality

Great! With this framework we can now dynamically generate unit tests for all of the HTTP verbs for all of the views in the application. This dynamic generation is handled by the StreetArtTestRunner class found in tests/runner.py , the contents of which are shown below:

StreetArtTestRunner contents from tests/runner.py class StreetArtTestRunner(DiscoverRunner): """ This is a custom discover runner for populating unit tests for the Street Art project. """ # Class Members ALL_HTTP_VERBS = [ "GET", "HEAD", "POST", "PUT", "DELETE", "OPTIONS", "TRACE", "PATCH", ] CSRF_VERBS = [ "POST", "PUT", "DELETE", "PATCH", ] # Instantiation def __init__(self, *args, **kwargs): self._url_patterns = None super(StreetArtTestRunner, self).__init__(*args, **kwargs) # Static Methods # Class Methods # Public Methods def build_suite(self, test_labels=None, extra_tests=None, **kwargs): """ Build the test suite to run for this discover runner. :param test_labels: A list of strings describing the tests to be run. :param extra_tests: A list of extra TestCase instances to add to the suite that is executed by the test runner. :param kwargs: Additional keyword arguments. :return: The test suite. """ extra_tests = extra_tests if extra_tests is not None else [] extra_tests.extend(self.__get_generated_test_cases()) return super(StreetArtTestRunner, self).build_suite( test_labels=test_labels, extra_tests=extra_tests, **kwargs ) def run_suite(self, suite, **kwargs): """ Override the run_suite functionality to populate the database. :param suite: The suite to run. :param kwargs: Keyword arguments. :return: The rest suite result. """ self.__populate_database() return super(StreetArtTestRunner, self).run_suite(suite, **kwargs) # Protected Methods # Private Methods def __get_authentication_enforcement_tests(self): """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) if not requestor.requires_auth: continue for supported_verb in requestor.supported_verbs: class AnonTestCase(AuthenticationEnforcementTestCase): pass to_return.append(AnonTestCase(view=view, verb=supported_verb)) return to_return def __get_csrf_enforcement_tests(self): """ Get a list of test cases that check to make sure that CSRF checks are being correctly enforced. :return: A list of test cases that check to make sure that CSRF checks are being correctly enforced. """ to_return = [] csrf_verbs = [x.lower() for x in self.CSRF_VERBS] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] supported_csrf_verbs = filter(lambda x: x in csrf_verbs, supported_verbs) for supported_csrf_verb in supported_csrf_verbs: class AnonTestCase1(CsrfEnforcementTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_csrf_verb)) return to_return def __get_dos_class_tests(self): """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for supported_verb in requestor.supported_verbs: class AnonTestCase1(RegularViewRequestIsSuccessfulTestCase): pass class AnonTestCase2(AdminViewRequestIsSuccessfulTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb)) to_return.append(AnonTestCase2(view=view, verb=supported_verb)) return to_return def __get_generated_test_cases(self): """ Get a list containing the automatically generated test cases to add to the test suite this runner is configured to run. :return: A list containing the automatically generated test cases to add to the test suite this runner is configured to run. """ # Ensure that all views are loaded import sectesting.urls to_return = [] if settings.TEST_FOR_REQUESTOR_CLASSES: to_return.extend(self.__get_requestor_class_tests()) if settings.TEST_FOR_DENIAL_OF_SERVICE: to_return.extend(self.__get_dos_class_tests()) if settings.TEST_FOR_UNKNOWN_METHODS: to_return.extend(self.__get_unknown_methods_tests()) if settings.TEST_FOR_AUTHENTICATION_ENFORCEMENT: to_return.extend(self.__get_authentication_enforcement_tests()) if settings.TEST_FOR_RESPONSE_HEADERS: to_return.extend(self.__get_response_header_tests()) if settings.TEST_FOR_OPTIONS_ACCURACY: to_return.extend(self.__get_options_accuracy_tests()) if settings.TEST_FOR_CSRF_ENFORCEMENT: to_return.extend(self.__get_csrf_enforcement_tests()) return to_return def __get_options_accuracy_tests(self): """ Get a list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. :return: A list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] for http_verb in self.ALL_HTTP_VERBS: if http_verb.lower() not in supported_verbs: class AnonTestCase1(RegularVerbNotSupportedTestCase): pass class AnonTestCase2(AdminVerbNotSupportedTestCase): pass to_return.append(AnonTestCase1(view=view, verb=http_verb)) to_return.append(AnonTestCase2(view=view, verb=http_verb)) return to_return def __get_response_header_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems(): for supported_verb in requestor.supported_verbs: class AnonTestCase1(HeaderKeyExistsTestCase): pass class AnonTestCase2(HeaderValueAccurateTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k)) to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v)) for excluded_header in settings.EXPECTED_RESPONSE_HEADERS["excluded"]: for supported_verb in requestor.supported_verbs: class AnonTestCase3(HeaderKeyNotExistsTestCase): pass to_return.append(AnonTestCase3(view=view, verb=supported_verb, header_key=excluded_header)) return to_return def __get_requestor_class_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. """ to_return = [] for _, _, callback in self.url_patterns: class AnonTestCase(ViewHasRequestorTestCase): pass to_return.append(AnonTestCase(self.__get_view_from_callback(callback))) return to_return def __get_unknown_methods_tests(self): """ Get a list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. :return: A list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. """ to_return = [] for _, _, callback in self.url_patterns: view = self.__get_view_from_callback(callback) class AnonTestCase1(RegularUnknownMethodsTestCase): pass class AnonTestCase2(AdminUnknownMethodsTestCase): pass to_return.append(AnonTestCase1(view)) to_return.append(AnonTestCase2(view)) return to_return def __get_view_from_callback(self, callback): """ Get the view associated with the given callback. :param callback: The callback to get the view from. :return: The view associated with the given callback. """ if hasattr(callback, "view_class"): return callback.view_class else: return callback def __get_view_and_requestor_from_callback(self, callback): """ Get a tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. :param callback: The URL pattern callback to process. :return: A tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. """ registry = TestRequestorRegistry.instance() view = self.__get_view_from_callback(callback) requestor = registry.get_requestor_for_view(view) return view, requestor def __populate_database(self): """ Populate the database with dummy database models. :return: None """ print("Now populating test database...") SaFaker.create_users() # Properties @property def url_patterns(self): """ Get a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. :return: a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. """ if self._url_patterns is None: self._url_patterns = UrlPatternHelper.get_all_streetart_views( include_admin_views=settings.INCLUDE_ADMIN_VIEWS_IN_TESTS, include_auth_views=settings.INCLUDE_AUTH_VIEWS_IN_TESTS, include_generic_views=settings.INCLUDE_GENERIC_VIEWS_IN_TESTS, include_contenttype_views=settings.INCLUDE_CONTENTTYPE_VIEWS_IN_TESTS, ) return self._url_patterns # Representation and Comparison def __repr__(self): return "<%s>" % (self.__class__.__name__,) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 class StreetArtTestRunner ( DiscoverRunner ) : """ This is a custom discover runner for populating unit tests for the Street Art project. """ # Class Members ALL_HTTP_VERBS = [ "GET" , "HEAD" , "POST" , "PUT" , "DELETE" , "OPTIONS" , "TRACE" , "PATCH" , ] CSRF_VERBS = [ "POST" , "PUT" , "DELETE" , "PATCH" , ] # Instantiation def __init__ ( self , * args , * * kwargs ) : self . _url_patterns = None super ( StreetArtTestRunner , self ) . __init__ ( * args , * * kwargs ) # Static Methods # Class Methods # Public Methods def build_suite ( self , test_labels = None , extra_tests = None , * * kwargs ) : """ Build the test suite to run for this discover runner. :param test_labels: A list of strings describing the tests to be run. :param extra_tests: A list of extra TestCase instances to add to the suite that is executed by the test runner. :param kwargs: Additional keyword arguments. :return: The test suite. """ extra_tests = extra_tests if extra_tests is not None else [ ] extra_tests . extend ( self . __get_generated_test_cases ( ) ) return super ( StreetArtTestRunner , self ) . build_suite ( test_labels = test_labels , extra_tests = extra_tests , * * kwargs ) def run_suite ( self , suite , * * kwargs ) : """ Override the run_suite functionality to populate the database. :param suite: The suite to run. :param kwargs: Keyword arguments. :return: The rest suite result. """ self . __populate_database ( ) return super ( StreetArtTestRunner , self ) . run_suite ( suite , * * kwargs ) # Protected Methods # Private Methods def __get_authentication_enforcement_tests ( self ) : """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) if not requestor . requires_auth : continue for supported_verb in requestor . supported_verbs : class AnonTestCase ( AuthenticationEnforcementTestCase ) : pass to_return . append ( AnonTestCase ( view = view , verb = supported_verb ) ) return to_return def __get_csrf_enforcement_tests ( self ) : """ Get a list of test cases that check to make sure that CSRF checks are being correctly enforced. :return: A list of test cases that check to make sure that CSRF checks are being correctly enforced. """ to_return = [ ] csrf_verbs = [ x . lower ( ) for x in self . CSRF_VERBS ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) supported_verbs = [ x . lower ( ) for x in requestor . supported_verbs ] supported_csrf_verbs = filter ( lambda x : x in csrf_verbs , supported_verbs ) for supported_csrf_verb in supported_csrf_verbs : class AnonTestCase1 ( CsrfEnforcementTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = supported_csrf_verb ) ) return to_return def __get_dos_class_tests ( self ) : """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) for supported_verb in requestor . supported_verbs : class AnonTestCase1 ( RegularViewRequestIsSuccessfulTestCase ) : pass class AnonTestCase2 ( AdminViewRequestIsSuccessfulTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = supported_verb ) ) to_return . append ( AnonTestCase2 ( view = view , verb = supported_verb ) ) return to_return def __get_generated_test_cases ( self ) : """ Get a list containing the automatically generated test cases to add to the test suite this runner is configured to run. :return: A list containing the automatically generated test cases to add to the test suite this runner is configured to run. """ # Ensure that all views are loaded import sectesting . urls to_return = [ ] if settings . TEST_FOR_REQUESTOR_CLASSES : to_return . extend ( self . __get_requestor_class_tests ( ) ) if settings . TEST_FOR_DENIAL_OF_SERVICE : to_return . extend ( self . __get_dos_class_tests ( ) ) if settings . TEST_FOR_UNKNOWN_METHODS : to_return . extend ( self . __get_unknown_methods_tests ( ) ) if settings . TEST_FOR_AUTHENTICATION_ENFORCEMENT : to_return . extend ( self . __get_authentication_enforcement_tests ( ) ) if settings . TEST_FOR_RESPONSE_HEADERS : to_return . extend ( self . __get_response_header_tests ( ) ) if settings . TEST_FOR_OPTIONS_ACCURACY : to_return . extend ( self . __get_options_accuracy_tests ( ) ) if settings . TEST_FOR_CSRF_ENFORCEMENT : to_return . extend ( self . __get_csrf_enforcement_tests ( ) ) return to_return def __get_options_accuracy_tests ( self ) : """ Get a list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. :return: A list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) supported_verbs = [ x . lower ( ) for x in requestor . supported_verbs ] for http_verb in self . ALL_HTTP_VERBS : if http_verb . lower ( ) not in supported_verbs : class AnonTestCase1 ( RegularVerbNotSupportedTestCase ) : pass class AnonTestCase2 ( AdminVerbNotSupportedTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = http_verb ) ) to_return . append ( AnonTestCase2 ( view = view , verb = http_verb ) ) return to_return def __get_response_header_tests ( self ) : """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) for k , v in settings . EXPECTED_RESPONSE_HEADERS [ "included" ] . iteritems ( ) : for supported_verb in requestor . supported_verbs : class AnonTestCase1 ( HeaderKeyExistsTestCase ) : pass class AnonTestCase2 ( HeaderValueAccurateTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = supported_verb , header_key = k ) ) to_return . append ( AnonTestCase2 ( view = view , verb = supported_verb , header_key = k , header_value = v ) ) for excluded_header in settings . EXPECTED_RESPONSE_HEADERS [ "excluded" ] : for supported_verb in requestor . supported_verbs : class AnonTestCase3 ( HeaderKeyNotExistsTestCase ) : pass to_return . append ( AnonTestCase3 ( view = view , verb = supported_verb , header_key = excluded_header ) ) return to_return def __get_requestor_class_tests ( self ) : """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. """ to_return = [ ] for _ , _ , callback in self . url_patterns : class AnonTestCase ( ViewHasRequestorTestCase ) : pass to_return . append ( AnonTestCase ( self . __get_view_from_callback ( callback ) ) ) return to_return def __get_unknown_methods_tests ( self ) : """ Get a list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. :return: A list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view = self . __get_view_from_callback ( callback ) class AnonTestCase1 ( RegularUnknownMethodsTestCase ) : pass class AnonTestCase2 ( AdminUnknownMethodsTestCase ) : pass to_return . append ( AnonTestCase1 ( view ) ) to_return . append ( AnonTestCase2 ( view ) ) return to_return def __get_view_from_callback ( self , callback ) : """ Get the view associated with the given callback. :param callback: The callback to get the view from. :return: The view associated with the given callback. """ if hasattr ( callback , "view_class" ) : return callback . view_class else : return callback def __get_view_and_requestor_from_callback ( self , callback ) : """ Get a tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. :param callback: The URL pattern callback to process. :return: A tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. """ registry = TestRequestorRegistry . instance ( ) view = self . __get_view_from_callback ( callback ) requestor = registry . get_requestor_for_view ( view ) return view , requestor def __populate_database ( self ) : """ Populate the database with dummy database models. :return: None """ print ( "Now populating test database..." ) SaFaker . create_users ( ) # Properties @ property def url_patterns ( self ) : """ Get a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. :return: a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. """ if self . _url_patterns is None : self . _url_patterns = UrlPatternHelper . get_all_streetart_views ( include_admin_views = settings . INCLUDE_ADMIN_VIEWS_IN_TESTS , include_auth_views = settings . INCLUDE_AUTH_VIEWS_IN_TESTS , include_generic_views = settings . INCLUDE_GENERIC_VIEWS_IN_TESTS , include_contenttype_views = settings . INCLUDE_CONTENTTYPE_VIEWS_IN_TESTS , ) return self . _url_patterns # Representation and Comparison def __repr__ ( self ) : return "<%s>" % ( self . __class__ . __name__ , )

To demonstrate how we dynamically generate tests, let’s take the RegularViewRequestIsSuccessfulTestCase as an example:

RegularViewRequestIsSuccessfulTestCase contents from tests/cases/dos.py class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, regular user)" % (self.view, self.verb, response.status_code) ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 class RegularViewRequestIsSuccessfulTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest ( self ) : """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , user_string = "user_1" ) self . _assert_response_successful ( response , "%s did not return a successful response for %s verb (%s status, regular user)" % ( self . view , self . verb , response . status_code ) )

This test case inherits from the BaseViewVerbTestCase class, which takes a view and a verb argument in the constructor. In order to generate instances of this test case for all of the verbs and views in the application, let’s take a look at the __get_dos_class_tests method in the StreetArtTestRunner class:

__get_dos_class_tests method in the StreetArtTestRunner class def __get_dos_class_tests(self): """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for supported_verb in requestor.supported_verbs: class AnonTestCase1(RegularViewRequestIsSuccessfulTestCase): pass class AnonTestCase2(AdminViewRequestIsSuccessfulTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb)) to_return.append(AnonTestCase2(view=view, verb=supported_verb)) return to_return 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 def __get_dos_class_tests ( self ) : """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) for supported_verb in requestor . supported_verbs : class AnonTestCase1 ( RegularViewRequestIsSuccessfulTestCase ) : pass class AnonTestCase2 ( AdminViewRequestIsSuccessfulTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = supported_verb ) ) to_return . append ( AnonTestCase2 ( view = view , verb = supported_verb ) ) return to_return

As shown above, we:

Iterate over all of the views found in the application (line 9) Get the view and the Requestor associated with the view (line 10) Iterate over all of the supported verbs listed in the Requestor (line 11) For each of the supported verbs, we create an anonymous subclass of RegularViewRequestIsSuccessfulTestCase (line 13) We add an instance of the anonymous subclass instantiated with the verb and view that we are iterating over (line 19)

This may look a bit odd – why are we creating an anonymous subclass? Well the way that the Python unit testing framework works we cannot have two instances of the same test case class in a test suite (it will only run one of them). As such, we create unique classes for each of the test cases that we need to run. It’s a bit of a quirk, but nothing we can’t handle!

And with that, we have the ability to dynamically generate unit tests for all of the functionality in our application. Let’s now take a look at how we can make good use of this capability.

Testing For Adherence To The Requestor Architecture

Since we are relying on our developers to add a little bit of extra functionality for all of the views that they author, a logical first step to test is that all of the code within our codebase does, in fact, follow our architecture. This is tested by the ViewHasRequestorTestCase :

ViewHasRequestorTestCase in tests/cases/requestor.py class ViewHasRequestorTestCase(BaseViewTestCase): """ This is a test case for testing whether or not a view has a corresponding requestor mapped to it. """ def runTest(self): """ Tests that the given view has a requestor mapped to it. :return: None """ registry = TestRequestorRegistry.instance() self.assertTrue( registry.does_view_have_mapping(self.view), "No requestor found for view %s." % self.view, ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 class ViewHasRequestorTestCase ( BaseViewTestCase ) : """ This is a test case for testing whether or not a view has a corresponding requestor mapped to it. """ def runTest ( self ) : """ Tests that the given view has a requestor mapped to it. :return: None """ registry = TestRequestorRegistry . instance ( ) self . assertTrue ( registry . does_view_have_mapping ( self . view ) , "No requestor found for view %s." % self . view , )

This test is rather simple – it takes the view that the test case was instantiated with and checks the Registry to make sure that a requestor for the view exists. To see the results of this test, check out the v0.1 tag:

git checkout tags/v0.1

Once checked out, modify the settings.py file so that only the Requestor check tests are enabled:

Modifying settings.py to enable only the Requestor checking test cases TEST_FOR_REQUESTOR_CLASSES = True TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False 1 2 3 4 5 6 7 TEST_FOR_REQUESTOR_CLASSES = True TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False

We can now run the following to verify that all of the views have a Requestor mapped to them:

python manage.py test

The result of running this command is shown below:

That’s great and all that all of our unit tests are passing, but let’s make sure that they’re testing what we intend them to. To do so, we remove the requested_by decorator from the MyPostsListView view as shown below:

Commenting out the requested_by decorator for MyPostsListView view #@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView ( BaseListView ) : """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset ( self ) : """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self . request . user . posts . all ( )

After commenting out requested_by as shown in line 1 above, we run the tests again:

Sure enough, as expected one of our unit tests fails indicating that the MyPostsListView does not have a Requestor mapped to it. Great! Let’s go ahead and uncomment the requested_by decorator and continue.

Testing For Denial Of Service

Now that we know all of our views have the necessary Requestor class mapped to them, let’s test to make sure that all of the functionality within our application is working properly (ie: returns an HTTP status code indicating a successful request). Testing this is handled by the RegularViewRequestIsSuccessfulTestCase and AdminViewRequestIsSuccessfulTestCase test cases:

Testing for denial of service (tests/cases/dos.py) class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, regular user)" % (self.view, self.verb, response.status_code) ) class AdminViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for an admin user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="admin_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, admin user)" % (self.view, self.verb, response.status_code) ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 class RegularViewRequestIsSuccessfulTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest ( self ) : """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , user_string = "user_1" ) self . _assert_response_successful ( response , "%s did not return a successful response for %s verb (%s status, regular user)" % ( self . view , self . verb , response . status_code ) ) class AdminViewRequestIsSuccessfulTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for an admin user. """ def runTest ( self ) : """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , user_string = "admin_1" ) self . _assert_response_successful ( response , "%s did not return a successful response for %s verb (%s status, admin user)" % ( self . view , self . verb , response . status_code ) )

These test cases simply send a request to the related view using the configured verb and check to see that the HTTP status code of the response indicates that the request was successful. To run these tests, configure the application to only run the denial of service test cases in settings.py :

Enabling only the denial of service test cases in settings.py TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = True TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False 1 2 3 4 5 6 7 TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = True TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False

We now run the unit tests:

python manage.py test

The result of running these tests is shown below:

As we did before, let’s modify our code to introduce a failing test case just to make sure that our unit tests are working as intended. To do this, raise a PermissionDenied exception in the EditPostView view’s get method:

Modified code in EditPostView to raise a PermissionDenied error (views/pages/post.py) @requested_by("streetart.tests.requestors.pages.EditPostViewRequestor") class EditPostView(BaseUpdateView): """ This is the page for editing the contents of a street art post object. """ template_name = "pages/edit_streetart_post.html" form_class = EditStreetArtPostForm model = StreetArtPost def get(self, request, *args, **kwargs): """ Handle a GET request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ raise PermissionDenied # if request.user != self.get_object().user and not request.user.is_superuser: # raise PermissionDenied # return super(EditPostView, self).get(request, *args, **kwargs) def post(self, request, *args, **kwargs): """ Handle a POST request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.post. """ if request.user != self.get_object().user and not request.user.is_superuser: raise PermissionDenied return super(EditPostView, self).post(request, *args, **kwargs) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 @ requested_by ( "streetart.tests.requestors.pages.EditPostViewRequestor" ) class EditPostView ( BaseUpdateView ) : """ This is the page for editing the contents of a street art post object. """ template_name = "pages/edit_streetart_post.html" form_class = EditStreetArtPostForm model = StreetArtPost def get ( self , request , * args , * * kwargs ) : """ Handle a GET request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ raise PermissionDenied # if request.user != self.get_object().user and not request.user.is_superuser: # raise PermissionDenied # return super(EditPostView, self).get(request, *args, **kwargs) def post ( self , request , * args , * * kwargs ) : """ Handle a POST request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.post. """ if request . user != self . get_object ( ) . user and not request . user . is_superuser : raise PermissionDenied return super ( EditPostView , self ) . post ( request , * args , * * kwargs )

We then run the tests again:

As expected, we see two failed unit tests indicating that the EditPostView view is returning an HTTP status code indicating a request error. Great! As before, let’s modify the EditPostView back to what it was beforehand and continue.

Testing For Unknown HTTP Verbs

We’ve now got some guarantees that our code is following the Requestor framework and that all of the tested functionality is indicating success, but what if our Requestor classes aren’t testing all of the HTTP verbs supported by our views? It is incredibly common for frameworks (especially ones like Django and Rails) to have all sorts of crazy functionality under the hood, and I’ve abused functionality that developers didn’t know about to nefarious ends on more than one occasion. To ensure that we are testing all of the functionality within our application, let’s make sure that the supported verbs reported by our views match the verbs that our Requestor classes are configured to invoke. Testing this is handled by the RegularUnknownMethodsTestCase :

RegularUnknownMethodsTestCase in tests/cases/hidden.py class RegularUnknownMethodsTestCase(BaseViewTestCase): """ This is a test case for testing whether or not a view returns the expected HTTP verbs through an OPTIONS request from a regular user. """ def runTest(self): """ Tests that the HTTP verbs returned by an OPTIONS request match the expected values. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_options(user_string="user_1") allowed_verbs = response._headers.get("allow", None) if not allowed_verbs: raise ValueError("No allow header returned by view %s." % self.view) allowed_verbs = [x.strip().lower() for x in allowed_verbs[1].split(",")] supported_verbs = [x.lower() for x in requestor.supported_verbs] self.assertTrue( all([x.lower() in supported_verbs for x in allowed_verbs]), "Unexpected verbs found for view %s. Expected %s, got %s." % (self.view, [x.upper() for x in supported_verbs], [x.upper() for x in allowed_verbs]) ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class RegularUnknownMethodsTestCase ( BaseViewTestCase ) : """ This is a test case for testing whether or not a view returns the expected HTTP verbs through an OPTIONS request from a regular user. """ def runTest ( self ) : """ Tests that the HTTP verbs returned by an OPTIONS request match the expected values. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_options ( user_string = "user_1" ) allowed_verbs = response . _headers . get ( "allow" , None ) if not allowed_verbs : raise ValueError ( "No allow header returned by view %s." % self . view ) allowed_verbs = [ x . strip ( ) . lower ( ) for x in allowed_verbs [ 1 ] . split ( "," ) ] supported_verbs = [ x . lower ( ) for x in requestor . supported_verbs ] self . assertTrue ( all ( [ x . lower ( ) in supported_verbs for x in allowed_verbs ] ) , "Unexpected verbs found for view %s. Expected %s, got %s." % ( self . view , [ x . upper ( ) for x in supported_verbs ] , [ x . upper ( ) for x in allowed_verbs ] ) )

As shown above, we issue an HTTP OPTIONS request to the view, parse the contents of the Allow HTTP response header (which contains a comma-separated list of the verbs supported by the endpoint), and then check those verbs against the verbs listed in the related Requestor . To run these tests, configure settings.py to only enable the unknown methods test cases:

Enabling only the unknown HTTP methods test cases TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = True TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False 1 2 3 4 5 6 7 TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = True TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False

We then run the tests:

python manage.py test

The result of running these tests is shown below:

As shown above, we have multiple Requestor classes that are not testing all of the HTTP verbs associated with their views! Funnily enough – this output was exactly what I saw when I was first authoring the test codebase for this talk. I had no idea that views that subclassed DeleteView supported both the GET and DELETE HTTP verbs!

Taking a look at the DeletePostViewRequestor class, we see that the class only tests for the POST verb:

DeletePostViewRequestor contents at tags/v0.1 class DeletePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the DeletePostView view. """ supported_verbs = ["POST"] requires_auth = True def get_url_path(self, user="user_1"): post = SaFaker.get_post_for_user(user) return "/delete-post/%s/" % (post.uuid,) 1 2 3 4 5 6 7 8 9 10 11 class DeletePostViewRequestor ( BaseRequestor ) : "" " This is a requestor class for sending requests to the DeletePostView view. " "" supported_verbs = [ "POST" ] requires_auth = True def get_url_path ( self , user = "user_1" ) : post = SaFaker . get_post_for_user ( user ) return "/delete-post/%s/" % ( post . uuid , )

Let’s go ahead and check out the next tag:

git checkout tags/v0.2

And taking a look at the DeletePostViewRequestor class again, we see that it is now updated to support the GET and DELETE verbs:

DeletePostViewRequestor contents at tags/v0.2 class DeletePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the DeletePostView view. """ supported_verbs = ["GET", "POST", "DELETE"] requires_auth = True def get_url_path(self, user="user_1"): post = SaFaker.get_post_for_user(user) return "/delete-post/%s/" % (post.uuid,) 1 2 3 4 5 6 7 8 9 10 11 class DeletePostViewRequestor ( BaseRequestor ) : "" " This is a requestor class for sending requests to the DeletePostView view. " "" supported_verbs = [ "GET" , "POST" , "DELETE" ] requires_auth = True def get_url_path ( self , user = "user_1" ) : post = SaFaker . get_post_for_user ( user ) return "/delete-post/%s/" % ( post . uuid , )

Let’s run the tests again and check to make sure we are now testing all of the HTTP verbs supported by our application’s views:

Boom! We now know that all of our Requestor classes are properly configured to test all of the HTTP verbs associated with all of our views!

Testing For Authentication Enforcement

Darn near every application contains functionality that is only available to users that are authenticated to the application. Wouldn’t it be great to ensure that all of our post-auth functionality is properly enforcing authentication checks? Well this case is tested for by the AuthenticationEnforcementTestCase . To see this test case, first checkout the v0.3 tag:

git checkout tags/v0.3

The contents of the AuthenticationEnforcementTestCase are shown below:

AuthenticationEnforcementTestCase contents in tests/cases/auth.py class AuthenticationEnforcementTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not authentication is properly enforced on a view. """ def runTest(self): """ Tests that the given view returns the expected HTTP response value when an unauthenticated request is submitted to it. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, do_auth=False) self._assert_response_redirect( response, "Response from unauthenticated %s request to view %s was %s. Expected %s." % (self.verb, self.view, response.status_code, [301, 302]) ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 class AuthenticationEnforcementTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not authentication is properly enforced on a view. """ def runTest ( self ) : """ Tests that the given view returns the expected HTTP response value when an unauthenticated request is submitted to it. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , do_auth = False ) self . _assert_response_redirect ( response , "Response from unauthenticated %s request to view %s was %s. Expected %s." % ( self . verb , self . view , response . status_code , [ 301 , 302 ] ) )

When generating instances of this unit test, we generate them only for endpoints that require authentication as shown in the __get_authentication_enforcement_tests method in the StreetArtTestRunner class:

Contents of the __get_authentication_enforcement_tests in StreetArtTestRunner def __get_authentication_enforcement_tests(self): """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) if not requestor.requires_auth: continue for supported_verb in requestor.supported_verbs: class AnonTestCase(AuthenticationEnforcementTestCase): pass to_return.append(AnonTestCase(view=view, verb=supported_verb)) return to_return 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 def __get_authentication_enforcement_tests ( self ) : """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) if not requestor . requires_auth : continue for supported_verb in requestor . supported_verbs : class AnonTestCase ( AuthenticationEnforcementTestCase ) : pass to_return . append ( AnonTestCase ( view = view , verb = supported_verb ) ) return to_return

To run these tests we configure settings.py to enable only the authentication check test cases:

Configuring settings.py to enable only the authentication check test cases TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = True TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False 1 2 3 4 5 6 7 TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = True TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False

We then run the unit tests:

python manage.py test

The results of running these tests are shown below:

We see that an error is being thrown indicating that an attribute is being accessed on a user object when that user is not authenticated. The offending code can be found in the MyPostsListView view on line 16:

Contents of MyPostsListView at tags/v0.3 @requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 @ requested_by ( "streetart.tests.requestors.pages.MyPostsListViewRequestor" ) class MyPostsListView ( BaseListView ) : """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset ( self ) : """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self . request . user . posts . all ( )

Let’s check out the next tag to see what has changed in the MyPostsListView :

git checkout tags/v0.4

The new contents of MyPostsListView are shown below:

Updated contents of MyPostsListView at tags/v0.4 @requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get(self, request, *args, **kwargs): """ Handle the processing of an HTTP GET request to this endpoint to ensure that the requesting user has sufficient permissions. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ if not request.user.is_authenticated: raise PermissionDenied return super(MyPostsListView, self).get(request, *args, **kwargs) def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 @ requested_by ( "streetart.tests.requestors.pages.MyPostsListViewRequestor" ) class MyPostsListView ( BaseListView ) : """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get ( self , request , * args , * * kwargs ) : """ Handle the processing of an HTTP GET request to this endpoint to ensure that the requesting user has sufficient permissions. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ if not request . user . is_authenticated : raise PermissionDenied return super ( MyPostsListView , self ) . get ( request , * args , * * kwargs ) def get_queryset ( self ) : """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self . request . user . posts . all ( )

Running the tests again, we see that all of our authentication checks are now being properly enforced:

Great! We now know that all of our post-auth endpoints are properly enforcing authentication checks.

Testing For HTTP Response Headers

For the uninitiated, there are a number of HTTP response headers that can greatly improve the security posture of the browsers used by your application’s clients. Even better, many of these headers require little-to-no configuration in your application to just work! To test our application for the presence of these headers, lets first check out the next tag:

git checkout tags/v0.5

Once checked out, we can find the test functionality for checking response headers in the HeaderKeyExistsTestCase and the HeaderValueAccurateTestCase test cases found in tests/cases/headers.py

Test cases for testing for the presence of HTTP response headers in tests/cases/headers.py class HeaderKeyExistsTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a header key is contained within all of the HTTP responses returned by the Street Art project. """ def __init__(self, header_key=None, *args, **kwargs): self.header_key = header_key super(HeaderKeyExistsTestCase, self).__init__(*args, **kwargs) def runTest(self): """ Tests that the HTTP response received from the view contains a header corresponding to self.header_key. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_has_header_key( response=response, header_key=self.header_key, message="Response from view %s with verb %s did not contain header key of %s. Keys were %s." % (self.view, self.verb, self.header_key, response._headers.keys()) ) class HeaderValueAccurateTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a response header has the expected value. """ def __init__(self, header_key=None, header_value=None, *args, **kwargs): self.header_key = header_key self.header_value = header_value super(HeaderValueAccurateTestCase, self).__init__(*args, **kwargs) def runTest(self): """ Tests that the HTTP response received from the view contains a header key and value corresponding to self.header_key and self.header_value. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_has_header_value( response=response, header_key=self.header_key, header_value=self.header_value, message="Response from view %s with verb %s did not contain expected header value of %s: %s. " "Headers were %s." % (self.view, self.verb, self.header_key, self.header_value, response._headers) ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 class HeaderKeyExistsTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not a header key is contained within all of the HTTP responses returned by the Street Art project. """ def __init__ ( self , header_key = None , * args , * * kwargs ) : self . header_key = header_key super ( HeaderKeyExistsTestCase , self ) . __init__ ( * args , * * kwargs ) def runTest ( self ) : """ Tests that the HTTP response received from the view contains a header corresponding to self.header_key. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , user_string = "user_1" ) self . _assert_response_has_header_key ( response = response , header_key = self . header_key , message = "Response from view %s with verb %s did not contain header key of %s. Keys were %s." % ( self . view , self . verb , self . header_key , response . _headers . keys ( ) ) ) class HeaderValueAccurateTestCase ( BaseViewVerbTestCase ) : """ This is a test case for testing whether or not a response header has the expected value. """ def __init__ ( self , header_key = None , header_value = None , * args , * * kwargs ) : self . header_key = header_key self . header_value = header_value super ( HeaderValueAccurateTestCase , self ) . __init__ ( * args , * * kwargs ) def runTest ( self ) : """ Tests that the HTTP response received from the view contains a header key and value corresponding to self.header_key and self.header_value. :return: None """ requestor = self . _get_requestor_for_view ( self . view ) response = requestor . send_request_by_verb ( self . verb , user_string = "user_1" ) self . _assert_response_has_header_value ( response = response , header_key = self . header_key , header_value = self . header_value , message = "Response from view %s with verb %s did not contain expected header value of %s: %s. " "Headers were %s." % ( self . view , self . verb , self . header_key , self . header_value , response . _headers ) )

These two test cases test (1) that a header key exists in every one of the HTTP responses and (2) that the value associated with each header key has the expected content. To generate these unit tests, we first have a list of the headers that we want to see in the application configured in settings.py :

HTTP response headers that we test for EXPECTED_RESPONSE_HEADERS = { "included": { "X-Frame-Options": "deny", "X-XSS-Protection": "1; mode=block", "X-Content-Type-Options": "nosniff", "X-Permitted-Cross-Domain-Policies": "none", }, "excluded": [ "X-Supah-Secret", "X-Supah-Dupah-Secret", ] } 1 2 3 4 5 6 7 8 9 10 11 12 EXPECTED_RESPONSE_HEADERS = { "included" : { "X-Frame-Options" : "deny" , "X-XSS-Protection" : "1; mode=block" , "X-Content-Type-Options" : "nosniff" , "X-Permitted-Cross-Domain-Policies" : "none" , } , "excluded" : [ "X-Supah-Secret" , "X-Supah-Dupah-Secret" , ] }

When generating the unit tests, we iterate over the contents of the expected HTTP response headers and the views/verbs in our application as shown in the __get_response_header_tests method in the StreetArtTestRunner class:

Generating unit tests for testing HTTP response headers in StreetArtTestRunner def __get_response_header_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems(): for supported_verb in requestor.supported_verbs: class AnonTestCase1(HeaderKeyExistsTestCase): pass class AnonTestCase2(HeaderValueAccurateTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k)) to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v)) return to_return 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 def __get_response_header_tests ( self ) : "" " Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. " "" to_return = [ ] for _ , _ , callback in self . url_patterns : view , requestor = self . __get_view_and_requestor_from_callback ( callback ) for k , v in settings . EXPECTED_RESPONSE_HEADERS [ "included" ] . iteritems ( ) : for supported_verb in requestor . supported_verbs : class AnonTestCase1 ( HeaderKeyExistsTestCase ) : pass class AnonTestCase2 ( HeaderValueAccurateTestCase ) : pass to_return . append ( AnonTestCase1 ( view = view , verb = supported_verb , header_key = k ) ) to_return . append ( AnonTestCase2 ( view = view , verb = supported_verb , header_key = k , header_value = v ) ) return to_return

In order to properly run these tests, configure the middleware in settings.py to exclude the middleware used for populating the headers:

Middleware configuration for testing the lack of HTTP response headers MIDDLEWARE = [ # 'streetart.middleware.HeaderCleaningMiddleware', 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', # 'streetart.middleware.SecurityHeadersMiddleware', # 'streetart.middleware.BadHeadersMiddleware', ] 1 2 3 4 5 6 7 8 9 10 11 12 MIDDLEWARE = [ # 'streetart.middleware.HeaderCleaningMiddleware', 'django.middleware.security.SecurityMiddleware' , 'django.contrib.sessions.middleware.SessionMiddleware' , 'django.middleware.common.CommonMiddleware' , 'django.middleware.csrf.CsrfViewMiddleware' , 'django.contrib.auth.middleware.AuthenticationMiddleware' , 'django.contrib.messages.middleware.MessageMiddleware' , 'django.middleware.clickjacking.XFrameOptionsMiddleware' , # 'streetart.middleware.SecurityHeadersMiddleware', # 'streetart.middleware.BadHeadersMiddleware', ]

Wit