The Ultimate Guide to Python Decorators, Part I: Function Registration

Posted by
on under

One of the signatures of the Flask framework is its clever use of decorators for common application tasks such as defining routes and error handlers. Decorators give a very concise and readable structure to your code, so much that most Flask extensions and many other Python packages follow the same pattern and expose core parts of their functionality through decorators.

Today I'm starting a series of in-depth posts about different ways in which you can incorporate custom decorators into your Python applications. In this first part I'm going to show you how to create simple decorators that register functions as callbacks for application-specific events.

For your reference, here is a list of all the posts in this series:

Registering Functions With Decorators

The most basic type of decorator is the one used to register a function as a handler for an event. This pattern is very common in Python applications, as it allows two or more subsystems to communicate without knowing much about each other, what is formally known as a "decoupled" design.

A function registration decorator that takes no arguments has the following structure:

def request_logger(f):
    # ...
    # insert custom decorator actions here
    # ...
    return f

In this example, request_logger is the name of the decorator, and is defined as a standard Python function. I said above that this decorator takes no arguments, so you may be confused about seeing that in fact there is an argument that I called f. When working with decorators you have to separate the implementation of the decorator (shown above) versus its usage. To help you get the whole picture, below you can see how this decorator would be used:

@request_logger
def log_a_request(request):
    pass

So as you see, the decorator has no arguments when it is used, but the function that implements the decorator does take an argument. This argument is actually required, because the decorator function is invoked indirectly by Python each time the decorator is used, and Python passes the decorated function (the log_a_request function in the example above) as this argument.

The effect that the decorator has on the decorated function can be better explained with a short snippet of Python code that achieves the same effect without using decorators:

def log_a_request(request):
    pass

log_a_request = request_logger(log_a_request)

Using this snippet as a guide, you can see that the decorator function is invoked with the decorated function as its argument, and the return value is a function that takes the place of the original function that was decorated. This allows advanced decorators to provide an alternative function to replace the decorated function. The simpler decorators that I'm covering in this chapter do not need to do any of that, so they just end with return f to keep the original function untouched.

The common pattern for this type of function registration decorator is to save a reference to the decorated function to invoke it later, when the event represented by the decorator occurs. Below you can see a complete implementation of the request_logger decorator. This is a decorator that registers one or more functions to act as a request loggers. The only thing that the decorator does is to add the decorated function to a list:

all_request_loggers = []

def request_logger(f):
    all_request_loggers.append(f)
    return f

Here is an example of how this decorator can be applied to a function, which can be located in another part of the application:

@request_logger
def log_a_request(request):
    print(request.method, request.path)

This usage of the decorator is going to cause the request_logger() decorator function defined above to execute with the log_a_request function passed as the f argument. The decorator function will then store a reference to this function in the all_request_loggers global list. If there are more functions decorated with this decorator, they are all going to be added to the all_request_loggers list.

The last part of this implementation is to invoke all the functions registered as request loggers when the event (in this case the arrival of a request) occurs. This can be done with a simple for-loop in a separate function:

def invoke_request_loggers(request):
    for logger in all_request_loggers:
        logger(request)

Here is the complete implementation of this example, using a simple Flask application:

from flask import Flask, request

app = Flask(__name__)
all_request_loggers = []

def request_logger(f):
    all_request_loggers.append(f)
    return f

def invoke_request_loggers(request):
    for logger in all_request_loggers:
        logger(request)

@request_logger
def log_a_request(request):
    print(request.method, request.path)

@app.route('/')
def index():
    invoke_request_loggers(request)
    return 'Hello World!'

You may wonder why all this effort if it is much easier to call the log_a_request() function directly from the index() view function. In some cases, particularly in simpler applications, that might be an acceptable solution, but using the decorator allows the application to decouple the view function from the function or functions registered to log requests, which is a good design practice.

In the above example the index() view function does not need to know what is the request logging function or if there is zero, one or many of them. You may also want to register different logging functions on a production deployment versus when you run locally for development. This design keeps your application code clean and independent of the decision of how to log a request. In a real-world application the decorator functions would be in their own Python module, separate from the other modules or packages of the application. Any module that needs to register a request logger would import the request_logger decorator and use it. Likewise, in any part of the application where a request needs to be logged, the invoke_request_loggers() function can be imported and called.

The Observer Pattern

You may have noticed that the ideas that I'm presenting in this article are suspiciously similar to the Observer Pattern. In fact, Python decorators used in the way I showed above are nothing more than a really nice way to implement this pattern!

The observer pattern can be used in a variety of situations. Some examples:

  • In a game, you could register collision handlers to be invoked when two game sprites touch.
  • In a desktop application, you could register background update handlers that are invoked while the application is idle.
  • In a command line application, you could register error handler functions that are invoked to clean up after an unexpected error occurs.
  • In a web application, well, I can just give you some examples from Flask. The route, before_request, after_request and teardown_request decorators along with a few others all use this pattern! These decorators are a bit more complex than the one I showed above, as some take arguments while others alter the behavior of the application based on what the decorated function returns. These are more advanced features that I'll discuss in future articles in this series.

Other Decorator Types

As I hinted above, function registration decorators with no arguments like the one I presented in this article are the simplest type of Python decorators. They are excellent as an introduction to the topic, but they just scratch the surface of the power decorators can bring to your application. In the next part of this series I'm going to discuss decorators that override or replace the decorated function!

Become a Patron!

Hello, and thank you for visiting my blog! If you enjoyed this article, please consider supporting my work on this blog on Patreon!

17 comments
  • #1 Jason said

    I see that in your code above,
    for logger in all_request_loggers:
    logger(request) # this line
    is where the

    @request_logger
    def log_a_request(request):
    print(request.method, request.path)

    is getting called... how is logger(request) associated with log_a_request?

  • #2 Miguel Grinberg said

    @Jason: the reference to the log_a_request() function is stored in the all_request_loggers list. This is what the decorator does.

  • #3 Fisher said

    I do test the code snippet, it works as expected. But I have something doubted, what's called in the index request is -

    invoke_request_loggers(request)

    It just take elements from the list named all_request_loggers and run them one by one.

    I don't think the request_logger or log_a_request has been explicitly or implicitly called, then how the log_a_request was put into the list all_request_loggers? And when? At the application was run or the request occured?

  • #4 Fisher said

    with a simple test, I saw that when the application ran, the decorator registered the f into the list all_request_loggers.
    I don't know why this happens since I thought the decorator should work at least when a decorated function is called - what I saw is weird.

  • #5 Miguel Grinberg said

    @Fisher: The decorator function runs at the time the decorated function is imported, not when it is called. So just by importing a module that has a function decorated with the @request_logger decorator you end up with the decorated function added to the list of loggers.

  • #6 Rushi said

    @Miguel as you replied to @Fisher comment : "just by importing a module that has a function decorated with the @request_logger decorator you end up with the decorated function added to the list of loggers" but it seems that func "log_a_request" is not imported or call anywhere.

  • #7 Miguel Grinberg said

    @Rushi: log_a_request is the decorated function. The decorator takes care of registering the function so that it is indirectly invoked when the invoke_request_loggers function is called. This is the same as in Flask with the @app.route decorator. The view functions are never invoked directly, they are just registered with the decorator. and then called indirectly by Flask when appropriate.

  • #8 Fisher said

    Thanks Miguel, I use simple decorators often, some have celery inside to log requests without influence(response time) to the decorated flask api. But this knowledge is fresh to know, appreciate that.

  • #9 Brandon said

    Thanks Miguel! I had no idea that decorator functions run when they are imported. That was confusing at first because I was expecting log_a_request to be called explicitly, but it makes sense now.

  • #10 Grayden said

    Curious about creating a wrapper function within a class.

    E.g. I have a class which maps a python object to a database table.

    I'd like to wrap my insert and update methods with @CheckRequiredFields that will loop through that object's properties and raise an error if a Non-Null field is set to None. I'd prefer the wrapper to be a method of the class so that it knows the required fields of the object, instead of establishing an outside function and passing the data as parameters.

    So, E.g.
    class Purchase():
    def CheckRequiredFields(self, func):
    def wrapped(args, kwargs):
    # implements logic to validate fields
    ...
    return func(
    args, **kwargs)
    return wrapped

    @self.CheckRequiredFields
    def insert(self):
    # logic to insert to db after passing validation

    I've struggled to make this work. Couldn't find much support for this on Stack Overflow, so I'm curious if you've ever used a pattern like this.

  • #11 Miguel Grinberg said

    @Grayden: you can't use self in a decorator for a method. At the time the decorator executes there is no object, so self does not have a value. You should be able to make it work by making the decorator a static class. Can't guarantee that this code works, but here is how this should more or less be:

    class Purchase():
    @staticmethod
    def CheckRequiredFields(func):
    def wrapped(self, args, kwargs):
    # implements logic to validate fields
    ...
    return self.func(
    args, **kwargs)
    return wrapped

    @CheckRequiredFields
    def insert(self):
    # logic to insert to db after passing validation

  • #12 yousef said

    i this tutorial, with decorator you just create a function to store refrences to other functions to call them in special part of code call them easyly.

    is this right?

  • #13 Miguel Grinberg said

    @yousef: Yes, basically that is what this does. It is a common pattern used by many libraries, Flask among them.

  • #14 Joe said

    Thanks for the article. It seems like a really useful pattern, but I can't figure out how to use it when the request_loggers are in a different module from the rest of the code.

    The point seems to be that we don't want to have to edit old code every time we add a new request_logger. With the pattern, the only thing we must edit is the new request_logger itself, with the help of the decorator. But if I add a new request_logger in its own "new" module, don't I then also have to edit the "old" module containing invoke_request_loggers to import the "new" module? Otherwise it won't get added to the all_request_loggers list. This seems to defeat the object. How can we add a new request_logger in a new module without having to go back and edit other code? Or otherwise how do we use the pattern in a realistic module setting?

  • #15 Miguel Grinberg said

    @Joe: When you add a new request logger you just need to import the decorator to register it. For example, let's say you have all this request logger functionality in a loggers.py module:

    from loggers import request_logger
    
    @request_logger
    def my_new_request_logger(request):
        # log your request here
    

    This is all you need to do. Now the my_new_request_logger() function is registered as a logger, and will be used when the invoke_request_loggers() function is called. There is no need to modify any code.

  • #16 Joe said

    Thanks for the response! So then the decorated my_new_request_logger() and the call to invoke_request_loggers() have to be in the same module -- or my_new_request_logger() has to be imported in the module calling invoke_request_loggers(). The point is that the definition of the decorator and invoke_request_loggers() don't have to be modified. I think I understand now. I was trying to work out how to register subclasses of a class and then write a factory that tried each subclass constructor in a loop, in such a way that I didn't have to edit the factory module (to add imports) every time I added a subclass. In that case the factory is sort of like invoke_request_loggers() above. I have been doing something like this, but all the subclasses and the factory have to be in one module, which is not great. I guess that's a different (and maybe wrongheaded) problem.

  • #17 Kyle Lawlor-Bagcal said

    Thanks for the article Miguel! This gave me some inspiration to write a flask-like class to handle discord bot interactions. It's still a work in progress but this pattern works quite nicely for this use-case. It's interesting how many places the flask-style registration pattern can be useful when you think about it. It certainly makes sense for most situations where you find yourself replying to something. I have an email equivalent of this discord bot and I can already see myself using a similar api there. Thanks as always for your writing and contributions.

Leave a Comment