Delimiter Collision

Overloading in Python

A Gentle Introduction

The word “overload” has a negative connotation in regular English. We say we are “overloaded” when we are overworked and close to burnout. If we overload an airplane, it may result in a less-than-optimal outcome for its passengers.

When we are programming and we “overload” a function, we mean something quite different – though reasonable minds can still differ about whether it is a good thing. In short, function overloading is defining one or more additional implementations of an existing function, with different arguments.¹

Why would anybody do this? Some reasons include²:

1. Greater uniformity in naming functions that do similar things;

2. Providing a mental aid to programmers using a library, so they don’t have to remember lots of permutations of function names;

3. Avoiding brittle, conditional logic to handle arguments of different types;

A common use-case for overloading is I/O. Say, for example, that we’d like a function or method to handle writing serialized data to disk. In Python, perhaps what we will pass to this function is an ElementTree that should become XML, or a dict that should be written as JSON. Maybe it will be a list of lists that should be written as CSV. You get the idea.

We could write functions, each with their own name (write_xml, write_json, etc.) that would do each of these things specifically; or we could write multiple implementations, all just called write_data. This latter approach requires overloading the function … errr, sort of.

While commonly used in other languages such as Java, Python doesn’t support function overloading in the usual way. But it does provide some features that are akin to overloading, which can prove useful depending on your situation.

A Bit of History

In a dynamically typed language like Python, it’s of course possible to pass different arguments of different types to the same function. There is no compiler that will bark at us when we, say, pass an int to an argument called name that the function expects to be a string.

The problem is the implementation. What should we do if we want to handle cases where name is an int, or perhaps some other type? This may require lots of calls to isinstance to perform type inspection, paired with conditional logic for the scenarios that the programmer can foresee.

This was one of the main reasons cited for implementing proper function overloading in Python cited by Phillip Eby, when he proposed the feature in PEP 3124 back in 2007³:

[I]t is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type.

Currently, the “obvious way” to do this is by type inspection, but this is brittle and closed to extension. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party.

What Eby proposed was to support multiple implementations of functions of the same name, via a new overloading module and @overload decorator. The PEP was ultimately deferred, for a variety of reasons.⁴

Nevertheless, some of the ideas live on in other modules of the Python standard library. Let’s take a look at them now.

Overloading (Sort Of) With typing

Let’s Get Decorating

Python is a dynamically typed language, but it offers something close to type safety via the standard library’s typing module. This allows programmers to indicate the expected types of variables and arguments in their code, via “type hints,” and then use tools such as mypy to ensure that those expectations are not violated.

Type hints were adopted in Python in 2015, via PEP 484.⁵ That change also included a new @overload decorator, but its behavior is quite different from what Eby proposed. What it allows is for a programmer to define neatly the different combinations of arguments and types that a function or method accepts, and to have type-checkers flag violations. But it still requires that there be a single implementation of the actual function.

A Real-Life Example

Let’s look at a non-trivial example. Suppose we want to implement the write_data function mentioned above, and accept different combinations of arguments depending on the data we are handling. We might implement a module like the following:

import csv
import json
import xml.etree.ElementTree as ET

from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import overload, Optional


@overload
def write_data(fp: Path, data: Mapping):
    # JSON
    ...


@overload
def write_data(
    fp: Path, data: Sequence[Sequence], *, header: Optional[Sequence] = None
):
    # CSV
    ...


@overload
def write_data(fp: Path, data: ET.Element, *, attrs: Optional[dict] = None):
    # XML
    ...


def write_data(*args, **kwargs) -> None:
    if not (fp := kwargs.get("fp")):
        fp = args[0]

    if not (data := kwargs.get("data")):
        data = args[1]

    # XML
    if isinstance(data, ET.Element):
        if attrs := kwargs.get("attrs"):
            for k, v in attrs.items():
                data.set(k, v)
        tree = ET.ElementTree(data)
        tree.write(fp)

    # JSON
    elif isinstance(data, Mapping):
        # Assumes data is JSON-serializable
        fp.write_text(json.dumps(data, indent=2))

    # CSV
    elif isinstance(data, Sequence):
        with open(fp, mode="w", newline="") as csvfile:
            writer = csv.writer(csvfile)
            if header := kwargs.get("header"):
                writer.writerow(header)
            writer.writerows(data)

    # Uh-oh
    else:
        raise NotImplementedError

    return None


def main() -> None:
    xml_data = ET.fromstring("<x>hello</x>")
    write_data(Path("output.xml"), xml_data)

    json_data = {"message": "hello"}
    write_data(Path("output.json"), json_data)

    csv_data = {"whoops": "I'm not CSV"}
    write_data(Path("output.csv"), csv_data, header=["col1", "col2"])


if __name__ == "__main__":
    main()

Picking It Apart

There’s a lot going on here, so let’s unpack it. First, we are defining several overloaded functions, all called write_data, with different signatures. These are just stubs (hence the ... in the function body). Note that two of the signatures force the third argument to be passed as a keyword, using the * notation as described in PEP 3102.⁶ This will make the parsing of varying keyword arguments somewhat simpler.

The actual implementation of write_data is defined below the @overload definitions; there must be just one. Here, we have a flexible (*args, **kwargs) signature, to allow for the different mixes of arguments when the function is actually called. In other words, we make sure that our implementation is consistent with the behavior described by our overloads.

The body of the actual implementation has to do the somewhat messy work of inspecting the types and parsing the arguments. This is exactly the kind of thing that Eby was trying to work around in his PEP; alas, the typing module’s implementation only gets us so far.

What this does allow, though, is for some sanity-checking using a tool like mypy. If I run this script, there is no error, even though I’ve written non-CSV data to a file that looks like CSV.

$ cat output.csv
{
  "whoops": "I'm not CSV"
}

But if I use mypy, it will warn me that I’ve passed something I shouldn’t have to my write_data function.

error: No overload variant of "write_data" matches argument types "Path", "dict[str, str]", "list[str]"  [call-overload]

Let’s fix our code:

diff --git a/overload_typing.py b/overload_typing.py
index 7f1913d..acc6149 100644
--- a/overload_typing.py
+++ b/overload_typing.py
@@ -69,7 +69,7 @@ def main() -> None:
     json_data = {"message": "hello"}
     write_data(fp=Path("output.json"), data=json_data)

-    csv_data = {"whoops": "I'm not CSV"}
+    csv_data = [["message", "hello"], ["foo", "bar"]]
     write_data(Path("output.csv"), csv_data, header=["col1", "col2"])

Now, mypy is happy.

$ mypy overload_typing.py
Success: no issues found in 1 source file

Why We Maybe Shouldn’t Do This

OK, this works. But let’s perhaps recall some key lines in “The Zen of Python”, the poem by Tim Peters⁷:

Explicit is better than implicit.
Simple is better than complex.

What we’ve done here is made it implicit how the data should be written, depending on the types of arguments passed. We also have some function signatures that are fairly complex to read.

Perhaps it would really be simpler just to have a module – let’s call it, oh, data_handlers – that defines all the functions we may need (write_json, etc.) and let the programmer choose which one is appropriate explicitly. This also would remove the need to parse the arguments to the function in a way that is fairly brittle and not exactly easy to reason about.

Overloading With functools

The other way that Python supports overloading is more “real,” in the sense that it supports actually writing separate implementations that are called at runtime depending on the types of the arguments passed.

This approach is supported by the standard library’s functools module, specifically its @singledispatch decorator.⁸ I found a post written by Martin Heinz on this topic; Heinz does a good job of explainining this, so I’d direct readers to that for a thorough explanation and real-world example of how it can be used.

Why Now?

None of this functionality is new to Python, so why am I writing about it now? Partly, because it’s just fairly new to me, and I imagine it’s a corner of the language that even many professional programmers don’t explore much. I’ve also observed that type hinting – though it’s also been around for a while – is becoming more and more commonly adopted throughout the Python ecosystem, which is a topic for another time.

It’s really hard not to end this post with some lazy pun using the word “overload”. So I’ll instead just suggest that readers meditate on how words in different contexts can mean different things. Isn’t that neat?