Master the weakref module in Python

The weakref module is a practical and very powerful module. Perhaps no module is treated as badly as this module because of its balance of practicality and name recognition. In this article, I'll show you how useful the weakref module is.

problem

Suppose you have a large number of objects with identity information (or unique hashable information). A dictionary for object management is prepared so that a large number of objects can be searched by using the ID as a key.

class Member:
    def __init__(self, id, name):
        self.id = id
        self.name = name

#Manage objects with a dictionary using ID as a key
members_dict = {
    x.id: x for x in [
        Member(1, "tim"),
        Member(2, "matthew"),
        Member(3, "Jack"),
        # ...
    ]
}

members_dict[1].name
>>> 'tim'

Here, there was a request to search not only by ʻid but also by name, so let's create a dictionary using the name as a key. (Here, for the sake of simplicity, name` is also unique information without duplication.)

#Create a dictionary using the name as a key
names_dict = {x.name: x for x in members_dict.values()}

names_dict["tim"].id
>>> 1

Since the member tim has withdrawn here, the data was deleted.

# ID:Delete tim data which is 1
del members_dict[1]

It is desirable that the object data of tim is collected by GC (garbage collection) and the memory is released. But the reality is not.

The object is referenced by the names_dict added in the sense of a search table, and the object of tim is not collected by the GC and continues to exist in the memory.

# names_GC leaks without releasing because there are references left in the dict!
names_dict['tim']
>>> <__main__.Member at 0x1744e889ba8>

If you repeat the creation and deletion of members, the memory will not be released and will accumulate, so you have to keep managing names_dict as well as members_dict at the same time. As the types of search tables increase, the number of dictionaries managed at the same time increases, which is a quagmire.

A simple data structure like this one would work if you designed it object-oriented. When it comes to complex data structures where objects refer to each other, a reference counter is required to determine whether an object is alive or dead, and the philosophical code of implementing a GC-like implementation on the language in which the GC is located Is completed.

The weakref module that appears dashingly there!

The weakref module can help you in that situation, just replace names_dict with weakref.WeakValuDictionary and it will solve the problem.

from weakref import WeakValueDictionary

names_dict = WeakValueDictionary(
    {_.name: _ for _ in members_dict.values()}
)

names_dict['tim'].id
>>> 1

# ID:Delete tim data which is 1
del members_dict[1]

names_dict['tim']
>>> KeyError: 'tim'

WeakValuDictionary is a dictionary-like object whose value is a weak reference. Weak references are references that do not interfere with GC recovery and are automatically deleted from the dictionary when there are no more strong references other than WeakValuDictionary.

There are many situations where you want to access a large number of objects using a fast cache / lookup table to improve performance. At such times, the Weak Valu Dictionary can be said to be an essential item.

Of course there is also a Weak Key Dictionary

If you have a WeakValuDictionary, of course there is also aWeakKeyDictionary whose key is a weak reference.

The WeakKeyDictionary can also be used to hold references and referenced.

from weakref import WeakKeyDictionary


class Member:
    def __init__(self, name, friend=None):
        self.name = name
        self._friend_dict = WeakKeyDictionary()
        self._referenced_dict = WeakKeyDictionary()
        if friend:
            self._friend_dict[friend] = None
            friend._referenced_dict[self] = None

    def referenced(self):
        referenced = list(self._referenced_dict.keys())
        return referenced[0] if referenced else None

    def friend(self):
        friends = list(self._friend_dict.keys())
        return friends[0] if friends else None

    def __str__(self):
        friend = self.friend()
        refer = self.referenced()
        return "{}: referenced={} friend={}".format(
            self.name,
            refer.name if refer else None,
            friend.name if friend else None
        )


tim = Member("tim")
matthew = Member("matthew", friend=tim)
jack = Member("jack", friend=matthew)

print(tim)
print(matthew)
print(jack)
# output:
#
# tim: referenced='matthew' friend='None'
# matthew: referenced='jack' friend='tim'
# jack: referenced='None' friend='matthew'

del matthew

print(tim)
print(jack)
# output:
#
# tim: referenced='None' friend='None'
# jack: referenced='None' friend='None'

I used it as a dictionary that automatically updates the references and references of other objects. You can see that the dictionary is updated automatically by deleting the object.

reference

-8.11. Weakref — Weak reference

Afterword

The weakref module is simple enough to require only two dictionary objects, the WeakValuDictionary and the WeakKeyDictionary, but it is a very powerful module. Depending on the device, there are likely to be many other uses.

It's a very low profile module, so I think it will be very helpful to those who don't know it. If you find any other effective usage, please write it in the article.