How to use __slots__ to achieve memory saving and performance optimization in Python

You've thought about it, a small one__slots__Can you directly reduce your Python class memory consumption by half? That’s right, what we are going to talk about today is this eye-catching technique that can save memory and improve performance during development. It is simply the "dream lover" in the hearts of programmers!

Background: A class full of memory

Have you ever encountered this situation? When developing a certain function, I clearly used one or two subcategories, but as the project gradually became more complex, the memory consumption gradually became outrageously high, which was almost broken into your computer's stomach - just like the crawler project I worked on last time, almost ate the entire server. It turns out that when we define a class, Python will assign each instance a dictionary to store attributes by default. This means that each instance will occupy more space in memory than you think. Well, you heard it right. This "dictionary" was originally designed for flexibility, but in some memory-sensitive scenarios, this "feature" will become a memory foodie.

For example: Do you think Python class memory does not take up much?

Suppose you have a simple class:

class Person:
    def __init__(self, name, age):
         = name
         = age

This class looks quite simple, but in fact, each instance will take up more memory, because Python will default toPersonCreate a dictionary for each instance of the class to storenameandage. That way, whenever you create aPersonWhen an instance is in progress, there will be an additional memory overhead, even if you only use two properties.

p1 = Person("Little Li", 25)
p2 = Person("Xiao Zhou", 28)

At this point, you may think: "Oh, my computer's memory is large enough, so there is no problem." But when you write dozens or hundreds of instances of the class, the memory overhead explodes directly, and the program runs slowly, the performance of the server may also be affected.

__slots__: Your memory management assistant

Speaking of this, you may think: "Is there any way to save some memory?" OK, I recommend a magic tool for you-__slots__！

__slots__It is a magic provided by Python, allowing you to limit the storage of its attributes when defining a class and reduce memory overhead. By using__slots__, you can tell the Python class not to use dictionaries to store instance properties, but to allocate fixed space for each property directly in memory.

• ​Memory structure comparison:

  • ​Noneslots: Instance memory = Object header +dictPointer + other metadata.
  • ​Yesslots: Instance memory = object header + attribute value array + other metadata.

• Give a life example:

  • Suppose you are a class teacher and every student (instance) in the class has a schoolbag (__dict__). You can put things (attributes) in your schoolbag, such as textbooks, water cups, snacks...Although flexible, each schoolbag has weight (memory occupancy), especially when you have 1,000 students, 1,000 schoolbags fill the classroom!

  • And used__slots__, equivalent to your regulations:Each student can only bring one fixed small bag, only a few specified things can be placed in it (such as only textbooks and pens).The bag is lighter,Without taking up space, 1,000 students can save a lot of space!

How to use __slots__

Modify your class like this

class Person:
    __slots__ = ['name', 'age']
    
    def __init__(self, name, age):
         = name
         = age

With this small change above, you will tell Python: "Hey! Only allow itnameandageThese two attributes! ” As a result, Python will allocate fixed space for these two properties in memory.No more dictionary, so that the memory usage can be greatly reduced.

Let me give you a rough example: see how it works?

1. Let me give you a simple example to see the effect

from pympler import asizeof

class WithoutSlots:
    def __init__(self):
         = 1
         = 2
         = 3

class WithSlots:
    __slots__ = ['a', 'b', 'c']
    def __init__(self):
         = 1
         = 2
         = 3

obj_big = WithoutSlots()
obj2_small = WithSlots()

print((obj_big))  # Output about 416 bytesprint((obj2_small))  # Output about 152 bytes

hint: pympleris a Python library for analyzing and measuring the memory usage of Python objects. passpip install pymplerCome to install

2. Simulate to create a million-level instance to observe the total amount of memory

obj_big = [WithoutSlots() for _ in range(1_000_000)]  
obj2_small = [WithSlots() for _ in range(1_000_000)]  

print((obj_big))  # Output approximately 160448992 bytesprint((obj2_small))  # Output approximately 64448824 bytes

Some things to note when using __slots__

OK, although the effect looks great, __slots__It is not omnipotent, we have to use it with caution. Here are a few points to note:

1.Don't add properties dynamically

use__slots__After that, you can no longer dynamically add other attributes to the instance. In other words, if you want to add another attribute later, you have to__slots__Definition is well in.

p = Person("Little Li", 25)
 = "Little Li"  # Can = "Beijing"  # Error: AttributeError: 'Person' object has no attribute 'address'

2. Classes that cannot be inherited from __slots__

If you want to have__slots__class inheritance, subclasses must also define their own__slots__, otherwise an error will be raised.

class Student(Person):
    __slots__ = ['school']

s = Student("Xiao Zhang", 22)
 = "Tsinghua"

3. Especially useful for subcategories

__slots__The most significant effect is when there are more objects and large memory usage. If you have few class instances or have little memory pressure, then use__slots__It doesn't make much sense.

Summary: Just be willing

This time about__slots__The sharing is actually to remind everyone that if the project you are doing is a high memory requirement or needs to create a large number of instances frequently, don't forget to add a class to the__slots__, it can save you a lot of memory overhead.

Of course, there are many things to be interesting about Python's memory management, but__slots__Definitely a good tool that cannot be ignored. Learning this little trick will not only make your code more efficient, but also make your program more "slimming". Maybe one day it will help you "slim" so that the system can run faster, even if it is a crawler, it will be lighter!

You may think, "Sister Hua, can this really greatly improve performance?" The answer is: **Yes! **But it depends on the scene. If you use the right place, it can help you save a lot of memory and improve performance; if you use the wrong place, it will add extra complexity. So, don't use it blindly! Remember, optimization should be targeted and not optimize for the sake of optimization.

This is the article about how Python uses __slots__ to achieve memory saving and performance optimization. For more related Python __slots__ content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!