SoFunction
Updated on 2025-04-06

Introduction to three implementation methods of memory monitoring in Python

Monitoring memory usage is a crucial task in Python development. Whether developing small scripts or large applications, unreasonable memory usage can lead to performance degradation and even cause program crashes. This article will introduce several common Python memory monitoring methods in detail, including the psutil library, the memory_profiler library and the tracemalloc module, and compare and analyze them, and explain their respective application scenarios.

1. Use the psutil library to monitor memory

1.1 Introduction and Installation

psutil (Process and System Utilities) is a cross-platform library that provides rich system information acquisition functions, including CPU, memory, disk, network, etc. With psutil, we can easily monitor the memory usage of the system and process.

Installing psutil is very simple, just run the following command on the command line:

pip install psutil

1.2 Sample code and function implementation

import psutil
 
# Get the system's memory usagedef get_system_memory_usage():
    """
     Get and print the system's memory usage.
     This function uses the psutil library to obtain the system's virtual memory information, including total memory, used memory, free memory, and memory usage.
     All memory values ​​are converted to GB and retained two decimal places.
     """
    memory = psutil.virtual_memory()
    total =  / (1024 ** 3)  # Convert to GB    used =  / (1024 ** 3)    # Convert to GB    free =  / (1024 ** 3)    # Convert to GB    percent = 
    print(f"Total memory: {total:.2f} GB")
    print(f"Memory used: {used:.2f} GB")
    print(f"Free memory: {free:.2f} GB")
    print(f"Memory usage: {percent}%")
 
# Get the memory usage of the current processdef get_process_memory_usage():
    """
     Gets and prints the memory usage of the current process.
     This function uses the psutil library to get the memory information of the current process, specifically the resident set size (RSS), and converts it to MB and retains two decimal places.
     """
    process = ()
    memory_info = process.memory_info()
    rss = memory_info.rss / (1024 ** 2)  # Convert to MB    print(f"The size of the current process's residency set（RSS）: {rss:.2f} MB")
 
if __name__ == "__main__":
    """
     Main program entry.
     When the script is run directly, the get_system_memory_usage and get_process_memory_usage functions are called to obtain and print the memory usage of the system and the current process.
     """
    print("System Memory Usage:")
    get_system_memory_usage()
    print("\nCurrent process memory usage:")
    get_process_memory_usage()

In the above code, the get_system_memory_usage function uses psutil.virtual_memory() to obtain the system's memory information, including total memory, used memory, free memory, and memory usage. The get_process_memory_usage function uses () to obtain the current process and obtains the memory information of the process through the memory_info() method. Here, it mainly focuses on the resident set size (RSS), which is the memory consumption of the current process.

The execution results are as follows:

C:\Users\gyfin\.conda\envs\pa001\ D:\pyproject\pa001\AAP01\ 
System memory usage:
Total memory: 7.92 GB
Memory used: 6.32 GB
Free memory: 1.60 GB
Memory usage: 79.8%
 
Current process memory usage:
The residency set size (RSS) of the current process: 15.12 MB
 
The process has ended, the exit code is 0

1.3 Application scenarios

System-level monitoring: psutil is a great choice when you need to monitor the memory usage of the entire system. For example, in a server environment, the system memory usage is regularly monitored so that the problem of insufficient memory is discovered in a timely manner and corresponding measures are taken, such as optimizing programs or increasing physical memory.

Process-level monitoring: For multi-process Python applications, you can use psutil to monitor the memory usage of each process, find processes with excessive memory usage, and perform targeted optimizations.

2. Use the memory_profiler library to monitor memory

2.1 Introduction and Installation

memory_profiler is a library dedicated to monitoring Python function memory usage. It can analyze the memory usage of functions line by line, helping us to accurately find out the key lines of code for memory allocation.

To install memory_profiler, you can also use the pip command:

pip install memory_profiler

2.2 Sample code and function implementation

from memory_profiler import profile
 
# Use the profile decorator of the memory_profiler library to monitor the memory usage of the function@profile
def test_function():
    """
     Test function to generate a list of 1 million integers and delete the list after use.
     This function uses a list comprehension to generate a list of 1 million integers, and then deletes the list to free up memory.
     """
    # Use list comprehension to generate a list of 1 million integers    data = [i for i in range(1000000)]
    # Delete the list to free the memory    del data
    # function returns    return
 
if __name__ == "__main__":
    """
     Main program entry.
     When the script is run directly, the test_function function is called to test memory usage.
     """
    # Call the test_function function    test_function()

In the above code, we use the @profile decorator to mark the function test_function that needs to be monitored. It should be noted that when running this code, you need to use the python -m memory_profiler your_script.py command to see detailed memory usage analysis.

The program execution results are as follows:

(pa001) D:\pyproject\pa001\AAP01>python -m memory_profiler
Filename:
 
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     3     50.1 MiB     50.1 MiB           1   @profile
     4                                         def test_function():
     5     88.5 MiB     35.0 MiB     1000003       data = [i for i in range(1000000)]
     6     50.8 MiB    -37.7 MiB           1       del data
     7     50.8 MiB      0.0 MiB           1       return
 

2.3 Application scenarios

Function-level memory analysis: When you suspect that a function has a memory leak or unreasonable memory usage, memory_profiler can help you analyze the memory usage of the function line by line, find out the key lines of code allocation and release, and thus perform targeted optimization.

Algorithm optimization: When developing algorithms, you can use memory_profiler to monitor the memory usage implemented by different algorithms and choose algorithms with higher memory efficiency.

3. Use the tracemalloc module to monitor memory

3.1 Introduction and use

tracemalloc is a module in the Python standard library that can be used to track memory allocation. It can count the size and quantity of memory allocations and find the line of code that has the most memory allocations.

3.2 Sample code and function implementation

# Import tracemalloc module to track memory allocationimport tracemalloc
 
# Start tracemalloc tracking()
 
# Simulate some memory allocation operations and create a list of 1 million integersdata = [i for i in range(1000000)]
 
# Get a snapshot of the current memory allocationsnapshot = tracemalloc.take_snapshot()
# Statistics the top 10 locations of memory allocationtop_stats = ('lineno')
 
# Print titleprint("[ Top 10 ]")
# traverse and print statistics about the first 10 memory allocation locationsfor stat in top_stats[:10]:
    print(stat)
 
# Clean the memory and delete the listdel data

In the above code, () is used to start memory tracking, tracemalloc.take_snapshot() is used to obtain the current memory allocation snapshot. Finally, through ('lineno'), the memory allocation can be counted by row and the information of the top 10 lines of code with the largest memory allocation is printed out.

The execution results are as follows:

C:\Users\gyfin\.conda\envs\pa001\ D:\pyproject\pa001\AAP01\ 
[ Top 10 ]
D:\pyproject\pa001\AAP01\:6: size=38.6 MiB, count=999745, average=40 B
 
The process has ended, the exit code is 0

3.3 Application scenarios

Memory Leak Detection: tracemalloc can help us find out possible memory leak problems in our programs. By taking a snapshot of memory allocations regularly and comparing, we can find out which objects' memory is not released correctly, thus positioning the location of memory leaks.

Performance optimization: When optimizing program performance, we can use tracemalloc to find the memory allocation hotspot, that is, the line of code that has the most memory allocations, and then optimize these codes to reduce unnecessary memory allocations.

4. Method comparison

4.1 Function comparison

psutil is mainly used to obtain the overall memory usage of the system and process, focusing on macro-level monitoring. It can conveniently obtain information such as the total system memory, used memory, free memory, and the size of the process's residency set.

memory_profiler focuses on function-level memory analysis, can display the memory usage of functions line by line, helping us accurately find the key lines of memory allocation.

tracemalloc focuses on tracking and statistics of memory allocations. It can count the size and quantity of memory allocations, and find the code lines with the most memory allocations, which is very helpful for detecting memory leaks and optimizing memory usage.

4.2 Use complexity comparison

The use of psutil is relatively simple, and you only need to call a few simple functions to obtain the memory information of the system and process.

memory_profiler requires the use of a decorator to mark functions that need to be monitored, and requires specific commands to be used at runtime, which is slightly more complicated to use.

tracemalloc is part of the Python standard library. When used, only a few functions are called to start and obtain memory allocation snapshots, but the analysis results may require some experience to interpret.

4.3 Performance overhead comparison

The performance overhead of psutil is relatively small because it mainly obtains memory information from the system level and will not have much impact on the normal operation of the program.

The performance overhead of memory_profiler is relatively high because it requires line by line analysis of the memory usage of functions, which will have a certain impact on the running speed of the program.

tracemalloc has a relatively large performance overhead, especially when memory allocation is frequent, because it requires tracking every memory allocation operation.

Summarize

In Python, psutil, memory_profiler, and tracemalloc can all be used for memory monitoring, but their functions and application scenarios are different. If you need to monitor the overall memory usage of the system and process, it is recommended to use psutil; if you need to perform line-by-line memory analysis of functions, memory_profiler is a good choice; if you need to detect memory leaks and optimize memory usage, tracemalloc will be more suitable for you. In practical applications, you can choose the appropriate method according to your specific needs, or use a combination of multiple methods to comprehensively monitor and optimize the memory usage of the program.

This is the article about the three implementation methods of memory monitoring in Python. For more related Python memory monitoring content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!