Profile and cProfile
There are two modules inside the Python standard library that can be used for performance testing.
1. One is Profile, which is a pure Python implementation, so it will be slower, and is more appropriate if you need to expand the module.
2. The second is cProfile, which, as you can tell from its name, is a C implementation and is officially recommended for most situations.
The interface and data output format of these two are exactly the same, and you can freely switch between the two, so we will only use cProfile as an example for the following introduction.
Performance Testing with cProfile
In cProfile, it is very easy to perform a performance test, just call the run method and pass it the function you want to test and its parameters, we will perform a performance test on fib(n) below.
import cProfile
def fib(n):
if n == 0:
return 0
if n == 1:
return 1
return fib(n-1) + fib(n-2)
if __name__ == '__main__':
('fib(30)')
The results of the performance test are shown below
You can see that a total of 2692539 function calls were made, taking a total of 0.815 seconds. Each line below corresponds to a function call where:
1. ncalls, the total number of function calls;
2. tottime, the total time spent on this function call;
3. percall, the average time spent per call;
4. cumtime, total cumulative time spent;
5. percall, the average cumulative time per call;
6. filename:lineno(function), information about the corresponding function.
So it is obvious from the graph that almost all the time consumed is on fib and the number of function calls is too high, this is mainly because the function is called recursively and creates a lot of redundant branches, so the program needs to be optimized. There are two ways to improve it, one is to cache the information of fib(n) without calculating it every time, and the other is to change the program to iterative.
And caching function values has a simple decorator in Python 3 called lru_cache, which automatically caches function values for you without having to manually store them yourself.
import functools @functools.lru_cache(maxsize=None) def fib(n): if n == 0: return 0 if n == 1: return 1 return fib(n-1) + fib(n-2)
The results of the run are as follows:
As you can see, the fib function is called only 31 times, and almost all of the extra calls hit the cache, which is far less than the number of previous calls, and the runtime is improved quite significantly. Also using the iterative version of the program below runs very fast, so I won't expand on that here.
def fib(n): prev, cur = 0, 1 if n == 0: return prev if n == 1: return cur count = 1 while count < n: count += 1 prev, cur = cur, prev + cur return cur
In addition to the run method mentioned earlier, there is another method called runctx that allows some contextual parameters to be supplied. For example, the ('fib(30)') can be changed to('fib', globals(), {'n':30})The final run results are the same.
Finally, in addition to printing directly to the command line, run and runctx can write the output to a file by passing a filename as the second argument.
Display control with pstats
Although cProfile can be used for simple performance testing of a program, when the program is too large and calls many functions, you need some tools to filter and sort the test results, and pstats is one such tool.
# fib_profile.py
import cProfile
import pstats
for i in range(5):
('fib(1000)', 'fib_profile_{}'.format(i))
stats = ('fib_profile_0')
for i in range(1, 5):
('fib_profile_{}'.format(i))
stats.strip_dirs()
stats.sort_stats('cumulative')
stats.print_stats('fib')
The above program first writes several test results, then initializes stats, you can add new files through the add method of stats, pstats will automatically aggregate the results; then strip_dirs will remove the paths in front of the filenames, only the filenames will be retained; sort_stats is to sort the results, that is, to select the lines in the aforementioned sort_stats is to sort the output, that is, to select the lines mentioned above (see the official documentation for details); finally, print_stats outputs the results, where you can filter the lines, for example, the above program only outputs the lines containing fib; the actual output results are as follows.
This is the whole content of this article.