Profiling Python Code Utilizing timeit and cProfile
Picture by Creator
As a software program developer, you’ll have probably heard the quote “Premature optimization is the root of all evil”—greater than as soon as—in your profession. Whereas optimization will not be tremendous useful (or completely obligatory) for small initiatives, profiling is usually useful.
After you’ve completed coding a module, it’s an excellent apply to profile your code to measure how lengthy every of the sections takes to execute. This can assist determine code smells and information optimizations to enhance code high quality. So all the time profile your code earlier than optimizing!
To take the primary steps, this information will allow you to get began with profiling in Python—utilizing the built-in timeit and cProfile modules. You’ll study to make use of each the command-line interface and the equal callables inside Python scripts.
The timeit module is a part of the Python normal library and affords a couple of comfort capabilities that can be utilized to time brief snippets of code.
Let’s take a easy instance of reversing a Python checklist. We’ll measure the execution occasions of acquiring a reversed copy of the checklist utilizing:
- the
reversed()perform, and - checklist slicing.
>>> nums=[6,9,2,3,7]
>>> checklist(reversed(nums))
[7, 3, 2, 9, 6]
>>> nums[::-1]
[7, 3, 2, 9, 6]
Working timeit on the Command Line
You may run timeit on the command line utilizing the syntax:
$ python -m timeit -s 'setup-code' -n 'quantity' -r 'repeat' 'stmt'
You’re required to supply the assertion stmt whose execution time is to be measured.
You may specify the setup code when wanted—utilizing the brief possibility -s or the lengthy possibility –setup. The setup code will probably be run solely as soon as.
The quantity of occasions to run the assertion: brief possibility -n or the lengthy possibility –number is non-obligatory. And the variety of occasions to repeat this cycle: brief possibility -r or the lengthy possibility –repeat is non-obligatory, too.
Let’s see the above in motion for our instance:
Right here creating the checklist is the setup code and reversing the checklist is the assertion to be timed:
$ python -m timeit -s 'nums=[6,9,2,3,7]' 'checklist(reversed(nums))'
500000 loops, better of 5: 695 nsec per loop
Once you don’t specify values for repeat, the default worth of 5 is used. And while you don’t specify quantity, the code is run as many occasions as wanted in order to succeed in a complete time of at least 0.2 seconds.
This instance explicitly units the variety of occasions to execute the assertion:
$ python -m timeit -s 'nums=[6,9,2,3,7]' -n 100Bu000 'checklist(reversed(nums))'
100000 loops, better of 5: 540 nsec per loop
The default worth of repeat is 5, however we are able to set it to any appropriate worth:
$ python3 -m timeit -s 'nums=[6,9,2,3,7]' -r 3 'checklist(reversed(nums))'
500000 loops, finest of three: 663 nsec per loop
Let’s additionally time the checklist slicing strategy:
$ python3 -m timeit -s 'nums=[6,9,2,3,7]' 'nums[::-1]'
1000000 loops, better of 5: 142 nsec per loop
The checklist slicing strategy appears to be quicker (all examples are in Python 3.10 on Ubuntu 22.04).
Working timeit in a Python Script
Right here’s the equal of operating timeit contained in the Python script:
import timeit
setup = 'nums=[9,2,3,7,6]'
quantity = 100000
stmt1 = 'checklist(reversed(nums))'
stmt2 = 'nums[::-1]'
t1 = timeit.timeit(setup=setup,stmt=stmt1,quantity=quantity)
t2 = timeit.timeit(setup=setup,stmt=stmt2,quantity=quantity)
print(f"Utilizing reversed() fn.: {t1}")
print(f"Utilizing checklist slicing: {t2}")
The timeit() callable returns the execution time of stmt for quantity of occasions. Discover that we are able to explicitly point out the variety of occasions to run, or make quantity take the default worth of 1000000.
Output >>
Utilizing reversed() fn.: 0.08982690000000002
Utilizing checklist slicing: 0.015550800000000004
This runs the assertion—with out repeating the timer perform—for the desired quantity of occasions and returns the execution time. Additionally it is fairly frequent to make use of time.repeat() and take the minimal time as proven:
import timeit
setup = 'nums=[9,2,3,7,6]'
quantity = 100000
stmt1 = 'checklist(reversed(nums))'
stmt2 = 'nums[::-1]'
t1 = min(timeit.repeat(setup=setup,stmt=stmt1,quantity=quantity))
t2 = min(timeit.repeat(setup=setup,stmt=stmt2,quantity=quantity))
print(f"Utilizing reversed() fn.: {t1}")
print(f"Utilizing checklist slicing: {t2}")
This may repeat the method of operating the code quantity of occasions repeat variety of occasions and returns the minimal execution time. Right here we now have 5 repetitions of 100000 occasions every.
Output >>
Utilizing reversed() fn.: 0.055375300000000016
Utilizing checklist slicing: 0.015101400000000043
We’ve seen how timeit can be utilized to measure the execution occasions of brief code snippets. Nevertheless, in apply, it is extra useful to profile a whole Python script.
This may give us the execution occasions of all of the capabilities and methodology calls—together with built-in capabilities and strategies. So we are able to get a greater thought of the costlier perform calls and determine alternatives for optimization. For instance: there may be an API name that is too sluggish. Or a perform might have a loop that may be changed by a extra Pythonic comprehension expression.
Let’s discover ways to profile Python scripts utilizing the cProfile module (additionally a part of the Python normal library).
Think about the next Python script:
# important.py
import time
def func(num):
for i in vary(num):
print(i)
def another_func(num):
time.sleep(num)
print(f"Slept for {num} seconds")
def useful_func(nums, goal):
if goal in nums:
return nums.index(goal)
if __name__ == "__main__":
func(1000)
another_func(20)
useful_func([2, 8, 12, 4], 12)
Right here we now have three capabilities:
func()that loops by way of a variety of numbers and prints them out.one other func()that comprises a name to thesleep()perform.useful_func()that returns the index of a goal quantity in checklist (if the goal is current within the checklist).
The above-listed capabilities will probably be known as every time you run the primary.py script.
Working cProfile on the Command Line
Run cProfile on the command line utilizing:
Right here we’ve named the file important.py:
Working this could provide the following output:
Output >>
0
...
999
Slept for 20 seconds
And the next profile:
Right here, ncalls refers back to the variety of calls to the perform and percall refers back to the time per perform name. If the worth of ncalls is bigger than one, then percall is the common time throughout all calls.
The execution time of script is dominated by another_func that makes use of the built-in sleep perform name (sleeps for 20 seconds). We see that print perform calls are fairly costly too.
Utilizing cProfile within the Python Script
Whereas operating cProfile on the command line works tremendous, it’s also possible to add the profiling performance to the Python script. You should utilize cProfile coupled with the pstats module for profiling and accessing statistics.
As a finest apply to deal with useful resource setup and teardown higher, use the with assertion and create a profile object that’s used as a context supervisor:
# important.py
import pstats
import time
import cProfile
def func(num):
for i in vary(num):
print(i)
def another_func(num):
time.sleep(num)
print(f"Slept for {num} seconds")
def useful_func(nums, goal):
if goal in nums:
return nums.index(goal)
if __name__ == "__main__":
with cProfile.Profile() as profile:
func(1000)
another_func(20)
useful_func([2, 8, 12, 4], 12)
profile_result = pstats.Stats(profile)
profile_result.print_stats()
Let’s take a better take a look at the output profile generated:
Once you’re profiling a big script, it’ll be useful to kind the outcomes by execution time. To take action you may name sort_stats on the profile object and kind based mostly on the execution time:
...
if __name__ == "__main__":
with cProfile.Profile() as profile:
func(1000)
another_func(20)
useful_func([2, 8, 12, 4], 12)
profile_result = pstats.Stats(profile)
profile_result.sort_stats(pstats.SortKey.TIME)
profile_result.print_stats()
Once you now run the script, it’s best to have the ability to see the outcomes sorted by time:
I hope this information helps you get began with profiling in Python. At all times bear in mind, optimizations ought to by no means come at the price of readability. For those who’re all for studying about different profilers, together with third-party Python packages, try this article on Python profilers.
Bala Priya C is a developer and technical author from India. She likes working on the intersection of math, programming, information science, and content material creation. Her areas of curiosity and experience embody DevOps, information science, and pure language processing. She enjoys studying, writing, coding, and occasional! Presently, she’s engaged on studying and sharing her information with the developer neighborhood by authoring tutorials, how-to guides, opinion items, and extra.
