Friedrich Ewald My Personal Website

Memory profiling with Memray

I am currently working on a small Python project with a relatively large memory footprint. I wanted to analyze the usage in detail and realized that I wasn’t aware of any good Python memory profiler. After some searching I found the builtin package tracemalloc (Trace memory allocation). Especially interesting to me is the capability to take snapshots at different points in time and then compare the delta between those snaphots. The following snippet illustrates this. 

import numpy
import tracemalloc

tracemalloc.start()
snapshot_before = tracemalloc.take_snapshot()
array = numpy.zeros((100, 100))
array2 = numpy.random.rand(1_000_000, 1_000)
print(array2.dtype)
array3 = array2.astype(numpy.float16)
snapshot_after = tracemalloc.take_snapshot()
stats = snapshot_after.compare_to(snapshot_before, 'lineno')

for s in stats[:10]:
    print(s)
print(tracemalloc.get_traced_memory())
This is fine for smaller code bases where I have full control over the source. It requires however a lot of editing and manual filtering of frames and taking snapshots at the right time. Another interesting solution is Bloombergs memray. This allows to run a whole Python application and save the snapshots into a file. From there memray can either create a table or a flamegraph to see the memory consumption over time. Memray can be used like this: 
pip3 install memray

# Run application and save snapshots
memray run my_app.py

# Display snapshots
memray flamegraph memray-test.py.94623.bin
This generates a HTML file which can be opened in any browser.

About the author

is an experienced Software Engineer with a Master's degree in Computer Science. He started this website in late 2015, mostly as a digital business card. He is interested in Go, Python, Ruby, SQL- and NoSQL-databases, machine learning and AI and is experienced in building scalable, distributed systems and micro-services at multiple larger and smaller companies.

Related Posts