# flamegraphs
Some tools for exploring and building intuition around the shape of [[Profiling]] performance data.
## Reference
- [Brendan Greg's Page on Flamegraphs](https://www.brendangregg.com/flamegraphs.html) - the now ubiquitous profile visualization; once upon a time, we only had tables.
- [Flame Graph Overview (Datadog)](https://www.datadoghq.com/knowledge-center/distributed-tracing/flame-graph/)
## Java Mission Control, [[Java Flight Recorder]] and [[Java]]
- [Java Mission Control](https://www.oracle.com/java/technologies/javase/products-jmc9-downloads.html) let's you explore and visualize events in [[Java Flight Recorder|JFR]] recordings. Viewing JFRs as flamegraphs and event streams is useful for developing intuition around what data is in a profile.
### Install
- <https://www.oracle.com/java/technologies/javase/products-jmc9-downloads.html>
### Usage
- Download a [[Java Flight Recorder|JFR]] [from Datadog](https://app.datadoghq.com/profiling/profile/AZsEsZRsAADrDfPijCgonAAA?query=service%3Aprof-analyzer&agg_m=%40prof_jvm_cpu_cores&agg_m_source=base&agg_t=sum&event=AwAAAZsEsZHs3wtIbQAAABhBWnNFc1pSc0FBRHJEZlBpakNnb25BQUEAAAAkZDE5YjA0YjEtYjAxZi00ZDk0LWI4ODQtMzQ0NjBiNWVjNzI0AAAAhA&extra_search_fields=%7B%22filters_query%22%3A%22%22%2C%22sample_type%22%3A%22cpu-time%22%7D&my_code=enabled&refresh_mode=paused&viz=stream&from_ts=1765306705993&to_ts=1765310305993&live=false), extract the archive and open `main.jfr`
- Click on the *Event Browser* tree nodes to see a flamegraph of all events for that type. In this case, `Datadog/Profiling/Method CPU Profiling Samples` is selected.
- Click on individual samples in the right pane to see information on the individual records that comprise the flamegraph.
![[Screen Shot 20251209T200709Z@2x.png]]
## [[pprof]]
[[pprof]] is a tool and [[protobuf]] definition for representing sets of profile samples.
### Installation
- [[golang]] has a version of the `pprof` cli tool accessible via `go tool pprof`, so install [[golang]]
- With go installed, also install <https://github.com/felixge/pprofutils>
- `go install github.com/felixge/pprofutils/v2/cmd/pprofutils@latest`
### Usage
- Download a [[pprof]] [from Datadog](https://app.datadoghq.com/profiling/profile/AZsEsZfQAADrDfPijCgpPgAA?query=service%3Anicky&agg_m=%40prof_core_cpu_cores&agg_m_source=base&agg_q=service&agg_q_source=base&agg_t=sum&event=AwAAAZsEsZaV361IbQAAABhBWnNFc1pmUUFBRHJEZlBpakNncFBnQUEAAAAkZjE5YjA0YjItNWI4ZS00ZGU5LWI4MzEtZTA2OGZiYTIyMWUxAAFT_Q&my_code=enabled&refresh_mode=paused&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1765306705993&to_ts=1765310305993&live=false) and extract it. Depending on the service, there will be multiple profile files. The examples focus on `cpu.pprof`
```bash
# Open in pprof viewer (browser)
go tool pprof -http localhost:8080 ~/Downloads/cpu.pprof
```
Juxtapose *Top*, *Graph*, and *Flame Graph* views.
![[Screen Shot 20251209T202420Z@2x.png]]
#### Alternate representations with `pprofutils`
```bash
# Create a JSON representation of all the samples
# This reifies location, function, stacks, and labels
# inline so this can be quite large.
# But this representation is useful for building an
# intuition around the data associated with samples.
pprofutils json ~/Downloads/cpu.pprof > /tmp/cpu.json
pprofutils raw ~/Downloads/cpu.pprof | head -n 10
#PeriodType: cpu nanoseconds
#Period: 10000000
#Time: 2025-12-09 14:57:23.213354907 -0500 EST
#Duration: 1m1.
#Samples:
#samples/count cpu/nanoseconds
# 1 10000000: 1 2 3 4 5 6 7 8 9 10 11
# local root span id:[6381255545276650881] span id:[4744457205950002483] trace endpoint:[process_unauthenticated]
# 1 10000000: 12 13 14 15 16 17 9 10 11
# local root span id:[5337040637023191697] span id:[1987669294245304205] trace endpoint:[process_unauthenticated]
pprofutils folded ~/Downloads/cpu.pprof | head -n 10
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject;runtime.findObject 1305
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject;runtime.greyobject;runtime.markBits.setMarked 761
#runtime._ExternalCode; 653
#runtime._ExternalCode; 627
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject;runtime.greyobject 598
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject 395
#runtime._ExternalCode; 201
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject;runtime.(*mspan).typePointersOfUnchecked;runtime.(*mspan).heapBitsSmallForAddr 154
#github.com/DataDog/dd-go/intake.(*QueueReaderV2).Run.func2;github.com/DataDog/dd-go/intake.(*QueueReaderV2).startReadWorker;github.com/DataDog/dd-go/intake.(*QueueReaderV2).batchReadAll;github.com/DataDog/dd-go/intake.(*QueueReaderV2).fillBatch;github.com/DataDog/dd-go/intake.(*Queue).PopSingle;github.com/gomodule/redigo/redis.(*activeConn).Do;github.com/gomodule/redigo/redis.(*conn).Do;github.com/gomodule/redigo/redis.(*conn).DoWithTimeout;bufio.(*Writer).Flush;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;internal/poll.ignoringEINTRIO;syscall.Write;syscall.write;syscall.Syscall;syscall.RawSyscall6;internal/runtime/syscall.Syscall6 131
#runtime.gcBgMarkWorker;runtime.systemstack;runtime.gcBgMarkWorker.func2;runtime.gcDrainMarkWorkerDedicated;runtime.gcDrain;runtime.scanobject;runtime.typePointers.next 114
```