diff --git a/.openpublishing.redirection.core.json b/.openpublishing.redirection.core.json index 3852a74f4b3a7..cf04edac6ef03 100644 --- a/.openpublishing.redirection.core.json +++ b/.openpublishing.redirection.core.json @@ -1113,6 +1113,14 @@ "source_path_from_root": "/docs/core/diagnostics/built-in-metrics-aspnetcore.md", "redirect_url": "/aspnet/core/log-mon/metrics/built-in" }, + { + "source_path_from_root": "/docs/core/diagnostics/dotnet-trace-collect-linux-scenarios.md", + "redirect_url": "/dotnet/core/diagnostics/dotnet-trace-collect-linux-performance" + }, + { + "source_path_from_root": "/docs/core/diagnostics/performance-diagnostics.md", + "redirect_url": "/dotnet/core/diagnostics/index#performance-tutorials" + }, { "source_path_from_root": "/docs/core/docker/build-docker-netcore-container.md", "redirect_url": "/dotnet/core/docker/build-container" diff --git a/docs/core/diagnostics/debug-deadlock.md b/docs/core/diagnostics/debug-deadlock.md index cafe425a96f55..49b95e0d6fd64 100644 --- a/docs/core/diagnostics/debug-deadlock.md +++ b/docs/core/diagnostics/debug-deadlock.md @@ -1,15 +1,14 @@ --- -title: Debugging deadlock - .NET Core -description: A tutorial that walks you through debugging a locking issue in .NET Core. +title: Debugging deadlock - .NET +description: A tutorial that walks you through debugging a locking issue in .NET. ms.topic: tutorial -ms.date: 07/20/2020 +ms.date: 09/08/2026 +ai-usage: ai-assisted --- -# Debug a deadlock in .NET Core +# Debug a deadlock in .NET -**This article applies to: ✔️** .NET Core 3.1 SDK and later versions - -In this tutorial, you'll learn how to debug a deadlock scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios) source code repository, you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how you can use various tools to analyze the problem, such as core dumps, core dump analysis, and process tracing. +In this tutorial, you'll learn how to debug a deadlock scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios), you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how to collect and analyze a process dump to identify the blocked threads, lock owners, and wait cycle. In this tutorial, you will: @@ -25,10 +24,9 @@ In this tutorial, you will: The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet) - [Sample debug target - web app](/samples/dotnet/samples/diagnostic-scenarios) to trigger the scenario -- [dotnet-trace](dotnet-trace.md) to list processes -- [dotnet-dump](dotnet-dump.md) to collect, and analyze a dump file +- [dotnet-dump](dotnet-dump.md) to list processes and collect and analyze a dump file ## Core dump generation @@ -41,14 +39,18 @@ dotnet run To find the process ID, use the following command: ```dotnetcli -dotnet-trace ps +dotnet-dump ps ``` Take note of the process ID from your command output. Our process ID was `4807`, but yours will be different. Navigate to the following URL, which is an API endpoint on the sample site: `https://localhost:5001/api/diagscenario/deadlock` -The API request to the site will stop responding. Let the request run for about 10-15 seconds. Then create the core dump using the following command: +The API request to the site will stop responding. Let the request run for about 10-15 seconds. + +A dump is the recommended artifact for an existing deadlock because it preserves the current threads, lock owners, and wait cycle. + +Create the core dump using the following command: ### [Linux](#tab/linux) @@ -259,7 +261,6 @@ The second thread is similar. It's also trying to acquire a lock that it already ## See also -- [dotnet-trace](dotnet-trace.md) to list processes - [dotnet-counters](dotnet-counters.md) to check managed memory usage - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file - [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/main/documentation/tutorial) @@ -267,4 +268,4 @@ The second thread is similar. It's also trying to acquire a lock that it already ## Next steps > [!div class="nextstepaction"] -> [What diagnostic tools are available in .NET Core](index.md) +> [What diagnostic tools are available in .NET](index.md) diff --git a/docs/core/diagnostics/debug-highcpu.md b/docs/core/diagnostics/debug-highcpu.md index 06e6c2a5ae658..87412ad4c9d35 100644 --- a/docs/core/diagnostics/debug-highcpu.md +++ b/docs/core/diagnostics/debug-highcpu.md @@ -1,15 +1,14 @@ --- -title: Debug high CPU usage - .NET Core -description: A tutorial that walks you through debugging high CPU usage in .NET Core. +title: Debug high CPU usage - .NET +description: A tutorial that walks you through debugging high CPU usage in .NET. ms.topic: tutorial -ms.date: 03/19/2026 +ms.date: 09/08/2026 +ai-usage: ai-assisted --- -# Debug high CPU usage in .NET Core +# Debug high CPU usage in .NET -**This article applies to: ✔️** .NET Core 3.1 SDK and later versions - -In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios) source code repository, you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how you can use various tools to diagnose this scenario with several key pieces of diagnostics data. +In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios), you can intentionally run CPU-intensive work and use metrics and platform-appropriate profiling tools to identify the expensive code. In this tutorial, you will: @@ -18,16 +17,16 @@ In this tutorial, you will: > - Investigate high CPU usage > - Determine CPU usage with [dotnet-counters](dotnet-counters.md) > - Use [dotnet-trace](dotnet-trace.md) for trace generation -> - Profile performance in PerfView +> - Profile performance in Visual Studio or PerfView > - Diagnose and solve excessive CPU usage ## Prerequisites The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version. +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet). - [Sample debug target](/samples/dotnet/samples/diagnostic-scenarios) to trigger the scenario. -- [dotnet-trace](dotnet-trace.md) to list processes and generate a profile. +- [dotnet-trace](dotnet-trace.md) to collect CPU profiles and runtime traces. - [dotnet-counters](dotnet-counters.md) to monitor cpu usage. ## CPU counters @@ -171,15 +170,17 @@ Throughout the duration of the request, the CPU usage will hover around the incr At this point, you can safely say the CPU is running higher than you expect. Identifying the effects of a problem is key to finding the cause. We will use the effect of high CPU consumption in addition to diagnostic tools to find the cause of the problem. -## Analyze High CPU with Profiler +## Analyze high CPU with a profiler -When analyzing an app with high CPU usage, use a profiler to understand what the code is doing. `dotnet-trace collect` works on all operating systems, but safe-point bias and managed-only callstacks limit it to more general information than a kernel-aware profiler like ETW for Windows or `perf` for Linux. Depending on your operating system and .NET version, improved profiling capabilities might be available—see the platform-specific tabs that follow for detailed guidance. +When analyzing an app with high CPU usage, use a profiler to understand what the code is doing. `dotnet-trace collect` works on all operating systems, but safe-point bias and managed-only call stacks limit it to more general information than kernel-aware profiling through ETW on Windows or `perf_events` on Linux. Depending on your operating system and .NET version, improved profiling capabilities might be available. See the platform-specific tabs that follow for detailed guidance. ### [Linux](#tab/linux) +For .NET 10 and later versions, use `dotnet-trace collect-linux`. If `collect-linux` isn't available, use `perf` for kernel-level CPU sampling. Also use `perf` when you need `perf.data`, perf-native analysis, or hardware performance counters. + #### Use `dotnet-trace collect-linux` (.NET 10+) -On .NET 10 and later, [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux) is the recommended profiling approach on Linux. It combines EventPipe with OS-level perf_events to produce a single unified trace that includes both managed and native callstacks, all without requiring a process restart. This requires root permissions and Linux kernel 6.4+ with `CONFIG_USER_EVENTS=y`. See [collect-linux prerequisites](dotnet-trace.md#prerequisites) for full requirements. +On .NET 10+, [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux) is the recommended Linux workflow. It retains .NET runtime and application event collection while adding kernel CPU samples, native call stacks, and selected Linux events through `perf_events`, all without requiring a process restart. This requires root permissions and Linux kernel 6.4+ with `CONFIG_USER_EVENTS=y`. See [collect-linux prerequisites](dotnet-trace.md#prerequisites) for full requirements. Ensure the [sample debug target](/samples/dotnet/samples/diagnostic-scenarios) is configured to target .NET 10 or later, then run it and exercise the high CPU endpoint (`https://localhost:5001/api/diagscenario/highcpu/60000`) again. While it's running within the 1-minute request, run `dotnet-trace collect-linux` to capture a machine-wide trace: @@ -189,13 +190,17 @@ sudo dotnet-trace collect-linux Let it run for about 20-30 seconds, then press Ctrl+C or Enter to stop the collection. The result is a `.nettrace` file that includes both managed and native callstacks. -Open the `.nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/main/documentation/Downloading.md) and use the **CPU Stacks** view to identify the methods consuming the most CPU time. +Copy the `.nettrace` file to a Windows machine. Use [Visual Studio](#analyze-high-cpu-data-with-visual-studio) for the managed CPU investigation. + +PerfView provides an alternative CPU stack workflow and Linux-native symbol lookup. -For information about resolving native runtime symbols in the trace, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). +PerfView 3.2.1 or later can resolve .NET native and R2R symbols at analysis time. In PerfView, select unresolved module frames and choose **Lookup Symbols**. For other native libraries, configure a local symbol path. For more information, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). + +For a worked diagnosis, see [Find a managed CPU hotspot](dotnet-trace-collect-linux-performance.md#example-find-a-managed-cpu-hotspot). #### Use `perf` -The `perf` tool can also be used to generate .NET Core app profiles. Exit the previous instance of the [sample debug target](/samples/dotnet/samples/diagnostic-scenarios). +Use `perf` when `collect-linux` isn't available and you need kernel-level CPU sampling. Use `perf` directly when the investigation requires the Linux perf ecosystem, such as `perf.data`, `perf report`, `perf annotate`, established flame graph scripts, or hardware performance counters. The following steps demonstrate the standard `perf record` and `perf report` workflow. Exit the previous instance of the [sample debug target](/samples/dotnet/samples/diagnostic-scenarios). Set the `DOTNET_PerfMapEnabled` environment variable to cause the .NET app to create a `map` file in the `/tmp` directory. This `map` file is used by `perf` to map CPU addresses to JIT-generated functions by name. For more information, see [Export perf maps and jit dumps](../runtime-config/debugging-profiling.md#export-perf-maps-and-jit-dumps). @@ -239,19 +244,23 @@ dotnet-trace collect -p 22884 --providers Microsoft-DotNETCore-SampleProfiler Let [dotnet-trace](dotnet-trace.md) run for about 20-30 seconds, and then press the Enter to exit the collection. The result is a `nettrace` file located in the same folder. The `nettrace` files are a great way to use existing analysis tools on Windows. -Open the `nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/main/documentation/Downloading.md) by navigating to samples/core/diagnostics/DiagnosticScenarios/ and clicking on the arrow by the `nettrace` file. Open the 'Thread Time (with StartStop Activities) Stacks' and choose the 'CallTree' tab near the top. After checking the box to the left of one of the threads, your file should look similar to the one pictured below. +Start with [Visual Studio](#analyze-high-cpu-data-with-visual-studio) to inspect CPU use and follow the expensive call path. + +Alternatively, open the `nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/main/documentation/Downloading.md) by navigating to samples/core/diagnostics/DiagnosticScenarios/ and clicking on the arrow by the `nettrace` file. Open the 'Thread Time (with StartStop Activities) Stacks' and choose the 'CallTree' tab near the top. After checking the box to the left of one of the threads, your file should look similar to the one pictured below. [![PerfView image](media/perfview.jpg)](media/perfview.jpg#lightbox) --- -## Analyzing High CPU Data with Visual Studio +## Analyze high CPU data with Visual Studio + +In Visual Studio on Windows, select **File** > **Open** > **File** and open the `.nettrace` file. Select **CPU Usage**, then **Open details**. Use **Current View** to select **Functions** and compare **Self CPU** with **Total CPU**, which includes callees. Use **Caller/Callee** and **Call Tree** to follow the expensive application path. -All \*.nettrace files can be analyzed in Visual Studio. To analyze a Linux \*.nettrace file in Visual Studio, transfer the \*.nettrace file, in addition to the other necessary documents, to a Windows machine, and then open the \*.nettrace file in Visual Studio. For more information, see [Analyze CPU Usage Data](/visualstudio/profiling/beginners-guide-to-performance-profiling?#step-2-analyze-cpu-usage-data). +For more information, see [Analyze CPU Usage Data](/visualstudio/profiling/beginners-guide-to-performance-profiling?#step-2-analyze-cpu-usage-data). ## See also -- [dotnet-trace](dotnet-trace.md) to list processes +- [dotnet-trace](dotnet-trace.md) to collect CPU profiles and runtime traces - [dotnet-counters](dotnet-counters.md) to check managed memory usage - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file - [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/main/documentation/tutorial) @@ -259,4 +268,4 @@ All \*.nettrace files can be analyzed in Visual Studio. To analyze a Linux \*.ne ## Next steps > [!div class="nextstepaction"] -> [Debug a deadlock in .NET Core](debug-deadlock.md) +> [Debug a deadlock in .NET](debug-deadlock.md) diff --git a/docs/core/diagnostics/debug-memory-leak.md b/docs/core/diagnostics/debug-memory-leak.md index 56cc9d217efc6..24c3eb0ae957c 100644 --- a/docs/core/diagnostics/debug-memory-leak.md +++ b/docs/core/diagnostics/debug-memory-leak.md @@ -2,13 +2,12 @@ title: Debug a memory leak tutorial description: Learn how to debug a memory leak in .NET. ms.topic: tutorial -ms.date: 11/13/2023 +ms.date: 09/08/2026 +ai-usage: ai-assisted --- # Debug a memory leak in .NET -**This article applies to:** ✔️ .NET Core 3.1 SDK and later versions - Memory can leak when your app references objects that it no longer needs to perform the desired task. Referencing these objects prevents the garbage collector from reclaiming the memory used. That can result in performance degradation and an exception being thrown. This tutorial demonstrates the tools to analyze a memory leak in a .NET app using the .NET diagnostics CLI tools. If you're on Windows, you may be able to [use Visual Studio's Memory Diagnostic tools](/visualstudio/profiling/memory-usage) to debug the memory leak. @@ -27,7 +26,7 @@ In this tutorial, you will: The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version. +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet). - [dotnet-counters](dotnet-counters.md) to check managed memory usage. - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file (includes the [SOS debugging extension](sos-debugging-extension.md)). - A [sample debug target](/samples/dotnet/samples/diagnostic-scenarios/) app to diagnose. @@ -266,7 +265,7 @@ You can also delete the dump file that was created. ## See also -- [dotnet-trace](dotnet-trace.md) to list processes +- [dotnet-trace](dotnet-trace.md) to collect runtime performance traces - [dotnet-counters](dotnet-counters.md) to check managed memory usage - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file - [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/main/documentation/tutorial) @@ -275,4 +274,4 @@ You can also delete the dump file that was created. ## Next steps > [!div class="nextstepaction"] -> [Debug high CPU in .NET Core](debug-highcpu.md) +> [Debug high CPU in .NET](debug-highcpu.md) diff --git a/docs/core/diagnostics/debug-threadpool-starvation.md b/docs/core/diagnostics/debug-threadpool-starvation.md index b518aed76566c..7515424b8282f 100644 --- a/docs/core/diagnostics/debug-threadpool-starvation.md +++ b/docs/core/diagnostics/debug-threadpool-starvation.md @@ -1,8 +1,9 @@ --- title: Debug ThreadPool Starvation -description: A tutorial that walks you through debugging and fixing a ThreadPool starvation issue on .NET Core +description: A tutorial that walks you through debugging and fixing a ThreadPool starvation issue on .NET. ms.topic: tutorial -ms.date: 04/19/2022 +ms.date: 09/04/2026 +ai-usage: ai-assisted --- # Debug ThreadPool starvation @@ -29,15 +30,15 @@ The tutorial uses: - [dotnet-counters](dotnet-counters.md) to observe performance counters - [dotnet-stack](dotnet-stack.md) to examine thread stacks - [dotnet-trace](dotnet-trace.md) to collect wait events -- Optional: [PerfView](https://github.com/microsoft/perfview/releases) to analyze the wait events +- Optional: [Visual Studio](#analyze-a-nettrace-with-visual-studio) or [PerfView](https://github.com/microsoft/perfview/releases) on Windows to analyze the wait events ## Run the sample app Download the code for the [sample app](/samples/dotnet/samples/diagnostic-scenarios) and run it using the .NET SDK: ```dotnetcli -E:\demo\DiagnosticScenarios>dotnet run -Using launch settings from E:\demo\DiagnosticScenarios\Properties\launchSettings.json... +dotnet run +Using launch settings from /path/to/DiagnosticScenarios/Properties/launchSettings.json... info: Microsoft.Hosting.Lifetime[14] Now listening on: https://localhost:5001 info: Microsoft.Hosting.Lifetime[14] @@ -47,7 +48,7 @@ info: Microsoft.Hosting.Lifetime[0] info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development info: Microsoft.Hosting.Lifetime[0] - Content root path: E:\demo\DiagnosticScenarios + Content root path: /path/to/DiagnosticScenarios ``` If you use a web browser and send requests to `https://localhost:5001/api/diagscenario/taskwait`, you should see the response `success:taskwait` returned after about 500 ms. This shows that the web server is serving traffic as expected. @@ -57,7 +58,7 @@ If you use a web browser and send requests to `https://localhost:5001/api/diagsc The demo web server has several endpoints which mock doing a database request and then returning a response to the user. Each of these endpoints has a delay of approximately 500 ms when serving requests one at a time but the performance is much worse when the web server is subjected to some load. Download the [Bombardier](https://github.com/codesenberg/bombardier/releases) load testing tool and observe the difference in latency when 125 concurrent requests are sent to each endpoint. ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait +bombardier https://localhost:5001/api/diagscenario/taskwait Bombarding https://localhost:5001/api/diagscenario/taskwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -73,7 +74,7 @@ Statistics Avg Stdev Max This second endpoint uses a code pattern that performs even worse: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/tasksleepwait +bombardier https://localhost:5001/api/diagscenario/tasksleepwait Bombarding https://localhost:5001/api/diagscenario/tasksleepwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -147,7 +148,7 @@ If your app is running a version of .NET older than .NET 9, the output UI of dot The preceding counters are an example while the web server wasn't serving any requests. Run Bombardier again with the `api/diagscenario/tasksleepwait` endpoint and sustained load for 2 minutes so there's plenty of time to observe what happens to the performance counters. ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/tasksleepwait -d 120s +bombardier https://localhost:5001/api/diagscenario/tasksleepwait -d 120s ``` ThreadPool starvation occurs when there are no free threads to handle the queued work items and the runtime responds by increasing the number of ThreadPool threads. The `dotnet.thread_pool.thread.count` value increases rapidly to 2-3x the number of processor cores on your machine, and then further threads are added 1-2 per second until stabilizing somewhere above 125. The key signals that ThreadPool starvation is currently a performance bottleneck are the slow and steady increase of ThreadPool threads and CPU Usage much less than 100%. The thread count increase will continue until either the pool hits the maximum number of threads, enough threads have been created to satisfy all the incoming work items, or the CPU has been saturated. Often, but not always, ThreadPool starvation will also show large values for `dotnet.thread_pool.queue.length` and low values for `dotnet.thread_pool.work_item.count`, meaning that there's a large amount of pending work and little work being completed. Here's an example of the counters while the thread count is still rising: @@ -202,7 +203,7 @@ Once the count of ThreadPool threads stabilizes, the pool is no longer starving. Starting in .NET 6, ThreadPool heuristics were modified to scale up the number of ThreadPool threads much faster in response to certain blocking Task APIs. ThreadPool starvation can still occur with these APIs, but the duration is much briefer than it was with older .NET versions because the runtime responds more quickly. Run Bombardier again with the `api/diagscenario/taskwait` endpoint: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` On .NET 6 you should observe the pool increase the thread count more quickly than before and then stabilize at a high number of threads. ThreadPool starvation is occurring while the thread count is climbing. @@ -216,7 +217,7 @@ To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked Run Bombardier again to put the web server under load: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` Then run dotnet-stack to see the thread stack traces: @@ -303,22 +304,39 @@ There's one particular event that helps diagnosing thread pool starvation: the W Run Bombardier again to put the web server under load: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` Then run dotnet-trace to collect wait events: ```dotnetcli -dotnet trace collect -n DiagnosticScenarios --clrevents waithandle --clreventlevel verbose --duration 00:00:30 +dotnet-trace collect -n DiagnosticScenarios --clrevents waithandle --clreventlevel verbose --duration 00:00:30 ``` -That should generate a file named `DiagnosticScenarios.exe_yyyyddMM_hhmmss.nettrace` containing the events. This nettrace can be analyzed using two different tools: +That should generate a file named `DiagnosticScenarios.exe_yyyyddMM_hhmmss.nettrace` containing the events. To analyze it in Visual Studio or PerfView, copy it to a Windows machine if you collected it elsewhere. +On .NET 10+ Linux, prefer the [`collect-linux` blocking configuration](dotnet-trace-collect-linux-performance.md#blocking-contention-and-threadpool-behavior). It records the same focused runtime signals together with native stacks and Linux scheduling context, which helps distinguish blocked workers from runnable workers that aren't receiving CPU. + +The EventPipe trace from `dotnet-trace collect` can be analyzed using three different tools: + +- Start with [Visual Studio](#analyze-a-nettrace-with-visual-studio) to inspect events, payloads, and stacks. - [PerfView](https://github.com/microsoft/perfview/releases): A performance analysis tool developed by Microsoft for Windows only. - [.NET Events Viewer](https://verdie-g.github.io/dotnet-events-viewer): A nettrace analysis [Blazor](https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor) web tool developed by the community. The following sections show how to use each tool to read the nettrace file. +#### Analyze a nettrace with Visual Studio + +Use Visual Studio's [Events Viewer](/visualstudio/profiling/events-viewer) to inspect the wait events and stacks recorded by the preceding command. + +1. Select **File** > **Open** > **File** and open the `.nettrace` file. +1. Open **Events**. In the event filter, select `Microsoft-Windows-DotNETRuntime` and `WaitHandleWait/Start`. +1. To focus on the application, right-click a column header, enable **Process ID** and **Thread ID**, and filter to the relevant IDs. +1. Select an event to inspect **Payload Properties**, including `WaitSource`, and the event stack. +1. Right-click an event and choose **Show Stacks For Event**. In **Event Stacks**, follow the application call path to `DiagScenarioController.TaskWait` and its synchronous task wait. + +These views identify the code that blocks the worker. Event counts alone don't measure time spent waiting. + #### Analyze a nettrace with Perfview 1. Download [PerfView](https://github.com/microsoft/perfview/releases) and run it. @@ -387,10 +405,10 @@ public async Task> TaskAsyncWait() } ``` -Running Bombadier to send load to the `api/diagscenario/taskasyncwait` endpoint shows that the ThreadPool thread count stays much lower and average latency remains near 500ms when using the async/await approach: +Running Bombardier to send load to the `api/diagscenario/taskasyncwait` endpoint shows that the ThreadPool thread count stays much lower and average latency remains near 500ms when using the async/await approach: ```dotnetcli ->bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskasyncwait +bombardier https://localhost:5001/api/diagscenario/taskasyncwait Bombarding https://localhost:5001/api/diagscenario/taskasyncwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -402,3 +420,9 @@ Statistics Avg Stdev Max others - 0 Throughput: 98.81KB/s ``` + +## See also + +- [Performance tutorials](index.md#performance-tutorials) +- [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md) +- [`dotnet-stack`](dotnet-stack.md) diff --git a/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md b/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md new file mode 100644 index 0000000000000..03d62ff6f9bf4 --- /dev/null +++ b/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md @@ -0,0 +1,152 @@ +--- +title: "Tutorial: Investigate Linux performance with dotnet-trace" +description: Collect Linux performance traces and diagnose a managed CPU hotspot and large object heap pressure with Visual Studio and PerfView. +ms.date: 09/08/2026 +ms.topic: tutorial +#Customer intent: As a .NET developer on Linux, I want to collect and analyze the right trace data to find the cause of a performance problem. +ai-usage: ai-assisted +--- + +# Tutorial: Investigate Linux performance with `dotnet-trace collect-linux` + +`dotnet-trace collect-linux` records .NET runtime events together with Linux CPU samples, native call stacks, process activity, scheduling data, and kernel events. By default, collection is machine-wide. This tutorial follows two problems from collection to diagnosis: a managed CPU hotspot and frequent collections caused by large object heap (LOH) allocations. + +## Prerequisites + +The tutorial requires: + +- Linux and .NET 10 or later that meet the [`collect-linux` prerequisites](dotnet-trace.md#prerequisites). +- The [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0). +- The latest [`dotnet-trace`](dotnet-trace.md) global tool. +- A Windows machine with [Visual Studio or PerfView](dotnet-trace.md#view-the-trace-captured-from-dotnet-trace) to analyze the trace. +- The [performance scenarios sample](/samples/dotnet/samples/dotnet-trace-collect-linux-performance-scenarios/). + +From the sample directory, build the application: + +```dotnetcli +dotnet build -c Release +``` + +Use one Linux terminal for the workload and another for collection. Each workload prints its process ID, symptom, and configured duration. Record that PID so you select the application rather than the `dotnet run` host or another process during analysis. + +## Example: Find a managed CPU hotspot + +This workload keeps one CPU core busy. CPU samples identify the method that consumes the CPU and the call path that reaches it. + +### Collect the CPU trace + +Start the workload: + +```dotnetcli +dotnet run -c Release --no-build -- cpu-hotspot 45 +``` + +While the workload runs, collect a 15-second trace in the other terminal: + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile dotnet-common,cpu-sampling \ + --duration 00:00:15 \ + --output cpu-hotspot.nettrace +``` + +`dotnet-common` provides lightweight .NET runtime context, while `cpu-sampling` provides Linux CPU samples and native call stacks. + +Wait for the collector to finish, then copy `cpu-hotspot.nettrace` to a local directory on the Windows analysis machine. + +### Interpret the CPU trace + +Use Visual Studio's [CPU Usage report](/visualstudio/profiling/cpu-usage#analyze-cpu-utilization) to analyze the trace. Select the sample's process ID and the interval in which the workload ran. **Self CPU** identifies samples in a function itself; **Total CPU** includes its callees. The call tree and caller/callee views show the application path that reaches the expensive function. + +Look for `Fibonacci` with high self CPU, and for `Fibonacci` appearing as both caller and callee. The following excerpt illustrates the relevant part of the sample's call tree; prefixes and counts vary with the build and selected interval: + +```text +CpuScenarios.HotspotAsync + CpuScenarios.Fibonacci + CpuScenarios.Fibonacci + CpuScenarios.Fibonacci +``` + +**Interpretation:** The process spends its CPU time in recursive Fibonacci computation. High self CPU locates the expensive method; the repeated frames establish recursion. A runtime startup frame with high *total* CPU is a caller of the expensive work, not evidence that startup caused the sustained CPU use. + +In the sample source, `CpuScenarios.HotspotAsync` repeatedly computes `Fibonacci(36)`, and `CpuScenarios.Fibonacci` calls itself recursively. An iterative algorithm or reuse of the result can avoid repeated recursive work. After an optimization, repeat the same workload and compare absolute CPU use as well as the stack profile. + +You can also analyze `cpu-hotspot.nettrace` with [PerfView](https://github.com/microsoft/perfview). For native symbol information, see [Get symbols for native runtime frames in PerfView](dotnet-trace.md#get-symbols-for-native-runtime-frames-in-perfview). + +In either tool, a high percentage means a large share of the *selected samples*, not that the process consumed every CPU on the machine. Check the process, interval, and absolute CPU usage before drawing that conclusion. + +## Example: Diagnose large object heap pressure + +This workload causes frequent full collections despite a modest retained object count. Collect allocation and GC events to distinguish large-object pressure from frequent small allocations or explicit calls to `GC.Collect`. + +### Collect allocation and GC data + +After the CPU workload exits, start the LOH workload: + +```dotnetcli +dotnet run -c Release --no-build -- loh-gc 45 +``` + +In the collection terminal, run: + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile gc-verbose,cpu-sampling \ + --duration 00:00:15 \ + --output loh-gc.nettrace +``` + +The `gc-verbose` profile adds detailed GC and sampled allocation events. After collection completes, copy `loh-gc.nettrace` to the Windows analysis machine. + +### Interpret allocation and GC data + +In Visual Studio, examine allocation and collection data for the sample process and workload interval. Correlate allocation types with GC generation, reason, and pause duration. An **Insights** result can suggest a starting point, but the diagnosis depends on the underlying data: + +| View | Result to look for | What it tells you | +| --- | --- | --- | +| **Allocations** | Repeated `System.Byte[]` allocations | Byte arrays contribute to the allocation workload. Sampled allocation records aren't an exact object count. | +| **Collections** | Generation 2, `AllocLarge`, and repeated pauses | Large-object allocations trigger full collections during the workload. | + +**Interpretation:** Large-array allocation pressure causes repeated full collections. The collection reason distinguishes this case from an `Induced` collection triggered by an explicit request. GC CPU alone wouldn't identify that distinction. + +In the sample source, `MemoryScenarios.LohGcAsync` allocates 200,000-byte arrays, which exceed the [85,000-byte LOH threshold](../../standard/garbage-collection/large-object-heap.md), and retains a rolling set. Reduce repeated large allocations, for example by reusing buffers when their lifetime permits, then compare allocation volume and GC pauses in another trace. + +For event-level detail, examine the `Microsoft-Windows-DotNETRuntime` events `GC/HeapStats`, `GC/Start`, and `GC/Stop` in [Visual Studio's Events Viewer](/visualstudio/profiling/events-viewer) or PerfView. Inspect payloads and timestamps for the workload. In either tool, check the process identity and any result limit before interpreting counts. + +Allocation pressure doesn't by itself prove a memory leak. If the remaining question is why objects stay alive, follow [Debug a memory leak](debug-memory-leak.md) to inspect retention paths in a process dump. + +## Choose a focused follow-up trace + +Use the first trace to choose a direction, and then collect only the detailed events required to answer the next question. CPU samples don't explain time spent sleeping or waiting because a thread that isn't running can't be sampled. + +### Blocking, contention, and ThreadPool behavior + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile thread-time,cpu-sampling \ + --clrevents threading+contention+waithandle \ + --clreventlevel Verbose +``` + +For the complete investigation, follow [Debug ThreadPool starvation](debug-threadpool-starvation.md). For example, if `dotnet-common` or `gc-verbose` configures the runtime provider, the tool prints a warning and ignores a supplied `--clrevents` list. Use kernel-only profiles such as `cpu-sampling` or `thread-time` with a focused `--clrevents`/`--clreventlevel` configuration. + +## Know when to use another artifact + +Use a process dump for an existing [deadlock](debug-deadlock.md), application or [distributed tracing](distributed-tracing.md) for logical request relationships, and a native memory profiler for native allocation ownership. Collecting more of the same data doesn't recover information that the artifact doesn't contain. + +## Manage collection overhead + +Trace overhead depends on event rate, enabled providers, stack capture, CPU count, and workload behavior. High-volume traces can lose events or perturb the application being measured. + +- Keep the initial trace short. +- Enable only the detailed events needed for the current question. +- Treat event totals as lower bounds when they disagree with application or operating-system counters. +- Repeat the capture with a narrower configuration when exact counts matter. +- Test production collection procedures and container limits before an incident. + +## See also + +- [Performance tutorials](index.md#performance-tutorials) +- [`dotnet-trace` reference](dotnet-trace.md) +- [Debug high CPU usage](debug-highcpu.md) +- [Collect diagnostics in Linux containers](diagnostics-in-containers.md) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 447b4a905a3fd..8b77256f36932 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -1,9 +1,10 @@ --- title: dotnet-trace diagnostic tool - .NET CLI -description: Learn how to install and use the dotnet-trace CLI tool to collect .NET traces of a running process without the native profiler, by using the .NET EventPipe. -ms.date: 06/10/2026 +description: Learn how to use dotnet-trace to collect .NET application traces and Linux system-wide performance traces. +ms.date: 09/04/2026 ms.topic: reference ms.custom: sfi-ropc-nochange +ai-usage: ai-assisted --- # dotnet-trace performance analysis utility @@ -295,6 +296,8 @@ dotnet-trace collect Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` enables the following additional features over [`collect`](#dotnet-trace-collect). +For a symptom-driven collection and analysis workflow, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). + | Feature | `collect` | `collect-linux` | |------------------------------------------|-----------|-----------------------------------| | Supported OS | Any | Linux only, kernel version >= 6.4 | @@ -703,13 +706,13 @@ However, when you want to gain a finer control over the lifetime of the app bein ## (Linux-only) Collect a machine-wide trace using dotnet-trace -### Get symbols for native runtime frames +`collect-linux` dynamically enables perf map generation for JIT-compiled code, so you don't need to restart .NET processes. -`collect-linux` captures native frames in callstacks. To resolve native method names for runtime libraries (such as `libcoreclr.so`), place the corresponding debug symbol files on disk beside the libraries. Without these symbols, native frames appear as unresolved addresses in the trace. +### Resolve native names during collection -`collect-linux` dynamically enables perf map generation for JIT-compiled code when the trace begins, so you don't need to restart any .NET processes. +This setup is optional. You can collect a trace without it and [resolve native symbols later in PerfView](#get-symbols-for-native-runtime-frames-in-perfview). -To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md): +Use [dotnet-symbol](./dotnet-symbol.md) before collection to place .NET native symbols beside the corresponding runtime libraries: 1. Install `dotnet-symbol`: @@ -727,6 +730,8 @@ To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md): After you place the symbols, `collect-linux` resolves native method names when it collects the trace. +### Collect the trace + This example captures CPU samples for all processes on the machine. Any processes running .NET 10+ will also include some additional lightweight events describing GC, JIT, and Assembly loading behavior. ```output @@ -778,13 +783,31 @@ For environments with multiple .NET versions installed, running `collect-linux` On Windows, you can view *.nettrace* files in [Visual Studio](/visualstudio/profiling/beginners-guide-to-performance-profiling?#step-2-analyze-cpu-usage-data) or [PerfView](https://github.com/microsoft/perfview) for analysis. -On Linux, you can view the trace by changing the output format of `dotnet-trace` to `speedscope`. Change the output file format by using the `-f|--format` option. You can choose between `nettrace` (the default option) and `speedscope`. The option `-f speedscope` will make `dotnet-trace` produce a `speedscope` file. `Speedscope` files can be opened at . +On Linux, you can view a trace from `dotnet-trace collect` by changing its output format to `speedscope`. Change the output file format by using the `-f|--format` option. You can choose between `nettrace` (the default option) and `speedscope`. The option `-f speedscope` will make `dotnet-trace collect` produce a `speedscope` file. `Speedscope` files can be opened at . For traces collected on non-Windows platforms, you can also move the trace file to a Windows machine and view it in Visual Studio or PerfView. +For a worked example of trace collection and interpretation, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). For Visual Studio's analysis features, see [CPU Usage](/visualstudio/profiling/cpu-usage) and [Events Viewer](/visualstudio/profiling/events-viewer). + > [!NOTE] > The .NET Core runtime generates traces in the `nettrace` format. The traces are converted to speedscope (if specified) after the trace is completed. Since some conversions may result in loss of data, the original `nettrace` file is preserved next to the converted file. + + +### Get symbols for native runtime frames in PerfView + +`collect-linux` captures native and ReadyToRun (R2R) frames in call stacks. [PerfView 3.2.1 or later](https://github.com/microsoft/perfview/releases/tag/v3.2.1) can download and resolve symbols when you analyze the trace: + +- .NET native and R2R runtime symbols are available from the Microsoft Symbol Server. +- Many Azure Linux native symbols are also available from the Microsoft Symbol Server. +- For native libraries from other Linux distributions, configure PerfView with a local symbol path that contains the matching distribution symbol files. + +In PerfView, open a stack view, select the unresolved module frames, and choose **Lookup Symbols**. If prompted, enable the Microsoft Symbol Server. No symbol setup is required before collection for symbols available from the configured server or local symbol paths. For native ELF modules, PerfView uses the ELF build ID to locate matching symbols. + +Internal Linux-native frames can require matching ELF debug symbols, which the Microsoft Symbol Server also serves for .NET. Application PDBs provide managed source information; they don't replace these native symbols. Managed names and native names embedded in the trace or available from library exports can already appear without full native symbol resolution. + +A saved trace can still use network symbol servers. For analysis without network access, obtain the matching files in advance and configure a local symbol path on the analysis machine. + ## Use .rsp file to avoid typing long commands You can launch `dotnet-trace` with an `.rsp` file that contains the arguments to pass. This can be useful when enabling providers that expect lengthy arguments or when using a shell environment that strips characters. diff --git a/docs/core/diagnostics/eventpipe.md b/docs/core/diagnostics/eventpipe.md index 7e4b8ce8434c2..d9cc7afb3a5df 100644 --- a/docs/core/diagnostics/eventpipe.md +++ b/docs/core/diagnostics/eventpipe.md @@ -1,8 +1,9 @@ --- title: EventPipe Overview description: Learn about EventPipe and how to use it for tracing your .NET applications to diagnose performance issues. -ms.date: 03/19/2026 +ms.date: 09/04/2026 ms.topic: overview +ai-usage: ai-assisted --- # EventPipe @@ -23,9 +24,9 @@ To learn more about the NetTrace format, see the [NetTrace format documentation] ## EventPipe vs. ETW/perf_events -EventPipe is part of the .NET runtime and is designed to work the same way across all the platforms .NET Core supports. This allows tracing tools based on EventPipe, such as `dotnet-counters`, `dotnet-gcdump`, and `dotnet-trace`, to work seamlessly across platforms. +EventPipe is part of the .NET runtime and is designed to work the same way across all the platforms .NET supports. This allows tracing tools based on EventPipe, such as `dotnet-counters`, `dotnet-gcdump`, and `dotnet-trace`, to work seamlessly across platforms. -However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stack traces with managed code frame information only. To get events from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events, use OS-specific tracing tools such as ETW or perf_events. On Linux, the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng). +However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stack traces with managed code frame information only. To collect events from other unmanaged user-mode libraries, CPU samples for native code, or kernel events, use platform tracing facilities and collectors. Examples include ETW on Windows and [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux) or `perf` on Linux. Starting in .NET 10, EventPipe on Linux can emit events as [user_events](https://docs.kernel.org/trace/user_events.html), enabling collection of managed events, OS/kernel events, and native callstacks in a single unified trace. This mode requires admin/root privileges and Linux kernel 6.4+. For more information, see [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux). @@ -35,7 +36,7 @@ The following table is a summary of the differences between EventPipe and ETW/pe |Feature|EventPipe|EventPipe (user_events)|ETW|perf_events| |-------|---------|----------------------|---|-----------| -|Cross-platform|Yes|No (only on supported Linux distros)|No (only on Windows)|No (only on supported Linux distros)| +|Cross-platform|Yes|No (Linux only)|No (Windows only)|No (Linux only)| |Require admin/root privilege|No|Yes|Yes|Yes| |Can get OS/kernel events|No|Yes|Yes|Yes| |Can resolve native callstacks|No|Yes|Yes|Yes| @@ -50,7 +51,9 @@ You can use EventPipe to trace your .NET application in many ways: * Use [environment variables](#trace-using-environment-variables) to start EventPipe. -After you've produced a `nettrace` file that contains your EventPipe events, you can view the file in [PerfView](https://github.com/Microsoft/perfview#perfview-overview) or Visual Studio. On non-Windows platforms, you can convert the `nettrace` file to a `speedscope` or `Chromium` trace format by using [dotnet-trace convert](./dotnet-trace.md#dotnet-trace-convert) command and view it with [speedscope](https://www.speedscope.app/) or Chrome DevTools. +After you've produced a `nettrace` file that contains your EventPipe events, you can view the file in Visual Studio or [PerfView](https://github.com/Microsoft/perfview#perfview-overview) on Windows. If you collected the trace on another platform, copy it to the Windows analysis machine. + +For traces from `dotnet-trace collect`, you can convert the `nettrace` file to a `speedscope` or `Chromium` trace format by using [dotnet-trace convert](./dotnet-trace.md#dotnet-trace-convert) command and view it on non-Windows platforms with [speedscope](https://www.speedscope.app/) or Chrome DevTools. You can also analyze EventPipe traces programmatically with [TraceEvent](https://github.com/Microsoft/perfview/blob/main/documentation/TraceEvent/TraceEventLibrary.md). diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index 29c9f6d720d11..204a4cb3e269a 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -3,6 +3,7 @@ title: Collect and View EventSource Traces description: A tutorial for collecting and viewing traces of EventSource events ms.topic: tutorial ms.date: 03/03/2022 +ai-usage: ai-assisted --- # Collect and View EventSource Traces @@ -183,7 +184,7 @@ To learn more about using PerfView, see the [PerfView video tutorials](/shows/Pe [dotnet-trace](./dotnet-trace.md) is a cross-platform command-line tool that can collect traces from .NET Core apps using [EventPipe](./eventpipe.md) tracing. It doesn't support viewing trace data, but the traces it collects can be viewed by other tools such -as [PerfView](#perfview) or [Visual Studio](#visual-studio). dotnet-trace also supports converting its default *.nettrace* format traces +as [Visual Studio](#visual-studio) or [PerfView](#perfview) on Windows. dotnet-trace also supports converting its default *.nettrace* format traces into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/). ### Collect a trace diff --git a/docs/core/diagnostics/index.md b/docs/core/diagnostics/index.md index 6d9a59c4969a4..33c24b831099c 100644 --- a/docs/core/diagnostics/index.md +++ b/docs/core/diagnostics/index.md @@ -1,9 +1,10 @@ --- -title: Diagnostics tools overview - .NET Core -description: An overview of the tools and techniques available to diagnose .NET Core applications. -ms.date: 10/20/2023 +title: Diagnostics tools overview - .NET +description: An overview of the tools and techniques available to diagnose .NET applications. +ms.date: 09/08/2026 ms.topic: overview -#Customer intent: As a .NET Core developer I want to find the best tools to help me diagnose problems so that I can be productive. +#Customer intent: As a .NET developer, I want to find the best tools to help me diagnose problems so that I can be productive. +ai-usage: ai-assisted --- # Diagnostics in .NET @@ -46,7 +47,7 @@ For most cases, whether adding logging to an existing project or creating a new There are multiple ways that the instrumentation data can be egressed from the application, including: - [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/getting-started-console/README.md) - a cross-platform, vendor-neutral standard for collecting and exporting telemetry -- [.NET CLI tools](./tools-overview.md) such as [dotnet-counters](./dotnet-counters.md) +- [.NET diagnostic tools](./tools-overview.md) such as [dotnet-counters](./dotnet-counters.md) - [dotnet-monitor](./dotnet-monitor.md) - an agent for collecting traces and telemetry - Third-party libraries or app code can read the information from the , , and APIs. @@ -56,38 +57,24 @@ If debugging or observability is not sufficient, .NET supports additional diagno ## Diagnostics tools -.NET supports a number of [CLI tools](./tools-overview.md) that can be used to diagnose your applications. +.NET supports a number of [diagnostic tools](./tools-overview.md) that can be used to diagnose your applications. To automate a custom diagnostic workflow, use the [diagnostics client library](diagnostics-client-library.md) and . -## .NET Core diagnostics tutorials +## Diagnostics tutorials -### Debug a memory leak +### Performance tutorials -[Tutorial: Debug a memory leak](debug-memory-leak.md) walks through finding a memory leak. The [dotnet-counters](dotnet-counters.md) tool is used to confirm the leak and the [dotnet-dump](dotnet-dump.md) tool is used to diagnose the leak. +To find out why an application consumes more resources or responds slowly, follow a tutorial for [high CPU usage](debug-highcpu.md), [memory leaks](debug-memory-leak.md), [ThreadPool starvation](debug-threadpool-starvation.md), or [deadlocks](debug-deadlock.md). For a complete Linux example, [collect a trace and diagnose CPU or allocation pressure](dotnet-trace-collect-linux-performance.md). -### Debug high CPU usage +If you aren't sure which symptom to investigate, use [`dotnet-counters`](dotnet-counters.md) to find the process and monitor its counters while you reproduce the problem. Replace `` with the process ID: -[Tutorial: Debug high CPU usage](debug-highcpu.md) walks you through investigating high CPU usage. It uses the [dotnet-counters](dotnet-counters.md) tool to confirm the high CPU usage. It then walks you through using [Trace for performance analysis utility (`dotnet-trace`)](dotnet-trace.md) or Linux `perf` to collect and view CPU usage profile. +```dotnetcli +dotnet-counters ps +dotnet-counters monitor --process-id --showDeltas +``` -### Debug deadlock +Use CPU time, managed heap growth, and ThreadPool queue and worker counts to choose a tutorial above. Counters help confirm the symptom, but they don't identify the application code responsible. -[Tutorial: Debug deadlock](debug-deadlock.md) shows you how to use the [dotnet-dump](dotnet-dump.md) tool to investigate threads and locks. +### Crash and dump tutorials -### Debug ThreadPool Starvation - -[Tutorial: Debug threadPool starvation](debug-threadpool-starvation.md) shows you how to use the [dotnet-counters](dotnet-counters.md) and [dotnet-stack](dotnet-stack.md) tools to investigate ThreadPool starvation. - -### Debug a StackOverflow - -[Tutorial: Debug a StackOverflow](debug-stackoverflow.md) demonstrates how to debug a on Linux. - -### Debug Linux dumps - -[Debug Linux dumps](debug-linux-dumps.md) explains how to collect and analyze dumps on Linux. - -### Measure performance using EventCounters - -[Tutorial: Measure performance using EventCounters in .NET](event-counter-perf.md) shows you how to use the API to measure performance in your .NET app. - -### Write your own diagnostic tool - -[The diagnostics client library](diagnostics-client-library.md) lets you write your own custom diagnostic tool best suited for your diagnostic scenario. For more information, see the [Microsoft.Diagnostics.NETCore.Client API reference](microsoft-diagnostics-netcore-client.md). +- [Debug a StackOverflow](debug-stackoverflow.md) demonstrates how to debug a on Linux. +- [Debug Linux dumps](debug-linux-dumps.md) explains how to collect and analyze dumps on Linux. diff --git a/docs/core/diagnostics/specialized-diagnostics-overview.md b/docs/core/diagnostics/specialized-diagnostics-overview.md index 16abe55f46313..761af6a76d697 100644 --- a/docs/core/diagnostics/specialized-diagnostics-overview.md +++ b/docs/core/diagnostics/specialized-diagnostics-overview.md @@ -1,7 +1,8 @@ --- title: Specialized Diagnostics description: A guide to more advanced diagnostics support in .NET -ms.date: 05/19/2023 +ms.date: 09/04/2026 +ai-usage: ai-assisted --- # Specialized diagnostics @@ -11,11 +12,11 @@ If debugging or observability is not sufficient, .NET supports additional diagno [Event Source](./eventsource.md) provides the ability to collect detailed diagnostic information about what's happening inside .NET processes. It includes telemetry information for the runtime, GC, libraries, and application code. -Event Source data can be collected in-process using the API or with external diagnostics tools such as [Visual Studio](/visualstudio/profiling), [dotnet-monitor](./dotnet-monitor.md), [dotnet-trace](./dotnet-trace.md), [PerfView](https://github.com/microsoft/perfview), and the [Perfcollect](./trace-perfcollect-lttng.md) scripts. Using the external tools to collect event source data in traces is commonly used for performance analysis. +Event Source data can be collected in-process using the API or with external diagnostics tools such as [Visual Studio](/visualstudio/profiling), [dotnet-monitor](./dotnet-monitor.md), [dotnet-trace](./dotnet-trace.md), [PerfView](https://github.com/microsoft/perfview), and [PerfCollect](./trace-perfcollect-lttng.md). On .NET 10 and later Linux systems, use [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux) when runtime events must be correlated with native call stacks and kernel events. ### EventPipe -[EventPipe](./eventpipe.md) is a runtime component that can be used to collect tracing data, similar to ETW or LTTng. The goal of EventPipe is to allow .NET developers to easily trace their .NET applications without having to rely on platform-specific, OS-native components, such as ETW or LTTng. +[EventPipe](./eventpipe.md) is a runtime component that can be used to collect tracing data, similar to ETW or `perf_events`. The goal of EventPipe is to allow .NET developers to easily trace their .NET applications without having to rely on platform-specific, OS-native components, such as ETW or `perf_events`. EventPipe is the mechanism behind many of the diagnostic tools. It can be used for consuming events emitted by the runtime as well as custom events written with [EventSource](xref:System.Diagnostics.Tracing.EventSource). @@ -46,4 +47,5 @@ The same diagnostics tools that are used in non-containerized Linux environments ## See also - [Debug high CPU usage](./debug-highcpu.md) +- [Collect a Linux trace with dotnet-trace](./dotnet-trace.md#dotnet-trace-collect-linux) - [Collect a performance trace in Linux with PerfCollect](./trace-perfcollect-lttng.md) diff --git a/docs/core/diagnostics/tools-overview.md b/docs/core/diagnostics/tools-overview.md index 5ab847b2ccacb..28b3d7cab66d3 100644 --- a/docs/core/diagnostics/tools-overview.md +++ b/docs/core/diagnostics/tools-overview.md @@ -1,9 +1,10 @@ --- title: .NET Diagnostic tools overview -description: An overview of the tools available to diagnose .NET Core applications. -ms.date: 06/8/2023 +description: An overview of the tools available to diagnose .NET applications. +ms.date: 09/04/2026 ms.topic: overview -#Customer intent: As a .NET Core developer I want to find the best tools to help me diagnose problems so that I can be productive. +#Customer intent: As a .NET developer I want to find the best tools to help me diagnose problems so that I can be productive. +ai-usage: ai-assisted --- # .NET diagnostic tools @@ -24,7 +25,7 @@ ms.topic: overview ### dotnet-counters -[dotnet-counters](dotnet-counters.md) is a performance monitoring tool for first-level health monitoring and performance investigation. It observes performance counter values published via the API. For example, you can quickly monitor things like the CPU usage or the rate of exceptions being thrown in your .NET Core application. +[dotnet-counters](dotnet-counters.md) is a performance monitoring tool for first-level health monitoring and performance investigation. It observes performance counter values published via the API. For example, you can quickly monitor things like the CPU usage or the rate of exceptions being thrown in your .NET application. ### dotnet-dump @@ -40,7 +41,7 @@ The [dotnet-monitor](dotnet-monitor.md) tool is a way to monitor .NET applicatio ### dotnet-trace -.NET Core includes `EventPipe`, which exposes diagnostics data. The [dotnet-trace](dotnet-trace.md) tool allows you to consume interesting profiling data from your app that can help in scenarios where you need to root-cause apps running that are running slowly. +The [dotnet-trace](dotnet-trace.md) tool is a cross-platform .NET diagnostic tool that collects traces from running applications without using a native profiler. On Linux with .NET 10 or later, its `collect-linux` command can also combine .NET runtime and application events with machine-wide CPU samples, native call stacks, and selected Linux kernel events through `perf_events`. For a collection-to-diagnosis walkthrough with Visual Studio and PerfView, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). ### dotnet-stack @@ -56,6 +57,10 @@ The [dotnet-stack](dotnet-stack.md) tool allows you to quickly print the managed ## Other tools +### PerfView + +[PerfView](https://github.com/microsoft/perfview) is a Windows tool for performance trace collection and analysis. It can also analyze `.nettrace` files collected by `dotnet-trace` on other platforms. + ### PerfCollect -[PerfCollect](trace-perfcollect-lttng.md) is a bash script you can use to collect traces with `perf` and `LTTng` for a more in-depth performance analysis of .NET apps running on Linux distributions. +[PerfCollect](trace-perfcollect-lttng.md) is the earlier bash-based workflow for collecting Linux CPU samples with `perf` and .NET runtime and EventSource events with LTTng. Prefer `dotnet-trace collect-linux` for new investigations. PerfCollect remains documented for existing workflows, but its runtime-event collection depends on an older LTTng ABI. diff --git a/docs/core/diagnostics/trace-perfcollect-lttng.md b/docs/core/diagnostics/trace-perfcollect-lttng.md index 328e6c413f4cb..c6f17d6ef6930 100644 --- a/docs/core/diagnostics/trace-perfcollect-lttng.md +++ b/docs/core/diagnostics/trace-perfcollect-lttng.md @@ -2,13 +2,16 @@ title: Tracing .NET applications with PerfCollect. description: A tutorial that walks you through collecting a trace with perfcollect in .NET. ms.topic: tutorial -ms.date: 04/10/2025 +ms.date: 09/04/2026 --- # Trace .NET applications with PerfCollect **This article applies to: ✔️** .NET Core 2.1 SDK and later versions +> [!IMPORTANT] +> For .NET 10+ Linux investigations, prefer [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). PerfCollect is the earlier .NET Linux tracing workflow. Its .NET runtime event collection requires LTTng 2.12; on distributions with LTTng 2.13 or later, the LTTng portion must be disabled as described later in this article. + When performance problems are encountered on Linux, collecting a trace with `perfcollect` can be used to gather detailed information about what was happening on the machine at the time of the performance problem. `perfcollect` is a bash script that uses [Linux Trace Toolkit: next generation (LTTng)](https://lttng.org) to collect events written from the runtime or any [EventSource](xref:System.Diagnostics.Tracing.EventListener), as well as [perf](https://perf.wiki.kernel.org/) to collect CPU samples of the target process. @@ -65,7 +68,6 @@ For resolving method names of native runtime DLLs (such as libcoreclr.so), `perf > [!NOTE] > LTTng had a breaking change between versions 2.12 and 2.13. The .NET runtime currently supports version 2.12. If your Linux distribution has adopted 2.13 or later then we recommend disabling the LTTng portion of the perfcollect functionality. To do this add the option '-nolttng' to the perfcollect command-line and in step 3 do not set the DOTNET_EnableEventLog environment variable. -1. **[App]** Set up the application shell with the following environment variables - this enables tracing configuration of CoreCLR. 1. **[App]** Set up the application shell with the following environment variables - this enables tracing configuration of CoreCLR. ```bash diff --git a/docs/navigate/tools-diagnostics/toc.yml b/docs/navigate/tools-diagnostics/toc.yml index 747c01c7edf78..709f725ae0313 100644 --- a/docs/navigate/tools-diagnostics/toc.yml +++ b/docs/navigate/tools-diagnostics/toc.yml @@ -606,6 +606,8 @@ items: href: ../../core/diagnostics/dotnet-dsrouter.md - name: .NET diagnostics tutorials items: + - name: "Tutorial: Investigate Linux performance" + href: ../../core/diagnostics/dotnet-trace-collect-linux-performance.md - name: Collect performance trace in Linux with PerfCollect href: ../../core/diagnostics/trace-perfcollect-lttng.md - name: Debug a memory leak