Skip to content

test(microbenchmarking): adding walk operation to the microbenchmarking list suite - #989

Merged
ankitaluthra1 merged 4 commits into
fsspec:mainfrom
raj-prince:microbench_walk
Jul 29, 2026
Merged

test(microbenchmarking): adding walk operation to the microbenchmarking list suite#989
ankitaluthra1 merged 4 commits into
fsspec:mainfrom
raj-prince:microbench_walk

Conversation

@raj-prince

@raj-prince raj-prince commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Adding walk operation to the microbenchmarking list suite.

Sample result:

------------------------------------------------------------------------------------------------------------------------------ benchmark 'listing': 8 tests -----------------------------------------------------------------------------------------------------------------------------
Name (time in s)                                                                                                               Min                Max               Mean            StdDev             Median               IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_listing_single_threaded[walk_flat_1procs_1threads_65536files_0depth_1folders_walk_regional]                            3.8861 (1.0)       3.8861 (1.0)       3.8861 (1.0)      0.0000 (1.0)       3.8861 (1.0)      0.0000 (1.0)           0;0  0.2573 (1.0)           1           1
test_listing_single_threaded[walk_flat_folders_1procs_1threads_65536files_0depth_256folders_walk_regional]                  8.1713 (2.10)      8.1713 (2.10)      8.1713 (2.10)     0.0000 (1.0)       8.1713 (2.10)     0.0000 (1.0)           0;0  0.1224 (0.48)          1           1
test_listing_single_threaded[walk_flat_1procs_1threads_131072files_0depth_1folders_walk_regional]                           8.2655 (2.13)      8.2655 (2.13)      8.2655 (2.13)     0.0000 (1.0)       8.2655 (2.13)     0.0000 (1.0)           0;0  0.1210 (0.47)          1           1
test_listing_single_threaded[walk_recursive_folders_deep_1procs_1threads_65536files_24depth_256folders_walk_regional]       8.6664 (2.23)      8.6664 (2.23)      8.6664 (2.23)     0.0000 (1.0)       8.6664 (2.23)     0.0000 (1.0)           0;0  0.1154 (0.45)          1           1
test_listing_single_threaded[walk_recursive_folders_1procs_1threads_65536files_8depth_256folders_walk_regional]             9.3824 (2.41)      9.3824 (2.41)      9.3824 (2.41)     0.0000 (1.0)       9.3824 (2.41)     0.0000 (1.0)           0;0  0.1066 (0.41)          1           1
test_listing_single_threaded[walk_flat_folders_1procs_1threads_131072files_0depth_256folders_walk_regional]                10.6405 (2.74)     10.6405 (2.74)     10.6405 (2.74)     0.0000 (1.0)      10.6405 (2.74)     0.0000 (1.0)           0;0  0.0940 (0.37)          1           1
test_listing_single_threaded[walk_recursive_folders_1procs_1threads_131072files_8depth_256folders_walk_regional]           10.7992 (2.78)     10.7992 (2.78)     10.7992 (2.78)     0.0000 (1.0)      10.7992 (2.78)     0.0000 (1.0)           0;0  0.0926 (0.36)          1           1
test_listing_single_threaded[walk_recursive_folders_deep_1procs_1threads_131072files_24depth_256folders_walk_regional]     11.4768 (2.95)     11.4768 (2.95)     11.4768 (2.95)     0.0000 (1.0)      11.4768 (2.95)     0.0000 (1.0)           0;0  0.0871 (0.34)          1           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Bucket Type Group Pattern Seq Prob Files Folders Threads Processes Depth Target Type File Size (MiB) Chunk Size (MiB) Min Chunk (MiB) Max Chunk (MiB) Block Size (MiB) MRD Pool Cache Size MRD Pool Size Mean Latency (s) Mean Throughput (MiB/s) Max CPU (%) Max Memory (MiB)
regional listing walk N/A 65536 1 1 1 0 N/A N/A N/A N/A N/A N/A N/A N/A 3.8861 N/A 0.81 477.49
regional listing walk N/A 131072 1 1 1 0 N/A N/A N/A N/A N/A N/A N/A N/A 8.2655 N/A 0.50 824.49
regional listing walk N/A 65536 256 1 1 0 N/A N/A N/A N/A N/A N/A N/A N/A 8.1713 N/A 0.65 816.99
regional listing walk N/A 131072 256 1 1 0 N/A N/A N/A N/A N/A N/A N/A N/A 10.6405 N/A 0.63 962.92
regional listing walk N/A 65536 256 1 1 8 N/A N/A N/A N/A N/A N/A N/A N/A 9.3824 N/A 0.25 902.80
regional listing walk N/A 131072 256 1 1 8 N/A N/A N/A N/A N/A N/A N/A N/A 10.7992 N/A 0.68 1024.73
regional listing walk N/A 65536 256 1 1 24 N/A N/A N/A N/A N/A N/A N/A N/A 8.6664 N/A 0.47 1081.18
regional listing walk N/A 131072 256 1 1 24 N/A N/A N/A N/A N/A N/A N/A N/A 11.4768 N/A 0.84 1116.96

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for benchmarking the walk operation in GCSFS microbenchmarks. It updates the documentation, adds new benchmark scenarios for flat and recursive directory walking, implements the walk pattern logic in the listing benchmark tests, and adds corresponding unit tests to verify the configuration. There are no review comments, so I have no feedback to provide.

@raj-prince raj-prince changed the title test(microbenchmarking): adding support for microbenchmarking walk operation test(microbenchmarking): adding walk operation to the microbenchmarking Jul 29, 2026
@raj-prince raj-prince changed the title test(microbenchmarking): adding walk operation to the microbenchmarking test(microbenchmarking): adding walk operation to the microbenchmarking list suite Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.68%. Comparing base (18afe60) to head (4b424f4).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #989   +/-   ##
=======================================
  Coverage   89.68%   89.68%           
=======================================
  Files          16       16           
  Lines        3579     3579           
=======================================
  Hits         3210     3210           
  Misses        369      369           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ankitaluthra1

Copy link
Copy Markdown
Collaborator

Can you add the result sample in PR description.

Comment thread gcsfs/tests/perf/microbenchmarks/listing/configs.yaml
@ankitaluthra1
ankitaluthra1 merged commit 370ca0b into fsspec:main Jul 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants