.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "_as_gen/benchmarks/run_digit_histogram_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr__as_gen_benchmarks_run_digit_histogram_performance.py: digit histogram performance benchmarks ====================================== This example benchmarks the digit histogram primitive (the upfront histogram of every radix digit place of a key buffer, as used by the Onesweep radix sort) for the different digit sizes available in Shamrock, on 3 key distributions: * ``zeros`` : every key is 0, so every digit place of every key lands in the same bin (worst case for the atomic contention on the histogram bins) * ``shuffled iota`` : a random permutation of ``0 .. N-1``, the low digit places are uniform while the high ones are concentrated on the few lowest bins * ``random`` : uniformly distributed keys over the full u32 range, every digit place is uniform .. GENERATED FROM PYTHON SOURCE LINES 15-32 .. code-block:: Python :lineno-start: 17 import time import matplotlib.pyplot as plt import numpy as np from shamrock.utils.plot import make_std_bench_plot import shamrock # If we use the shamrock executable to run this script instead of the python interpreter, # we should not initialize the system as the shamrock executable needs to handle specific MPI logic if not shamrock.sys.is_initialized(): shamrock.change_loglevel(1) shamrock.sys.init("0:0") .. GENERATED FROM PYTHON SOURCE LINES 34-35 Use shamrock documentation style for matplotlib .. GENERATED FROM PYTHON SOURCE LINES 35-37 .. code-block:: Python :lineno-start: 35 shamrock.matplotlib.set_shamrock_mpl_style() .. GENERATED FROM PYTHON SOURCE LINES 38-39 Digit sizes (in bits) supported by the digit histogram .. GENERATED FROM PYTHON SOURCE LINES 39-41 .. code-block:: Python :lineno-start: 39 radix_bits_list = [1, 2, 4, 8] .. GENERATED FROM PYTHON SOURCE LINES 42-43 Key distributions to benchmark .. GENERATED FROM PYTHON SOURCE LINES 43-64 .. code-block:: Python :lineno-start: 43 key_cases = ["zeros", "shuffled iota", "random"] def make_keys(case, N, seed=111): if case == "zeros": keys = shamrock.backends.DeviceBuffer_u32() keys.resize(N) keys.fill(0) return keys elif case == "shuffled iota": rng = np.random.default_rng(seed) keys = shamrock.backends.DeviceBuffer_u32() keys.resize(N) keys.copy_from_stdvec(rng.permutation(N).astype(np.uint32).tolist()) return keys elif case == "random": return shamrock.algs.mock_buffer_u32(seed, N, 0, 2**32 - 1) else: raise ValueError(f"unknown key case {case}") .. GENERATED FROM PYTHON SOURCE LINES 65-66 Check the result against numpy .. GENERATED FROM PYTHON SOURCE LINES 66-92 .. code-block:: Python :lineno-start: 66 def reference_digit_histogram(keys, radix_bits): keys_np = np.array(keys.copy_to_stdvec(), dtype=np.uint64) nbuckets = 2**radix_bits npasses = 32 // radix_bits return np.concatenate( [ np.bincount((keys_np >> (p * radix_bits)) & (nbuckets - 1), minlength=nbuckets) for p in range(npasses) ] ) def compute_digit_histogram(keys, radix_bits): N = keys.get_size() return np.array(shamrock.algs.digit_histogram(keys, radix_bits, N).copy_to_stdvec()) for case in key_cases: keys = make_keys(case, 100003) for radix_bits in radix_bits_list: ok = np.array_equal( compute_digit_histogram(keys, radix_bits), reference_digit_histogram(keys, radix_bits) ) print(f"{case:>13s}, radix_bits={radix_bits} : result matches numpy = {ok}") .. rst-class:: sphx-glr-script-out .. code-block:: none zeros, radix_bits=1 : result matches numpy = True zeros, radix_bits=2 : result matches numpy = True zeros, radix_bits=4 : result matches numpy = True zeros, radix_bits=8 : result matches numpy = True shuffled iota, radix_bits=1 : result matches numpy = True shuffled iota, radix_bits=2 : result matches numpy = True shuffled iota, radix_bits=4 : result matches numpy = True shuffled iota, radix_bits=8 : result matches numpy = True random, radix_bits=1 : result matches numpy = True random, radix_bits=2 : result matches numpy = True random, radix_bits=4 : result matches numpy = True random, radix_bits=8 : result matches numpy = True .. GENERATED FROM PYTHON SOURCE LINES 93-94 Main benchmark function .. GENERATED FROM PYTHON SOURCE LINES 94-109 .. code-block:: Python :lineno-start: 94 def benchmark_u32(keys, radix_bits, nb_repeat=10, max_cumulated_time=2.0): N = keys.get_size() times = [] cumulated_time = 0.0 for _ in range(nb_repeat): t = shamrock.algs.benchmark_digit_histogram(keys, radix_bits, N) times.append(t) cumulated_time += t if cumulated_time > max_cumulated_time: break return min(times), max(times), sum(times) / len(times) .. GENERATED FROM PYTHON SOURCE LINES 110-112 Run the performance test for all parameters (the keys of a given case and size are generated once and reused for every digit size) .. GENERATED FROM PYTHON SOURCE LINES 112-142 .. code-block:: Python :lineno-start: 112 def run_performance_sweep(): # logspace as array, deliberately not restricted to powers of 2 particle_counts = np.logspace(2, 7, 20).astype(int).tolist() results = {case: {radix_bits: [] for radix_bits in radix_bits_list} for case in key_cases} print(f"Particle counts: {particle_counts}") total_runs = len(particle_counts) for current_run, N in enumerate(particle_counts, start=1): for case in key_cases: keys = make_keys(case, N) for radix_bits in radix_bits_list: print( f"[{current_run:2d}/{total_runs}] Running N={N:8d}, {case:>13s}, " f"radix_bits={radix_bits}...", end=" ", ) start_time = time.time() min_time, max_time, mean_time = benchmark_u32(keys, radix_bits) results[case][radix_bits].append(min_time) elapsed = time.time() - start_time print(f"mean={mean_time:.3e}s (took {elapsed:.1f}s)") return particle_counts, results .. GENERATED FROM PYTHON SOURCE LINES 143-144 Run the performance benchmarks for all key distributions and digit sizes .. GENERATED FROM PYTHON SOURCE LINES 144-148 .. code-block:: Python :lineno-start: 145 particle_counts, results = run_performance_sweep() .. rst-class:: sphx-glr-script-out .. code-block:: none Particle counts: [100, 183, 335, 615, 1128, 2069, 3792, 6951, 12742, 23357, 42813, 78475, 143844, 263665, 483293, 885866, 1623776, 2976351, 5455594, 10000000] [ 1/20] Running N= 100, zeros, radix_bits=1... mean=6.500e-05s (took 0.0s) [ 1/20] Running N= 100, zeros, radix_bits=2... mean=4.938e-05s (took 0.0s) [ 1/20] Running N= 100, zeros, radix_bits=4... mean=4.060e-05s (took 0.0s) [ 1/20] Running N= 100, zeros, radix_bits=8... mean=3.706e-05s (took 0.0s) [ 1/20] Running N= 100, shuffled iota, radix_bits=1... mean=5.473e-05s (took 0.0s) [ 1/20] Running N= 100, shuffled iota, radix_bits=2... mean=4.540e-05s (took 0.0s) [ 1/20] Running N= 100, shuffled iota, radix_bits=4... mean=4.078e-05s (took 0.0s) [ 1/20] Running N= 100, shuffled iota, radix_bits=8... mean=3.796e-05s (took 0.0s) [ 1/20] Running N= 100, random, radix_bits=1... mean=5.621e-05s (took 0.0s) [ 1/20] Running N= 100, random, radix_bits=2... mean=4.567e-05s (took 0.0s) [ 1/20] Running N= 100, random, radix_bits=4... mean=4.117e-05s (took 0.0s) [ 1/20] Running N= 100, random, radix_bits=8... mean=3.784e-05s (took 0.0s) [ 2/20] Running N= 183, zeros, radix_bits=1... mean=7.473e-05s (took 0.0s) [ 2/20] Running N= 183, zeros, radix_bits=2... mean=5.404e-05s (took 0.0s) [ 2/20] Running N= 183, zeros, radix_bits=4... mean=4.376e-05s (took 0.0s) [ 2/20] Running N= 183, zeros, radix_bits=8... mean=3.890e-05s (took 0.0s) [ 2/20] Running N= 183, shuffled iota, radix_bits=1... mean=7.268e-05s (took 0.0s) [ 2/20] Running N= 183, shuffled iota, radix_bits=2... mean=5.222e-05s (took 0.0s) [ 2/20] Running N= 183, shuffled iota, radix_bits=4... mean=4.575e-05s (took 0.0s) [ 2/20] Running N= 183, shuffled iota, radix_bits=8... mean=4.065e-05s (took 0.0s) [ 2/20] Running N= 183, random, radix_bits=1... mean=7.273e-05s (took 0.0s) [ 2/20] Running N= 183, random, radix_bits=2... mean=5.327e-05s (took 0.0s) [ 2/20] Running N= 183, random, radix_bits=4... mean=4.451e-05s (took 0.0s) [ 2/20] Running N= 183, random, radix_bits=8... mean=4.091e-05s (took 0.0s) [ 3/20] Running N= 335, zeros, radix_bits=1... mean=1.052e-04s (took 0.0s) [ 3/20] Running N= 335, zeros, radix_bits=2... mean=7.007e-05s (took 0.0s) [ 3/20] Running N= 335, zeros, radix_bits=4... mean=5.265e-05s (took 0.0s) [ 3/20] Running N= 335, zeros, radix_bits=8... mean=4.234e-05s (took 0.0s) [ 3/20] Running N= 335, shuffled iota, radix_bits=1... mean=1.063e-04s (took 0.0s) [ 3/20] Running N= 335, shuffled iota, radix_bits=2... mean=7.043e-05s (took 0.0s) [ 3/20] Running N= 335, shuffled iota, radix_bits=4... mean=5.272e-05s (took 0.0s) [ 3/20] Running N= 335, shuffled iota, radix_bits=8... mean=4.469e-05s (took 0.0s) [ 3/20] Running N= 335, random, radix_bits=1... mean=1.099e-04s (took 0.0s) [ 3/20] Running N= 335, random, radix_bits=2... mean=6.919e-05s (took 0.0s) [ 3/20] Running N= 335, random, radix_bits=4... mean=5.233e-05s (took 0.0s) [ 3/20] Running N= 335, random, radix_bits=8... mean=4.919e-05s (took 0.0s) [ 4/20] Running N= 615, zeros, radix_bits=1... mean=1.685e-04s (took 0.0s) [ 4/20] Running N= 615, zeros, radix_bits=2... mean=1.005e-04s (took 0.0s) [ 4/20] Running N= 615, zeros, radix_bits=4... mean=6.759e-05s (took 0.0s) [ 4/20] Running N= 615, zeros, radix_bits=8... mean=4.996e-05s (took 0.0s) [ 4/20] Running N= 615, shuffled iota, radix_bits=1... mean=1.663e-04s (took 0.0s) [ 4/20] Running N= 615, shuffled iota, radix_bits=2... mean=1.014e-04s (took 0.0s) [ 4/20] Running N= 615, shuffled iota, radix_bits=4... mean=6.616e-05s (took 0.0s) [ 4/20] Running N= 615, shuffled iota, radix_bits=8... mean=5.276e-05s (took 0.0s) [ 4/20] Running N= 615, random, radix_bits=1... mean=1.663e-04s (took 0.0s) [ 4/20] Running N= 615, random, radix_bits=2... mean=9.897e-05s (took 0.0s) [ 4/20] Running N= 615, random, radix_bits=4... mean=6.804e-05s (took 0.0s) [ 4/20] Running N= 615, random, radix_bits=8... mean=5.717e-05s (took 0.0s) [ 5/20] Running N= 1128, zeros, radix_bits=1... mean=2.831e-04s (took 0.0s) [ 5/20] Running N= 1128, zeros, radix_bits=2... mean=1.537e-04s (took 0.0s) [ 5/20] Running N= 1128, zeros, radix_bits=4... mean=9.503e-05s (took 0.0s) [ 5/20] Running N= 1128, zeros, radix_bits=8... mean=6.553e-05s (took 0.0s) [ 5/20] Running N= 1128, shuffled iota, radix_bits=1... mean=2.750e-04s (took 0.0s) [ 5/20] Running N= 1128, shuffled iota, radix_bits=2... mean=1.534e-04s (took 0.0s) [ 5/20] Running N= 1128, shuffled iota, radix_bits=4... mean=9.591e-05s (took 0.0s) [ 5/20] Running N= 1128, shuffled iota, radix_bits=8... mean=7.640e-05s (took 0.0s) [ 5/20] Running N= 1128, random, radix_bits=1... mean=2.848e-04s (took 0.0s) [ 5/20] Running N= 1128, random, radix_bits=2... mean=1.631e-04s (took 0.0s) [ 5/20] Running N= 1128, random, radix_bits=4... mean=1.033e-04s (took 0.0s) [ 5/20] Running N= 1128, random, radix_bits=8... mean=7.883e-05s (took 0.0s) [ 6/20] Running N= 2069, zeros, radix_bits=1... mean=4.678e-04s (took 0.0s) [ 6/20] Running N= 2069, zeros, radix_bits=2... mean=2.516e-04s (took 0.0s) [ 6/20] Running N= 2069, zeros, radix_bits=4... mean=1.430e-04s (took 0.0s) [ 6/20] Running N= 2069, zeros, radix_bits=8... mean=8.863e-05s (took 0.0s) [ 6/20] Running N= 2069, shuffled iota, radix_bits=1... mean=4.708e-04s (took 0.0s) [ 6/20] Running N= 2069, shuffled iota, radix_bits=2... mean=2.496e-04s (took 0.0s) [ 6/20] Running N= 2069, shuffled iota, radix_bits=4... mean=1.447e-04s (took 0.0s) [ 6/20] Running N= 2069, shuffled iota, radix_bits=8... mean=8.927e-05s (took 0.0s) [ 6/20] Running N= 2069, random, radix_bits=1... mean=4.704e-04s (took 0.0s) [ 6/20] Running N= 2069, random, radix_bits=2... mean=2.513e-04s (took 0.0s) [ 6/20] Running N= 2069, random, radix_bits=4... mean=1.420e-04s (took 0.0s) [ 6/20] Running N= 2069, random, radix_bits=8... mean=9.431e-05s (took 0.0s) [ 7/20] Running N= 3792, zeros, radix_bits=1... mean=4.680e-04s (took 0.0s) [ 7/20] Running N= 3792, zeros, radix_bits=2... mean=2.506e-04s (took 0.0s) [ 7/20] Running N= 3792, zeros, radix_bits=4... mean=1.382e-04s (took 0.0s) [ 7/20] Running N= 3792, zeros, radix_bits=8... mean=8.455e-05s (took 0.0s) [ 7/20] Running N= 3792, shuffled iota, radix_bits=1... mean=4.660e-04s (took 0.0s) [ 7/20] Running N= 3792, shuffled iota, radix_bits=2... mean=2.785e-04s (took 0.0s) [ 7/20] Running N= 3792, shuffled iota, radix_bits=4... mean=1.413e-04s (took 0.0s) [ 7/20] Running N= 3792, shuffled iota, radix_bits=8... mean=8.897e-05s (took 0.0s) [ 7/20] Running N= 3792, random, radix_bits=1... mean=4.661e-04s (took 0.0s) [ 7/20] Running N= 3792, random, radix_bits=2... mean=2.461e-04s (took 0.0s) [ 7/20] Running N= 3792, random, radix_bits=4... mean=1.430e-04s (took 0.0s) [ 7/20] Running N= 3792, random, radix_bits=8... mean=9.486e-05s (took 0.0s) [ 8/20] Running N= 6951, zeros, radix_bits=1... mean=8.794e-04s (took 0.0s) [ 8/20] Running N= 6951, zeros, radix_bits=2... mean=4.403e-04s (took 0.0s) [ 8/20] Running N= 6951, zeros, radix_bits=4... mean=2.383e-04s (took 0.0s) [ 8/20] Running N= 6951, zeros, radix_bits=8... mean=1.377e-04s (took 0.0s) [ 8/20] Running N= 6951, shuffled iota, radix_bits=1... mean=8.548e-04s (took 0.0s) [ 8/20] Running N= 6951, shuffled iota, radix_bits=2... mean=4.403e-04s (took 0.0s) [ 8/20] Running N= 6951, shuffled iota, radix_bits=4... mean=2.504e-04s (took 0.0s) [ 8/20] Running N= 6951, shuffled iota, radix_bits=8... mean=1.422e-04s (took 0.0s) [ 8/20] Running N= 6951, random, radix_bits=1... mean=8.492e-04s (took 0.0s) [ 8/20] Running N= 6951, random, radix_bits=2... mean=4.832e-04s (took 0.0s) [ 8/20] Running N= 6951, random, radix_bits=4... mean=2.631e-04s (took 0.0s) [ 8/20] Running N= 6951, random, radix_bits=8... mean=1.580e-04s (took 0.0s) [ 9/20] Running N= 12742, zeros, radix_bits=1... mean=1.814e-03s (took 0.0s) [ 9/20] Running N= 12742, zeros, radix_bits=2... mean=8.414e-04s (took 0.0s) [ 9/20] Running N= 12742, zeros, radix_bits=4... mean=4.516e-04s (took 0.0s) [ 9/20] Running N= 12742, zeros, radix_bits=8... mean=2.436e-04s (took 0.0s) [ 9/20] Running N= 12742, shuffled iota, radix_bits=1... mean=1.751e-03s (took 0.0s) [ 9/20] Running N= 12742, shuffled iota, radix_bits=2... mean=8.535e-04s (took 0.0s) [ 9/20] Running N= 12742, shuffled iota, radix_bits=4... mean=4.579e-04s (took 0.0s) [ 9/20] Running N= 12742, shuffled iota, radix_bits=8... mean=2.521e-04s (took 0.0s) [ 9/20] Running N= 12742, random, radix_bits=1... mean=1.649e-03s (took 0.0s) [ 9/20] Running N= 12742, random, radix_bits=2... mean=8.450e-04s (took 0.0s) [ 9/20] Running N= 12742, random, radix_bits=4... mean=4.480e-04s (took 0.0s) [ 9/20] Running N= 12742, random, radix_bits=8... mean=2.650e-04s (took 0.0s) [10/20] Running N= 23357, zeros, radix_bits=1... mean=2.488e-03s (took 0.1s) [10/20] Running N= 23357, zeros, radix_bits=2... mean=1.237e-03s (took 0.0s) [10/20] Running N= 23357, zeros, radix_bits=4... mean=6.517e-04s (took 0.0s) [10/20] Running N= 23357, zeros, radix_bits=8... mean=3.489e-04s (took 0.0s) [10/20] Running N= 23357, shuffled iota, radix_bits=1... mean=2.518e-03s (took 0.1s) [10/20] Running N= 23357, shuffled iota, radix_bits=2... mean=1.230e-03s (took 0.0s) [10/20] Running N= 23357, shuffled iota, radix_bits=4... mean=7.318e-04s (took 0.0s) [10/20] Running N= 23357, shuffled iota, radix_bits=8... mean=4.588e-04s (took 0.0s) [10/20] Running N= 23357, random, radix_bits=1... mean=2.637e-03s (took 0.1s) [10/20] Running N= 23357, random, radix_bits=2... mean=1.229e-03s (took 0.0s) [10/20] Running N= 23357, random, radix_bits=4... mean=6.622e-04s (took 0.0s) [10/20] Running N= 23357, random, radix_bits=8... mean=4.058e-04s (took 0.0s) [11/20] Running N= 42813, zeros, radix_bits=1... mean=4.523e-03s (took 0.1s) [11/20] Running N= 42813, zeros, radix_bits=2... mean=2.288e-03s (took 0.0s) [11/20] Running N= 42813, zeros, radix_bits=4... mean=1.167e-03s (took 0.0s) [11/20] Running N= 42813, zeros, radix_bits=8... mean=6.142e-04s (took 0.0s) [11/20] Running N= 42813, shuffled iota, radix_bits=1... mean=4.520e-03s (took 0.1s) [11/20] Running N= 42813, shuffled iota, radix_bits=2... mean=2.320e-03s (took 0.0s) [11/20] Running N= 42813, shuffled iota, radix_bits=4... mean=1.177e-03s (took 0.0s) [11/20] Running N= 42813, shuffled iota, radix_bits=8... mean=6.638e-04s (took 0.0s) [11/20] Running N= 42813, random, radix_bits=1... mean=4.642e-03s (took 0.1s) [11/20] Running N= 42813, random, radix_bits=2... mean=2.295e-03s (took 0.0s) [11/20] Running N= 42813, random, radix_bits=4... mean=1.185e-03s (took 0.0s) [11/20] Running N= 42813, random, radix_bits=8... mean=7.089e-04s (took 0.0s) [12/20] Running N= 78475, zeros, radix_bits=1... mean=8.249e-03s (took 0.2s) [12/20] Running N= 78475, zeros, radix_bits=2... mean=4.025e-03s (took 0.1s) [12/20] Running N= 78475, zeros, radix_bits=4... mean=2.050e-03s (took 0.0s) [12/20] Running N= 78475, zeros, radix_bits=8... mean=1.091e-03s (took 0.0s) [12/20] Running N= 78475, shuffled iota, radix_bits=1... mean=8.299e-03s (took 0.2s) [12/20] Running N= 78475, shuffled iota, radix_bits=2... mean=4.042e-03s (took 0.1s) [12/20] Running N= 78475, shuffled iota, radix_bits=4... mean=2.146e-03s (took 0.0s) [12/20] Running N= 78475, shuffled iota, radix_bits=8... mean=1.172e-03s (took 0.0s) [12/20] Running N= 78475, random, radix_bits=1... mean=8.373e-03s (took 0.2s) [12/20] Running N= 78475, random, radix_bits=2... mean=4.067e-03s (took 0.1s) [12/20] Running N= 78475, random, radix_bits=4... mean=2.110e-03s (took 0.0s) [12/20] Running N= 78475, random, radix_bits=8... mean=1.252e-03s (took 0.0s) [13/20] Running N= 143844, zeros, radix_bits=1... mean=1.473e-02s (took 0.3s) [13/20] Running N= 143844, zeros, radix_bits=2... mean=7.233e-03s (took 0.1s) [13/20] Running N= 143844, zeros, radix_bits=4... mean=3.665e-03s (took 0.1s) [13/20] Running N= 143844, zeros, radix_bits=8... mean=1.936e-03s (took 0.0s) [13/20] Running N= 143844, shuffled iota, radix_bits=1... mean=1.487e-02s (took 0.3s) [13/20] Running N= 143844, shuffled iota, radix_bits=2... mean=7.335e-03s (took 0.1s) [13/20] Running N= 143844, shuffled iota, radix_bits=4... mean=3.758e-03s (took 0.1s) [13/20] Running N= 143844, shuffled iota, radix_bits=8... mean=2.064e-03s (took 0.0s) [13/20] Running N= 143844, random, radix_bits=1... mean=1.479e-02s (took 0.3s) [13/20] Running N= 143844, random, radix_bits=2... mean=7.436e-03s (took 0.1s) [13/20] Running N= 143844, random, radix_bits=4... mean=3.781e-03s (took 0.1s) [13/20] Running N= 143844, random, radix_bits=8... mean=2.207e-03s (took 0.0s) [14/20] Running N= 263665, zeros, radix_bits=1... mean=2.722e-02s (took 0.5s) [14/20] Running N= 263665, zeros, radix_bits=2... mean=1.338e-02s (took 0.3s) [14/20] Running N= 263665, zeros, radix_bits=4... mean=6.617e-03s (took 0.1s) [14/20] Running N= 263665, zeros, radix_bits=8... mean=3.441e-03s (took 0.1s) [14/20] Running N= 263665, shuffled iota, radix_bits=1... mean=2.743e-02s (took 0.5s) [14/20] Running N= 263665, shuffled iota, radix_bits=2... mean=1.356e-02s (took 0.3s) [14/20] Running N= 263665, shuffled iota, radix_bits=4... mean=6.874e-03s (took 0.1s) [14/20] Running N= 263665, shuffled iota, radix_bits=8... mean=3.709e-03s (took 0.1s) [14/20] Running N= 263665, random, radix_bits=1... mean=2.723e-02s (took 0.5s) [14/20] Running N= 263665, random, radix_bits=2... mean=1.341e-02s (took 0.3s) [14/20] Running N= 263665, random, radix_bits=4... mean=6.824e-03s (took 0.1s) [14/20] Running N= 263665, random, radix_bits=8... mean=3.961e-03s (took 0.1s) [15/20] Running N= 483293, zeros, radix_bits=1... mean=5.050e-02s (took 1.0s) [15/20] Running N= 483293, zeros, radix_bits=2... mean=2.435e-02s (took 0.5s) [15/20] Running N= 483293, zeros, radix_bits=4... mean=1.220e-02s (took 0.2s) [15/20] Running N= 483293, zeros, radix_bits=8... mean=6.274e-03s (took 0.1s) [15/20] Running N= 483293, shuffled iota, radix_bits=1... mean=5.012e-02s (took 1.0s) [15/20] Running N= 483293, shuffled iota, radix_bits=2... mean=2.460e-02s (took 0.5s) [15/20] Running N= 483293, shuffled iota, radix_bits=4... mean=1.263e-02s (took 0.3s) [15/20] Running N= 483293, shuffled iota, radix_bits=8... mean=6.683e-03s (took 0.1s) [15/20] Running N= 483293, random, radix_bits=1... mean=4.990e-02s (took 1.0s) [15/20] Running N= 483293, random, radix_bits=2... mean=2.434e-02s (took 0.5s) [15/20] Running N= 483293, random, radix_bits=4... mean=1.248e-02s (took 0.3s) [15/20] Running N= 483293, random, radix_bits=8... mean=7.094e-03s (took 0.1s) [16/20] Running N= 885866, zeros, radix_bits=1... mean=9.253e-02s (took 1.9s) [16/20] Running N= 885866, zeros, radix_bits=2... mean=4.495e-02s (took 0.9s) [16/20] Running N= 885866, zeros, radix_bits=4... mean=2.244e-02s (took 0.4s) [16/20] Running N= 885866, zeros, radix_bits=8... mean=1.145e-02s (took 0.2s) [16/20] Running N= 885866, shuffled iota, radix_bits=1... mean=9.203e-02s (took 1.8s) [16/20] Running N= 885866, shuffled iota, radix_bits=2... mean=4.523e-02s (took 0.9s) [16/20] Running N= 885866, shuffled iota, radix_bits=4... mean=2.303e-02s (took 0.5s) [16/20] Running N= 885866, shuffled iota, radix_bits=8... mean=1.226e-02s (took 0.2s) [16/20] Running N= 885866, random, radix_bits=1... mean=9.208e-02s (took 1.8s) [16/20] Running N= 885866, random, radix_bits=2... mean=4.509e-02s (took 0.9s) [16/20] Running N= 885866, random, radix_bits=4... mean=2.284e-02s (took 0.5s) [16/20] Running N= 885866, random, radix_bits=8... mean=1.284e-02s (took 0.3s) [17/20] Running N= 1623776, zeros, radix_bits=1... mean=1.707e-01s (took 3.4s) [17/20] Running N= 1623776, zeros, radix_bits=2... mean=8.322e-02s (took 1.7s) [17/20] Running N= 1623776, zeros, radix_bits=4... mean=4.143e-02s (took 0.8s) [17/20] Running N= 1623776, zeros, radix_bits=8... mean=2.097e-02s (took 0.4s) [17/20] Running N= 1623776, shuffled iota, radix_bits=1... mean=1.704e-01s (took 3.4s) [17/20] Running N= 1623776, shuffled iota, radix_bits=2... mean=8.458e-02s (took 1.7s) [17/20] Running N= 1623776, shuffled iota, radix_bits=4... mean=4.340e-02s (took 0.9s) [17/20] Running N= 1623776, shuffled iota, radix_bits=8... mean=2.279e-02s (took 0.5s) [17/20] Running N= 1623776, random, radix_bits=1... mean=1.692e-01s (took 3.4s) [17/20] Running N= 1623776, random, radix_bits=2... mean=8.391e-02s (took 1.7s) [17/20] Running N= 1623776, random, radix_bits=4... mean=4.306e-02s (took 0.9s) [17/20] Running N= 1623776, random, radix_bits=8... mean=2.356e-02s (took 0.5s) [18/20] Running N= 2976351, zeros, radix_bits=1... mean=3.364e-01s (took 4.0s) [18/20] Running N= 2976351, zeros, radix_bits=2... mean=1.639e-01s (took 3.3s) [18/20] Running N= 2976351, zeros, radix_bits=4... mean=8.186e-02s (took 1.6s) [18/20] Running N= 2976351, zeros, radix_bits=8... mean=4.153e-02s (took 0.8s) [18/20] Running N= 2976351, shuffled iota, radix_bits=1... mean=3.287e-01s (took 4.6s) [18/20] Running N= 2976351, shuffled iota, radix_bits=2... mean=1.654e-01s (took 3.3s) [18/20] Running N= 2976351, shuffled iota, radix_bits=4... mean=8.461e-02s (took 1.7s) [18/20] Running N= 2976351, shuffled iota, radix_bits=8... mean=4.334e-02s (took 0.9s) [18/20] Running N= 2976351, random, radix_bits=1... mean=3.283e-01s (took 4.6s) [18/20] Running N= 2976351, random, radix_bits=2... mean=1.646e-01s (took 3.3s) [18/20] Running N= 2976351, random, radix_bits=4... mean=8.373e-02s (took 1.7s) [18/20] Running N= 2976351, random, radix_bits=8... mean=4.453e-02s (took 0.9s) [19/20] Running N= 5455594, zeros, radix_bits=1... mean=5.854e-01s (took 4.7s) [19/20] Running N= 5455594, zeros, radix_bits=2... mean=2.840e-01s (took 4.5s) [19/20] Running N= 5455594, zeros, radix_bits=4... mean=1.453e-01s (took 2.9s) [19/20] Running N= 5455594, zeros, radix_bits=8... mean=7.340e-02s (took 1.5s) [19/20] Running N= 5455594, shuffled iota, radix_bits=1... mean=5.845e-01s (took 4.7s) [19/20] Running N= 5455594, shuffled iota, radix_bits=2... mean=2.898e-01s (took 4.1s) [19/20] Running N= 5455594, shuffled iota, radix_bits=4... mean=1.503e-01s (took 3.0s) [19/20] Running N= 5455594, shuffled iota, radix_bits=8... mean=7.531e-02s (took 1.5s) [19/20] Running N= 5455594, random, radix_bits=1... mean=5.992e-01s (took 4.8s) [19/20] Running N= 5455594, random, radix_bits=2... mean=3.214e-01s (took 4.5s) [19/20] Running N= 5455594, random, radix_bits=4... mean=1.658e-01s (took 3.3s) [19/20] Running N= 5455594, random, radix_bits=8... mean=8.269e-02s (took 1.7s) [20/20] Running N=10000000, zeros, radix_bits=1... mean=1.075e+00s (took 4.3s) [20/20] Running N=10000000, zeros, radix_bits=2... mean=5.287e-01s (took 4.2s) [20/20] Running N=10000000, zeros, radix_bits=4... mean=2.680e-01s (took 4.3s) [20/20] Running N=10000000, zeros, radix_bits=8... mean=1.342e-01s (took 2.7s) [20/20] Running N=10000000, shuffled iota, radix_bits=1... mean=1.081e+00s (took 4.3s) [20/20] Running N=10000000, shuffled iota, radix_bits=2... mean=5.379e-01s (took 4.3s) [20/20] Running N=10000000, shuffled iota, radix_bits=4... mean=2.762e-01s (took 4.4s) [20/20] Running N=10000000, shuffled iota, radix_bits=8... mean=1.359e-01s (took 2.7s) [20/20] Running N=10000000, random, radix_bits=1... mean=1.070e+00s (took 4.3s) [20/20] Running N=10000000, random, radix_bits=2... mean=5.340e-01s (took 4.3s) [20/20] Running N=10000000, random, radix_bits=4... mean=2.738e-01s (took 4.4s) [20/20] Running N=10000000, random, radix_bits=8... mean=1.352e-01s (took 2.7s) .. GENERATED FROM PYTHON SOURCE LINES 149-150 Plot the digit histogram performance benchmarks, one figure per key distribution .. GENERATED FROM PYTHON SOURCE LINES 150-184 .. code-block:: Python :lineno-start: 151 color_cycle = plt.rcParams["axes.prop_cycle"].by_key()["color"] def before_plot(ax_plot): Nobj = np.array(particle_counts) Time1G = Nobj / 1e9 ax_plot.plot( particle_counts, Time1G, color="grey", linestyle="-", alpha=0.7, label="1G obj/sec" ) for case in key_cases: plot_data = {} for i, radix_bits in enumerate(radix_bits_list): plot_data[f"radix_bits={radix_bits}"] = { "x": particle_counts, "y": results[case][radix_bits], "color": color_cycle[i % len(color_cycle)], "label": f"radix_bits={radix_bits} (u32)", "linestyle": "--", "marker": ".", } make_std_bench_plot( plot_data, xlabel="Number of elements", ylabel="Time (s)", title=f"digit histogram performance benchmarks ({case} keys)", end_label_fmt=lambda y: f"{y:.2e} s", before_plot_func=before_plot, ) plt.show() .. rst-class:: sphx-glr-horizontal * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_001.png :alt: digit histogram performance benchmarks (zeros keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_001.png :class: sphx-glr-single-img * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_002.png :alt: digit histogram performance benchmarks (shuffled iota keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_002.png :class: sphx-glr-single-img * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_003.png :alt: digit histogram performance benchmarks (random keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 185-188 Plot the digit histogram performance benchmarks (bandwidth), one figure per key distribution. The keys are read only once whatever the number of digit places, the histogram itself being negligible in size. .. GENERATED FROM PYTHON SOURCE LINES 188-213 .. code-block:: Python :lineno-start: 189 for case in key_cases: plot_data = {} for i, radix_bits in enumerate(radix_bits_list): Nobj = np.array(particle_counts) Bytes = 4 * Nobj # 1 u32 key read per element (sizeof = 4) BW = Bytes / np.array(results[case][radix_bits]) plot_data[f"radix_bits={radix_bits}"] = { "x": particle_counts, "y": BW, "color": color_cycle[i % len(color_cycle)], "label": f"radix_bits={radix_bits} (u32)", "linestyle": "--", "marker": ".", } make_std_bench_plot( plot_data, xlabel="Number of elements", ylabel="Bandwidth (B.s^-1)", title=f"digit histogram performance benchmarks ({case} keys)", end_label_fmt=lambda y: f"{y / 1e9:.2f} GB.s^-1", ) plt.show() .. rst-class:: sphx-glr-horizontal * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_004.png :alt: digit histogram performance benchmarks (zeros keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_004.png :class: sphx-glr-single-img * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_005.png :alt: digit histogram performance benchmarks (shuffled iota keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_005.png :class: sphx-glr-single-img * .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_006.png :alt: digit histogram performance benchmarks (random keys) :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 214-215 Plot the histograms of the 4 digit places of 8 bits for each key distribution .. GENERATED FROM PYTHON SOURCE LINES 215-230 .. code-block:: Python :lineno-start: 216 N_plot = 1000000 fig, axs = plt.subplots(1, len(key_cases), figsize=(15, 4), layout="constrained") for ax, case in zip(axs, key_cases): hist = compute_digit_histogram(make_keys(case, N_plot), 8) for p in range(4): ax.plot(hist[p * 256 : (p + 1) * 256], label=f"digit place {p} (bits {8 * p}-{8 * p + 7})") ax.set_xlabel("digit value") ax.set_ylabel("count") ax.set_yscale("symlog") ax.set_title(f"{case} keys") axs[0].legend() fig.suptitle(f"digit histogram (radix_bits=8, {N_plot} u32 keys)") plt.show() .. image-sg:: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_007.png :alt: digit histogram (radix_bits=8, 1000000 u32 keys), zeros keys, shuffled iota keys, random keys :srcset: /_as_gen/benchmarks/images/sphx_glr_run_digit_histogram_performance_007.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (2 minutes 46.277 seconds) **Estimated memory usage:** 540 MB .. _sphx_glr_download__as_gen_benchmarks_run_digit_histogram_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: run_digit_histogram_performance.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: run_digit_histogram_performance.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: run_digit_histogram_performance.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_