Shamrock 2025.10.0
Astrophysical Code
Loading...
Searching...
No Matches
shamalgs::primitives::impl Namespace Reference

namespace to control implementation behavior More...

Classes

struct  Reference
 CPU reference implementation, computed on a host copy of the buffers. More...
struct  NaiveGpu
 Portable GPU kernel, one work-item per bin. More...
struct  GpuTeamFetching
 GPU kernel using a team-local cache to coalesce input reads. More...
struct  GpuOversubscribe
 GPU kernel oversubscribing a work-group per bin, reducing locally. More...
struct  Host
 Check all elements on host after copying the buffer back. More...
struct  SumReduction
 Check all elements via a sum reduction on device. More...
struct  AtomicEarlyExit
 Check all elements via a sum reduction on device. More...
struct  Fallback
 Fallback USM reduction (portable, no group reduction support required). More...
struct  StdScan
 std::exclusive_scan on a host copy of the buffer (portable fallback) More...
struct  LocalInsertionSort
 Sort each segment locally with an insertion sort, one kernel work-item per segment. More...
struct  MultiStdSort
 Copy back to host and sort each segment with std::sort, parallelized over OpenMP. More...
struct  BitonicSort
 Bitonic sort, updated USM kernel (see bitonicSort_updated_usm.hpp). More...
struct  StdSort
 Copy the buffers to host, std::sort the zipped key/value pairs, and copy back. More...
struct  LsdRadixSortBasic
struct  BatcherOddEvenHostSerial
 Copy the buffers to host, sort with Batcher's odd-even merge sort, and copy back. More...
struct  BatcherOddEven
 Copy the buffers to host, sort with Batcher's odd-even merge sort, and copy back. More...

Typedefs

using ComputeHistogramImpl = shamalgs::ImplVariantGlobal<Reference, NaiveGpu, GpuTeamFetching, GpuOversubscribe>
 Implementation selector type for compute_histogram.

Enumerations

enum class  MaxStencilSize : u32 { Size16 = 16 , Size32 = 32 }

Functions

template<class T, class Tbins, class... Targs, class Tfunctor>
void compute_histogram_reference (const sham::DeviceBuffer< Tbins > &bin_edge_inf, const sham::DeviceBuffer< Tbins > &bin_edge_sup, size_t nbins, size_t element_count, Tfunctor &&functor, sham::DeviceBuffer< T > &result, const sham::DeviceBuffer< Targs > &...input_data)
template<class T, class Tbins, class... Targs, class Tfunctor>
void compute_histogram_naive_gpu (const sham::DeviceScheduler_ptr &dev_sched, const sham::DeviceBuffer< Tbins > &bin_edge_inf, const sham::DeviceBuffer< Tbins > &bin_edge_sup, size_t nbins, size_t element_count, Tfunctor &&functor, sham::DeviceBuffer< T > &result, const sham::DeviceBuffer< Targs > &...input_data)
template<class T, class Tbins, class... Targs, class Tfunctor>
void compute_histogram_gpu_team_fetching (const sham::DeviceScheduler_ptr &dev_sched, const sham::DeviceBuffer< Tbins > &bin_edge_inf, const sham::DeviceBuffer< Tbins > &bin_edge_sup, size_t nbins, size_t element_count, Tfunctor &&functor, sham::DeviceBuffer< T > &result, const sham::DeviceBuffer< Targs > &...input_data)
template<class T, class Tbins, class... Targs, class Tfunctor>
void compute_histogram_gpu_oversubscribe (const sham::DeviceScheduler_ptr &dev_sched, u32 group_size, const sham::DeviceBuffer< Tbins > &bin_edge_inf, const sham::DeviceBuffer< Tbins > &bin_edge_sup, size_t nbins, size_t element_count, Tfunctor &&functor, sham::DeviceBuffer< T > &result, const sham::DeviceBuffer< Targs > &...input_data)
 SHAMALGS_REGISTER_IMPL (compute_histogram_impl_name, compute_histogram_impl)
 SHAMALGS_REGISTER_IMPL (is_all_true_impl_name, is_all_true_impl)
 SHAMALGS_REGISTER_IMPL (reduction_impl_name, reduction_impl)
 SHAMALGS_REGISTER_IMPL (scan_exclusive_sum_in_place_impl_name, scan_exclusive_sum_in_place_impl)
 SHAMALGS_REGISTER_IMPL (segmented_sort_in_place_impl_name, segmented_sort_in_place_impl)
 SHAMALGS_REGISTER_IMPL (sort_by_key_pow2_len_impl_name, sort_by_key_pow2_len_impl)
template<class Tkey, class Tval>
void sort_by_key_pow2_len_bitonic_dispatch (const sham::DeviceScheduler_ptr &sched, sham::DeviceBuffer< Tkey > &buf_key, sham::DeviceBuffer< Tval > &buf_values, u32 len, MaxStencilSize stencil_size)
 SHAMALGS_REGISTER_IMPL (sort_by_keys_impl_name, sort_by_keys_impl)

Variables

ComputeHistogramImpl compute_histogram_impl
 Implementation selector for compute_histogram (defined in compute_histogram.cpp).
constexpr std::string_view compute_histogram_impl_name = "compute_histogram"
 Registry name, shared by its registration and the dispatch site(s).
constexpr std::string_view is_all_true_impl_name = "is_all_true"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< Host, SumReduction, AtomicEarlyExit > is_all_true_impl
constexpr std::string_view reduction_impl_name = "reduction"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< Fallback > reduction_impl
constexpr std::string_view scan_exclusive_sum_in_place_impl_name = "scan_exclusive_sum_in_place"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< StdScan > scan_exclusive_sum_in_place_impl
constexpr std::string_view segmented_sort_in_place_impl_name = "segmented_sort_in_place"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< LocalInsertionSort, MultiStdSort > segmented_sort_in_place_impl
constexpr std::string_view sort_by_key_pow2_len_impl_name = "sort_by_key_pow2_len"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< BitonicSort, StdSort, LsdRadixSortBasic > sort_by_key_pow2_len_impl
constexpr std::string_view sort_by_keys_impl_name = "sort_by_keys"
 Registry name, shared by its registration and the dispatch site(s).
shamalgs::ImplVariantGlobal< StdSort, BatcherOddEvenHostSerial, BatcherOddEven, LsdRadixSortBasic > sort_by_keys_impl

Detailed Description

namespace to control implementation behavior

Typedef Documentation

◆ ComputeHistogramImpl

Implementation selector type for compute_histogram.

Definition at line 57 of file compute_histogram.hpp.

Enumeration Type Documentation

◆ MaxStencilSize

Max stencil size (tile width) used by the updated USM bitonic sort kernel Sizes 2, 4 and 8 are currently disabled (kept for easy re-enabling if needed)

Definition at line 31 of file sort_by_key_pow2_len.cpp.

Function Documentation

◆ compute_histogram_gpu_oversubscribe()

template<class T, class Tbins, class... Targs, class Tfunctor>
void shamalgs::primitives::impl::compute_histogram_gpu_oversubscribe ( const sham::DeviceScheduler_ptr & dev_sched,
u32 group_size,
const sham::DeviceBuffer< Tbins > & bin_edge_inf,
const sham::DeviceBuffer< Tbins > & bin_edge_sup,
size_t nbins,
size_t element_count,
Tfunctor && functor,
sham::DeviceBuffer< T > & result,
const sham::DeviceBuffer< Targs > &... input_data )
inline

Definition at line 241 of file compute_histogram.hpp.

◆ compute_histogram_gpu_team_fetching()

template<class T, class Tbins, class... Targs, class Tfunctor>
void shamalgs::primitives::impl::compute_histogram_gpu_team_fetching ( const sham::DeviceScheduler_ptr & dev_sched,
const sham::DeviceBuffer< Tbins > & bin_edge_inf,
const sham::DeviceBuffer< Tbins > & bin_edge_sup,
size_t nbins,
size_t element_count,
Tfunctor && functor,
sham::DeviceBuffer< T > & result,
const sham::DeviceBuffer< Targs > &... input_data )
inline

Definition at line 148 of file compute_histogram.hpp.

◆ compute_histogram_naive_gpu()

template<class T, class Tbins, class... Targs, class Tfunctor>
void shamalgs::primitives::impl::compute_histogram_naive_gpu ( const sham::DeviceScheduler_ptr & dev_sched,
const sham::DeviceBuffer< Tbins > & bin_edge_inf,
const sham::DeviceBuffer< Tbins > & bin_edge_sup,
size_t nbins,
size_t element_count,
Tfunctor && functor,
sham::DeviceBuffer< T > & result,
const sham::DeviceBuffer< Targs > &... input_data )
inline

Definition at line 109 of file compute_histogram.hpp.

◆ compute_histogram_reference()

template<class T, class Tbins, class... Targs, class Tfunctor>
void shamalgs::primitives::impl::compute_histogram_reference ( const sham::DeviceBuffer< Tbins > & bin_edge_inf,
const sham::DeviceBuffer< Tbins > & bin_edge_sup,
size_t nbins,
size_t element_count,
Tfunctor && functor,
sham::DeviceBuffer< T > & result,
const sham::DeviceBuffer< Targs > &... input_data )
inline

Definition at line 66 of file compute_histogram.hpp.

◆ sort_by_key_pow2_len_bitonic_dispatch()

template<class Tkey, class Tval>
void shamalgs::primitives::impl::sort_by_key_pow2_len_bitonic_dispatch ( const sham::DeviceScheduler_ptr & sched,
sham::DeviceBuffer< Tkey > & buf_key,
sham::DeviceBuffer< Tval > & buf_values,
u32 len,
MaxStencilSize stencil_size )

Dispatch to the updated USM bitonic sort kernel, picking its MaxStencilSize non-type template parameter at runtime from the enum value stored in BitonicSort

Definition at line 122 of file sort_by_key_pow2_len.cpp.

Here is the call graph for this function:

Variable Documentation

◆ compute_histogram_impl

ComputeHistogramImpl shamalgs::primitives::impl::compute_histogram_impl
Initial value:
{
[](const sham::DeviceScheduler_ptr &dev_sched, auto &self) {
if (dev_sched->ctx->device->prop.type == sham::DeviceType::GPU) {
self.set(GpuOversubscribe{});
} else {
self.set(NaiveGpu{});
}
}}
GPU kernel oversubscribing a work-group per bin, reducing locally.

Implementation selector for compute_histogram (defined in compute_histogram.cpp).

Definition at line 21 of file compute_histogram.cpp.

◆ compute_histogram_impl_name

std::string_view shamalgs::primitives::impl::compute_histogram_impl_name = "compute_histogram"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 63 of file compute_histogram.hpp.

◆ is_all_true_impl

shamalgs::ImplVariantGlobal<Host, SumReduction, AtomicEarlyExit> shamalgs::primitives::impl::is_all_true_impl
Initial value:
{
[](const sham::DeviceScheduler_ptr &, auto &self) {
self.set(Host{});
}}
Check all elements on host after copying the buffer back.

Definition at line 212 of file is_all_true.cpp.

◆ is_all_true_impl_name

std::string_view shamalgs::primitives::impl::is_all_true_impl_name = "is_all_true"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 210 of file is_all_true.cpp.

◆ reduction_impl

shamalgs::ImplVariantGlobal< Fallback > shamalgs::primitives::impl::reduction_impl
Initial value:
{[](const sham::DeviceScheduler_ptr &, auto &self) {
self.set(Fallback{});
}}
Fallback USM reduction (portable, no group reduction support required).
Definition reduction.cpp:34

Definition at line 88 of file reduction.cpp.

◆ reduction_impl_name

std::string_view shamalgs::primitives::impl::reduction_impl_name = "reduction"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 79 of file reduction.cpp.

◆ scan_exclusive_sum_in_place_impl

shamalgs::ImplVariantGlobal< StdScan > shamalgs::primitives::impl::scan_exclusive_sum_in_place_impl
Initial value:
{[](const sham::DeviceScheduler_ptr &, auto &self) {
self.set(StdScan{});
}}
std::exclusive_scan on a host copy of the buffer (portable fallback)

Definition at line 159 of file scan_exclusive_sum_in_place.cpp.

◆ scan_exclusive_sum_in_place_impl_name

std::string_view shamalgs::primitives::impl::scan_exclusive_sum_in_place_impl_name = "scan_exclusive_sum_in_place"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 141 of file scan_exclusive_sum_in_place.cpp.

◆ segmented_sort_in_place_impl

shamalgs::ImplVariantGlobal<LocalInsertionSort, MultiStdSort> shamalgs::primitives::impl::segmented_sort_in_place_impl
Initial value:
{
[](const sham::DeviceScheduler_ptr &, auto &self) {
self.set(MultiStdSort{});
}}
Copy back to host and sort each segment with std::sort, parallelized over OpenMP.

Definition at line 125 of file segmented_sort_in_place.cpp.

◆ segmented_sort_in_place_impl_name

std::string_view shamalgs::primitives::impl::segmented_sort_in_place_impl_name = "segmented_sort_in_place"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 123 of file segmented_sort_in_place.cpp.

◆ sort_by_key_pow2_len_impl

shamalgs::ImplVariantGlobal<BitonicSort, StdSort, LsdRadixSortBasic> shamalgs::primitives::impl::sort_by_key_pow2_len_impl
Initial value:
{[](const sham::DeviceScheduler_ptr &dev_sched, auto &self) {
if (dev_sched->ctx->device->prop.type == sham::DeviceType::GPU) {
self.set(BitonicSort{});
} else {
self.set(LsdRadixSortBasic{});
}
}}
Bitonic sort, updated USM kernel (see bitonicSort_updated_usm.hpp).

Definition at line 108 of file sort_by_key_pow2_len.cpp.

◆ sort_by_key_pow2_len_impl_name

std::string_view shamalgs::primitives::impl::sort_by_key_pow2_len_impl_name = "sort_by_key_pow2_len"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 105 of file sort_by_key_pow2_len.cpp.

◆ sort_by_keys_impl

shamalgs:: ImplVariantGlobal<StdSort, BatcherOddEvenHostSerial, BatcherOddEven, LsdRadixSortBasic> shamalgs::primitives::impl::sort_by_keys_impl
Initial value:
{[](const sham::DeviceScheduler_ptr &dev_sched, auto &self) {
if (dev_sched->ctx->device->prop.type == sham::DeviceType::CPU) {
self.set(LsdRadixSortBasic{});
} else {
self.set(StdSort{});
}
}}

Definition at line 85 of file sort_by_keys.cpp.

◆ sort_by_keys_impl_name

std::string_view shamalgs::primitives::impl::sort_by_keys_impl_name = "sort_by_keys"
constexpr

Registry name, shared by its registration and the dispatch site(s).

Definition at line 81 of file sort_by_keys.cpp.