-
Notifications
You must be signed in to change notification settings - Fork 484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to implement ParallelIterator for a custom Range? #1086
Comments
The smallest entry point for writing your own kind of parallel iterator is to use |
Ok, but what do i do with it next? |
Sure, you can use a regular iterator within |
Ok, I changed this: use rayon::iter::{IntoParallelIterator, ParallelIterator};
#[derive(Debug, Clone)]
pub struct NumberHash<N, H> {
pub number: N,
pub hash: H,
}
pub type Number = u128;
pub fn gen_range_of_nums<H, F, A, R>(start: Number, end: Number, filter: F, apply: A)
where
F: Fn(Number) -> Option<NumberHash<Number, H>> + Sync + Send,
A: Fn(NumberHash<Number, H>) -> R + Sync + Send,
{
(start..=end).into_par_iter().for_each(|number| {
if let Some(num_hash) = filter(number) {
apply(num_hash);
}
})
} to this: use rayon::iter::{IntoParallelIterator, ParallelIterator};
use std::ops::{Add, RangeInclusive};
#[derive(Debug, Clone)]
pub struct NumberHash<N, H> {
pub number: N,
pub hash: H,
}
pub type Number = u128;
fn splitter<N>(data: RangeInclusive<N>) -> (RangeInclusive<N>, Option<RangeInclusive<N>>)
where
N: num_traits::Num + Copy,
{
let (start, end) = (*data.start(), *data.end());
let middle = end.div(N::one() + N::one());
let next = middle + N::one();
(start..=middle, Some(next..=end))
}
pub fn gen_range_of_nums<N, H, F, A, R>(start: N, end: N, filter: F, apply: A)
where
N: num_traits::Num + num_traits::ToPrimitive + Add<N, Output = N> + PartialOrd + Copy + Send,
F: Fn(N) -> Option<NumberHash<N, H>> + Sync + Send,
A: Fn(NumberHash<N, H>) -> R + Sync + Send,
{
rayon::iter::split(start..=end, splitter)
.into_par_iter()
.for_each(|sub_range| {
let (start, end) = (*sub_range.start(), *sub_range.end());
num_iter::range_inclusive(start, end).for_each(|number| {
if let Some(num_hash) = filter(number) {
apply(num_hash);
}
})
})
} Works fine, haven't notice any performance degradation. Is it OK that my splitter may return an emoty range? |
I expect these are related. If you keep splitting, then at the tail end of computation you'll have threads racing to steal each others' empty ranges, and stealing also triggers the heuristic to reset its split count and try to split even more. With a terminating case that returns |
Yes, you are right. |
Can you share your implementation, with enough for someone to run and reproduce that result? |
Sure, the code is here. |
Oh, this isn't a correct |
Yeah, makes sense.
I wrote some tests to compare fixed and unfixed version. I was able to bench the two versions versus each other (old ver usually overflows the stack, but sometimes it works, just with terrible performance) and new version if far better. However, benchmarks show that the two 'middle' lines above do not differ in performance (even though this one line changes so much in the end). Of course, this has nothing to do with |
Update: some of the info is incorrect, I'll check again. |
OK, analyzing So now I'm completely lost. |
No, wait I do see the difference when I compare |
Hello!
I want to iterate over a range of
T
s that implementNum
fromnum_traits
crate.rayon
implementsParallelIterator
for ranges using privateTraits
andmacros
, but I'm not very familiar with macros and they use integer types anyway.I wasn't able examples or tutorials for my use case.
Thank you
The text was updated successfully, but these errors were encountered: