New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Iterable to support streaming #38
Merged
Merged
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
f7e4c8e
add iterable
huyuncong f85dff8
Update src/iterable/mod.rs
mmaker 8c42cdf
Add reference to a paper that will be published soon.
mmaker 81008b1
Marker commit for authorship.
mmaker 6938144
Update src/iterable/mod.rs
weikengchen 20aeb1f
Update src/iterable/mod.rs
weikengchen 8676c4f
Update src/iterable/mod.rs
mmaker 8ccaf55
Update src/iterable/mod.rs
mmaker 3a16ede
Update src/iterable/mod.rs
mmaker 261ac34
Update src/iterable/mod.rs
mmaker ab3c4c0
Update src/iterable/mod.rs
mmaker File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,106 @@ | ||||||
//! A base library for interfacing with streams of vectors and matrices. | ||||||
//! | ||||||
//! This library presents the abstraction layer for the _streaming model_. | ||||||
//! Essentially, it provides a set of handy utilities as a wrapper around | ||||||
//! iterators. | ||||||
|
||||||
/// The trait [`Iterable`] represents a streamable object that can produce | ||||||
/// an arbitrary number of streams of length [`Iterable::len`](Iterable::len). | ||||||
/// | ||||||
/// An Iterable is pretty much like an [`IntoIterator`] that can be copied over | ||||||
/// and over, and has an hint of the length. | ||||||
/// In other words, these two functions are pretty much equivalent. | ||||||
/// | ||||||
/// # Examples | ||||||
/// | ||||||
/// ``` | ||||||
/// use ark_std::borrow::Borrow; | ||||||
/// use ark_std::iterable::Iterable; | ||||||
/// | ||||||
/// // Relying only on standard library | ||||||
/// fn f(xs: impl IntoIterator<Item=impl Borrow<u32>> + Clone) -> u32 { | ||||||
/// xs.clone().into_iter().fold(1, |x, y| x.borrow() * y.borrow()) + | ||||||
/// xs.clone().into_iter().fold(0, |x, y| x.borrow() + y.borrow()) + | ||||||
/// xs.into_iter().size_hint().0 as u32 | ||||||
/// } | ||||||
/// | ||||||
/// // Relying on the trait below | ||||||
/// fn g(xs: impl Iterable<Item=impl Borrow<u32>>) -> u32 { | ||||||
/// xs.iter().fold(1, |x, y| x.borrow() * y.borrow()) + | ||||||
/// xs.iter().fold(0, |x, y| x.borrow() + y.borrow()) + | ||||||
/// xs.len() as u32 | ||||||
/// } | ||||||
/// | ||||||
/// // Test over a slice (which implements both traits). | ||||||
/// let xs = &[1, 2, 3, 4]; | ||||||
/// assert_eq!(f(xs), g(xs)); | ||||||
/// ``` | ||||||
/// | ||||||
/// # Efficency | ||||||
/// | ||||||
/// For efficiency, functions using iterables are often times relying on | ||||||
/// [`Borrow`](std::borrow::Borrow) in order to avoid copying the entire items. | ||||||
weikengchen marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// | ||||||
/// The `Iter` associated type has a lifetime that is independent from that of the | ||||||
/// [`Iterable`] object. This means that implicitly a copy of the relevant | ||||||
/// contents of the object will happen whenever | ||||||
/// [`Iterable::iter`](crate::iterable::Iterable::iter) is called. This might | ||||||
/// change in the future as associated type constructors [[RFC1598](https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md#declaring--assigning-an-associated-type-constructor)] | ||||||
/// stabilize. | ||||||
/// | ||||||
/// # Future implementation | ||||||
/// | ||||||
/// A lot of stream operations must be performed symbolycally. | ||||||
mmaker marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// We expect that, in the future, this trait will accomodate for additional | ||||||
mmaker marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// streaming function, e.g. `Iterable::hadamard(&self, other: &Iterable)` to | ||||||
/// perform the hadamard product of two streams, or `Iterable::add(&self, other: | ||||||
mmaker marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// &Iterable)` to perform the addition of two streams. | ||||||
pub trait Iterable { | ||||||
/// The type of the element being streamed. | ||||||
type Item; | ||||||
/// The type of the iterator being generated. | ||||||
type Iter: Iterator<Item = Self::Item>; | ||||||
|
||||||
/// Returns the iterator associated to the current instance. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
mmaker marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// | ||||||
/// In the so-called _streaming model_ [BCHO22], this is equivalent to | ||||||
/// instantiating a new stream tape. | ||||||
/// For base types, this acts in the same way as the `.iter()` method. | ||||||
/// | ||||||
/// ``` | ||||||
/// use ark_std::iterable::Iterable; | ||||||
/// | ||||||
/// let x = &[1, 2, 4]; | ||||||
/// let mut iterator = x.iter(); | ||||||
/// ``` | ||||||
fn iter(&self) -> Self::Iter; | ||||||
|
||||||
/// Returns a hint on the length of the stream. | ||||||
mmaker marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
/// | ||||||
/// Careful: different objects might have different indications of what | ||||||
/// _length_ means; this might not be the actual size in terms of | ||||||
/// elements. | ||||||
fn len(&self) -> usize; | ||||||
|
||||||
/// Return `true` if the stream is empty, else `false`. | ||||||
fn is_empty(&self) -> bool { | ||||||
self.len() == 0 | ||||||
} | ||||||
} | ||||||
|
||||||
impl<I> Iterable for I | ||||||
where | ||||||
I: IntoIterator + Copy, | ||||||
I::IntoIter: ExactSizeIterator, | ||||||
{ | ||||||
type Item = <I as IntoIterator>::Item; | ||||||
type Iter = <I as IntoIterator>::IntoIter; | ||||||
|
||||||
fn iter(&self) -> Self::Iter { | ||||||
self.into_iter() | ||||||
} | ||||||
|
||||||
fn len(&self) -> usize { | ||||||
self.into_iter().len() | ||||||
} | ||||||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This sounds like the entire Iterable trait is not necessary, and it is the duty for Gemini to not use Iterable but instead to use the standard library one.
What is the main difference that Iterable provides but not by
IntoIterator
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair point: the main difference is that we need to iter over the same stream multiple times, by design.
This means that already we require
IntoIterator + Copy
orIntoIterator + Clone
if we want to go down that avenue.From there, it's pretty much the same, except that now to get the length of a stream it becomes a bit more verbose, and the length function might disagree with the actual length (this is needed for instance when streaming a matrix).
Here's a bit of an overview of how the code might differ: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=c7d9d2fca7d1d308601bd5b32d770409
To be fair, on the one hand, the downside is that
Iterable::iter
will have to copy all relevant data from the initial structure (until https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md hits the streets). On the other hand, for the long-term vision we have a trait that can be extended with other functions on the top of normal iterables, and that we skip having to type things likevec.into_iter().len()
.