Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iterable to support streaming #38

Merged
merged 11 commits into from Jan 14, 2022
106 changes: 106 additions & 0 deletions src/iterable/mod.rs
@@ -0,0 +1,106 @@
//! A base library for interfacing with streams of vectors and matrices.
//!
//! This library presents the abstraction layer for the _streaming model_.
//! Essentially, it provides a set of handy utilities as a wrapper around
//! iterators.

/// The trait [`Iterable`] represents a streamable object that can produce
/// an arbitrary number of streams of length [`Iterable::len`](Iterable::len).
///
/// An Iterable is pretty much like an [`IntoIterator`] that can be copied over
/// and over, and has an hint of the length.
/// In other words, these two functions are pretty much equivalent.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like the entire Iterable trait is not necessary, and it is the duty for Gemini to not use Iterable but instead to use the standard library one.

What is the main difference that Iterable provides but not by IntoIterator?

Copy link
Member

@mmaker mmaker Jan 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point: the main difference is that we need to iter over the same stream multiple times, by design.
This means that already we require IntoIterator + Copy or IntoIterator + Clone if we want to go down that avenue.

From there, it's pretty much the same, except that now to get the length of a stream it becomes a bit more verbose, and the length function might disagree with the actual length (this is needed for instance when streaming a matrix).
Here's a bit of an overview of how the code might differ: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=c7d9d2fca7d1d308601bd5b32d770409

To be fair, on the one hand, the downside is that Iterable::iter will have to copy all relevant data from the initial structure (until https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md hits the streets). On the other hand, for the long-term vision we have a trait that can be extended with other functions on the top of normal iterables, and that we skip having to type things like vec.into_iter().len().

///
/// # Examples
///
/// ```
/// use ark_std::borrow::Borrow;
/// use ark_std::iterable::Iterable;
///
/// // Relying only on standard library
/// fn f(xs: impl IntoIterator<Item=impl Borrow<u32>> + Clone) -> u32 {
/// xs.clone().into_iter().fold(1, |x, y| x.borrow() * y.borrow()) +
/// xs.clone().into_iter().fold(0, |x, y| x.borrow() + y.borrow()) +
/// xs.into_iter().size_hint().0 as u32
/// }
///
/// // Relying on the trait below
/// fn g(xs: impl Iterable<Item=impl Borrow<u32>>) -> u32 {
/// xs.iter().fold(1, |x, y| x.borrow() * y.borrow()) +
/// xs.iter().fold(0, |x, y| x.borrow() + y.borrow()) +
/// xs.len() as u32
/// }
///
/// // Test over a slice (which implements both traits).
/// let xs = &[1, 2, 3, 4];
/// assert_eq!(f(xs), g(xs));
/// ```
///
/// # Efficency
///
/// For efficiency, functions using iterables are often times relying on
/// [`Borrow`](std::borrow::Borrow) in order to avoid copying the entire items.
weikengchen marked this conversation as resolved.
Show resolved Hide resolved
///
/// The `Iter` associated type has a lifetime that is independent from that of the
/// [`Iterable`] object. This means that implicitly a copy of the relevant
/// contents of the object will happen whenever
/// [`Iterable::iter`](crate::iterable::Iterable::iter) is called. This might
/// change in the future as associated type constructors [[RFC1598](https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md#declaring--assigning-an-associated-type-constructor)]
/// stabilize.
///
/// # Future implementation
///
/// A lot of stream operations must be performed symbolycally.
mmaker marked this conversation as resolved.
Show resolved Hide resolved
/// We expect that, in the future, this trait will accomodate for additional
mmaker marked this conversation as resolved.
Show resolved Hide resolved
/// streaming function, e.g. `Iterable::hadamard(&self, other: &Iterable)` to
/// perform the hadamard product of two streams, or `Iterable::add(&self, other:
mmaker marked this conversation as resolved.
Show resolved Hide resolved
/// &Iterable)` to perform the addition of two streams.
pub trait Iterable {
/// The type of the element being streamed.
type Item;
/// The type of the iterator being generated.
type Iter: Iterator<Item = Self::Item>;

/// Returns the iterator associated to the current instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Returns the iterator associated to the current instance.
/// Returns the iterator associated to the current instance.

mmaker marked this conversation as resolved.
Show resolved Hide resolved
///
/// In the so-called _streaming model_ [BCHO22], this is equivalent to
/// instantiating a new stream tape.
/// For base types, this acts in the same way as the `.iter()` method.
///
/// ```
/// use ark_std::iterable::Iterable;
///
/// let x = &[1, 2, 4];
/// let mut iterator = x.iter();
/// ```
fn iter(&self) -> Self::Iter;

/// Returns a hint on the length of the stream.
mmaker marked this conversation as resolved.
Show resolved Hide resolved
///
/// Careful: different objects might have different indications of what
/// _length_ means; this might not be the actual size in terms of
/// elements.
fn len(&self) -> usize;

/// Return `true` if the stream is empty, else `false`.
fn is_empty(&self) -> bool {
self.len() == 0
}
}

impl<I> Iterable for I
where
I: IntoIterator + Copy,
I::IntoIter: ExactSizeIterator,
{
type Item = <I as IntoIterator>::Item;
type Iter = <I as IntoIterator>::IntoIter;

fn iter(&self) -> Self::Iter {
self.into_iter()
}

fn len(&self) -> usize {
self.into_iter().len()
}
}
17 changes: 11 additions & 6 deletions src/lib.rs
Expand Up @@ -49,6 +49,8 @@ pub use rand_helper::*;

pub mod perf_trace;

pub mod iterable;

pub use num_traits::{One, Zero};

/// Returns the ceiling of the base-2 logarithm of `x`.
Expand All @@ -75,8 +77,9 @@ pub fn log2(x: usize) -> u32 {
}

/// Creates parallel iterator over refs if `parallel` feature is enabled.
/// Additionally, if the object being iterated implements `IndexedParallelIterator`,
/// then one can specify a minimum size for iteration.
/// Additionally, if the object being iterated implements
/// `IndexedParallelIterator`, then one can specify a minimum size for
/// iteration.
#[macro_export]
macro_rules! cfg_iter {
($e: expr, $min_len: expr) => {{
Expand All @@ -100,8 +103,9 @@ macro_rules! cfg_iter {
}

/// Creates parallel iterator over mut refs if `parallel` feature is enabled.
/// Additionally, if the object being iterated implements `IndexedParallelIterator`,
/// then one can specify a minimum size for iteration.
/// Additionally, if the object being iterated implements
/// `IndexedParallelIterator`, then one can specify a minimum size for
/// iteration.
#[macro_export]
macro_rules! cfg_iter_mut {
($e: expr, $min_len: expr) => {{
Expand All @@ -125,8 +129,9 @@ macro_rules! cfg_iter_mut {
}

/// Creates parallel iterator if `parallel` feature is enabled.
/// Additionally, if the object being iterated implements `IndexedParallelIterator`,
/// then one can specify a minimum size for iteration.
/// Additionally, if the object being iterated implements
/// `IndexedParallelIterator`, then one can specify a minimum size for
/// iteration.
#[macro_export]
macro_rules! cfg_into_iter {
($e: expr, $min_len: expr) => {{
Expand Down