Arno Schödl | 29 Aug 12:10
Favicon

lifetime of ranges vs. iterators

Hello,

some adaptor ranges need certain constant information about their underlying ranges. For example,
iterating over a forward "difference_range", the difference between two sorted ranges, requires
access to the ends of the underlying ranges. These two additional iterators are constant, they do not
change when the difference_iterator is incremented.

Now, where should this additional constant information be stored?

a) In the difference_range, and each iterator holds a reference to the difference_range? That requires
the difference_range to be alive as long as its iterators are alive. This may be difficult to ensure if the
difference_range is a temporary.

b) In the iterators, so the lifetime of adaptor iterators is independent of the adaptor range. Of course,
one can use shared_ptrs and so on, but still the iterators are ultimately responsible to keep that
information available.

This initially seemed more natural to me, but if implemented naively by storing the additional iterators
in the iterator itself, it leads to storage bloat: each difference_iterator needs 4 instead of 2 of its
underlying iterators. When stacking difference iterators, the difference in required storage grows
exponentially: instead of 2^N, you need 4^N storage...

I feel that in order to solidify the range concept, one needs to make a choice whether a) or b) is "standard",
because any user of ranges would need to know. Any thoughts on this?

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 
(Continue reading)

Neil Groves | 29 Aug 12:59

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 11:12 AM, Arno Schödl <aschoedl <at> think-cell.com>wrote:

> Hello,
>
>
>
> some adaptor ranges need certain constant information about their
> underlying ranges. For example, iterating over a forward "difference_range",
> the difference between two sorted ranges, requires access to the ends of the
> underlying ranges. These two additional iterators are constant, they do not
> change when the difference_iterator is incremented.
>
>
>
> Now, where should this additional constant information be stored?
>
>
>
> a) In the difference_range, and each iterator holds a reference to the
> difference_range? That requires the difference_range to be alive as long as
> its iterators are alive. This may be difficult to ensure if the
> difference_range is a temporary.
>
>
Option a, is the solution that I prefer due to the lower memory overhead. I
am of the opinion that there is no need to provide a uniform lifetime
arrangement since it would often make optimal implementations impossible.
The difference_range should be implemented with the requirement that it be
held for as long as the iterators. The iterator invalidation of the standard
containers being different for same named operations on different classes is
(Continue reading)

David Abrahams | 29 Aug 13:00
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Aug 29 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

> Hello,
>
>  
>
> some adaptor ranges need certain constant information about their
> underlying ranges. For example, iterating over a forward
> "difference_range", the difference between two sorted ranges, requires
> access to the ends of the underlying ranges. These two additional
> iterators are constant, they do not change when the
> difference_iterator is incremented.
>
>  
>
> Now, where should this additional constant information be stored?
>
>  
>
> a) In the difference_range, and each iterator holds a reference to the
> difference_range? That requires the difference_range to be alive as
> long as its iterators are alive. This may be difficult to ensure if
> the difference_range is a temporary.
>
> b) In the iterators, so the lifetime of adaptor iterators is
> independent of the adaptor range. Of course, one can use shared_ptrs
> and so on, but still the iterators are ultimately responsible to keep
> that information available.
>
(Continue reading)

Neil Groves | 29 Aug 13:05

Re: lifetime of ranges vs. iterators

>
> Aren't standard containers supposed to satisfy the Range concepts?  When
> the container is destroyed, its iterators are invalidated, right?
>

Yes, containers satisfy Range concepts, but this question is more related to
the opposite relationship. There is no need for a range destruction to
invalidate the iterators, and this leads to simpler code for example when
creating a temporary range to pass to a range algorithm.

Dave Abrahams
> BoostPro Computing
> http://www.boostpro.com

Neil Groves

>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

David Abrahams | 29 Aug 15:56
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Aug 29 2008, "Neil Groves" <neil-AT-grovescomputing.com> wrote:

>>
>> Aren't standard containers supposed to satisfy the Range concepts?  When
>> the container is destroyed, its iterators are invalidated, right?
>>
>
> Yes, containers satisfy Range concepts, but this question is more related to
> the opposite relationship. There is no need for a range destruction to
> invalidate the iterators, and this leads to simpler code for example when
> creating a temporary range to pass to a range algorithm.

So?  No algorithm could possibly _rely_ on the fact that the iterators
will be invalidated, so what does it matter?  You can do whatever's most
efficient.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Arno Schödl | 29 Aug 14:50
Favicon

Re: lifetime of ranges vs. iterators

For iterators, the make_ helpers are great:

vector<int> vecnA;
vector<int> vecnB;

DoSomethingWithRangeThatStoresIterators( 
	make_difference_iterator( vecnA.begin(), vecnA,end(), vecnB.begin(), vecnB.end() ),
	make_difference_iterator( vecnA.end(), vecnA,end(), vecnB.end(), vecnB.end() )
);

The equivalent with ranges, supposedly easier, would now look like this:

vector<int> vecnA;
vector<int> vecnB;

difference_range< int, int > diffrng( vecnA, vecnB ); // watch out with scope
DoSomethingWithRangeThatStoresIterators( diffrng ); 

as opposed to the much nicer:

vector<int> vecnA;
vector<int> vecnB;

DoSomethingWithRangeThatStoresIterators( make_difference_iterator( vecnA, vecnB ) );

If we drive this stacking to one or more levels, things pretty quickly become pretty ugly. Do we really want that?

Arno

-----Original Message-----
(Continue reading)

Neil Groves | 29 Aug 14:56

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 1:50 PM, Arno Schödl <aschoedl <at> think-cell.com>wrote:

> For iterators, the make_ helpers are great:
>
> vector<int> vecnA;
> vector<int> vecnB;
>
> DoSomethingWithRangeThatStoresIterators(
>        make_difference_iterator( vecnA.begin(), vecnA,end(), vecnB.begin(),
> vecnB.end() ),
>        make_difference_iterator( vecnA.end(), vecnA,end(), vecnB.end(),
> vecnB.end() )
> );
>
> The equivalent with ranges, supposedly easier, would now look like this:
>
> vector<int> vecnA;
> vector<int> vecnB;
>
> difference_range< int, int > diffrng( vecnA, vecnB ); // watch out with
> scope
> DoSomethingWithRangeThatStoresIterators( diffrng );
>
> as opposed to the much nicer:
>
> vector<int> vecnA;
> vector<int> vecnB;
>
> DoSomethingWithRangeThatStoresIterators( make_difference_iterator( vecnA,
> vecnB ) );
(Continue reading)

Arno Schödl | 29 Aug 15:13
Favicon

Re: lifetime of ranges vs. iterators

I have seen your | operator. It is o.k. for unary things, like rng | filter( predicate ), but for binary
things, it is a bit weird:

rngA | difference(rngB)

I find the alternatives clearer: rngA - rngB would be nice (but requires concept checking), or difference(
rngA, rngB ).

But regardless of notation, doesn't this suffer from the same problem that these objects are temporaries?

Arno

-----Original Message-----
From: boost-bounces <at> lists.boost.org [mailto:boost-bounces <at> lists.boost.org] On Behalf Of Neil Groves
Sent: Friday, August 29, 2008 2:56 PM
To: boost <at> lists.boost.org
Subject: Re: [boost] lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 1:50 PM, Arno Schödl <aschoedl <at> think-cell.com>wrote:

> For iterators, the make_ helpers are great:
>
> vector<int> vecnA;
> vector<int> vecnB;
>
> DoSomethingWithRangeThatStoresIterators(
>        make_difference_iterator( vecnA.begin(), vecnA,end(), vecnB.begin(),
> vecnB.end() ),
>        make_difference_iterator( vecnA.end(), vecnA,end(), vecnB.end(),
> vecnB.end() )
(Continue reading)

Neil Groves | 29 Aug 15:25

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 2:13 PM, Arno Schödl <aschoedl <at> think-cell.com>wrote:

> I have seen your | operator. It is o.k. for unary things, like rng |
> filter( predicate ), but for binary things, it is a bit weird:
>
> rngA | difference(rngB)
>
> I find the alternatives clearer: rngA - rngB would be nice (but requires
> concept checking), or difference( rngA, rngB ).
>

Indeed I can definately see your point about binary operations.

>
> But regardless of notation, doesn't this suffer from the same problem that
> these objects are temporaries?
>

Yes, it does not help at all with the 'difference' issue. Would you be able
to supply code for the difference solution you have proposed? I would be
happy to look at it. It would help me understand the problem more
concretely. If you have specific suggestions for changes to range / range_ex
then I would be happy to consider them. With my currently level of
non-understanding of the actual difference algorithm I wonder if the result
is not more appropriately a model of a Container rather than a Range?

Neil

>
> Arno
(Continue reading)

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 3:13 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
> I have seen your | operator. It is o.k. for unary things, like rng | filter( predicate ), but for binary
things, it is a bit weird:
>
> rngA | difference(rngB)
>
> I find the alternatives clearer: rngA - rngB would be nice (but requires concept checking), or
difference( rngA, rngB ).

What about Infix a-la FC++?

  (rngA ^difference^ rngB)

I love the syntax, but maybe this is too much operator abuse.

>
> But regardless of notation, doesn't this suffer from the same problem that these
> objects are temporaries?

I'm still missing something: where is the problem in:

vector<int> vecnA;
vector<int> vecnB;

DoSomethingWithRangeThatStoresIterators( difference_range< int, int >
diffrng( vecnA, vecnB ) );

This works, as long as the range doesn't escape from DSWRTSI, or a copy is made.

[of course a make_difference_range would make the code much cleaner]
(Continue reading)

Arno Schödl | 29 Aug 16:12
Favicon

Re: lifetime of ranges vs. iterators

Indeed, as long as the range does not escape, everything is fine. But here we are in trouble:

some_stacked_adaptor_range<...> rng( make_difference_range( vecnA, vecnB ) );

Again, notation makes no difference, but Neil's idea, which I fully endorse, namely easy stacking, makes
the problem all the more apparent:

some_stacked_adaptor_range<...> rng( vecnA | difference( vecnB ) ); // wrong

What makes this really annoying is that there is an easy way out: Wouldn't it be more natural to represent a
forward range as a single iterator, with an extra .at_end() check? This would eliminate the problem,
because a difference_range would only need two such iterators, not four. Stacking them naively would
have 2^N storage growth, rather than 4^N, and we still don't need to worry about range lifetime.

For other classes of iterators, the situation is similar. bidirectional ranges have at_end() and
at_begin(), and random_access_ranges a size() and an operator[].

I know that overthrowing such an established concept as ranges is not terribly popular, but I don't see any
other way to combine easy stacking with no worry about lifetimes.

Arno

-----Original Message-----
From: boost-bounces <at> lists.boost.org [mailto:boost-bounces <at> lists.boost.org] On Behalf Of Giovanni
Piero Deretta
Sent: Friday, August 29, 2008 3:28 PM
To: boost <at> lists.boost.org
Subject: Re: [boost] lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 3:13 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
(Continue reading)

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 4:12 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
> Indeed, as long as the range does not escape, everything is fine. But here we are in
> trouble:
>
> some_stacked_adaptor_range<...> rng( make_difference_range( vecnA, vecnB ) );
>
> Again, notation makes no difference, but Neil's idea, which I fully endorse, namely
> easy stacking, makes the problem all the more apparent:
>
> some_stacked_adaptor_range<...> rng( vecnA | difference( vecnB ) ); // wrong
>
> What makes this really annoying is that there is an easy way out: Wouldn't it be more
> natural to represent a forward range as a single iterator, with an extra .at_end()
> check? This would eliminate the problem, because a difference_range would only need
> two such iterators, not four. Stacking them naively would have 2^N storage growth,
> rather than 4^N, and we still don't need to worry about range lifetime.
>

Ok, now I understand.

The lack of auto and decltype usually means that you use range
adaptors in a strictly LIFO way, i.e. you usually never have named
range adaptors, and iterator validity is never an issue as the scope
of the wrapping range is always a subset of the wrapped range.

Anyways, the solution is simple: if you want your adaptor to survive
the original wrapped range, you have to store a copy of the wrapped
range in the adaptor. As an added benefit, the adaptor size will be
the smallest possible for deep sequences of stacked adaptors.

(Continue reading)

David Abrahams | 29 Aug 16:48
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Aug 29 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

> Wouldn't it be more natural to represent a forward range as a single
> iterator, with an extra .at_end() check? This would eliminate the
> problem, because a difference_range would only need two such
> iterators, not four. 

As part of
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1873.html one
of the ideas we discussed extensively was allowing the end iterator of a
range to be a different type from the begin iterator.  That would allow
a sort of "universal end iterator" to be used in situations such as
this:

  struct end
  {
      // These definitions are only useful for infinite ranges
      template <class T>
      bool operator==(T const&) { return false; }
      template <class T>
      bool operator=!(T const&) { return true; }
  };

Effectively the data representation of the begin iterator would be the
same as what you're proposing, plus comparison operators against the
universal end type above, but the end iterator type could be empty (and
optimized away).  That allows a generic description of algorithms that
are optimally efficient on pairs of pointers, but could simplify many
classes of iterator (ifstream_iterators, for example) where every
(Continue reading)

Re: lifetime of ranges vs. iterators

On Fri, Aug 29, 2008 at 4:48 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Fri Aug 29 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:
>
>> Wouldn't it be more natural to represent a forward range as a single
>> iterator, with an extra .at_end() check? This would eliminate the
>> problem, because a difference_range would only need two such
>> iterators, not four.
>
> As part of
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1873.html one
> of the ideas we discussed extensively was allowing the end iterator of a
> range to be a different type from the begin iterator.
>

<snip>

> Effectively the data representation of the begin iterator would be the
> same as what you're proposing, plus comparison operators against the
> universal end type above, but the end iterator type could be empty (and
> optimized away).  That allows a generic description of algorithms that
> are optimally efficient on pairs of pointers, but could simplify many
> classes of iterator (ifstream_iterators, for example) where every
> iterator needs all the information required to detect whether it can
> advance anyway.
>

Nice!

>> Stacking them naively would have 2^N storage
(Continue reading)

David Abrahams | 30 Aug 01:51
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Aug 29 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> On Fri, Aug 29, 2008 at 4:48 PM, David Abrahams <dave <at> boostpro.com> wrote:
>>
>> I've been trying to say that generic algorithms *can't* worry about
>> the range lifetime; they have to assume that the iterators they're
>> passed will not be invalidated before they're destroyed.
>
> I think that the OP is talking about a different problem: let say you
> want to name the result of stacked applications of range adaptors:
>
> template<typename Range>
> void my_algo(Range& r) {
>   auto stacked_range = r | filtered(filter) | mapped(mapper);
>   // use stacked_range
> }
>
> This may lead iterator lifetime problems, not because r iterators may
> be invalid, but because the iterators of the range returned by
> filtered(...) may become invalid as that temporary has been
> destructed. There are many solutions:
>
> The easiest is splitting apart the expression, but is ugly.
>
> Then you could use expression templates to make the whole pipe
> expression return a range only at the end.
>
> Finally, both filtered and mapped could guarantee that the iterators
> are not invalidated when the range is destructed, but that may prevent
(Continue reading)

Re: lifetime of ranges vs. iterators

On Sat, Aug 30, 2008 at 1:51 AM, David Abrahams <dave <at> boostpro.com> wrote:
>
>> I'm not sure this is as revolutionary as you think.  Every other
>>> language I've looked at has a get_next()/at_end() iterator abstraction,
>>> which is a lot like what you're proposing.  My advice: make sure you
>>> keep in mind the whole problem domain.  You can't understand the reason
>>> that C++ iterators are unlike those in other languages without looking
>>> at the algorithms.
>>>
>>
>> BTW, ranges are equivalent to GoF iterators:
>
> Well, they're informationally equivalent...
>
>> you just spell 'at_end'
>> as 'empty(r)';
>> 'next()' as 'r = range(next(begin(r)), end(r))'   (equivalently: 'r =
>> rest(r))'; and '.get()' as
>> '*begin(r)';
>
> ...but that is so ugly as to make them non-equivalent abstractions in my
> book.  And that's a good thing for ranges ;-)

when I need a simple for loop (and can't reach for BOOST_FOREACH), the syntax
I sometimes use is:

for(range<container> r (c);  r; r = rest(r)) {
   ....
}

(Continue reading)

Re: lifetime of ranges vs. iterators

On Sat, Aug 30, 2008 at 1:51 AM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Fri Aug 29 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>> On Fri, Aug 29, 2008 at 4:48 PM, David Abrahams <dave <at> boostpro.com> wrote:
>>>
>>> I've been trying to say that generic algorithms *can't* worry about
>>> the range lifetime; they have to assume that the iterators they're
>>> passed will not be invalidated before they're destroyed.
>>
>> I think that the OP is talking about a different problem: let say you
>> want to name the result of stacked applications of range adaptors:
>>
>> template<typename Range>
>> void my_algo(Range& r) {
>>   auto stacked_range = r | filtered(filter) | mapped(mapper);
>>   // use stacked_range
>> }
>>
>> This may lead iterator lifetime problems, not because r iterators may
>> be invalid, but because the iterators of the range returned by
>> filtered(...) may become invalid as that temporary has been
>> destructed. There are many solutions:
>>
>> The easiest is splitting apart the expression, but is ugly.
>>
>> Then you could use expression templates to make the whole pipe
>> expression return a range only at the end.
>>
>> Finally, both filtered and mapped could guarantee that the iterators
(Continue reading)

Arno Schödl | 30 Aug 11:26
Favicon

Re: lifetime of ranges vs. iterators

I am following up on Giovanni's idea of storing ranges by value. To minimize the impact on existing code, and
to make the frequent case of wrapping an adaptor range around another container like vector or set easy, we
could shift the burden to specify which ranges should be stored by reference and which by value onto the
ranges themselves.

A template container_reference would designate a reference to the underlying container that holds the
data. The data is assumed to remain valid, but any adaptors around it may go out of scope.

Storage by reference would be default, so everything works as it did so far, but any wrapped ranges may not go
out of scope:

template< class T >
struct container_reference {
	typedef T& type;
};

Adaptor ranges could decide to store themselves by value:

template< class RngA, class RngB >
struct container_reference< difference_range<RngA, RngB> > {
	typedef T type;
};

Adaptors would use it to store a reference to the underlying data:

template< class RngA, class RngB >
class difference_range {
	typename container_reference< RngA >::type m_rngA;
	typename container_reference< RngB >::type m_rngB;

(Continue reading)

David Abrahams | 30 Aug 20:17
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sat Aug 30 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> Hum, this would means that every range needs to know how to construct
> and deconstruct iterators of other ranges, at least if it wants to
> guarantee that the minimum space possible is used.

It would be done with associated types and functions of the iterators.
Few iterators need to store that kind of redundant information, but when
you build an iterator type that *does*, it would be deemed "polite" to
specialize an appropriate trait and/or define the corresponding
overloads to allow it to be decomposed/re-composed.

I've been itching to rework Boost.Iterator recently, and this could make
a very nice enhancement.

> It seems a bit complex, and there would be a need to come up with some
> non obvious protocol to do that (some sort of compile time
> introspection on ranges).

Not on ranges; just on iterators.

> Is it really worth it? Are you advocating it, or it was just another
> solution to consider?

I think I may be advocating it :-)

I think there may be some relationship between this and hierarchical
iterator representation, so I'd like to explore it further from that
angle as well.
(Continue reading)

Steven Watanabe | 30 Aug 23:18

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
> on Sat Aug 30 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>   
>> Hum, this would means that every range needs to know how to construct
>> and deconstruct iterators of other ranges, at least if it wants to
>> guarantee that the minimum space possible is used.
>>     
>
> It would be done with associated types and functions of the iterators.
> Few iterators need to store that kind of redundant information, but when
> you build an iterator type that *does*, it would be deemed "polite" to
> specialize an appropriate trait and/or define the corresponding
> overloads to allow it to be decomposed/re-composed.
>
> I've been itching to rework Boost.Iterator recently, and this could make
> a very nice enhancement.
>   

How would this work:

concept IteratorWithEnd<typename Iterator> {
    Iterator get_end(Iterator);
    Iterator set_end(Iterator, Iterator);
}

Taking filter_iterator, for example, this would allow the space to be 
minimized,
(Continue reading)

David Abrahams | 31 Aug 02:24
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sat Aug 30 2008, Steven Watanabe <watanabesj-AT-gmail.com> wrote:

> How would this work:
>
> 
> template<IteratorWithEnd Iterator, class F>
> class filter_iterator<Iterator, F>
> {
> public:
>    //...
>    filter_iterator(Iterator begin, Iterator end, F f)
>      : iter(set_end(begin,end)), f(f)
>    {}
>     
>    filter_iterator& operator++() {
>        ++iter;
>        while(iter != get_end(iter) && !f(*iter)) ++iter;
>        return *this;
>    }
>     
>    friend Iterator get_end(const filter_iterator& self) {
>        return
>            filter_iterator(
>                get_end(self.iter), get_end(self.iter), self.f);
>    }
>     
>    friend Iterator set_end(
>        const filter_iterator& self, const filter_iterator& other)
>    {
(Continue reading)

Steven Watanabe | 31 Aug 02:39

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
>>    friend Iterator get_end(const filter_iterator& self) {
>>        return
>>            filter_iterator(
>>                get_end(self.iter), get_end(self.iter), self.f);
>>    }
>>     
>>    friend Iterator set_end(
>>        const filter_iterator& self, const filter_iterator& other);
>>     
>
> Hi Steven,
>
> Say for example Iterator is strided_iterator<int>.  Then the return type
> of set_end above is strided_iterator<int>, right?  So how will its body
> typecheck?  Seems like this needs a little more thought.
>   

Oops.  I intended the return type to be filter_iterator.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

David Abrahams | 31 Aug 06:15
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sat Aug 30 2008, Steven Watanabe <watanabesj-AT-gmail.com> wrote:

> AMDG
>
> David Abrahams wrote:
>>>    friend Iterator get_end(const filter_iterator& self) {
>>>        return
>>>            filter_iterator(
>>>                get_end(self.iter), get_end(self.iter), self.f);
>>>    }
>>>        friend Iterator set_end(
>>>        const filter_iterator& self, const filter_iterator& other);
>>>     
>>
>> Hi Steven,
>>
>> Say for example Iterator is strided_iterator<int>.  Then the return type
>> of set_end above is strided_iterator<int>, right?  So how will its body
>> typecheck?  Seems like this needs a little more thought.
>>   
>
> Oops.  I intended the return type to be filter_iterator.

Okay then, I take it you meant something like:

    template<IteratorWithEnd Iterator, class F>
    class filter_iterator<Iterator, F>
    {
    public:
(Continue reading)

Arno Schödl | 31 Aug 16:09
Favicon

Re: lifetime of ranges vs. iterators

>  // Gimme a better name, please!  The abstraction is a thing that
>  // is always used in pairs, where each instance contains some 
>  // redundant information in common.
>  auto concept Factorable<typename T>
>  {
>       typename common_data = void;
>       typename unique_data = T;
>
>       common_data common(T) {}
>       unique_data unique(T x) { return x; }
>       T reconstruct(common_data, unique_data x) { return x; }
>  };

How is the separation of common_data and unique_data different from a separation of ranges and iterators?
If iterators of ranges can rely on their range to exist, this is where common data like functors, end
iterators etc. can be stored.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

(Continue reading)

David Abrahams | 31 Aug 21:19
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sun Aug 31 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>>  // Gimme a better name, please!  The abstraction is a thing that
>>  // is always used in pairs, where each instance contains some 
>>  // redundant information in common.
>>  auto concept Factorable<typename T>
>>  {
>>       typename common_data = void;
>>       typename unique_data = T;
>>
>>       common_data common(T) {}
>>       unique_data unique(T x) { return x; }
>>       T reconstruct(common_data, unique_data x) { return x; }
>>  };
>
> How is the separation of common_data and unique_data different from a separation
> of ranges and iterators? If iterators of ranges can rely on their range to
> exist, this is where common data like functors, end iterators etc. can be
> stored.

Naturally the point is to store the common data once when you need to
store more than one iterator to the same sequence -- in a range, for
example.

If you're asking why I bother reconstituting the whole iterator out of
its parts, instead of simply referring to the common_data stored in the
range: an adapted iterator is a useful concept regardless of whether
anyone is using a range abstraction.

(Continue reading)

Arno Schödl | 31 Aug 23:59
Favicon

Re: lifetime of ranges vs. iterators

> > How is the separation of common_data and unique_data different from a separation
> > of ranges and iterators? If iterators of ranges can rely on their range to
> > exist, this is where common data like functors, end iterators etc. can be
> > stored.

> Naturally the point is to store the common data once when you need to
> store more than one iterator to the same sequence -- in a range, for
> example.

> If you're asking why I bother reconstituting the whole iterator out of
> its parts, instead of simply referring to the common_data stored in the
> range: an adapted iterator is a useful concept regardless of whether
> anyone is using a range abstraction.

You could provide an adapted_iterator and also an adapted_range. If we subscribe to the rule that ranges
must stay valid for their iterators to be valid, the adapted_range::iterator can use the common data
stored in the range, while the adapted_iterator stores the common data itself. Both could even be derived
from the same source code. Do you then still need a factored iterator?

Or do you want to avoid people having to use the range abstraction? If you pass iterator pairs to algorithms
instead of ranges, at least this parameter passing would have to pass the common data redundantly, even if
inside the algorithm, say when many iterators have to be stored, the common data is stripped and stored separately.

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229
(Continue reading)

David Abrahams | 1 Sep 05:24
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sun Aug 31 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > How is the separation of common_data and unique_data different from a
> separation
>> > of ranges and iterators? If iterators of ranges can rely on their range to
>> > exist, this is where common data like functors, end iterators etc. can be
>> > stored.
>
>> Naturally the point is to store the common data once when you need to
>> store more than one iterator to the same sequence -- in a range, for
>> example.
>
>> If you're asking why I bother reconstituting the whole iterator out of
>> its parts, instead of simply referring to the common_data stored in the
>> range: an adapted iterator is a useful concept regardless of whether
>> anyone is using a range abstraction.
>
> You could provide an adapted_iterator and also an adapted_range. 

My point is that the adapted_range would then have semantically
equivalent iterators with a different representation (and an extra
indirection), which seems like a waste.

> If we subscribe to the rule that ranges must stay valid for their
> iterators to be valid,

I don't.  I do subscribe to the rule that generic code cannot afford to
destroy an arbitrary range while its iterators are still in use.

(Continue reading)

Arno Schödl | 1 Sep 08:57
Favicon

Re: lifetime of ranges vs. iterators

> > If we subscribe to the rule that ranges must stay valid for their
> > iterators to be valid,

> I don't.  I do subscribe to the rule that generic code cannot afford to
> destroy an arbitrary range while its iterators are still in use.

Isn't that what I said? There may be iterators that work without their ranges, but in general they don't.

> > the adapted_range::iterator can use the common data stored in the
> > range, while the adapted_iterator stores the common data itself. Both
> > could even be derived from the same source code. 

> Yeah, that's still a lot of coding effort.

I think you could write it generically, a la iterator_facade/adaptor, so it is a one-time fixed cost.

> > Do you then still need a factored iterator?

> You need to be able to take two adapted iterators and turn them into a
> range.  Do you want that range to contain redundant data?  I don't.

> > Or do you want to avoid people having to use the range abstraction? 

> I certainly don't want to force it on them.

Ok, now I understand. The debate is about primacy of ranges or iterators. You propose that iterators stay
the primary vehicle and to convert them to/from ranges by stripping the common information. But that
would mean that there is no "lean" iterator, all iterators would contain the redundant information. 

When stacking N difference_ranges, the size difference between "fat" and "lean" iterators is 2^N. Thus in
(Continue reading)

David Abrahams | 1 Sep 18:51
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Mon Sep 01 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > If we subscribe to the rule that ranges must stay valid for their
>> > iterators to be valid,
>
>> I don't.  I do subscribe to the rule that generic code cannot afford to
>> destroy an arbitrary range while its iterators are still in use.
>
> Isn't that what I said? 

No, but it might be what you meant.

> There may be iterators that work without their ranges,
> but in general they don't.
>
>> > the adapted_range::iterator can use the common data stored in the
>> > range, while the adapted_iterator stores the common data itself. Both
>> > could even be derived from the same source code. 
>
>> Yeah, that's still a lot of coding effort.
>
> I think you could write it generically, a la iterator_facade/adaptor,
> so it is a one-time fixed cost.

I'm not sure if it's worth the effort.

>> > Do you then still need a factored iterator?
>
>> You need to be able to take two adapted iterators and turn them into a
(Continue reading)

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 5:24 AM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Sun Aug 31 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:
>
>>> > How is the separation of common_data and unique_data different from a
>> separation
>>> > of ranges and iterators? If iterators of ranges can rely on their range to
>>> > exist, this is where common data like functors, end iterators etc. can be
>>> > stored.
>>
>>> Naturally the point is to store the common data once when you need to
>>> store more than one iterator to the same sequence -- in a range, for
>>> example.
>>
>>> If you're asking why I bother reconstituting the whole iterator out of
>>> its parts, instead of simply referring to the common_data stored in the
>>> range: an adapted iterator is a useful concept regardless of whether
>>> anyone is using a range abstraction.
>>
>> You could provide an adapted_iterator and also an adapted_range.
>
> My point is that the adapted_range would then have semantically
> equivalent iterators with a different representation (and an extra
> indirection), which seems like a waste.

hum, which indirection?

<...>
>> Do you then still need a factored iterator?
>
(Continue reading)

Arno Schödl | 1 Sep 13:56
Favicon

Re: lifetime of ranges vs. iterators

> >
> > You need to be able to take two adapted iterators and turn them into a
> > range.  Do you want that range to contain redundant data?  I don't.

> Hei, maybe this is all that we need! Let's have a metafunction that
> given an iterator type returns its preferred range type (the
> associated range). The default might be boost.range or even std::pair.
>  The developer of an iterator adaptor, can specialize the metafunction
> so that the associate range is the optimum representation for an
> iterator pair.

> Do you think this would be enough? It seems too simple to me, so
> probably I'm missing something...

I wouldn't know what to do when my iterator_adaptor needs three iterators to the same range. I could store
two efficiently as a range and retrieve them when needed by querying range::begin()/end(), but what do I
do with the third? Dave's factored iterators go further in the same direction, basically dropping the
requirement that the common data must be a range, it can be anything. 

Both proposals have the disadvantage that I need to wrap/unwrap the [common data (Dave)/range
(Giovanni)] whenever I change one of the wrapped iterators. I'd rather store the common data explicitly,
and have indirections inside the iterators that point to it.

Correct me if I misunderstand one of the two proposals.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 
(Continue reading)

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 1:56 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
>> >
>> > You need to be able to take two adapted iterators and turn them into a
>> > range.  Do you want that range to contain redundant data?  I don't.
>
>> Hei, maybe this is all that we need! Let's have a metafunction that
>> given an iterator type returns its preferred range type (the
>> associated range). The default might be boost.range or even std::pair.
>>  The developer of an iterator adaptor, can specialize the metafunction
>> so that the associate range is the optimum representation for an
>> iterator pair.
>
>> Do you think this would be enough? It seems too simple to me, so
>> probably I'm missing something...
>
> I wouldn't know what to do when my iterator_adaptor needs three iterators to the >same range. I could store
two efficiently as a range and retrieve them when needed >by querying range::begin()/end(), but what do I
do with the third? Dave's factored
>terators go further in the same direction, basically dropping the requirement that the >common data must
be a range, it can be anything.

You are right. It wouldn't work with three iterators.

But three iterators are a range of ranges. Maybe segmented iterators
(or, better, segmented ranges)  can come to the rescue?

Need to think about it. Could you give a practical example of an
iterator adapter needing three iterators?

>
(Continue reading)

Arno Schödl | 1 Sep 16:08
Favicon

Re: [SPAM (Bayesian)] - Re: lifetime of ranges vs. iterators - Bayesian Filter detected spam

> Need to think about it. Could you give a practical example of an
> iterator adapter needing three iterators?

I never needed them in practice. But I would forward this to Dave. He likes the complete generality of
factored iterators, I think. I think it is a cleaner concept than saying the common data must be
representable as a range, but I cannot really back this up with practical examples. The
wrapping/unwrapping (see below) troubles me in either case.

> You would still have to allow for storing the wrapped iterator in the
> wrapper if the wrapped iterator does not support refactoring. I do not
> think there is a way around it.

But in my proposal (actually, yours), all iterators would be "lean", and each iterator would have access to
its range, all the way up the iterator/range stack. So each iterator would itself only wrap other lean
iterators. Storage bloat for stack size N would only be O(N) for the indirections, rather than O(2^N).

> Also, I have this (completely unfounded) feeling that an iterator that
> relies on accessing common data in the owning range might be harder to
> optimize.

> I think that ranges should be used for long term storage, so if the
> iterator size itself is not optimal shouldn't be a big problem.

Without any kind of storage optimization like your preferred-range templates or Dave's
factored-iterators, the storage bloat would be _factor_ 2^N, so ever handling completely unwrapped
iterators just does not scale.

So it comes down to:

a) with storage optimization, at every iterator access, unwrap/wrap the iterator out of their storage:
(Continue reading)

Re: [SPAM (Bayesian)] - Re: lifetime of ranges vs. iterators - Bayesian Filter detected spam

On Mon, Sep 1, 2008 at 4:08 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
>> Need to think about it. Could you give a practical example of an
>> iterator adapter needing three iterators?
>
> I never needed them in practice. But I would forward this to Dave. He likes the
> complete generality of factored iterators, I think. I think it is a cleaner concept than
> saying the common data must be representable as a range, but I cannot really back
> this up with practical examples. The wrapping/unwrapping (see below) troubles me in > either case.
>

Sure, full generality is always good, but if ranges cover 90% of the
cases, maybe it is good enough.

>> You would still have to allow for storing the wrapped iterator in the
>> wrapper if the wrapped iterator does not support refactoring. I do not
>> think there is a way around it.
>
> But in my proposal (actually, yours), all iterators would be "lean", and each iterator
> would have access to its range, all the way up the iterator/range stack. So each
> iterator would itself only wrap other lean iterators. Storage bloat for stack size N
> would only be O(N) for the indirections, rather than O(2^N).

You can't assume that all iterators are lean. In fact most won't.
Unless you drop compatibility with existing iterators, which is a
no-no.

My proposal was aimed at making ranges as lean as possible, not
necessarily iterators.
Anyways, as long as all parts of the stack are 'lean' ranges, you
should have only a 2x storage bloat when using iterators.
(Continue reading)

Arno Schödl | 1 Sep 20:28
Favicon

Re: [SPAM (Bayesian)] - Re: lifetime of ranges vs.iterators - Bayesian Filter detected spam

> Sure, full generality is always good, but if ranges cover 90% of the
> cases, maybe it is good enough.

I agree.

> > But in my proposal (actually, yours), all iterators would be "lean", and each iterator
> > would have access to its range, all the way up the iterator/range stack. So each
> > iterator would itself only wrap other lean iterators. Storage bloat for stack size N
> > would only be O(N) for the indirections, rather than O(2^N).

> You can't assume that all iterators are lean. In fact most won't.
> Unless you drop compatibility with existing iterators, which is a
> no-no.

Old iterators will have neither the indirection I propose, nor will they offer a non-trivial
factorization nor internally store preferred ranges. So any of the schemes discussed make no difference
to old iterators. But old iterators are often trivial, so the whole problem should not really apply to them.

> My proposal was aimed at making ranges as lean as possible, not
> necessarily iterators.
> Anyways, as long as all parts of the stack are 'lean' ranges, you
> should have only a 2x storage bloat when using iterators.

The bloat is potentially much worse with your implementation below:

>  iterator begin() const{
>       return iterator(begin(m_range), end(m_range), m_f);
>  }

This triggers recursive unwrapping all the way to the top, causing factor 2^N extra storage to be
(Continue reading)

David Abrahams | 1 Sep 19:01
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Mon Sep 01 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> Need to think about it. Could you give a practical example of an
> iterator adapter needing three iterators?

E.g., a bidirectional filtered_iterator needs to be prevented from
stepping off both ends of the underlying data to avoid undefined
behavior.

On the other hand, you shouldn't have to pay for those endpoint
comparisons all the time, which tells me that something may be wrong
with the design.

Iterators are supposed to be lightweight things.  If we're seriously
worried about the cost of building iterator adaptors, we might consider
the idea that there's something wrong with the bigger picture.  Maybe
these endpoint comparisons need to be dealt with differently, somehow.
The situation with deque iterators that led to segmented iterators is
very similar, and it resulted in a system of abstractions that allows
one to operate, once again, close to the machine model.

I wouldn't know how to address this issue without some real-life use
cases, though.  This has all become a bit too abstract a discussion for
me, and Generic Programming is supposed to be driven by concrete
examples.

--

-- 
Dave Abrahams
BoostPro Computing
(Continue reading)

Arno Schödl | 1 Sep 19:53
Favicon

Re: lifetime of ranges vs. iterators

> I wouldn't know how to address this issue without some real-life use
> cases, though.  This has all become a bit too abstract a discussion for
> me, and Generic Programming is supposed to be driven by concrete
> examples.

Isn't my problem concrete enough? I need a stackable difference_range without exponential storage
bloat. How should I implement it?

I am not sure that you appreciate the problem: 

Let's consider two implementations of a forward difference_iterator.

A) Implemented with iterators alone, it contains 4 iterators:

- the iterator to the range being traversed (mutable)
- the iterator to the range being removed (mutable)
- the end iterator to the range being traversed (constant, common)
- the end iterator to the range being removed (constant, common)

Total size: 4*sizeof(ptr)

B) Implemented with iterators + indirection to common data:

- the iterator to the range being traversed (mutable)
- the iterator to the range being removed (mutable)
- a reference to the range

Total size: 3*sizeof(ptr)

Now we stack these things, creating difference_range( difference_range, difference_range ) and so on.
(Continue reading)

Steven Watanabe | 1 Sep 20:32

Re: lifetime of ranges vs. iterators

AMDG

Arno Schödl wrote:
> So any implementation that even temporarily expands iterators to their original size, like Giovanni's
implementation, will need vastly more storage. IMO, relying on the compiler to fix this is hazardous.
Non-optimized code should be a constant factor slower than optimized code, and need a constant factor
more memory, not exponentially slower and exponentially more memory.
>   

If I understand Giovanni's implementation correctly, it only needs to 
expand one layer at a time.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Arno Schödl | 1 Sep 20:55
Favicon

Re: lifetime of ranges vs. iterators

> If I understand Giovanni's implementation correctly, it only needs to 
> expand one layer at a time.

Let's expand it:

  iterator begin() const{
       return iterator(begin(m_range), end(m_range), m_f);
  }

-->

  return iterator( iterator( begin(subrange), end(subrange), m_f ), iterator( end(subrange),
end(subrange), m_f ), m_f);

-->

  ...

The expansion will terminate when begin/end(sub...subrange) is the base iterator. Then the call is made
to the filter_iterator ctor, which calls the associated_range<filter_iterator> ctor, which
condenses everything back down.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
(Continue reading)

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 8:55 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
>> If I understand Giovanni's implementation correctly, it only needs to
>> expand one layer at a time.
>
> Let's expand it:
>
>  iterator begin() const{
>       return iterator(begin(m_range), end(m_range), m_f);
>  }
>
> -->
>
>  return iterator( iterator( begin(subrange), end(subrange), m_f ), iterator( end(subrange),
end(subrange), m_f ), m_f);
>
> -->
>
>  ...
>
>
> The expansion will terminate when begin/end(sub...subrange) is the base iterator.
>Then the call is made to the filter_iterator ctor, which calls the
> associated_range<filter_iterator> ctor, which condenses everything back down.
>

I see the problem.

I think that for non complex iterators, the optimizer can still manage
to fix the problem.

(Continue reading)

Arno Schödl | 1 Sep 23:35
Favicon

Re: lifetime of ranges vs. iterators

I fixed filter_range::begin() and filter_range::end() as you said. iterator::equal was still
exponential, but can be fixed by only comparing begin(). I think the exponential cases are now gone.

Most things are still quadratic in the stack size, though... end() for example constructs a parameter of
size N-1, which to be constructed, must construct a parameter of size N-2 and so on. NRVO does not help, as
far as I understand it. You would need "upward" NRVO, constructing parameters right into the respective
members. Instead of passing a pointer to the function where it wants the return value to be constructed
(NRVO), the caller would need to get a pointer from the callee where the callee wants its parameters
constructed. I don't think compilers do that, at least not reliably enough to build a whole paradigm on it:-(

> [btw, for long chains of filter and map iterators it is not really a
> problem because you can always transform them exactly to one each of
> map and filter iterator]

That's why the problem occurred to me looking at difference_iterator.

------

struct both_end {};

template<class Iter, class F>
struct filter_iterator : iterator_facade<blah,blah...> {  
  filter_iterator(associated_range<Iter>::type const& rng, F f ) :
   m_range(rng), m_f(f) {}

  filter_iterator(iter e, F f, both_end ) :
   m_range(e,e), m_f(f) {} // if m_range takes e by value, e is expanded twice here, but no more, so no problem
private:
  void increment() {
      while(!m_f(front(m_range))
(Continue reading)

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 11:35 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
> I fixed filter_range::begin() and filter_range::end() as you said. iterator::equal was still
exponential, but can be fixed by only comparing begin(). I think the exponential cases are now gone.
>

it should be ok to compare only begin. If the two iterators have
different ends, probably things would be a "little" messed up anyway.

> Most things are still quadratic in the stack size, though... end() for example constructs a parameter of
size N-1, which to be constructed, must construct a parameter of size N-2 and so
> on. NRVO does not help, as far as I understand it. You would need "upward" NRVO, constructing parameters
right into the respective members. Instead of passing a pointer to the >
> function where it wants the return value to be constructed (NRVO), the caller would need to get a pointer
from the callee where the callee wants its parameters constructed. I don't think > compilers do that, at
least not reliably enough to build a whole paradigm on it:-(

Hum, I've played a little with some mockup iterators, and  I can
definitely see the N^2 stack growth, and with some expression template
heavy program, I wouldn't be surprised if the  growth would actually
become noticeable (even if it weren't enough to blow the stack, it
could still produce cache misses and such).   I can't seem to come-up
with any clever scheme to reduce the growth (short of using pointers
to the original range of course).

Enough for today. Maybe tomorrow I'll get some idea.

--

-- 
gpd
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
(Continue reading)

Steven Watanabe | 2 Sep 05:34

Re: lifetime of ranges vs. iterators

AMDG

Giovanni Piero Deretta wrote:
> Hum, I've played a little with some mockup iterators, and  I can
> definitely see the N^2 stack growth, and with some expression template
> heavy program, I wouldn't be surprised if the  growth would actually
> become noticeable (even if it weren't enough to blow the stack, it
> could still produce cache misses and such).   I can't seem to come-up
> with any clever scheme to reduce the growth (short of using pointers
> to the original range of course).
>
> Enough for today. Maybe tomorrow I'll get some idea.
>   

I've been playing around with this a bit.
See attached.

In Christ,
Steven Watanabe

Attachment (scratch.cpp): text/x-c++src, 7401 bytes
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 2 Sep 12:25
Favicon

Re: [SPAM (Bayesian)] - Re: lifetime of ranges vs. iterators - Bayesian Filter detected spam

> I've been playing around with this a bit.
> See attached.

> In Christ,
> Steven Watanabe

    iterator begin() {
        return iterator(boost::begin(m_range), boost::end(m_range), m_f);
    }
    iterator end() {
        return iterator(boost::end(m_range), boost::end(m_range), m_f);
    }

This would blow up, I believe. It gets expanded on both begin and end, causing exponential storage growth.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Steven Watanabe | 2 Sep 16:55

Re: lifetime of ranges vs. iterators

AMDG

Arno Schödl wrote:
> This would blow up, I believe. It gets expanded on both begin and end, causing exponential storage growth.
>   

Ok.  Right.  Incidentally, the storage required is not exponential 
because it is not
all needed at once, but I see your point.

I like the idea of having the filter_iterator contain a copy of the range.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Arno Schödl | 2 Sep 17:02
Favicon

Re: [SPAM (Bayesian)] - Re: lifetime of ranges vs. iterators - Bayesian Filter detected spam

> > This would blow up, I believe. It gets expanded on both begin and end, causing exponential storage growth.
> >   

> Ok.  Right.  Incidentally, the storage required is not exponential 
> because it is not
> all needed at once, but I see your point.

You are right, running time is exponential, but storage is not.

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Re: lifetime of ranges vs. iterators

On Tue, Sep 2, 2008 at 5:34 AM, Steven Watanabe <watanabesj <at> gmail.com> wrote:
> AMDG
>
> Giovanni Piero Deretta wrote:
>>
>> Hum, I've played a little with some mockup iterators, and  I can
>> definitely see the N^2 stack growth, and with some expression template
>> heavy program, I wouldn't be surprised if the  growth would actually
>> become noticeable (even if it weren't enough to blow the stack, it
>> could still produce cache misses and such).   I can't seem to come-up
>> with any clever scheme to reduce the growth (short of using pointers
>> to the original range of course).
>>
>> Enough for today. Maybe tomorrow I'll get some idea.
>>
>
> I've been playing around with this a bit.
> See attached.
>

I can't compile it because I lack a recent phoenix. What numbers are
you getting?

Also, isn't end() still O(N^2) on the stacking depth?

--

-- 
gpd
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

(Continue reading)

Arno Schödl | 2 Sep 08:47
Favicon

Re: lifetime of ranges vs. iterators

> Enough for today. Maybe tomorrow I'll get some idea.

Let me try:

First of all, associated_range has the semantics of "[efficiently (who wouldn't want that?)] holds pair
of iterators of certain type". This is the same semantics as iterator_range. The existing
iterator_ranges are the default implementation of associated_range, and the associate_ranges we want
to implement are specialized iterator_ranges for certain types of iterators. The two types are the same.
So let's use this iterator_range throughout, since we already have it.

To eliminate quadratic running times, we must eliminate any linear-space expansion of function
parameters. To do this, let's introduce some helper functions that make an iterator_range look like an
iterator. Since we just discovered the dual life of iterator_range as associated_range, I will add them
to iterator_range:

template< class I >
class iterator_range {
   ...other iterator_range stuff...

   // regular copy ctor for begin iterator

   ////////////////////////////////////
   // associated_range interface

   // end iterator ctor
   iterator_range( iterator_range const& range, create_end_tag ) 
   : m_begin( range.m_end ), m_end( range.m_end )
   {}

   // ctor from pair of ranges
(Continue reading)

Re: lifetime of ranges vs. iterators

On Tue, Sep 2, 2008 at 8:47 AM, Arno Schödl <aschoedl <at> think-cell.com> wrote:
>> Enough for today. Maybe tomorrow I'll get some idea.
>
> Let me try:
>
> First of all, associated_range has the semantics of "[efficiently (who wouldn't want
> that?)] holds pair of iterators of certain type". This is the same semantics as
> iterator_range. The existing iterator_ranges are the default implementation of
> associated_range, and the associate_ranges we want to implement are specialized
> iterator_ranges for certain types of iterators. The two types are the same. So let's
> use this iterator_range throughout, since we already have it.

The reason to use a trait it to make it easier to adapt existing
ranges. Also, containers are ranges, but do not have your additional
functions, so you need to explicitly wrap them in iterator_ranges
before wrapping them in another adapter range. With associated_range,
the wrapping comes implicitly for free.

>
> To eliminate quadratic running times, we must eliminate any linear-space expansion of
> function parameters. To do this, let's introduce some helper functions that make an
> iterator_range look like an iterator. Since we just discovered the dual life of
> iterator_range as associated_range, I will add them to iterator_range:

I'll add some comments inline in the code.

>
> template< class I >
> class iterator_range {
>   ...other iterator_range stuff...
(Continue reading)

Arno Schödl | 2 Sep 15:22
Favicon

Re: lifetime of ranges vs. iterators

> > First of all, associated_range has the semantics of "[efficiently (who wouldn't want
> > that?)] holds pair of iterators of certain type". This is the same semantics as
> > iterator_range. The existing iterator_ranges are the default implementation of
> > associated_range, and the associate_ranges we want to implement are specialized
> > iterator_ranges for certain types of iterators. The two types are the same. So let's
> > use this iterator_range throughout, since we already have it.

> The reason to use a trait it to make it easier to adapt existing
> ranges. Also, containers are ranges, but do not have your additional
> functions, so you need to explicitly wrap them in iterator_ranges
> before wrapping them in another adapter range. With associated_range,
> the wrapping comes implicitly for free.

We can also use a trait. Keep in mind though that from a client's standpoint, both classes are really
equivalent. In fact, if you instantiate an iterator_range for one of the associated_range::iterators,
you really want the associated_range because it is more efficient. Likewise, the corresponding
sub_range should really derive from associated_range. In our company's library, sub_range is a trait so
that sub_range( transform_range ).base() returns the base subrange. I see iterator_range along
similar lines, with the semantics mattering more than the implementation.

This issue is somewhat of a side-track, though.

> I think that these functions should all be friend functions, with
> reasonable default. The reason is the usual one: adaptability of
> existing ranges.

Fine with me. I am not so familiar with the boost.range stuff.

> >   // [actually I am not so sure, I have not tried to eliminate them]
> >   // They could be implented via
(Continue reading)

David Abrahams | 2 Sep 05:50
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Mon Sep 01 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> I wouldn't know how to address this issue without some real-life use
>> cases, though.  This has all become a bit too abstract a discussion for
>> me, and Generic Programming is supposed to be driven by concrete
>> examples.
>
> Isn't my problem concrete enough? I need a stackable difference_range without
> exponential storage bloat. 

Yeah, but why?  What algorithm are you trying to implement that leads
you to think about stacking difference_range?  I don't imagine that a
stacked difference_range is a good way to do anything efficiently.

For example, if you're trying to do compute something like

   A - B - C - D

where A,B,C, and D are sorted ranges, and you're doing it without a
heap, it *will* be needlessly inefficient.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 2 Sep 07:26
Favicon

Re: lifetime of ranges vs. iterators

> > Isn't my problem concrete enough? I need a stackable difference_range without
> > exponential storage bloat. 

> Yeah, but why?  What algorithm are you trying to implement that leads
> you to think about stacking difference_range?  I don't imagine that a
> stacked difference_range is a good way to do anything efficiently.

> For example, if you're trying to do compute something like

>    A - B - C - D

> where A,B,C, and D are sorted ranges, and you're doing it without a
> heap, it *will* be needlessly inefficient.

The RangeEx documentation writes something like "what algorithms are to containers, adaptor ranges are
to ranges". Whenever you are applying one (non-permutating) algorithm to a container, and then another
and then another, you can stack range adaptors. It's like lazy evaluation of a Haskell list. 

Looking through our code for adaptor ranges, I find:

unique_range
union_range
difference_range
filter_range
concat_range

These ranges are all actually used in our program. All these ranges need an end iterator, and any stacking of
any combination of these ranges leads to the problems we are discussing here. I don't find this impractical.

-Arno
(Continue reading)

David Abrahams | 2 Sep 17:02
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>
> The RangeEx documentation writes something like "what algorithms are to
> containers, adaptor ranges are to ranges". Whenever you are applying one
> (non-permutating) algorithm to a container, and then another and then another,
> you can stack range adaptors. It's like lazy evaluation of a Haskell list.

Sheesh; give me a little credit, please.  I know what range adaptors do.

> Looking through our code for adaptor ranges, I find:
>
> unique_range
> union_range
> difference_range
> filter_range
> concat_range
>
> These ranges are all actually used in our program. All these ranges need an end
> iterator, and any stacking of any combination of these ranges leads to the
> problems we are discussing here. I don't find this impractical.

You're missing my point.  Are you indeed making such stacks?  If so,
what are you doing with them?

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
(Continue reading)

Re: lifetime of ranges vs. iterators

On Tue, Sep 2, 2008 at 5:02 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Tue Sep 02 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:
>
>>
>> The RangeEx documentation writes something like "what algorithms are to
>> containers, adaptor ranges are to ranges". Whenever you are applying one
>> (non-permutating) algorithm to a container, and then another and then another,
>> you can stack range adaptors. It's like lazy evaluation of a Haskell list.
>
> Sheesh; give me a little credit, please.  I know what range adaptors do.
>
>> Looking through our code for adaptor ranges, I find:
>>
>> unique_range
>> union_range
>> difference_range
>> filter_range
>> concat_range
>>
>> These ranges are all actually used in our program. All these ranges need an end
>> iterator, and any stacking of any combination of these ranges leads to the
>> problems we are discussing here. I don't find this impractical.
>
> You're missing my point.  Are you indeed making such stacks?  If so,
> what are you doing with them?

The deepest stack I have in my application is 4 adapters deep. The
reasons I do not have deeper stack are:

(Continue reading)

David Abrahams | 2 Sep 20:27
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

>> You're missing my point.  Are you indeed making such stacks?  If so,
>> what are you doing with them?
>
> The deepest stack I have in my application is 4 adapters deep. The
> reasons I do not have deeper stack are:
>
> 1) compile time cost
> 2) lack of auto and decltype for easy factorization (need to 'reify'
> the ranges at function boundaries)
> 3) lack of time to make additional lazy wrappers for standard algorithm.
>
> I expect that with a C++0x compiler 1 [*] and 2 won't be a problem,
> while RangeEx should take care of 3.
>
> The usage is in a real world text processing heavy application.

Great, but what are you doing with these stacks?  Normally, the answer
would be something like "I'm applying the X algorithm to a range of
elements that represent Y but have been adapted to look like Z"

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

(Continue reading)

Re: lifetime of ranges vs. iterators

On Tue, Sep 2, 2008 at 8:27 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Tue Sep 02 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>>> You're missing my point.  Are you indeed making such stacks?  If so,
>>> what are you doing with them?
>>
>> The deepest stack I have in my application is 4 adapters deep. The
>> reasons I do not have deeper stack are:
>>
>> 1) compile time cost
>> 2) lack of auto and decltype for easy factorization (need to 'reify'
>> the ranges at function boundaries)
>> 3) lack of time to make additional lazy wrappers for standard algorithm.
>>
>> I expect that with a C++0x compiler 1 [*] and 2 won't be a problem,
>> while RangeEx should take care of 3.
>>
>> The usage is in a real world text processing heavy application.
>
> Great, but what are you doing with these stacks?  Normally, the answer
> would be something like "I'm applying the X algorithm to a range of
> elements that represent Y but have been adapted to look like Z"
>

Usually is a matter of converting a text document to a point in a
feature vector space as a preprocessing stage:

Most pipelines start with tokenization, normalization, filtering and
hashing, with a 'take' operation at the end.
(Continue reading)

David Abrahams | 3 Sep 15:11
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

>> Great, but what are you doing with these stacks?  Normally, the answer
>> would be something like "I'm applying the X algorithm to a range of
>> elements that represent Y but have been adapted to look like Z"
>>
>
> Usually is a matter of converting a text document to a point in a
> feature vector space as a preprocessing stage:

e.g., for the purposes of search?  Just trying to get a feel for what
you're describing.

> Most pipelines start with tokenization, normalization, filtering and
> hashing, with a 'take' operation at the end.

By 'take' you mean a destructive read?

> The resulting range is usually sorted (which implies breaking the
> laziness), 

Yep.

> unique-ed and augmented with score information (another map).
>
> I very often need to compute set union and intersection of pair of
> these ranges.
> [I do not have yet a lazy adaptor for this (actually I do, but is kind
> of experimental)].
(Continue reading)

Re: lifetime of ranges vs. iterators

On Wed, Sep 3, 2008 at 3:11 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Tue Sep 02 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>>> Great, but what are you doing with these stacks?  Normally, the answer
>>> would be something like "I'm applying the X algorithm to a range of
>>> elements that represent Y but have been adapted to look like Z"
>>>
>>
>> Usually is a matter of converting a text document to a point in a
>> feature vector space as a preprocessing stage:
>
> e.g., for the purposes of search?  Just trying to get a feel for what
> you're describing.

Near duplicate elimination and document clustering in general. Most
clustering algorithms work on representations of documents as points
in a very high dimension space.

>
>> Most pipelines start with tokenization, normalization, filtering and
>> hashing, with a 'take' operation at the end.
>
> By 'take' you mean a destructive read?
>

What do you mean with 'destructive'?

'take(range, n)' returns a range adaptor that iterates over the first
'n' elements of 'range'. This way I do not have to decide how much of
(Continue reading)

David Abrahams | 3 Sep 19:21
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> On Wed, Sep 3, 2008 at 3:11 PM, David Abrahams <dave <at> boostpro.com> wrote:
>>
>> on Tue Sep 02 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>>
>>>> Great, but what are you doing with these stacks?  Normally, the answer
>>>> would be something like "I'm applying the X algorithm to a range of
>>>> elements that represent Y but have been adapted to look like Z"
>>>>
>>>
>>> Usually is a matter of converting a text document to a point in a
>>> feature vector space as a preprocessing stage:
>>
>> e.g., for the purposes of search?  Just trying to get a feel for what
>> you're describing.
>
> Near duplicate elimination and document clustering in general. Most
> clustering algorithms work on representations of documents as points
> in a very high dimension space.

OK.

>>> Most pipelines start with tokenization, normalization, filtering and
>>> hashing, with a 'take' operation at the end.
>>
>> By 'take' you mean a destructive read?
>>
>
(Continue reading)

Arno Schödl | 3 Sep 19:42
Favicon

Re: lifetime of ranges vs. iterators


> > We are simply trying to come up with a simple and clean trick to keep
> > iterator size under control.

> Really?  IIUC so far we're only discussing compressing the size of
> ranges, unless we're willing to pay for indirection inside of iterators
> (I'm not).

No, we also had minimum size iterators. By "minimum size" I mean that each iterator contains the begin and
end base (e.g., all the way up the stack) iterators + all functors involved, stored by value. This is really
equivalent to a single iterator + boost::bind.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

David Abrahams | 3 Sep 20:27
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > We are simply trying to come up with a simple and clean trick to keep
>> > iterator size under control.
>
>> Really?  IIUC so far we're only discussing compressing the size of
>> ranges, unless we're willing to pay for indirection inside of iterators
>> (I'm not).
>
> No, we also had minimum size iterators. By "minimum size" I mean that each
> iterator contains the begin and end base (e.g., all the way up the stack)
> iterators + all functors involved, stored by value. This is really equivalent to
> a single iterator + boost::bind.

Please be specific.  Do you have some way to avoid my what happens in my
description of stacking filter_iterator and strided_iterator?  If so,
what do you get and how is it done?

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Re: lifetime of ranges vs. iterators

On Wed, Sep 3, 2008 at 7:21 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>>
>> Do you think we are breaking the iterator abstraction? How? Or I've
>> misunderstood the question?
>
> Well, if we add "Factorable" or something like it, the iterator
> abstraction gets broken down into smaller bits for the purpose of
> storage, complicating some code [like a generic range for example ;-)]
> and it imposes an extra coding duty on writers of high-quality iterator
> adaptors.  I'm asking if its worth it.
>
>> We are simply trying to come up with a simple and clean trick to keep
>> iterator size under control.
>
> Really?  IIUC so far we're only discussing compressing the size of
> ranges, unless we're willing to pay for indirection inside of iterators
> (I'm not).

But iterators like filter_iterator that need to know iteration
boundaries, can store a range instead of an iterator, so they can
benefit from the compression.

>
>>>> I'm interested in using dynamic iterators in the near future for code
>>>> decoupling
>>>
>>> You mean, like, any_iterator?
(Continue reading)

David Abrahams | 4 Sep 01:36
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> On Wed, Sep 3, 2008 at 7:21 PM, David Abrahams <dave <at> boostpro.com> wrote:
>>
>> on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>>
>>>
>>> Do you think we are breaking the iterator abstraction? How? Or I've
>>> misunderstood the question?
>>
>> Well, if we add "Factorable" or something like it, the iterator
>> abstraction gets broken down into smaller bits for the purpose of
>> storage, complicating some code [like a generic range for example ;-)]
>> and it imposes an extra coding duty on writers of high-quality iterator
>> adaptors.  I'm asking if its worth it.
>>
>>> We are simply trying to come up with a simple and clean trick to keep
>>> iterator size under control.
>>
>> Really?  IIUC so far we're only discussing compressing the size of
>> ranges, unless we're willing to pay for indirection inside of iterators
>> (I'm not).
>
> But iterators like filter_iterator that need to know iteration
> boundaries, can store a range instead of an iterator, so they can
> benefit from the compression.

Oh, I totally missed that; sorry!  Let's see...

(Continue reading)

Re: lifetime of ranges vs. iterators

On Thu, Sep 4, 2008 at 1:36 AM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>> On Wed, Sep 3, 2008 at 7:21 PM, David Abrahams <dave <at> boostpro.com> wrote:
>>>
>>> on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>>>
>>>>
>>>> Do you think we are breaking the iterator abstraction? How? Or I've
>>>> misunderstood the question?
>>>
>>> Well, if we add "Factorable" or something like it, the iterator
>>> abstraction gets broken down into smaller bits for the purpose of
>>> storage, complicating some code [like a generic range for example ;-)]
>>> and it imposes an extra coding duty on writers of high-quality iterator
>>> adaptors.  I'm asking if its worth it.
>>>
>>>> We are simply trying to come up with a simple and clean trick to keep
>>>> iterator size under control.
>>>
>>> Really?  IIUC so far we're only discussing compressing the size of
>>> ranges, unless we're willing to pay for indirection inside of iterators
>>> (I'm not).
>>
>> But iterators like filter_iterator that need to know iteration
>> boundaries, can store a range instead of an iterator, so they can
>> benefit from the compression.
>
> Oh, I totally missed that; sorry!  Let's see...
(Continue reading)

David Abrahams | 4 Sep 04:08
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

>>> But iterators like filter_iterator that need to know iteration
>>> boundaries, can store a range instead of an iterator, so they can
>>> benefit from the compression.
>>
>> Oh, I totally missed that; sorry!  Let's see...
>>
>>  strided_iterator<int*,5> stores range<int*>
>>
>>  sizeof(range<int*>) == 2*sizeof(int*)
>>
>>  filter_iterator<strided_iterator<int*,5> > stores
>>    range<strided_iterator<int*,5> >
>>
>>  range<strided_iterator<int*,5> > is represented as
>>
>>    unique_data<strided_iterator<int*,5> >::type start;
>>    unique_data<strided_iterator<int*,5> >::type finish;
>>    common_data<strided_iterator<int*,5> >::type common;
>>
>>  unique_data<strided_iterator<int*,5> >::type == int*
>>  common_data<strided_iterator<int*,5> >::type == int*
>>
>>  so its size is 3*sizeof(int*)
>>
>> ??  Did I get that right?
>>
>>
(Continue reading)

Re: lifetime of ranges vs. iterators

2008/9/4 David Abrahams <dave <at> boostpro.com>:
>>>> But iterators like filter_iterator that need to know iteration
>>>> boundaries, can store a range instead of an iterator, so they can
>>>> benefit from the compression.
>>>
>>> Oh, I totally missed that; sorry!  Let's see...
>>>
>>>  strided_iterator<int*,5> stores range<int*>
>>>
>>>  sizeof(range<int*>) == 2*sizeof(int*)
>>>
>>>  filter_iterator<strided_iterator<int*,5> > stores
>>>    range<strided_iterator<int*,5> >
>>>
>>>  range<strided_iterator<int*,5> > is represented as
>>>
>>>    unique_data<strided_iterator<int*,5> >::type start;
>>>    unique_data<strided_iterator<int*,5> >::type finish;
>>>    common_data<strided_iterator<int*,5> >::type common;
>>>
>>>  unique_data<strided_iterator<int*,5> >::type == int*
>>>  common_data<strided_iterator<int*,5> >::type == int*
>>>
>>>  so its size is 3*sizeof(int*)
>>>
>>> ??  Did I get that right?
>>>
>>>
>>> If so, it seems like there's still redundant storage.  finish and common
>>> will be representing essentially the same information.  If I were to
(Continue reading)

Arno Schödl | 5 Sep 13:38
Favicon

Re: lifetime of ranges vs. iterators

> void incement() { //similar thing for decrement
>     offset += N;
> }

> void advance(distance_type n) {
>     offset += n * N;
> }

> reference dereference() const {
>     return base[offset];
> }

> distance_type distance_to(filter_iterator rhs) {
>      return ((base - rhs.base) + (offset - rhs.offset)) / N ; // I
might have got the sign wrong
> }

In the case of a base forward_iterator, we can do without any space overhead with the same trick used for the
filter_iterator.  We implement a strided_range and then derive the strided_iterator from it. The
strided_range stores its underlying random_access range, and adapted_range
interface::increment/advance can check the underlying range for empty. It would be like:

increment() {
   for( difference_type i=0; i<N && !base.empty(); ++i ) {
      base.increment(); // the adapted_range::increment which increments begin, base is a range
   }
}

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
(Continue reading)

David Abrahams | 5 Sep 15:39
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Sep 05 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

> In the case of a base forward_iterator, we can do without any space overhead
> with the same trick used for the filter_iterator.  We implement a strided_range
> and then derive the strided_iterator from it. The strided_range stores its
> underlying random_access range, and adapted_range interface::increment/advance
> can check the underlying range for empty. It would be like:
>
> increment() {
>    for( difference_type i=0; i<N && !base.empty(); ++i ) {
>       base.increment(); // the adapted_range::increment which increments begin,
> base is a range
>    }
> }

I don't think you're fully appreciating what a huge impact such a
complex implementation of increment() would have on performance.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 6 Sep 15:52
Favicon

Re: lifetime of ranges vs. iterators

> > increment() {
> >    for( difference_type i=0; i<N && !base.empty(); ++i ) {
> >       base.increment(); // the adapted_range::increment which increments begin,
> > base is a range
> >    }
> > }

> I don't think you're fully appreciating what a huge impact such a
> complex implementation of increment() would have on performance.

Sorry, I meant this as the implementation for forward range. You don't want to use increment for
random_access ranges, obviously. 

Thinking about it some more, I believe the associated_range abstraction is not quite the right one yet. It
works for forward_ranges, which is why I did not notice earlier, but not for bidirectional and random_access_ranges.

The correct abstration is bounded_iterator, which is an iterator + the knowledge of whatever bounds it
needs. The extra functionality offered over iterators is to test overrun of begin/end in
increment/decrement/advance. begin/end may have a bool return value whether increment/decrement
succeeded or not, and advance may return the difference_type that actually got executed until begin/end
was reached.

The bounded_forward_iterator is a bit similar to a range. Its base iterator implementation must store the
current iterator + the end. This is essentially what Giovanni introduced as associated_range.

The bounded_bidrectional_iterator and bounded_random_access_iterator base implementation must
store begin, current and end base iterators.

All iterators needing the underlying iterator bounds can then run on top of bounded_iterators, and are
implement the bounded_iterator interface themselves. I think this eliminates all space overhead of any
(Continue reading)

David Abrahams | 7 Sep 01:32
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sat Sep 06 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > increment() {
>> >    for( difference_type i=0; i<N && !base.empty(); ++i ) {
>> > base.increment(); // the adapted_range::increment which increments begin,
>> > base is a range
>> >    }
>> > }
>
>> I don't think you're fully appreciating what a huge impact such a
>> complex implementation of increment() would have on performance.
>
> Sorry, I meant this as the implementation for forward range. You don't want to
> use increment for random_access ranges, obviously.

I don't see what the traversal category has to do with anything.

> Thinking about it some more, I believe the associated_range abstraction is not
> quite the right one yet. It works for forward_ranges, which is why I did not
> notice earlier, but not for bidirectional and random_access_ranges.
>
> The correct abstration is bounded_iterator, which is an iterator + the knowledge
> of whatever bounds it needs. 

Only if you don't care about eliminating other redundancies.

--

-- 
Dave Abrahams
BoostPro Computing
(Continue reading)

Arno Schödl | 7 Sep 12:02
Favicon

Re: lifetime of ranges vs. iterators

>> > increment() {
>> >    for( difference_type i=0; i<N && !base.empty(); ++i ) {
>> > base.increment(); // the adapted_range::increment which increments begin,
>> > base is a range
>> >    }
>> > }
>
>> I don't think you're fully appreciating what a huge impact such a
>> complex implementation of increment() would have on performance.
>
> Sorry, I meant this as the implementation for forward range. You don't want to
> use increment for random_access ranges, obviously.

>I don't see what the traversal category has to do with anything.

A strided_iterator should use advance on random_access_ranges rather than increment.

> Thinking about it some more, I believe the associated_range abstraction is not
> quite the right one yet. It works for forward_ranges, which is why I did not
> notice earlier, but not for bidirectional and random_access_ranges.
>
> > The correct abstration is bounded_iterator, which is an iterator + the knowledge
> > of whatever bounds it needs. 

> Only if you don't care about eliminating other redundancies.

What are your referring to specificly? If you are referring to storing the predicate or stride length, this
is taken care of because the iterator adaptor stack stores this information only once:

template<I, F>
(Continue reading)

David Abrahams | 7 Sep 14:18
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Sun Sep 07 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> Thinking about it some more, I believe the associated_range abstraction is not
>> quite the right one yet. It works for forward_ranges, which is why I did not
>> notice earlier, but not for bidirectional and random_access_ranges.
>>
>> > The correct abstration is bounded_iterator, which is an iterator + the
> knowledge
>> > of whatever bounds it needs. 
>
>> Only if you don't care about eliminating other redundancies.
>
> What are your referring to specificly? If you are referring to storing the
> predicate or stride length, this is taken care of because the iterator adaptor
> stack stores this information only once:

I'm talking about "vertical" redundancies between layers of the stack.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 7 Sep 17:02
Favicon

Re: lifetime of ranges vs. iterators

>> What are your referring to specificly? If you are referring to storing the
>> predicate or stride length, this is taken care of because the iterator adaptor
>> stack stores this information only once:

>I'm talking about "vertical" redundancies between layers of the stack.

Could you give a specific example? 

Each bounded_iterator stores the information pertinent to its own operation (predicate, stride length,
transform functor, whatever) + its base bounded iterator. The base bounded_iterator stores either two
(for forward iterators: current and end) or three (for bidirectional and random_access: begin, current
and end) base iterators.

The only redundancy that I still see is that you may start with a random_access iterator at the base, but
somewhere in the stack the traversal category is weakened to forward. Or the user of the iterator only uses
forward traversal, although the iterator type would also support bidirectional traversal. Then the
base bounded_iterator stores a begin that is never used. 

I am not sure this is worth optimizing for, but you could put the desired traversal category into the
bounded_iterator template interface:

bounded_iterator< I, traversal_category >

bounded_iterator< char*, forward_traversal > would then be a forward bounded_iterator running on
char*, of size 2x sizeof(char*). It only stores current char* and end char*, instead of begin char*,
current char* and end char*.

Arno

--
(Continue reading)

David Abrahams | 5 Sep 15:38
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Sep 05 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> 2008/9/4 David Abrahams <dave <at> boostpro.com>:
>>>>> But iterators like filter_iterator that need to know iteration
>>>>> boundaries, can store a range instead of an iterator, so they can
>>>>> benefit from the compression.
>>>>
>>>> Oh, I totally missed that; sorry!  Let's see...
>>>>
>>>>  strided_iterator<int*,5> stores range<int*>
>>>>
>>>>  sizeof(range<int*>) == 2*sizeof(int*)
>>>>
>>>>  filter_iterator<strided_iterator<int*,5> > stores
>>>>    range<strided_iterator<int*,5> >
>>>>
>>>>  range<strided_iterator<int*,5> > is represented as
>>>>
>>>>    unique_data<strided_iterator<int*,5> >::type start;
>>>>    unique_data<strided_iterator<int*,5> >::type finish;
>>>>    common_data<strided_iterator<int*,5> >::type common;
>>>>
>>>>  unique_data<strided_iterator<int*,5> >::type == int*
>>>>  common_data<strided_iterator<int*,5> >::type == int*
>>>>
>>>>  so its size is 3*sizeof(int*)
>>>>
>>>> ??  Did I get that right?
>>>>
(Continue reading)

Re: lifetime of ranges vs. iterators

On Fri, Sep 5, 2008 at 3:38 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Fri Sep 05 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>> 2008/9/4 David Abrahams <dave <at> boostpro.com>:
>>>>>> But iterators like filter_iterator that need to know iteration
>>>>>> boundaries, can store a range instead of an iterator, so they can
>>>>>> benefit from the compression.
>>>>>
>>>>> Oh, I totally missed that; sorry!  Let's see...
>>>>>
>>>>>  strided_iterator<int*,5> stores range<int*>
>>>>>
>>>>>  sizeof(range<int*>) == 2*sizeof(int*)
>>>>>
>>>>>  filter_iterator<strided_iterator<int*,5> > stores
>>>>>    range<strided_iterator<int*,5> >
>>>>>
>>>>>  range<strided_iterator<int*,5> > is represented as
>>>>>
>>>>>    unique_data<strided_iterator<int*,5> >::type start;
>>>>>    unique_data<strided_iterator<int*,5> >::type finish;
>>>>>    common_data<strided_iterator<int*,5> >::type common;
>>>>>
>>>>>  unique_data<strided_iterator<int*,5> >::type == int*
>>>>>  common_data<strided_iterator<int*,5> >::type == int*
>>>>>
>>>>>  so its size is 3*sizeof(int*)
>>>>>
>>>>> ??  Did I get that right?
(Continue reading)

David Abrahams | 5 Sep 18:55
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Sep 05 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

>> You're completely right about that, and there are even additional
>> reasons that you didn't mention, as I realized a couple days ago.  I
>> didn't want to bring it up, though, because it doesn't change the
>> fundamental reality.  Just replace strided_iterator by another layer of
>> filter_iterator.
>
> Hum, but as long as an iterator contains only a range, you can always
> encapsulate it with no space overhead. a
> filter_iterator<filter_iterator<T* > >  has sizeof(T*2) (assuming a
> stateless predicate), which is optimal.

First of all, there's other kinds of data that's redundant across pairs
of iterators in a range hey, like the predicate!

But let's ignore that for a moment.  The range compression formula is:

      sizeof(range<X>) == 2*sizeof(X) - sizeof(common_data<X>::type)

    so when X is filter_iterator<Y>:

       sizeof(X) is 2*sizeof(Y)
       common_data<X>::type == Y

    so:

       sizeof(range<X>) == 2*(2*sizeof(Y)) - sizeof(Y) == 3*sizeof(Y)

(Continue reading)

Re: lifetime of ranges vs. iterators

On Fri, Sep 5, 2008 at 6:55 PM, David Abrahams <dave <at> boostpro.com> wrote:
>
> on Fri Sep 05 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>>> You're completely right about that, and there are even additional
>>> reasons that you didn't mention, as I realized a couple days ago.  I
>>> didn't want to bring it up, though, because it doesn't change the
>>> fundamental reality.  Just replace strided_iterator by another layer of
>>> filter_iterator.
>>
>> Hum, but as long as an iterator contains only a range, you can always
>> encapsulate it with no space overhead. a
>> filter_iterator<filter_iterator<T* > >  has sizeof(T*2) (assuming a
>> stateless predicate), which is optimal.
>
> First of all, there's other kinds of data that's redundant across pairs
> of iterators in a range hey, like the predicate!

that's not a problem, a filter_range would store the predicate only once.

>
> But let's ignore that for a moment.  The range compression formula is:
>
>      sizeof(range<X>) == 2*sizeof(X) - sizeof(common_data<X>::type)
>
>    so when X is filter_iterator<Y>:
>
>       sizeof(X) is 2*sizeof(Y)
>       common_data<X>::type == Y

(Continue reading)

David Abrahams | 5 Sep 20:36
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Fri Sep 05 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

> Oh! now I see where is where our inpedence mismatch is!. You want to
> store three pointers in filter_range because you correctly want this
> transformation to be lossless:
>
> pair<filter_iterator, filter_iterator> a;
>
>   filter_range f(a.begin, a.end)

I think you mean

    filter_range f(a.first, a.second)

And I'll assume from what you write below this not adding an additional
layer of filtering; it is just storing a.first and a.second (compressed).

>   assert(f.begin() == a);
>   assert(f.end() == b);

I think you mean

   assert(f.begin() == a.first);
   assert(f.end() == a.second);

> i.e. the hidden common knowledge  of a.begin and a.end of the
> effective end of underlying range (let's call it real_end) is
> preserved.

(Continue reading)

Arno Schödl | 2 Sep 17:38
Favicon

Re: lifetime of ranges vs. iterators

> > The RangeEx documentation writes something like "what algorithms are to
> > containers, adaptor ranges are to ranges". Whenever you are applying one
> > (non-permutating) algorithm to a container, and then another and then another,
> > you can stack range adaptors. It's like lazy evaluation of a Haskell list.

> Sheesh; give me a little credit, please.  I know what range adaptors do.

So why are you asking these questions then? With smart compilers, generic programming is supposed to
produce code that is pretty close to handcrafted code. That is what people expect, and that is what generic
programming is sold on. The current range adaptors were far from it. And even if stacks don't exceed size 3
or 4 in practice, which is all I have in my code, why produce code that is factor 8 or 16 slower or more
storage-bloated than necessary? As we have seen, the reason was not that the problem is intractable with
generic programming. It just took a little thought to do it right.

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Neil Groves | 2 Sep 18:00

Re: lifetime of ranges vs. iterators

Arno,

On Tue, Sep 2, 2008 at 4:38 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:

> > > The RangeEx documentation writes something like "what algorithms are to
> > > containers, adaptor ranges are to ranges". Whenever you are applying
> one
> > > (non-permutating) algorithm to a container, and then another and then
> another,
> > > you can stack range adaptors. It's like lazy evaluation of a Haskell
> list.
>
> > Sheesh; give me a little credit, please.  I know what range adaptors do.
>
> So why are you asking these questions then? With smart compilers, generic
> programming is supposed to produce code that is pretty close to handcrafted
> code. That is what people expect, and that is what generic programming is
> sold on. The current range adaptors were far from it. And even if stacks
> don't exceed size 3 or 4 in practice, which is all I have in my code, why
> produce code that is factor 8 or 16 slower or more storage-bloated than
> necessary? As we have seen, the reason was not that the problem is
> intractable with generic programming. It just took a little thought to do it
> right.
>

Please do not get frustrated. We all want to improve Range / RangeEx. I have
to apologise for not offering to modify the code sooner. I have been
watching, and learning. Once I have understood the design alternatives and
have fully comprehended the optimum solution I will implement the suggested
improvements. It looks like I will need to coordinate with the Iterator
(Continue reading)

Arno Schödl | 2 Sep 18:32
Favicon

Re: lifetime of ranges vs. iterators

> optimum solution they are at least implementations of the functionality. I
> would gladly accept superior contributions ;-). I am also happy to spend
> more time improving them myself. I have to apologise for slow response
> times, my full-time job simply doesn't allow me enough time to contribute to
> this discussion as much as I probably ought to. You have made some very
> valid technical points, let's focus on these please.

My last post was not intended to put pressure on anyone to change anything, and I am very grateful for having
boost. But Dave evidently thinks I am wasting everyone's time, and I beg to differ.

If you agree that the general direction, self-contained iterators that use funky ranges to represent
themselves efficiently, is right, I would bit by bit change my implementations of adaptor ranges and
share them with you. I have not yet contributed to Boost, so probably my code is non-conformant all over the
place, but it is a start.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Neil Groves | 2 Sep 18:45

Re: lifetime of ranges vs. iterators

On Tue, Sep 2, 2008 at 5:32 PM, Arno Schödl <aschoedl <at> think-cell.com> wrote:

> > optimum solution they are at least implementations of the functionality.
> I
> > would gladly accept superior contributions ;-). I am also happy to spend
> > more time improving them myself. I have to apologise for slow response
> > times, my full-time job simply doesn't allow me enough time to contribute
> to
> > this discussion as much as I probably ought to. You have made some very
> > valid technical points, let's focus on these please.
>
> My last post was not intended to put pressure on anyone to change anything,
> and I am very grateful for having boost. But Dave evidently thinks I am
> wasting everyone's time, and I beg to differ.
>

> If you agree that the general direction, self-contained iterators that use
> funky ranges to represent themselves efficiently, is right, I would bit by
> bit change my implementations of adaptor ranges and share them with you. I
> have not yet contributed to Boost, so probably my code is non-conformant all
> over the place, but it is a start.

I would really like to see some of the changes that you make, and that would
help me apply a consistent set of improvements to other ranges and adaptors
that could benefit. Once things are a bit more concrete we can measure and
compare the storage and performance benefits. There will be very little
argument with objective measurements. We can work together to make them work
and compliant with the intention of incorporating the changes into RangeEx.

>
(Continue reading)

David Abrahams | 2 Sep 20:39
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> optimum solution they are at least implementations of the functionality. I
>> would gladly accept superior contributions ;-). I am also happy to spend
>> more time improving them myself. I have to apologise for slow response
>> times, my full-time job simply doesn't allow me enough time to contribute to
>> this discussion as much as I probably ought to. You have made some very
>> valid technical points, let's focus on these please.
>
> My last post was not intended to put pressure on anyone to change
> anything, and I am very grateful for having boost. But Dave evidently
> thinks I am wasting everyone's time,

I don't think I said anything of the sort.  

I think you're asking interesting questions that might lead to some
important work.  However, it seems to me that you already have a general
answer in mind, and it might be better to back up and look at the real
problems you want to solve before moving on to the answer.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Abrahams | 2 Sep 20:36
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > The RangeEx documentation writes something like "what algorithms are to
>> > containers, adaptor ranges are to ranges". Whenever you are applying one
>> > (non-permutating) algorithm to a container, and then another and then
> another,
>> > you can stack range adaptors. It's like lazy evaluation of a Haskell list.
>
>> Sheesh; give me a little credit, please.  I know what range adaptors do.
>
> So why are you asking these questions then? 

I'm trying to find out what the specific example applications are.  I
have a suspicion that range adapters or single-level iterator adaptors
may not be the best way to address these applications and may even
sacrifice significant performance, but I can't tell until I know what
the applications are.  Just so, single-level iterators (or ordinary
range adapters, which don't add anything conceptually to iterators) are
not the best way to address iteration over hierarchical data structures,
which is why we have segmented iterators.

> With smart compilers, generic programming is supposed to produce code
> that is pretty close to handcrafted code. That is what people expect,
> and that is what generic programming is sold on.

Generic Programming is supposed to be driven by concrete implementations
and real use cases.  http://www.stlport.org/resources/StepanovUSA.html:

  It is as if mathematicians would start with axioms. You do not start
(Continue reading)

Arno Schödl | 2 Sep 23:39
Favicon

Re: lifetime of ranges vs. iterators

> I'm trying to find out what the specific example applications are.  I
> have a suspicion that range adapters or single-level iterator adaptors
> may not be the best way to address these applications and may even
> sacrifice significant performance, but I can't tell until I know what
> the applications are.  Just so, single-level iterators (or ordinary
> range adapters, which don't add anything conceptually to iterators) are
> not the best way to address iteration over hierarchical data structures,
> which is why we have segmented iterators.

This is a typical one, copied straight from our code:

		grdlns->InsertGridlines( 
			vecgvepsSource,
			make_first_range(vecpairanchorsrcgrdln), 
			GridlineInsertionPositions( 
				make_transform_range(
					make_transform_range( 
						make_first_range( vecpairanchorsrcgrdln ),
						boost::mem_fn(&_vector< Ref<CGridlineAnchor> >::begin)
					),
					boost::mem_fn( &CGridlineAnchor::Binding )
				), 
				vecgrdlnSnapped),
			vecgrdlnSnapped 
		);

This is a case in point: I did not write this code, and in the absence of specific guidelines against, the
programmer decided to stack transform_range. In this case, performance-wise, this is no better or worse
than using a single transform_range with boost::bind. Likewise, people may innocently write

(Continue reading)

David Abrahams | 3 Sep 15:05
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Tue Sep 02 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> I'm trying to find out what the specific example applications are.  I
>> have a suspicion that range adapters or single-level iterator adaptors
>> may not be the best way to address these applications and may even
>> sacrifice significant performance, but I can't tell until I know what
>> the applications are.  Just so, single-level iterators (or ordinary
>> range adapters, which don't add anything conceptually to iterators) are
>> not the best way to address iteration over hierarchical data structures,
>> which is why we have segmented iterators.
>
> This is a typical one, copied straight from our code:
>
> 		grdlns->InsertGridlines( 
> 			vecgvepsSource,
> 			make_first_range(vecpairanchorsrcgrdln), 
> 			GridlineInsertionPositions( 
> 				make_transform_range(
> 					make_transform_range( 
> 						make_first_range(
> vecpairanchorsrcgrdln ),
> 						boost::mem_fn(&_vector<
> Ref<CGridlineAnchor> >::begin)
> 					),
> 					boost::mem_fn( &CGridlineAnchor::Binding
> )
> 				), 
> 				vecgrdlnSnapped),
> 			vecgrdlnSnapped 
(Continue reading)

Arno Schödl | 3 Sep 17:08
Favicon

Re: lifetime of ranges vs. iterators

> > rng | filtered( funcA ) 
> >     | filtered( funcB ) 
> >     | filtered( funcC ) 
> >     | filtered( funcD ) 
> >     | filtered( funcE )

> 1. _Do_ they write that?  

Why not? They did for transform_range, and the notation is getting easier...

> 2. If they do that, I believe they are still paying an enormous
>    unnecessary runtime penalty even with your optimization because of
>    the endpoint checks in each layer of the composite iterator or
>    range.  The only way IIUC to avoid such a penalty is to compress the
>    filter adapters into one, essentially by using expression
>    templates.

See answer to 4.

> 3. I am not yet convinced that you are suffering a significant
>    performance penalty due to the wasted space in real applications.
>    Have you measured it?  Unless you're actually *storing* lots of
>    these stacked iterators or passing them around by value in your
>    inner loops, it seems unlikely to make much of a difference.

Giovanni has seen the quadratic stack expansions. If you are worried about performance of pointers, why
not worry about the performance of wrappers around pointers?

> 4. I can understand wanting to improve these components so that they
>    can be efficient under more conditions, and avoiding the redundant
(Continue reading)

David Abrahams | 3 Sep 19:09
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > rng | filtered( funcA ) 
>> >     | filtered( funcB ) 
>> >     | filtered( funcC ) 
>> >     | filtered( funcD ) 
>> >     | filtered( funcE )
>
>> 1. _Do_ they write that?  
>
> Why not? They did for transform_range, and the notation is getting easier...

Yes, I never contested that they *could* write it.

I'm still asking you the same question: what real life computations
demand a solution to this problem?

>> 2. If they do that, I believe they are still paying an enormous
>>    unnecessary runtime penalty even with your optimization because of
>>    the endpoint checks in each layer of the composite iterator or
>>    range.  The only way IIUC to avoid such a penalty is to compress the
>>    filter adapters into one, essentially by using expression
>>    templates.
>
> See answer to 4.
>
>> 3. I am not yet convinced that you are suffering a significant
>>    performance penalty due to the wasted space in real applications.
>>    Have you measured it?  Unless you're actually *storing* lots of
(Continue reading)

Arno Schödl | 3 Sep 19:16
Favicon

Re: lifetime of ranges vs. iterators

> I'm not there yet, but I know enough about the problem now that I
> wouldn't "buy" a partial solution.

Before I start playing around with the problem, is my throwing increment_and_dereference optimization
fair game? I believe I can't do without.

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 3 Sep 19:19
Favicon

Re: lifetime of ranges vs. iterators

> > I'm not there yet, but I know enough about the problem now that I
> > wouldn't "buy" a partial solution.

> Before I start playing around with the problem, is my throwing increment_and_dereference optimization
fair game? I believe I can't > do without.

Since dereference does not need to end-check, throwing increment() is probably the basic building block.
Fair game?

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Abrahams | 3 Sep 19:24
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > I'm not there yet, but I know enough about the problem now that I
>> > wouldn't "buy" a partial solution.
>
>> Before I start playing around with the problem, is my throwing
> increment_and_dereference optimization fair game? I believe I can't > do
> without.
>
> Since dereference does not need to end-check, throwing increment() is
> probably the basic building block. Fair game?

Sorry, I really don't know what you're asking.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 3 Sep 19:36
Favicon

Re: lifetime of ranges vs. iterators

>>> > I'm not there yet, but I know enough about the problem now that I
>>> > wouldn't "buy" a partial solution.
>>
>>> Before I start playing around with the problem, is my throwing
>> increment_and_dereference optimization fair game? I believe I can't > do
>> without.
>>
>> Since dereference does not need to end-check, throwing increment() is
>> probably the basic building block. Fair game? 
>
>Sorry, I really don't know what you're asking.

A basic multiple-filter loop is this:

filter_increment {
  for(;;) {
    ++i;
    if( empty() || predA( dereference() && predB( dereference ) && predC( dereference ) && predD( dereference
) ) break;
    // <--- towards base_range    *** function stack ***    towards outermost filter_iterator ---->
  }
}

If empty() evaluates to true, all pred* are skipped. This "skipping" in recursion needs exceptions.
Exceptions are the only means C++ gives us to "jump" in a recursive environment. At least in Win32,
exceptions are slow, probably too slow to be practical. I heard that table-based exceptions are much
faster, but I have no practical experience.

Before spending time working on the problem you posed, I want to make sure than my solution does not get
trashed because I use exceptions. If we want to avoid them, I would say that there is no way to get optimal
(Continue reading)

David Abrahams | 3 Sep 20:26
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>>>> > I'm not there yet, but I know enough about the problem now that I
>>>> > wouldn't "buy" a partial solution.
>>>
>>>> Before I start playing around with the problem, is my throwing
>>> increment_and_dereference optimization fair game? I believe I can't > do
>>> without.
>>>
>>> Since dereference does not need to end-check, throwing increment() is
>>> probably the basic building block. Fair game? 
>>
>>Sorry, I really don't know what you're asking.
>
> A basic multiple-filter loop is this:
>
> filter_increment {
>   for(;;) {
>     ++i;
>     if( empty() || predA( dereference() && predB( dereference ) && predC(
> dereference ) && predD( dereference ) ) break;
>     // <--- towards base_range *** function stack *** towards outermost
> filter_iterator ---->
>   }
> }
>
> If empty() evaluates to true, all pred* are skipped. 

Uh-huh...
(Continue reading)

Arno Schödl | 3 Sep 21:14
Favicon

Re: lifetime of ranges vs. iterators

> > This "skipping" in recursion needs exceptions. Exceptions are the only
> > means C++ gives us to "jump" in a recursive environment.

> I don't know what recursion you are referring to.

The stack of adaptors calling each other.

> Only for the non-exceptional path.

Too bad.

> but I have no practical experience.

> My advice: forget exceptions.  They're totally inappropriate in this
> context IMO.

IMO, you need some sort of jump to solve the repeated end-checking problem. If the base iterator end check is
nested into functions, the only way to transport this information out is via exceptions. If you don't like
exceptions, the "I am at the end of the underlying sequence" information must be generated at the
outermost level of the function tree. If it is generated anywhere else, there is simply no way to transport
the information out without checking the return value of a function, which is the extra check we want to
avoid. Do we agree?

Arno

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
(Continue reading)

David Abrahams | 4 Sep 01:46
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> > This "skipping" in recursion needs exceptions. Exceptions are the only
>> > means C++ gives us to "jump" in a recursive environment.
>
>> I don't know what recursion you are referring to.
>
> The stack of adaptors calling each other.

That's not recursion, FWIW.

>> Only for the non-exceptional path.
>
> Too bad.
>
>> but I have no practical experience.
>
>> My advice: forget exceptions.  They're totally inappropriate in this
>> context IMO.
>
> IMO, you need some sort of jump to solve the repeated end-checking
> problem. 

I think if you hand-code an iterator that is semantically equivalent to
the one that results from one of your stackings, you will see that no
such "jump" is required.

> If the base iterator end check is nested into functions, the only way
> to transport this information out is via exceptions. If you don't like
(Continue reading)

Steven Watanabe | 4 Sep 05:29

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
> I think the optimizer can remove some redundant tests if the functions
> are inlined.  But, that said, I am becoming more and more convinced as
> this discussion goes on that compressing stacked iterator adaptors is
> barking up the wrong tree if you care about efficiency.  The difference
> between the code generated in the abstracted case and the code you'd
> write by hand is simply too great.
>
> No, I don't have a picture of a better solution yet.  But it's an
> interesting problem.
>   

Here are some measurements for my current version of filter_iterator

In brief:

4 function objects containing two integers each:  3.39 seconds
4 stateless function objects: 2.813 seconds
4 stateless function objects combined using phoenix's && rather than 
stacked iterators: 2.563 seconds
1 stateless function object 2.015 seconds
raw for loop using vector iterators 2.516 seconds
raw for loop using pointers: 1.703 seconds

These were all doing the same calculation over the same underlying vector.

In Christ,
Steven Watanabe
(Continue reading)

Michael Marcin | 4 Sep 20:51

Re: lifetime of ranges vs. iterators

Steven Watanabe wrote:
> AMDG
> 
> David Abrahams wrote:
>> I think the optimizer can remove some redundant tests if the functions
>> are inlined.  But, that said, I am becoming more and more convinced as
>> this discussion goes on that compressing stacked iterator adaptors is
>> barking up the wrong tree if you care about efficiency.  The difference
>> between the code generated in the abstracted case and the code you'd
>> write by hand is simply too great.
>>
>> No, I don't have a picture of a better solution yet.  But it's an
>> interesting problem.
>>   
> 
> Here are some measurements for my current version of filter_iterator
> 
> In brief:
> 
> 4 function objects containing two integers each:  3.39 seconds
> 4 stateless function objects: 2.813 seconds
> 4 stateless function objects combined using phoenix's && rather than 
> stacked iterators: 2.563 seconds
> 1 stateless function object 2.015 seconds
> raw for loop using vector iterators 2.516 seconds
> raw for loop using pointers: 1.703 seconds
> 
> These were all doing the same calculation over the same underlying vector.
> 

(Continue reading)

Michael Marcin | 4 Sep 21:11

Re: lifetime of ranges vs. iterators

Michael Marcin wrote:
> 
> I ran your attached test on VC9 SP1 default console app release mode 
> with _SECURE_SCL=0 on my Core 2 Quad Q6600 2.4Ghz with 4GB of RAM
> 
> 50.702 seconds
> 19.984 seconds
> 48.453 seconds
> 13.343 seconds
> 4.313 seconds
> 2.953 seconds
> Press any key to continue . . .
> 
> Are these all truly doing the same work?
> 

I looked at the generated code to see why the timing were so drastically 
different and I realized the compiler wasn't inlining a lot of trivial 
functions.

I ran it again after I flipped inline function expansion from default to 
Any Suitable

9.297 seconds
5.812 seconds
8.75 seconds
4.688 seconds
2.922 seconds
3.016 seconds
Press any key to continue . . .
(Continue reading)

Steven Watanabe | 4 Sep 05:11

Re: lifetime of ranges vs. iterators

AMDG

Arno Schödl wrote:
> Before spending time working on the problem you posed, I want to make sure than my solution does not get
trashed because I use exceptions. If we want to avoid them, I would say that there is no way to get optimal
performance with stacked iterators, and I would not even try.
>   

It ought to be possible to propagate a bool up the stack
indicating whether the increment just reached the end.

In principle, if the compile inlines all the functions, then
it ought to be able to avoid any extra boolean test.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Arno Schödl | 4 Sep 10:55
Favicon

Re: lifetime of ranges vs. iterators

> It ought to be possible to propagate a bool up the stack
> indicating whether the increment just reached the end.

> In principle, if the compile inlines all the functions, then
> it ought to be able to avoid any extra boolean test.

I thought about what can be done in principle, without restricting myself to iterators, to optimize the
kind of nested functions on sequences we are discussing. I want to start with the end() check problem in
increment. Let's consider a toy example:

... F2( T2 ( F1( T1( R ) ) ) )

Handcrafted code would look like this. I wrote it with gotos to avoid pre-judgement:

increment() {
next:
   if( base.empty() ) return;
   base.increment();

   T1::reference t1=T1( *base );
   if( !F1( t1 ) ) goto next;

   T2::reference t2=T2( t1 );
   if( !F2( t2 ) ) goto next;

   ....
};

Since we will use function templates to generate this code, we must now organize the code into the blocks we
want to create. We basially have two choices, first the obvious one:
(Continue reading)

David Abrahams | 3 Sep 19:23
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

>> I'm not there yet, but I know enough about the problem now that I
>> wouldn't "buy" a partial solution.
>
> Before I start playing around with the problem, is my throwing
> increment_and_dereference optimization fair game? I believe I can't do
> without.

Fair game, sure, but AFAICT it gains us nothing.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Arno Schödl | 3 Sep 17:12
Favicon

Re: lifetime of ranges vs. iterators

As with these end checks, what if dereference would throw exceptions? End checks gone, all the way up the stack...

Arno 

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Steven Watanabe | 3 Sep 17:24

Re: lifetime of ranges vs. iterators

AMDG

Arno Schödl wrote:
> As with these end checks, what if dereference would throw exceptions? End checks gone, all the way up the stack...
>   

Unfortunately not.

This simply moves the check into the dereference operation
Also, remember that a filter_iterator dereferences iterator
in it's operator++, so you the number of comparisons is unchanged.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Abrahams | 3 Sep 18:47
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:

> As with these end checks, what if dereference would throw exceptions? End checks
> gone, all the way up the stack...

The problem happens way before dereference: it's the undefined behavior
that results from even moving the underlying iterator past the end of
the underlying sequence.

--

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Steven Watanabe | 3 Sep 20:37

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
> on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:
>
>   
>> As with these end checks, what if dereference would throw exceptions? End checks
>> gone, all the way up the stack...
>>     
>
> The problem happens way before dereference: it's the undefined behavior
> that results from even moving the underlying iterator past the end of
> the underlying sequence.
>   

Using Arno's suggestion, dereferencing the end iterator
ought to throw, which happens before we increment
off the end, right?  It sounds like he's proposing making
iterators work the way they do in python.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Abrahams | 3 Sep 20:55
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Wed Sep 03 2008, Steven Watanabe <watanabesj-AT-gmail.com> wrote:

> AMDG
>
> David Abrahams wrote:
>> on Wed Sep 03 2008, Arno Schödl <aschoedl-AT-think-cell.com> wrote:
>>
>>   
>>> As with these end checks, what if dereference would throw exceptions? End
> checks
>>> gone, all the way up the stack...
>>>     
>>
>> The problem happens way before dereference: it's the undefined behavior
>> that results from even moving the underlying iterator past the end of
>> the underlying sequence.
>>   
>
> Using Arno's suggestion, dereferencing the end iterator
> ought to throw, which happens before we increment
> off the end, right?  

The *underlying* iterator?  How are you going to get that to throw if
it's a pointer?

> It sounds like he's proposing making iterators work the way they do in
> python.

Sorta, yeah.
(Continue reading)

Steven Watanabe | 3 Sep 21:13

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
>> Using Arno's suggestion, dereferencing the end iterator
>> ought to throw, which happens before we increment
>> off the end, right?  
>>     
>
> The *underlying* iterator?  How are you going to get that to throw if
> it's a pointer?

You would have to wrap it.  Any solution using iterators adapters
is going to have to rely on detecting whether an iterator knows
it's own end.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Steven Watanabe | 1 Sep 20:37

Re: lifetime of ranges vs. iterators

AMDG

David Abrahams wrote:
> on Mon Sep 01 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:
>
>   
>> Need to think about it. Could you give a practical example of an
>> iterator adapter needing three iterators?
>>     
>
> E.g., a bidirectional filtered_iterator needs to be prevented from
> stepping off both ends of the underlying data to avoid undefined
> behavior.
>   

Do we really have to worry about stepping off the beginning?
I mean, you aren't allowed to decrement the iterator at the
beginning of a range, anyway, and if you decrement any other
iterator, the predicate is guaranteed to stop it before it goes
off the beginning.

> On the other hand, you shouldn't have to pay for those endpoint
> comparisons all the time, which tells me that something may be wrong
> with the design.
>   

Hmm.  Iterating over a range using a filtered_iterator
on an underlying range of size n for which the predicate
is true on m elements results in m + n + 2 comparisons
of the underlying iterator and n calls to the predicate.
(Continue reading)

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 8:37 PM, Steven Watanabe <watanabesj <at> gmail.com> wrote:
> AMDG
>
> David Abrahams wrote:
>>
>> on Mon Sep 01 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com>
>> wrote:
>>
>>
>>>
>>> Need to think about it. Could you give a practical example of an
>>> iterator adapter needing three iterators?
>>>
>>
>> E.g., a bidirectional filtered_iterator needs to be prevented from
>> stepping off both ends of the underlying data to avoid undefined
>> behavior.
>>
>
> Do we really have to worry about stepping off the beginning?
> I mean, you aren't allowed to decrement the iterator at the
> beginning of a range, anyway, and if you decrement any other
> iterator, the predicate is guaranteed to stop it before it goes
> off the beginning.

That's right! It can only be one of three cases: or the range is empty
and you can't move at all; or you are at the beginning and you can't
go back anyway; or it exist at least one element before the current
one such as the predicate is true.

(Continue reading)

David Abrahams | 1 Sep 18:53
Favicon
Gravatar

Re: lifetime of ranges vs. iterators


on Mon Sep 01 2008, "Giovanni Piero Deretta" <gpderetta-AT-gmail.com> wrote:

>>> You could provide an adapted_iterator and also an adapted_range.
>>
>> My point is that the adapted_range would then have semantically
>> equivalent iterators with a different representation (and an extra
>> indirection), which seems like a waste.
>
> hum, which indirection?

The iterators have to refer to the common data somehow.

> <...>
>>> Do you then still need a factored iterator?
>>
>> You need to be able to take two adapted iterators and turn them into a
>> range.  Do you want that range to contain redundant data?  I don't.
>
> Hei, maybe this is all that we need! Let's have a metafunction that
> given an iterator type returns its preferred range type (the
> associated range). The default might be boost.range or even std::pair.
>  The developer of an iterator adaptor, can specialize the metafunction
> so that the associate range is the optimum representation for an
> iterator pair.
>
> Do you think this would be enough? It seems too simple to me, so
> probably I'm missing something...

It is quite attractive, I must say.  I'll give it some thought.
(Continue reading)

Thorsten Ottosen | 1 Sep 10:18

Re: lifetime of ranges vs. iterators

Giovanni Piero Deretta skrev:

>>> Stacking them naively would have 2^N storage
>>> growth, rather than 4^N, and we still don't need to worry about range
>>> lifetime.
>> I've been trying to say that generic algorithms *can't* worry about the
>> range lifetime; they have to assume that the iterators they're passed
>> will not be invalidated before they're destroyed.
> 
> I think that the OP is talking about a different problem: let say you
> want to name the result of stacked applications of range adaptors:
> 
> template<typename Range>
> void my_algo(Range& r) {
>   auto stacked_range = r | filtered(filter) | mapped(mapper);
>   // use stacked_range
> }
> 
> This may lead iterator lifetime problems, not because r iterators may
> be invalid, but because the iterators of the range returned by
> filtered(...) may become invalid as that temporary has been
> destructed. 

I don't see how this happens. The iterators are stored by value in each 
new range that is generated.

-Thorsten
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

(Continue reading)

Arno Schödl | 1 Sep 10:38
Favicon

Re: lifetime of ranges vs. iterators

> This may lead iterator lifetime problems, not because r iterators may
> be invalid, but because the iterators of the range returned by
> filtered(...) may become invalid as that temporary has been
> destructed. 

> I don't see how this happens. The iterators are stored by value in each 
> new range that is generated.

Correct, but in the case of difference_range, storing iterators by value leads to 2^N storage bloat when
stacking N difference_iterators. You can avoid this with factorable iterators (Dave) or by storing
ranges by value and avoiding copying containers either by explicitly wrapping containers as "by ref"
(Giovanni) or by range trait (me).

--
Dr. Arno Schoedl · aschoedl <at> think-cell.com 
Technical Director 

think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany 
http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091
Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Re: lifetime of ranges vs. iterators

On Mon, Sep 1, 2008 at 10:18 AM, Thorsten Ottosen
<thorsten.ottosen <at> dezide.com> wrote:
> Giovanni Piero Deretta skrev:
>
>>>> Stacking them naively would have 2^N storage
>>>> growth, rather than 4^N, and we still don't need to worry about range
>>>> lifetime.
>>>
>>> I've been trying to say that generic algorithms *can't* worry about the
>>> range lifetime; they have to assume that the iterators they're passed
>>> will not be invalidated before they're destroyed.
>>
>> I think that the OP is talking about a different problem: let say you
>> want to name the result of stacked applications of range adaptors:
>>
>> template<typename Range>
>> void my_algo(Range& r) {
>>  auto stacked_range = r | filtered(filter) | mapped(mapper);
>>  // use stacked_range
>> }
>>
>> This may lead iterator lifetime problems, not because r iterators may
>> be invalid, but because the iterators of the range returned by
>> filtered(...) may become invalid as that temporary has been
>> destructed.
>
> I don't see how this happens. The iterators are stored by value in each new
> range that is generated.
>

(Continue reading)

Arno Schödl | 29 Aug 17:21
Favicon

Re: lifetime of ranges vs. iterators

David says:

>I've been trying to say that generic algorithms *can't* worry about the
>range lifetime; they have to assume that the iterators they're passed
>will not be invalidated before they're destroyed. ...
>Bleah (sorry).  Iterators in general shouldn't be burdened with the
>requirement to know the limits of the sequence, and if you go back and ...
>My advice: make sure you keep in mind the whole problem domain.  You can't 
>understand the reason that C++ iterators are unlike those in other languages 
>without looking at the algorithms.

What you are saying about iterators is all true, and they work wonderfully. The problem that you
acknowledged, which is exactly the problem that I have when designing difference_range, is that a _pair_
of iterators often has redundant information, which in the case of stacking difference_ranges leads
either to storage bloat or to storage lifetime issues because we need a place to store this redundant
information. The universal end iterator fixes this problem for forward ranges, but not for
bidirectional or random access ranges.

Giovanni says:

>Anyways, the solution is simple: if you want your adaptor to survive the original wrapped range, you have
>to store a copy of the wrapped range in the adaptor. As an added benefit, the adaptor size will be the
>smallest possible for deep sequences of stacked adaptors.

This is a good solution, but RangeEx is not designed that way, and it is a big departure from regular
containers. I am happy with it if we decide that this is the way to push for the next C++ standard:-)

Arno

--
(Continue reading)


Gmane