Pass iterator into thread

[bitrex]

The poster "Johannes Goller" here gives an example of some code to
"chunk" a STL container so a different section could be passed to a
different thread of execution for processing, for example.

http://stackoverflow.com/questions/14226952/partitioning-batch-chunk-a-container-into-equal-sized-pieces-using-std-algorithm

However, he doesn't give any example on how this is accomplished in
practice with say a boost::thread or a std::thread. I've been trying to
pass what's returned by accessing "chunks[0].first, chunks[0].second" as
a parameter into a boost::thread callback just by value and am getting
difficult to interpret errors like:

/usr/include/c++/5/functional|1505|error: no type named ‘type’ in ‘class
std::result_of<void
(*(__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>...

etc.

[Ben Bacarisse]

You might get lucky with an answer but you will stand a much better
chance by posting the code here. If possible, prepare a minimal example
that produces the error. Very often simply trying to produce that small
example leads you to the solution yourself.

--
Ben.

[bitrex]

Thanks, okay. Consider the following example:

#include <vector>
#include <utility>
#include <algorithm>
#include <cstdint>
#include "boost/thread/thread.hpp"

template <typename T>
std::vector<std::pair<T, T>> chunk(T range_from, T range_to,
const std::ptrdiff_t num);

void process(std::vector<std::string>::iterator first,
std::vector<std::string>::iterator last,
std::vector<std::string>& thread_data);

int main() {
using std::vector;
using std::string;
using boost::thread;

const size_t container_size = 10;
const size_t num_threads = 2;

vector<vector<string>> vec_string(container_size);

std::generate_n(std::begin(vec_string), container_size, []() {
vector<string> v{"A", "B", "C", "D"};
return v;
});

std::array<thread, num_threads> t;

auto chunks =
chunk(std::begin(vec_string), std::end(vec_string), num_threads);

for (size_t i = 0; i < num_threads; ++i) {
t[i] = thread(process, chunks[i].first, chunks[i].second,
std::ref(vec_string));
}

// Join the threads with the main thread
for (size_t i = 0; i < num_threads; ++i) {
t[i].join();
}

return 0;
}

void process(std::vector<std::string>::iterator& first,
std::vector<std::string>::iterator& last,
std::vector<std::string>& thread_data) {
// read from the chunks here
}

template <typename T>
std::vector<std::pair<T, T>> chunk(T range_from, T range_to,
const std::ptrdiff_t num) {
using std::vector;
using std::pair;
using std::make_pair;
using std::distance;
using diff_t = std::ptrdiff_t;

/* Total Tem number and portion input_size. */
const diff_t total{distance(range_from, range_to)};
const diff_t portion{total / num};

vector<pair<T, T>> chunks(num);

T portion_end{range_from};

/* Use the 'generate' algorThm to create portions. */
std::generate(begin(chunks), end(chunks), [&portion_end, portion]() {
T portion_start{portion_end};

portion_end += portion;
return make_pair(portion_start, portion_end);
});

/* The last portion's end must always be 'range_to'. */
chunks.back().second = range_to;

return chunks;
}

I believe the "chunk" function should return a vector of pairs, with
"first" containing an iterator to the beginning of a chunk of the vector
"vec_string", and "second" containing an iterator to the end. I try to
pass the two of them into a thread constructor by value so thread 1 can
read from say the first five vectors of strings within the outer vector
"vec_string" and the second can read from the rest. However I get the
following errors on compilation:

||=== Build: Debug in ThreadTest (compiler: GNU GCC Compiler) ===|
/usr/include/boost/bind/bind.hpp||In instantiation of ‘void
boost::_bi::list3<A1, A2, A3>::operator()(boost::_bi::type<void>, F&,
A&, int) [with F = void (*)(const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&, const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&,
std::vector<std::__cxx11::basic_string<char> >&); A = boost::_bi::list0;
A1 =
boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >;
A2 =
boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >;
A3 =
boost::_bi::value<std::reference_wrapper<std::vector<std::vector<std::__cxx11::basic_string<char>
> > > >]’:|
/usr/include/boost/bind/bind.hpp|893|required from
‘boost::_bi::bind_t<R, F, L>::result_type boost::_bi::bind_t<R, F,
L>::operator()() [with R = void; F = void (*)(const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&, const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&,
std::vector<std::__cxx11::basic_string<char> >&); L =
boost::_bi::list3<boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >,
boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >,
boost::_bi::value<std::reference_wrapper<std::vector<std::vector<std::__cxx11::basic_string<char>
> > > > >; boost::_bi::bind_t<R, F, L>::result_type = void]’|
/usr/include/boost/thread/detail/thread.hpp|116|required from ‘void
boost::detail::thread_data<F>::run() [with F = boost::_bi::bind_t<void,
void (*)(const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&, const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&,
std::vector<std::__cxx11::basic_string<char> >&),
boost::_bi::list3<boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >,
boost::_bi::value<std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > > >,
boost::_bi::value<std::reference_wrapper<std::vector<std::vector<std::__cxx11::basic_string<char>
> > > > > >]’|
/home/matt/Documents/C++/ThreadTest/main.cpp|98|required from here|
/usr/include/boost/bind/bind.hpp|392|error: invalid initialization of
reference of type ‘const
__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char>*,
std::vector<std::__cxx11::basic_string<char> > >&’ from expression of
type
‘std::reference_wrapper<__gnu_cxx::__normal_iterator<std::vector<std::__cxx11::basic_string<char>
>*, std::vector<std::vector<std::__cxx11::basic_string<char> > > > >’|
||=== Build failed: 1 error(s), 4 warning(s) (0 minute(s), 5 second(s)) ===|

[bitrex]

On 01/02/2017 01:39 PM, bitrex wrote:

> void process(std::vector<std::string>::iterator& first,
> std::vector<std::string>::iterator& last,
> std::vector<std::string>& thread_data) {
> // read from the chunks here
> }

Sorry, these should not be references.

[Paavo Helde]

On 2.01.2017 20:39, bitrex wrote:

> Thanks, okay. Consider the following example:

It looks like you are mixing up vector<string> and
vector<vector<string>> in several places. You have to make up your mind
what you want to use where.

This code could also benefit from a couple of typedefs (or their newer
incarnation via 'using'). Inconsistent usage of std:: prefix does not
help either.

>
> #include <vector>
> #include <utility>
> #include <algorithm>
> #include <cstdint>
> #include "boost/thread/thread.hpp"
>
> template <typename T>
> std::vector<std::pair<T, T>> chunk(T range_from, T range_to,
> const std::ptrdiff_t num);
>
> void process(std::vector<std::string>::iterator first,
> std::vector<std::string>::iterator last,
> std::vector<std::string>& thread_data);

In process() you expect vector<string>::iterator and vector<string>.

> int main() {
> using std::vector;
> using std::string;
> using boost::thread;
>
> const size_t container_size = 10;
> const size_t num_threads = 2;
>
> vector<vector<string>> vec_string(container_size);
>
> std::generate_n(std::begin(vec_string), container_size, []() {
> vector<string> v{"A", "B", "C", "D"};
> return v;
> });
>
> std::array<thread, num_threads> t;
>
> auto chunks =
> chunk(std::begin(vec_string), std::end(vec_string), num_threads);

Here you pass in vector<vector<string>>::iterators and consequently
these are the iterators stored in chunks.

> for (size_t i = 0; i < num_threads; ++i) {
> t[i] = thread(process, chunks[i].first, chunks[i].second,
> std::ref(vec_string));
> }

Thus, here you pass vector<vector<string>>::iterator instead of expected
vector<string>::iterator, and a vector<vector<string>> instead of a
vector<string>.

[bitrex]

I see, thank you - yes I should be passing an iterator to the proper
thing, I suppose...;-)

The desired functionality is to subdivide the outer std::vector into n
blocks of the inner std::vector<std::string> and pass each block of the
latter to the "process" function.