[Boost-users] boost::tokenizer problem, maybe a bug.

[Jarl Friis]

Hi.

I discovered the following problem using boost::tokenizer. The
following program should print "hello", but it prints "hell":

#include <iostream>
#include <boost/tokenizer.hpp>

int main(){
typedef boost::tokenizer<boost::char_separator<char> > tokenizer;

boost::char_separator<char> sep(",", "");
tokenizer values(std::string(",hello,"), sep);
for( tokenizer::iterator valueI = values.begin();
valueI != values.end();
++valueI){
std::cout << "\"" << *valueI << "\"" << std::endl;
}
}

Problem is confirmed using boost 1.32.0 and both
GCC 3.3.4 and GCC 3.3.5

If anybody have a newer GCC installed, I would like to hear if the
problems also exists in later GCC releases.

I believe it is a bug in the std::string::assign implementation
(i.e. not boost directly), but a the workaround for msvc in
boost/token_functions.hpp, line 215-228 has also been tried for the
GCC but it just moves the problem; it will mibehave when the line
tokenizer values(std::string(",hello,"), sep); is replaced with
tokenizer values(std::string("hello,"), sep); since it will then print
"hel". So maybe another boost workaround is needed for GCC for the
lines 215-228 in boost/token_functions.hpp

Jarl

--
Jarl Friis
Softace ApS
Omøgade 8, 2.sal
2100 København Ø.
Denmark
Phone: +45 26 13 20 90
E-mail: ja...@softace.dk


_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Jeff Flinn]

Jarl Friis wrote:
> Hi.
>
> I discovered the following problem using boost::tokenizer. The
> following program should print "hello", but it prints "hell":
>
> #include <iostream>
> #include <boost/tokenizer.hpp>
>
> int main(){
> typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
>
> boost::char_separator<char> sep(",", "");
> tokenizer values(std::string(",hello,"), sep);

Your string is a temporary and goes out of scope. Try:

std::string test(",hello,");

tokenizer values( test, sep );

> for( tokenizer::iterator valueI = values.begin();
> valueI != values.end();
> ++valueI){
> std::cout << "\"" << *valueI << "\"" << std::endl;
> }
> }
>

Jeff

[Serge Skorokhodov]

Jeff Flinn wrote:

>Jarl Friis wrote:
>
>
>>Hi.
>>
>>I discovered the following problem using boost::tokenizer. The
>>following program should print "hello", but it prints "hell":
>>
>>#include <iostream>
>>#include <boost/tokenizer.hpp>
>>
>>int main(){
>> typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
>>
>> boost::char_separator<char> sep(",", "");
>> tokenizer values(std::string(",hello,"), sep);
>>
>>
>
>
>Your string is a temporary and goes out of scope. Try:
>
> std::string test(",hello,");
>
> tokenizer values( test, sep );
>
>

hm, why thiis works with vc-7_1 then? ;)

>> for( tokenizer::iterator valueI = values.begin();
>> valueI != values.end();
>> ++valueI){
>> std::cout << "\"" << *valueI << "\"" << std::endl;
>> }
>>}
>>
>>
>>

--
Serge

[Jarl Friis]

Serge Skorokhodov <serge.sk...@tochka.ru> writes:

> Jeff Flinn wrote:
>
>>Jarl Friis wrote:
>>
>>
>>>Hi.
>>>
>>>I discovered the following problem using boost::tokenizer. The
>>>following program should print "hello", but it prints "hell":
>>>
>>>#include <iostream>
>>>#include <boost/tokenizer.hpp>
>>>
>>>int main(){
>>> typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
>>>
>>> boost::char_separator<char> sep(",", "");
>>> tokenizer values(std::string(",hello,"), sep);
>>>
>>>
>>
>>
>>Your string is a temporary and goes out of scope. Try:
>>
>> std::string test(",hello,");
>>
>> tokenizer values( test, sep );

Thanks, that helped.

> hm, why thiis works with vc-7_1 then? ;)

I gues accoriding to the standard, it is undefined behaviour, so it
might work, you just can't rely on it.

Jarl