readLine() and newline problem

[mike]

Hi,

We have the following code for reading a NETCONF "hello" message from a NETCONF Device.

public class MessageHandler {

public StringBuilder readLines(InputStream is) {
final StringBuilder buffer = new StringBuilder();
final String delimiter = "]]>]]>";
String tmp;

try (BufferedReader reader = new BufferedReader(new InputStreamReader(is))) {
do {

tmp = reader.readLine();

if (tmp != null) {
buffer.append(tmp);
}
} while (tmp != null && !tmp.endsWith(delimiter));

} catch (IOException e) {
LOGGER.info("Failed to read <rpc-reply> message ",
e);
}

return buffer;

}




}

We get the following string from the device ( just an example):

<?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<capabilities>
<capability>
urn:ietf:params:netconf:base:1.1
</capability>
<capability>
urn:ietf:params:ns:netconf:capability:startup:1.0
</capability>
</capabilities>
<session-id>4</session-id>
</hello>
]]>]]>

The problem here is that some devices add a newline after "]]>]]>" which is the message separator. And of course some devices don't since it is not required in the RFC6242.

So when there is no newline from device then the above method will hang. Then I thought it would be easy to create a unit test that recreates the problem. One with a hello message with message separator and newline and one with only message separator. But problem is that I get no exception in the test without the newline. Any ideas what I am missing or how I can re-create the issue?

br,

//mike

public class MessageHandlerTest {

private MessageHandler messageHandler;

@BeforeClass
public void setup() {
messageHandler = new MessageHandler();
}

@Test
public void testHelloServerWithOutNewLine() throws Exception {
InputStream is = getHelloWithoutNewline();
StringBuilder actual = messageHandler.readLines(is);
Assert.assertTrue(actual.length() > 0);

}

@Test
public void testHelloServerWithNewline() throws Exception {
InputStream is = getHelloWithNewline();
StringBuilder actual = messageHandler.readLines(is);
Assert.assertTrue(actual.length() > 0);

}


private InputStream getHelloWithoutNewline() {
final String HELLO_REPLY = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
" <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">" +
" <capabilities>" +
" <capability>" +
" urn:ietf:params:netconf:base:1.1" +
" </capability>" +
" <capability>" +
" urn:ietf:params:ns:netconf:capability:startup:1.0" +
" </capability>" +
" </capabilities>" +
" <session-id>4</session-id>" +
" </hello>" +
" ]]>]]>";

return new ByteArrayInputStream(HELLO_REPLY.getBytes());

}

private InputStream getHelloWithNewline() {
final String HELLO_REPLY = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
" <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">\n" +
" <capabilities>\n" +
" <capability>\n" +
" urn:ietf:params:netconf:base:1.1\n" +
" </capability>\n" +
" <capability>\n" +
" urn:ietf:params:ns:netconf:capability:startup:1.0\n" +
" </capability>\n" +
" </capabilities>\n" +
" <session-id>4</session-id>\n" +
" </hello>\n" +
" ]]>]]>\n";

return new ByteArrayInputStream(HELLO_REPLY.getBytes());

}

}

[Daniele Futtorovic]

On 2019-05-10 08:16, mike wrote:
> The problem here is that some devices add a newline after "]]>]]>" which is the message separator. And of course some devices don't since it is not required in the RFC6242.
>
> So when there is no newline from device then the above method will hang. Then I thought it would be easy to create a unit test that recreates the problem. One with a hello message with message separator and newline and one with only message separator. But problem is that I get no exception in the test without the newline. Any ideas what I am missing or how I can re-create the issue?

In your test, an EOF takes the place of the otherwise expected EOL and
makes it work.

As for the problem overall, in my view you definitely shouldn't be using
readLine() here. You should be reading with read(char[]) and keep doing
so until you encounter the separator.

--
DF.

[mike]

So how can I update test to simulate my error?

Yes I am also thinking of using read(char []). I think it is more appropriate. The only thing that will be a bit tricky is to find my separator char.I have to check if we have ] then another ] so there will be many ifs.

[Martin Gregorie]

Two suggestions:

1) Continue to use readLine() but use Scanner.findInLine() to recognise
the end pattern and DON'T include a newline in the end pattern.

2) Read everything into a String and then use a Scanner or
String.indexOf(String.str) to find the end pattern and truncate the
string at that point.

Which one is better depends on how much unwanted stuff follows the
terminator and how much time would be wasted reading it in and discarding
it.


--
Martin | martin at
Gregorie | gregorie dot org

[Daniele Futtorovic]

On 2019-05-10 10:29, mike wrote:
> So how can I update test to simulate my error?

That's a bit trickier, but the code below should give you an idea.

> Yes I am also thinking of using read(char []). I think it is more appropriate. The only thing that will be a bit tricky is to find my separator char.I have to check if we have ] then another ] so there will be many ifs.

Save yourself trouble and use a java.util.Scanner. See below.

Code:
````
import org.junit.Test;

import java.io.*;
import java.util.Objects;
import java.util.Scanner;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class NoEol {

static final String DATA =

"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ " <hello xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\">\n"
+ " <capabilities>\n"
+ " <capability>\n"
+ " urn:ietf:params:netconf:base:1.1\n"
+ " </capability>\n"
+ " <capability>\n"
+ " urn:ietf:params:ns:netconf:capability:startup:1.0\n"
+ " </capability>\n"
+ " </capabilities>\n"
+ " <session-id>4</session-id>\n"
+ " </hello>\n"

+ " ]]>]]>"; //NOTE no newline at the end

@Test
public void testBufferedReaderWithNewLine(){
try(Supplier s = Supplier.newInstance(DATA + "\n")){
globWithBufferedReader( s.input(), "]]>]]>");
}
catch ( IOException e ) {
e.printStackTrace();
}
}

@Test
public void testBufferedReaderWithoutNewLine(){
try(Supplier s = Supplier.newInstance(DATA)){
globWithBufferedReader( s.input(), "]]>]]>");
}
catch ( IOException e ) {
e.printStackTrace();
}
}

@Test
public void testScannerWithNewLine(){
try(Supplier s = Supplier.newInstance(DATA + "\n")){
globWithScanner( s.input(), "]]>]]>");
}
}

@Test
public void testScannerWithoutNewLine(){
try(Supplier s = Supplier.newInstance(DATA)){
globWithScanner( s.input(), "]]>]]>");
}
}

static void globWithBufferedReader( Reader r, String sep) throws
IOException {
BufferedReader br = new BufferedReader(r);
StringBuilder sb = new StringBuilder();

for(String line; null != (line = br.readLine()); ){
sb.append(line);

if( line.contains( sep ) ){
break;
}
}

System.out.println("Read finished: " + sb);
}

static void globWithScanner(Reader r, String sep){
Scanner sc = new Scanner(r);
sc.useDelimiter( sep );

System.out.println("Read finished: " + sc.next());
}

private static final class Supplier implements Runnable, AutoCloseable {
static final ExecutorService executor =
Executors.newCachedThreadPool();

private final String data;

private final PipedReader pipeIn;
private final PipedWriter pipeOut;

private volatile Thread thread;

Supplier( String data )
{
this.data = Objects.requireNonNull( data, "data" );

pipeOut = new PipedWriter();

try {
pipeIn = new PipedReader( pipeOut );
}
catch ( IOException x ) {
throw new AssertionError( "Could not cerate pipe", x );
}
}

public Reader input() {
return pipeIn;
}

@Override
public void close() {
if ( thread != null ) {
thread.interrupt();
thread = null;
}
}

@Override
public void run() {
thread = Thread.currentThread();

try {
pipeOut.write( data );

synchronized ( this ) {
wait();
}
}
catch ( InterruptedException e ) {
System.err.println( "Interrupted." );
Thread.currentThread().interrupt();
}
catch ( IOException e ) {
e.printStackTrace();
}

System.err.println( "Supplier exiting." );
}

public static Supplier newInstance( String data ) {
Supplier ret = new Supplier(data);
executor.submit( ret );

return ret;
}
}
}
````
--
DF.