Fscan() eats 1st char in scanning values

[Ivan Burak]

Fscan() eats 1^st char in scanning values if the values aren't a first value in a line.

The documentation tells: "Fscanln is similar to Fscan, but stops scanning at a newline and after the final item there must be a newline or EOF."

 

The test discovers a bug in Fscanln() - https://go.dev/play/p/Jem2lWVn3H5

 Result of the test you can see below.

                 The string for scanning (2 lines below)

Go version go1.22.11 windows/amd64
Build simple, secure, scalable systems with Go

--------------------------------------------------------
| Fscan()           | Fscanln()         | Equal        |
--------------------------------------------------------
| Go                | Go                | true         |
| version           | ersion            | false   - !? |
| go1.22.11         | o1.22.11          | false   - !? |
| windows/amd64     | indows/amd64      | false   - !? |
| Build             | Build             | true         |
| simple,           | imple,            | false   - !? |
| secure,           | ecure,            | false   - !? |
| scalable          | calable           | false   - !? |
| systems           | ystems            | false   - !? |
| with              | ith               | false   - !? |
| Go                | o                 | false   - !? |
--------------------------------------------------------

   First words in the beginning of every line are equal for Fscan() and Fscanln()

   Fscanln() eats a first char in every 2nd, 3rd and other words in any line

 

Thanks
Ivan

[Ian Lance Taylor]

On Sun, Jan 26, 2025 at 4:51 PM 'Ivan Burak' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> Fscan() eats 1st char in scanning values if the values aren't a first value in a line.
>
> The documentation tells: "Fscanln is similar to Fscan, but stops scanning at a newline and after the final item there must be a newline or EOF."
>
>
>
> The test discovers a bug in Fscanln() - https://go.dev/play/p/Jem2lWVn3H5

Check the error result from Fscanln.

Ian

[Ivan Burak]

Hi Ian

>> Check the error result from Fscanln.

Thank you for the advice but I still have some arguments about wrong behavior of Fscanln() or not fully correct documentation for it..

Let’s look at the next lines from the documentation.

func Fscan
func Fscan(r io.Reader, a ...any) (n int, err error)
Fscan scans text read from r, storing successive space-separated values into successive arguments. Newlines count as space. 
It returns the number of items successfully scanned. If that is less than the number of arguments, err will report why.

func Fscanln 
func Fscanln(r io.Reader, a ...any) (n int, err error)

Fscanln is similar to Fscan, but stops scanning at a newline and after the final item there must be a newline or EOF.

---

So it means that Fscanln() scans text read from r, storing successive space-separated values into successive arguments, but stops scanning at a newline. 

I prepared the new code, please look at it.

https://go.dev/play/p/xDGYbwdzMpL

In the table below you can see the result of the code.

In the row 1 from the table we see that Fscanln() scans till the space and reads the right value "Go" and returns also an error "expected newline".

The error message in this case means that the reading was successful but the newline wasn't achieved - "expected newline".

For all other values in this line the behavior of Fscanln() differs (row 2-4). The function reads the wrong values (without 1st char) 

with the same error message except the row 4 in the table where the newline was found and the error is nil..

For the second line of the source string we see the same results (row 5 with right value, rows 6-11 with shrinked values).

The Fscanln() behaves differently for the first and other values in a line, and this is the error because documentation tells: "Fscanln is similar to Fscan".

That's the problem.

The error "expected newline" resulting from Fscan() is only a message not an error similar to io.EOF.

We can interpret "expected newline" as nothing dangerous just continue to read till the newline.

So concerning the documentation after successful reading the first value in a line we should continue the scanning.

 

`ivan`

[Jan Mercl]

On Mon, Jan 27, 2025 at 6:54 PM 'Ivan Burak' via golang-nuts
<golan...@googlegroups.com> wrote:

WAI: https://go.dev/play/p/4bWHGemItL_9

[Howard C. Shaw III]

In your example, you are repeatedly reading one 'item' from the line. This is the source of the confusion, as this is not how Fscanln works - when you do this, *each* item is the documented 'final item' after which there must be a newline.

That is, your data would look more like this:

space separated column headers

first second third last

some data with the

same pattern of columns

and your read would be pulling all of those items from one line:

fmt.Fscanln(in2,&one,&two,&three,&four)

It is expected/designed to read one line of data at a time, and for you to know in advance the number of items in the line, and to expect to receive an error if they don't match. It is also expected that each column will have a consistent datatype, and for that to match the types of your parameters, and to error if the data can't be parsed.

For parsing arbitrary string data, you might be better off just using bufio.Scanner with bufio.ScanWords.

[Ivan Burak]

  Hi Howard

Thank you for the answer.

My data is easier, it contains only 2 lines:

Go version go1.22.11 windows/amd64
Build simple, secure, scalable systems with Go

I did some tests and to speedup coding decided to use Fscanln().

I don't use this function in real programming, it's too slow.

There are many ways to parse texts in Golang that fast and stable.

My test algorithm didn't work, and when I realized what the reason was, I was just surprised.

The documentation tells that Fscanln() is similar to Fscan(), my test shows that it isn't true for some cases. 

In my opinion the documentation or the function should be corrected. That's it.  :)

>> fmt.Fscanln(in2,&one,&two,&three,&four)

That works but you know in our profession, usually we don't know the exact amount of data being processed 

(number of tokens, records, data volumes etc.), so we have to code based on estimates n, n^2, n^3 ... and we don't know the real n.

thanks

ivan

[Ivan Burak]

Hi Jan

Thank you for the test enhancing.

Please read my answer to Howard

https://groups.google.com/g/golang-nuts/c/tzvKTyaugEk/m/5h2Sm03bDgAJ

thanks

ivan

[Brian Candler]

> The documentation tells that Fscanln() is similar to Fscan(), my test shows that it isn't true for some cases. 

"similar" does not mean "the same as".

Fscanln chomps the fields given, then chomps the next character. If the next character is newline, it's happy. If the next character is not a newline, then it's unhappy and returns an error. At this point, you know the input was bad.

No guarantees are made about the state of the input after an error, and as you've found, the next character (which *should* have been a newline) has been chomped.

[tapi...@gmail.com]

On Tuesday, January 28, 2025 at 7:07:15 PM UTC+8 Brian Candler wrote:

> The documentation tells that Fscanln() is similar to Fscan(), my test shows that it isn't true for some cases. 

"similar" does not mean "the same as".

Fscanln chomps the fields given, then chomps the next character. If the next character is newline, it's happy. If the next character is not a newline, then it's unhappy and returns an error. At this point, you know the input was bad.

No guarantees are made about the state of the input after an error, and as you've found, the next character (which *should* have been a newline) has been chomped.

It looks the eating/chomping behavior is not well documented.

[Brian Candler]

I guess not, but since it uses a plain io.Reader which doesn't support peek or pushback, I can't think of any other behavior which is possible.

[tapi...@gmail.com]

The documents are too simple to explain clearly on anything.

Besides the eating/chomping behavior, does an item include the space/newline following it?

Either answer will not satisfy the behavior of the following code,

by either answer, there should one case reports an error.

https://go.dev/play/p/XQFwxxiz2NA

package main

import "fmt"
import "strings"

func main() {
{
var r = strings.NewReader("xxx yyy\n")
var x, y string
n, err := fmt.Fscanln(r, &x, &y)
fmt.Println(n, err, x, y, len(y)) // 2 <nil> xxx yyy 3
}
{
var r = strings.NewReader("xxx yyy \n")
var x, y string
n, err := fmt.Fscanln(r, &x, &y)
fmt.Println(n, err, x, y, len(y)) // 2 <nil> xxx yyy 3
}
}

[Ivan Burak]

The correct documentation for current Fscanln() behavior shoud be like this
Fscanln scans text read from r, storing successive space-separated values into successive arguments. It stops scanning at a newline and after the final item there must be a newline or EOF. If number of arguments  for scanning into is less than the number of values in a text, err will be reported.
So every time it needs to know the real number of values in a scanning text, not good limitation for real programming.

Another way is to fix some code for Fscanln(). 
The fix might be as follows

------------------------------------------------------------------------------------------------------
To see the code pls go to  https://go.dev/play/p/-LQhT0zK9_N

With code changes Fscanln() works without char eating (it was tested locally), the number of values in a scanning text and arguments for scanning into could be different.
scan_test.go falls on 2 functions, but the reason is behavior changes for Fscanln(). 

The result of code execution

[Brian Candler]

You've missed the point of how Fscanln is *supposed* to be different to Fscan.

Fscanln is when you want to scan *the entire line* in one go, and you know in advance exactly the right number of placeholders to capture.  If there is extra data on the line after these placeholders, then an error is returned. That is: it does the checking for spurious trailing junk for you. (If there are insufficient columns then it treats the last ones as empty).

In other words, to take your example number 2: Fscanln(in, &a) is *supposed* to return an error with the given input. If you don't know how many items to expect in the line, then Fscanln is not what you should be using.

[Ivan Burak]

>>  Fscanln is when you want to scan *the entire line* in one go, and you know in advance exactly the right number of placeholders to capture.

Yes, Brain. This means the documentation needs to be changed a bit, like the blue text in my previous message.

And of course, my example of the fix is not complete, it lacks something like EOL error ("end of line" similar to EOF), but it will another function Fscanln_v2    :)