Upgrade to Go1.16 from Go1.14 broke my query

[Hugh Myrie]

I recently updated my Go version to 1.16 and for all my queries I now get an error:

{"error":"Dynamic SQL Error\nSQL error code = -303\narithmetic exception, numeric overflow, or string truncation\nstring right truncation\n"}

I'm using  the package:   _ "github.com/nakagami/firebirdsql"

The following is a simple function:

func getSystem(w http.ResponseWriter, r *http.Request) {

type system struct {

Tax1    float64  `json:"tax1"`

Fees1  float64  `json:"fees1"`

Fees2   float64  `json:"fees2"`

Minus_stk string `json:"minus_stk"`

Discount1 float64 `json:"discount1"`

Discount2 float64 `json:"discount2"`

Discount3 float64 `json:"discount3"`

Discount4 float64 `json:"discount4"`

Elderly float64 `json:"elderly"`

Message NullString `json:"message"`

Binary4 NullString `json:"binary4"`

}

sysdata := []system{}

sql2 := "select tax1, fees1, fees2, minus_stk, discount1, discount2, discount3, discount4, elderly, cast(mesg as varchar(400)), binary4 from system"

    conn, _ := sql.Open("firebirdsql",  datapath)

    defer conn.Close()

rows, err := conn.Query(sql2)

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

return

}

defer rows.Close()

for rows.Next() {

var p system

err := rows.Scan(&p.Tax1, &p.Fees1, &p.Fees2, &p.Minus_stk, &p.Discount1, &p.Discount2, &p.Discount3, &p.Discount4, &p.Elderly, &p.Message, &p.Binary4)

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

return

}

sysdata = append(sysdata, p)

}

err = rows.Err()      //  <--- errors seem to thrown here.

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

return

}

respondWithJSON(w, http.StatusOK, sysdata)

}

Your help would be appreciated.

I'm using Firebird 2.5. Everything worked under the previous version of Go. I am scanning for null values so I don't believe that is the issue.

[Roland Müller]

Hello,

ust a wild guess not "biased" by any close knowledge of Firebird DB or the above mentioned Go module: there has something changed with handling of rows.Next() when the last row of data has been passed. This guess is based on the fact that your mark the error only occurs after reading all data rows.

I would try to log a warning to server log and try to return the JSON data instead of returning http.StatusBadRequest. It may be that the data is valid.

 

BR,

Roland

[Hugh Myrie]

Bypassing the error returns an empty result set. 

 Additional information: "string right truncation /expected length 0, actual 3". In the DB the Code field is varchar(3) and Description is Varchar(30).

It seems to have something to do with the return field size and the size expected in Golang.

[Hugh Myrie]

Yes. It seems that's the case. I'm floored.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/BHxTxzCpUB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9f77d3d8-a666-48f1-8da8-43b0a076a132n%40googlegroups.com.

--

http://www.jaxtr.com/blessed_hope

[jake...@gmail.com]

Is it possible that the version of the library you are using also changed when you changed versions of Go?

Have you tried building again, on the same machine, using Go 1.14 to make sure that the only difference is the Go language version?

[Hugh Myrie]

That is a possibility. I didn't try to rebuild under Go1.14., but I will give that a try.  Interestingly, one "select query" worked but the others failed with similar error messages. After upgrading to G1.16 I ran "go tidy" which downloaded the libraries and create a "mod" file in the working folder, so I take it that the libraries are correct. I'll try your suggestion and let you know.

  I am able to access and read MySQL and MariaDB tables with the updated Go version. 

[Roland Müller]

When the query works with other DBs & their Go  drivers this strongly suggest that there is some incompatibility of the Firebird vs. Go 16.x.

Thus you should check the suggestion from Jake to update the Firebird driver for Go.

In the Github project https://github.com/nakagami/firebirdsql/commits/master 

In Commits on Jun 21, 2021 used go version in go.mod was change to 1.15 and the readme file now startes that go 1.15 or newer is supported. May be this helps.

BR,

Roland

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/90a32eff-32c0-4d6d-a822-e843bbc5b31an%40googlegroups.com.

[Hugh Myrie]

I agree. So far, I've Installed Go1.15, reinstalled the Firebird library. Compiled using Go1.15 and the problem persist. Trying to install Go1.14.9 but I'm not yet successful. I suppose Go1.14 is now deprecated so the normal procedure to have it coexist with the later versions doesn't work.

[Brian Candler]

On Sunday, 27 June 2021 at 03:27:28 UTC+1 hugh....@gmail.com wrote:

I agree. So far, I've Installed Go1.15, reinstalled the Firebird library.

Which version?  The latest tagged version (v0.9.1) or the tip of the master branch?

Having said that: the commit which changes the current supported version to 1.15, does nothing except change the version number in go.mod; and it was applied immediately after the v0.9.1 tag. 

 

Compiled using Go1.15 and the problem persist.

If you can make a standalone test program which replicates the problem, then I suggest you open an issue on the firebirdsql repo.  All developers love good reproducible bug reports with code that reproduces the bug :-)

[Hugh Myrie]

The version I saw in the go.mod file was v0.9.1. However, looking at the modified date of the files in the library folder was last year. So I deleted the library and reinstalled it . I suppose doing a "go get" without specifying a version would retrieve the latest version. Correct me if I'm wrong here.

I'm trying to get back to go1.14 to conduct further tests before making a standalone test.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/BHxTxzCpUB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6bb8c44e-e564-4685-aecf-7c8bffb3269cn%40googlegroups.com.

--

http://www.jaxtr.com/blessed_hope

[Hugh Myrie]

Update: 

I did a re-installation of Go1.14.15. Reinstalled the FirebirdSQL driver, yet the problem persist. 

[Brian Candler]

Then it sounds to me like the problem is with your application, and/or with the data in your database, and that the problem has always been there.

The error talks about "string right truncation". My suggestion would be to take your SQL query and start removing some fields from it, starting with the "message" and "binary4" fields.  In particular, is it possible that some value in your "mesg" column has more than 400 characters?

sql2 := "select tax1, fees1, fees2, minus_stk, discount1, discount2, discount3, discount4, elderly, cast(mesg as varchar(400)), binary4 from system"

...

err := rows.Scan(&p.Tax1, &p.Fees1, &p.Fees2, &p.Minus_stk, &p.Discount1, &p.Discount2, &p.Discount3, &p.Discount4, &p.Elderly, &p.Message, &p.Binary4)

-->

sql2 := "select tax1, fees1, fees2, minus_stk, discount1, discount2, discount3, discount4, elderly from system"

...

err := rows.Scan(&p.Tax1, &p.Fees1, &p.Fees2, &p.Minus_stk, &p.Discount1, &p.Discount2, &p.Discount3, &p.Discount4, &p.Elderly)

(Aside: those values have type "NullString" but your code snippet didn't show how that type was defined)

If that doesn't make a difference, start chopping out the other fields.

[Hugh Myrie]

Update: 

I created a new database with only one table with two fields. Added three records. I have also created a stripped down version of the Go program with a function that query the table. The problem persist.

Table:

 CREATE TABLE TESTR

(

  CODE Varchar(5) NOT NULL,

  DESCRIPTION Varchar(30) NOT NULL

);

The Data.

insert into testr (code, description) values ('123', 'Data 1');

insert into testr (code, description) values ('163', 'Data 5');

insert into testr (code, description) values ('653', 'Data 2');

The Go function:

func getTest(w http.ResponseWriter, r *http.Request) {

type test struct {

Code  string  `json:"code"`

Description   string  `json:"description"`

}

tests := []test{}

conn, _ := sql.Open("firebirdsql", datapath)

    defer conn.Close()

rows, err := conn.Query("select code, description from testr")

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

logError(err.Error())

return

}

defer rows.Close()

for rows.Next() {

var t test

err := rows.Scan(&t.Code, &t.Description)

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

logError(err.Error())

return

}

tests = append(tests, t)

}

err = rows.Err()

if err != nil {

respondWithError(w, http.StatusBadRequest, err.Error())

logError(err.Error())

return

}

respondWithJSON(w, http.StatusOK, tests)

}

The call (from the browser):

192.168.0.13:8080/test

The Error:

{"error":"Dynamic SQL Error\nSQL error code = -303\narithmetic exception, numeric overflow, or string truncation\nstring right truncation\nexpected length 1, actual 5\n"}

[Brian Candler]

Can you make that into a standalone program, which doesn't use a HTTP exchange but just does the query and exits?  If the problem is still there, then you have a very nice test case to submit upstream (along with details of exactly what version of firebird you're using, what platform you're running on etc)

If the problem goes away, but comes back when it's run as a HTTP server, then make a standalone program that contains a HTTP server and also demonstrates the problem.  What you've shown so far is just a snippet which isn't runnable as-is.

[Hugh Myrie]

Let me do that.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3f8b65ed-dc7c-4807-bd5a-7044f67d5b60n%40googlegroups.com.

--

http://www.jaxtr.com/blessed_hope

[Hugh Myrie]

Stripped down file is attached. 

Go version 1.14.15

DB was created in Firebird 3.0.

[Brian Candler]

I suggest the issue tracker for the firebirdsql library is the right place.

[Hugh Myrie]

I opened an issue there. Thanks for your help.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/BHxTxzCpUB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5dbecbbf-fe1f-4020-9a53-6b6fc8a0f977n%40googlegroups.com.

--

http://www.jaxtr.com/blessed_hope