"unsafe" roadblock

[Arne Hormann]

Hi,

I'm hitting a roadblock with package unsafe, maybe someone here can help.

I'd like to get a copy of a slice of structs from another struct passed as an interface.

None of the structs are exported and I need the copy in another package.

This is probably a little hard to understand, so here's an example.

Any help is highly appreciated!

Arne

[minux]

You can first dig the slice out using reflect, however, I'm not sure why you have such

a strange requirement. Could you please elaborate your use case here? perhaps there

are better solutions that the unsafe and hacky solutions.

I have a feel that you're designing your solution against the language....

just a feeling, though.

[Arne Hormann]

It's not just a feeling, you're right and this is not my prefered solution.

I need column type information from database/sql drivers and everything in there is wrapped.

I proposed a safe workaround in an earlier post, but that requires changing the runtime.

I don't know if I should have posted that to golang-dev instead of golang-nuts or maybe filed a bug (which I can still do),

but I'd like to provide a way to use it with older Go version anyway.

[Arne Hormann]

A little more detail: my blockade is getting the internal rows implementation for github.com/go-sql-driver/mysql (mysqlRows) and extract the []mysqlField slice into my own copy.

I get it as driver.Driver (an interface) and both structs aren't exported.

Though I could probably get an implementation based on a safe way into the driver, Julien is strictly against anything using unsafe (and I think he's right).

That's why I have to use an external package and that's why I can't access the two unexported structs.

[John Nagle]

On 5/31/2013 6:07 AM, Arne Hormann wrote:
> Am Freitag, 31. Mai 2013 14:59:06 UTC+2 schrieb minux:

>> I have a feel that you're designing your solution against the language....
>> just a feeling, though.
>>
>

> It's not just a feeling, you're right and this is not my preferred solution.

> I need column type information from database/sql drivers and everything in
> there is wrapped.
> I proposed a safe workaround in an earlier post

> <https://groups.google.com/forum/#!topic/golang-nuts/2aLctcVyp6Q>,

> but that requires changing the runtime.

There's a lot of dissatisfaction with Go's SQL interface.
That's why there are so many alternatives to it that do almost
the same thing. The Go SQL interface itself isn't a good fit
to Go; it's basically a warmed-over interface from another
language.

The fundamental problem is that what you get from a SQL
query is a sequence of typed values, each of which has a
type, a value, and a column name. In some languages,
like Python, this is easily represented as a tuple.
For Go, it would logically be a struct, but the form
of that struct is query dependent and not known to the SQL
package. So the Go SQL package returns an opaque type,
a "Row", for which a few functions are provided.

(The description for "func (*Row) Scan" at
"http://golang.org/pkg/database/sql/#Row" makes
no sense. It says: "Scan copies the columns from the matched row into
the values pointed at by dest. If more than one row matches the query,
Scan uses the first row and discards the rest. If no row matches the
query, Scan returns ErrNoRows." A "Row" only contains one row.
There's a type Rows, but that's different. This description may
have been modified from that for Scan() for type Rows.)

The original poster needs to convert a Row into a struct with
the matching fields, and there's no general way to do that
without reflection. Maybe there should be.

The "...T" syntax in function calls turns a list of
parameters into a slice of type []T. But you can only do
that in function calls. It would be useful to have a general
function that can turn a struct into a slice of interface{},
and the reverse operation. Those are safe operations, and
can be written (inefficiently) with reflection.

There some other simple generic operations like that which
would run fast if implemented at compile time but are
painfully slow when done via reflections. Deep copy
for nonrecursive types is an example.

John Nagle

[Arne Hormann]

The original poster needs to convert a Row into a struct with
the matching fields, and there's no general way to do that
without reflection.   Maybe there should be.

In my other post, I linked to Issue 5606 created for exactly that purpose.

 

[Kevin Gillette]

On Friday, May 31, 2013 7:07:24 AM UTC-6, Arne Hormann wrote:

I proposed a safe workaround in an earlier post, but that requires changing the runtime.

Why would it require changing the runtime? database/sql hardly does anything deep in the runtime. Doing what you want in an official way may require awkward API changes, but should be fully achievable given the current runtime implementation. 

[Arne Hormann]

Sorry, scrap that. I'm not interested in any row.

I'm interested in the metadata for each column in a result - the type, flags, comments, whatever is available.

What I want would only have to be extracted once per query.

In some cases not even necessarily per query but per unique sql query string.

My proposal is from a technical perspective:

How do I add the functionality in a way that causes the least changes in the api and move/assign/provide as much responsibility and flexibility as possible to the driver implementations.

[Arne Hormann]

Ok, I probably confused the terminology. In the standard library, not in the runtime.

[John Nagle]

On 5/31/2013 12:17 PM, Arne Hormann wrote:

> Sorry, scrap that. I'm not interested in any row.
> I'm interested in the metadata for each column in a result - the type,
> flags, comments, whatever is available.
> What I want would only have to be extracted once per query.
> In some cases not even necessarily per query but per unique sql query
> string.

You can do a SHOW COLUMNS query in SQL to get table structure
and data type info.

John Nagle

[Arne Hormann]

Thanks, but that's not that easy for aggregated columns, joins, SHOW queries, ...

[Kevin Gillette]

SHOW COLUMNS is by no means standard SQL (or anywhere close). Besides which, it's reasonable that the OP would want to fetch this information from an arbitrary query result.

[Kevin Gillette]

Ah, then yes indeed. In Go the runtime refers to garbage collection, allocation, goroutine/channel scheduling, signal handling, etc (the stuff that's lower level than what the application or library programmer needs to be concerned with).

[Carlos Castillo]

Isn't this a very driver specific function? Different DB's have different types, different features for types, different ways of representing these features, and different ways those features can be used. Expecting a general package like database/sql to handle them all seems like a bad idea. You will either have a huge API, most of which won't apply, or a very small one, limited to features that mostly appear everywhere.

Your previous proposal already put the main logic into the driver (ie: the user had to manually call driver Functions), so you already know the driver and have to use it's code. If you want to see how a driver deals with a given query you might as well ask it directly, instead of doing a back and forth with database/sql.

Also you still haven't answered the question "What are you trying to do?". Are you trying to build an ORM? A tool for analyzing DB's?

Here is an example of type and database agnostic code that converts a rows object into a []map[string]interface{} or [][]interface{}: http://play.golang.org/p/RJQgrLBX2K

[Arne Hormann]

Am Samstag, 1. Juni 2013 01:09:25 UTC+2 schrieb Carlos Castillo:

Isn't this a very driver specific function? Different DB's have different types, different features for types, different ways of representing these features, and different ways those features can be used. Expecting a general package like database/sql to handle them all seems like a bad idea. You will either have a huge API, most of which won't apply, or a very small one, limited to features that mostly appear everywhere.

Your previous proposal already put the main logic into the driver (ie: the user had to manually call driver Functions), so you already know the driver and have to use it's code. If you want to see how a driver deals with a given query you might as well ask it directly, instead of doing a back and forth with database/sql.

Also you still haven't answered the question "What are you trying to do?". Are you trying to build an ORM? A tool for analyzing DB's?

Here is an example of type and database agnostic code that converts a rows object into a []map[string]interface{} or [][]interface{}: http://play.golang.org/p/RJQgrLBX2K

I only have one proposal, the one in the other thread. This one is an attempt at a workaround that is usable without an api change in Go.

Your example will probably return a string representation as []byte (or sql.RawBytes) in most cases. That way you still lose the information available to the driver.

[Carlos Castillo]

That's the result of a choice made by the database/sql pakage, but there is no information lost if you convert a []byte to a string if that's what you wanted.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/_wig_BFKXC8/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Carlos Castillo

[Arne Hormann]

Am Samstag, 1. Juni 2013 02:25:47 UTC+2 schrieb Carlos Castillo:

That's the result of a choice made by the database/sql pakage, but there is no information lost if you convert a []byte to a string if that's what you wanted.

I know that. But I'm not interested in a string or a []byte, I'm interested in getting the closest matching go type from the driver and causing the smallest amount of internal type conversions that is possible.

Today, there's just no way to get there. If I use dynamic queries (specified by the user) or aggregated queries or anything synthetic (like SHOW in MySQL), I can't get the type from the database.

Ok, maybe there is a way, but that means replicating a lot of infrastructure across projects to determine the column types and provide guesses for all synthetic stuff not using a table directly. Maybe there could be a companion support library for each driver, but who wants to write and maintain them? 

Additionaly, using []byte every time causes a lot of allocations and inspecting the table structure causes some queries and a slower startup, which in some cases can't be prevented by caching, the database may have changed. If I use SHOW COLUMNS or maybe parse the sql to get that info, I still have to construct something to store it and have to remember the matches between Go and the database types. Moving that to the driver makes life easier for everyone who wants to efficiently serialize rows in the smallest possible binary form, create ORM tools, build code generators (my own use case here), build tools for any database instead of the one one her own server, ...

Having the ability to know the types would be a gift to everyone building tools. Maybe not the users with their own database and existing tables they already know, but I guess they won't complain when better tools are created.

And saying better I also mean causing less queries and putting less strain on the GC. Or not having to implement tricks like a lazy buffer in every driver (which I didn't even know about until I saw it in a pull request from bradfitz).

[Arne Hormann]

If you came here to discuss my reflect problem, feel free to post here.

If you want to discuss column type retrieval in database/sql, please reply in the thread with my proposal: https://groups.google.com/forum/#!topic/golang-nuts/2aLctcVyp6Q

I linked the other thread in Issue 5606, anything here may be overlooked.