database/sql driver for flat file?

[Boris]

I am curious whether any database/sql driver exists for reading from flat files. It seems that it would be fairly straightforward to access flat files (csv, space-separated, fixed-width) through the database/sql interface. Of course, it would not support all of the features of a relational database, but it would be less mental overhead to do this:

db, err := sql.Open("flatfile", "rows=newline,cols=newline")

rows, err := db.Query("/path/to/my/file.txt")

for rows.Next() {

}

If one is already using database/sql'ish concepts to access row-oriented data, this seems like it would keep the number of different concepts in the code smaller. Anyone know of such a driver, or am I Doing It Wrong?

Thanks

Boris

[Rodrigo Kochenburger]

IMO, database/sql interface is defined w/ SQL interaction in mind. I don't think implementing on top of it would be a good idea.

By the way you're trying to read though, I'd suggest using CSV files using http://golang.org/pkg/encoding/csv/.

Cheers

- RK

Boris

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[John Nagle]

That's quite feasible. Not clear if it's a good idea, but you
can certainly implement SQL queries for flat files. Updating
is going to be kind of lame (you'd probably have to write a
new file and move it to the old name) but it's certainly possible.
This could be a good way to work on configuration files and such.

You could even keep indices in RAM and speed it up.

Note that most SQL-based database support LOAD INFILE,
which can read CSV files and such. So if your problem is that
you have a CSV file and need to work with its data, you can
load it into a real database and go to work.

John Nagle

[DisposaBoy]

IIRC someone did something similar not too long ago. Search the ml or reddit /r/golang . all I can remember is that it was hosted github and I think it used JSON files

[Robert Johnstone]

You may be able to achieve the same effect by using ODBC.  However, after dealing with all the ODBC configuration, I'm not certain if you'll find it all that elegant in the end...

[Jim Robinson]

It's certainly possible, I'm not sure why you want a flat

file though?  Are you interested in doing this because

you want to keep the data files simple for access from

programs other than your own?

If the latter, if cross-program access is important to you,

you might consider just using an embedded database

like sqlite instead.

That said, I was just working on some stuff where I

needed to maintain a key/value index into a multi-gigabyte

file, and I decided to use a TSV file as the lowest common

denominator, reading it into a persistent binary KV store

on startup but writing it back out to TSV on close.

For inspiration in implementing a Go version of a relational

database program, you could take a look at the 'qawk'

database implementation in The AWK Programming

Language.

  http://www.cs.bell-labs.com/cm/cs/awkbook/index.html

[Boris]

Reason I want flat file is for making mocking easy for tests. Actually I don't want to mock whole sql interface, probably just mock sql.Rows. Then keep test datasets into flat files. Maybe CSV is easiest -- write sql.Rows compatible wrapper around, define interface matching sql.Rows, define functions to accept arguments of this interface. In real code pass in sql.Rows, in test code pass in csvRowsWrapper or whatever.

[Rob Merrell]

If you just want it for testing you may be looking for something like this: https://github.com/erikstmartin/go-testdb

- Rob

On Sun, Aug 4, 2013 at 10:23 AM, Boris <boris.s...@gmail.com> wrote:

Reason I want flat file is for making mocking easy for tests. Actually I don't want to mock whole sql interface, probably just mock sql.Rows. Then keep test datasets into flat files. Maybe CSV is easiest -- write sql.Rows compatible wrapper around, define interface matching sql.Rows, define functions to accept arguments of this interface. In real code pass in sql.Rows, in test code pass in csvRowsWrapper or whatever.

--