Re: [R-sig-DB] R and PostgreSQL - Writing data?

858 views
Skip to first unread message

NISHIYAMA Tomoaki

unread,
Dec 6, 2012, 9:57:22 AM12/6/12
to Dirk Eddelbuettel, NISHIYAMA Tomoaki, James David Smith, r-sig-db, rpostgr...@googlegroups.com
Hi,

PostgreSQL have library function PQexecParams and also supports prepared statements.
String expansion in the SQL statement is cumbersome for escaping special characters and
therefore error prone.

I just commited to the SVN repository a very simple and primitive implementation that
allows to pass vector of characters as parameters.

A sample statement is like:

res <- dbGetQuery(con, "SELECT * FROM rockdata WHERE peri > $1 AND shape < $2 LIMIT $3", c(4000, 0.2, 10))
print(res)

The syntax for a positional parameter is a dollar sign ($) followed by digits
rather than a colon followed by digits in PostgreSQL.
http://www.postgresql.org/docs/9.2/static/sql-syntax-lexical.html#SQL-SYNTAX-SPECIAL-CHARS

This mechanism is required for the support of prepared statements.
It is nicer if I could make automatic conversions for various type and binary transfer,
but this is not implemented right now.
So all parameters are simply passed as strings at the moment.

Note this is the very initial implementation and the interface may change.

Any enhancement, feedback, or test case/program is welcome.
Especially, on what would be the best interface/syntax.

Best regards,
--
Tomoaki NISHIYAMA

Advanced Science Research Center,
Kanazawa University,
13-1 Takara-machi,
Kanazawa, 920-0934, Japan


On 2012/12/01, at 0:26, Dirk Eddelbuettel wrote:

>
> On 30 November 2012 at 15:05, James David Smith wrote:
> | Hi all,
> |
> | Sorry for the thread re-activation. I was wondering if anyone has
> | successfully used the syntax below with the library RPostgreSQL?
>
> Nope.
>
> I always expand the strings explicitly. It would be news to me of that
> worked. Good news, for sure, but still news...
>
> Dirk
>
>
> | dbGetQuery(con, "update foo set sal = :1 where empno = :2",
> | data = dat[,c("SAL","EMPNO")])
> |
> | I've been messing about with it but can't get it to work. I get the error:
> |
> | Error in postgresqlQuickSQL(conn, statement, ...) :
> | unused argument(s) (data = list(bc = c(NA, NA, NA etc.
> |
> | Thanks
> |
> | James
> |
> |
> |
> | On 28 September 2012 17:13, Denis Mukhin <denis.x...@oracle.com> wrote:
> | > James,
> | >
> | > I have never tried RPostgreSQL before but in ROracle which is also a DBI based interface you can do something like this:
> | >
> | > library(ROracle)
> | > con <- dbConnect(Oracle(), "scott", "tiger")
> | > dbGetQuery(con, "create table foo as select * from emp")
> | >
> | > dat <- dbGetQuery(con, "select * from foo")
> | > dat$SAL <- dat$SAL*10
> | > dbGetQuery(con, "update foo set sal = :1 where empno = :2",
> | > data = dat[,c("SAL","EMPNO")])
> | > dbCommit(con)
> | > dbGetQuery(con, "select * from foo")
> | >
> | > dbGetQuery(con, "drop table foo purge")
> | > dbDisconnect(con)
> | >
> | > Denis
> | >
> | > -----Original Message-----
> | > From: Sean Davis [mailto:sda...@mail.nih.gov]
> | > Sent: Friday, September 28, 2012 11:43 AM
> | > To: James David Smith
> | > Cc: r-si...@r-project.org
> | > Subject: Re: [R-sig-DB] R and PostgreSQL - Writing data?
> | >
> | > On Fri, Sep 28, 2012 at 10:36 AM, James David Smith <james.da...@gmail.com> wrote:
> | >> Hi Sean,
> | >>
> | >> Thanks for the reply. I'm familiar with UPDATE queries when working in
> | >> PostgreSQL, but not from within R. Would it look something like this?
> | >>
> | >> dbWriteTable(con, " UPDATE table SET ucam_no2 =
> | >> 'ucam_no2$interpolated_data' ")
> | >>
> | >> My problem is how to get the R data 'within' my SQL statement I think.
> | >
> | > To do an update, you'll need to loop through your data.frame and issue a dbSendQuery(). To create the SQL string, I often use something
> | > like:
> | >
> | > sprintf("UPDATE originalTable SET ucam_no2=%f WHERE originalTable.id = %d",....)
> | >
> | > You can't do this in one step, unfortunately. This is how UPDATE works and has nothing to do with R.
> | >
> | > Sean
> | >
> | >
> | >>
> | >> On 28 September 2012 15:19, Sean Davis <sda...@mail.nih.gov> wrote:
> | >>> On Fri, Sep 28, 2012 at 10:14 AM, James David Smith
> | >>> <james.da...@gmail.com> wrote:
> | >>>> Dear all,
> | >>>>
> | >>>> Sorry if this isn't quite the right place, but it's the first time
> | >> SendSave NowDiscardDraft autosaved at 15:36 (0 minutes ago) 33% full
> | >> Using 3.4 GB of your 10.1 GB
> | >> ©2012 Google - Terms & Privacy
> | >> Last account activity: 50 minutes ago
> | >> Details
> | >> People (2)
> | >> Sean Davis
> | >> Add to circles
> | >>
> | >> Show details
> | >> Ads – Why these ads?
> | >> Big Data Too Slow?
> | >> Real-Time Analytics for Big Data. Visual Drag & Drop UI. Quick & Easy
> | >> PentahoBigData.com Talend Open Source ESB Open Source ESB Based on
> | >> Apache CXF and Apache Camel. Free Download!
> | >> www.talend.com/Free_ESB_Software
> | >> Warp I/O for SQL Server
> | >> Speed SQL Server performance 3x Faster I/O, reduced storage
> | >> www.confio.com/warp-io Storage Container Sussex Ex-Shipping Containers
> | >> Sale & Hire Storage Container 0800 043 6311
> | >> www.CsShippingContainers.co.uk More about...
> | >> MS Access Database SQL »
> | >> Database »
> | >> Excel Database Query »
> | >> Oracle Database Problems »
> | >>
> | >>>> I've posted here. My issue is to do with writing to a PostgreSQL
> | >>>> database from within R. My situation is best explained by some R
> | >>>> code to start:
> | >>>>
> | >>>> #Connect to the database
> | >>>> con <- dbConnect(PostgreSQL(), user="postgres", password="password",
> | >>>> dbname="database")
> | >>>>
> | >>>> #Get some data out of the database.
> | >>>> ucam_no2$original_data <- dbGetQuery(con, "select ucam_no2 FROM
> | >>>> table")
> | >>>>
> | >>>> This returns say 10000 rows of data, but there is only data in about
> | >>>> half of those rows. What I want to do is interpolate the missing
> | >>>> data so I do this:
> | >>>>
> | >>>> #Generate some data
> | >>>> ucam_no2$interpolated_data <- na.approx(ucam_data$ucam_no2, na.rm =
> | >>>> FALSE)
> | >>>>
> | >>>> This works well and I now have 10000 rows of data with no empty cells.
> | >>>> I now want to write this back into my PostgresSQL database. Into the
> | >>>> same row that I took the data from in the first place. But I don't
> | >>>> know how. I can write to a new table with something like the below,
> | >>>> but what I'd really like to do is put the data back into the table I
> | >>>> got it from.
> | >>>>
> | >>>> # Try to write the data back
> | >>>> dbWriteTable(con, "new_data", ucam_no2$interpolated_data)
> | >>>
> | >>> Hi, James.
> | >>>
> | >>> You'll need to look into doing a SQL UPDATE. That is the standard
> | >>> way to "put data back into the table I got it from".
> | >>>
> | >>> Sean
> | >
> | > _______________________________________________
> | > R-sig-DB mailing list -- R Special Interest Group R-si...@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-db
> | >
> | > _______________________________________________
> | > R-sig-DB mailing list -- R Special Interest Group
> | > R-si...@r-project.org
> | > https://stat.ethz.ch/mailman/listinfo/r-sig-db
> |
> | _______________________________________________
> | R-sig-DB mailing list -- R Special Interest Group
> | R-si...@r-project.org
> | https://stat.ethz.ch/mailman/listinfo/r-sig-db
>
> --
> Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
>
> _______________________________________________
> R-sig-DB mailing list -- R Special Interest Group
> R-si...@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-db

hora...@ucr.edu

unread,
Apr 16, 2013, 7:19:30 PM4/16/13
to rpostgr...@googlegroups.com, Dirk Eddelbuettel, NISHIYAMA Tomoaki, James David Smith, r-sig-db

Hi, I would be interested in implementing what ever is required to support prepared queries. I was thinking of allowing dbSendQuery take a data frame instead of a vector, and then prepare the query once and run it on all rows of the data frame. This is basically what RSQLite does. I have already made a quick modification to RS_PostgreSQL_pqexecParams to call PQexecPrepared instead on an already prepared statement, and that worked. So it seems its mostly a case of modifying the C code to prepare the query first and then read through the data frame calling PQexecPrepared.
   The problem I'm trying to solve right now is being able to efficiently load 70 million chemical compounds into postgres. I know there are other avenues for accomplishing this, but using R is the best solution in this case.
   Please let me know how I can best help, how you want things done, etc. Thanks.

Kevin

NISHIYAMA Tomoaki

unread,
Apr 16, 2013, 8:13:01 PM4/16/13
to hora...@ucr.edu, NISHIYAMA Tomoaki, rpostgr...@googlegroups.com, Dirk Eddelbuettel, James David Smith, r-sig-db
Dear Kevin,

   The problem I'm trying to solve right now is being able to efficiently load 70 million chemical compounds into postgres. I know there are other avenues for accomplishing this, but using R is the best solution in this case. 

dbWriteTable() should be used to load all rows of a data frame to PostgreSQL.
This uses a single COPY and should be much faster than calling PQexecPrepared many times.

For prepared statement in RPostgreSQL, I think we should implement some mechanism to
access the prepared statement from R and make use of it by dbGetQuery or dbApply?
functions.

Best regards,

-- 

Tomoaki NISHIYAMA


Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi, 

Kanazawa, 920-0934, Japan



kho...@globalrecordings.net

unread,
Apr 17, 2013, 1:50:30 PM4/17/13
to rpostgr...@googlegroups.com, hora...@ucr.edu, NISHIYAMA Tomoaki, Dirk Eddelbuettel, James David Smith, r-sig-db
Tomoaki,
   dbWriteTable could work, I'd not looked too closely at it before. The problem I have though is that I need to leave the primary key field un-specified so that it will fill in the key from a sequence. Using a column of NA values does not work. This could be fixed by having dbWriteTable explicitly state the list of columns given in the data frame in the COPY command.  So, right now it executes "COPY tablename FROM", but could it be changed to grab the list of fields from the given data frame and then add them to the copy command, like so: "COPY tablename (col1, col2, ... ) FROM"? Then I would not need any prepared statements though ....

Thanks

Kevin

NISHIYAMA Tomoaki

unread,
Apr 17, 2013, 8:38:35 PM4/17/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu, Dirk Eddelbuettel, James David Smith, r-sig-db
Dear Kevin,

The problem I have though is that I need to leave the primary key
field un-specified so that it will fill in the key from a sequence.

I do not understand why you need to leave the primary key unspecified.
You can change the field characteristics with ALTER TABLE, ALTER SEQUENCE,
and so on.  To see what operation is needed you can prepare a small
example table and use pg_dump.

"row.names" are copied by dbWriteTable
to ensure that every record is distinguishable.
It is by default a text field.
If you need a serial primary key, you may create a new column, and compute
the initial values and add constraints and index. 
(or just alter the column may work depending on the data.frame)

Note that pg_dump/restore should have been very well tested by 
PostgreSQL developers to ensure the reliability and the speed.
So, it would be a very hard to invent a better method.

  So, right now it executes "COPY tablename FROM", but could it be changed to grab
the list of fields from the given data frame and then add them to the copy command,
like so: "COPY tablename (col1, col2, ... ) FROM"?

What would be the calling convention of dbWriteTable, then?
Current one is very simple, that is,
dbWrtieTable(con, tablename, data.frame)

If you want to write the list of columns, then wouldn't 
it easier to make a data.frame that have only those columns?

-- 

Tomoaki NISHIYAMA


Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi, 

Kanazawa, 920-0934, Japan



--
You received this message because you are subscribed to the Google Groups "RPostgreSQL Development and Discussion List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rpostgresql-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

kho...@globalrecordings.net

unread,
Apr 20, 2013, 12:58:50 AM4/20/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu, Dirk Eddelbuettel, James David Smith, r-sig-db


On Wednesday, April 17, 2013 5:38:35 PM UTC-7, Tomoaki wrote:
Dear Kevin,

The problem I have though is that I need to leave the primary key
field un-specified so that it will fill in the key from a sequence.

I do not understand why you need to leave the primary key unspecified.
You can change the field characteristics with ALTER TABLE, ALTER SEQUENCE,
and so on.  To see what operation is needed you can prepare a small
example table and use pg_dump.

"row.names" are copied by dbWriteTable
to ensure that every record is distinguishable.
It is by default a text field.
If you need a serial primary key, you may create a new column, and compute
the initial values and add constraints and index. 
(or just alter the column may work depending on the data.frame)

I want to use the serial primary key provided by postgres. I intend to 
do parallel inserts into the table and it is not practical to have the 
application generate unique ids. Thus, I need to leave it un-specified 
and let postgres generate valid ids.  This is not really a "one time 
load". It needs to work under normal database usages and be very fast at 
the same time, to the extent possible.
 
Note that pg_dump/restore should have been very well tested by 
PostgreSQL developers to ensure the reliability and the speed.
So, it would be a very hard to invent a better method.

  So, right now it executes "COPY tablename FROM", but could it be changed to grab
the list of fields from the given data frame and then add them to the copy command,
like so: "COPY tablename (col1, col2, ... ) FROM"?

What would be the calling convention of dbWriteTable, then?
Current one is very simple, that is,
dbWrtieTable(con, tablename, data.frame)

If you want to write the list of columns, then wouldn't 
it easier to make a data.frame that have only those columns?
Yes, that was my intention. The calling convention would be the same, 
just see what column names are in the given data frame

( sorry for the delay, I actually sent this from my email client 2 days ago and then the bounce got sent to my junk bin, just now found it)
 

NISHIYAMA Tomoaki

unread,
Apr 23, 2013, 10:00:04 AM4/23/13
to hora...@ucr.edu, NISHIYAMA Tomoaki, rpostgr...@googlegroups.com, Dirk Eddelbuettel, James David Smith, r-sig-db
Dear Kevin,

In case you do need computation to construct the data to store in the
database that is under normal work load, it is unlikely that 
the way the SQL is issued dominate the overall time.
The DBMS have to lock the record while it is accepting an access from
one client, and during that time other clients should just wait.

Thus, I need to leave it un-specified 
and let postgres generate valid ids. 
You can still let postgres generate a valid id by a specialized table
and use it to specify the primary key.  
There are quite a lot more of ways than what you state "I need".

So, while implementing prepared statement more useful is welcome,
I recommend not go such detail for now for your aim.
More important are to analyze what is the real bottleneck.

-- 

Tomoaki NISHIYAMA


Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi, 

Kanazawa, 920-0934, Japan



kho...@globalrecordings.net

unread,
Apr 23, 2013, 11:38:24 AM4/23/13
to rpostgr...@googlegroups.com, hora...@ucr.edu, NISHIYAMA Tomoaki
Ok. Could you perhaps apply the following patch to allow dbWriteTable to work with a sub-set of table columns. This would just make my life easier. Thanks.

diff --git a/RPostgreSQL/R/PostgreSQLSupport.R b/RPostgreSQL/R/PostgreSQLSupport.R
index ade695b..0b1dbf6 100644
--- a/RPostgreSQL/R/PostgreSQLSupport.R
+++ b/RPostgreSQL/R/PostgreSQLSupport.R
@@ -649,7 +658,7 @@ postgresqlWriteTable <- function(con, name, value, field.types, row.names = TRUE
     })
     oldenc <- dbGetQuery(new.con, "SHOW client_encoding")
     postgresqlpqExec(new.con, "SET CLIENT_ENCODING TO 'UTF8'")
-    sql4 <- paste("COPY", postgresqlTableRef(name), "FROM STDIN")
+    sql4 <- paste("COPY  ", postgresqlTableRef(name),"(",paste(names(value),collapse=","),") FROM STDIN")
     postgresqlpqExec(new.con, sql4)
     postgresqlCopyInDataframe(new.con, value)
     rs<-postgresqlgetResult(new.con)

NISHIYAMA Tomoaki

unread,
Apr 24, 2013, 9:47:28 PM4/24/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu
Dear Kevin,

I don't think this patch would make anyone happy.
Please attach a sample code that benefit from this change.
You should apply locally and test the result:- that's easy.

-- 

西山智明


金沢大学学際科学実験センター

ゲノム機能解析分野

(920-0934 金沢市宝町13−1)


Tomoaki NISHIYAMA

Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi

Kanazawa, 920-0934 Japan


kho...@globalrecordings.net

unread,
Apr 24, 2013, 10:12:52 PM4/24/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu
I have been using this patched code already and it makes me very happy. I can pass dbWriteTable a data frame without the primary key column and postgres happily fills it in for me using the default values. I don't see any way at all to accomplish the same thing with the existing dbWriteTable function.
Here is an example:

Table definition:   "compounds"
      Column        |  Type   |                            Modifiers                           
---------------------+---------+-----------------------------------------------------------------
 compound_id         | integer | not null default nextval('compounds_compound_id_seq'::regclass)
 name                | text    | not null default ''::text
 definition          | text    | not null
 definition_checksum | text    | not null
 format              | text    | not null

Data I'd like to insert:

>df = data.frame(name=c("a","b"),definition = c("d1","d2"), definition_checksum = c("dc1","dc2"), format=c("f1","f2") )

Note that this data frame does not have the compound_id column, this is a requirement for me.

With standard dbWriteTable:
>conn = dbConnect( <connection arguments here> )
>dbWriteTable(conn,"compounds",df,append=TRUE,row.names=FALSE)
Error in postgresqlgetResult(new.con) :
  RS-DBI driver: (could not Retrieve the result : ERROR:  invalid input syntax for integer: "a"
CONTEXT:  COPY compounds, line 1, column compound_id: "a"
)

This error is caused because it thinks that the first column in the data frame, a string, is actually the first column in the compounds table, which is an integer.

WIth patched dbWriteTable:
>dbWriteTable(conn,"compounds",df,append=TRUE,row.names=FALSE)                                                                                                                                                                           
[1] TRUE

This works, and the database fills in the value for the compound_id field automatically.

The given patch also should not break anything because prior to the patched version, all columns in the table must have been given in the data frame, so simply listing the columns in the COPY statement will not change anything. The patch allows future users to leave out some columns from the data frame and let the database fill in default values, when possible.

Kevin

NISHIYAMA Tomoaki

unread,
Apr 25, 2013, 12:48:33 AM4/25/13
to kho...@globalrecordings.net, rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu
Hi Kevin,

The given patch also should not break anything because prior to the patched version, all columns in the table must have been given in the data frame, so simply listing the columns in the COPY statement will not change anything. 

This patch, in fact, destroys many of the existing use of dbWriteTable.
(you can run the tests with sh check_with_vars.sh after installing modified RPostgreSQL)
Your usecase doesnot seem to fit the concept of dbWriteTable, which is 
intended to write the R data.frame to a table whose structure is defined
by the driver rather than a table that preexisted.

Perhaps, you are better coping the function under different name 
that represent what you want to do.

-- 

西山智明


金沢大学学際科学実験センター

ゲノム機能解析分野

(920-0934 金沢市宝町13−1)


Tomoaki NISHIYAMA

Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi

Kanazawa, 920-0934 Japan



kho...@globalrecordings.net

unread,
Apr 25, 2013, 1:17:34 AM4/25/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, hora...@ucr.edu
Yes, it seems I should just write my own function then to handle this. Thanks for working with me on this.

Kevin

Daniel Lister

unread,
Jun 20, 2013, 11:46:45 AM6/20/13
to rpostgr...@googlegroups.com, kho...@globalrecordings.net, NISHIYAMA Tomoaki, hora...@ucr.edu
I would support Kevin's request for a patch as I have the same issue, and had to resort to the same solution of forking the contents of postgresqlWriteTable into my own dbWriteData function (with default row.names=FALSE, append=TRUE since that suits my use case). IMHO it would be much better if a solution could be provided within the RPostgreSQL package and it would make dbWriteTable much more flexible. If other code is effected by specifying the list of columns to be copied, then couldn't it simply be made an extra parameter with appropriate default (eg use.col.names=FALSE).

My use case is where I want to write data from R to an existing database table (append), but the data.frame contains only a subset of the columns in the table - the remainder are columns which I'm happy to be populated with default values. These are identifiers from a sequence, timestamps etc, but could equally be a column which I don't want to cbind a lot of extra values to my data frame. If a column list is provided to the COPY FROM command then the defaults are handled gracefully, otherwise all columns and associated values must be provided.

You state that this use case doesn't fit the concept of dbWriteTable (the intention being to write to a new table, although append parameter contradicts this), but I cannot see any other function in the RPostgreSQL package which enables you to simply write a data frame to an existing table. The functionality is nearly there already, postgresqlWriteTable just needs to be a little more flexible and a extra parameter would be much appreciated.

Thanks in advance

Daniel


On Thursday, 25 April 2013 05:48:33 UTC+1, Tomoaki wrote:
The given patch also should not break anything because prior to the patched version, all columns in the table must have been given in the data frame, so simply listing the columns in the COPY statement will not change anything. 

This patch, in fact, destroys many of the existing use of dbWriteTable.
(you can run the tests with sh check_with_vars.sh after installing modified RPostgreSQL)
Your usecase doesnot seem to fit the concept of dbWriteTable, which is 
intended to write the R data.frame to a table whose structure is defined
by the driver rather than a table that preexisted.

Perhaps, you are better coping the function under different name 
that represent what you want to do.

<snip>

NISHIYAMA Tomoaki

unread,
Jun 20, 2013, 9:36:59 PM6/20/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, kho...@globalrecordings.net, hora...@ucr.edu
Hi,

You state that this use case doesn't fit the concept of dbWriteTable (the intention being to write to a new table, although append parameter contradicts this),

append does not contradict to the concept that the structure is defined
by the driver. Note that I did not write "new" table.
You can append additional data to a table created by dbWriteTable
from a data.frame with the same column structure. 

So being able to append to arbitrary preexisting database is not specification.
It may happen to work if you match the implementation detail.

Shortcoming of Kevin's patch was that it did not properly quote the column names.
The patch at the end does not break the test codes. 
Does this help?

Tomoaki
-- 

Tomoaki NISHIYAMA


Advanced Science Research Center,

Kanazawa University,

13-1 Takara-machi, 

Kanazawa, 920-0934, Japan


Index: RPostgreSQL/R/PostgreSQLSupport.R
===================================================================
--- RPostgreSQL/R/PostgreSQLSupport.R (revision 249)
+++ RPostgreSQL/R/PostgreSQLSupport.R (working copy)
@@ -588,11 +588,6 @@
 ## it with the values of the data.frame "value"
 ## TODO: This function should execute its sql as a single transaction,
 ##       and allow converter functions.
-## TODO: In the unlikely event that value has a field called "row_names"
-##       we could inadvertently overwrite it (here the user should set
-##       row.names=F)  I'm (very) reluctantly adding the code re: row.names,
-##       because I'm not 100% comfortable using data.frames as the basic
-##       data for relations.
 postgresqlWriteTable <- function(con, name, value, field.types, row.names = TRUE,
                                  overwrite = FALSE, append = FALSE, ..., allow.keywords = FALSE) {
     if(overwrite && append)
@@ -649,7 +644,7 @@
     })
     oldenc <- dbGetQuery(new.con, "SHOW client_encoding")
     postgresqlpqExec(new.con, "SET CLIENT_ENCODING TO 'UTF8'")
-    sql4 <- paste("COPY", postgresqlTableRef(name), "FROM STDIN")
+    sql4 <- paste("COPY", postgresqlTableRef(name),"(",paste(postgresqlQuoteId(names(value)),collapse=","),") FROM STDIN")
     postgresqlpqExec(new.con, sql4)
     postgresqlCopyInDataframe(new.con, value)
     rs<-postgresqlgetResult(new.con)
Index: RPostgreSQL/tests/dbWriteTableTest.R
===================================================================
--- RPostgreSQL/tests/dbWriteTableTest.R (revision 249)
+++ RPostgreSQL/tests/dbWriteTableTest.R (working copy)
@@ -32,13 +32,18 @@
         dbRemoveTable(con, "rockdata")
     }
 
-    dbWriteTable(con, "rockdata", rock)
+    dbWriteTable(con, "rockdata", rock[1:5,])
 
     ## run a simple query and show the query result
     res <- dbGetQuery(con, "select * from rockdata limit 10")
     print(res)
 
+    dbWriteTable(con, "rockdata", rock[6:15,], append=T)
 
+    ## run a simple query and show the query result
+    res <- dbGetQuery(con, "select * from rockdata limit 10")
+    print(res)
+
     ## cleanup
     if (dbExistsTable(con, "rockdata")) {
         print("Removing rockdata\n")

Daniel Lister

unread,
Jun 21, 2013, 6:11:07 AM6/21/13
to rpostgr...@googlegroups.com, NISHIYAMA Tomoaki, kho...@globalrecordings.net
On 21/06/2013 02:36, NISHIYAMA Tomoaki wrote:
> append does not contradict to the concept that the structure is defined
> by the driver. You can append additional data to a table created by dbWriteTable.

Point taken on append.

> So being able to append to arbitrary preexisting database is not
> specification. It may happen to work if you match the implementation detail.

For my case, reading and writing from/to an existing database is what
makes the package useful. It would be good if it's flexibility can be
extended to support this.

> Shortcoming of Kevin's patch was that it did not properly quote the
> column names. The patch at the end does not break the test codes. Does this help?

Yes, this works well for me. I can see why this version is correct and
it would be great to have it included in the code base.

Many thanks

Daniel

NISHIYAMA Tomoaki

unread,
Jun 21, 2013, 7:15:22 AM6/21/13
to rpostgr...@googlegroups.com, kho...@globalrecordings.net, NISHIYAMA Tomoaki
Hi,

> Yes, this works well for me. I can see why this version is correct and it would be great to have it included in the code base.


Daniel,
Thank you for testing the patch.
Committed to the svn at r251.

Kevin,
Thank you for the original patch close to the final. Major difference was just one step.

I am happier if the code works well for more people.

However, please note that the interface for such use is not determined yet
and we might decide to introduce optional argument like col.names for assignment,
or change in accordance with DBI interface or other drivers.
db* functions will be designed to fulfil the request by DBI and
if DBI spec is changed we will also change.

Best regards,
--
Tomoaki NISHIYAMA

Advanced Science Research Center,
Kanazawa University,
13-1 Takara-machi,
Kanazawa, 920-0934, Japan


Daniel Lister

unread,
Jun 21, 2013, 8:01:33 AM6/21/13
to rpostgr...@googlegroups.com
On 21/06/2013 12:15, NISHIYAMA Tomoaki wrote:
> Committed to the svn at r251.

Great! Thanks for this.

> However, please note that the interface for such use is not determined yet
> and we might decide to introduce optional argument like col.names for assignment,
> or change in accordance with DBI interface or other drivers.

Noted, and I'm happy to update when new db* functions become available.
However, it's good to have something which works for now.

jangorecki

unread,
Sep 26, 2015, 7:27:30 PM9/26/15
to RPostgreSQL Development and Discussion List
Sorry for resurrecting such an old discussion...
"append data to table which has an ID auto-incremented field" - was it addressed somehow since that discussion?
Regards,
Jan
Reply all
Reply to author
Forward
0 new messages