Adding data to stardog container from another container on the same server

[Alex]

Hi,

I read that "Data add command sends the server only the file path when both are running on the same machine" [https://groups.google.com/a/clarkparsia.com/forum/#!searchin/stardog/$20%22data$20add%22$20--server-side/stardog/1BV2mJegUCg/DpnT6u11BDUJ]

Now for stardog running on a container technically I assume it's determined to be running on the host server with the same ip address and as another container on the same host even though from a logical stand point they would not have access to file paths on the server.

I solved issue a bit on a dev server with only 1 container by copying data to a volume that my stardog container has access to and then giving stardog the internal volume path to find the data.

But really what I would like to do in the production server is add data from a jenkins container to a stardog container running on the same server. This I haven't been able to do using stardog add. I assume because it's considered the same ip and thus only passes the file path.

Is that correct?

I note "-server-side: Server-side file IO. If this flag is set, it will be assumed that the input files passed reside on the server and the client will just send the file path to the server. "

Could an update to stardog data add  have a  -different-server  option that forces data transfer?

Let me know if I'm not correct at anything here and there is simple fix to add data from one container to another.

Regards
Alex

[Michael Grove]

On Tue, Oct 27, 2015 at 9:27 AM, Alex <alex....@gmail.com> wrote:

Hi,

I read that "Data add command sends the server only the file path when both are running on the same machine" [https://groups.google.com/a/clarkparsia.com/forum/#!searchin/stardog/$20%22data$20add%22$20--server-side/stardog/1BV2mJegUCg/DpnT6u11BDUJ]

Now for stardog running on a container technically I assume it's determined to be running on the host server with the same ip address and as another container on the same host even though from a logical stand point they would not have access to file paths on the server.

I solved issue a bit on a dev server with only 1 container by copying data to a volume that my stardog container has access to and then giving stardog the internal volume path to find the data.

But really what I would like to do in the production server is add data from a jenkins container to a stardog container running on the same server. This I haven't been able to do using stardog add. I assume because it's considered the same ip and thus only passes the file path.

It only considers the client to be on the same machine as the server if you're addressing the server via a `localhost` address.

 

Is that correct?

I note "-server-side: Server-side file IO. If this flag is set, it will be assumed that the input files passed reside on the server and the client will just send the file path to the server. "

Could an update to stardog data add  have a  -different-server  option that forces data transfer?

Any server address that is not `localhost` will transfer the files.

Cheers,

Mike

 

Let me know if I'm not correct at anything here and there is simple fix to add data from one container to another.

Regards
Alex

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Alex Muir]

On Tue, Oct 27, 2015 at 1:40 PM, Michael Grove <mi...@complexible.com> wrote:

It only considers the client to be on the same machine as the server if you're addressing the server via a `localhost` address.

 

Is that correct?

I note "-server-side: Server-side file IO. If this flag is set, it will be assumed that the input files passed reside on the server and the client will just send the file path to the server. "

Could an update to stardog data add  have a  -different-server  option that forces data transfer?

Any server address that is not `localhost` will transfer the files.

Thanks Mike that did work.. I thought I tested that but clearly my STARDOG_JAVA_ARGS="-Dstardog.default.cli.server=$1" was not properly set at the time of the testing or something...

One other question "I note that "File arguments are not allowed for server-side exports"... why the restriction?

It would have been nice from jenkins to pass file arguments to stardog data export to specify the filename and directory on the stardog container where I wanted the data. In end I pulled the data back into jenkins container o save it which does waste some time.

Regards

Alex
www.tilogeo.com

[Michael Grove]

On Wed, Oct 28, 2015 at 5:32 AM, Alex Muir <alex....@gmail.com> wrote:

On Tue, Oct 27, 2015 at 1:40 PM, Michael Grove <mi...@complexible.com> wrote:

It only considers the client to be on the same machine as the server if you're addressing the server via a `localhost` address.

 

Is that correct?

I note "-server-side: Server-side file IO. If this flag is set, it will be assumed that the input files passed reside on the server and the client will just send the file path to the server. "

Could an update to stardog data add  have a  -different-server  option that forces data transfer?

Any server address that is not `localhost` will transfer the files.

Thanks Mike that did work.. I thought I tested that but clearly my STARDOG_JAVA_ARGS="-Dstardog.default.cli.server=$1" was not properly set at the time of the testing or something...

One other question "I note that "File arguments are not allowed for server-side exports"... why the restriction?

Mostly, organization.  Easier to manage if they're all kept in a central place.  You can override the location on the server where they are stored.  Normally, exports are kept in `.exports` in your STARDOG_HOME, but you can set the property `export.dir` in your `stardog.properties` file to provide a different location.

Cheers,

Mike

 

It would have been nice from jenkins to pass file arguments to stardog data export to specify the filename and directory on the stardog container where I wanted the data. In end I pulled the data back into jenkins container o save it which does waste some time.

Regards

Alex
www.tilogeo.com

--

[Alex Muir]

On Wed, Oct 28, 2015 at 12:13 PM, Michael Grove <mi...@complexible.com> wrote:

Mostly, organization.  Easier to manage if they're all kept in a central place.  You can override the location on the server where they are stored.  Normally, exports are kept in `.exports` in your STARDOG_HOME, but you can set the property `export.dir` in your `stardog.properties` file to provide a different location.

Right,, I noted that in the documentation. I have a requirement to export files in three formats with specific names. So I have easily set that up without the server-side export however that works a bit slower,  bringing the data between containers.

We would prefer the server-side options saving data to a volume for efficiency. So with that setup we have to then create logic to mv the files to rename them after export I assume with some combinations of *'s and letters. Is that correct? There is really no way to give these exports a filename?

Regards

Alex
www.tilogeo.com

[Alex Muir]

On Thu, Oct 29, 2015 at 8:57 AM, Alex Muir <alex....@gmail.com> wrote:

We would prefer the server-side options saving data to a volume for efficiency. So with that setup we have to then create logic to mv the files to rename them after export I assume with some combinations of *'s and letters. Is that correct? There is really no way to give these exports a filename?

Just to be clear,, it's an automated process that is setup to run daily.

Regards

Alex
www.tilogeo.com

[Michael Grove]

Correct, you can only configure where they are stored.  If you need them stored in multiple separate places or want to change the names, you have to do that yourself.

Cheers,

Mike