Upload local media files via API (v9.2)

[Tom Göller]

Hi,

I'm writing a python script to import photos into RS. There are around 15k of media files to import from a local folder on the client side to a remote RS server.

I had several approaches to get files uploaded but the only one that worked was referencing the media file via HTTP on the RS server.

I couldn't figure out how to upload local files.

It should be done by API calls as I can set metadata fields automatically extracted from the folder structure.

Folders media is stored in will be used as collection name to reflect the folder structure from local storage in RS.

What I tried already:

- created a resource (photo) via 'create_resource' call (worked)

- upload/assign a media file to the previous created resource via 'upload_file_by_url'

'upload_file_by_url' works only when I use a HTTP reference. If I use 'file://' to upload it fails.

What would be the best approach to upload files to a remote RS server via API calls?

As I'm new to RS help and hints appreciated!

Thx, Tom

[Charlie Alonso]

Hi Tom, perhasp staticsync could be a good method for massive upload as you have some features abut path folders mapping on fields or store the directories structures

If you could explain more about the organization of those 15k resources I could help you a bit more

Regards

Charlie Alonso

https://zonadam.com

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/5e9be71f-e8e0-46ca-affb-03af13b44ac4%40googlegroups.com.

[Tom Göller]

Hi Charlie,

what I have in mind is a script to push local stored media into RS without having any further requirements. Means, you configure where the media files are stored (local folder, external hard drive) and files are pushed into RS via the API.

So getting a import tool which I can adjust the way I need it.

I had a look at the staticsync as well. I'm not sure if that fits my need as I have no clue what kind of files are stored in the folder structure.

What I found out is that e.g. proprietary RAW files are not supported (e.g. Olympus ORF, Nikon NEF - just to name a few) and I'd like to preprocess them with dcraw (or a like) and import them afterwards into RS.

Did I get it right that with staticsync I'm able to keep the existing folder structure and just reference the files? And create the resources with extracted data from the path and/or the file names?

Maybe it would be an option if I do a pre-processing of the files and then configure staticsync.

Nevertheless, is it possible to push files via API into RS?

Thx, Tom

Am Montag, 4. Mai 2020 16:49:12 UTC+2 schrieb Charlie Alonso:

Hi Tom, perhasp staticsync could be a good method for massive upload as you have some features abut path folders mapping on fields or store the directories structures

If you could explain more about the organization of those 15k resources I could help you a bit more

Regards

Charlie Alonso

https://zonadam.com

El 4 may 2020 10:06 +0200, Tom Göller <th.g...@gmail.com>, escribió:

Hi,

I'm writing a python script to import photos into RS. There are around 15k of media files to import from a local folder on the client side to a remote RS server.

I had several approaches to get files uploaded but the only one that worked was referencing the media file via HTTP on the RS server.

I couldn't figure out how to upload local files.

It should be done by API calls as I can set metadata fields automatically extracted from the folder structure.

Folders media is stored in will be used as collection name to reflect the folder structure from local storage in RS.

What I tried already:

- created a resource (photo) via 'create_resource' call (worked)

- upload/assign a media file to the previous created resource via 'upload_file_by_url'

'upload_file_by_url' works only when I use a HTTP reference. If I use 'file://' to upload it fails.

What would be the best approach to upload files to a remote RS server via API calls?

As I'm new to RS help and hints appreciated!

Thx, Tom

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

To unsubscribe from this group and stop receiving emails from it, send an email to resour...@googlegroups.com.

[Jeff Nova]

I’m surprised to hear you say raw files aren’t supported. You can certainly use dcraw on the back end if you desire. I haven’t looked in a while at this but experience no problems with our system. - J

On May 4, 2020, at 10:23 AM, Tom Göller <th.go...@gmail.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/755ae5e9-a90d-4747-b795-6fe0c86d0620%40googlegroups.com.

[Tom Göller]

ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).

[Jeff Nova]

I don’t believe this is true but maybe we forked. But I am open to sharing the code. Can you send me some sample files to test? - J

> On May 4, 2020, at 3:20 PM, Tom Göller <th.go...@gmail.com> wrote:
>
> ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).

>
> --
> ResourceSpace: Open Source Digital Asset Management
> http://www.resourcespace.com
> ---
> You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/bd650646-fea5-4c74-bda4-a40a5b82f136%40googlegroups.com.

[Charlie Alonso]

Hi Tom, yes, you can push files from local via API but you should install a light http server so the API just specify a url for the resource you want to ingest.

A simple solution could be...

Set up a ftp service in RS server. The root directory of the FTP could be the Syncdir of RS and you could use a automatic sync file transfer app from your local pc to RS server

Here you have a link of a open source tool

https://freefilesync.org/

If you want some help to do a test, let me know

Cheers

Charlie Alonso

https://zonadam.com

El 5 may 2020 0:20 +0200, Tom Göller <th.go...@gmail.com>, escribió:

ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/bd650646-fea5-4c74-bda4-a40a5b82f136%40googlegroups.com.

[Tom Göller]

Sure. I will select some and post you a link to GDrive to download them. As I said I'm totally new to RS, so maybe there's still some config issues.

For my test I converted them to DNG via Adobe DNG converter after that it worked.

Am Dienstag, 5. Mai 2020 00:28:20 UTC+2 schrieb Jeff Nova:

I don’t believe this is true but maybe we forked. But I am open to sharing the code. Can you send me some sample files to test?  - J

> On May 4, 2020, at 3:20 PM, Tom Göller <th.g...@gmail.com> wrote:
>
> ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).
>
> --
> ResourceSpace: Open Source Digital Asset Management
> http://www.resourcespace.com
> ---
> You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to resour...@googlegroups.com.

[Tom Göller]

Hmm, if I get you right the RS server needs to know the location to pull the files from, right? FTP? Better not! At least SFTP or set up a share and mount it to the local client device.

I had some tests with staticsync which nearly worked but I had the following issues:

- the imported resources had no dimensions

- error messages in the Apache log saying exiftool not found even though it is installed at the correct path

- themes were not working - only after commenting the code (first IF statement) in themes.php

  enable_themes in the configs (config.php and cpnfig.default.php was set to TRUE)

previews of the imported resources were created and visible, also adding the path from the source as title.

But downloaded files were empty - so there's something fishy!

Am Dienstag, 5. Mai 2020 00:33:27 UTC+2 schrieb Charlie Alonso:

Hi Tom, yes, you can push files from local via API but you should install a light http server so the API just specify a url for the resource you want to ingest.

A simple solution could be...

Set up a ftp service in RS server. The root directory of the FTP could be the Syncdir of RS and you could use a automatic sync file transfer app from your local pc to RS server

Here you have a link of a open source tool

https://freefilesync.org/

If you want some help to do a test, let me know

Cheers

Charlie Alonso

https://zonadam.com

El 5 may 2020 0:20 +0200, Tom Göller <th.g...@gmail.com>, escribió:

ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

To unsubscribe from this group and stop receiving emails from it, send an email to resour...@googlegroups.com.

[Charlie Alonso]

Which version of RS are you using? What OS is RS installed on?

You can use SFTP as your prefer, if your synced files has no dimension it’s a problem of exiff tool . Did you check the system configuration check??

Charlie Alonso

https://zonadam.com

To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/92fdd6f0-5443-4fb4-b3c6-33003e8aab5a%40googlegroups.com.

[Jeff Nova]

Yes, we are using Exiftool to extract previews from NEF and ORF files, setting the ExifTool config to use BigImage, which automatically utilizes the largest preview available when there are more than one.  I can share the code with you.  The extracted previews are often better than what you might get from dcraw and certainly WAYYYY faster.  And can be done easily server-side.  Still, send me the samples so I can confirm all of the above; we generally dont see those file types.  - J

To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/68e92ec2-31bf-415d-9f6d-cd3f00d482f8%40googlegroups.com.

[Tom Göller]

Jeff, I think you're totally right! I had a closer look at ExifTool and I'm pretty sure it can handle my RAW files, if I set the right parameters. So I will have some trials to make it work myself - to learn how it works. But thank you for offering your config!

Another question according to my project - is it possible to read my folder structure, leave it untouched and only create the data needed for RS on the server side?

So it would be possible to keep the legacy structure but benefit from RS.

Does this work?

Thx, Tom

Am Dienstag, 5. Mai 2020 01:21:06 UTC+2 schrieb Jeff Nova:

Yes, we are using Exiftool to extract previews from NEF and ORF files, setting the ExifTool config to use BigImage, which automatically utilizes the largest preview available when there are more than one.  I can share the code with you.  The extracted previews are often better than what you might get from dcraw and certainly WAYYYY faster.  And can be done easily server-side.  Still, send me the samples so I can confirm all of the above; we generally dont see those file types.  - J

On Mon, May 4, 2020 at 3:37 PM Tom Göller <th.g...@gmail.com> wrote:

Sure. I will select some and post you a link to GDrive to download them. As I said I'm totally new to RS, so maybe there's still some config issues.

For my test I converted them to DNG via Adobe DNG converter after that it worked.

Am Dienstag, 5. Mai 2020 00:28:20 UTC+2 schrieb Jeff Nova:

I don’t believe this is true but maybe we forked. But I am open to sharing the code. Can you send me some sample files to test?  - J

> On May 4, 2020, at 3:20 PM, Tom Göller <th.g...@gmail.com> wrote:
>
> ORF, NEF e.g. is not supported. I have to convert them into DNG to be visible (preview image).
>
> --
> ResourceSpace: Open Source Digital Asset Management
> http://www.resourcespace.com
> ---
> You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to resour...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/bd650646-fea5-4c74-bda4-a40a5b82f136%40googlegroups.com.

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resour...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/resourcespace/68e92ec2-31bf-415d-9f6d-cd3f00d482f8%40googlegroups.com.

[Fred]

How did you do it? Would u mind to share your script?