[JIRA] (JENKINS-40654) Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

5 views
Skip to first unread message

cleclerc@cloudbees.com (JIRA)

unread,
Dec 23, 2016, 4:12:03 AM12/23/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc created an issue
 
Jenkins / Improvement JENKINS-40654
Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)
Issue Type: Improvement Improvement
Assignee: Alexander A
Components: s3-plugin
Created: 2016/Dec/23 9:11 AM
Priority: Minor Minor
Reporter: Cyrille Le Clerc

Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

To do this with the AWS SDK, you have to provide a file classpath://com/amazonaws/partitions/override/endpoints.json using the JSON format defined in [com/amazonaws/partitions/endpoints.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json)

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

cleclerc@cloudbees.com (JIRA)

unread,
Dec 23, 2016, 4:15:01 AM12/23/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc updated an issue
Change By: Cyrille Le Clerc
Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

To do this with the AWS SDK, you have to provide a file {{classpath://com/amazonaws/partitions/override/endpoints.json}} using the JSON format defined in  [com/amazonaws/partitions/endpoints.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json ) ]

cleclerc@cloudbees.com (JIRA)

unread,
Dec 23, 2016, 4:15:02 AM12/23/16
to jenkinsc...@googlegroups.com
Cyrille Le Clerc updated an issue
Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

To do this with the AWS SDK, you have to provide a file {{classpath://com/amazonaws/partitions/override/endpoints.json}} using the JSON format defined in   [com/amazonaws/partitions/endpoints.json| https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json ]

cleclerc@cloudbees.com (JIRA)

unread,
Dec 23, 2016, 4:22:01 AM12/23/16
to jenkinsc...@googlegroups.com

scm_issue_link@java.net (JIRA)

unread,
Dec 23, 2016, 7:15:02 AM12/23/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Cyrille Le Clerc
Path:
src/main/java/hudson/plugins/s3/ClientHelper.java
src/main/java/hudson/plugins/s3/Entry.java
src/main/java/hudson/plugins/s3/S3BucketPublisher.java
http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8
Log:
JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider, stop relying on a static lists and constants.

  • Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com`
  • Replace
  • `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")`
  • `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region`
  • For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region.

The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json .

A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

scm_issue_link@java.net (JIRA)

unread,
Dec 23, 2016, 7:15:02 AM12/23/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Akbashev Alexander
Path:
README.md
src/main/java/hudson/plugins/s3/ClientHelper.java
src/main/java/hudson/plugins/s3/Entry.java
src/main/java/hudson/plugins/s3/S3BucketPublisher.java
http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2
Log:
Merge pull request #100 from cyrille-leclerc/master

JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

scm_issue_link@java.net (JIRA)

unread,
Dec 23, 2016, 7:15:02 AM12/23/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Cyrille Le Clerc
Path:
README.md
src/main/java/hudson/plugins/s3/ClientHelper.java
src/main/java/hudson/plugins/s3/Entry.java
http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6
Log:
JENKINS-40654 @Jimilian’s recommendations

  • Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md
  • Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once)
  • Cleanup imports

Simon.Richter@hogyros.de (JIRA)

unread,
Jul 3, 2019, 5:23:03 AM7/3/19
to jenkinsc...@googlegroups.com

We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally.

It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

cleclerc@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 9:45:02 AM7/8/19
to jenkinsc...@googlegroups.com

@simonrichter can't you overwrite the endpoints.json file to add your S3 endpoint? You may have to introduce and "s3.endpoint" ; you may also have to define an "s3-control.endpoint".

https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

cleclerc@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 9:46:04 AM7/8/19
to jenkinsc...@googlegroups.com
Cyrille Le Clerc edited a comment on Improvement JENKINS-40654
@ [~ simonrichter ] can't you overwrite the {{endpoints.json}} file to add your S3 endpoint? You may would have to introduce and "s3.endpoint" I think ; you may also have to define an "s3-control.endpoint".

https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367
Reply all
Reply to author
Forward
0 new messages