Bug#966573: awscli: aws cli v2

[Franck Hamelin]

Package: awscli
Severity: normal

Dear Maintainer,

Amazon recommands to upgrade to V2, and my S3 service provider requires a v2. That leads me to investigate a bit around V2 and its absence on Debian (even in SID).
I’ve seen that amazon change their orientation (https://github.com/aws/aws-cli/issues/4947) making python hidden.
The tracker page points to the upstream new version 1.18.107 (https://tracker.debian.org/pkg/awscli) that's the last of the v1 branch but the v2 branch offers many newer versions, at writing time the latest is 2.0.35 (https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst). If the package awscli will « specialized » on V1 branch it would be a plus to state it explicitly in the package description.

Installation the amazon way is easy but probably not in line with the idea of Debian packaging system. So my question is : what are the plan for AWS CLI V2 on Debian ?

-- System Information:
Debian Release: 10.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-9-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages awscli depends on:
pn groff <none>
ii python3 3.7.3-1
pn python3-botocore <none>
pn python3-colorama <none>
pn python3-docutils <none>
pn python3-pyasn1 <none>
pn python3-rsa <none>
pn python3-s3transfer <none>
pn python3-yaml <none>

awscli recommends no packages.

awscli suggests no packages.

[Noah Meyerhans]

On Thu, Jul 30, 2020 at 10:12:54PM +0200, Franck Hamelin wrote:
>
> Installation the amazon way is easy but probably not in line with the idea of Debian packaging system. So my question is : what are the plan for AWS CLI V2 on Debian ?
>

You're the second one to ask about this recently, so I think the plan is
to start looking into it with greater urgency.

Thanks
noah

[Noah Meyerhans]

I spent a little bit of time on this today. Here are my findings so
far.

First, awscli v2 depends on a python-botocore v2, which isn't yet
officially released upstream. It does seem reasonable to package it,
but the blast radius is large enough that it's important to be careful
to avoid regressions.

Second, after updating python-botocore, there are still a few failing
tests that need investigation:

======================================================================
FAIL: test_with_year (tests.unit.customizations.codeartifact.test_adapter_login.TestRelativeExpirationTime)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/src/aws-cli/.pybuild/cpython3_3.8/build/tests/unit/customizations/codeartifact/test_adapter_login.py", line 454, in test_with_year
self.assertEqual(message, '11 months and 30 days')
AssertionError: '11 months and 31 days' != '11 months and 30 days'
- 11 months and 31 days
? ^
+ 11 months and 30 days
? ^


======================================================================
FAIL: test_can_set_default_yes_no_value (tests.unit.customizations.wizard.test_core.TestPlanner)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/src/aws-cli/.pybuild/cpython3_3.8/build/tests/unit/customizations/wizard/test_core.py", line 436, in test_can_set_default_yes_no_value
self.assertEqual(
AssertionError: Lists differ: [{'display': 'Yes', 'actual_value': 'yes'}, {'display':[24 chars]no'}] != [{'display': 'No', 'actual_value': 'no'}, {'display': '[24 chars]es'}]

First differing element 0:
{'display': 'Yes', 'actual_value': 'yes'}
{'display': 'No', 'actual_value': 'no'}

+ [{'actual_value': 'no', 'display': 'No'},
- [{'actual_value': 'yes', 'display': 'Yes'},
? ^ ^

+ {'actual_value': 'yes', 'display': 'Yes'}]
? ^ ^

- {'actual_value': 'no', 'display': 'No'}]

======================================================================
FAIL: test_getpreferredencoding_with_env_var (tests.unit.test_compat.TestGetPreferredEncoding)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/mock/mock.py", line 1800, in _inner
return f(*args, **kw)
File "/src/aws-cli/.pybuild/cpython3_3.8/build/tests/unit/test_compat.py", line 149, in test_getpreferredencoding_with_env_var
self.assertEqual(encoding, 'cp1252')
AssertionError: 'UTF-8' != 'cp1252'
- UTF-8
+ cp1252


----------------------------------------------------------------------
Ran 40615 tests in 1623.119s

FAILED (SKIP=3, errors=260, failures=6)
E: pybuild pybuild:352: test: plugin distutils failed with: exit code=1: cd /src/aws-cli/.pybuild/cpython3_3.8/build; python3.8 -m nose -v tests
dh_auto_test: error: pybuild --test --test-nose -i python{version} -p 3.8 returned exit code 13
make: *** [debian/rules:8: build] Error 13

[Gregor Riepl]

AWS has now locked the discussion on their bug report:
https://github.com/aws/aws-cli/issues/4947#issuecomment-793192340

Sadly, they will not change their stance towards publishing proper
source packages of awscli and botocore on PyPI. On the positive side,
they promised to bundle the source code in a manner that is easier to
build and package.

I suppose this will mean that the packaging scripts will need to be
overhauled when they do publish bundled sources...

[Noah Meyerhans]

Control: severity -1 wishlist
Control: forwarded -1 https://github.com/aws/aws-cli/issues/6186

awscli v2 remains quite difficult to package, but it seems that upstream
is looking to address this. See
https://github.com/aws/aws-cli/issues/6186 for details and tracking.

We'll continue to track upstream developments and see where things go.

noah

[Ross Vandegrift]

On Thu, Jun 17, 2021 at 09:56:10AM -0700, Noah Meyerhans wrote:
> awscli v2 remains quite difficult to package, but it seems that upstream
> is looking to address this. See
> https://github.com/aws/aws-cli/issues/6186 for details and tracking.

Using the source dist poc from https://github.com/aws/aws-cli/pull/6352 I've
made enough progress to get a packaged aws-cli v2 working. There's a lot more
that needs to be done, but idea of the above linked PR could work for us. I'm
going to document my findings here.

The branch is based on 2.2.1. All of its python dependencies except
aws-crt-python are packaged (though the versions in sid don't match the
requirements). I ignored this with no ill side effects so far to focus on
aws-crt-python.

aws-crt-python is the hard part. It depends on a bunch of C libraries which
follow a more modern development style. They:
- provide no abi/api stability guarantees.
- have versioned releases, but no one cares about the versions. The intention
is that everyone should be following the main branch.
- only nominally support dynamic linking (it can be enabled, but is not used or
tested by upstream).
- seem to be mostly used via source inclusion.

My first pass only produces -dev packages with headers and static libraries.
To test them out, build the debian/sid branch from these repos, in this order:
- https://salsa.debian.org/rvandegrift/aws-c-common
- https://salsa.debian.org/rvandegrift/aws-lc
- https://salsa.debian.org/rvandegrift/s2n-tls
- https://salsa.debian.org/rvandegrift/aws-c-cal
- https://salsa.debian.org/rvandegrift/aws-c-io
- https://salsa.debian.org/rvandegrift/aws-c-compression
- https://salsa.debian.org/rvandegrift/aws-checksums
- https://salsa.debian.org/rvandegrift/aws-c-http
- https://salsa.debian.org/rvandegrift/aws-c-mqtt
- https://salsa.debian.org/rvandegrift/aws-c-event-stream
- https://salsa.debian.org/rvandegrift/aws-c-auth
- https://salsa.debian.org/rvandegrift/aws-c-s3
- https://salsa.debian.org/rvandegrift/aws-crt-python
- https://salsa.debian.org/rvandegrift/aws-cli2-temp

With gbp and sbuild, you should be able to build each with:
gbp buildpackage --git-builder=sbuild -d unstable --extra-package=$PWD/../build-area/


A few known issues:

- some repos have tests disabled due to failing during builds. So far, I don't
know if these are real failures, or if upstream's build method.

- copyright attribution for aws-lc is very hard. It's a fork of Google's
BoringSSL, which is a fork of pre-3.0 OpenSSL.

- That also means that aws-lc inherits the openssl gpl incompatibility.

- the aws-cli2-temp repo is based on upstream, not our awscli repo. I was
intentionally being sloppy to quickly get through a test.

- aws-lc and s2n-tls may be hard to maintain. Both are complicated, security
critical crypto libraries.

Ross

[Ross Vandegrift]

On Tue, Oct 05, 2021 at 11:10:48PM -0600, Ross Vandegrift wrote:
> A few known issues:
>
> - some repos have tests disabled due to failing during builds. So far, I don't
> know if these are real failures, or if upstream's build method.
>
> - copyright attribution for aws-lc is very hard. It's a fork of Google's
> BoringSSL, which is a fork of pre-3.0 OpenSSL.
>
> - That also means that aws-lc inherits the openssl gpl incompatibility.
>
> - the aws-cli2-temp repo is based on upstream, not our awscli repo. I was
> intentionally being sloppy to quickly get through a test.
>
> - aws-lc and s2n-tls may be hard to maintain. Both are complicated, security
> critical crypto libraries.

I forgot to mention one more issue:

- this branch avoids the challenges with botocore by embedding a copy. This
decision is defended by the author in the github conversation, and I'm
convinced it's the best option right now. Still, it's not ideal.

Ross

[Noah Meyerhans]

Hi Ross,

On Tue, Oct 05, 2021 at 11:10:43PM -0600, Ross Vandegrift wrote:
> > awscli v2 remains quite difficult to package, but it seems that upstream
> > is looking to address this. See
> > https://github.com/aws/aws-cli/issues/6186 for details and tracking.
>
> Using the source dist poc from https://github.com/aws/aws-cli/pull/6352 I've
> made enough progress to get a packaged aws-cli v2 working. There's a lot more
> that needs to be done, but idea of the above linked PR could work for us. I'm
> going to document my findings here.

So, only a year later, I've picked this up and made some additional
progress:

I have no name!@b02f1db79f9e:/src$ aws --version
aws-cli/2.8.7 Python/3.10.8 Linux/6.0.0-2-amd64 source/x86_64.debian prompt/off
I have no name!@b02f1db79f9e:/src$ dpkg -s awscli | grep Version
Version: 2.8.7-1

> aws-crt-python is the hard part. It depends on a bunch of C libraries which
> follow a more modern development style. They:
> - provide no abi/api stability guarantees.
> - have versioned releases, but no one cares about the versions. The intention
> is that everyone should be following the main branch.
> - only nominally support dynamic linking (it can be enabled, but is not used or
> tested by upstream).
> - seem to be mostly used via source inclusion.

Indeed, none of this has improved in the past year. In fact, we've
actually gained a few more such dependencies that didn't exist last year
(aws-c-sdkutils, at least).

We might actually be able to make a case that all the aws-c-* libraries
end up being easier to maintain if we ship them as private internal
components of the aws-crt-python package rather than as first class
packages that could end up becoming dependencies of other downstream
packages.

> - some repos have tests disabled due to failing during builds. So far, I don't
> know if these are real failures, or if upstream's build method.

I think I've got most of these fixed.

> - copyright attribution for aws-lc is very hard. It's a fork of Google's
> BoringSSL, which is a fork of pre-3.0 OpenSSL.
>
> - That also means that aws-lc inherits the openssl gpl incompatibility.

Here's the good news: We don't actually need aws-lc at all. awscli v2
and its various dependencies (including s2n-tls) can build against
OpenSSL 3.

> - the aws-cli2-temp repo is based on upstream, not our awscli repo. I was
> intentionally being sloppy to quickly get through a test.

Same. I essentially Debianized the upstream v2 repo from scratch,
pulling in some of your packaging metadata as it made sense. Given that
v2 is developed on a different branch and by now differs quite
significantly from v1, a case could be made for introducing a new
awscli2 package as a new source package and retiring the original awscli
package. However, the debian package metadata isn't really all that
complex, so it may not actually be necessary.

> - aws-lc and s2n-tls may be hard to maintain. Both are complicated, security
> critical crypto libraries.

Fortunately, aws-lc isn't an issue. But s2n-tls remains one. Not sure
we're going to be able to do anything about that. The difficult thing
is that it's typically expected to be used as a statically linked
library, which means updates end up being tedious.

I haven't pushed my changes anywhere, yet. Once I do, the remaining
tasks will be to any lintian issues or other obvious problems and get
these packages into NEW. I think they're in reasonably good shape, but
we don't have a lot of time before bookworm starts freezing, so I'd love
any help with these steps.

We have to get at least the following packages into the archive:

aws-c-auth_0.6.18-1_amd64.changes
aws-c-cal_0.5.20-1_amd64.changes
aws-c-common_0.8.4-1_amd64.changes
aws-c-compression_0.2.15-1_amd64.changes
aws-c-event-stream_0.2.15-1_amd64.changes
aws-c-http_0.6.23-1_amd64.changes
aws-c-io_0.13.6-1_amd64.changes
aws-c-mqtt_0.7.12-1_amd64.changes
aws-c-s3_0.1.51-1_amd64.changes
aws-c-sdkutils_0.1.4-1_amd64.changes
aws-checksums_0.1.13-1_amd64.changes
aws-crt-python_0.15.1-1_amd64.changes
s2n-tls_1.3.26-1_amd64.changes

[Noah Meyerhans]

On Mon, Oct 31, 2022 at 10:50:19PM -0700, Noah Meyerhans wrote:
> > aws-crt-python is the hard part. It depends on a bunch of C libraries which
> > follow a more modern development style. They:
> > - provide no abi/api stability guarantees.
> > - have versioned releases, but no one cares about the versions. The intention
> > is that everyone should be following the main branch.
> > - only nominally support dynamic linking (it can be enabled, but is not used or
> > tested by upstream).

Actually, taking a look at Amazon Linux 2022's packaging of these
libraries, I see that they're distributed as .so files, and the
aws-crt-python packagin declares runtime dependencies on these
dynamically loaded libraries. So I don't think we're in uncharted
waters if we decide to ship .so libraries.

noah

[Ross Vandegrift]

Hi Noah,

On Mon, Oct 31, 2022 at 10:50:19PM -0700, Noah Meyerhans wrote:

> On Tue, Oct 05, 2021 at 11:10:43PM -0600, Ross Vandegrift wrote:
> > > awscli v2 remains quite difficult to package, but it seems that upstream
> > > is looking to address this. See
> > > https://github.com/aws/aws-cli/issues/6186 for details and tracking.
> >
> > Using the source dist poc from https://github.com/aws/aws-cli/pull/6352 I've
> > made enough progress to get a packaged aws-cli v2 working. There's a lot more
> > that needs to be done, but idea of the above linked PR could work for us. I'm
> > going to document my findings here.
>
> So, only a year later, I've picked this up and made some additional
> progress:
>
> I have no name!@b02f1db79f9e:/src$ aws --version
> aws-cli/2.8.7 Python/3.10.8 Linux/6.0.0-2-amd64 source/x86_64.debian prompt/off
> I have no name!@b02f1db79f9e:/src$ dpkg -s awscli | grep Version
> Version: 2.8.7-1

Great news!

> > - some repos have tests disabled due to failing during builds. So far, I don't
> > know if these are real failures, or if upstream's build method.
>
> I think I've got most of these fixed.

🤘

> > - copyright attribution for aws-lc is very hard. It's a fork of Google's
> > BoringSSL, which is a fork of pre-3.0 OpenSSL.
> >
> > - That also means that aws-lc inherits the openssl gpl incompatibility.
>
> Here's the good news: We don't actually need aws-lc at all. awscli v2
> and its various dependencies (including s2n-tls) can build against
> OpenSSL 3.

🤘

> > - the aws-cli2-temp repo is based on upstream, not our awscli repo. I was
> > intentionally being sloppy to quickly get through a test.
>
> Same. I essentially Debianized the upstream v2 repo from scratch,
> pulling in some of your packaging metadata as it made sense. Given that
> v2 is developed on a different branch and by now differs quite
> significantly from v1, a case could be made for introducing a new
> awscli2 package as a new source package and retiring the original awscli
> package. However, the debian package metadata isn't really all that
> complex, so it may not actually be necessary.

Is there any reason to have both versions available to install at once?

> > - aws-lc and s2n-tls may be hard to maintain. Both are complicated, security
> > critical crypto libraries.
>
> Fortunately, aws-lc isn't an issue. But s2n-tls remains one. Not sure
> we're going to be able to do anything about that. The difficult thing
> is that it's typically expected to be used as a statically linked
> library, which means updates end up being tedious.

Agreed, there's nothing we can do about it. But I think it's good news to be
able to use OpenSSL and just have one highly security sensitive package.

> I haven't pushed my changes anywhere, yet. Once I do, the remaining
> tasks will be to any lintian issues or other obvious problems and get
> these packages into NEW. I think they're in reasonably good shape, but
> we don't have a lot of time before bookworm starts freezing, so I'd love
> any help with these steps.

I might have some time to help. Would it be useful to transfer my original
repos to the cloud-team group?

Ross

[Noah Meyerhans]

On Wed, Nov 02, 2022 at 07:55:51PM -0700, Ross Vandegrift wrote:
> > > - the aws-cli2-temp repo is based on upstream, not our awscli repo. I was
> > > intentionally being sloppy to quickly get through a test.
> >
> > Same. I essentially Debianized the upstream v2 repo from scratch,
> > pulling in some of your packaging metadata as it made sense. Given that
> > v2 is developed on a different branch and by now differs quite
> > significantly from v1, a case could be made for introducing a new
> > awscli2 package as a new source package and retiring the original awscli
> > package. However, the debian package metadata isn't really all that
> > complex, so it may not actually be necessary.
>
> Is there any reason to have both versions available to install at once?

No, there isn't. And both should be installed as /usr/bin/aws anyway.

At this point my v2 package builds as awscli_2.8.7-1_all.deb and should
be a direct upgrade from the current 1.x packages.

> > Fortunately, aws-lc isn't an issue. But s2n-tls remains one. Not sure
> > we're going to be able to do anything about that. The difficult thing
> > is that it's typically expected to be used as a statically linked
> > library, which means updates end up being tedious.
>
> Agreed, there's nothing we can do about it. But I think it's good news to be
> able to use OpenSSL and just have one highly security sensitive package.

I just uploaded s2n, so it should be in the NEW queue momentarily.
Sources are at https://salsa.debian.org/cloud-team/s2n-tls. So we'll
see how this goes.

> > I haven't pushed my changes anywhere, yet. Once I do, the remaining
> > tasks will be to any lintian issues or other obvious problems and get
> > these packages into NEW. I think they're in reasonably good shape, but
> > we don't have a lot of time before bookworm starts freezing, so I'd love
> > any help with these steps.
>
> I might have some time to help. Would it be useful to transfer my original
> repos to the cloud-team group?

I don't think so. There were a couple repos where you had made a commit
on the upstream branch, which I blew away in order to stay in sync with
upstream commits. So I'd need to force push. Either way, I think the
repos I have locally can be pushed to empty salsa repos.

Honestly, filing some of the ITPs would be quite helpful at this point.
We'll need to get the following projects packaged:

aws-c-auth
aws-c-cal
aws-c-common
aws-c-compression
aws-c-event-stream
aws-c-http
aws-c-io
aws-c-mqtt
aws-c-s3
aws-c-sdkutils
aws-checksums
aws-crt-python

[Ross Vandegrift]

On Wed, Nov 02, 2022 at 10:26:50PM -0700, Noah Meyerhans wrote:
> Honestly, filing some of the ITPs would be quite helpful at this point.
> We'll need to get the following projects packaged:
>
> aws-c-auth
> aws-c-cal

> aws-c-compression
> aws-c-event-stream
> aws-c-http

I got this far, and then:

> SMTP send failure: {'sub...@bugs.debian.org': (451, b'sorry, only 5 reports per hour for
> submission')}. You can retry, or save the report and exit. Do you want to retry [Y|n|q|?]?

Ross

[Noah Meyerhans]

On Thu, Nov 03, 2022 at 09:35:14PM -0700, Ross Vandegrift wrote:
> > Honestly, filing some of the ITPs would be quite helpful at this point.
> > We'll need to get the following projects packaged:
> >
> > aws-c-auth
> > aws-c-cal
> > aws-c-compression
> > aws-c-event-stream
> > aws-c-http
>
> I got this far, and then:
>
> > SMTP send failure: {'sub...@bugs.debian.org': (451, b'sorry, only 5 reports per hour for
> > submission')}. You can retry, or save the report and exit. Do you want to retry [Y|n|q|?]?

That's hilarious.

I guess the rust team has been uploading their random crate dependencies
without an ITP bug at all, so we're probably not going to generate too
much unhappiness if we skip them.

One nice thing about having the wnpp bugs is that we can add them as
blockers to this bug, which is helpful for tracking progress:

$ bts block 966573 with <wnpp bug>

I uploaded aws-c-common and aws-checksums today. All the git repos
backing my uploads are owned by the cloud-team on salsa. I'll try to
get another couple uploads done tomorrow.

noah

[Bastian Blank]

On Tue, Oct 05, 2021 at 11:10:43PM -0600, Ross Vandegrift wrote:

> My first pass only produces -dev packages with headers and static libraries.
> To test them out, build the debian/sid branch from these repos, in this order:
> - https://salsa.debian.org/rvandegrift/aws-c-common

Are you sure this library can have a 1 as ABI? Can you please reproduce
the ABI stability promisses?

I see in the public JSON headers the following interface changes:

| -typedef bool(aws_json_on_member_encountered_const_fn)(
| +typedef int(aws_json_on_member_encountered_const_fn)(
| const struct aws_byte_cursor *key,
| const struct aws_json_value *value,
| + bool *out_should_continue,
| void *user_data);

That's neither source nor binary compatible.

I don't think any of those libraries are reasonable ABI stable to be
maintained as shaed libraries.

Bastian

--
Warp 7 -- It's a law we can live with.

[Noah Meyerhans]

On Fri, Nov 04, 2022 at 08:32:25AM +0100, Bastian Blank wrote:
> > My first pass only produces -dev packages with headers and static libraries.
> > To test them out, build the debian/sid branch from these repos, in this order:
> > - https://salsa.debian.org/rvandegrift/aws-c-common
>
> Are you sure this library can have a 1 as ABI? Can you please reproduce
> the ABI stability promisses?
>
> I see in the public JSON headers the following interface changes:
>
> | -typedef bool(aws_json_on_member_encountered_const_fn)(
> | +typedef int(aws_json_on_member_encountered_const_fn)(
> | const struct aws_byte_cursor *key,
> | const struct aws_json_value *value,
> | + bool *out_should_continue,
> | void *user_data);
>
> That's neither source nor binary compatible.
>
> I don't think any of those libraries are reasonable ABI stable to be
> maintained as shaed libraries.

Allegedly upstream has recently committed to proper SONAME and ABI
management in support of efforts to get these packages accepted into
Fedora. I'll see if I can find meaningful evidence of this...

[Bastian Blank]

On Fri, Nov 04, 2022 at 09:08:22AM -0700, Noah Meyerhans wrote:
> > Are you sure this library can have a 1 as ABI? Can you please reproduce
> > the ABI stability promisses?

> Allegedly upstream has recently committed to proper SONAME and ABI
> management in support of efforts to get these packages accepted into
> Fedora. I'll see if I can find meaningful evidence of this...

The SOVERSION was set two years ago:

https://github.com/awslabs/aws-c-common/pull/702

So some more information would be nice.

Bastian

--
War is never imperative.
-- McCoy, "Balance of Terror", stardate 1709.2

[Noah Meyerhans]

On Fri, Nov 04, 2022 at 05:37:21PM +0100, Bastian Blank wrote:
> > > Are you sure this library can have a 1 as ABI? Can you please reproduce
> > > the ABI stability promisses?
> > Allegedly upstream has recently committed to proper SONAME and ABI
> > management in support of efforts to get these packages accepted into
> > Fedora. I'll see if I can find meaningful evidence of this...
>
> The SOVERSION was set two years ago:
>
> https://github.com/awslabs/aws-c-common/pull/702
>
> So some more information would be nice.

According to the person who handled the Fedora review for aws-c-common
(https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2049379), davdunc
had indicated he'd take care for SONAME management until upstream gets
things worked out. I haven't been able to get in touch with davdunc
yet to confirm this.

Given the number of packages involved, though, I expect it'll take a
while for everything to have a properly managed SONAME. Given that, I
see a few alternatives:

1. We package aws-crt-python with all the aws-c-* packages included in a
single source package. Given that upstream maintains this structure
in a single repo using git submodules and that their build system
supports it directly, it appears to be the way they expect the package
to be consumed.

2. We package the individual aws-c-* dependencies but only ship static
libraries, and handle them as standard buid-deps for aws-crt-python.

3. We manage the SONAME versions ourselves until upstream does it for
us.

4. We ignore awscli v2 and continue shipping v1.

I actually prefer #1 and suggest we do that. My reasoning is:

1. It's well supported by upstream,

2. It prevents other packages from picking up the aws-c-* packages as
dependencies before they expose a stable API/ABI,

3. It is simple to split out the submodules into standalone packages
one-by-one as their interfaces stabilize.

4. It's quite simple to implement

What do folks think of this idea?

noah

[Ross Vandegrift]

On Fri, Nov 04, 2022 at 02:56:45PM -0700, Noah Meyerhans wrote:
> Given the number of packages involved, though, I expect it'll take a
> while for everything to have a properly managed SONAME. Given that, I
> see a few alternatives:
>
> 1. We package aws-crt-python with all the aws-c-* packages included in a
> single source package. Given that upstream maintains this structure
> in a single repo using git submodules and that their build system
> supports it directly, it appears to be the way they expect the package
> to be consumed.
>
> 2. We package the individual aws-c-* dependencies but only ship static
> libraries, and handle them as standard buid-deps for aws-crt-python.
>
> 3. We manage the SONAME versions ourselves until upstream does it for
> us.
>
> 4. We ignore awscli v2 and continue shipping v1.
>
> I actually prefer #1 and suggest we do that. My reasoning is:
>
> 1. It's well supported by upstream,
>
> 2. It prevents other packages from picking up the aws-c-* packages as
> dependencies before they expose a stable API/ABI,
>
> 3. It is simple to split out the submodules into standalone packages
> one-by-one as their interfaces stabilize.
>
> 4. It's quite simple to implement
>
> What do folks think of this idea?

Sounds reasonable. My initial thought was #2, but I hadn't considered the
value of insulating others from upstream's changes.

Ross