the future of pgxn-tester
24 views
Skip to first unread message
Tomas Vondra
Nov 17, 2016, 3:28:46 AM11/17/16
to pgxn-...@googlegroups.com
Hi,
A rather long time ago I've developed pgxn-tester [1] with the primary
goal to provide feedback to developers of extensions, and to some extent
also to users. There's a bunch of issues that I think need to be
addressed, but I've been unable to do so - both because of lack of time
and clear ideas how. I'm considering what to do with the project.
I see about three issues with how the testing is done.
Firstly there's the security, or lack of it - the extensions compiled
and tested without any sandboxing, so it wouldn't be difficult to
compromise the system with a bit of C code. I've neglected the issue so
far, but that's a poor approach. While the system is not particularly
valuable, it'd still be a hassle. I've stopped the testing job until a
reasonable sandboxing solution is put into place.
Sadly, I don't have a good idea how to do such sandboxing, so this may
mean "forever".
There are other issues - lack of diversity of the test systems (single
Linux system, while it'd be good to also have Windows, etc.) and
imperfect automation, but those are secondary issues - people are
unlikely deploy the client until addressing the security thing.
So if you find pgxn-tester useful, have an idea how to address the
issues (particularly the sandboxing one) and want to help with it, let
me know.
regards
Tomas
[1] http://pgxn-tester.org
A rather long time ago I've developed pgxn-tester [1] with the primary
goal to provide feedback to developers of extensions, and to some extent
also to users. There's a bunch of issues that I think need to be
addressed, but I've been unable to do so - both because of lack of time
and clear ideas how. I'm considering what to do with the project.
I see about three issues with how the testing is done.
Firstly there's the security, or lack of it - the extensions compiled
and tested without any sandboxing, so it wouldn't be difficult to
compromise the system with a bit of C code. I've neglected the issue so
far, but that's a poor approach. While the system is not particularly
valuable, it'd still be a hassle. I've stopped the testing job until a
reasonable sandboxing solution is put into place.
Sadly, I don't have a good idea how to do such sandboxing, so this may
mean "forever".
There are other issues - lack of diversity of the test systems (single
Linux system, while it'd be good to also have Windows, etc.) and
imperfect automation, but those are secondary issues - people are
unlikely deploy the client until addressing the security thing.
So if you find pgxn-tester useful, have an idea how to address the
issues (particularly the sandboxing one) and want to help with it, let
me know.
regards
Tomas
[1] http://pgxn-tester.org
David E. Wheeler
Nov 17, 2016, 11:54:04 AM11/17/16
to pgxn-...@googlegroups.com
On Nov 17, 2016, at 12:28 AM, Tomas Vondra <t...@fuzzy.cz> wrote:
> Sadly, I don't have a good idea how to do such sandboxing, so this may mean "forever”.
:-( I suspect the simplest thing to do is to use a VM and maybe replace it once a day or something. Or maybe there’s a way to use a fresh Docker image for each build? I honestly don’t know; I suspect the Travis folks could make some suggestions based on their experience. They got a lot of GitHub tests running on their systems now.
> Sadly, I don't have a good idea how to do such sandboxing, so this may mean "forever”.
> There are other issues - lack of diversity of the test systems (single Linux system, while it'd be good to also have Windows, etc.) and imperfect automation, but those are secondary issues - people are unlikely deploy the client until addressing the security thing.
> So if you find pgxn-tester useful, have an idea how to address the issues (particularly the sandboxing one) and want to help with it, let me know.
Best,
David
David Fetter
Nov 17, 2016, 12:44:48 PM11/17/16
to pgxn-...@googlegroups.com
On Thu, Nov 17, 2016 at 09:28:39AM +0100, Tomas Vondra wrote:
> Hi,
>
> A rather long time ago I've developed pgxn-tester [1] with the primary goal
> to provide feedback to developers of extensions, and to some extent also to
> users. There's a bunch of issues that I think need to be addressed, but I've
> been unable to do so - both because of lack of time and clear ideas how. I'm
> considering what to do with the project.
>
> I see about three issues with how the testing is done.
>
> Firstly there's the security, or lack of it - the extensions compiled and
> tested without any sandboxing, so it wouldn't be difficult to compromise the
> system with a bit of C code. I've neglected the issue so far, but that's a
> poor approach. While the system is not particularly valuable, it'd still be
> a hassle. I've stopped the testing job until a reasonable sandboxing
> solution is put into place.
What does CircleCI do? I'm guessing there's something along the lines
> Hi,
>
> A rather long time ago I've developed pgxn-tester [1] with the primary goal
> to provide feedback to developers of extensions, and to some extent also to
> users. There's a bunch of issues that I think need to be addressed, but I've
> been unable to do so - both because of lack of time and clear ideas how. I'm
> considering what to do with the project.
>
> I see about three issues with how the testing is done.
>
> Firstly there's the security, or lack of it - the extensions compiled and
> tested without any sandboxing, so it wouldn't be difficult to compromise the
> system with a bit of C code. I've neglected the issue so far, but that's a
> poor approach. While the system is not particularly valuable, it'd still be
> a hassle. I've stopped the testing job until a reasonable sandboxing
> solution is put into place.
of a container or virtual machine. Could something use CircleCI's (or
similar) machinery?
> There are other issues - lack of diversity of the test systems
> (single Linux system, while it'd be good to also have Windows, etc.)
> and imperfect automation, but those are secondary issues - people
> are unlikely deploy the client until addressing the security thing.
they're using, even if it's something along the lines of, "this is
known to be a popular piece of software, and I haven't heard any
concerns about it that concern me." Apart from taking steps to
protect our infrastructure, which a VM-based approach would probably
cover, we aren't in the business of certifying things.
> So if you find pgxn-tester useful, have an idea how to address the issues
> (particularly the sandboxing one) and want to help with it, let me know.
all the source and docs in a place where others can help if they're
not already there.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
Jim Nasby
Nov 17, 2016, 1:13:22 PM11/17/16
to pgxn-...@googlegroups.com
On 11/17/16 11:44 AM, David Fetter wrote:
>> > Firstly there's the security, or lack of it - the extensions compiled and
>> > tested without any sandboxing, so it wouldn't be difficult to compromise the
>> > system with a bit of C code. I've neglected the issue so far, but that's a
>> > poor approach. While the system is not particularly valuable, it'd still be
>> > a hassle. I've stopped the testing job until a reasonable sandboxing
>> > solution is put into place.
> What does CircleCI do? I'm guessing there's something along the lines
> of a container or virtual machine. Could something use CircleCI's (or
> similar) machinery?
I'm no ops guy, but Vagrant would presumably make this pretty easy.
>> > Firstly there's the security, or lack of it - the extensions compiled and
>> > tested without any sandboxing, so it wouldn't be difficult to compromise the
>> > system with a bit of C code. I've neglected the issue so far, but that's a
>> > poor approach. While the system is not particularly valuable, it'd still be
>> > a hassle. I've stopped the testing job until a reasonable sandboxing
>> > solution is put into place.
> What does CircleCI do? I'm guessing there's something along the lines
> of a container or virtual machine. Could something use CircleCI's (or
> similar) machinery?
I think the bigger problem here is that PGXN needs some folks that spend
their time doing ops (and webdev) to step up and help. There's a bunch
of issues open against the website that David W. doesn't have time to
get to, but it's hard for anyone else to hack on them because that
requires getting a bunch of perl environment setup. That's an ops
problem. It's also hard for anyone that doesn't do a fair amount of web
development to grok how all the site fits together... that's why we need
some webdev folks.
It'd also be nice if the "official" Postgres community supported PGXN
better, but that's a different issue.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
Tomas Vondra
Nov 17, 2016, 3:04:24 PM11/17/16
to pgxn-...@googlegroups.com
Hi,
On 11/17/2016 06:44 PM, David Fetter wrote:
> On Thu, Nov 17, 2016 at 09:28:39AM +0100, Tomas Vondra wrote:
>> Hi,
>>
>> A rather long time ago I've developed pgxn-tester [1] with the primary goal
>> to provide feedback to developers of extensions, and to some extent also to
>> users. There's a bunch of issues that I think need to be addressed, but I've
>> been unable to do so - both because of lack of time and clear ideas how. I'm
>> considering what to do with the project.
>>
>> I see about three issues with how the testing is done.
>>
>> Firstly there's the security, or lack of it - the extensions
>> compiled and tested without any sandboxing, so it wouldn't be
>> difficult to
compromise the
>> system with a bit of C code. I've neglected the issue so far, but
that's a
>> poor approach. While the system is not particularly valuable, it'd
still be
>> a hassle. I've stopped the testing job until a reasonable
>> sandboxing solution is put into place.
>
> What does CircleCI do? I'm guessing there's something along the
> lines of a container or virtual machine. Could something use
> CircleCI's (or similar) machinery?
>
I have no idea how CircleCI works.
A lot of CI tools were designed for internal development purposes,
assuming only known good code (produced by the team) will be tested. So
while they provide orchestration, there's not much built-in protection.
At least that was the case 2 years ago, I think.
At least that was my experience with tools like Jenkins. But I admit
I've never done any serious research, and perhaps the situation has
changed with improvements in the area of containers (e.g. Docker).
FWIW This is also partially caused by available infrastructure. The
tests are currently running on RH 6.x machine with 2.6.32 kernel, where
the LXC probably lacks the new stuff (namespaces etc.).
That's fairly simple to fix (upgrade to RH 7.3 or whatever is the
current version), I simply never got to do that due to lack of clear
idea of what needs to be done.
>> There are other issues - lack of diversity of the test systems
>> (single Linux system, while it'd be good to also have Windows, etc.)
>> and imperfect automation, but those are secondary issues - people
>> are unlikely deploy the client until addressing the security thing.
>
> People vet software for "security," whatever that means in the
> context they're using, even if it's something along the lines of,
> "this is known to be a popular piece of software, and I haven't heard
> any concerns about it that concern me." Apart from taking steps to
> protect our infrastructure, which a VM-based approach would probably
> cover, we aren't in the business of certifying things.
>
I think there's a major difference when considering a particular popular
software package, and a tool that is intended to download, compile and
execute arbitrary packages from the Internet.
pgbuildfarm does not have this problem, because the code is passing
through a trusted repository, which is something pgxn-tester does not
have (for the tested packages).
While it's probably safe to assume people would deploy pgxn-tester into
a VM isolated from the rest of the system/network (at least that's what
I'd do, no matter if it's using something smart internally), it would be
fairly trivial to use pgxn-tester to establish "proxy" nodes, and route
arbitrary traffic through that. No fun.
>> So if you find pgxn-tester useful, have an idea how to address the
>> issues (particularly the sandboxing one) and want to help with it,
>> let me know.
>
> I don't know how many tuits I'll have on this, but it'd be good to get
> all the source and docs in a place where others can help if they're
> not already there.
>
Well, everything is available on github from the very beginning:
https://github.com/tvondra/pgxn-tester-client
https://github.com/tvondra/pgxn-tester-server
Maybe there were some minor tweaks that I've made directly on the
machine - I'll check.
I'm also fine with granting access to other people, assuming they're not
entirely anonymous and are considering contributing. Being the single
guy managing a repository is not much fun.
regards
Tomas
On 11/17/2016 06:44 PM, David Fetter wrote:
> On Thu, Nov 17, 2016 at 09:28:39AM +0100, Tomas Vondra wrote:
>> Hi,
>>
>> A rather long time ago I've developed pgxn-tester [1] with the primary goal
>> to provide feedback to developers of extensions, and to some extent also to
>> users. There's a bunch of issues that I think need to be addressed, but I've
>> been unable to do so - both because of lack of time and clear ideas how. I'm
>> considering what to do with the project.
>>
>> I see about three issues with how the testing is done.
>>
>> Firstly there's the security, or lack of it - the extensions
>> compiled and tested without any sandboxing, so it wouldn't be
>> difficult to
compromise the
>> system with a bit of C code. I've neglected the issue so far, but
that's a
>> poor approach. While the system is not particularly valuable, it'd
still be
>> a hassle. I've stopped the testing job until a reasonable
>> sandboxing solution is put into place.
>
> What does CircleCI do? I'm guessing there's something along the
> lines of a container or virtual machine. Could something use
> CircleCI's (or similar) machinery?
>
A lot of CI tools were designed for internal development purposes,
assuming only known good code (produced by the team) will be tested. So
while they provide orchestration, there's not much built-in protection.
At least that was the case 2 years ago, I think.
At least that was my experience with tools like Jenkins. But I admit
I've never done any serious research, and perhaps the situation has
changed with improvements in the area of containers (e.g. Docker).
FWIW This is also partially caused by available infrastructure. The
tests are currently running on RH 6.x machine with 2.6.32 kernel, where
the LXC probably lacks the new stuff (namespaces etc.).
That's fairly simple to fix (upgrade to RH 7.3 or whatever is the
current version), I simply never got to do that due to lack of clear
idea of what needs to be done.
>> There are other issues - lack of diversity of the test systems
>> (single Linux system, while it'd be good to also have Windows, etc.)
>> and imperfect automation, but those are secondary issues - people
>> are unlikely deploy the client until addressing the security thing.
>
> People vet software for "security," whatever that means in the
> context they're using, even if it's something along the lines of,
> "this is known to be a popular piece of software, and I haven't heard
> any concerns about it that concern me." Apart from taking steps to
> protect our infrastructure, which a VM-based approach would probably
> cover, we aren't in the business of certifying things.
>
software package, and a tool that is intended to download, compile and
execute arbitrary packages from the Internet.
pgbuildfarm does not have this problem, because the code is passing
through a trusted repository, which is something pgxn-tester does not
have (for the tested packages).
While it's probably safe to assume people would deploy pgxn-tester into
a VM isolated from the rest of the system/network (at least that's what
I'd do, no matter if it's using something smart internally), it would be
fairly trivial to use pgxn-tester to establish "proxy" nodes, and route
arbitrary traffic through that. No fun.
>> So if you find pgxn-tester useful, have an idea how to address the
>> issues (particularly the sandboxing one) and want to help with it,
>> let me know.
>
> I don't know how many tuits I'll have on this, but it'd be good to get
> all the source and docs in a place where others can help if they're
> not already there.
>
https://github.com/tvondra/pgxn-tester-client
https://github.com/tvondra/pgxn-tester-server
Maybe there were some minor tweaks that I've made directly on the
machine - I'll check.
I'm also fine with granting access to other people, assuming they're not
entirely anonymous and are considering contributing. Being the single
guy managing a repository is not much fun.
regards
Tomas
Tomas Vondra
Nov 17, 2016, 3:11:51 PM11/17/16
to pgxn-...@googlegroups.com
On 11/17/2016 07:13 PM, Jim Nasby wrote:
> On 11/17/16 11:44 AM, David Fetter wrote:
>>> > Firstly there's the security, or lack of it - the extensions
>>> compiled and
>>> > tested without any sandboxing, so it wouldn't be difficult to
>>> compromise the
>>> > system with a bit of C code. I've neglected the issue so far, but
>>> that's a
>>> > poor approach. While the system is not particularly valuable, it'd
>>> still be
>>> > a hassle. I've stopped the testing job until a reasonable sandboxing
>>> > solution is put into place.
>> What does CircleCI do? I'm guessing there's something along the lines
>> of a container or virtual machine. Could something use CircleCI's (or
>> similar) machinery?
>
> I'm no ops guy, but Vagrant would presumably make this pretty easy.
>
1) Installing pgxn-tester easily (with dependencies etc.).
2) Running the tests, ideally in a way that isolates each test so that
it can only access selected network, run for limited amount of time etc.
And after terminating the test, discarding all the changes.
My understanding is Vagrant would solve (1) fairly easily, but it's the
(2) item that bothers me way more.
> I think the bigger problem here is that PGXN needs some folks that
> spend their time doing ops (and webdev) to step up and help. There's
> a bunch of issues open against the website that David W. doesn't have
> time to get to, but it's hard for anyone else to hack on them because
> that requires getting a bunch of perl environment setup. That's an
> ops problem. It's also hard for anyone that doesn't do a fair amount
> of web development to grok how all the site fits together... that's
> why we need some webdev folks.
>
> It'd also be nice if the "official" Postgres community supported PGXN
> better, but that's a different issue.
regards
Tomas
Tomas Vondra
Nov 17, 2016, 3:19:51 PM11/17/16
to pgxn-...@googlegroups.com
On 11/17/2016 05:53 PM, David E. Wheeler wrote:
> On Nov 17, 2016, at 12:28 AM, Tomas Vondra <t...@fuzzy.cz> wrote:
>
>> Sadly, I don't have a good idea how to do such sandboxing, so this may mean "forever”.
>
> :-( I suspect the simplest thing to do is to use a VM and maybe replace it once a day or something. Or maybe there’s a way to use a fresh Docker image for each build? I honestly don’t know; I suspect the Travis folks could make some suggestions based on their experience. They got a lot of GitHub tests running on their systems now.
>
of operating systems - I'm a Linux user, and it's not that difficult do
write a bunch of bash scripts to test an extension on a number of
PostgreSQL versions. But testing the extension on OS X/macOS (or
whatever is the name today) or Windows is much more difficult.
I'm not sure if that is compatible with Docker - I recall it did not
work particularly well on non-Linux systems, but I know people were
working on improving the situation.
>> There are other issues - lack of diversity of the test systems (single Linux system, while it'd be good to also have Windows, etc.) and imperfect automation, but those are secondary issues - people are unlikely deploy the client until addressing the security thing.
>
> Agreed.
>
>> So if you find pgxn-tester useful, have an idea how to address the issues (particularly the sandboxing one) and want to help with it, let me know.
>
> Thank you for this, and I’d love it if someone was able to step up and help out with this. I wish I had the tuits to contribute, but I don’t even have time to put into PGXN improvements these days. Life yo, it goes places with or without you.
>
I think this is partially due to me doing much less extension
development, which somewhat limits the benefits of pgxn-tester for me
personally.
regards
Tomas
Jim Nasby
Nov 18, 2016, 1:23:05 AM11/18/16
to pgxn-...@googlegroups.com
On 11/17/16 2:11 PM, Tomas Vondra wrote:
> 2) Running the tests, ideally in a way that isolates each test so that
> it can only access selected network, run for limited amount of time etc.
> And after terminating the test, discarding all the changes.
>
> My understanding is Vagrant would solve (1) fairly easily, but it's the
> (2) item that bothers me way more.
AFAIK vagrant can handle that (though it might depend on the underlying
> 2) Running the tests, ideally in a way that isolates each test so that
> it can only access selected network, run for limited amount of time etc.
> And after terminating the test, discarding all the changes.
>
> My understanding is Vagrant would solve (1) fairly easily, but it's the
> (2) item that bothers me way more.
virtualization it's using).
BTW, it'd definitely be good to get windows testing happening... someone
recently posted some patches for the pgxn client on windows, but I don't
know that anyone can really do anything with them.
Reply all
Reply to author
Forward
0 new messages