pgtap and sqitch

[Matt DeLuco]

I know there’s been past conversation on this topic, but how to use pgtap in sqitch is still not clear to me.

I’ve written some comprehensive tests using pgtap and as I move my scripts to sqitch I’m wondering how I can integrate my tests into the verify script for each change added with sqitch.

I’ve read this article: https://justatheory.com/2014/01/templating-tests-with-sqitch/ where a template is added for testing, but it’s not clear to me how I can have sqitch run additional scripts during the verify step. Is that what’s happening, or should I be including and running each test script from the verify script for each change?

Or are the use cases between pgtap and sqitch very different - that is, the sqitch verify scripts are intended only to verify the expected changes have been made, not to verify that things work as I expect (i.e. unit test expectations; pgtap’s purpose.)

Matt

[David E. Wheeler]

Hi Matt.

On Aug 29, 2021, at 11:53, Matt DeLuco <ma...@deluco.net> wrote:

> I’ve written some comprehensive tests using pgtap and as I move my scripts to sqitch I’m wondering how I can integrate my tests into the verify script for each change added with sqitch.

In general I don’t recommend this. The Sqitch verify scripts are intended to demonstrate that changes were successfully applied, while pgTAP tests are designed for testing schema structure and correctness of business logic. I also tend to design pgTAP tests to run against a new database, while verify scripts should be runnable against a production server without changing anything.

So I set up CI pipelines that deploy changes with --verify (or, more likely `deploy.verify=on in config) and immediately (but separately) run pgTAP tests through pg_prove. pgTAP tests are better for testing a whole schema, as well, while verify changes are just to check a specific change.

> I’ve read this article: https://justatheory.com/2014/01/templating-tests-with-sqitch/ where a template is added for testing, but it’s not clear to me how I can have sqitch run additional scripts during the verify step. Is that what’s happening, or should I be including and running each test script from the verify script for each change?

It can’t. The only scripts that Sqitch runs are deploy, verify, and revert. Anything else you run yourself. I wrote that post to help encourage writing tests at the same time. They’re not integrated in Sqitch deployments, just a way to automate their creation with scaffolding to help remind your developers to write tests for their changes

> Or are the use cases between pgtap and sqitch very different - that is, the sqitch verify scripts are intended only to verify the expected changes have been made, not to verify that things work as I expect (i.e. unit test expectations; pgtap’s purpose.)

Aaaaand I guess I should have read your message to the end, because, that’s it *exactly*.

That said, I think some people have used pgTAP in their verify scripts. The scripts have to load the pgTAP extension, then just do their thing. But pgTAP functions don’t exit psql on failure, do you would have to add some logic to the end of the test and throw an error if there are any failures. Again, very different use cases, and I recommend keeping as separate processes.

At least, that’s my opinion. Others may disagree.

Best,

David

[Matt DeLuco]

That all makes sense to me, David, thank you for the clarification!

Matt

> On Aug 29, 2021, at 19:19, David E. Wheeler <da...@justatheory.com> wrote:
>
> Hi Matt.