Jira (PUP-9726) Refactor long running acceptance tests to exectute in less than 45 seconds

[John Duarte (JIRA)]

John Duarte created an issue

Puppet / PUP-9726 Refactor long running acceptance tests to exectute in less than 45 seconds Issue Type: Task Assignee: Unassigned Created: 2019/05/30 11:59 AM Priority: Normal Reporter: John Duarte As a puppet tester I want to keep the execution time of individual acceptance tests under 45 seconds So that I can get test feedback faster Acceptance tests are expensive. The following acceptance tests take longer than 45 seconds to execute on the reference RHEL 7 platform. These tests should be refactored to reduce there execution time below 45 seconds (preferably below 30 seconds). Tests that take longer than 45 seconds to run on RHEL7 along with there execution time: acceptance/tests/direct_puppet/cached_catalog_remediate_local_drift.rb 52.35 acceptance/tests/direct_puppet/catalog_uuid_correlates_catalogs_with_reports.rb 55.81 acceptance/tests/direct_puppet/static_catalog_env_control.rb 114.42 acceptance/tests/environment/agent_runs_pluginsync_with_proper_environment.rb 55.61 acceptance/tests/environment/custom_type_provider_from_same_environment.rb 47.64 acceptance/tests/environment/enc_nonexistent_directory_environment.rb 58.88 acceptance/tests/environment/environment_scenario-bad.rb 106.58 acceptance/tests/environment/use_agent_environment_when_enc_doesnt_specify.rb 88.39 acceptance/tests/environment/use_agent_environment_when_no_enc.rb 81.42 acceptance/tests/environment/use_enc_environment.rb 76.63 acceptance/tests/environment/use_enc_environment_for_files.rb 46.97 acceptance/tests/environment/use_enc_environment_for_pluginsync.rb 67.49 acceptance/tests/environment/use_environment_from_environmentpath.rb 71.33 acceptance/tests/language/exported_resources.rb 48.25 acceptance/tests/language/pcore_generate_env_isolation.rb 63.53 acceptance/tests/loader/autoload_from_resource_type_decl.rb 64.57 acceptance/tests/lookup/lookup.rb 57.79 acceptance/tests/lookup/lookup_rich_values.rb 56.28 acceptance/tests/lookup/merge_strategies.rb 78.82 acceptance/tests/lookup/v3_config_and_data.rb 68.38 acceptance/tests/ordering/master_agent_application.rb 53.52 acceptance/tests/parser_functions/puppet_lookup_cmd.rb 171.06 acceptance/tests/pluginsync/4420_pluginfacts_should_be_resolvable_on_agent.rb 81.22 acceptance/tests/reports/cached_catalog_status_in_report.rb 66.33 acceptance/tests/reports/corrective_change_outside_puppet.rb 82.52 acceptance/tests/reports/corrective_change_via_puppet.rb 81.63 acceptance/tests/reports/failover_master.rb 118.09 acceptance/tests/reports/submission.rb 79.67 acceptance/tests/security/cve-2013-4761_injection_of_class_names_loading_code.rb 54.48 acceptance/tests/ssl/autosign_command.rb 116.72 acceptance/tests/ssl/certificate_extensions.rb 178.28 acceptance/tests/utf8/utf8-in-catalog.rb 158.18 acceptance/tests/utf8/utf8-in-function-args.rb 54.22 acceptance/tests/i18n/modules/puppet_agent.rb 113.80 acceptance/tests/i18n/modules/puppet_agent_cached_catalog.rb 108.79 acceptance/tests/i18n/modules/puppet_agent_with_multiple_environments.rb 90.68 acceptance/tests/i18n/modules/puppet_apply.rb 48.16 acceptance/tests/i18n/modules/puppet_apply_module_lang.rb 56.36 acceptance/tests/i18n/modules/puppet_apply_unsupported_lang.rb 46.78 acceptance/tests/resource/file/source_attribute.rb 70.15 acceptance/tests/resource/file/symbolic_modes.rb 50.20 acceptance/tests/resource/service/init_on_systemd.rb 58.47 acceptance/tests/resource/service/puppet_service_management.rb 157.51 acceptance/tests/resource/user/utf8_user_comments.rb 48.55   Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[John Duarte (JIRA)]

John Duarte updated an issue

Puppet / PUP-9726 Refactor long running acceptance tests to exectute in less than 45 seconds

Change By: John Duarte *As a* puppet tester *I want* to keep the execution time of individual acceptance tests under 45 seconds *So that* I can get test feedback faster

Acceptance tests are expensive. The following acceptance tests take longer than 45 seconds to execute on the reference RHEL 7 platform. These tests should be refactored to reduce there execution time below 45 seconds (preferably below 30 seconds).

Total execution time for the tests in the sample below was 3506.51 seconds.  If all of these tests were successfully executed in 30 secs each, then the overall test suite execution time would be reduced by about 30% (on RHEL 7 as of 2019.05.31).

Tests that take longer than 45 seconds to run on RHEL7 along with there execution time:

{noformat}

{noformat}

Add Comment

[John Duarte (JIRA)]

John Duarte commented on PUP-9726

Re: Refactor long running acceptance tests to exectute in less than 45 seconds Of the above list, only the following have a risk tag of "audit:high". This ticket should be re-scoped to only include the tests below. acceptance/tests/environment/agent_runs_pluginsync_with_proper_environment.rb acceptance/tests/security/cve-2013-4761_injection_of_class_names_loading_code.rb acceptance/tests/ssl/autosign_command.rb acceptance/tests/ssl/certificate_extensions.rb acceptance/tests/utf8/utf8-in-catalog.rb acceptance/tests/resource/file/source_attribute.rb acceptance/tests/resource/file/symbolic_modes.rb

Add Comment

[John Duarte (JIRA)]

John Duarte assigned an issue to John Duarte

Puppet / PUP-9726

Refactor long running acceptance tests to exectute in less than 45 seconds

Change By: John Duarte Assignee: John Duarte

Add Comment

[John Duarte (JIRA)]

John Duarte commented on PUP-9726

Re: Refactor long running acceptance tests to exectute in less than 45 seconds PR 8004 opened for symbolic_modes test.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9726

Re: Refactor long running acceptance tests to exectute in less than 45 seconds

The environment, ssl, utf8, and source attribute tests are still very valid and important. I don't mind that they don't complete in less than 45 seconds. The "cve-2013-4761_injection_of_class_names_loading_code" could be modified to not start puppetserver twice. Also puppet acceptance is no longer the slowest part of the pipeline (pxp now is). It might be good for us to resync with Mihai Buzgau to see what the slowest parts of the pipeline are.

Add Comment

[John Duarte (Jira)]

John Duarte commented on PUP-9726

Re: Refactor long running acceptance tests to exectute in less than 45 seconds

Of the tests that are tagged as "audit:high" only "acceptance/tests/resource/file/symbolic_modes.rb" is not also tagged as a "server" test. Given this context, I agree Josh Cooper that the "server" tests do not warrant refactoring. I would say that reduces the scope of this effort to the point that it is not worth pursuing. Therefore, I am closing this issue.

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)