Jira (PUP-10781) If file resource content changes during puppet run, it uses original file checksum although content is retrieved several times

[Patrick Grant (Jira)]

Patrick Grant created an issue

Puppet / PUP-10781 If file resource content changes during puppet run, it uses original file checksum although content is retrieved several times Issue Type: Bug Assignee: Unassigned Created: 2020/11/12 2:56 AM Priority: Normal Reporter: Patrick Grant Puppet Version: 6.17.0 Puppet Server Version: 6.12.1 OS Name/Version: Rhel7 If file resource content changes during a puppet run, although content of the file is retrieved several times, its continues to use the first retrievals checksum Steps from customer testing: Starting with empty file (checksum d41d8) 2020-11-05 12:57:34 +0100 Puppet (debug): HTTP GET https://testmaster:8140/puppet/v3/file_metadata/skv_files/bigfile?links=manage&checksum_type=md5&source_permissions=ignore&environment=che_mini returned 200 OK 2020-11-05 12:57:34 +0100 Puppet (debug): Caching connection for https://testmaster:8140 2020-11-05 12:57:34 +0100 Puppet (debug): Using cached connection for https://testmaster:8140 2020-11-05 12:57:34 +0100 Puppet (debug): HTTP GET https://testmaster:8140/puppet/v3/file_content/skv_files/bigfile?environment=che_mini returned 200 OK note that the file content was fetched here. at exactly this point in time I did the mv 4bf04783777bafe2284f66dcac603a41 bigfile 6d5f49c6fd6ea5291693d649785776de bigfile2 # mv bigfile bigfile1; mv bigfile2 bigfile; md5sum bigfile* 6d5f49c6fd6ea5291693d649785776de bigfile 4bf04783777bafe2284f66dcac603a41 bigfile1 And then the log continues with 2020-11-05 12:58:11 +0100 Puppet (debug): Caching connection for https://testmaster:8140 2020-11-05 12:58:11 +0100 Puppet (debug): Executing: 'diff -u /var/crash/puppet_big_testfile /tmp/puppet-file20201105-12000-12l62g2' 2020-11-05 12:58:11 +0100 /Stage[main]/Che_test/File[/var/crash/puppet_big_testfile]/content (notice): Binary files /var/crash/puppet_big_testfile and /tmp/puppet-file20201105-12000-12l62g2 differ 2020-11-05 12:58:12 +0100 Puppet (debug): Using cached connection for https://testmaster:8140 2020-11-05 12:58:12 +0100 Puppet (debug): HTTP GET https://testmaster:8140/puppet/v3/file_content/skv_files/bigfile?environment=che_mini returned 200 OK The content is fetched again here. 2020-11-05 12:58:49 +0100 Puppet (debug): Caching connection for https://testmaster:8140 2020-11-05 12:58:49 +0100 Puppet (err): File written to disk did not match desired checksum; discarding changes ({md5}6d5f49c6fd6ea5291693d649785776de vs {md5}4bf04783777bafe2284f66dcac603a41) And yes, the checksum 4bf04 doesn't match the checksum the file fetched 12:57:34 had. 2020-11-05 12:58:49 +0100 Puppet (debug): Using cached connection for https://testmaster:8140 2020-11-05 12:58:49 +0100 Puppet (debug): HTTP GET https://testmaster:8140/puppet/v3/file_content/skv_files/bigfile?environment=che_mini returned 200 OK 2020-11-05 12:59:26 +0100 Puppet (debug): Caching connection for https://testmaster:8140 2020-11-05 12:59:26 +0100 Puppet (debug): Executing: 'diff -u /var/crash/puppet_big_testfile /tmp/puppet-file20201105-12000-25edl0' 2020-11-05 12:59:26 +0100 /Stage[main]/Che_test/File[/var/crash/puppet_big_testfile]/content (notice): Binary files /var/crash/puppet_big_testfile and /tmp/puppet-file20201105-12000-25edl0 differ 2020-11-05 12:59:26 +0100 /Stage[main]/Che_test/File[/var/crash/puppet_big_testfile]/content (err): change from '{md5}d41d8cd98f00b204e9800998ecf8427e' to '{md5}4bf04783777bafe2284f66dcac603a41' failed: File written to disk did not match desired checksum; discarding changes ({md5}6d5f49c6fd6ea5291693d649785776de vs {md5}4bf04783777bafe2284f66dcac603a41) (corrective) The log state that it wanted to change content from d41d8 (empty file) to 4bf04 but the new content is actualy 6d5f4 Desired Behavior: if puppet agent fetches the content first 12:57:34, and then again 12:58:12, it should not expect the checksum to be the one from the first content fetch but from the second. Actual Behavior: Agent uses the checksum from first file content fetch Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10781

Re: If file resource content changes during puppet run, it uses original file checksum although content is retrieved several times The downloading twice issue is PUP-3916 and is due to the way puppet determines if a resource is insync and if so, enforces compliance. The file diff is calculated in the first step, because we want to show what would happen when running in noop mode, as step 2 never happens. Puppet has always worked this way, but is only a hard failure due to PUP-10368, as puppet strictly enforces that the file it is about to commit matches the desired checksum, either specified in the manifest or retrieved from the source via file_metadata. Previously puppet was "lenient". Since puppet makes two requests for metadata and content, there is always a race condition where the content can change in between. I'd argue that setting the checksum type to mtime is probably the correct thing here, since you can't definitively say what the md5/sha256 checksum is.

Add Comment

[Patrick Grant (Jira)]

Patrick Grant commented on PUP-10781

Re: If file resource content changes during puppet run, it uses original file checksum although content is retrieved several times

Closing this in favor of PUP-3916, will add the support case there for traction

Add Comment