End of year survey for how much storage you typically use for your Windows program files & program data

[Arlen Holder]

End of year survey for how much storage you typically use
for your Windows installed program files & program data

Happy New Year!

Here's my quick survey of space used (none of which needs backing up)
o <https://i.postimg.cc/1z1YYcD5/app01.jpg> ~28 GB installed apps
o <https://i.postimg.cc/kMHspvm3/app02.jpg> ~1 GB Program Data
o <https://i.postimg.cc/L5S1c5NP/app03.jpg> ~2 GB Program Files
o <https://i.postimg.cc/tCHKcbyH/app04.jpg> ~18 GB Windows files
o <https://i.postimg.cc/rFVkmTHN/app05.jpg> ~500 GB personal data

Note that I don't willingly store _anything_ in default Microsoft folders
o Which is something I learned in the Win95/WinXP days works rather well

Hence, _anything_ in any default Windows folder can be trashed
o When a new system is set up from scratch

Given I store everything that matters into C:\data, there's nothing
anywhere else on my Windows systems which needs to be backed up at the
annual EOY backup archive.

To be clear, I use a philosophy which, by design, enables this efficiency

For example:
1. All program installers are backed up at the moment they are downloaded
2. Such that they are easily re-installed on the new machine when needed
3. And where a USB-based syslog file is kept of each apps' setting changes

Given the menus and the syslogs and the installers are all maintained at
the time the program is first installed, there's nothing to back up other
than your own personal data (which I keep in, not surprisingly, C:\data).
o <https://i.postimg.cc/rFVkmTHN/app05.jpg> ~500 GB personal data

Specifically:
a. Syslogs are maintained on an external USB stick (for use with many PCs)
b. Data is the only folder that matters (it's all you need to archive)
c. Installers are saved at the time of download to external storage

All hierarchies on all machines match for complete consistency:
o <https://i.postimg.cc/yY74z87s/taskbarmenu06.jpg>

In summary, once a year (or so), you copy your "c:\data" & you're done
o Nothing else matters on a typical Windows system if you planned ahead

See also:
o Tutorial for setting up Windows in a well organized KISS philosophy
<https://groups.google.com/forum/#!topic/alt.comp.freeware/i9Cz3POZFCo>
--
Posted out of the goodness of my heart to help others & to learn from them.

[Arlen Holder]

On 30 Dec 2020 16:20:26 GMT, Frank Slootweg wrote:

> If the Restore Points are included in the SVI, then nearly all is
> Restore Points, because the 'Current Usage:' of the Restore Points is
> 12.35 GB (according to the System Protection tab of the System
> Properties control panel applet).

Hi Frank,

By way of providing another helpful datapoint, my RPs are about 16GB:
o Win+R > systempropertiesprotection > [Configure]

Unlike the trolls on Usenet, I don't bullshit (I prove what I claim):
o <https://i.postimg.cc/FKkC2YjZ/app06.jpg> ~16 GB restore point

In my case, that's only five restore points, which is odd, as I don't
remember deleting any, where I've noticed that whatever is labeled as:
o Windows Module Installer
is, perhaps, deleting all prior restore points (it may be one of my
cleaner programs, e.g., Revo, AdvancedUninstall, iOBit, ZSoft, etc.).
o Win+R > systempropertiesprotection > [System Restore] > [Next]
o [x]Show more restore points

Whatever "Windows Modules Installer" is I don't know though...
o <https://i.postimg.cc/hhFVQGJS/app07.jpg> Something deleted my RPs!

In summary, I'm glad you brought up this system restore point space
o As I just noticed "something" is auto-deleting my many restore points!
--
Posted to learn from others and to disseminate knowledge to all who care.

[Auric__]

Arlen Holder wrote:

> End of year survey for how much storage you typically use
> for your Windows installed program files & program data
>
> Happy New Year!

Windows 10 1909, clean install 2 weeks ago. Installer somehow nuked my
storage drive; 20 years of downloads -- 4 TB! -- gone. (I have backups, but
the last one was over a year ago, and they're on a Mac-formatted drive,
because reasons.)

=Apps=
Program Files: 1.16 GB
Program Files (x86): 2.53 GB
ProgramData: 436 MB
Windows: 20.8 GB
Wintools: 5.23 GB
Win64tools: 2.9 GB
%USERPROFILE%\AppData: 1.84 GB (and increasing as I watch...)
TOTAL: 35 GB
=Personal=
Desktop: 21.4 GB
Books: 123 GB
Games: 28.2 GB
Movies: 8.84 GB (formerly more than 1 TB, dammit)
Music: 362 GB
Storage: 126 GB (and growing)
TOTAL: 671.6 GB

> a. Syslogs are maintained on an external USB stick (for use with many PCs)
> b. Data is the only folder that matters (it's all you need to archive)

That's similar to how I setup my *nix systems, with /home (equivalent to
\users) on one partition and everything else on another.

--
You're going to spend several years of your life in some cold dark cubicle
with a crazy boss who couldn't program a twenty out of an ATM.

[GlowingBlueMist]

On 12/31/2020 1:43 PM, Arlen Holder wrote:
o <https://i.postimg.cc/hhFVQGJS/app07.jpg> Something deleted my RPs!
>
> In summary, I'm glad you brought up this system restore point space
> o As I just noticed "something" is auto-deleting my many restore points!
>

Check the actual configuration of your restore points (how much disk
space you reserve for them). As new restore points are created the
oldest gets deleted if the allotted space is used up.

When a major Microsoft update occurs Microsoft may delete all of your
old restore points -IF- Microsoft determines they can not be used after
the update.

At least on the last few major updates Microsoft started leaving the
creation of restore points option turned on, if you had it on. In the
past, after a major update Microsoft quite often would silently delete
all the old restore points and not tell you they also turned the option
off entirely.

[Arlen Holder]

On Thu, 31 Dec 2020 17:35:46 -0600, GlowingBlueMist wrote:

> Check the actual configuration of your restore points (how much disk
> space you reserve for them). As new restore points are created the
> oldest gets deleted if the allotted space is used up.

Hi GlowingBlueMist,

Happy New Year!

Only a handful of people not only voluntarily offer purposefully helpful
advice, but even fewer offer _good_ advice (where I'm of the former, but
perhaps not even yet in the latter).

For 2021, I'm gonna keep a list of those who, like you, are well worth
reading since you offer purposefully helpful _good_ advice!
o Win+R > usenet
Where in the Registry is the definition of what it does (opens a text file)
o HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths
o Usenet.exe === c:\path\to\the\text\file\you\wish\to\edit.txt

Moving on to your suggestion of checking my Restore Point Configuration...
o Win+R > systempropertiesprotection > [Configure]
The "Max Usage" is set to the max possible, but that was a good guess
which _everyone_ should take note of (who _wants_ restore points).
o <https://i.postimg.cc/FKkC2YjZ/app06.jpg> ~16 GB restore points

I know Mayayana doesn't "like" restore points, but I find them fantastic
as I install a _lot_ of freeware (I test for the best); where sometimes,
they just maliciously screw up the system like you can't believe.

> When a major Microsoft update occurs Microsoft may delete all of your
> old restore points -IF- Microsoft determines they can not be used after
> the update.

Hi GlowingBlueMist,

Happy New Year!
o What's great about your answer is that it will be archived for posterity!

This is interesting, as I don't remember the date I updated to
o Win+R > winver ===Reports===> 2H02 (OS Build 19042.685)

Hmmm... how can I tell _when_ I updated to 2H02?
o Win+I > Update & Security > View update history

Bingo!
A. Notice all restore points prior to 12/14/2020 mysteriously gone:
<https://i.postimg.cc/hhFVQGJS/app07.jpg> All prior RPs are gone!
Notice a cumulative Windows update was on 12/14/2020 at 10:14:12PM
B. Notice a cumulative Windows update was on 12/14/2020 (dunno what time)
<https://i.postimg.cc/bJ5BqRFD/app08.jpg> Windows update history
C. Also notice a 12/ı27/ı2020 "Feature update to Windows 10, version 20H2"
Which didn't seem to be the culprit that destroyed my prior RPs
But notice _another_ mysterious "Windows Modules Installer" on 12/27!

Both "Windows Modules Installers" coincide with the Windows' updates!

I looked at what's in this cumulative update but it says nothing relevant:
o <https://support.microsoft.com/en-us/help/4592438/windows-10-update-kb4592438>

In tentative agreement, it "could be" that the cumulative update on
12/14/2020 at 10:14:22PM is the mysterious "Windows Modules Installer"
which seems to have wiped out all my prior restore points sans asking.

Do others _also_ have a "Windows Modules Installer" timed to the same
date as their recent Windows updates? (That would help to confirm.)

> At least on the last few major updates Microsoft started leaving the
> creation of restore points option turned on, if you had it on. In the
> past, after a major update Microsoft quite often would silently delete
> all the old restore points and not tell you they also turned the option
> off entirely.

This is _also_ a great purposefully helpful point to remember!

Come to think of it, I think more than once I have looked at my restore
point log only to find the entire set wiped out.

I'll start looking for this mysterious "Windows Modules Installer" to see
if it's always the last fingerprint left when the Restore Points are on.

Note: I'm trying to figure out a way to tell at what time of day I ran
the update, but it _could_ have been at 10pm so that information jives.
--
Happy New Year to the handful of purposefully helpful people on this ng!

[Arlen Holder]

On Thu, 31 Dec 2020 22:55:59 -0000 (UTC), Auric__ wrote:

> Windows 10 1909, clean install 2 weeks ago.

Hi Auric,
Happy New Year!

Thanks for being purposefully helpful, which is a rarity on this ng,
and therefore very much appreciated is your Windows install survey data.
o People like you and GlowingBlueMist are how we get smarter every day!

Like you, I back up my claims with real world data, which is an improvement
over having just knowledge vs having knowledge _plus_ empirical experience!

o <https://i.postimg.cc/1z1YYcD5/app01.jpg> ~28 GB installed apps
o <https://i.postimg.cc/kMHspvm3/app02.jpg> ~1 GB Program Data
o <https://i.postimg.cc/L5S1c5NP/app03.jpg> ~2 GB Program Files
o <https://i.postimg.cc/tCHKcbyH/app04.jpg> ~18 GB Windows files
o <https://i.postimg.cc/rFVkmTHN/app05.jpg> ~500 GB personal data

o <https://i.postimg.cc/FKkC2YjZ/app06.jpg> ~16 GB restore points

> Installer somehow nuked my
> storage drive; 20 years of downloads -- 4 TB! -- gone. (I have backups, but
> the last one was over a year ago, and they're on a Mac-formatted drive,
> because reasons.)

Yikes. I'm sorry about that. I'm sure you tried the various recovery
options, where it has happened, at some point in time, to all of us.

Interestingly, a Windows update seems to have nuked all my prior restore
points, which, luckily for me, isn't anywhere near as bad as what it did to
you!

o <https://i.postimg.cc/hhFVQGJS/app07.jpg> Something deleted my RPs!

o <https://i.postimg.cc/bJ5BqRFD/app08.jpg> Windows update history

Note the loss of restore points coincides with "Windows Modules Installer"
o And note the "Windows Modules Installer" coincides with Windows' updates!

> Program Files: 1.16 GB
> Program Files (x86): 2.53 GB

Your program files are similar in scope to mine, particularly given you
likely haven't had a chance to install a lot of programs in just two weeks
(and where you likely spent a lot of that elapsed time on HDD recovery).
o I had about 2GB of Program Files, while you have about 3 1/2 GB

Note though that I've been running my system for six months
o Hence my "installed programs" are about 28GB but I install a lot of apps!

> ProgramData: 436 MB

That's also about the same, as I have 1GB to your 1/2 GB Program Data.

> Windows: 20.8 GB

That's also roughly about the same as I have 18GB to your 21GB Windows.

> Wintools: 5.23 GB
> Win64tools: 2.9 GB
> %USERPROFILE%\AppData: 1.84 GB (and increasing as I watch...)
> TOTAL: 35 GB
> =Personal=
> Desktop: 21.4 GB
> Books: 123 GB
> Games: 28.2 GB
> Movies: 8.84 GB (formerly more than 1 TB, dammit)
> Music: 362 GB
> Storage: 126 GB (and growing)
> TOTAL: 671.6 GB

Your total of about 700 GB of "personal stuff" is similar to mine at 500GB.

>> a. Syslogs are maintained on an external USB stick (for use with many PCs)
>> b. Data is the only folder that matters (it's all you need to archive)
>
> That's similar to how I setup my *nix systems, with /home (equivalent to
> \users) on one partition and everything else on another.

Like you, I come from the old school, which, in the olden days, was prior
to Linux (my immersion to computers was in the late 60s) so I weaned myself
from the IBM systems, to DEC VAX VMS PDP11, to SunOS & Solaris, etc., prior
to personal computers taking over (mostly Redhat & Ubuntu Linux's).

Hence, like you, I'm extremely familiar to organizing a system by putting
stuff "where it belongs", where we get to define "where it belongs".

I think all well organized systems keep the programs separate from the data
o Where Mayayana has an apt "tractor trailer" characterization

Mayayana describes the "tractor" as the programs (e.g., C:\apps)
o And the "trailer" as the data (e.g., c:\data)

You can have lots of trailers, whereas you often use the same tractor
o You can have many users, where you often use the same programs

In summary, thanks for your addition end-of-year survey of software needs
o Where yours is very similar to mine in terms of what uses how much space
--
Happy New Year to all those who are purposefully helpful on this newsgroup!

[Arlen Holder]

On Fri, 1 Jan 2021 02:56:02 -0000 (UTC), Arlen Holder wrote:

> Note the loss of restore points coincides with "Windows Modules Installer"
> o And note the "Windows Modules Installer" coincides with Windows' updates!

Happy New Year!

To help us get smarter every day, here's some info on
o "Windows Modules Installer"
Which seems to be what is auto-deleting all my prior System Restore Points!

o Windows Modules Installer listed System Restore Point fails
<https://www.thewindowsclub.com/windows-modules-installer-system-restore-point-fails>
"The Windows Modules Installer Worker or WMIW or TiWorker.exe checks
for new updates from the Windows server and installs them on your
computer system.

So this process may create a system restore point before installing
Windows Updates."

o What do "windows modules installer" restore point mean?
<https://www.reddit.com/r/Windows10/comments/94lfhc/what_do_windows_modules_installer_restore_point/>
"Windows Modules Installer is the thing that installs Windows Updates,
so it's just Windows Update, basically."

o How to Disable Windows Modules Installer Worker
<https://www.techcrises.com/tips-and-tricks-2/disable-windows-modules-installer-worker-windows10/>
"Windows Modules Installer Worker (TiWorker.exe) is a system process
that is responsible for installing both Windows modules
and operating system updates."

According to this site:
o Windows Modules Installer Shown Instead Of Restore Points
<https://forums.majorgeeks.com/threads/windows-modules-installer-shown-instead-of-restore-points.319160/>
"Microsoft is phasing out Restore Points in favor of doing a system refresh"
--
Happy New Year! Let's strive to get smarter every day!

[Paul]

There is TestDisk, if a partition got deleted (partition entry
removed from partition table).

Using an UnDelete utility, can sometimes recover recently
deleted files. On NTFS, only one byte per deleted file is
flipped, to mark they have a deleted status. Then the
$MFT entry storage space gets reused at the first opportunity,
and the clusters the file used are allocated when new file
writes call for them. There is an incentive then, to stop
using the disk when trouble is first detected. If you
continue to use the partition, the quality of the recovery
becomes poorer and poorer with time.

https://en.wikipedia.org/wiki/Recuva

A good UnDelete utility, tells you how many clusters have
been overwritten since the "event". A file with a "Good" status
or a green dot, none of the clusters are damaged, and simply
flipping the byte in the $MFT entry, brings back that file.
Files which have a yellow dot or red dot could be suffering
severe damage, and some other file is using the same clusters
and has overwritten what the previous owner of the clusters
put there.

As for TestDisk, it requires that the user remember what the
partition setup looked like. Many times (in my experience),
the reconstructed partition table is not the same as the
original partition table. Using a "bad" reconstruction
would result in damage to the data. If you have a backup, it has
a copy of the protective MBR, a copy of the GPT tables (there is
a primary and a backup table), and those are some of the items
that a person might put back as part of data recovery.

It really depends on what happened, as to how close to recovery
the thing might be. For example, an innocent CHKDSK run on a
partition with problems, can erase data too. Windows 10 has
a background process that verifies structures on disk, but
with the lack of documentation, we don't really know if there
are any corner cases where that makes things worse.

If I had to guess, some feature like "Storage Spaces" or
"Storage Manager", some sort of rubbish like that, it
spots when a partition has only 5% space remaining. It
then prompts you with an "I can help you clean off this shit"
prompt. When the user doesn't think it's shit :-/

Now, the CleanMgr.exe utility (which might be what the Storage
feature uses), it has a *very dangerous* box, or what was
a box there at one time. It offered to *delete your Downloads*.
That's the Downloads folder in your home directory.

Any mechanism of that sort, is a terror. I'm always very careful
in that tool, to clear all boxes, before I start selecting the
boxes I'm thinking of using. A dead giveaway would be a box that
says "potential savings 4TB", as then you'd know "hey, wait a minute,
that's my Downloads!". If CleanMgr got them, then Recuva might work
for some of it (like if you stopped using the "empty" partition
immediately).

There are also file scavengers like Photorec. But as far as I know,
those would work best if the file clusters were sequential. Such as
if a very good defragmenter had put all the files in precise order
before the "event". It's a shame that it's so much hard work to
prepare volumes for "events", and as a result, partitions are
hardly ever ideal candidates for the Photorec approach.

Paul

[Arlen Holder]

On Fri, 01 Jan 2021 00:23:12 -0500, Paul wrote:

> There is TestDisk, if a partition got deleted (partition entry
> removed from partition table).

Hi Paul,
Happy New Year!

You're one of the handful of purposefully helpful people on this ng
o Where people like you are what make this Usenet newsgroup worthwhile

Looking up the links for TestDisk to put in my software archive
TestDisk
o TestDisk & PhotoRec 7.2
<https://www.techspot.com/downloads/4171-testdisk.html>

Apparently this is the canonical URL (AFAICT):
o <https://www.cgsecurity.org/>

o TestDisk & PhotoRec 7.2
<https://www.techspot.com/downloads/4171-testdisk.html>

It's not easy to find the canonical location for testdisk or photorec
o Maybe this?

<https://www.cgsecurity.org/>
o TestDisk: Partition recovery, file undelete

PhotoRec: File recovery
o <https://www.cgsecurity.org/wiki/PhotoRec>
o <https://www.cgsecurity.org/wiki/PhotoRec_Step_By_Step>

TestDisk & PhotoRec:
o <https://www.cgsecurity.org/wiki/TestDisk>
o <https://www.cgsecurity.org/testdisk.pdf>
o <https://www.cgsecurity.org/wiki/TestDisk_Download>
o <https://www.cgsecurity.org/Download_and_donate.php/testdisk-7.2-WIP.win64.zip>
Name: testdisk-7.2-WIP.win64.zip
Size: 26407735 bytes (25 MiB)
CRC32: D82F2413
CRC64: E2267B4C9455CD69
SHA256: CA9539FD9BA18A3903D48989C35553546D768296AABDC336C354F2F8019C1D73
SHA1: 3C5CF15920B82D833662EF2D32243E25E2D79A39
BLAKE2sp: D9EB98406057BFE40BE6C67916DF0CC99E2FB8E683E8FC5F5039192EA3D686DE

> Using an UnDelete utility, can sometimes recover recently
> deleted files.

Likewise, finding the canonical location is problematic
o Is this the canonical location?

Recuva UnDelete
o <https://www.ccleaner.com/recuva>
o <https://www.ccleaner.com/recuva/download>
o <https://download.ccleaner.com/rcsetup153.exe?cc2020>
Name: rcsetup153.exe
Size: 7638224 bytes (7459 KiB)
CRC32: CAD6818D
CRC64: 049F3BA80A38D26A
SHA256: 189A868B9A175A3C303AED14688F33848A19E439D7B333DC5AE24853DAEE0870
SHA1: 37FC209D42EC49595BEDC73CD0A698E4200EDDE6
BLAKE2sp: 12DBB548F0F1EE3733295D78146D0BA04726D4781045F429CA7D36A5F46A0F50

> Now, the CleanMgr.exe utility (which might be what the Storage
> feature uses), it has a *very dangerous* box, or what was
> a box there at one time. It offered to *delete your Downloads*.
> That's the Downloads folder in your home directory.

CleanMgr.exe?
o Is this that?
<https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/cleanmgr>
--
Happy New Year! Let's all try to add value in every thread and in every post!

[Paul]

Most likely links:

TestDisk - partition table repair (attempt) https://www.cgsecurity.org/
Photorec - scavenger https://www.cgsecurity.org/
Recuva - UnDelete https://www.ccleaner.com/recuva

CleanMgr - Win10, backport to Win7 (Part of OS)
- Excels at deleting Windows.old
- But could delete your Downloads

Piriform was bought out, Recuva likely still supported by its author,
not really sure.

Paul

[Arlen Holder]

On Fri, 01 Jan 2021 07:17:30 -0500, Paul wrote:

> Most likely links:
> TestDisk - partition table repair (attempt) https://www.cgsecurity.org/
> Photorec - scavenger https://www.cgsecurity.org/
> Recuva - UnDelete https://www.ccleaner.com/recuva

> Piriform was bought out, Recuva likely still supported by its author,
> not really sure.

Hi Paul,

Thanks again for being one of the handful of people on this newsgroup who
are purposefully helpful, where these are important tools to have on hand
in our archives, and yet they all have innocuous names; hence it behooves
us all to archive the canonical files for emergency use on our flash drive.

By way of confirmation, and resolve to try to add value in every post,
I agree with you those are the most likely canonical links.

My observations are, mostly unintuitively so:
1. It seems "TestDisk" & "PhotoRec" are in the same "TestDisk" download.
2. There are many undeletes, Recuva seems best (but damn Piriform).

Side note: I used to diligently use CCleaner but, like Chrome, it's no
longer allowed on any system that I have anything to say about the matter.

> CleanMgr - Win10, backport to Win7 (Part of OS)
> - Excels at deleting Windows.old
> - But could delete your Downloads

With respect to %windir%\system32\cleanmgr.exe here is my quick survey:

o Win+R > cleanmgr > [C:] > [OK] > [Clean up system files] > [C:] > [OK]
[x]Download Program Files
[x]Temporay Internet Files
[x]Windows error reports and feedback di...
[x]DirectX Shader Cache
[x]Delivery Optimization Files
[x]Recycle Bin
[x]Temporary Files
[x]Thumbnails

o Win+R > cleanmgr > [C:] > [OK] > [Clean up system files] > [C:] > [OK]
[x]Windows Update Cleanup
[x]Microsoft Defender Andivirus
[x]Download Program Files
[x]Temporay Internet Files
[x]Windows error reports and feedback di...
[x]DirectX Shader Cache
[x]Delivery Optimization Files
[x]Device driver packages
[x]Language Resource Files
[_]Previous Windows installation(s)
[x]Recycle Bin
[x]Temporary Files
[x]Thumbnails

Total amount you can gain = 124MB
--
Every thread & every post to this Usenet ng should strive to add value.

[Arlen Holder]

UPDATE:
o <https://i.postimg.cc/bN875p8b/apk01.jpg> 1600 APKs extracted
o <https://i.postimg.cc/cLXyGjTH/apk02.jpg> 16GB of archive storage

Automatically saving all your program installers is part of a well designed
plan to allow easy (almost trivial) re-installation after a system crash.

To that end, I'm saving my EOY archive of Android & Windows-based apps
installed over the past year, where, on my one Android year-old $100
64GB/4GB Moto G7 phone alone, I amassed 1,600 APKs over the year
(this count includes pre-installed apps & all versions that I installed and
updated, plus any programs which were tested & then deleted over the year):
<https://i.postimg.cc/bN875p8b/apk01.jpg>

The disk space for those 1,600 extracted Android APKs (including
pre-installed system apps) is 16GB which is stored along with Windows
installers (given Android & Windows both have similar app archive needs).
<https://i.postimg.cc/cLXyGjTH/apk02.jpg>

Windows:
a. You generally download the app off the net nowadays (most apps anyway)
b. Where you save that download to your archive at that very same time
c. Then you install a copy of that download (and delete that installer)

Android:
A. You set Google Play apps to automatically save the APK upon download
B. You set F-Droid apps to automatically save the APK upon download
C. The only thing left is to "extract" the existing pre-installed apps
(which includes system apps, if so desired)

Note: Android has a huge advantage over Windows in that it's trivial to
"extract" a pre-installed app, even a system app; and it's trivial to
set Android to automagically save all the APKs you install (and every
version of every app APK you install, if you set the right switches)
at the very time you install that app.

Hence, on Android, the archive is completely automatic (if you know how).
o See the Android ng for my tutorials on how to accomplish app autoarchive

In summary, at the end of each year, I suggest it's a good idea to archive
on an external media, all your Windows & Android program installers, both
of which are saved, automatically, at the moment you downloaded and
installed the programs (plus, if desired, all pre-installed programs).

Automatically saving all your program installers is part of a well designed
plan to allow easy (almost trivial) re-installation after a system crash.
--
Posted out of the goodness of my heart to disseminate useful information.

[Auric__]

Paul wrote:

> There is TestDisk, if a partition got deleted (partition entry
> removed from partition table).

No, it's more than that. I've had to use testdisk in the past, and I wish it
was that simple.

My 4tb storage drive got overwritten with a (blank) ~100GB NTFS partition, at
the very beginning of the disk. Not sure what happened or why, but I am not
exactly thrilled, as you can imagine.

> Using an UnDelete utility, can sometimes recover recently
> deleted files. On NTFS, only one byte per deleted file is
> flipped, to mark they have a deleted status. Then the
> $MFT entry storage space gets reused at the first opportunity,
> and the clusters the file used are allocated when new file
> writes call for them. There is an incentive then, to stop
> using the disk when trouble is first detected. If you
> continue to use the partition, the quality of the recovery
> becomes poorer and poorer with time.
>
> https://en.wikipedia.org/wiki/Recuva

I'll look into that one. I've been using the free version of R-Studio:

https://www.r-studio.com/free-linux-recovery/

I wish I knew more about how ext2/3/4 works. I have a very basic user-level
understanding of it, but I'm not sure how to use the info presented to me to
recover my files (and believe me, if I can recover instead of restore from
backup, I'll be a happy man).

Man, I miss the simple days of UNDELETE.EXE.

--
Your mileage may vary. Results not guaranteed.
Warranty void in the United States, England, and the planet Earth.

[David Brooks]

Auric

I've not followed this thread but it might help you to know about a
facility called Disk Drill. I have used it with GREAT success!

https://www.cleverfiles.com

It works on both Windows AND Macs.

Good luck! :-)

https://en.wikipedia.org/wiki/Disk_Drill_Basic

(Follow-up set to ACW)

[Paul]

Then R-Studio might be your best bet.

If Microsoft sniffed GPT, it might wipe the two GPT tables )primary
and secondary), the first track and the MBR, then do a "quick" format of the
partition they had in mind. While the partition size is 100GB, that
doesn't mean there is a 100GB hole. The hole might only be a hundred
megabytes or two.

Using nfi.exe from Microsoft in the year 2000-2003 or so, will
give a "map" of sectors in use on the partition right now. and the
quantity described there, is an approximation to the level of damage.
The pi chart Properties for the partition, gives the same size info,
as an easier to understand single number.

But naturally, scavenging is no fun. I've tried it before in the
past, and the destination hard drive ended up with 100,000 fragments
of unknown parentage, and a quick examination shows there aren't
enough hours in the day to "make" files out of what you're given.

It's possible EXT is more decentralized than NTFS and there might be
portions of the tree that R-Studio can walk.

I guess you'll know soon enough, as long as R-Studio doesn't
give you the usual "teaser" response, where they pretend to
show you the names of the files they "could" recover, but
don't. And then three payments of $39.95, you can have
your files back. With no way of knowing whether they really
know how to recover the inodes belonging to the file.

Paul

[Paul]

Auric__ wrote:

>
> My 4tb storage drive got overwritten with a (blank) ~100GB NTFS partition, at
> the very beginning of the disk. Not sure what happened or why, but I am not
> exactly thrilled, as you can imagine.

One of the rules of installing, is to disconnect all drives
that do not need to be touched by the install. For example, I
have three drives on this computer, and during an OS install,
two get disconnected as they have nothing to do with it.

That way, only one drive needs to be backed up just before
the install attempt.

I''ve lost a couple drives (target drives) while setting
up multi-boots. I would recommend being especially careful
with Debian itself.

Paul

[Paul]

Auric__ wrote:

>
> I wish I knew more about how ext2/3/4 works. I have a very basic user-level
> understanding of it, but I'm not sure how to use the info presented to me to
> recover my files (and believe me, if I can recover instead of restore from
> backup, I'll be a happy man).
>
> Man, I miss the simple days of UNDELETE.EXE.

https://www.sciencedirect.com/science/article/pii/S1742287617300270

"Contribution

It can be concluded from the findings of this work that, by using
the described approach, files can be reconstructed from Ext4 file
systems even without knowledge about the particular structure of
the file system. By separating the inode search from the inode
reconstruction, it is possible to find inodes when there is no
given information about the file system layout available. Neither
the file system size, nor the examined Ext4 file system's offset
to the begin of the partition are necessary in order to locate
inodes (and thus gain first information about potential files).
"

So what that says, is EXT4 has some special properties. Especially
if it wasn't a conversion from EXT3 to EXT4. Apparently EXT4 likes
to use Extents, and then if the recovery software searches for
inode signatures, that makes it easier to recover complete files.

The files may be sans path information, but for a download folder,
getting the file itself back is a start.

The recovery then, doesn't have to be as bad as Photorec just
randomly grabbing blocks and slapping them together. There
may be enough "local structure" to what is on the disk, to
allow recovery of most everything still there. All it takes is
an Extent Aware recovery tool. (Apparently the following
code fits into the SleuthKit framework, which is command line.)

https://www.cs1.tf.fau.de/research/archive/ext4-file-recovery/

# source code

http://faui1s205.informatik.uni-erlangen.de/Thesis-Seufert.rar

Paul

[Auric__]

Paul wrote:

> https://www.sciencedirect.com/science/article/pii/S1742287617300270
[snip]

> https://www.cs1.tf.fau.de/research/archive/ext4-file-recovery/
>
> # source code
>
> http://faui1s205.informatik.uni-erlangen.de/Thesis-Seufert.rar

Wow, thank you very much. I'd never heard of SleuthKit, much less this. This
looks very, very promising. R-Studio seems to be able to at least detect the
inodes (or whatever), but you get a pretty heavy information overload. I'll
have to spend some time reading and dinking around.

Elsewhere, Paul also wrote:

> One of the rules of installing, is to disconnect all drives
> that do not need to be touched by the install. For example, I
> have three drives on this computer, and during an OS install,
> two get disconnected as they have nothing to do with it.

I know, I know, after 35+ years you'd think I'd be doing it religiously, but
no, I have to be an idiot...

--
There is no rest for me in my search for peace.

[Paul]

I finally got a result on my attempts to recover EXT4 files.

To simulate your disaster, I formatted a 500GB drive as EXT4, copied
over 5000 files (from my Downloads folder), then changed the first
partition to NTFS and "formatted". This causes about 100MB of writes
to the partition, guaranteed to cause at least some file loss for
where the NTFS $MFT metadata file is stored.

I got the Sleuthkit file source, and added the inode carving module.
The package build did not go well at all. It turns out the "framework"
module, would be something a person writing a program would use, rather
than being immediately useful to an end user. so this turned out to be a
dead end.

I then tried working on "tsk_recover", part of the SleuthKit you can
install from Synaptic package manager.

So now, imagine the situation. I have an EXT4 partition but it's disguised
as NTFS.

1) First step, was to use FDISK or GDISK, to change the partition
number from 0x0700 to 0x8300. So the partition table type would
change from NTFS to EXT. This really didn't do much of anything.
Sleuthkit tsk_recover said its internal use of some e2fs utility
was unable to find an EXT partition. Something like that.

2) Next step, was to make an identical disk, and mkfs.ext4 to the partition,
so I would have partition header (the first few sectors) plus all the
Superblock locations. Turns out, I didn't need any Superblock info.

I took a look with a Hex editor (from Windows, the HxD one), and
I copied 4096 bytes from the newly minted empty partition, over
to the damaged partition. So now I've replaced the first few sectors
(8 sectors of 512 bytes), with what should be a very similar file system
header to what used to be there.

At this point tsk_recover indicated it scanned and found 177 files
of around 5000 files total. Still not doing good enough.

3) Next, with backup in hand, I tried a few fsck runs. I figured,
maybe some genius fixed fsck to make use of the new extent features
and all that sweet redundant information, and I could "make" a partition
out of the thing. I used Gnome-disks and used the star menu to
select "scan disk" and "repair disk". These whined and it didn't
look like they did anything.

Next, I did a simple "sudo fsck /dev/sda1" kind of command from Terminal,
and the command seemed to complete.

Now when I ran tsk_recover, the utility claimed it had copied most everything
that used to be on there. With fsck, you never know whether the wiring
will be better or worse than it was before (that's why you make a backup
with "dd" beforehand - I just cloned the original disk to a scratch disk to
run this test case).

I haven't checked sizes, nor verified anything yet. Because the process
finished relatively quickly (should have been maybe 40GB of files), I suspect
I'm in for a rude surprise when I compare the "before and after" file sets.

But at least it claims to have done something.

When I tried the RStudio trial, it promised to return more files
than I'd put there in the first place.

Picture:

https://i.postimg.cc/qMF2CMKQ/its-fun-to-pretend.gif

Paul

[Paul]

Paul wrote:

>
> Picture:
>
> https://i.postimg.cc/qMF2CMKQ/its-fun-to-pretend.gif
>
> Paul

Ran hashdeep on the recovered data from the tsk_recover
and the files recovered seem good. The dates are wrong,
and have todays date. But the size ahd MD5s are correct.
There are the expected missing files, which correspond to
the approx 100MB of metadata NTFS writes out during formatting.
So whatever files were hit by that, aren't recovered.

The size of the recovered data is 44GB, and perhaps the
original test dir was 46GB or so. Perhaps one of my
downloaded ISOs did not survive.

By combining your (older) backup, with the recovered data,
you'd have good coverage for the damaged files in the
recovery (written long ago and backed up), as well as
the files up near the end of the partition which are
recoverable.

Paul