HOW TO: Create a file listing, and count total number of lines VERY quickly

[-PC- (formerly kssxs)]

I currently have a task in which I run a terminal command to write the name and path of the files in that folder and its sub-folders into a text file.  That command outputs that text file incredibly fast (a matter of maybe 5 - 10 seconds for 7500 files).  I then populate a global variable with the number of files listed in that text file.

This is how I do it EXTREMELY quickly:

A1: Anchor 

A2: Flash [ Text:Building new file list... Long:Off ] 

A3: Run Shell [ Command:find /mnt/extSdCard/files/. -type f -name *.db -exec rm {} \; Timeout (Seconds):0 Use Root:On Store Output In: Store Errors In: Store Result In: Continue Task After Error:On ] 

A4: Run Shell [ Command:find /mnt/extSdCard/files/ -type f  > /mnt/extSdCard/files/List.txt Timeout (Seconds):0 Use Root:On Store Output In: Store Errors In: Store Result In: Continue Task After Error:On ] 

A5: Run Shell [ Command:find /mnt/extSdCard/files -not -type d | wc -l Timeout (Seconds):0 Use Root:On Store Output In:%NumberOfDFs Store Errors In: Store Result In: Continue Task After Error:On ] 

A6: Flash [ Text:New Default file list built! Files found: %NumberOfDFs Long:Off ] 

A3 - deletes any Windows thumbnail .db files that got copied into the SDCard (when the phone was attached to the PC to transfer pictures files, for example)

A4 - this shell command creates the text file that contains the complete file listing, including path to each file, and it's individual extension (so in my case, I could have .PNG and/or .JPG and/or .BMP files for wallpaper, without worrying about what type of file it is, what it's called, and where it is located (which also allows me to have a heirarchy of files, sorted to the way I like ;) ))

A5 - this shell command gives you the total number of files in the entire folder heirarchy you have setup, and counts only files, and not the number of folders.  This is useful for those of you, like me, who use a 'Variable Randomize' action to pick an individual line number from a group of numbers between 1 and the total number of files to randomize which file, for example, is used to set your wallpaper.

I don't think you need Root, but I have it on my phone, so put a check mark in the option to use it.

Those 2 commands create a file with the listing contained within, AND the total number of files in any specified folder, within SECONDS.  I use these 2 commands to list and count over 7500 files in... get this... 10 seconds.  Tasker, using 'For' loops and whatnot, can take over 25 MINUTES to do the same.  This isn't a knock on Tasker at all (so Pent, don't be upset with me for saying that ;) ), but instead is a compliment to the number of things that Tasker allows you to do.

I hope someone else finds this as useful as I did when I figured out how to do it.  :)

[TomL]

That's pretty cool.  I've been meaning to do something similar to find which subdir has the highest number of files.  (Because sometimes I find that a dir with over several thousand files slows the filesystem down.)

If you're trying to squeeze the very last bit of raw speed out of your task, you can chain all the shell commands in steps 3,4,5 into a single RunShell action.  And that would probably save you half a second. :)

Tom

[-PC- (formerly kssxs)]

Thanks to your suggestion, I now am saving an additional 1 second of time every time this runs ;p .

[-PC- (formerly kssxs)]

Just as a follow-up, I tried your suggestion of chaining the commands together into one statement, and found that you'll need to put in a 'sleep 0.1' command (I used 0.1, but you may need to use a higher or lower value, probably depending on the size of the folder you're processing) in-between the commands in order for them to process properly.  Without the 'sleep' command, I wasn't getting a text file outputted properly, and then that messes up anything that refers to that text file in other actions.

So, I'll change what I said... I'm now saving an additional 0.6 seconds of time when this runs :D .

[TomL]

Can you paste the entire chained set of commands?

Something isn't right. A set of cmds like

aaa ; bbb ; ccc

... should each finish before the next start.

Tom

[-PC- (formerly kssxs)]

Sure, here you go:

A1: Anchor 

A2: Flash [ Text:Building new Default file list... Long:Off ] 

A3: Run Shell [ Command:find /mnt/extSdCard/files/. -type f -name *.db -exec rm {} \; sleep 0.1; find /mnt/extSdCard/files -type f  > /mnt/extSdCard/files/List.txt; sleep 0.1; find /mnt/extSdCard/files -not -type d | wc -l Timeout (Seconds):0 Use Root:On Store Output In:%NumberOfDFs Store Errors In: Store Result In: Continue Task After Error:On ] 

A4: Flash [ Text:New Default file list built! Files found: %NumberOfDFs Long:Off ] 

The only difference between your suggestions and what I have above is the 'sleep 0.1' command.  Without it, the text file I was outputting was 0 bytes in length (obviously indicating it didn't store any data).

The only thing that made logical sense to me is that the next command is being run before the previous one completes, but at the same time, those commands are all doing something unique that is not directly related to the previous command.  Each command is its own separate thing, so I don't know the exact reason why it's doing it, but regardless, adding the 'sleep' command fixed the problem (so I'm not complaining ;) ).

Let me know what you think!

[-PC- (formerly kssxs)]

Sorry Tom, I should have added that the last command in the chain (the file counting command) is completing properly, as the flash alert that pops up is showing the correct number of files.  So I'll stick to my previous thought that the size of the folder determines how much time (even though 0.1 works for me for over 7500 files) the shell needs to write out the file (maybe it's the time it takes to write the file, and the contents are in memory, so if the next command runs, and the file hasn't completed writing out to the filesystem, it corrupts mid-way and writes out 0 bytes in length instead, IE my 'folder processing time' ;) ).

Thoughts?

[TomL]

Ok, found it. You're missing a semicolon after the:
-exec rm {} \;

It should be:

-exec rm {} \; ;

The \; is part of the -exec argument, then you need the next ; to indicate that you have another shell cmd coming up.

Tom

[-PC- (formerly kssxs)]

There we go!  That did it.  I'm, by tech skills, a Windows guy, and even though I know the command line very well, I don't know the Linux/Unix command line, so that would have escaped me.

Thanks again!

[Ticiano Damore]

Hi, sorry to bother, I have a newbie doubt: What´s does the ANCHOR action? 

Thanx for the attention!

[Rich D]

> Hi, sorry to bother, I have a newbie doubt: What´s does the ANCHOR action? 

It is a "Anchor" for the Goto action.. you can specify a label then "goto" it. It can also just be used for a label for organizational use..

[Ticiano Damore]

Nice, thanx!

Ticiano D'Amore
@ticianodamore
www.ticianodamore.com
Enviado do meu xingling

Em 03/09/2013 10:43, "Rich D" <ricp...@gmail.com> escreveu:

> Hi, sorry to bother, I have a newbie doubt: What´s does the ANCHOR action? 

It is a "Anchor" for the Goto action.. you can specify a label then "goto" it. It can also just be used for a label for organizational use..

--
You received this message because you are subscribed to a topic in the Google Groups "Tasker" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tasker/Tb7kx4NWbX0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tasker+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/tasker.
For more options, visit https://groups.google.com/groups/opt_out.