[example] ListFolderSH - advanced

[alex.symbroson]

Here's an advanced version of my ListFolderSH function I've written in the past.

It creates a list of file objects which includes type, permissions, links [dirs only], owner, group, size [files only], date, time and of course, the name.

In addition to the file types "File", "Folder" and "Link", I have added various extensions of sound, image and video formats that allow you to switch between different icons in the file list.

In the example, I also added a sort function that sorts the list by a specific property.

In addition, there is a complete function that converts the object list into a list that can be read by DS list controls.

Feel free to report or even fix bugs and post them in this thread.

This demo is a basic template for file browsers needed by many applications - suggestions for improvements or even implementations are always welcome.

best regards,

Symbroson

[alex.symbroson]

I've noticed that files with (many) spaces can cause problems. This version should fix this

I've also added a FoldersFirst function which moves all folders to the front of the list.

the file types of files, folders and links are now renamed to "fil", "fld" and "lnk" to have a standard type string length

regards

[Unik]

It does not work...

It says
error at line 70

script error: cannot read property of 'replace' of undefined

and I removed the replace from the line..
now it shows no error but
the most of list items do not contain filename as title but contain only size, date etc as body...
only some files in the list have filename as title.

and when I touch the list item which is folder,
it do not browse the touched folder neither it has the option to go back to parent folder.

[alex.symbroson]

try this one - it replaces all whitespace characters with spaces - this may cause problems if a filename includes a whitespace character other than space

if "." or ".." doesn't exist for some reason it adds it automatically - (with ".." you can switch one directory up)

The script error shouldn't appear too

[alex.symbroson]

I have implemented an other (probably better) way to split the file data from the shell output string. Should be more secure

Please anyone test this one because I don't know how the android shell behaves on different devices - thanks guys

[BareK]

I teste the latest spk you provided, but it doesn't seems to work properly.

I attached the screens I have (at first only '..', then mostly single letters if I touch '..').

Here is an extract of the output of "ls -la" on my device (numbers are file size):

drwxrwx--- root     everybody          2018-04-18 11:11 .AppySite
drwxrwx--- root     everybody          2018-07-17 14:24 .DroidScript
drwxrwx--- root     everybody          2018-04-09 02:44 .EveryplayCache
drwxrwx--- root     everybody          2018-06-11 11:24 .OrMicPark
-rw-rw---- root     everybody        5 2018-07-17 12:41 .appodeal
drwxrwx--- root     everybody          2018-04-09 02:44 .chartboost
drwxrwx--- root     everybody          2018-05-15 16:41 .estrongs
drwxrwx--- root     everybody          2018-03-29 21:51 .smartlauncher
-rw-rw---- root     everybody       72 2018-07-17 14:24 .userReturn
drwxrwx--- root     everybody          2018-03-09 10:02 846b44643ec609f507828878741e1f9a
drwxrwx--- root     everybody          2018-03-18 10:40 Algoid
drwxrwx--- root     everybody          2018-01-31 00:36 Android
drwxrwx--- root     everybody          2018-04-17 22:36 AppySite

My device is OnePlus One CM 13.1 Android 6.0.1 DS 1.61

[alex.symbroson]

thanks

this must be because on mobiles the folder size won't be shown...

will work on that

[alex.symbroson]

Check out this one:

I'm now using a regular expression to get all file data at once!

I also improved the "." / ".." add if missing function

and for images the preview will now be loaded instead of the image image

If anybody has still problems please send me a snippet of your "ls -la ~" output as Unik did. Then I can adjust the regex.

best regards

[alex.symbroson]

I think this should be the last update for now:

- crop file name of links when opening (remove " -> link location")

- check isfolder for links when opening

- if shell list is empty after 3 tries use app.ListFolder instead

- if list is still empty pretend the folder is empty

  (I don't know a way to detect 'not existant' or 'no permission' - would need a plugin for that)

- added specific handling for empty path (can be app folder)

Please test it and give some feedback wether it works for your devices.

[BareK]

I tested it and I find it unstable:

"Something went wrong" is fired arbitrary (even when visiting the same folder multiple times).

App.ListFolder() is called sometimes, but not others.

Clicking on the dots '..' sometimes goes one level up, sometimes reloads the actual folder.

Clicking on a folder sometimes loads it, sometimes reloads the actual folder (or even brings me back to the "sdcard" folder).

Images (as "thumbnails") doesn't always appears on the list.

Did you try to navigate a lot with it?

Also: I don't think that the single dot '.' link is necessary, as it should bring you back into the same folder (or maybe you wanted some kind of "refresh" behavior).

Great work anyway :)

[alex.symbroson]

Yes I did - my whole filesystem up and down ^^

Could you please show me your ls -al /sdcard/ output

in function ListFolderSH
put app.WriteFile(lst, app.GetAppPath()+"/log.txt")
before
if(lst.length === 0) {
// use app.ListFolder instead
lst = app.ListFolder(path)

[alex.symbroson]

Or try to output the path variable after each dir switch

[BareK]

For the log I already gave them to you here:

https://groups.google.com/d/msg/androidscript/xOTWRP8ZUm0/blu5luWlBwAJ

I will try to output the path variable as you suggested :)

[alex.symbroson]

Oh right - but I definitely tried it and it worked perfectly. But it was just the one list without subdirs or anything. So maybe there's something else.

Please try to figure out the reason - I can't reproduce it in any way

btw be sure there's always a '/' at the end of the path - maybe I have to add another test which checks for it

[BareK]

I'll give you the actions I did and the result I obtained.

The '[DEBUG] (...)' lines are produced by this code:

app.Debug( "<b>[DEBUG] path before: " + path );
app.Debug( "<b>[DEBUG] path after: " + path );

Placed both at the begining and the end of List_OnTouch.

[DEBUG] path before: /sdcard/
[DEBUG] path after: /sdcard/
-> displayed: /sdcard/
-> click on: 'Music'
[DEBUG] path before: /sdcard/
[DEBUG] path after: /sdcard/Music/
-> displayed: /sdcard/
-> click on: '.'
[DEBUG] path before: /sdcard/Music/
[DEBUG] path after: /sdcard/Music/
-> displayed: /sdcard/Music/
-> click on: '..'
[DEBUG] path before: /sdcard/Music/
[DEBUG] path after: /sdcard/
-> displayed: /sdcard/
-> click on: 'DCIM'
[DEBUG] path before: /sdcard/
[DEBUG] path after: /sdcard/DCIM/
-> displayed: /sdcard/
-> click on: 'DCIM'
[DEBUG] path before: /sdcard/DCIM/
[DEBUG] path after: /sdcard/DCIM/DCIM/
-> displayed: /sdcard/DCIM/
-> click on: 'Camera'
[DEBUG] path before: /sdcard/DCIM/DCIM/
[DEBUG] path after: /sdcard/DCIM/DCIM/Camera/
-> displayed: empty folder
-> click on: '..'
[DEBUG] path before: /sdcard/DCIM/DCIM/Camera/
[DEBUG] path after: /sdcard/DCIM/DCIM/
-> displayed: empty folder
-> click on: '..'
[DEBUG] path before: /sdcard/DCIM/DCIM/
[DEBUG] path after: /sdcard/DCIM/
-> displayed: /sdcard/DCIM/
-> click on: 'Camera'
[DEBUG] path before: /sdcard/DCIM/
[DEBUG] path after: /sdcard/DCIM/Camera/
-> displayed: /sdcard/DCIM/Camera
-> note: no permissions displayed (might be app.ListFolder)
-> click on: '.'
[DEBUG] path before: /sdcard/DCIM/Camera/
[DEBUG] path after: /sdcard/DCIM/Camera/
-> displayed: /sdcard/DCIM/Camera/
-> note: permissions displayed (might be sysproc)
-> click on: '..'
[DEBUG] path before: /sdcard/DCIM/Camera/
[DEBUG] path after: /sdcard/DCIM/
-> displayed: /sdcard/DCIM/Camera/
-> note: images icons/thumbnails are gone for images files
-> clickon: '.'
[DEBUG] path before: /sdcard/DCIM/
[DEBUG] path after: /sdcard/DCIM/
-> displayed: /sdcard/DCIM/
-> click on: '..'
[DEBUG] path before: /sdcard/DCIM/
[DEBUG] path after: /sdcard/
-> displayed: /sdcard/DCIM/
-> click on: 'Camera'
[DEBUG] path before: /sdcard/
[DEBUG] path after: /sdcard/Camera/
-> displayed: /sdcard/
-> note: /sdcard/Camera/ does not exists, it's in DCIM

Tried many times, the bug always occurs, but not always at the same moment.

It's a bit random.

Hope it helps.

[alex.symbroson]

This is strange.
Can you give me the code you have so that I see where and what you're debugging
thanks

[alex.symbroson]

Maybe there's smth else different in your code what you've accidently changed. I will give a file comparer a try.
or it is smth related to escape sequences - is there a \0 escape somewhere? Try to replace it with an other null width character or just use "" instead if possible

[BareK]

I didn't modify anything :)

I'll post it here, so people can have a quick look at it without having to download the spk.

So here's your code:

/*
    Advances ListFolderSH example by Symbroson
    WTFPL license  -  no warranties
   
    if you find any bugs:
        feel free to fix them yourself or report them at least
        post it in the related thread in the DroidScript forum
    https://groups.google.com/d/topic/androidscript/xOTWRP8ZUm0/discussion
*/

var sys, files = [], path = "/sdcard/";

function error( msg, line, file ) {
    app.Alert( file + ": error at line " + line + "\n\n" + msg, "notitle" );
}

function OnStart() {
    app.SetScreenMode("full");
    app.SetOnError(error);
    sys = app.CreateSysProc("sh");
   
    var lay = app.CreateLayout("linear", "VCenter,FillXY");
   
    lstFiles = app.CreateList('', 1, 1);
    lstFiles.SetOnTouch(List_OnTouch);
    lay.AddChild(lstFiles);
   
    List_OnTouch('.');
    if(files.length === 0) {
        alert('Couldn\'t load "' + path + '". Stop.');
        app.Exit();
    }
   
    app.AddLayout(lay);
}

function List_OnTouch(item, body, img, i) {

app.Debug( "<b>[DEBUG] path before: " + path );

    var tpath = path;
    if(item === "..") path = path.slice(0, path.slice(0, -1).lastIndexOf('/') + 1);
    else if(item !== '.') {
        if(files[i].type === "lnk") item = item.slice(0, item.lastIndexOf("->") - 1);
        if(files[i].type === "dir" || (files[i].type === "lnk" &&
            app.IsFolder(tpath + item + '/'))) path += item + (item === '/'? '' : '/');
        else return app.OpenFile(path + item);
    }
   
    files = ListFolderSH(path, true, true);
    if(files.length === 0) {
        app.ShowPopup("couldn't switch directory");
        files = ListFolderSH(path = tpath, true);
    }
    files = SortFolderList(files, "name", "asc");
    files = FoldersFirst(files);
    lstFiles.SetList(MakeFolderList(files).join(";;;"), ";;;");

app.Debug( "<b>[DEBUG] path after: " + path );
}

function OnFResume() {
    List_OnTouch('.');
}

function MakeFolderList(list) {
   
    return list.map(function(file) {
        var date = file.date.toString();
       
        return file.name + ':' + file.permissions + ' ' +
            (file.type == "dir" && file.links > 0? file.links + ' ' : '') +
            file.owner + ' ' + file.group +
            (file.type != "dir"? ' ' + Math.round(file.size / 1024) + "kB " : ' ') +
            date.slice(0, date.lastIndexOf(':')).replace(/:/g, "^c^") +
            ':' + ({
                fil: "Img/file.png",  dir: "Img/folder.png",
                lnk: "Img/link.png",  img: path + file.name,
                snd: "Img/music.png", vid: "Img/video.png"
            }[file.type] || ' ');
    });
}

function SortFolderList(list, sortby, order) {
    var order = {asc: 1, inc: 1, dec: -1, desc: -1}[order];

    return list.sort( function(file1, file2) {
        var res;
        switch(sortby) {
            case "name":
                res = file1.name.toLowerCase() > file2.name.toLowerCase();
            break;
           
            default:
                res = file1[sortby] < file2[sortby];
        }
        return res ? order : -order;
    });
}

function FoldersFirst(list) {
    var lstDir = [], lstFil = [];
    list.forEach(function(v, i, l) {
        if(v.type == "dir") lstDir.push(v);
        else lstFil.push(v);
    });
    return lstDir.concat(lstFil);
}

function File(type, permissions, links, owner, group, size, date, name) {
    return {
        name: name, type: type,
        permissions: permissions,
        links: links, owner: owner,
        group: group, size: size,
        date: date
    };
}

var formats = {
        img: ".png.jpg.jpeg.bmp.dds.tiff.svg.cgm",
        snd: ".mp3.wav.wma.pcm.aac.ac3.ogg.tta.wma.msv",
        vid: ".mp4.vob.flv.mkv.ogv.gif.mng.avi.wmv.amv.mpg.mpeg.svi"
    };

function extractType(type, name) {
    var ext = name.lastIndexOf('.'), ttype = type;
    if(ext > 0) ext = name.slice(name.lastIndexOf('.'));
    else ext = null;
   
    // set file type
    if(type == '-' && ext !== null)
        for(var i in formats)
            if(formats[i].indexOf(ext) >= 0)
                return i;
   
    if(type === '-') return "fil";
    else if(type === 'd') return "dir";
    else if(type === 'l') return "lnk";
    return type;
}

function ListFolderSH(path, showHidden) {
    var s, lst = [], i = 0, files = [];
   
    if(path.length > 0) {
        //try shell 3 times
        for(var i = 0; lst.length === 0 && i < 2; i++) {
            // execute ls command
            sys.Out('ls -la "' + path + '"\n');
       
            //read command output
            while((lst = sys.In(10, "nowait")) == '' && ++i < 10);
            while(s = sys.In(100, "nowait")) lst += s;
        }
    }
   
    if(lst.length === 0) {
        // uae app.ListFolder instead
        lst = app.ListFolder(path);
        if(lst.length > 0) {
            lst = (".,.." + (path.length === 0? ",/" : '')).split(',').concat(lst);
           
            //create and return file object list
            return lst.map(function(file) {
                var isFld = app.IsFolder(path + file + '/') ? 1 : 0;
                return File(
                    extractType("-d"[isFld], file), "", "", "", "",
                    app.GetFileSize(path + file + (isFld? '/' : '')),
                    app.GetFileDate(path + file + (isFld? '/' : '')), file
                );
            });
        }
    } else {
        // create list of file objects
        lst.replace(
            /(.)(([r-][w-][x-]){3})\.?([\d\s]*?)\s*(\w*)\s*(\w*)\s*(\d*)\s*(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2})\s(.*)/g,
            function(match, type, permissions, _, links, owner, group, size, year, month, day, hours, mins, name) {
               
                type = extractType(type, name);
               
                //assign date object
                var date = new Date();
                date.setFullYear(year);
                date.setMonth(month - 1);
                date.setDate(day);
                date.setHours(hours);
                date.setMinutes(mins);
                date.setSeconds(0);
                date.setMilliseconds(0);
               
                files.push(File(
                    type, permissions, 1 * links, owner,
                    group, 1 * size, date, name
                ));
            }, ''
        );
       
        if(!showHidden) {
            files = files.filter(function(v, i, l) {
                return v.name[0] == '.' || v.name == '.' || v.name == "..";
            });
        }
    }
   
    if(files.length === 0 || files.filter(function(file) {return file.name === "..";}).length === 0)
        files.unshift(File("dir", "", "", "", "", app.GetFileSize('.'), app.GetFileDate('.'), ".."));
   
    if(files.length === 0 || files.filter(function(file) {return file.name === '.';}).length === 0)
        files.unshift(File("dir", "", "", "", "", app.GetFileSize('.'), app.GetFileDate('.'), '.'));
   
    return files;
};

[alex.symbroson]

I couldn't find a way to prevent this behaviour. I've just increased the max amount of sys.In and some other things and it seems to help a little but but it doesn't really solve the problem.

[BareK]

I'll investigate that when I'll have some time and post back my result here.

Take a break, this is already a nice sample ;)

[alex.symbroson]

Thanks BareK. I appreciate your help

[BareK]

Hi Alex.
I tested your code on 3 different devices and I didn't manage to reproduce the bug!
Did you?
But interesting thing: I notice that on one of my devices (a lenovo tablet on Android 5.1) some directories don't appear if I list from /sdcard/.
I noticed that with the 'Droidscript' directory.
If I set path to /sdcard/Droidscript/ in the code I access it without problem.
But then touching '..' brings me back to /sdcard/ with 'Droidscript' directory missing.
I'll investigate more tomorrow, but it's globally very good :)

[alex.symbroson]

Maybe this is related to the search pattern again - what does the shell output look like?

I couldn't reproduce it too but that doesn't mean that it is gone for sure - I think my adjustments just decreased the probability...

[BareK]

The shell output is the same as the one I posted already (except that the group is 'sdcard_r' instead of 'everyone').
So it might be something else.

[alex.symbroson]

can you try this one:

function ListFolderSH(path, showHidden) {


    console.log("ListFolderSH(\"" + path + "\")");
    var s, slst = '', i = 0, list;
   
    if(path.length > 0) {
        //try shell 2 times
        for(var i = 0; slst.length === 0 && i < 2; i++) {
            // execute ls command
            sys.Out('ls -la "' + path + '";echo \"#*end%!\"\n');
           
            //read command output
            while((slst = sys.In(10, "nowait")) == '' && ++i < 50);
            while(!slst.endsWith("#*end%!\n")) slst += sys.In(100, "nowait");
        }
    }
   
    slst = slst.slice(0, -9);
   
    if(slst.length === 0) { /* ... */

[BareK]

It gives the same result.

[BareK]

Mail:

Hi Alex :)

I worked on your code and I think I came to something.

I tested it on 4 different devices and it worked very well.

I've attached the files to the post (spk, and individual files to allow to read them online).

I've focused on readability / understandability by adding spacing and spliting long lines.

This way it might be easier to understand and adapt (as an example code).

Now the core of it is the regex (which can be improved as I'm not an expert). 

As it is a bit long and complicated, I put a 'regex.txt' file with informations about it.

I tried to couple with most of the cases but I'm affraid that more can occur.

Seen those examples (based on some googling):

Total 20M

drwx------+  1  Spencer  61k   oct  2  09:54  file.ext

drwx--x--x  1  r8wx  d768   61   oct  2   2014  dir/

drwx------  1  Spencer  root   61k   oct  2  09:54  link*

drwx------  1  Spencer  root   61   oct.  2  09:54  link*

drwx------  1  Spencer  root   61   2  sep.  09:54  file

drwx------  1  Spencer  root   61   2  sept.  2018  dir

drwx------  1  漢字   汉字   61   7汉   12   0:54.10  link*

There are also a lot of warnings about parsing LS output:

https://unix.stackexchange.com/questions/128985/why-not-parse-ls

https://mywiki.wooledge.org/ParsingLs

I think a better approach could be to use the shell but not using ls:

https://blog.sleeplessbeastie.eu/2014/12/30/how-to-check-file-metadata-using-shell-script/

Here is the summary of what I did:

Added spacing

Splitted long code lines

Added some comments / doc

Added debug overlay that shows ls command results

   -> toggle it by typing 'Debug( true | false )' in the web ide debug view

Improved logs with colors & a prefix '[D]' to filter them easily

Improved regexp to catch the more

Added optionnal timezone parameter

Dropped the new Date() object due to variations in date format

Added physical back button navigation

Removed the '.' element and the '..' informations

Ensured that '..' is always on top of sorted list 

   -> had issues with special characters such as '(' at the beginning

Added behavior related to links (open and using back key)

   -> deserves more test, but I can't make links on my device

Improved sort function using device's locale

I also droped the app.ListFolder part as it seems out of scope (not a ListFolder using sh).

[alex.symbroson]

Thanks a lot BareK - good work. I didn't knew that there are that many different time formats in ls ls…
I will have a closer look at all that if I'm back in our holiday apartment (I'm in Iceland for the next two weeks)

best regards,
Alex

[BareK]

You're welcome ;)
And you did most of it ^^
Have a nice holiday!

[alex.symbroson]

Well, after reading these articles I'm not sure if writing a ShellUtils plugin was a good idea - was almost done with it :(

However I think we can get at least close to a stable ls parser even if it's very complicated. And such things like linebreaks or tab characters in filenames are rare enough to just ignore them because most android filemanagers don't allow that at all. (That doesn't mean we can't try to handle them)

Also getting each file metadata by a command may be difficult because the tools may be not available (ie I don't have the 'stat' command by default)

And at last I hope that android has at least some standardization in it's shell. I won't try to deal with '?' replaced characters - that would be too much..

I will spend some more time and thoughts in the ListFolderSH function since I know about the problems it has

best regards

[BareK]

Something that could help a lot would be a "busybox" plugin (or another compilation of shell tools).
But I saw it was discussed whith Chirs in the Icode thread and seems to be very complicated.
Anyway, I believe it would be a great addition to DS :)

[alex.symbroson]

An other way would be to just write a c program which can format all data as we want it and print it to stdout. This way we can avoid to parse ls and deal with many different formats.
But maybe it's just too much for a little ShellUtils plugin which is supposed to allow other DS devs to execute some shell commands. I don't know. I could even write everything in c and the JS part would just call it. But thats not the purpose of a shell plugin

[alex.symbroson]

Found a bug for block and charakter specific files regarding size after adding b and c to the type regex ([dlcbf\-]).
Sample output of /dev/input/:

_start_
crw-rw---- root     input     13,  64 2015-01-01 00:00 event0
crw-rw---- root     input     13,  65 2015-01-01 00:00 event1
crw-rw---- root     input     13,  66 2015-01-01 00:00 event2
crw-rw---- root     input     13,  67 2015-01-01 00:00 event3
crw-rw---- root     input     13,  68 2015-01-01 00:00 event4
crw-rw---- root     input     13,  69 2015-01-01 00:00 event5
crw-rw---- root     input     13,  70 2015-01-01 00:00 event6
_ls_ok_
_end_

The comma seperated values are the minor and major number of these files

So we'd have to adjust the size regex to
(\d+\,\s*\d+\s|\d+\s?\w*\s)?\s*

I also had a problem for items with name length 1 (ie. I had a 'c' dir)
For that remove the \S from the name regex. This won't have any effect because the previous \s* of the timezone regex takes as many whitespace characters as possible.

I didn't found any fifo files so I can't test them. Added the type anyways.
But all the rest looks great so far - even if I'm confused about some changes you made in other functions ^^ Don't mind.

At last I found a way to support multiline files - even if they shouldn't appear in normal fs's. Have a look on it