Polly (3) -- Downloads as HUB

[Mark S.]

Just uploaded a new branch:

https://github.com/Marxsal/polly/tree/core-enabled

This version is hopefully forward compatible with Powershell Core 6. That is, you

can use it with PS 5, and also with PS 6 -- which means it can be used with

Linux and maybe Mac OS.

Looking for feedback to make sure nothing has broken with it running under standard PS 5.

For use under Linux, there is no documentation yet. But you can just run "pwsh" to

start powershell, and then from there invoke the script with  polly.ps1. After invoking,

a starter settings.ini file will be created, but still using windows defaults. Quit out of the

script and use your favorite editor to modify settings.ini. Then start the script again.

Not sure if environmental variable expansion works, but it should be able to make use of

directory names that start with / or ~ .

I have no way of testing how things work on Mac, so am dependent on feedback. I suspect

that it should work very similarly to Linux.

I would like to incorporate this branch going forward as quickly as possible, so developments

in polly will immediately carry over to the other major platforms.

TT and I will be holding (we think) more of our technical discussions over on GH, so hopefully

this thread will be quieter than the previous two.

Thanks!

-- Mark

[@TiddlyTweeter]

Great news on cross-platform PSCore possibility!

Mark S. wrote:

I've been experimenting with Powershell Core on linux. It took less than 10 minutes to install, and takes about 150megs of space. So
the barrier to usage is pretty low. It will be even lower if PS gets into the major distributions. For those who aren't familiar with
Linux distributions, in Linux most distributions come with a package manager. With a package manager you don't have to hunt down
software at individual web sites. Instead you just open the manager, tell it that you want Open Office, Firefox, and Gnumeric, and in 

a few minutes it's all installed.

 The good news is that Polly's menus looked the same right out of the gate. Somehow PS handled the directory structures well enough

that nothing special had to be done to see the usage file or load settings.ini. Obviously, the part of TT's code that invokes notepad would
need to be tweaked. 

So far the only thing PS has completely rejected is the BufferSize setting. 

@TT -- how important is this setting? The screens looked OK. I'm thinking it would be possible to wrap the code with a check for platform 

so it doesn't throw an error.

The BufferSize is under the # CONSOLE section. That entire console section may not be required on Linux. 

The BufferSize sets the "internal" size of Windows console (not the actual size of displayed window which I set in the batch file launchers). 
I set it it wide and high so (1) long lines don't wrap; (2) you can scroll up what has happened without the buffer running out.
 

There were some conflicts between linux alias names and PS names. They can be resolved by using full PS names.

OK. Changing most aliases to full PS names might be a good idea. 
Makes code longer but perhaps more readable cross-platform

There may or may not be some issues relating to the path separator, but further testing is required.

This is an interesting issue. \ v. /. 
I noticed on Windows a lot of time you can use them interchangeably.
Hopefully PSCore supports that so we don't need to change too much.
 

At the moment it looks like it might be possible to make Polly forward compatible with Powershell Core 6, 

Sounds like it could be done! 

Mostly, in future, I'll comment on tech on GHub.

TT

[@TiddlyTweeter]

Mark S. wrote:

Just uploaded a new branch:

https://github.com/Marxsal/polly/tree/core-enabled

This version is hopefully forward compatible with Powershell Core 6. That is, you

can use it with PS 5, and also with PS 6 -- which means it can be used with

Linux and maybe Mac OS.

Looking for feedback to make sure nothing has broken with it running under standard PS 5.

For use under Linux, there is no documentation yet. But you can just run "pwsh" to

start powershell, and then from there invoke the script with  polly.ps1. After invoking,

a starter settings.ini file will be created, but still using windows defaults. Quit out of the

script and use your favorite editor to modify settings.ini. Then start the script again.

Not sure if environmental variable expansion works, but it should be able to make use of

directory names that start with / or ~ .

I'll comment about this on GHub in a bit. 

I think its worth thinking through how to cater for variant platforms.

For instance, maybe platform dependent # INTERNAL SETTINGS might be moved to secondary scripts or .\lib\win-internal.ini, .\lib\mac-internal.ini & .\lib\unix-internal.ini?

Thoughts

 

TT

[@TiddlyTweeter]

General question on Linux.

What is the equivalent of a Windows "batch" file?

TT

[@TiddlyTweeter]

For those on electronic mail I updated my penultimate post https://groups.google.com/d/msg/tiddlywiki/lMAdx3Xrqow/ehuI5HUBDAAJ

[@TiddlyTweeter]

For instance, maybe platform dependent # INTERNAL SETTINGS might be moved to secondary scripts or .\lib\win-internal.ini, .\lib\mac-internal.ini & .\lib\unix-internal.ini?

My point about this is a wondering if we could enable ONE system that auto-supports 3 o/s.

So, for instance, a user who uses both Linux & Windows ONLY downloads ONE set of PSCore scripts that can auto adapt to either run environment?

Too ambitious?

Simply my thoughts on now.

TT

[BurningTreeC]

General question on Linux.

What is the equivalent of a Windows "batch" file?

Hi, I'd say a bash script

TT

[@TiddlyTweeter]

Mark S. wrote:

especially in the context of USB testing, that you might

want to try some larger files to give a more realistic experience.

@TiddlyTweeter wrote: 

Right. I overlooked that so far!

Now there is testing structure I'll make a "Performance" version as it will be easy.

The gory details of a serious performance check are here: https://github.com/Marxsal/polly/issues/2

Polly performed well on this basic challenge test (8 wiki updated every 10 seconds; restore process running every 3 seconds) from a USB stick.

Hence also the random thoughts about robocopy, which is supposedly more

robust than standard copy, but actually I don't know if it's more robust than

the copy function built into PS.

I don't think RoboCopy is needed.

TT 

[@TiddlyTweeter]

Thanks BTC

Say I write this in Windows BATCH ...

   powershell -file .\polly.ps1 -run "menu" -ini "t_relative-USB.ini"

... the bat file can be clicked to run from file explorer GUI. File called "t_relative-USB.bat"

In BASH would it be as simple as? ...

   powershell -file ./polly.ps1 -run "menu" -ini "t_relative-USB.ini"

What would I call a clickable GUI file in under bash? "t_relative-USB.bash"?

i know NOTHING :(

TT

[BurningTreeC]

Am Montag, 12. August 2019 14:37:36 UTC+2 schrieb @TiddlyTweeter:

Thanks BTC

Say I write this in Windows BATCH ...

   powershell -file .\polly.ps1 -run "menu" -ini "t_relative-USB.ini"

... the bat file can be clicked to run from file explorer GUI. File called "t_relative-USB.bat"

I would call it t_relative-USB.sh

In BASH would it be as simple as? ...

I'm sorry I don't know much about batch-to-linux conversion

[@TiddlyTweeter]

@TiddlyTweeter:

Say I write this in Windows BATCH ...

   powershell -file .\polly.ps1 -run "menu" -ini "t_relative-USB.ini"

... the bat file can be clicked to run from file explorer GUI. File called "t_relative-USB.bat"

BTC 

I would call it t_relative-USB.sh

TX! That is all  I needed to see :-)

TT 

[Mark S.]

This code kind of does that already.

(Update: Just pushed an update to core-enabled to fix a problem in the variables discussed below.)

If you're running PS Core 6, then there are 3 helpful variables:

$isWindows

$isMacOS

$isLinux

To tell you what platform you are on. These variables don't exist in PS 5, so I faked them at the

top, under the assumption that 5 will only run on Windows.

There is also a platform separator character (long complicated name) which I put into $SEPARATOR and

use everywhere in the new version (even though in many cases, but not all, PS can figure out the pathing).

So far there haven't been any separate *.ini settings.

The main thing is that users will need a different settings.ini file for each environment because of the pathing.

But it kind of looks like if users use "/" for their pathing, and not absolute pathing (e.g. "C:/...") then it might

work for both platforms without changes.

Thanks!

-- Mark

[Mark S.]

Quick comment about pathing. "/" Works better as a universal path separator because "\" works as an escape character in

many string interpolations. PS seems to be able to use "/" in paths even on Windows, which is pretty handy.

[@TiddlyTweeter]

I think the "\" on Windows 10, at least, is happily also "/" in many situations. 

There are global substitutions going on to do with MS wanting to share, I think.

If you run "Get-ChildItem Env:" in PS5 you get interesting things like this showing up ...

USERPROFILE                    C:\Users\Josiah                                                                                                                 

USERPROFILE:DoubleBackslash    C:\\Users\\Josiah                                                                                                               

USERPROFILE:Forwardslash       C:/Users/Josiah                                                                                                                 

USERPROFILE:java.util.prefs    /C:///Users///Josiah  

[@TiddlyTweeter]

Mark S. wrote:

(Update: Just pushed an update to core-enabled to fix a problem in the variables discussed below.)

If you're running PS Core 6, then there are 3 helpful variables:

$isWindows

$isMacOS

$isLinux

To tell you what platform you are on. These variables don't exist in PS 5, so I faked them at the

top, under the assumption that 5 will only run on Windows.

There is also a platform separator character (long complicated name) which I put into $SEPARATOR and

use everywhere in the new version (even though in many cases, but not all, PS can figure out the pathing).

So far there haven't been any separate *.ini settings.

The main thing is that users will need a different settings.ini file for each environment because of the pathing.

 Right. Maybe we should try on 5 IF "/" works already on Windows?

TT

[Mark S.]

Hmm. It worked for a download directory with a relative path, but didn't work with a wiki name as a relative path.

*Because* the path is split by the current official separator in order to determine the base file name and the base directory path.

Have to decide if we need to split by both separators (covering all bases) OR ask users to always use forward

slash "/".

Think ... think ... think ...

More research into string splitting needed.

[Mark S.]

On Monday, August 12, 2019 at 8:47:50 AM UTC-7, Mark S. wrote:

More research into string splitting needed.

Just pushed update to core-enabled that splits wiki paths by / and \ .

Getting there

[@TiddlyTweeter]

Ciao Mark

I'll make a test set for that over next couple of days.

TT

[@TiddlyTweeter]

Mark

I see what you doing with that.

I will go though the code and try deal with as much of the Windows hard-coded pathing as possible.

We can certainly dump the manual trailing slashes I added on many directory addresses.

And all libraries I think can be referenced through Unix style pathing without problem. I don't think it would confuse a Win user.

I'll do a version so we can test that.

TT

[@TiddlyTweeter]

Mark S. wrote:

This code kind of does that already.

$isWindows

$isMacOS

$isLinux

Excellent. I can incorporate that into do-settings easily.

These variables don't exist in PS 5.

In do-settings I think it would be easy. An IF for $IsMacOS or $IsLinux else Windows regardless.

 

There is also a platform separator character (long complicated name) which I put into $SEPARATOR and 

use everywhere in the new version (even though in many cases, but not all, PS can figure out the pathing).

So far there haven't been any separate *.ini settings.

I think it can be done all from do-settings. As well as creating bat/sh files? 

 

But it kind of looks like if users use "/" for their pathing, and not absolute pathing (e.g. "C:/...") then it might 

work for both platforms without changes.

That is the one thing I'm grappling with a bit.

I'll work on it over next few days.

TT

[@TiddlyTweeter]

Ciao Mark

Wanted you to have this up-date sooner rather than later.

NB: Its an edit of https://github.com/Marxsal/polly/tree/core-enabled-with-menu-loop, polly.ps1

Only the polly.ps1 file is changed.

polly-01N-TT.ps1

Removed: Unneeded "\"
Changed: Pathing to scripts to Unix style
Changed: Many aliases changed to full cmdlet names
Added: Additional timing setting for "once-menu" refresh cycle
Cosmetic changes

https://drive.google.com/file/d/17LfXDfGDJD5HV-c-f_IHb6EoKqsHHbzT/view?usp=sharing

TT

[Mark S.]

Ut-oh. I just pushed a big "parrot menu" change. Hopefully we can fit the pieces together, because that was a bit of work.

Also, need permission to see your file (submitted request).

Thanks!

[@TiddlyTweeter]

Don't worry. I'll integrate with your latest too tomorrow. You can leave this till then.

TT

[Mark S.]

Ok. Standing by ..

[@TiddlyTweeter]

Mark S. wrote:

Ok. Standing by ..

Ciao Mark

Confirm "Parrot Now" works well. It is a VERY useful advanced feature of Polly.  (For instance my test included using Polly "Parrot Now" as an easy way to maintain a partial mirror of Polly herself that is archived to the cloud along with some wiki via clone to an auto-upload dir).

I need do longer term, more complex, testing to fully confirm.

There are minor issues that I think are just cosmetic. I'll document them on GitHub.

I hope my attempt to bring into one, the 2 generations of you and the 1.5 of me in polly-01Na-TT hasn't damaged anything! 

My additions, apart from in main script, includes tools related to understanding working with settings under the three PowerShell Core o/s

.\dev\do-pathcheck.ps1 tool to check any path to dirs or files, allows relative addressing on check and wildcards;

.\dev\do-grabwiki.ps1 experimental tool to capture directory listings for wikis and bulk convert into .ini data, allows relative addressing on check and wildcards

I doubt we'd put these in a release but may use the code in .\lib\do-settings.ps1 later

They are useful in testing at least.

I'll comment on GitHub on details.

polly-01Na-TT here: https://drive.google.com/file/d/12auDrY6Kg5PFkyZjqAR2JRDEaF4WZB_x/view?usp=sharing (its changed/added items ONLY)

Best wishes

TT 

[Mark S.]

TT's updates have been incorporated. Available as a release from:

https://github.com/Marxsal/polly/releases

New features

• Expands use to PS Core 6 (inc Linux, Mac ?)
• Expands env variables under Windows
• Menu loops for Restore Once and Parrot Now
• Allows relative paths in most situations

It works for me on Linux, but we need feedback from Mac users.

As always, make backups of any files before using.

Thanks!

[Mohammad]

Hi developers,

In the setting.ini is it possible to use wildcards like

[wikis]   

--- WIKIS FOR RESTORE: one per line in format: "label1=path-to-wiki-file" 

file1=C:\TW\Polly\tw-wikidir\*.html

Of course then the label is may be need to be changed to something else!

--Mohammad

[Mohammad]

I have asked this question earlier! After backup and zip why the files still in Downloads dir?

Why not keep Downloads dir tidy?

On Thursday, August 15, 2019 at 9:43:38 PM UTC+4:30, Mark S. wrote:

[@TiddlyTweeter]

Ciao Mohammad

Do you mean that whilst creating setting you could add multiple wiki based on a full path with wildcards. If so, its possible.

IF you mean auto-detection at run-time of wiki within a directory (including recursively through sub-folders) that would not be for the [wikis] section. The wikis section should remain one wiki per line. But I could add tools to automate their addition when first creating an .ini file. 

BUT Mark S. is looking into having a dynamic, run-time, way to add all wikis (recursively) within a folder structure using a wikidirs approach. This issue on GH: https://github.com/Marxsal/polly/issues/16 is a starting on that theme.

TT

[@TiddlyTweeter]

Ciao Mohammad

Mohammad wrote:

I have asked this question earlier! After backup and zip why the files still in Downloads dir?

Why not keep Downloads dir tidy?

I'm not sure. I want to explain some implications. 

1 -- It is important to understand that Polly backups and zip-archives (by design) are always one step behind their current wiki. 

That means for the latest restored the only backup is the Downloads version.

2 -- It is possible (not documented yet) to run a second .ini file that restores to a secondary address. If you auto-deleted Downloads that would prevent that from working.

A possible solution might be a "maintenance" tool that tidies up when explicitly launched.

But, I'm not sure what Mark thinks about all this?

TT

[Mohammad]

Hi Josiah,

On Friday, August 16, 2019 at 2:39:51 PM UTC+4:30, @TiddlyTweeter wrote:

Ciao Mohammad

Do you mean that whilst creating setting you could add multiple wiki based on a full path with wildcards. If so, its possible.

Yes, that,s correct. I mean when creating setting, I want to add all wikis in a folder like C:\TW\Polly\tw-wikidir\*.html

 

IF you mean auto-detection at run-time of wiki within a directory (including recursively through sub-folders) that would not be for the [wikis] section. The wikis section should remain one wiki per line. But I could add tools to automate their addition when first creating an .ini file. 

BUT Mark S. is looking into having a dynamic, run-time, way to add all wikis (recursively) within a folder structure using a wikidirs approach. This issue on GH: https://github.com/Marxsal/polly/issues/16 is a starting on that theme.

This is would be great!

[Mohammad]

On Friday, August 16, 2019 at 2:53:55 PM UTC+4:30, @TiddlyTweeter wrote:

Ciao Mohammad

Mohammad wrote:

I have asked this question earlier! After backup and zip why the files still in Downloads dir?

Why not keep Downloads dir tidy?

I'm not sure. I want to explain some implications. 

1 -- It is important to understand that Polly backups and zip-archives (by design) are always one step behind their current wiki. 

That means for the latest restored the only backup is the Downloads version.

Okay, so we can have only one file per wiki in Downloads or when Polly takes backup it can delete the files from two steps before.

 

2 -- It is possible (not documented yet) to run a second .ini file that restores to a secondary address. If you auto-deleted Downloads that would prevent that from working.

A possible solution might be a "maintenance" tool that tidies up when explicitly launched.

But, I'm not sure what Mark thinks about all this?

Great!

 

TT

[@TiddlyTweeter]

ADVERTISEMENT ...

Before Polly there were problems on automation.

 

 https://www.youtube.com/watch?v=Yx520hQweY0

No longer. Polly solves them all.

 

By Roosting right in.

TT

[@TiddlyTweeter]

ADVERTISEMENT

Unlocking the potential of TiddlyWiki ...

https://www.youtube.com/watch?v=Yy0WiXNFRv4

With Polly.

[@TiddlyTweeter]

PROMO

Polly performing at the ballet  ...

https://youtu.be/WJSj3MxkyZQ?t=95

TT

[Mark S.]

For your consideration, branch:

https://github.com/Marxsal/polly/tree/Wiki-Directories

that enables a section [wikidirs] where you can list entire

directories of wikis to be restored. To avoid conflict, and because

there will be massive numbers of files, you can not specify the

download directory (though you can specify individual files from the

download directory).

As always, be sure to have backups before trying.