Polly (2) -- Downloads as HUB

[@TiddlyTweeter]

This is a continuation of https://groups.google.com/forum/#!topic/tiddlywiki/tMLfNs1K3JU%5B1-25%5D that got too long.

Its an open development of a tool (Windows) to save TW5 & TWC without the need for any plugin. Via ANY browser. Its a near universal method to restore wiki.

It works by engaging the default downloads system to leverage TW. 

Its still highly experimental. Not yet for casual use.

But so far we managed to restore from downloads to any wiki. To have backup that are zipped to save spce. To restore not just back to the original authoring wiki, but copy to mutiple outputs. 

In effect its a wiki mangement system.

Slowly its shaping. 

Best wishes

TT

[@TiddlyTweeter]

Repeat for email ...

[@TiddlyTweeter]

@TiddlyTweeter wrote:

This is a continuation of https://groups.google.com/forum/#!topic/tiddlywiki/tMLfNs1K3JU%5B1-25%5D that got too long.

Ciao Mark

I made a wiki (Telmiger ToDoNow + Mal's version control made it easy) . 

The most useful part is the top section at https://tidbits.wiki/polly/polly.html#pollyCODE-01, "Compare Versions", has tiddlers with versioning. Makes it a bit easier to see each other's changes. You can see changes opening the "Info > Versions" panel.

Best wishes

Josiah

 

[Mohammad]

Hi Josiah,

I have not followed this discussion! Have you tried the TiddlywikiWatcher

https://github.com/MircoBabin/TiddlyWikiWatcher

It does a similar job!

--Mohammad

[@TiddlyTweeter]

Tx Mohammad. I'll take a look.

Josiah

Mohammad wrote:

... Have you tried the TiddlywikiWatcher

https://github.com/MircoBabin/TiddlyWikiWatcher

[Mark S.]

https://drive.google.com/file/d/1lIORd9f_qpVqt-WZAEsdedvLLRFIIFGy/view?usp=sharing

polly-01h-MS.zip

* Adds suppression to flashed zip archive

* Adds date-stamped backup to specified backup directory (different from zip archive)

* Adds new entry in INI for backups

* Adds an archive directory in polly directory path for older versions of the ps1 files for reference.

* Backup was tested once. So, beware

I'm not exactly sure how to use the PollyDev wiki. Some explanation may be in order.

Thanks!

[@TiddlyTweeter]

Ciao Mark

You doing so much I can't keep up :-) f & g were good and clear. So now I'll concentrate on h before responding.

Mark: I'm not exactly sure how to use the PollyDev wiki. 

I doubt you need to use it. Only the first section ("Compare Versions") is evolving code shared between us. I track the code when we share it in by using a versioning system in the tiddlers flagged "-versioned". 

Under Info > Versions you can click through the versions to see differences. I made it mainly for myself to see what you changed, but thought you might find it useful. This one thing I try keep up to date. For instance I just added your latest "h" to it. Its in "polly-01.ps1 -versioned" as the most recent.

Best wishes

Josiah

[Mark S.]

polly-01i-MS.zip

https://drive.google.com/file/d/1P03hvjOEmzbBQPKpEdAhiNU_SCueuYX7/view?usp=sharing

* Added "parrots"

* Added more to polly.txt, but still very incomplete

Barely tested, so be careful out there!

[Mark S.]

Oops. I see I left in a lot of debugging messages. They'll have to come out in the next go-around.

[@TiddlyTweeter]

Ciao Mark

Excellent! And there I was about to send my demo version!

I've made a lot of cosmetic changes because of the new features. But I'll integrate i-MS too now before sending. With "parrots" you have basically covered most scenarios. I think parrots will prove particularly useful.

I think documentation is not that easy for it. Maybe we should eventually include some diagrams? 

I started to play with idea of making a "setup" tool that on first run guides the user through the creation of an .ini. On install Polly would lack settings.ini and auto-launch the setup script. Early idea.

Best wishes

Josiah

[Mark S.]

Hi TT,

You're going to write an entire setup wizard in PS ?

For documentation, if you're going to have diagrams, you could launch a TiddlyWiki file with the default browser. I suppose it could be used for setup too.

[@TiddlyTweeter]

 

Mark: You're going to write an entire setup wizard in PS ?

I was thinking about it. A reason to do so is that precise pathings are very important in it. A setup script up would allow easy path checking. Also on first run without a suited .ini it may be confusing. 

For documentation, if you're going to have diagrams, you could launch a TiddlyWiki file with the default browser. I suppose it could be used for setup too.

I was thinking maybe a release could include a docs TW that will be able to get restored using polly automatically? A thought. 

The various wizards in TW I think would make the procedural aspect of setup easy. The problem is there is no interaction with the OS (you could do it in Bob, but that's overkill to install that for this). PS does make things like path checking very easy, so maybe the setup part is easier in the end in PS? Not sure yet.

TT

[@TiddlyTweeter]

For those on email: I updated my last post (final para). TT

[@TiddlyTweeter]

Mark S.

A presto.

One of the things that interests me about Polly is being able to (eventually) provide film students of mine with a pre-configured system in which all they need to do is confirm the Download address. Having to ask them to switch browser model in order to use a saver they have to additionally download and install in software they don't know is all too much for a student principally interested in film, not computers.

The default browser on latest Windows is Edge. Firefox holds a minority report. Chrome & Chromium hold most ground. Both chrom* and FF are extra  steps. Edge does the job, mainly. 

I'd rather the user didn't have to choose simply to enable saving. If the defualt Edge works, why not?

Best wishes

TT

[@TiddlyTweeter]

Email users: eek! Thinking too much. Added last two paras to last post. 

[Mark S.]

Original Edge is a bit of a non-starter for me with TIddlyWiki. It won't read/write HTML files on a networked drive! Or at least on my drive.

I found lots of references to this problem, but no solutions. It has something to do with over-the-top security. None of the other browsers,

including IE had this problem. The new (Chromium) Edge doesn't have this problem (yet).

If your educational environment uses a networked drive, you might verify in advance that your default version of Edge will work with it.

Thanks!

[@TiddlyTweeter]

TBH Mark my students are on vanilla systems. One person on one computer. The lonely film-maker. I'm not sure they would ever see the issues you hit?

Let's see. I'd love to get Polly to the point I can test it with them. Its coming.

Josiah

[@TiddlyTweeter]

I want to COMPLAIN

My parrot is dead. Its a late parrot. Its joined the choir invisible.

Where can i get my money back?

https://www.youtube.com/watch?v=vnciwwsvNcc

[Mark S.]

That's not the one we sold you. It's not the real Macaw.

[@TiddlyTweeter]

Ciao Mark S.

In this email I say what I did in main scripts; in the next I'll give details of the "test/demo tool" and download link.

I've integrated all your changes in polly-01i-MS into polly-01i-TT

   and ...

 

                            polly-01i-TT.ps1 -----------------------------------------

       Added: Console Buffer setting so very long paths don't wrap and

              ruin the layout

  Simplified: Menu options hiding. Now if settings don't have options 

              set for them the display of them is always hidden 

       Added: Menu item "B - Backups" to list content of Backup & Zip Archive folders

       Added: Menu item "P - Parrots", just a **placeholder** for now

       Added: Two new URLS to web links

       Added: DESCRIPTION to the settings (optional): useful if you have more than one .ini file
       Added: "#region Name ... #endregion Name" psuedo-comment pairs (in PowerShell IDE they 
              let you collapse/expand sections). Purely for my own use. In run they are ignored.

     Revised: Directory displays to be more relevantly laid out and compact

     Removed: "Run Info" from the "U - Usage" screen to simplify it

    Replaced: "E - Edit Settings" with "S - My Settings" that displays a 

              settings sub-screen 

       Moved: "Run Info" display into "S - My Settings" sub-screen

     Revised: Many cosmetic aspects of the interface, especially feedback when 

              The Restorer is running

     Removed: Most of my "dead wood" comments

              Get-IniContent.ps1 --------------------------------------

     Revised: (a) "-" in addition to ";" as a comment start

              (b) discard whitespace both before and after [section]

              (c) discard trailing whitespace on "key=value"

              polly.txt -----------------------------------------------

     Revised: Trying to combine your useful text with mine (**half-done**)

              But, I'm not sure how much text is workable paged in PowerShell. 

              It maybe, other than a summary note, that using TW would be much better??

   
              Knowing how much info is needed is quite difficult.

    Note for Mark S.: In your central Restorer "while" loop I've added  

                      slightly different messaging for the end user. I hope you think

                      its okay. Its left-offset at the moment so you can quickly see

                      what it is. Let me know what you think.

Next post, subito

TT

[@TiddlyTweeter]

Ciao Mark S., and interested avians

I realised, calling this bit a "demo" is not right. Its a serious "test system". 

Though the "touch" mechanism it uses could be used to create a user friendly demo I think.

The thing is under .\dev\test\

You start it clicking polly-test.bat, it then (1) opens a file explorer in the test replication of Polly in its application folder so you can launch Polly;  (2) runs polly-touch.ps1 which auto creates "downloads" in the simulation download directory. The tool explains what its doing and how to use it. I hope its clear!

You may find it "over-noisey" in that the test settings.ini enables many "wiki" to be rapidly Backed up, Archived, Parotted & Restored. It creates a lot of fast console traffic. 

To reduce noise comment out some of the lines in settings.ini.

Important: ALL pathing in the test settings.ini are relative. You should not need (fingers-crossed) to change anything to get it to run.

Note: the "test" version has two temporary hacks in it to enable the display of directories at relative addresses. 

Otherwise its identical to polly-01i-TT.ps1.

I wanted it to resemble what a user would likely see on first use.

----

I forgot to mention in the last post that I ...

 
      Added: DESCRIPTION to the settings (optional): useful if you have more than one .ini file
      Added: "#region Name ... #endregion Name" psuedo-comment pairs (in PowerShell IDE they
             let you collapse/expand sections). Purely for my own use. In run they are ignored.

---

In this version will be, this time, only the full test and the files I changed for the main app. Ask if anything is unclear.

You can get it here: https://drive.google.com/file/d/1c-AcYxFuKYsmhUtQlxjOUwm2CC5XlD4M/view?usp=sharing

Let me know if it works for you! :-)

Best wishes

TT

[@TiddlyTweeter]

Ciao Mark S. & interested avians

ISSUES

19/07/20 -- There are issues with .\lib\Get-IniContent related to whitespace handling. In the "test" version of Polly I discovered that IF a "[parrots]" setting line has trailing spaces it will fail to parrot. I'll change the module's regular expressions for next version so this is, hopefully, no longer an issue. The draft code changes are at: https://tidbits.wiki/polly/polly.html#Get-IniContent.ps1%20-versioned under "Info > Versions > item 2"

TT

[@TiddlyTweeter]

Ciao Mohammad & interested avians

 Mohammad asked:

.... Have you tried the TiddlywikiWatcher

https://github.com/MircoBabin/TiddlyWikiWatcher

It does a similar job!

Thanks for the pointer!

 

Its interesting. The interface is very simple and works well ... for example ...

 How it works ...

• One wiki at a time
• Only Download address avaialble is the Windows default
• On restore it creates one backup copy that overwrites the previous backup

The actual change monitor is very effective. And definitely of interest.

But, note, Polly is much broader and already works to allow ...

• Any number of wiki to be monitored and auto-restored
• User determined "Download address" (you could run different settings files for different downloads folders if needed)
• Provides multiple copy redundant Backup and/or Zip file archives that are time-stamped
• Will clone ("Parrot") your wiki to multiple locations if needed on change

Polly is develping into an interesting tool, I think, that is more than a substitute for not having browser saving for some browser brands. It does more.

Best wishes

Josiah

[Mohammad]

Many thanks Josiah,

 If I want to use Polly on Windows 10, is there any documentation / tutorial to help me!
I understood there is no installation required!

Thank you

[@TiddlyTweeter]

Hi Mohammad

Thanks for your interest!

Polly is still in experimental testing. 

I wouldn't use it on production wiki yet. There don't seem to be any major problems, but we need to be sure.

Some basic docs are about half written by Mark & I. I'll let you know when they are ready.

Best wishes

Josiah

[Mark S.]

There's an additional issue in GIC. At the bottom of the "comments" section there needs to be a "continue" statement. Otherwise

the code will drop through to the next "switch" test and then the commented-out files will be passed back along with the regular

files. I don't know why it requires "continue" when it should technically require "break", but that's the only way that seemed to

work. I also don't know how this escaped the original author's notice, unless they never used comments.

To be more clear, it needs "continue" either just before or just after the line:

#$ini[$section][$name] = $value  

In my own copy I've used your GIC from tidbits but adding "continue". So if I'm the next one to update
both sets of changes will be there. I've also listed your changes and my changes in the comments at the
top to ease confusion.

                      1.4 - 2019/07/18 - Various changes to section tests (TT)
                      1.5 - 2019/07/22 - Needs 'continue' at bottom of section test (MAS)  

Thanks!

[@TiddlyTweeter]

Thanks.

I'll wait and store your version of GIC in "versions" on your next update. Better not to randonly add changes.

Your way of noting changes is more economical than mine too.

TT

[@TiddlyTweeter]

Ciao Mark S.

An interesting potential "caught on your own petard" issue came up :-).

I wanted to change the online-name of the current "polly.html" to "polly-dev.html". And create a new "polly.html" for basic documentation. But, according to our own rules, I can't do that. 

But It seems a bit odd to have to create "polly-doc.html" to address what may be a broad need. "Polly.html" would be a lot better I think for first contact?

Yours in the stem :-)

TT

[Mark S.]

It's probably time to reconsider the file-globbing versus regular expression approach. File-globbing was good, but it does limit the name comparisons.

You're good with regular expressions ;-)

What would be a good matching rule so that  stem matches

stem(1)

stem (1)

stem (1)(2)

BUT NOT

stem (old)

stem (john)

I'm thinking that a match that searches for    stem<space>*<left brace><number><non-greedy-indicator>

This would mean people couldn't deliberately use (number) in their stem names, but would open up stems that

currently would match (e.g. polly vs polly-dev)

Thanks!

[@TiddlyTweeter]

Mark S. wrote:

It's probably time to reconsider the file-globbing versus regular expression approach. File-globbing was good, but it does limit the name comparisons.

You're good with regular expressions ;-)

 I love them! They are so annoying to get right :-) 

What would be a good matching rule so that  stem matches

stem(1)

stem (1)

stem (1)(2)

BUT NOT

stem (old)

stem (john)

I'm thinking that a match that searches for    stem<space>*<left brace><number><non-greedy-indicator>

 

 This would mean people couldn't deliberately use (number) in their stem names, but would open up stems that

currently would match (e.g. polly vs polly-dev)

I'll do a test page. In the end it will be easiest to be sure we get it right. 

A domani

TT 

[@TiddlyTweeter]

Mark S.

As I look at it, working the regex, with wiki-name now honoured. I realised that ...

   (a) describing Polly got easier; 

   (b) we do not need to save anyone from the disaster of "stem (2)" naming. Merely advise them its not a good idea. 

The NAMING LIMITS REVERT solely to "downloads", which is exactly where the issue is and should only be.

TT

[Mat]

@TiddlyTweeter wrote:

   (a) describing Polly got easier; 

When it is finalized, please do describe it fully. This one of those creations that I feel might be useful but I don't quite yet understand what it solves... so kindly also describe the problem.

<:-)

[@TiddlyTweeter]

Mat

Thanks for asking!

Its a Minority Report. 

It saves wiki, does backup (timestamped, zip or normal, or both), plus auto-cloning of wiki to secondary locations. 

Regardless of browser. Any modern browser will do. No plugins.  

It works by monitoring downlòads to restore wiki and more. Windows based ATM.

Its complimentary to other TW saving methods. Its additive, or it can be used just on its own to restore downloads.

Its still experimental. Documentation is coming.

TT

[Mat]

@TiddlyTweeter wrote:

Its a Minority Report. 

It saves wiki, does backup (timestamped, zip or normal, or both), plus auto-cloning of wiki to secondary locations. 

Automatically, right?

 

Regardless of browser. Any modern browser will do. No plugins.  

No plugins in the browser, right? Beause it is a TW plugin... or is it a separate software that I need to run every time I start up TW? If the latter, does it do this for all active TWs or must I manually "add" each TW that it is supposed to work for?

 

Its complimentary to other TW saving methods. Its additive, or it can be used just on its own to restore downloads.

You mean the features that it has that other savers do not? And no conflicts with other savers?

What does "restore a download" mean? Maybe overwrite the latest file with a backup file?

Thank you.

<:-)

[@TiddlyTweeter]

Mat asked:

@TiddlyTweeter wrote:

Its a Minority Report. 

It saves wiki, does backup (timestamped, zip or normal, or both), plus auto-cloning of wiki to secondary locations. 

Automatically, right?

Ciao Mat

Can be auto.

It has three modes ...

    -- "Auto", 

    -- "Once" 

   and 

    --"Menu". 

"Auto" runs a monitor that restores wiki at user defined intervals.  "Menu" gives a way to change settings and see what is happening as well as to switch into Auto or Once mode. "Once" mode does an instant restore and exits.

Regardless of browser. Any modern browser will do. No plugins.  

 

No plugins in the browser, right? Beause it is a TW plugin... or is it a separate software that I need to run every time I start up TW?

Its a Windows PowerShell script.

 

If the latter, does it do this for all active TWs or must I manually "add" each TW that it is supposed to work for?

Yes. 

Multiple configurations are possible using settings files.

Its complimentary to other TW saving methods. Its additive, or it can be used just on its own to restore downloads.

You mean the features that it has that other savers do not? And no conflicts with other savers?

It has no conflict with any other saver because it only works when you use fallback download saving. 

What does "restore a download" mean? Maybe overwrite the latest file with a backup file?

It means you "save" (to downloads) via TW and the file is copied back and overwrites the wiki to keep it up to date.

Its a slightly differnet approach than plugin savers. 

It supports TW5 & TWC. Most any browser. 

In a way, its inspired by Eric Shulman who has long argued the merits of default saving. 

All Polly does is leverage download saving and add dimension to it. Make the restore automatic, not a manual labour. And add backups.

"Parrots" (auto-cloning of wiki to other locations) are an interesting innovation it has that other methods don't have.

TT

[@TiddlyTweeter]

Mat asked:

...  must I manually "add" each TW that it is supposed to work for?

Good query! 

I do think the overhead of setup could be an issue.

On the other hand, the fact you can configure (once you "get" what it can do)  in many ways for different purposes is useful.

If I can figure it out I'll write a settings tool to ease the process.

TT

[@TiddlyTweeter]

Ciao Mark

I played around with it. 3 main tests are here: https://tidbits.wiki/reginald/tw_regex.html

The solution ALLOWS for the "Falcon(7)" case. (BTW the visual presentation of the case makes it look like the first (7) is removed--actually that is just a limitation of the difference engine in TW. Its the second (7) that actually matches.) 

My feeling here is ADVISE against "Falcon(7)", but not prevent it matching. 

The tests assumes that the basic pattern is ...

[wikiname-literal][regex-of-number-cruft][.extension-literal]

The regex I came up with is ...

(\s*\(\d+\))*

This means ...

    Match 0 or more spaces as many times as possible

    Match left bracket

    Match any number

    Match right bracket

    And repeat the pattern 0 or as many times as possible

This a JavaScript regex.  

I'm pretty sure it compatible with PowerShell. Let me know if you have any problems.

Best

TT

[Mark S.]

Overhead? I think it's comparable to other solutions.

You unzip to a directory. You configure a simple ini. You run a batch file.

The setup by INI file is easier than the JSON configuration used in tiddlyserver or Bob. The latest versions of both of these products

increased in complexity for some reason, adding levels of indenture that seem unnecessary from a user point of view. Not sure if that

reflects some underlying node.js change.

In one of the upcoming editions, we hope to add a listing of monitored directories. All the *.htm, *.html, and *.tw files in those directories will

be automatically "restored".

The problem this solves --

If you're perfectly happy with solutions that only work in Firefox and possibly Chrome, require an (usually unsigned) executable, that may break

the next time FF changes its rules AGAIN, or that only work with sub-directories of the download directory, then it may not solve any problem for you.

But if you want to see other browsers, not use extensions, and not use (or can't use) executables, then Polly might be a good solution for you.

If you want to save copies of your changes in more than one place at once, then Polly might be a good solution for you.

If you want a solution that might (some future edition) allow you to run an executable and send a copy of your TW off to a server, then Polly might be a good solution for you. More to the point, if you want a solution where the code is simple enough for you to add on your own processes, then Polly might be a good solution for you.

For me, I think the Extension Apocalypse from a couple months ago rekindled my interest in this project. I had no idea

that FF was re-evaluating my extensions every single day. There was total lack of transparency in the process. So one

morning we get up, and all the extensions we've been depending on are gone. They never explained exactly how they

could allow this to happen. The incredible amount of carelessness involved in this fiasco got me rethinking about FF.

At this point, most of the native features I really care about in FF are gone -- RSS feed, extensions, tab groups. Despite the

"improvements", FF continues to be a memory hog, gradually sucking up 60% of available memory. Over the past

year, FF's share has dropped 1.3% and continues downhill.

There are other browsers out there -- Brave, Vivaldi, Edge (two versions), IE, Palemoon, Waterfox -- so having more options

is a good thing, right? Also, Powershell is allegedly coming to the Linux and Mac platforms, meaning that Polly (or some variant)

might become a universal platform solution as well as universal browser solution.

HTH!

[Mark S.]

Hi TT

This just has the changes to the ps1 file with the new regular expression filter you suggested:

https://drive.google.com/file/d/1q-yFuBiWnA30ZMaICFfljDSfCgZKAgVN/view?usp=sharing

I tried using your test environment, but I think that it may have to be rewritten for our new rule (no stems like "mystem (100)" ).

I was kind overwhelmed by the feedback, and wasn't sure if some files were not being updated or if they already had on

a prior cycle. Are all the test files "touched" on every reloop? Also, what is the point between cycle 2 and 3 ? Are they

the same but with different times, or something else? Anyway, I'm sure you'll know how to use it to check results and

see if anything has to be tweaked in the code.

BTW, your reporting in the PS1 file looks better than mine, at least everything I've seen so far.

Thanks!

[Joshua Fontany]

Commenting so I can come back and find the thread later.


This is really neat! Well done.

Best,

Josh F

[@TiddlyTweeter]

Mark S.

01j-MS is ... Looking good!

Over next few days I'll do second test set that is less noisy and gives better feedback on outcome.

TT

[@TiddlyTweeter]

Ciao Mark S.

Comment from test on BACKUPS ...

• Your choice to make Backups and Zips be of the "previous" version (i.e. prior to RESTORE) of the wiki is, I think, a very good one.

• It matches well what backups are for. And better ensures viability.

• But I I think its worth our mentioning the behaviour in documentation.

TT

[@TiddlyTweeter]

Ciao Mark S.

Additional Run Mode for Parrots Only?

I was thinking about adding a further "-run" mode to fully support Parrots. Say ...

 "-run parrot-now"

This would be like "-run once" but that performs  a "forced clone" (i.e. it ignores download status for the wiki) and simply copies to addresses under "[parrots]" (only).

The advantage of this would it could be used to maintain, say, a website structure that copies additional files (listed under [wikis]) that have never passed through downloads.

Hope this is clear!

Any thoughts?

TT

[Mark S.]

Hi TT,

Interesting. I think it would mainly take some additional "if" loops to jump over the code where the download directory is checked.

It would definitely need to be in the documentation that parroted files might get overwritten. Hmm ... guess that should be there already.

I could see someone being interested in this even if they weren't interested in backing up TW. For instance, you might have a password file that you want to put on gdrive or dropbox.

[Mark S.]

The backup choice certainly makes sense for me. My working files are on a network drive, which may occasionally flake out. The most dangerous moment is when copying from the download directory which is on a reliable (knock on silicon) hard drive to the home on the network drive. So making a copy before that to the (hopefully) solid hard drive first makes the most sense. That way even if something bad happens in transit, there's one or more backups.

Whom or what are these dock Men of which you speak?

[@TiddlyTweeter]

Ciao Mark S. & interested avians,

I made a second test sequence that focuses more tightly on CHECKING that Polly basics work ... I will upload it soon.

Overall everything is looking VERY good. 

BUT, there IS an **informational issue** on Backups & Zip Archives. To do with the name/time stamp format. 

This is the easiest way I can put it ...

Q: SHOULD they be stamped with the moment of CREATION of the backup or Archive?

Or should they be stamped with a cloned time of the LASTWRITETIME of the Wiki Backed-up or Zip Archived? 

Thoughts!

On the one hand it makes sense to mark backups with timestamps of exactly when they were created. 

On the other hand it ALSO makes sense to replicate the exact time of the Wiki that is backed-up.

I'm trying to think this through.

Your thoughts?

TT

[@TiddlyTweeter]

In the process of working on POLLY I recalled the excellent experimental film by Peter Greenaway, "THE FALLS", in which anyone whose name prefix is "Fall" got inflicted by a BIRD VIRUS that changes their languge to "Miceleese", a bird language, especially of parrots ...

THE FALLS enumerates 92 documented cases of the "Falls". Here is one ...

Pollie Fallory (74): https://www.youtube.com/watch?v=kG4ukbx97Vw

TT 

[Mark S.]

At the time I made it I was mostly worried about creating unique file names that wouldn't clobber each other.

At some point, if we can figure how to do it, it would make more sense to stamp them with the original date stamp, since that's

what you're going to be looking for if you have to do a restore-from-backup.

[@TiddlyTweeter]

Mark S. wrote:

At the time I made it I was mostly worried about creating unique file names that wouldn't clobber each other.

With the improved testing system its easier to evaluate what is going on and experiment quickly to better see results of our choices.

I am hopeful it will be helpful.

The encouraging thing is that everything (appears) to be running as we expect.

I just need make sure on one aspect its running as intended. Soon.

TT

[@TiddlyTweeter]

Ciao Mark S. & interested avians

I worked on the test system.

Its far less busy than before. It works through these tests ...

1 - basic tests on name & number in downloads removal of numbers for save (files are given content on phase 1 so you can see time AND size in results)

2 - advanced tests on name and number (basically, the regex works)

3 - basic reporting on backups and zip archives

4 - parroting check.

[@TiddlyTweeter]

Ciao Mark S.

This is a revised test system under .\dev. Its not a complete update, just polly-test-01j-TT

https://drive.google.com/file/d/1kKmBv3XatVVr8buxSlmhKpewZ1D6vyZB/view?usp=sharing

Hope its useful! It comes with instructions. You have to run polly-once.bat twice to activate restores etc whilst using it. This time I gave basic notes on what the outcomes should look like.

Best wishes

TT

[@TiddlyTweeter]

Mark S. wrote:

At some point, if we can figure how to do it, it would make more sense to stamp them with the original date stamp, since that's

what you're going to be looking for if you have to do a restore-from-backup.

Okay.  Agreed.

I just wanted to addionally mention that I noticed in tests that if you have more than a couple of wiki the backups and zip-archive listing get very noisy.

Might it be better to do ...???

...\backupdir\wikiname-ext\wikiname-datestamp.ext

... so each wiki has its own sub-dir in backups?

Same with zips?

Q: Do you know if the PowerShell zip system supports incremental adds to an archive? 

If possible, what I'm thinking of, is to have one zip file per wiki, that date-stamped wiki get added to? 

Thoughts?

TT

[Mark S.]

I'm working on the new timestamp and subdirectories routines.

I think adding zips to a single archive would be inadvisable since as it increases in size it will take

longer and longer for the backup to process. Eventually it will probably crash.

An ambitious enhancement would be to delete all but the last x number of backups. Of course, it's

always a little worrying to be deleting things.

Thanks!

[@TiddlyTweeter]

Mark S. wrote:

I'm working on the new timestamp and subdirectories routines.

I think adding zips to a single archive would be inadvisable since as it increases in size it will take

longer and longer for the backup to process. Eventually it will probably crash.

Okay. Understood.

 

An ambitious enhancement would be to delete all but the last x number of backups. Of course, it's

always a little worrying to be deleting things.

I agree that it will eventually need some kind of ("periodic"??, once a session?) backup management. If a user fully went Polly with a bunch of wiki especially, with auto-save on, running in auto-mode at 60 secs, the numbers of backups could get very high quickly.

FWIW, I had a quick look around at .ps1 backup related scripts that manage "spread" of "preserved" backups (i.e. schemas like 1 a day, 1 a week, 1 a month etc) based on date matching. I didn't find one that fitted our use case perfectly but there may be one that could handle the problem?

Another approach might be, rather than delete, to move all files that would be deleted into  zip-archive (in this case I think possible failure on zip would be lower as its performed once) in a different directory? The user then manually decides to delete them?? 

Thoughts

TT

[@TiddlyTweeter]

Ciao Mark S.

Just wanted you know I'm working on documentation and the "Usage" notes.

As you pointed out before, a "help wiki" is a good idea. I'm just beginning to think into what needs to be in it.

I have a couple of technical Q's on how I can add "full parroting" to your magic restore loops. 

But I'll formulate the issues better first so you are clear what I'm asking.

Best wishes

TT

[Mark S.]

polly-01k-MS.zip

https://drive.google.com/file/d/1irHB15L1NIi2qo6JwU0PHM2hgFKaEV46/view?usp=sharing

This should be the "complete" kit for anyone following along. Not complete in terms of documentation.

General overview of usage.

Make sure you have backups somewhere else of any files you will be "restoring" with this system.

You need powershell 5 or greater. Powershell is available for free on windows 7 and 10. It is probably already

on your Windows 10 system.

Download zip.

The following instruction is likely different from other installs you may have done

Right click on zip file and unblock it. Without this step you will probably get a barrage of messages

when you attempt to run the scripts.

Change polly-01k-MS directory to something more useful like "polly".

Configure "settings.ini" for your needs.

Open a command box and go to the polly directory.

Use polly-menu.bat to see a menu of options

Use polly-once.bat to do a one-time restore of your files.

It is likely that the first time you run these files you will have to agree to running one or more scripts.

Hopefully this will be a one-time task.

[@TiddlyTweeter]

Mark S. wrote:

polly-01k-MS.zip

https://drive.google.com/file/d/1irHB15L1NIi2qo6JwU0PHM2hgFKaEV46/view?usp=sharing

This should be the "complete" kit for anyone following along. Not complete in terms of documentation.

Right now you and I know its powerful. And that it already does many things easily.

But without a settings editor or method to make an .ini and instructions on how to I don't think it will be a happy experience trying to use it :-(. Maybe I'm wrong?

I working on a settings editor. It will take me some time as I'm very slow, not being a programmer by trade. I looked at the same library that Get-IniContent came from and see there was a proto-editor. I might try use it?

Best

Josiah

[@TiddlyTweeter]

Ciao Mark

Below my various bits of docs (a lot of which was informed by you). None finished. But, on the whole a lot further on than before. They make clearer, I think, the scope of documentation needed.

My feeling is the documentation IN Polly be minimal and most of this would go into a TW?

  

  |       WHAT DOES POLLY DO?

  |

  |       Restore - Without any plugin saver TiddlyWiki uses the browser "downloads" folder

  |       -------   to save files like this ...

  |

  |                      mywiki (3).html

  |                      mywiki (2).html

  |                      mywiki.html 

  |

  |                 With each new save the number increases by 1. 

  |

  |                 Polly works out what wiki saved which file and RESTORES the 

  |                 latest version, removing the number and overwriting the original.

  |

  |                 When the user RELOADS their wiki in the browser the restored version

  |                 loads.

  |                         

  |        Backup - Polly can make a copy of your wikis and time-stamp them to a 

  |        ------   backup directory. 

  |

  |                 The latest backup is always of the previous version of a wiki, 

  |                 not the latest restored version.

  | 

  |   Zip Archive - Polly can create a zip archive of your wiki and archive a 

  |   -----------   time-stamped version to a Zip file.

  |

  |                 The latest archive is always of the previous version of a wiki, 

  |                 not the latest restored version.

  |   

  |        Parrot - A "parrot" is a clone, or copy, of a wiki to another

  |        ------   location. Parrots include TiddlyWiki based websites being

  |                 prepared for upload to the internet. Automating this

  |                 process through Polly is labour saving.

  |

  |                 Note: Parroted wikis get over-written by the latest 

  |                 restored version of the wiki.      

  |

  |                 Parrot Mode - There is also a "parrot" (only) mode. This "parrots" 

  |                 any files under [wikis] listed for cloning in [parrots].

  |                 It copies the files regardless of whether they are restored or not 

  |                 Its most useful if you are maintaining a secondary repository. [NOT YET DONE].

  |       INSTALLATION

  |

  |                 TiddlyWiki 5 - Polly works with TW5 on all modern browsers. Any browser 

  |                                with "download saving" should work with Polly. 

  |

  |                   TW Classic - Polly works with TWC on most modern browsers. [NEEDS CHECKING]

  |                                An exception is Edge. Chromium/ Edge (expand) work with TWC. 

  |

  |  Requirements - ??? Windows ver? Powershell lowest ver?

  |  ------------

  |                 Polly installation makes NO changes to your computer's

  |                 setup. 

  | 

  |                 Note: you may need to grant permissions, after installing, 

  |                 to let it run.

  |

  |       STEPS

  |

  |   1. Download - ??? What, where?. 

  |      --------   

  |                 Downloads of Polly carry their version as a 3 character string. 

  |                 For instance: "polly-01p.zip" is "version '01', edition 'p'. 

  |                 The version will be displayed when you run Polly. You can

  |                 ignore it, unless you have a problem to report.

  |

  |   2.  Unblock - Once you have downloaded "polly-xxx.zip", navigate to 

  |       -------   downloads in a file explorer and right-click the file.

  |                 Click on "Properties" and look for "Unblock" and select and

  |                 Apply it. 

  |   

  |                 This will "Unblock" all scripts in the downloaded zip file.

  |

  |   3.   Un-Zip - Once you have "Unblocked" the downloaded the zip, right click 

  |        ------   to "Extract All". 

  | 

  |   4.   Folder - Create a folder where you want to install Polly. (It can

  |        ------   be anywhere.) Copy the contents of the un-zipped "polly-xxx" 

  |                 folder to it.

  |

  |                 Polly should now be installed on your computer

  |

  |       PERMISSIONS POLICY CODA

  |

  |   Polly is written in the Windows scripting language "PowerShell".

  |

  |   In can happen that, initially, running scripts may not be allowed due to 

  |   an existing "ExecutionPolicy" on the computer.

  |   

  |   Its easy to change the policy to be able to run Polly. There are 

  |   notes on Permissions here ... [EXPAND]

  |       FIRST USE: CREATE INITIAL SETTINGS.INI FILE

  |

  |         Start - Click "polly.bat" to start Polly. On first run you may see a

  |         -----   message saying ... [NOT DONE YET]

  | 

  |                      "Now, let's create your first settings file!"

  |

  |                 Your first settings file must be called "settings.ini". Polly detects 

  |                 if it exists and will prompt you to make it if it doesn't. 

  |                 Once created you can have as many additional ".ini" setting files 

  |                 as you want. 

  |

  |        Naming - BEFORE you add any settings for Polly its important to

  |        ------   understand it utilises the "downloads" folder. And the downloads

  |                 folder can only accurately save files with unique names.

  |    

  |                 So, all wikis monitored by Polly, MUST have *UNIQUE* names. 

  |

  |         Paths - All *wiki files* you want Polly to monitor and their *directories* should

  |         -----   exist before you enter them in settings.

  |

  |                 All *directories* for Backups, Zip Archives and Parrots should exist before. 

  |                 you enter them in settings.

  | 

  |                 If paths don't exist when Polly runs a restore PowerShell will present error messages.

  | 

  |      Portable - Polly can be used portably from a USB stick if paths [NOT WORKING YET]

  |       Pathing   in settings to wikis use relative addressing. All relative pathing needs

  |       -------   to be relative to the application directory.

  |

  |                 Note: the "Downloads" folder still needs to be at an 

  |                 absolute address. This is because most browsers require it.

  |

  |       .INI FILE

  |

  |                 "*" indicates minimal settings required for Polly to function. 

  |

  |   * [general] - Required. Section header for basic settings 

  |

  |                 inidescription=

  |                    Optional. Useful if you have more than one .ini file.

  |

  |                 waitseconds=

  |                    Optional. Seconds to wait between checks in "auto" mode.

  |                              Defaults to 60 seconds.

  |

  |   *             downloaddir= 

  |                    Required. Directory where browser downloads are saved.

  |    

  |                 wikidir=

  |                    Optional. Useful if your wikis are nested under one directory.

  |

  |                 backupdir=

  |                    Optional. Directory to create date-stamped backups in.

  |

  |                 backupzipdir=

  |                    Optional. Directory to create date-stamped zip archives in.

  |

  |                 A basic settings file might look like this ...

  |

  |                    [general] 

  |                    inidescription=My Basic Settings

  |                    downloaddir=C:\Users\Polly\Downloads

  |                    wikidir=C:\Users\Polly\Documents\tw-wikis

  |

  |                 If a setting is not defined its corresponding menu

  |                 item in Polly in "menu" mode is not shown. 

  |

  |   *   [wikis] - Required. And you need at least one wiki under it. In this section you 

  |                 add the paths to the wikis you want Polly to monitor & 

  |                 restore. One wiki per line, for example ...

  |

  |                    [wikis]

  |                    file1=C:\users\jon\notes.html

  |                    toDoWiki=C:\users\jon\mywikis\todo.html

  |                    ;file2=C:\photos\photo-album.htm

  |

  |                 Wiki names must be UNIQUE.

  |

  |                 Each file is preceded by a unique identifer, e.g.

  |                 "file1" followed by "=". The identifier should 

  |                 only contain letters or numbers. No spaces or 

  |                 special punctuation.

  |

  |                 Lines starting ";" are ignored. This lets you 

  |                 switch off restoration for that wiki.

  | 

  |    [parrots] - Optional. Under this section you add paths in the same way  

  |                as for [wikis] ...

  |

  |                    [parrots]

  |                    parrot1=C:\users\jon\Dropbox\notes.html

  |                    parrot2=C:\website\photo-album.htm

  |

  |                The names of parrot wiki need to match the names of

  |                items in the [wikis] section since [parrots] clone items

  |                in [wikis]

  |

  |                Parrots can include any type of file. 

  

  |       USAGE

  |

  |  powershell -file .\polly.ps1 [-ini "settingsfile.ini"] [-run "menu"|"auto"|"once"|"parrot"]

  |

  |             -ini   "settingsfile.ini", a user defined settings file 

  |

  |                    without an -ini parameter Polly loads "settings.ini"

  |                    without "settings.ini" present Polly prompts the user to create it [NOT YET DONE]

  |

  |             -run   "menu"    displays menu (which can start all other modes)

  |                    "auto"    checks & restores wikis at user set intervals

  |                    "once"    checks & restores wikis, then exists

  |                    "parrot"  runs extended parroting (only), then exits [NOT YET DONE]

  |

  |                    without a -run parameter Polly runs "menu" mode

  |

  |                    Polly comes with three ready to run batch files ...

  |

  |                    "polly.bat"      runs Polly in "menu" mode

  |                    "polly-auto.bat" runs Polly in "auto" mode

  |                    "polly-once.bat" runs Polly in "once" mode 

  |

  |                    all load the default "settings.ini" file

 

[Mark S.]

On Monday, July 29, 2019 at 11:47:19 AM UTC-7, @TiddlyTweeter wrote:

But without a settings editor or method to make an .ini and instructions on how to I don't think it will be a happy experience trying to use it :-(. Maybe I'm wrong?

The settings are easier to read and set than for Bob or TiddlyServer. So at least some people should be OK with it. But maybe the people you're thinking of are

different than the people I'm thinking of.

 

I working on a settings editor. It will take me some time as I'm very slow, not being a programmer by trade. I looked at the same library that Get-IniContent came from and see there was a proto-editor. I might try use it?

Shouldn't hurt ;-)

 

[@TiddlyTweeter]

On Monday, 29 July 2019 21:07:33 UTC+2, Mark S. wrote:

On Monday, July 29, 2019 at 11:47:19 AM UTC-7, @TiddlyTweeter wrote:

But without a settings editor or method to make an .ini and instructions on how to I don't think it will be a happy experience trying to use it :-(. Maybe I'm wrong?

The settings are easier to read and set than for Bob or TiddlyServer.

Agree. But that is not exactly that I meant :-). Its that Polly doesn't really function at all without settings defined (I guess you could stare at the menu :). But if you try option A without having defined them you'll get swamped in Errorase. :-)

[Mark S.]

It could just post a warning at the end of the menu if key settings are missing. And the various routines refuse to run (with a message) if the downloads or

other essential settings are missing.

My sense is that anyone who is challenged by a simple ini file (pre-filled with sample data) won't be helped much by a wizard or step-by-step process ...

or am I overly pessimistic?

Thanks!

[Mark S.]

That looks very nice. Are you using notepad for all this?

|                 wikidir=

|                    Optional. Useful if your wikis are nested under one directory.

I've been thinking that the way it might work is instead to have a section "wikidirs". Then under

the section you could have

mydir1=c:\temp\mywikis

mydir2=d:\other\path\wikis

At start-up, the process would examine each of these directories, and then add any htm, html, tw files to the existing list

of files to be restored. The obvious caveat would be that the directories could not contain any htm, html, tw files that

you would not want to be written over from their corresponding file in the downloads directory. TiddlyWiki.html and empty.html

files would be the more obvious examples. And of course the stem names should not clash with names that are already being

restored via file=...\my.html.

Thanks!

[@TiddlyTweeter]

Mark S. wrote:

That looks very nice. Are you using notepad for all this?

 No, I use EditPadPro, a programmers text editor that you can configure for layouts like this.

I did the layout that way because reading stuff in a console I think needs a very clear layout. 

But, having done it,  actually think its too much to have all that in the console?? Most of it would go to a help wiki??

|                 wikidir=

|                    Optional. Useful if your wikis are nested under one directory.

I've been thinking that the way it might work is instead to have a section "wikidirs". Then under

the section you could have

mydir1=c:\temp\mywikis

mydir2=d:\other\path\wikis

At start-up, the process would examine each of these directories, and then add any htm, html, tw files to the existing list

of files to be restored. The obvious caveat would be that the directories could not contain any htm, html, tw files that

you would not want to be written over from their corresponding file in the downloads directory. TiddlyWiki.html and empty.html

files would be the more obvious examples. And of course the stem names should not clash with names that are already being

restored via file=...\my.html.

Very neat idea! Basically does what TonyM wanted to.

Making it dynamic has the real advantage that if you rename, move, add or delete any wiki under any  [wikidir]  Polly will register it. Simplifies things very well.

TT

[@TiddlyTweeter]

Mark S. wrote:

It could just post a warning at the end of the menu if key settings are missing. And the various routines refuse to run (with a message) if the downloads or

other essential settings are missing.

Okay. Makes sense. I can probably figure out how to do that bit.

 

My sense is that anyone who is challenged by a simple ini file (pre-filled with sample data) won't be helped much by a wizard or step-by-step process ...

or am I overly pessimistic?

Actually I broadly agree. And the introduction of   [wikidir]  section would ease setup & settings maintenance considerably.

Part of my interest in creating settings is for path testing. Parthly because I myself do typos often. But largely because I want all ".ini" to hold relative addresses (I try stay portable) and if you have complex relative pathing it can be cumbersome not having an immediate path check.

So. Agree your approach. But maybe I'll add some kind of setup related tool later?

I'm going to write you a seperate note on overall structure of  settings.ini  based on what's emerging.

TT

[@TiddlyTweeter]

Ciao Mark

Just thoughts about the .ini structure. For wiki (and other files  too) content it looks like we may well have ...

[wikis]

;--- WIKIS FOR RESTORE

file6=..\..\..\tw-wikidir\parrot\parrot-red.html

file7=..\..\..\tw-wikidir\parrot\parrot-green.html

[wikidir]

;--- WIKIDIR (??optional??): directories that hold wikis that get auto-added, should not contain any html files other than wikis.

mydir1=c:\temp\mywikis

mydir2=d:\other\path\wikis

[parrots]

;--- PARROTS (optional): auto-updated secondary copies of wikis 

parrot1=..\..\..\tw-sites\www.polly.net\parrot-red.html

parrot2=..\..\..\tw-sites\www.polly.net\parrot-green.html

I assume that in your "Magic Loops" that the dynamically found wiki in [wikidir] dirs will be added to the same "pool" (?hashtable?) as [wikis] at exact addesses?

IF SO, I have a suggestion, that we add an [otherfiles] section. Like this ....

[wikis]

;--- WIKIS FOR RESTORE

file6=..\..\..\tw-wikidir\parrot\parrot-red.html

file7=..\..\..\tw-wikidir\parrot\parrot-green.html

[otherfiles]

;--- OTHER FILES (optional): non-wiki files Polly manages for Parrots

other1=..\..\..\tw-wikidir\parrot\parrot-red.ico

other2=..\..\..\tw-wikidir\parrot\parrot-green.ico

other3=..\..\..\tw-wikidir\parrot\parrot-red.pdf

other4=..\..\..\tw-wikidir\parrot\parrot-green.pdf

 

[wikidir]

;--- WIKIDIR (??optional??): directories that hold wikis that get auto-added, should not contain any html files other than wikis.

mydir1=c:\temp\mywikis

mydir2=d:\other\path\wikis

[parrots]

;--- PARROTS (optional): auto-updated secondary copies of wikis 

parrot1=..\..\..\tw-sites\www.polly.net\parrot-red.html

parrot2=..\..\..\tw-sites\www.polly.net\parrot-green.html

parrot3=..\..\..\tw-sites\www.polly.net\parrot-red.ico

parrot4=..\..\..\tw-sites\www.polly.net\parrot-green.ico

parrot5=..\..\..\tw-sites\www.polly.net\parrot-red.pdf

parrot6=..\..\..\tw-sites\www.polly.net\parrot-green.pdf

This is so that visually when you look at your settings you can see non-wiki files you want to use for a "full parrot". These also get added to the "pool". (In reality you could still add non-wiki files to [wikis] or wiki files to [otherfiles] and it would work. But I think the conceptual and visual distinction is useful??

Hope this is clear!

Yours thoughts??

TT

[@TiddlyTweeter]

Finally Mark, & anyone interested in Polly

I been thinking about possible ways to further leverage Polly. The .ini settings approach + Mark's "Magic Loops" is enormously flexible. These are ideas I'm looking at ...

1 - Open wiki. Using a settings file run in a new "launch" mode, i.e.  -run "launch" 

                 An optional [general] setting would be added  called,  browser= 

                Which would be the path to a specific browser.

                Without the setting the O/S default browser would be used.

                This would launch the browser and open the wiki listed.

I been testing the idea in PowerShell. There are some issues in that I haven't yet found a reliable single way to assemble a command line all browsers accept. There are some differences between them. But perhaps its possible.

You could then get Polly to start a session in a specific browser, open wiki, & start monitoring.

            2 - Handling regular downloads. Using a settings file run in a new "fetch" mode, i.e.  -run "fetch" 

                 For example you list in a settings files you frequently need from the net like the TW pre-release.

                 Fetching files is very easy in PowerShell. And you can both rename and save places other than in downloads ...

                 Like this ...

 function fetchTests {

    # --- TT FETCH

    iwr -uri https://tiddlywiki.com/prerelease/ -OutFile ..\..\P\Users\Polly\Downloads\tw_fetch\tw5-pre-$(get-date -f yyyy-MM-dd).html

 }

IF either looks worthwhile pursuing let me know!

It would not need to be part of standard Polly. 

Maybe they could be implemented as "extension libraries"??

Thoughts!

Josiah

[@TiddlyTweeter]

Ciao Mark S.

UPDATE

Integrated your K with my latest. In testing now.

I will add stuff to do with initiating settings edit in next couple of days.

TT

[@TiddlyTweeter]

Ciao Mark

Both via test and running your latest directly I get this error ...

Yours in hope it may be ME.

TT

[Mark S.]

Well drat. I should have known that was too easy.

Though yes, it is probably "you". I'm guessing that you've done something somewhere to change your own local (locale?) date format string.

Do you recall making some setting to that effect? And/or changing locale?

Not that you should have to change, just curious if my theory is right.

[@TiddlyTweeter]

Unfortunately I changed nothing. 

Other than my computer is Italian.

TT

[Mark S.]

It's just that PS should be interpreting dates in local format. But I see that the day is in M/D/Y format, which

isn't the default for Italy or GB. Hmmm ... was it always an Italian computer, or was it something else once upon a time?

Oh well. Since it doesn't break for me, you'll need to tell me if this fix works. It's just for the standard backup dir at the moment.

Find the line

      $destinationTimestamp = $destination.LastWriteTime

and insert below it the new line:

      $dt = $destinationTimestamp

Then down below that find the first line that starts

      $archiveFilename = ...

And replace it with:

        $archiveFilename = "$stem-$exten-$(get-date -year $dt.year -month $dt.month -day $dt.day -hour $dt.hour -minute $dt.minute -second $dt.second -millisecond $dt.millisecond -f yyyy-MM-dd_HHmmss).$exten"

I don't know how GG will format this, but it apparently needs to be all one line.

Now when you test it, if the fix works there should be half as many errors (because there will still be errors from the zip backup).

Let me know!

Thanks!

[@TiddlyTweeter]

Ciao Mark S.

These are changes I made ...

Its working for Backups. Great!

Its also creating zip archives but unstamped like ...

tricky-polly-.zip

And throws big error for them during creation, as you said it would.

Best

TT

[Mark S.]

Ok. Great. I would just patch my own version, but I know you've already made changes to yours. So I'll describe what to do next. Since

we need the new naming routine twice, and maybe more, I've put it in a function.

Up above the RUN INFO section, paste in a new function:

# MAKE ARCHIVE STRING FROM DATE, STEM, AND EXTEN
function generate-archivestring {
    param( [datetime]$dt, [string]$pStem, [string]$pExten)
             return  "$pStem-$pExten-$(get-date -year $dt.year -month $dt.month -day $dt.day -hour $dt.hour -minute $dt.minute -second $dt.second -millisecond $dt.millisecond -f yyyy-MM-dd_HHmmss).$pExten"
}

Comment out the line that starts

$dt = ...

Comment out the the first line that starts

$archiveFilename = ....

and paste in

$archiveFilename = generate-archivestring $destinationTimestamp $stem $exten  

Find the next line (down in the zip dir section) that starts with

$archiveFilename = ...

and comment it out. Paste in

$archiveFilename = generate-archivestring $destinationTimestamp $stem "zip"

Hopefully this all makes sense.

Thanks!

[@TiddlyTweeter]

Tx Mark! I'm updating now ... I'll test and respond later.

One thing I wanted to comment, slightly OT, but interesting ...

Mark S. wrote:

Ok. Great. I would just patch my own version, but I know you've already made changes to yours. So I'll describe what to do next. 

I often wonder if TW, either through use of the diff. engine or via raw regex, could be used to apply patches to Tiddlers?

TT 

[Scott Kingery]

All,

Polly is interesting to me. This thread is long and I am just diving in. Observations and questions:

* Are you planning to allow the launch of a wiki from within the menu?

* Instead of using C:\Users\joeuser\downloads I'd love to be able to use system variables like this: %userprofile%\downloads

* Also appears relative paths don't work today. Like if I have c:\polly and my wikis are in c:\polly\tiddlywiki I was thinking I could do .\tiddlywiki\mywiki.html but that doesn't work. Actually, this tries to work but I think it looks for a path relative to the downloads folder.

* Another enhancement would be to check for the existence of the backup and parrots folders and create it if it doesn't exist.

Thanks for building this!

Scott

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/d0fe6222-ca4d-46af-be34-9b2daa8fa550%40googlegroups.com.

[Mark S.]

Hello Scott,

It's possible that you are our entire user base ;-)

On Thursday, August 1, 2019 at 5:18:16 PM UTC-7, Scott Kingery wrote:

* Are you planning to allow the launch of a wiki from within the menu?

Would that be helpful? If there's 20 wikis, would the user need to select and launch?

 

* Instead of using C:\Users\joeuser\downloads I'd love to be able to use system variables like this: %userprofile%\downloads

Have you tried this? I'm wondering if it might accidentally work already.

 

* Also appears relative paths don't work today. Like if I have c:\polly and my wikis are in c:\polly\tiddlywiki I was thinking I could do .\tiddlywiki\mywiki.html but that doesn't work. Actually, this tries to work but I think it looks for a path relative to the downloads folder.

We probably need to do a refactor to make this possible, though I never envisaged that anyone would want to run from the polly directory.

 

* Another enhancement would be to check for the existence of the backup and parrots folders and create it if it doesn't exist.

Some of that is already happening for the sub-directories for backup and zip.

It's making directories all the way down that worries me.

I cringe a little, because what if a user accidentally specifies a path to some place unintended? It's likely they'll complain that their

backup directory is empty, while meanwhile the actual backup is filling up their USB drive or something. Or filling up a "download" directory instead of a "downloads" directory.

But in general it's a good idea at some point.

Thanks!

[@TiddlyTweeter]

Ciao Scott and Mark

Mark S. wrote:

Hello Scott,

It's possible that you are our entire user base ;-)

lol! 

A few footnotes to Mark's replies ...

On Thursday, August 1, 2019 at 5:18:16 PM UTC-7, Scott Kingery wrote:

* Are you planning to allow the launch of a wiki from within the menu?

Would that be helpful? If there's 20 wikis, would the user need to select and launch?

I been looking at and thinking about this. It would be very easy to launch from Polly. The issue is what is launched and how. 

Say you had one settings file (you could have several for different situation) with, say, 4 wiki listed and they all got launched on a menu choice or, say, a -run mode of "launch"? Launch would be in the default browser. Is that the kind of thing you mean? Would that do it?

Being able to launch individual wiki by single selectable choice? Not so sure on that. I think it could over-complicate Polly.

FWIW, I been looking at whether you can relaibly set a non default different browser and launch. The way different browsers launch have some variations (though Chromium and Geko browsers work pretty much the same on launch commands) which, overall, may make selected Browser launch with Wikis more difficult. But I'm still experimenting to see if we could get a unified launch mechanism for other than default browser.

 

 

* Instead of using C:\Users\joeuser\downloads I'd love to be able to use system variables like this: %userprofile%\downloads

Have you tried this? I'm wondering if it might accidentally work already.

That should be quite easy to do. One approach is if you leave "downloaddir" $null it will revert to the Env variable for current users profile?

Mark, would you like me to try that approach? 

 

* Also appears relative paths don't work today. Like if I have c:\polly and my wikis are in c:\polly\tiddlywiki I was thinking I could do .\tiddlywiki\mywiki.html but that doesn't work. Actually, this tries to work but I think it looks for a path relative to the downloads folder.

We probably need to do a refactor to make this possible, though I never envisaged that anyone would want to run from the polly directory.

 That is maybe my bad as I, I think, set it that way without thinking it through. At the moment the first part of script runs in the app dir. The second half runs in downloads. That breaks portability and relative addressing.

I definitely think its something we should address and fix so that the script always runs in the app dir.

Mark, could you look into that? I ask because changing to always run in app dir will effect the "Restore" section most. That's where the change would have to be?

 

* Another enhancement would be to check for the existence of the backup and parrots folders and create it if it doesn't exist.

Some of that is already happening for the sub-directories for backup and zip.

It's making directories all the way down that worries me.

I cringe a little, because what if a user accidentally specifies a path to some place unintended? It's likely they'll complain that their

backup directory is empty, while meanwhile the actual backup is filling up their USB drive or something. Or filling up a "download" directory instead of a "downloads" directory.

I agree with Mark. I'd be nervous of auto-creating the  top level in a hierachy of folders because of the potential for unanticipated results. But any sub-dir below an existing directory I think is fine. So, for instance we maybe could eventually create sub-folder structure for parrots--provided the top dir exists?

Scott, great questions! Thanks.

TT

[PMario]

Hi folks,

Is there a github repo, where I can have a look at the project / code? ... Is there a way to contribute?

have fun!

mario

[@TiddlyTweeter]

Thanks Mark

I updated and its working well.

I believe there may be an issue on Zip Archive naming?

You drop the file extension in the "date stamp text" and replace it with "zip". 

Would that mean that wiki "tricky.tw", "tricky.html", "tricky.htm" would all end up stamped the same, e.g. as "tricky-zip-2019-08-02_184838.zip"?

Best

TT

[@TiddlyTweeter]

PMario wrote:

Is there a github repo, where I can have a look at the project / code? ... Is there a way to contribute?

There isn't yet a github for it. I maintain code in a TW using versioning. Its kinda works okay for this just being me and Mark so far.

I think with enough demand a github will probably emerge.

For myself I am reluctant to actually run a github for it. This is likely the only thing like this I will ever do,  so I'm reluctant having to learn yet another system that I'll use once.

But I can see longer term there may be enough interest for one and value in it.

Best wishes

Josiah

[Mark S.]

On Thursday, August 1, 2019 at 10:06:27 PM UTC-7, @TiddlyTweeter wrote:

Being able to launch individual wiki by single selectable choice? Not so sure on that. I think it could over-complicate Polly.

That's what I was thinking. It doesn't have a real graphical interface, and that wasn't the primary object.

 

That should be quite easy to do. One approach is if you leave "downloaddir" $null it will revert to the Env variable for current users profile?

We should find out if some version of "%userprofile%/downloads" will work out of the box. Then that can be our default in the settings,

and the user can change it if they have some special download location.

Mark, could you look into that? I ask because changing to always run in app dir will effect the "Restore" section most. That's where the change would have to be?

We'll definitely need to co-ordinate, since it's likely to need changes all over the main PS file.

 

I agree with Mark. I'd be nervous of auto-creating the  top level in a hierachy of folders because of the potential for unanticipated results. But any sub-dir below an existing directory I think is fine. So, for instance we maybe could eventually create sub-folder structure for parrots--provided the top dir exists?

The make-directory function already handles making new sub-directories, so we should be able to add it for parrots. Hmm ... not sure if it knows to make for subdirectory/subdirectory/sub...

Scott, great questions! Thanks.

Yes!

 

[Mark S.]

Github on the web is clunky because you can't make your changes, run, and test.

It becomes more useful in conjunction with a desktop version of Git. But not everyone is comfortable with that.

So, so far we've just been handing the baton back and forth. It seems to work, at least with two people.

I suppose we should have something like GitHub for followers to download from. Up to now, we haven't had any followers ;-)

[@TiddlyTweeter]

On Friday, 2 August 2019 19:21:24 UTC+2, Mark S. wrote:

On Thursday, August 1, 2019 at 10:06:27 PM UTC-7, @TiddlyTweeter wrote:

Being able to launch individual wiki by single selectable choice? Not so sure on that. I think it could over-complicate Polly.

That's what I was thinking. It doesn't have a real graphical interface, and that wasn't the primary object.

 

That should be quite easy to do. One approach is if you leave "downloaddir" $null it will revert to the Env variable for current users profile?

We should find out if some version of "%userprofile%/downloads" will work out of the box. Then that can be our default in the settings,

and the user can change it if they have some special download location.

I got something working on that. IF no setting in ini is made then the settings will be to the Powershell "$Env:userprofile\downloads", like this ...

  # --- DOWNLOADS DIRECTORY: where browser downloads go

  $downloaddir = $general["downloaddir"]

  # If not defined use o/s "userprofile" 

  if ([string]::IsNullOrEmpty($downloaddir)){$downloaddir = "$Env:userprofile\Downloads"}

Would that be enough to cover the case?

[@TiddlyTweeter]

@TiddlyTweeter wrote:

Being able to launch individual wiki by single selectable choice? Not so sure on that. I think it could over-complicate Polly.

 

Mark S. wrote:

That's what I was thinking. It doesn't have a real graphical interface, and that wasn't the primary object.

Right. In any case the ability to run many setting profiles if you want I think addresses that issue well enough without need for a menu picker. 

If a user wanted to load two wiki then I'm sure we could think about a settings file to automate that. 

I do like the idea, in principle, you could start Polly, open your wiki, automate their backup and restore.  

But the "invocation" of the loads needs thinking about. Cause what a browser does is not always the same (tab? or window? etc) and different browsers do vary too.

Thoughts

TT 

[@TiddlyTweeter]

Sorry Mark, I should clarify this ...

The zips ARE going into thir correct directories, but I think it would be better too if the individual zip files carried the name and extension of the wiki.

TT

[Mark S.]

On Friday, August 2, 2019 at 10:42:16 AM UTC-7, @TiddlyTweeter wrote:

I got something working on that. IF no setting in ini is made then the settings will be to the Powershell "$Env:userprofile\downloads", like this ...

  # --- DOWNLOADS DIRECTORY: where browser downloads go

  $downloaddir = $general["downloaddir"]

  # If not defined use o/s "userprofile" 

  if ([string]::IsNullOrEmpty($downloaddir)){$downloaddir = "$Env:userprofile\Downloads"}

Would that be enough to cover the case?

It would handle the general case of using the user's download directory as default. But not Scott's concern about using

environmental variables. Say a music lover had their download dir set to CSIDL_MYMUSIC because all they usually downloaded was

MP3s?

Thanks! 

[Mark S.]

On Friday, August 2, 2019 at 10:11:00 AM UTC-7, @TiddlyTweeter wrote:

Would that mean that wiki "tricky.tw", "tricky.html", "tricky.htm" would all end up stamped the same, e.g. as "tricky-zip-2019-08-02_184838.zip"?

No, unless I made a mistake, ...

tricky-tw becomes tricky-tw-2019-08-02_184838.zip

tricky-htm becomes tricky-htm-2019-08-02_184838.zip

tricky-html becomes tricky-html-2019-08-02_184838.zip

Thanks!

[@TiddlyTweeter]

So in his case he wants to pass it through from settings and have it expand when it reaches Polly?

[@TiddlyTweeter]

I don't that pattern in tests here. I see the ".ext" in the stamp as always "zip".

TT

[Mark S.]

Oops. Ok, changes ...

first $archiveFilename ...

$archiveFilename = generate-archivestring $destinationTimestamp "$stem-$exten" $exten  

second $archiveFilename

$archiveFilename = generate-archivestring $destinationTimestamp "$stem-$exten" "zip"

and new function generate-archivestring

# MAKE ARCHIVE STRING FROM DATE, STEM, AND EXTEN
function generate-archivestring {

    param( [datetime]$dt, [string]$pFilename, [string]$pExten)
             return  "$pFilename-$(get-date -year $dt.year -month $dt.month -day $dt.day -hour $dt.hour -minute $dt.minute -second $dt.second -millisecond $dt.millisecond -f yyyy-MM-dd_HHmmss).$pExten"
}

I guess the main question is whether I should be using a "-" or a "." for the extension separator in the first part of the file name. I guess it doesn't matter

as long as all strings are unique.

Thanks!

[Mark S.]

That would be the original request. After some research, it looks like if we bring in the settings like this:

  $backupdir= $general["backupdir"]
  $backupdir= [System.Environment]::ExpandEnvironmentVariables($backupdir)

(which we would probably put into a function to simplify)*

And have a setting entry like:

backupdir=%userprofile%\Downloads\polly-backup

That it works. Maybe you want to try that and see what you think. Is the extra complexity

worth the results?

Thanks!

* We should probably make our own routine library file at this point

[Scott Kingery]

Thanks for taking my suggestions into consideration!

You are right, Polly is more of a saving/restoring solution than a launcher and development should go in that direction

The %userprofile%/downloads was just an example. I could easily see where I'd put my wikis in Google Drive (for example) and reference them locally as %userprofile%\Google Drive\tiddlywiki  This has the potential of only needing 1 .ini file across my synced devices. 

Can I run Polly.bat with different ini files? I see that is available with polly-once.bat

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/0457248b-5a4f-4041-8a76-c65cfe630419%40googlegroups.com.

[@TiddlyTweeter]

Scott Kingery wrote:

You are right, Polly is more of a saving/restoring solution than a launcher and development should go in that direction

Actually auto-launch of a few wiki is not a bad idea at all! We're thinking about it. Its just being able to launch one when settings has more than one that is the issue--since Polly is designed to recurse over all entries for wiki in settings.  

The %userprofile%/downloads was just an example. I could easily see where I'd put my wikis in Google Drive (for example) and reference them locally as %userprofile%\Google Drive\tiddlywiki  This has the potential of only needing 1 .ini file across my synced devices. 

Right. But Mark & I just testing on this. It raised some interesting, relevant,  questions.

 

Can I run Polly.bat with different ini files? I see that is available with polly-once.bat

Yes you can have as many different settings files as you want. And all of them can be run in any one of the current -run modes ("auto", "once", "menu"). Later we'll probably write a few examples of bat files that load a specific settings profile. Testing indicates that you can, if needed, run more than one at a time.

Best

TT 

[@TiddlyTweeter]

Mark S. wrote:

Oops. Ok, changes ...

Thanks. Zip archives are named as expected now. But I'll do a bit more testing too.

I revised the test system to list the new nested backups and zips. I'll include that in my next update.

Re "." or "-" in names. I prefer the way you did it with "-". Just a preference not to see an ".ext" in the middle of a file name. 

I think its good though you used "." in the sub dir names. Makes explicit the connection to the source wiki.

FYI, I'm slightly delayed on settings creation thing. I got it working but need to now integrate it properly.

TT

[@TiddlyTweeter]

Mark: * We should probably make our own routine library file at this point

You show what that would look like and I'll follow :-).

I do think hiving off repeated things is a good idea.

I also thinking about "extensions". The heart of Polly is your "while" looping magic. 

I'm trying to better understand if we could, maybe, add an entry point that isn't for the more complex restore part but for things like "forced parrot" or "launch" that are not restore infinite processes, just a  single loop over settings. 

For instance in "menu" mode be able to run "once-menu", that would be "once" mode, but after completion returns to the menu. The point being ONE loop over the settings and return to menu. Hope this is clear!

Thoughts!

TT

[@TiddlyTweeter]

FYI, we now have two people interested ...

[@TiddlyTweeter]

Ciao Mark

I'm not sure its worth it on bespoke %env%. I toyed around a bit. I kinda felt it was bending a bit backwards on it. A user who needs that special path only needs enter it once, (literally) right, anyway in Polly settings? So its seems like overkill to worry the point? But maybe I''m missing something?

But I do think automating a default o/s determined downloads dir does have traction in that its a common situation and (hopefully) makes immediate sense to many users (when we have more than two).

TT

[Mark S.]

Did you find any actual downside? Once we have a function in place, it shouldn't be hard to implement. We just have to

"sanitize" the four+ directory types. Unless of course ... it breaks relative addressing or something.

Thanks!

[@TiddlyTweeter]

Mark S. wrote:

Did you find any actual downside? Once we have a function in place, it shouldn't be hard to implement. We just have to

"sanitize" the four+ directory types. Unless of course ... it breaks relative addressing or something.

No downside I can see at all. And I doubt it would break the relative thing in that, even with full relative addressing for everything else, its more than likely that downloads would remain absolute anyway but that would not be a problem.

TT