Backend for NON-flat file store

[Randy]

Frank

If a backend for non-flat file store was created
(i.e. store the page text files in a directory tree),
what are the issues that should be considered?

For example:

1) Should a new Python module (?term?) be created or
should the existing one be enhanced with an option?
I like the idea of it being an option.

2) How should the directory and file names be mangled?
Can mangling be disabled (or made optional)?
Would the Wiki remain portable across platforms if
users limited themselves to portable naming conventions?

Should ".txt" suffixes be implicitly added to Wiki names
to derive the text filename?
Should this be optional?

3) What is needed for security?
Should the page directory tree be limited to a root,
i.e. only text files in the .../wiki/pages/... can be served?
This would be similar to the .../wiki/files/... convention.

Should the root name be configurable?
I don't see much need for this, since .../wiki/pages/
could be a symbolic link to wherever.

4) Should Wiki paths be relative to the root or
the page containing the link or both?

For example, paths that begin with "/" could use
.../wiki/ as the "root", while other paths are
relative to the page's directory, i.e. pages within
the same directory could refer to each other without
any path prefix, while foo_section/foo.txt walks deeper
into the tree, and ../bar.txt would walk out (but
not past the "root").

5) Could the Wiki name for the page default to the
text file name ? and visa versa ?

How would this interact with relative path names?
Perhaps all relative path names should require
at least a "./" path prefix to distinguish them from
simple Wiki page names. Example:

"FooBar" (AKA "/FooBar") is a simple Wiki page name in the
".../wiki/" directory.

"/pages/FooBar" refers to the FooBar text file in the
directory for Wiki pages, i.e. .../wiki/pages/FooBar

"./FooBar" refers to the FooBar text file in the
same directory.

"../FooBar" refers to the FooBar text file in the
parent directory.

"sect/FooBar" refers to the FooBar text file in the
sect subdirectory.

6) Should the Wiki engine manage revisions?
Implicitly ? Explicitly?

The Wiki engine could punt and simply assume
the Wiki site admin will make backups and
revision snapshots as desired (not under user control).

The Wiki engine could offer an explicit check-in button,
for the Wiki repository as a whole or for individual pages.
This may be a good compromise.

The Wiki engine could implicitly check-in every
change, no matter how trivial it might be... yuk.

- RLS

[Randy]

Another issue...

7) Where should the meta data be kept?

Top of the page file itself?
Sibling file (foo.meta or hidden as .foo.meta)?
Parallel tree (.../wiki/meta/foo for page .../wiki/pages/foo) ?

Keeping meta data at the top of the page is simple,
but it might constrain future enhancements.
For example, program documentation might be kept in
in a comment block that has funky delimiters.

... C code
/* blah, blah...
!!!{{{BEGIN WIKKYTEXT|
contents //processed// by ''Wiki'' go __here__.
END WIKKYTEXT}}}!!!
*/
... more C code

The Wiki engine might ignore everything in the source code
outside the funky delimiters and display the inside as documentation.
The engine might also provide an option to display
everything outside the funky delimiters in a preformatted
block that is highlighted for a specified language,
i.e. push this button to see my source code.

Wiki engine policies might limit browser editing to
within the funky blocks, or allow source code editing
(given sufficient permission).

In this case, the Wiki files should not have a ".txt" suffix
imposed upon them. They would need the normal suffix
for their respective programming language, i.e. ".c", ".py", ".f77",
etc.

Perhaps the meta data should be kept within their own
special funky delimiters (which is within comments as
appropriate for the language). This overall scheme would
be close to the Knuth literate programming concept.

[Frank McIngvale]

On Sun, Sep 27, 2009 at 10:16 AM, Randy <Randy...@data-warp.com> wrote:

Another issue...

7) Where should the meta data be kept?

Top of the page file itself?
Sibling file (foo.meta or hidden as .foo.meta)?
Parallel tree (.../wiki/meta/foo  for page .../wiki/pages/foo) ?

The flatfile store puts metadata in the .txt file directly. Is that what you are referring to when you say metadata here?

Wiki engine policies might limit browser editing to
within the funky blocks, or allow source code editing
(given sufficient permission).

In this case, the Wiki files should not have a ".txt" suffix
imposed upon them.  They would need the normal suffix
for their respective programming language, i.e. ".c", ".py", ".f77",
etc.

OK, I think I see what you are getting at here -- so instead of storing wiki items in .txt files, you want the wiki to read EVERY .c, .py, etc file and pull out the markup that's intermixed with code? That's quite a bit different than the current wiki concept where each "item" is purely markup, not markup mixed with other stuff that should be ignored. This sounds more useful in a read-only sense -- pulling the markup out, not trying to allow editing the mixed file within the wiki. At least it would be much simpler to implement that way.
 
frank

[Randy]

On Sep 27, 6:00 pm, Frank McIngvale <fmcingv...@gmail.com> wrote:

> On Sun, Sep 27, 2009 at 10:16 AM, Randy <RandyTid...@data-warp.com> wrote:
>
> > Another issue...
>
> > 7) Where should the meta data be kept?
>
> > Top of the page file itself?
> > Sibling file (foo.meta or hidden as .foo.meta)?
> > Parallel tree (.../wiki/meta/foo  for page .../wiki/pages/foo) ?
>
> The flatfile store puts metadata in the .txt file directly. Is that what you
> are referring to when you say metadata here?

YES,
What is the proper name for that stuff at the top
that I see after a page is stored?

>
> Wiki engine policies might limit browser editing to
>
> > within the funky blocks, or allow source code editing
> > (given sufficient permission).
>
> > In this case, the Wiki files should not have a ".txt" suffix
> > imposed upon them.  They would need the normal suffix
> > for their respective programming language, i.e. ".c", ".py", ".f77",
> > etc.
>
> OK, I think I see what you are getting at here -- so instead of storing wiki
> items in .txt files, you want the wiki to read EVERY .c, .py, etc file and
> pull out the markup that's intermixed with code? That's quite a bit
> different than the current wiki concept where each "item" is purely markup,
> not markup mixed with other stuff that should be ignored. This sounds more
> useful in a read-only sense -- pulling the markup out, not trying to allow
> editing the mixed file within the wiki. At least it would be much simpler to
> implement that way.
>
> frank

I'm not sure what the proper terminology is,
but I would want [[foo.c]] markup to be a link.
If clicked, the Wiki engine would read the file "foo.c".

The first stage of parsing would distinguish 3 things:
1) Stuff inside funky meta-data markup.
2) Stuff inside funky text-data markup.
3) Everything outside of (1) and (2).

(1) and (2) would be subsequently processed
much like the existing engine to render a page.

(2) could be edited, if the user had basic write privileges.
(1) would be automatically maintained by the Wiki engine.
(3) could be viewed, if the user had read-code privileges
and a [[foo.c#use-the-source-luke]] link was provided
(or some other trickery). If the user had write-code
privileges, then (3) could be edited.

If editing is allowed, the Wiki engine must remember
the relative position of (1) and (2) within the whole.
When saved, the pieces would be put back together.

[Frank McIngvale]

On Sun, Sep 27, 2009 at 9:27 PM, Randy <Randy...@data-warp.com> wrote:

On Sep 27, 6:00 pm, Frank McIngvale <fmcingv...@gmail.com> wrote:

> On Sun, Sep 27, 2009 at 10:16 AM, Randy <RandyTid...@data-warp.com> wrote:
>
> > Another issue...
>
> > 7) Where should the meta data be kept?
>
> > Top of the page file itself?
> > Sibling file (foo.meta or hidden as .foo.meta)?
> > Parallel tree (.../wiki/meta/foo  for page .../wiki/pages/foo) ?
>
> The flatfile store puts metadata in the .txt file directly. Is that what you
> are referring to when you say metadata here?

YES,
What is the proper name for that stuff at the top
that I see after a page is stored?

You're right in calling it metadata, just wanted to make sure we were talking about the same thing.
 

I'm not sure what the proper terminology is,
but I would want [[foo.c]] markup to be a link.
If clicked, the Wiki engine would read the file "foo.c".

The first stage of parsing would distinguish 3 things:
1) Stuff inside funky meta-data markup.
2) Stuff inside funky text-data markup.
3) Everything outside of (1) and (2).

(1) and (2) would be subsequently processed
much like the existing engine to render a page.

(2) could be edited, if the user had basic write privileges.
(1) would be automatically maintained by the Wiki engine.
(3) could be viewed, if the user had read-code privileges
and a [[foo.c#use-the-source-luke]] link was provided
(or some other trickery).  If the user had write-code
privileges, then (3) could be edited.

If editing is allowed, the Wiki engine must remember
the relative position of (1) and (2) within the whole.
When saved, the pieces would be put back together.

Still pondering this but here are my current thoughts ...

a. foo.c is inherently a source code file with some wiki content scattered throughout, NOT a true wiki item. So making it look like a wiki item by allowing [[foo.c]] to work seems to get confusing.
b. The splitting apart/selective editing/putting back together is a neat idea but it also sounds like a lot of work for something I can't see myself using that much. In other words, I'm ALWAYS going to edit "foo.c" or any source code file inside my regular editor. I would NEVER edit source inside a wiki environment on a regular basis.
c. However, the "pulling out wiki content" is worth doing I think. But again this is sounding better to implement as a plugin, not creating a new store type. To give you an idea, the Python docs plugin defines a macro <<pydoc>> where ...

<<pydoc name>> -- produce documentation for entire module 'name'
<<pydoc name func>> -- produce documentation for function 'func' inside module 'name'
<<pydoc name class>> -- produce documentation for class 'class' inside module 'name'

So (from what I understand) by using ctags you could accomplish the same sort of lookups with C, C++, etc. code. So then you could make a general <<srcdoc>> macro that does ...

<<srcdoc foo.c>> -- make docs for entire foo.c
<<srcdoc lib/foo/bar.cpp myfunc>> -- make docs for function myfunc inside bar.cpp

This seems like the cleanest way to implement it -- no new store type or deep URL magic, just a regular plugin like any other.

Would vim be able to understand the "foo.c" inside a macro call, or would it only work inside a [[link]]?

frank

[Randy]

On Sep 28, 6:07 am, Frank McIngvale <fmcingv...@gmail.com> wrote:

What do you think of this idea.
If the filename extension is ".txt",
then its processed exactly like WikklyText.

If the extension is anything else,
then only text within funky markers is processed.

> b. The splitting apart/selective editing/putting back together is a neat
> idea but it also sounds like a lot of work for something I can't see myself
> using that much. In other words, I'm ALWAYS going to edit "foo.c" or any
> source code file inside my regular editor. I would NEVER edit source inside
> a wiki environment on a regular basis.

Yes, I agree.

I can't see myself changing source code this way either,
at least not now and not with my current projects.
Editing source via the Wiki might be useful in the future
for some projects. I'm attempting to look into my crystal
ball and envision what might be useful. The objective is
to avoid design decisions that might hinder evolution.

> c. However, the "pulling out wiki content" is worth doing I think. But again
> this is sounding better to implement as a plugin, not creating a new store
> type. To give you an idea, the Python docs plugin defines a macro <<pydoc>>
> where ...

I trust your instinct as to what should be a plugin
versus what requires a new store... for me, these
distinctions blur together because of my ignorance
on this topic.

> <<pydoc name>> -- produce documentation for entire module 'name'
> <<pydoc name func>> -- produce documentation for function 'func' inside
> module 'name'
> <<pydoc name class>> -- produce documentation for class 'class' inside
> module 'name'
>
> So (from what I understand) by using ctags you could accomplish the same
> sort of lookups with C, C++, etc. code. So then you could make a general
> <<srcdoc>> macro that does ...

The ctags concept is independent of funky markup
within source code... neither is dependent upon the other.

Yes, something like a general <<srcdoc foo.c>> may be appropriate.

Question: can <<srcdoc ...>> markup result in a link?

When I was exploring TiddlyWiki functionality, a reply by Eric
suggested that [[<<srcdoc ...>>]] was not valid syntax
(if I understood him correctly) and the output from a macro
was not Wikified further (i.e. if the macro output "[[something]]",
it would not result in a link).

>
> <<srcdoc foo.c>> -- make docs for entire foo.c
> <<srcdoc lib/foo/bar.cpp myfunc>> -- make docs for function myfunc inside
> bar.cpp
>

Yes.
The <<srcdoc lib/foo/bar.cpp myfunc>> case may require that the
myfunc portion of the documentation be identified by the
funky markup in the source, i.e.
!!!{{{func myfunc| Documentation for myfunc}}}!!!
or
!!!{{{class myclass| Documentation for myclass}}}!!!

The general <<srcdoc ...>> and funky markup should be language
independent to whatever extent possible, otherwise the
Wiki-engine/plugin will have to understand the language
syntax... which is possible and could provide richer
documentation (but would require parsing the language,
source code preprocessing, and other difficulties).

> This seems like the cleanest way to implement it -- no new store type or
> deep URL magic, just a regular plugin like any other.
>
> Would vim be able to understand the "foo.c" inside a macro call, or would it
> only work inside a [[link]]?

Vim works either way.
>
> frank

[Frank McIngvale]

On Mon, Sep 28, 2009 at 8:22 AM, Randy <Randy...@data-warp.com> wrote:

On Sep 28, 6:07 am, Frank McIngvale <fmcingv...@gmail.com> wrote:

> a. foo.c is inherently a source code file with some wiki content scattered
> throughout, NOT a true wiki item. So making it look like a wiki item by
> allowing [[foo.c]] to work seems to get confusing.

What do you think of this idea.
If the filename extension is ".txt",
then its processed exactly like WikklyText.

If the extension is anything else,
then only text within funky markers is processed.

Something about it is bothering me -- mixing metaphors too much between wiki items and non-wiki items that contain wiki markup. Not saying it couldn't be done, but I think the right way to evolve this would be to start with a macro (<<srcdoc>>, etc.). Then if that gets to a good mature state, it may become clearer that either that's the desired end state, or make it clearer the right way to wire it up to get to [[foo.c]] working.

> <<pydoc name>> -- produce documentation for entire module 'name'
> <<pydoc name func>> -- produce documentation for function 'func' inside
> module 'name'
> <<pydoc name class>> -- produce documentation for class 'class' inside
> module 'name'
>
> So (from what I understand) by using ctags you could accomplish the same
> sort of lookups with C, C++, etc. code. So then you could make a general
> <<srcdoc>> macro that does ...

The ctags concept is independent of funky markup
within source code... neither is dependent upon the other.

Yes, something like a general <<srcdoc foo.c>> may be appropriate.

Question: can <<srcdoc ...>> markup result in a link?

 

When I was exploring TiddlyWiki functionality, a reply by Eric
suggested that [[<<srcdoc ...>>]] was not valid syntax
(if I understood him correctly) and the output from a macro
was not Wikified further (i.e. if the macro output "[[something]]",
it would not result in a link).

If that's the case, then that's another difference between TiddlyWiki & WikklyText, because WikklyText DOES reparse the return value from macros for markup. So you can make links programmatically. However, it is true that text inside links is not parsed, so  [[<<macro>>]] does NOT result in macro call. (Originally WikklyText parsed text inside links, but that seemed to cause more problems than it was worth, so I ended up doing what TiddlyWiki does). Here is an example that shows both cases -- just create a new item in your wiki and paste in this text (only works in WikklyText b/c of embedded Python code):

----------------------
<?py
# this is how I make download links @ wikklytext.com
VER = "1.6.0"

def ZIP(context):
  return "[[WikklyText-%s.zip|http://wikklytext.com/files/download/WikklyText-%s.zip]]" % (VER,VER)
?>

Text inside links is NOT parsed: [[<<not called>>]] just like TiddlyWiki

However, you can create links programmatically: <<ZIP>>
----------------------

So bottom line, yes, <<srcdoc foo.c>> can generate any arbitrary markup -- links, tables, whatever.

>
> <<srcdoc foo.c>> -- make docs for entire foo.c
> <<srcdoc lib/foo/bar.cpp myfunc>> -- make docs for function myfunc inside
> bar.cpp
>

Yes.
The <<srcdoc lib/foo/bar.cpp myfunc>> case may require that the
myfunc portion of the documentation be identified by the
funky markup in the source, i.e.
!!!{{{func myfunc|  Documentation for myfunc}}}!!!
or
!!!{{{class myclass|  Documentation for myclass}}}!!!

The general <<srcdoc ...>> and funky markup should be language
independent to whatever extent possible, otherwise the
Wiki-engine/plugin will have to understand the language
syntax... which is possible and could provide richer
documentation (but would require parsing the language,
source code preprocessing, and other difficulties).

This is where Python is great -- every function, class, etc., gets an attribute __doc__ that contains its docstring. So you just grab that and render it. But yeah, there would need to be some standard way to do it for other languages. The reason I thought of ctags is that it looks like ctags gives you a regex that lets you find the line in the source file where the function, etc. is defined. So the standard could be "look for the comments immediately before/after that source line".
 
It would be nice if there were a way to hook into an existing doctool (doxygen, etc.) so that functionality could just be reused instead of inventing yet another docs-mixed-with-code system. But I'm not very familiar with what exists and what could be easily integrated/embedded within WikklyText.

> This seems like the cleanest way to implement it -- no new store type or
> deep URL magic, just a regular plugin like any other.
>
> Would vim be able to understand the "foo.c" inside a macro call, or would it
> only work inside a [[link]]?

Vim works either way.

OK good, sounding like that may be a good way to go then.

frank

[Randy]

I'm hacking at a directory server...
Tiny baby steps, for now.

On Sep 28, 6:57 pm, Frank McIngvale <fmcingv...@gmail.com> wrote: