Adding comments in the Plover dict.json file.

[Ed]

If you want to add a comment in the Plover dict.json file,
just make a line of text where the steno part is an invalid entry, by
including a character that is not a recognized steno character, such
as X, or any other character that is not used for steno. It can never
be activated, so it just exists as a comment.
The English side can be anything.
Of course you have to use the mandatory format with quotes, colon,
space, and ending comma ("X": "X",) but that's just so that Plover
will continue to see the dictionary file as being correct.
And be very careful when messing with the 1st few and the last few
lines.
Warning: it's a good idea to make a backup of your known good working
dictionary file before experimenting, because if Plover does not like
your dictionary additions or changes, it could just stop, and you may
need to go back to a known good working dictionary in order to get
Plover working again.
Cheers!
Ed

.

[Hesky Fisher]

This workaround will work for a little while but the new version of plover modifies the dictionary and reorders it as well so your comments may disappear or move to different lines.

Once we have the dictionary manager integrated into plover then users shouldn't need to look at the raw dictionary file. Then I'll look into updating the format so we can store meta data about translations: how often it's used, when it was added, comment, etc.

Ed

.

--
You received this message because you are subscribed to the Google Groups "Plover" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ploversteno...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mirabai Knight]

I might be alone in this, but I hope that even if we change the dictionary format away from json it will continue to be editable/searchable in a text editor. The dictionary manager in my proprietary steno software is pretty powerful, but I still resent not being able to just open my Eclipse-formatted dictionary in Vim and go to town on it. 

[Hesky Fisher]

I'll keep that in mind.

Eclipse can export and import RTF dictionaries, which can easily be edited in vim. Would that work for plover too or do you want the native format to be text editable?

On Fri, Apr 26, 2013 at 8:57 AM, Mirabai Knight <askel...@gmail.com> wrote:

I might be alone in this, but I hope that even if we change the dictionary format away from json it will continue to be editable/searchable in a text editor. The dictionary manager in my proprietary steno software is pretty powerful, but I still resent not being able to just open my Eclipse-formatted dictionary in Vim and go to town on it. 

--

[Jeremy Kahn]

Let me suggest YAML as a transitional format: for the purposes here, it is effectively a more forgiving JSON - you can drop a y'all parser in place of the JSON parser.

But YAML also supports comments, and far more.

Jeremy

[Mirabai Knight]

On Fri, Apr 26, 2013 at 9:03 AM, Hesky Fisher <hesky....@gmail.com> wrote:
> Eclipse can export and import RTF dictionaries, which can easily be edited
> in vim. Would that work for plover too or do you want the native format to
> be text editable?

I feel strongly that I *do* want to Plover's native format to be text
editable without importing/exporting, but I'm not sure how rational
that feeling is. It might just be an artifact of frustration from the
general closed format style of proprietary steno software. Input from
cooler heads definitely welcome.

[Michael Wilber]

Here are some possibilities for alternate dictionary formats:

Bare JSON dictionary:
Example document:
{
"TKE/SAU": "Dessau",
"SRES/PA": "Vespa",
"-F/KOS/TAOUPL": "of costume",
"AF/TKPWAPB": "Afghan"
}
Advantages:
- Already built in to plover
Disadvantages:
- Ordering is unimportant, meaning that Plover might reshuffle entries
if it modifies the dictionary
- Entries containing " or other awkward unicode characters must be
escaped appropriately
- Syntax can cause problems (eg. a comma must come after every
key/value pair *except* the last one, so if you append a new entry,
you have to remember to insert the comma on the previous entry)



Bare JSON, but if we just ignore lines beginning with #:
Example document:
{
"TKE/SAU": "Dessau",
"SRES/PA": "Vespa",
# I had trouble stroking this one so I'm just going to add a correction
"-F/KOS/TAOUPL": "of costume",
"AF/TKPWAPB": "Afghan",
}
Advantages:
- Inherits all the advantages of JSON
- Easy to implement -- we just ignore lines beginning with #
Disadvantages:
- This is no longer valid JSON because the JSON spec doesn't allow
comments
- Inherits all the disadvantages of JSON. In particular, ordering is
still undefined, which means that:
- if Plover decides to add or reshuffle entries to the dictionary, it
will either forget all comments, or we have to have some complicated
way of associating comments to the proper JSON entries.



Bare JSON, but with the "workaround" comments:
Example document:
{
"TKE/SAU": "Dessau",
"SRES/PA": "Vespa",
"X": "I had trouble stroking this one so I'm just going to add a correction"
"-F/KOS/TAOUPL": "of costume",
"AF/TKPWAPB": "Afghan",
}
Advantages:
- Still valid JSON, which is nice for philosophy reasons (ie. we
don't need to write our own parser)
Disadvantages:
- Inherits all the disadvantages of JSON
- Ordering is still undefined, this still has trouble if plover
reshuffles dictionary entries.



YAML:
Example document:
-F/KOS/TAOUPL: of costume
AFR/KA: Africa
TPHOEBL/W*EUPL: noblewomen
# not sure about this one
PHUG: mug
Advantages:
- Valid JSON dictionaries are *also* valid YAML dictionaries, which
means we wouldn't have to ask users to convert their dictionary
- YAML syntax is "simpler" -- things are strings by default, so we
don't need to escape "of costume" in the above document, for
example. We do need to escape anything with {}s in it though.
- Comments are allowed by the spec
Disadvantages:
- PyYAML doesn't come built-in to python, which means we'd have to add
another dependency
- YAML syntax is "more complex" -- whitespace is important, things may
be surrounded in quotes or not, etc.
- PyYAML throws away comments when parsing the document, so if we
add/edit dictionary entries, we won't be able to preserve comments.
Example code:
>>> import yaml
>>> yaml.load("""
... -F/KOS/TAOUPL: of costume
... AFR/KA: Africa
... TPHOEBL/W*EUPL: noblewomen
...
... # not sure about this one
... PHUG: mug
... """)
{'PHUG': 'mug', 'TPHOEBL/W*EUPL': 'noblewomen', 'AFR/KA': 'Africa', '-F/KOS/TAOUPL': 'of costume'}



Writing our own parser:
Example document:
-F/KOS/TAOUPL of costume
AFR/KA Africa
TPHOEBL/W*EUPL noblewomen
PHUG mug
# Not sure about that one
Advantages:
- Appending entries could be easy
- We'd have full control over the syntax
- We could keep track of the comments if we desire
Disadvantages:
- We would have to write our own parser
- We would have to either convert dictionaries automatically or ask
users to convert their JSON dictionaries to the new format
Example code:
>>> def load_dict(document):
... dict={}
... for line in document.split("\n"):
... if line.strip() and not line.startswith("#"):
... stroke,space,translation = line.partition(" ")
... dict[stroke] = translation.strip()
... return dict
>>> load_dict("""
... -F/KOS/TAOUPL of costume
... AFR/KA Africa
... TPHOEBL/W*EUPL noblewomen
... PHUG mug
... # Not sure about that one
... """)
{'TPHOEBL/W*EUPL': 'noblewomen', 'PHUG': 'mug', '-F/KOS/TAOUPL': 'of costume', 'AFR/KA': 'Africa'}

[Daniel Langlois]

Features:

-Simple syntax

-Round trip - read, modify, write with comment retention

Advanced Features:

-Support allowing user control of files used

--Environment variables may be used to allow user to specify a default file

--Files may be nested heirarchically by including files from within a section or subsection

This one:

Bare JSON, but if we just ignore lines beginning with #: 

Example document: 

    { 

    "TKE/SAU": "Dessau", 

    "SRES/PA": "Vespa", 

    # I had trouble stroking this one so I'm just going to add a correction 

    "-F/KOS/TAOUPL": "of costume", 

    "AF/TKPWAPB": "Afghan", 

    } 

If comments in the file start with the '#' character and continue until the end of that line, I think that is

simple syntax. But, among the Disadvantages, ordering is undefined. So, this doesn't support read/modify/write

of files with comment retention. Or, it doesn't support 'round trip'. 

This one:

Writing our own parser: 

Example document: 

    -F/KOS/TAOUPL    of costume 

    AFR/KA           Africa 

    TPHOEBL/W*EUPL   noblewomen 

    PHUG             mug 

    # Not sure about that one 

Disadvantages: 

  - We would have to write our own parser 

  - We would have to either convert dictionaries automatically or ask 

    users to convert their JSON dictionaries to the new format 

While we are contemplating writing our own parser, I'm inclined to suggest that we take a step further back.

Say that want to check whether a file is a Python source, how do I list python 

source files? Something is built in. And, this might go further than checking whether a file name ends in '.py[w]'.

It might involve checking whether the file is not binary, and maybe it doesn't have the '.py[w]' extension, but the

first line contains the word 'python'. And maybe, you really need to check whether the file can be compiled.

Python can do all of this, internally, somehow. Consider its use in documenting Python source files.

Now, as Python uses dictionaries, they are similar to what their name suggests - a dictionary. In a dictionary, you have an 

'index' of words, and for each of them a definition. In Python, the word is called a 'key', and the definition a 'value'. 

And, you can add, remove, and modify the values in dictionaries.

So how do we make a dictionary?

Remember, dictionaries have keys, and values. So, creating a dictionary looks like this:

#Make the phone book:

phonebook = {'Andrew Parson':8806336, 

'Emily Everett':6784346, 'Peter Power':7658344, 

'Lewis Lame':1122345}

Ok, I've created a new phone book. Now I want to add new numbers to the book. What do I do?

#Add the person 'Gingerbread Man' to the phonebook:

phonebook['Gingerbread Man'] = 1234567

# Didn't think I would give you

# my real number now, would I?

Okay, so I'm adding entries to a dictionary.

What about removing entries from a dictionary?

del phonebook['Andrew Parson']

Again, very easy. 

Then, what about this kind of thing:

#A few examples of a dictionary

#First we define the dictionary

#it will have nothing in it this time

ages = {}

#Add a couple of names to the dictionary

ages['Sue'] = 23

ages['Peter'] = 19

ages['Andrew'] = 78

ages['Karren'] = 45

I'm saying, that dict.josn is an associative array, right? A hash table? Well, what about using the general syntax of a dictionary,

and having it actually be a dictionary. 

dict = {'Alice': '2341', 'Beth': '9102', 'Cecil': '3258'}

Or this:

dict1 = { 'abc': 456 };

dict2 = { 'abc': 123, 98.6: 37 }

Here, each key is separated from its value by a colon (:), the items are separated by commans, and the whole thing is enclosed in curly braces.

And then, there is this kind of thing:

dict['Age'] = 8; # update existing entry

dict['School'] = "DPS School"; # Add new entry

[Daniel Langlois]

About this suggestion, which, here is the code:

import ast

ast.literal_eval("{'x':1, 'y':2}")

=> {'y': 2, 'x': 1}

And, you can use 'eval' if you trust the input string (while the above method is more secure than JSON):

>>>a=eval('{"a":1,"b":2}')

>>> a

{'a': 1, 'b': 2}

-It does not break backwards-compatibility with the existing scripts for importing and exporting Plover dictionaries, for data interchange, as discussed here:

Import and convert dictionaries from other programs

https://github.com/plover/plover/issues/39

As Plover is already using a dictionary internally, so we can already encode/decode json files. 

-This is handled:

Multiple dictionaries running at the same time with priority

https://github.com/plover/plover/issues/42

How to add two Python dictionaries:

domains.update(domains2)

That is, the domains2 dictionary is added to the domains dictionary with the update() method.

-This is handled:

Invalid dictionary crashes Plover

https://github.com/plover/plover/issues/27

For syntax errors, or parsing errors, the parser naturally would, in this case too, repeat the offending line and display a little 'arrow', pointing at the earliest point in the line, where the error was detected.  

[Jeremy Kahn]

The two big disadvantages of encoding dictionaries as python files:

1) security: python code can include arbitrary python code, which would mean trying out dictionaries from an untrusted source is risky.

(Ruby went through a fiasco this last season because Ruby on Rails sites were all accepting arbitrary YAML as data upload, and the "unsafe" YAML superset includes the ability to include executable code.)

2) Portability: I love python, but if I ever wanted to edit or otherwise work with a plover dictionary with a tool, I might want to be able to use tools that aren't themselves in python. One list member just this week even used Java!  But if the dictionaries are in python syntax, we'd need to parse them with python, all the time.

Also, automatic generation and pretty printing of python-syntax dictionaries, even *in* python, is surprisingly hard, compared to YAML or JSON.

Jeremy

[Daniel Langlois]

On Saturday, April 27, 2013 8:59:16 AM UTC-7, Jeremy K wrote:

The two big disadvantages of encoding dictionaries as python files:

1) security: python code can include arbitrary python code, which would mean trying out dictionaries from an untrusted source is risky.

Except that Python actually includes the cabability to parse itself, which is what is happening in this example:

import ast

ast.literal_eval("{'x':1, 'y':2}")

=> {'y': 2, 'x': 1}

And I offered to characterize this as 'safer than JSON'. Although, in this case, comment blocks are not preserved by the Python ast Parser.

 

(Ruby went through a fiasco this last season because Ruby on Rails sites were all accepting arbitrary YAML as data upload, and the "unsafe" YAML superset includes the ability to include executable code.)

2) Portability: I love python, but if I ever wanted to edit or otherwise work with a plover dictionary with a tool, I might want to be able to use tools that aren't themselves in python. One list member just this week even used Java!  But if the dictionaries are in python syntax, we'd need to parse them with python, all the time.

Here, I'll point out that I'm saying the format of dict.json, doesn't necessarily have to change, even if we strip out JSON. You can parse the file as Python, as is. Or even deserialize it as Python, which is going to town, and would be worth pausing over, as an evil security hole. On the other hand, the Python source is generally already available to Plover users, they can do what they like. Is dict.json really an 'untrusted' source? My point is that the dict.json format is an abstract format, which a JSON parser can handle, as JSON is a subset of Javascript--but also, Python can handle it natively. 

Why strip out JSON? To be able to add comments to the dict.json file, was the original idea, but really, that may not add up to much of a reason. It's still interesting that the JSON dependency is somewhat more of a mirage, than it may seem. 

 

Also, automatic generation and pretty printing of python-syntax dictionaries, even *in* python, is surprisingly hard, compared to YAML or JSON.

You mean, indentation? The JSON parser does provide such, but Python has a pretty-printing module. I don't know about pretty printing of python dictionaries being hard--this is a flat dictionary, even. 

[Snowy Wilderness]

What about encoding Plover dictionaries in OPML (Outline Processor Markup Language) and then programming a dictionary manager skin in CSS, to be used via Dave Winer's rapidly evolving OPML viewer Fargo?

http://dev.opml.org/

http://fargo.io/

Dave Winer (the inventor of RSS and podcasting, also NYC-based): http://scripting.com/

Yours Truly,

Johnny Sagan aka Snowy Wilderness

[Daniel Langlois]

XML never felt like a great fit for data (or configuration, for that matter)

[Josh Lifton]

I agree - I try to avoid XML if I can.

On Sun, 28 Apr 2013, Daniel Langlois wrote:

> XML never felt like a great fit for data (or configuration, for that matter)
>
> On Sunday, April 28, 2013 9:40:26 AM UTC-7, Snowy Wilderness wrote:
>>
>> What about encoding Plover dictionaries in OPML (Outline Processor Markup
>> Language) and then programming a dictionary manager skin in CSS, to be used
>> via Dave Winer's rapidly evolving OPML viewer Fargo?
>>
>> http://dev.opml.org/
>>
>> http://fargo.io/
>>
>> Dave Winer (the inventor of RSS and podcasting, also NYC-based):
>> http://scripting.com/
>>
>>

>> On Sun, Apr 28, 2013 at 12:22 AM, Daniel Langlois <danlan...@gmail.com<javascript:>

>>>>> https://github.com/plover/**plover/issues/39<https://github.com/plover/plover/issues/39>

>>>>> As Plover is already using a dictionary internally, so we can already
>>>>> encode/decode json files.
>>>>>
>>>>> -This is handled:
>>>>> Multiple dictionaries running at the same time with priority

>>>>> https://github.com/plover/**plover/issues/42<https://github.com/plover/plover/issues/42>

>>>>>
>>>>> How to add two Python dictionaries:
>>>>>
>>>>> domains.update(domains2)
>>>>>
>>>>> That is, the domains2 dictionary is added to the domains dictionary
>>>>> with the update() method.
>>>>> -This is handled:
>>>>> Invalid dictionary crashes Plover
>>>>>

>>>>> https://github.com/plover/**plover/issues/27<https://github.com/plover/plover/issues/27>

>>>>>>>>>> email to ploversteno...@googlegroups.**co**m.
>>>>>>>>>> For more options, visit https://groups.google.com/**grou**
>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.

>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups
>>>>>>>>> "Plover" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an

>>>>>>>>> email to ploversteno...@googlegroups.**co**m.
>>>>>>>>> For more options, visit https://groups.google.com/**grou**
>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.

>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Plover" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>> send an email to ploversteno...@googlegroups.**co**m.
>>>>>>>> For more options, visit https://groups.google.com/**grou**
>>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>.

>>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Plover" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>> an email to ploversteno...@**googlegroups.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>>> .

>>>>>
>>>>>
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Plover" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to ploversteno...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>>
>>
>> --
>> Yours Truly,
>>

>> Johnny <https://vimeo.com/27826162> Sagan<http://issuu.com/overflow.magazine/docs/summer11-web/43>
>> aka<http://snowywilderness.tumblr.com/post/21046433446/new-yorker-screenshot-by-armando>
>> Snowy <http://exhibition-group.tumblr.com/> Wilderness<http://bcallalooksproject.com/>

[Zachary Brown]

Hi Hesky, Mirabai, and folks,

Personally I like the idea of plain text instead of RTF. It's nice to be able to write scripts to filter data; and that's harder and slower if the data has to be imported and exported for each filtration.

Plain text is also easier to use with revision control systems like git, if anyone in the future wants to start a project that maintains and enhances files of this type.

Closed formats are yucky; but even open formats that are hard to work with can be yucky for similar reasons. The value of an easy-to-use format is the same value UNIX commands have: it can be used in arbitrary ways that we don't have to be able to predict right now. UNIX allows arbitrary command chaining precisely because the creators of UNIX didn't want to constrain the actions of future users. They didn't want to create special-purpose use-cases that everyone would have to conform to - instead they recognized that everyone would have their own brilliant, unpredictable ideas; and they wanted to enable those people to realize those visions.

Be well,

Zack