Brython and literal_eval from ast

[zd]

Hello,

we are planning to use Brython on our service pripravto.cz (which is Python app).  We like to idea to have Python in browser. It will ease our develompent, but we found out that we cannot use

from ast import literal_eval

We use it to convert string dict like "{'hello':'world'}" to convert it to real dict. I know we can use normal eval in Brython, but we would like to use literal_eval, because it is much safer. Is it possible to do something like that in Brython without eval?

https://docs.python.org/2/library/ast.html#ast.literal_eval

Thanks

[Kiko]

2015-03-10 14:49 GMT+01:00 zd <zdne...@gmail.com>:

Hello,

we are planning to use Brython on our service pripravto.cz (which is Python app).  We like to idea to have Python in browser. It will ease our develompent, but we found out that we cannot use

from ast import literal_eval

We use it to convert string dict like "{'hello':'world'}" to convert it to real dict. I know we can use normal eval in Brython, but we would like to use literal_eval, because it is much safer. Is it possible to do something like that in Brython without eval?

The json module would be useful for your use case?

 

https://docs.python.org/2/library/ast.html#ast.literal_eval

Thanks

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/667eb588-8935-4c5c-9fec-1a6b4cf8d575%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[zd]

Hello,
well right now we are using json in JS, but it's not really good because there are limitations such as "  ' " apostrophes. Our dicts are much complex, have list and other dicts inside with more complex values and these are not really workable in JSON. That's why we were thinking that Python would be better, because we use them internally.


Dne úterý 10. března 2015 14:59:09 UTC+1 kiko (on pybonacci) napsal(a):

[Billy Earney]

from brython do the following:

import json

json.loads(myjsonstring)

If myjsonstring is a valid json string, it should work, if not, give us an example, and we will try to fix it.

Billy

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/b794f4e9-46ee-41ae-ad2d-97682ee67090%40googlegroups.com.

[Glenn Linderman]

On 3/10/2015 9:26 AM, zd wrote:

well right now we are using json in JS, but it's not really good because there are limitations such as "  ' " apostrophes. Our dicts are much complex, have list and other dicts inside with more complex values and these are not really workable in JSON.

It is my understanding that JSON supports infinitely (memory limits only) nested Python lists and dicts, corresponding to Javascript Arrays and object literals.

You mention  " '" as a limitation... Could you give an example of such a limitation? I have converted Python data containing "straight apostrophes" to JSON quite successfully, and back again to Python.

Specific examples of what doesn't work, or even things that work but have some sort of issue, when using JSON, would be more useful than a vague statement regarding limitations and complexities that are not perceived by the community.

You mention converting string dict to real dict... JSON can do that.

You don't mention how you convert real dict to string dict... JSON can do that, too.

For my own work, I heavily use the JSON module to convert Python data structures to/from string form. I save these string form data structures in UTF-8 encoded files, and use the parameter  ensure_ascii=false to avoid converting characters beyond the first 128 (or is it 256) to escaped notation \uXXXX. Works very well, and can be read by most programming languages as most have a JSON support module these days.

Glenn

[zd]

Ok... well if the answear is to use JSON, I don't need Brython, as I can use JS and it will be similar as I have it now. My question is if I can use literal_eval func or ast module. Because that would actually make it easier.

For example  I want to have dictionary which have key test and value just "

In python it is just {'test':'"'} but in JSON this is not so easy and if I would like to have more complex strings it's almost imposible to have it right... when there would be string formating and more it gets easily broken in JSON.

Anyway thanks for information

[zd]

And one next reason, we would not need to actually convert anything... if it would possible, which would be perfect.

[Glenn Linderman]

OK, so now you are being a little more specific... it is not the inclusion of straight apostrophes in data that is your concern, but the ability to use straight apostrophes as string delimiters (whereas JSON always uses straight quotation mark for its string delimiter).

And in your next message, you mentioned conversions.... implying that perhaps you have a bunch of data already in the format of using straight apostrophe for delimiters.

In JSON, your example is:

>>> foo = {'test':'"'}
>>> print(json.dumps( foo ))
{"test": "\""}

Python would print that natively as:

>>> print( foo )
{'test': '"'}

When using json.dumps to generate your strings, it is not particularly harder to add a call to "json.dumps" to generate your strings, and any complexity of the nested data structures is automatically handled by "json.dumps".

Since you are concerned about it being "not so easy" for "more complex strings", it makes me wonder if you are trying to have people generate the strings?

So is your concern for manually generating complex strings, or for converting the existing repository of saved strings in non-JSON format, the extra characters required to call "json.dumps( ...., ensure_ascii=False)", or something else?

An existing repository can obviously be converted with a Python program using literal_eval on the input, and json.dumps on the output, to convert the data... of course if the repository is extremely large, or extremely distributed, that can be a complex issue, too.

By the way, "threats" such as your bold text above are seldom effective... an exposition of the technical issues, and reasons why the first proffered alternative of JSON format data is unacceptable would are much more effective.  The "threat" to not use Brython sounds an awful lot like a little kid crying and saying that he is going to take his blocks and go home. Brython isn't, and never will be, the solution for all people, and all problems. A clear explanation of some concerns regarding some limitation might inspire a solution to that limitation, but if it doesn't, then you (and anyone that faces such a limitation) has 3 choices... (1) code the solution themselves, or hire someone to code the solution (2) use a different system that doesn't have the limitation [but may have other limitations] (3) wait for more responses that might be more closely aligned to your needs or (4) give up on solving the problem. Threatening to not use Brython before even a single day has passed without a response that you like doesn't seem like an effective technique.  It seems likely that the exact solution you are looking for doesn't presently exist, but there may be a solution somewhere in the bowels of Brython that may solve your problem without forcing a conversion to JSON format.... none of the current responders apparently know about it, though.  But the more detail you can provide about your needs, beyond "I want literal_eval", helps to understand the real problem, which may be solvable in ways not requiring literal_eval. Or may not. Time will tell.

[zd]

I am sorry, that I said that. The problem there is that I do not want to use JSON, becasue we are already using it is not so effective. Because every time we make a conversion something can go wrong and it happens for example with complex strings where are more of these quotes and double quotes sometimes with example down these and so on.

"""write something '{template}' here with "  """

And the problem there is it does not stop just here. We have custom functions for recursive dict reconstructions (automatic fixes) and checking and they are build upon AST and literal eval. That's why I was asking in first place about AST and not JSON.

So if i get it right, AST and literal_eval right now are not possible. Do you know what is needed to make it possible? I was looking on part of code and seems that some parts are from PyPy, but i guess that ast module from pypy is not so easy to start working on because it is quite different from (CPython implementation) and what  I know about architecture.

[Joao S. O. Bueno]

The matter is that "ast" is needed in cPython to parse Python code
itself - while in Brython other methods are used to translate python
to javascript that make the Python AST prescindeble at this time.

If there was a Python implementation of AST's it might just work as is
in Brython right now - but it is implemented in C, instead. Maybe
there is a Python "ast" module in Pypy? It cold possibly e made to
work as is in Brython. Apparently. there are no Brython contributor
resources to check for that right now.

So, while one of Brythpon's ains is to have most of the stdlib
available client side, that goal is not reached, and theast module is
not a priority.


If your input data is trusted - i.e. not genereated by 3rd parties,
you could simply use Brython's eval -
it will be no different from AST in that case. If your data is machine
generated (i.e. not typed by humans, but formed
by fecthign data at some source - escaping it to properly conform to
json should not be an issue.

Otherwise, if writting Python data structures is preferred that much,
an option is to take that as a string in Brython, use an Ajax call to
a server-side Python application that can run native Python's AST and
return porperly encoded json to use in your Brython application -
maybe that is the easiest way to go, since, onece this is in polace
you could work around other Brython bumps using this technique.

I could point you in the right directions if you opt for this last option.

> --
> You received this message because you are subscribed to the Google Groups
> "brython" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to brython+u...@googlegroups.com.
> To post to this group, send email to bry...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/brython/8bfeb451-b2cc-45a3-8ee4-0d6862bcfd18%40googlegroups.com.

[Glenn Linderman]

On 3/10/2015 10:36 PM, zd wrote:

Because every time we make a conversion something can go wrong

I'm still trying to grasp what kinds of manipulations are being done on the string form of the data, and why, to comprehend the issues.

The rules for Python literals allow for multiple forms of quoting:  '  "  '''  """  which is convenient manually for writing data structures in source code, but once a properly formed data structure exists in Python, the algorithms for converting it to a string-form Python Literal (and back) and to a string-form JSON Literal (and back) are well-known.  I don't know that anyone has written code to directly convert one to the other, other than by loading one form into a Python data structure and writing out the other form, but while perhaps a bit tricky (two sets of rules to conform to), it would be possible.

But the talk of "reconstructions" and "automatic fixes" and "checking" that you make is somewhat baffling.  String form data structures are primarily useful for storage and retrieval, or for transmission between cooperating programs.  Neither use case would seem to need "reconstruction".

If you have a complex, nested data structure in string form as you keep mentioning, my question is how and why is it getting corrupted such that "reconstruction" is necessary?

If manual editing of the string form is occurring, and corrupting the structure of the data, that would seem to be a very error-prone technique for an operational system.  If manual editing of data values within a string form representation is required, would it not be better to code an "data structure editor" Python program for the data structure?  Load it in to Python, allow browsing of the structure, manual identification of the data value to update, allow updating of the value via some interface, and then rewrite the string form data structure as a total unit?  This would seem to be much less error prone than manual editing with a text editor of the string form data structure, and then heuristic attempts to "reconstruction" to correct whatever the person using the text editor screwed up by not knowing or not adhering to the correct rules for the string form data structure.

With a "data structure editor", there would be no need for people to even understand the string form data structure, and except for about two lines (load and dump, literal_eval and print or pprint), there would be no concern about whether JSON or Python Literal format is used as the storage format, and there would be no need to worry about whether there is only one string literal delimiter (JSON) or if there are 4 different string literal delimiters (Python Literal).

[zd]

Thank you for suggestions, right now this is probably solved. Together with team we agreed that AST is the most important thing for us right now. We will give it shot to see what would be needed to work with PYPY version or if it is possible by some easy hacks. Anyway,  thanks for a lot of information.