CSV file, list from csv table failure

[Chris Ward]

Hi All, I know I'm doing something wrong and it's probably right under my nose but I can't see it :)

File load works fine, but then [list from csv table] fails:

Syntax Error: quote in unquoted cell

Cannot parse text argument to "list from csv table" as a CSV-formatted table

There isn't a quote character in any cell.........

I have decorated the file to turn it into a CSV for AI2 - in the real world, commas are not used as separators for these files,  vertical bars are used. So in fact I'm only interested in Column 1, I added Column 2 to see if it made a positive difference (nope).

apk file removed, see the posting guidelines! Boban

[Ghica]

It beats me why this happens.
Since your original file is supposed to have only one column, you could use this the block shown.

Additional advantage: The elements of the resulting list are text, not lists with one element.

Cheers, Ghica.

[Evan Patton]

Thanks Chris for the report.

I took a look and it appears that the myfile.txt has a byte order mark at the beginning that is causing the problem. After I stripped out the BOM, the operation behaved as expected. I've filed this as an issue for us to look into handling the BOM in text loaded from files.

Regards,

Evan

[Chris Ward]

Hi Evan

I should have realised that myself (really need emoticons for this forum!).

Food for thought - either in addition to the functions dedicated to CSV or as an upgrade, it should be possible to specify any cell delimiter. In addition to Commas, Vertical Bars and Tabs are in common use but there are others where the file is specific to an industry or machine/device, including delimiters consisting of Unicode symbols or more than one character. Of course AI2 can handle such files now, but the 'list from file table' function is very elegant.

[Chris Ward]

Hi Ghica

Yes in fact I have been working on being more compatible with the files received, rather than decorating the files to suit the AI functions. Hit a few snags, all of which I think are due to my inexperience with AI, but generally a lot of problems could be solved if AI had a FileReadString function so that lines (rows) could be read one-at-a-time in a loop. It's often the case that engineering information has been written to file with that expectation, so one or more lines are actually headers giving information about the data in subsequent lines.

[Ghica]

Hi Chris,
Before you settle for an awkward solution like you are planning to do, with or without BOM markers, maybe you should investigate other options.
- How big is your file when the app is in actual use?
- How often are there updates to the data you need to process?
- Do you need to share the data with other users of the app?

As answers to these questions you could explore other options:
- Use the spreadsheet as-is as Google spreadsheet, you can query the contents with simplified SQL quite easily, update it from multiple sources, etc. but you need internet connection. Seatch puravidaapps for more information.
- Import your data into a TinyWebDB, or a Fusion table, but of course web access required
- Use Firebase. In a limited way you can also use it offline. It works much like a TinyDB.
- If your data is purely local, you could, on your PC, convert it to an SQLite database and import it using Taifun’s SQLite extension.

Without knowing more, I cannot advise you. I agree with your conclusions about desirable functional enhancements to AI though.
Cheers, Ghica

[Ghica]

Hi Evan,
Yes, of course! The infamous BOM-marker as ABG calls it. Search for BOM in the forum and you will see that there were quite a few people having problems with it. It would be nice if this bug could be solved.
Cheers, Ghica.

[Evan Patton]

Hi Ghica,

I think the real challenge with this is that in some cases one might want to preserve the BOM, but without the user specifying it's impossible to know. We might have to make it the case that the BOM will always be stripped out and we will convert the text to UTF-8 representation. Of course, any change might break people who are already working around the presence of the BOM.

Cheers,

Evan

[Ghica]

Hi Evan,
For all questions I have seen, the person who asked had no clue what a BOM was (In my country everybody would think I am talking about an explosive).
A while ago I had a problem with it, and I stripped those using notepad++. The texts say the marker is actually useless for UTF-8, but some software seems to depend on it.
As these markers give problems that are not understandable to most people now in AI, (and how could you get rid of them within AI?), it is probably better to strip them.
Cheers, Ghica.

[Evan Patton]

Thanks for your feedback, Ghica. I think you would be able to strip them using the segment operator to skip the first few bytes, but one would need to know in advance how long the BOM is. I agree the reasonable thing for us to do is to also strip the BOM since most users won't be aware of it.

Evan

[Chris Ward]

Yes indeed. Probably the most-used BOM is for UTF-8, and the reason for that is to distinguish the file text from plain ASCII, which remains a popular format. Since AI2 handles UTF-8 by default, stripping-out the BOM by default would be good in most cases, but perhaps that could be an additional function or an option added to the existing function, to cover all horses......