Beginner question: Importing CSV

[jaco erasmus]

I am trying to learn J, and basic descriptive stats, by writing verbs for the basic stats calculations, and applying them to vectors.

I have a simple CSV file: header line for names and totals, then rows with the relevant data.

I am able to import the csv file, and also import the file with my verbs.

Now, I want to get the values in the second column into a vector so that I can run my verbs on them, get the answers I need and smile because J isn't that difficult after all...

Here be dragons.

I think my biggest problem is that, rather than reading "Learning J" or "J Primer" or "The Idiot's Guide To Working With CSV Files In J", I am asking Gemini to teach me what I need to know. Free Tier Gemini-cli. I know, I know, don't judge.

The first problem is boxes. Why is it so difficult to get the numbers from my CSV? Here's the code for that: 

data =. 1!:1 < 'C:\Users\Jaco\Documents\j_statistics_gemini_cli\gr9_inf_41.csv'

lines =. (CR,LF) cut data

data_lines =. }. lines

score_strings =. ({: @ (','&cut)@>)"1 data_lines

usable_data =. _1 }. score_strings

unboxed_scores =. > usable_data

That part seems to work. Read in the file, cut into individual lines, cut into cells and put the last cells into a noun, drop the empty box at the end, get rid of the boxes. Might have been easier if there were no boxes, but I'm sure there is a reason for them. At least it let me see the final, open box.

My last line, for now, is this:

numeric_scores =. ". unboxed_scores

which promptly throws an error:

|spelling error: scriptd (invalid character in sentence, code unit 13)

|      9

|      ^

| numeric_scores=. ".unboxed_scores

|[-20] c:\users\jaco\documents\j_statistics_gemini_cli\gr9_stats.ijs

My LLM overlord is just as flummoxed as I am.

Any ideas? I don't know too much about programming, so go slowly. I know it could probably be done much more easily in python, but where is the f(rustration)un in that?

Thanks

[jaco erasmus]

I forgot to mention, the first score in my CSV file is 9, so there's something there, but I don't know what.

[Clifford Reiter]

Does your data still have CR or LF in it?

".LF,'9'

|spelling error (invalid character in sentence, code unit 10)

|

9

| ^

| ".LF,'9'

You could try numeric_scores=. ".unboxed_scores-.CR,LF

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@jsoftware.com.

[Henry Rich]

The message says that it is trying to execute a sentence containing ASCII character 13, which is CR.  Cliff's fix works for that. 

Tip: when you execute

>array

it usually indicates a design error, because if you didn't need the boxing why did you have it in the first place?  When you open the array you will be adding fill to the lines to bring them to a common length. That's harmless here, but you could avoid it with

".@> array 

which execute the lines one by one and assembles the results into an array. You would need to have removed the CRLF earlier, or you could use

".@(-.&CRLF)@> array

Henry Rich

[jaco erasmus]

Thank you Henry and Clifford for the help.

First, I tried Clifford's suggestion. I had a line: numeric_scores =. ".unboxed_scores-.CR,LF but that gave another error:

|spelling error: scriptd (invalid character in sentence, code unit 13)

| 9

| ^

| numeric_scores=. ".unboxed_scores-.CR,LF

I then tried Henry's idea: numeric_scores =. ".@(-.&CRLF) @ > unboxed_scores (which I think means something like "unbox unboxed_scores, and then remove? CRLF, and then convert to numbers). This gave me an output with everything in triads: 900, 100, 800, etc.

I could calculate the mean and the median with my defined verbs and I got answers, albeit weird ones: mean numeric_scores

5.69231

0

0

However, variance doesn't work now, because of some shape issue: variance numeric_scores

|length error in variance, executing dyad -

|shapes 13 3 1 and 3 1 do not conform

| (+/(y-m)*(y -m))%#y

I think it might be time to read a book!

Henry, you asked that, if I did not need boxing, why did I use it in the first place. Is there another, better way of doing it?

Again, thank you both for the kind help.

Jaco

[Henry Rich]

You are lacking some basic concepts, notably verb rank.  I recommend you read the NuVoc articles for beginners rather than a book. Look in the ancillary pages at the bottom of the NuVoc screen. 

Henry Rich

[jaco erasmus]

Shall give it a look, thank you!

[More Rice]

> I had a line:

>  numeric_scores =. ".unboxed_scores-.CR,LF

> but that gave another error:

I think just missing “1 ; i.e., 

numeric_scores =. ".unboxed_scores-.”1 CR,LF

—

To start, it may be easier to use csv package. You’re trying to do math on the last “numeric” column after removing its header?  So, something like:

   load’csv’

   domath =. +/%#

   domath “. > {:”1 }. readcsv ‘path/to/test.csv’

Sent from mobile

[jaco erasmus]

Thank you! Using load'csv' worked!

So I can do basic stats now. I guess the next step will be to figure out rank. And boxes.

(I don't know the etiquette of this forum: whether or not I should have a big SOLVED! at the bottom of this post, but that it is.)

Thanks again, More Rice, and everyone else.