ArrayCast Episode 124: J 9.7 with Henry Rich & Marcin Żołek

[Eric Iverson]

A new array cast episode starts in about 30 minutes.

This is a video and should not be missed. Catch it live or later!

https://www.youtube.com/watch?v=8MYdCF4za

[Yuvaraj Athur Raghuvir]

Hello,

Good presentation. Thanks. Leaned a lot.

After watching the ArrayCast I am trying to reproduce thread behaviors in my Win 11 J9.7 install. I don't see general/threads in my package manager. I believe this is causing errors.

Is general/threads addon generally available? How can I install and test threads addon?

Thanks,

Yuva

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@jsoftware.com.

[Henry Rich]

You don't need an addon for threading.  Any 64-bit system that supports pthreads can use the task features.

HOWEVER, when I recorded the ArrayCast I was using my working version of J, which included a shorthand not in the 9.7.1 release.  My bad.

To reproduce the results in the ArrayCast, you can:

* most easily, install 9.8.0-beta2, which is what I was using

OR

* replace the assignment of allthreads with the 2 lines

coinsert 'base' [ cocurrent loc =: 0 ". > cocreate''
allthreads =: t. (< 'locales' ;< 16bff ; 8 # loc )

In 9.7 the 'locales' keyword requires a locale-list as above.  9.8 allows it to be omitted.  Same function.

Henry Rich

PS. Note that if you have a Mac the line setting 'coremask' will fail, but you can run without it.

[Yuvaraj Athur Raghuvir]

<Please move this thread to appropriate channel if required>

JVERSION

Engine: j9.8.0-beta2/j64avx2/windows

Build: commercial/2026-05-08T16:41:51/clang-19-1-5/SLEEF=1

Library: 9.8.2

Qt IDE: 2.6.3/6.10.3(6.10.3)

OS Ver: Windows 11 Version 25H2 10.0.26200

Platform: Win 64

Installer: j9.8 install

InstallPath: d:/programs/j9.8

Contact: www.jsoftware.com

]'corecount maxthreads' =: 8 T. ''

16 63

Sample Code [1] runs with one line commented out:

NB. anchor 'resarea' NB. anchor it

Follow-up questions:

1/ allthreads=: t. (< 'locales' ; 16bff) What value should I pass for running on 16 cores?

2/ Domain error - how can I fix it? What is the consequence of not fixing resarea?

 anchor 'resarea'
|domain error in anchor, executing monad 1&(15!:24)
|       anchor'resarea'
|[-1]

3/ All smoutput calls are buffered and written after the program exits. How can I get J to output progress as the execution proceeds?

4/ What is a pyx that is referenced in the https://code.jsoftware.com/wiki/Vocabulary/tcapdot wiki?

Thanks,

Yuva

[1] Handwritten using image from video - may have errors

NB. Library definitions
smoutput 'Library functions'

amv =: 16&T. NB. Create AMV with starting value y
NB. AMV stands for Atomically Modifiable Value (AMV)

amvadd =: 17 T. (,<) NB. add y to AMV x, return old x

delth =: {{while. 1 T. '' do. 55 T. '' end. EMPTY }} NB. delete all worker threads anchor=1&(15!:24)
anchor =: 1&(15!:24) NB. Indicate that y~ is not to be moved or aliased
ts =: 6!:2 NB. time sentence

NB. Machine-dependent one-time thread initialization
22 T. 0;1 NB. Run the master thread in core 0

NB. Clear threads, then allocate one thread for each P core, plus 4 others
(0 T. [: < 'coremask';2&^)"0 (2 4 6 8 10 12 14) [ delth''

NB. coremask is not supported for Mac or OpenBSD

NB. Adverb to run on 8 cores
allthreads=: t. (< 'locales' ; 16bff)


NB. The code:
resarea =: 1000$00 NB. Allocate result area
NB. anchor 'resarea' NB. anchor it

smoutput 'Test functions'
NB. y is number, result is # Collatz iterations required to finish
collatz =: {{
t =. 0
while. 1 < y do.
 t=. >: t NB. increment loop count
 if. 2 | y do. NB. Test even/odd
  y=. 1 3 p. y NB. odd, replace y by 3y+1
  else. y =. y % 2x NB. even, replace y by y/2
 end.
end.
t}} NB. Return loop count when y goes to 1

NB. x is start of collatz interval, y is AMV of #values to calculate
NB. Repeatedly take a value, run collatz, store the result in (resarea)
NB. Result is empty

colint =: {{
 while. 0 <: i =. <: y amvadd _1 do. NB. Get next # to try in descending sequence
  resarea =: (collatz x+i) i} resarea NB. store its Collatz count into the result
 end.
''}} NB. return empty


NB. Single thread compute
smoutput 'Single Thread Compute'
smoutput ts '(3x ^ 10) colint amv # resarea' NB. init # results and run
NB. x in 3x stands for extended precision

smoutput ts '(3x ^ 1000) colint amv # resarea' NB. init # results and run
smoutput +/resarea

NB. Multi-thread compute
smoutput 'Multi-Thread Compute'
smoutput ts '(3x ^ 1000) colint allthreads amv # resarea' NB. init # results and run

[Henry Rich]

2. The code runs as given.  It fails when you /rerun/ it.  I didn't want to add extra stuff to the video.

It is vital that you anchor (resarea) so that when the different threads run they will be able to modify the noun concurrently.  But, once you anchor it, you must not try to move it.  When you reload the script, the reassignment fails because it is reassigning the anchored noun but not in place.

To allow rerunning the script, precede the assignment to (resarea) with

erase 'resarea'

1. Use 16bffff for 16 threads.  You must then change the line that executes (0 T.) to create 15 worker threads.  The coremask there is correct for my machine, which has 4 hyperthreaded cores.  If you don't want to trouble looking up your core details you could create 15 worker threads with

0 T."1 i. 15 0 [ delth''

and let the OS assign them to cores.

3. use (wd 'msgs') to emit queued messages.  Example:
6!:3 (2) [ wd 'msgs' [ smoutput 'xxx'

Formatting and printing messages may affect your timings.

4. See https://code.jsoftware.com/wiki/Vocabulary/tdot


Timings when multiple threads are running depend on system load and clock rate.  CPUs reduce the clock rate when you try to do more, unless you have a cooler.  The JE adds very little overhead for threading.

Henry Rich

[Yuvaraj Athur Raghuvir]

Ack. I am trying to understand JE threading and I have a few more questions. 

In the explanation above, I read this:

If you don't want to trouble looking up your core details you could create 15 worker threads with

0 T."1 i. 15 0 [ delth''

and let the OS assign them to cores.

Here is what I get on my machine

22 T.0

65535

1/ Does this mean 'coremask' on my machine is 65535?

2/ What is the following expression trying to do? How to understand 'coremask' in this expression?

(0 T. [: < 'coremask';2&^)"0 (2 4 6 8 10 12 14) [ delth''

3/ And how is this expression a simplification of the above for 16 cores?

0 T."1 i. 15 0 [ delth''

[Henry Rich]

1. Yes.  But what that fact means depends on your OS's interpretation of the coremask.  That may be OS-dependent.

2. The sentence creates 7 threads with each thread having the indicated bit# in its coremask set.  On my machine, the P-cores are hyperthreaded and get 2 bits of coremask each, while the E-cores get 1 bit each.  I chose the mask to use only one thread per P-core, and to separate the E-cores to try to use different D2$.

3. The sentence creates 15 threads, letting the OS manage which cores they run on.  The coremask is not used.

Henry Rich

[Yuvaraj Athur Raghuvir]

Thanks—with the proposed changes I can run and re-run these lines. And emit queued messages to monitor progress. 

I want to use threads to run ffmeg.exe as an external program. I see a way to use addons or dlls. It is not clear to me how to spawn threads to run external programs on Windows. 

Note: As per https://code.jsoftware.com/wiki/Vocabulary/Foreigns 2!:n foreigns are Unix only. Is there a plan to implement them in Windows now that we have Threads and Tasks?

[Henry Rich]

NO current plan.

Henry Rich

[Yuvaraj Athur Raghuvir]

Ack.

If 2!:n foreigns are not planned for Windows is there a way to call external programs (*.exe) using command line string from J in Windows? 

[Eric Iverson]

Yuvaraj,

Not sure what you are looking for, but does this help:

   load'jtask'

   shell_jtask_'....' NB. run windows command

[Yuvaraj Athur Raghuvir]

Yes, this is the clue I am looking for to run media transformation programs like ffmpeg from my J script. 

 shell_jtask_'....' NB. run windows command

Thanks.