Efficient `over` with a large state? (in-place state updates)

Skip to first unread message

My Phone

May 25, 2021, 3:46:07 PMMay 25
to Kdb+ Personal Developers
How can I efficiently iterate over some values while making small updates to a large state vector (in my actual application it's a dictionary).

For example, if I use `over` it's very slow because the entire state vector is copied at each iteration:
q)n:prd 7#10
q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
2108 402655360

But if I were to use a global state (what I'm trying to avoid), it's much more efficient:
q)\ts {xx[x]:rand 1f}'[100?n]
0 2912


Osvaldo Fuica

May 25, 2021, 4:16:38 PMMay 25
to personal...@googlegroups.com
Hi John, 

You can store your large state vector in the global state and then update it in place.

q)n:prd 7#10
q)`xx set n#1f
q)\ts @[;;:;rand 1f]/[`xx;100?n]
0 1872


You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/c4d24c96-ba6e-4bcc-8d7c-b0038bea8866n%40googlegroups.com.

Ajay Rathore

May 25, 2021, 4:49:08 PMMay 25
to personal...@googlegroups.com
No need to create the projection in this case

q)\ts @/[`xx;100?n;:;rand 1f]
0 1648



May 25, 2021, 4:49:14 PMMay 25
to Kdb+ Personal Developers
If you can generate the values you'd like to update to separately then it'd be easier to use replace as demonstrated in https://code.kx.com/q/ref/amend/#replacement

q)n:prd 7#10
q)v:n#1f  // the initial state of your vector
q)indx:100?n // the indices to update
q)nv:count[indx]?1f // new values
q)\ts @[v;indx;:;nv]
//11 134219232

Osvaldo Fuica

May 26, 2021, 12:42:08 AMMay 26
to personal...@googlegroups.com
q)\ts {@[x;y;:;rand 1f]}/[`xx;100?n]
0 2160
q)count distinct xx

Any other idea with over?


My Phone

May 28, 2021, 8:04:37 AMMay 28
to Kdb+ Personal Developers
Thanks for the suggestions, but I should have clarified two points earlier:

1. The actual calculation is path-dependent and non-trivial, so the solution should work for any arbitrary function of the current state and an element of the iterated vector.
2. I want to avoid updating globals because ultimately I want to be able to run this calculation many times in parallel with peach.  And AFAIK, it's not possible to use thread local globals directly from q.

Eoin Cunning

May 28, 2021, 10:20:24 AMMay 28
to personal...@googlegroups.com
I think you are not including the creation of the variable. when you make it a global. When you do your timings.
Making it hard to see what your really comparing and wanting to optimize.
Using global will reduce memory but dont think there is much difference in speed.

q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
1117 402655360
q)\ts xx:n#1f ; {x[y]:rand 1f; x}/[xx;100?n]
1168 268437936

Reply all
Reply to author
0 new messages