Efficient `over` with a large state? (in-place state updates)

[My Phone]

How can I efficiently iterate over some values while making small updates to a large state vector (in my actual application it's a dictionary).

For example, if I use `over` it's very slow because the entire state vector is copied at each iteration:
q)n:prd 7#10

q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]

2108 402655360

But if I were to use a global state (what I'm trying to avoid), it's much more efficient:
q)xx:n#1f

q)\ts {xx[x]:rand 1f}'[100?n]

0 2912

Thanks,

John

[Osvaldo Fuica]

Hi John, 

You can store your large state vector in the global state and then update it in place.

q)n:prd 7#10

q)`xx set n#1f

q)\ts @[;;:;rand 1f]/[`xx;100?n]

0 1872

Best, 

Osvaldo

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/c4d24c96-ba6e-4bcc-8d7c-b0038bea8866n%40googlegroups.com.

[Ajay Rathore]

No need to create the projection in this case

q)\ts @/[`xx;100?n;:;rand 1f]

0 1648

-Ajay

To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/CAG4Kp0R4k%3DF9cdotRWfdpnfVPP%3DJCFk5QR1BCzkJFSoTF8n_Pg%40mail.gmail.com.

[jfsaw...@gmail.com]

If you can generate the values you'd like to update to separately then it'd be easier to use replace as demonstrated in https://code.kx.com/q/ref/amend/#replacement

q)n:prd 7#10

q)v:n#1f  // the initial state of your vector

q)indx:100?n // the indices to update

q)nv:count[indx]?1f // new values
q)\ts @[v;indx;:;nv]
//11 134219232

[Osvaldo Fuica]

q)\ts {@[x;y;:;rand 1f]}/[`xx;100?n]

0 2160

q)count distinct xx

101

Any other idea with over?

Osvaldo

To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/c78fac4e-7732-4586-a9b9-b57bfd9bdc9bn%40googlegroups.com.

[My Phone]

Thanks for the suggestions, but I should have clarified two points earlier:

1. The actual calculation is path-dependent and non-trivial, so the solution should work for any arbitrary function of the current state and an element of the iterated vector.

2. I want to avoid updating globals because ultimately I want to be able to run this calculation many times in parallel with peach.  And AFAIK, it's not possible to use thread local globals directly from q.

[Eoin Cunning]

I think you are not including the creation of the variable. when you make it a global. When you do your timings.

Making it hard to see what your really comparing and wanting to optimize.

Using global will reduce memory but dont think there is much difference in speed.

q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]

1117 402655360
q)\ts xx:n#1f ; {x[y]:rand 1f; x}/[xx;100?n]
1168 268437936

To view this discussion on the web visit https://groups.google.com/d/msgid/personal-kdbplus/1e6e4393-031c-4a0d-ac75-b156c4f1ace4n%40googlegroups.com.