Segmentation questions

[Vadim Skubilin]

Hi all.

I'm looking improvements in my Kdb+ usage and I have question about segmentation.

Can segment name be used in query?

Eg we have X segments each Y rows and query require data only from one segment(we know from which).

Will kdb be looking data in all segments and it will X times slower than can be(if one thread used)?

Or we can use all information we know and search only in necessary segment?

[mark....@aquaq.co.uk]

Hi Vadim,

There was a discussion about this (or at least a similar issue) on the k4 listbox a few months back. The solution
is to just drop in the full path of the segment into your query. You will need to build a dictionary
that can map your query inputs to the correct segments.

e.g.

Suppose that you had some trades segmented by symbol range (A-M and N-Z) and then partitioned by date.

Instead of running a standard query like:

    select from trades where date=2015.01.07, sym=`AAPL

you could run something along the lines of:

    select from `$":/AtoM/2015.01.07/trades" where sym=`AAPL

The lookup of the segment could then be wrapped up into a function so that you could write something like:

    select from findsegment[2015.01.07;`AAPL;`trades] where sym=`AAPL

Credit to Arthur Whitney and Pavel Hejbal on the k4 listbox for the solution!
http://www.listbox.com/member/archive/1080/2014/10/sort/time_rev/page/5/entry/17:136/

Thanks

--
Mark Rooney
Financial Software Developer
AQUAQ Analytics

[Vadim Skubilin]

Thank Mark, it's work (even w/o converting to sym `$"...").

Not good that we can not use partition column in query, but it better than nothing.

[Vadim Skubilin]

1)I just curious if can I append string to text file without rewriting it fully?

                q.Sync("SymPathToString:{[SymPath]\r\n" +

                        "String: string SymPath;\r\n" +

                        "String: (1- count String)#String};");

                q.Sync("saveSegment:{[SegmentName;hdbdir]\r\n" +

                        "SegmentPath: ` sv (hdbdir;`dbdata;SegmentName);\r\n" +

                        "if[((.Q.P?SegmentPath)=(count .Q.P))=1b;ParPath: ` sv (hdbdir;`dbhead;`par.txt);ParPath 0: SymPathToString each (.Q.P,SegmentPath)]};");

[Sean O'Hagan]

> can I append string to text file without rewriting it fully?

yes, certainly.

http://code.kx.com/wiki/Reference/hopen#File

[Vadim Skubilin]

Thank, it's look better.

Now I see strange :: after using my func for list of elements and no see them when use func once per time.

                q.Sync("SymbolPathToString:{[SymbolPath]\r\n" +
                        "String: string SymbolPath;\r\n" +

                        "String: (1- count String)#String};");

                q.Sync("SaveSegment:{[SegmentName;hdbdir]\r\n" +

                        "SegmentPath: ` sv (hdbdir;`dbdata;SegmentName);\r\n" +

                        "if[((.Q.P?SegmentPath)=(count .Q.P))=1b;ParPath: ` sv (hdbdir;`dbhead;`par.txt); file: hopen ParPath; file (SymbolPathToString SegmentPath), \"\n\";hclose file]};");

[Meredith Grieve]

Hi Vadim,

That is expected behaviour when the last line of your function has a semi-colon at the end (generic null is returned).

Regards, 

Meredith Grieve 

Financial software developer

AquaQ Analytics

q)/ -returns a generic null (can't see it on the command line)
q){x+1;} 2
q)/ -but the output is a generic null
q)(::) ~ {x+1;} 2
1b
q)/ -shows a list of generic nulls if you run function on a list using an adverb

q) {x+1;} each til 4
::
::
::
:: 

[Vadim Skubilin]

One simple question about functions syntax:

For example I have func:

f: {[arg1;arg2] 1 (string arg1), " ", (string arg2), "\n"}

and table :

table: ([] col1: `foo1`foo2`foo3; col2: 100,101,102)

How can I use each row of table on my func?

for single value vector I simple use:

f[`foo;] each exec col2 from table

but exec col1,col2 from table give me dictionary and I somehow should associate arg1, arg2 with col1,col2 of this dictionary.

[Glen Smith]

Hey Vadim,

Try this... f'[table`col1;table`col2]

--
Glen Smith

Financial Software Developer
AQUAQ Analytics

[Vadim Skubilin]

Can it be hided? 

anyway I'll better replace :: to my output text -)

[Vadim Skubilin]

It got error, because table`col1 and table`col2 is vectors.

[Glen Smith@AquaQ]

q)table:([] col1: `foo1`foo2`foo3; col2: 100,101,102)

q)f: {[arg1;arg2] 1 (string arg1), " ", (string arg2), "\n"}

q)f'[table`col1;table`col2]

foo1 100

foo2 101

foo3 102

1 1 1

I don't get an error. Did you leave the ' out?

--

Glen Smith

Financial Software Developer
AQUAQ Analytics

[Vadim Skubilin]

Oh yes, thanks, I missed '

[Vadim Skubilin]

Sadly that this approach works only for vectors.

table:([] col1: `foo1`foo2

`foo3; col2: 100,101,102)
f: {[arg1;arg2;arg3] 1 (string arg1), " ", (string arg2), " ", (string arg3), "\n"}
f'[`xfoo;table`col1;table`col2]
 'lenght

In my case only some of arguments will from tables and other will be constants.

[Glen Smith@AquaQ]

f[`xfoo]'[table`col1;table`col2]

--
Glen Smith

Financial Software Developer
AQUAQ Analytics

[Vadim Skubilin]

Not sure that it works:

[Glen Smith@AquaQ]

Your f function is incorrect

It should be f: {[arg1;arg2;arg3] 1 (string arg1), " ", (string arg2), " ", (string arg3), "\n"}

not It should be f: {[arg1;arg2;arg3] 1 (string arg1), " ", (string arg2), " ", (string arg3, "\n")}

--
Glen Smith
Financial Software Developer
AQUAQ Analytics

[Charles Skelton]

or just use -1 instead and don't bother with that new line append.

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[Vadim Skubilin]

Thanks, my fault.

Sometimes is too hard to understand K syntax and not make any errors. 

[Charles Skelton]

Mark's links to the k4 listbox appear to be enticing non-licensed users to request access, so to clarify -

Please note that the kdb+/k4 listbox is a forum for licensed customers only. 

If you work at a company that has licensed kdb+, you may request to join the kdb+ Listbox by submitting your company email address via the link in the kdb+ listbox section at

http://kx.com/community.php

Private email address (gmail, yahoo, hotmail etc) will not be considered for membership.

Non-licensed users should continue to use this google group.

thanks

--

[Vadim Skubilin]

In my func:

q.Sync("SaveSegment:{[SegmentName;hdbdir]\r\n" +
                        "SegmentPath: ` sv (hdbdir;`dbdata;SegmentName);\r\n" +

                        "if[((.Q.P?SegmentPath)=(count .Q.P))=1b;ParPath: ` sv (hdbdir;`dbhead;`par.txt); file: hopen ParPath; file (SymbolPathToString SegmentPath), \"\n\";hclose file]};");

I'm checking that my segment not exist in .Q.P, but when DB is empty, there is not .Q.P and I got error.

How can I check this?

For usual variables, I can check them by (system"v")?`foo, but .Q.P isn't system variable...

[Vadim Skubilin]

It strange but 

\l .

works alright and

system"l" .

not works, I receive: 

.,["\\"].["l"]

and my \cd:

"c:\\db\\db2.5\\dbhead"

PS how can I reload my DB by other way?
I know path but don't know how convert symbol/string to directory type.

[Sean O'Hagan]

everything must be in a string for the system command to execute...

system"l ", string PATH

@[{system"l ",string x};PATH;`error]

value "\\l ",string PATH

http://code.kx.com/wiki/Reference/SystemCommands#.5Cl_name_-_load_file_or_directory

http://code.kx.com/wiki/Reference/SystemCommands#System_Commands

http://code.kx.com/wiki/Startingkdbplus/hdb#5.3_Sample_Segmented_Database

HTH,

Sean

[Vadim Skubilin]

Thanks, it helps.

system"l ",system"cd"