output csv files

[Gijs Dekkers]

Dear all,

In many cases, when we create a csv file, we would like to have the same command in the header.

For example

            immigration_output_init():            procedure name, called in the init phase of the model
                - csv('period',
                      'avg(nationality_ext == 0) * 100',
                      'avg(nationality_ext == 1) * 100',
                      'avg(nationality_ext == 0, filter=year_registered == period) * 100',
                      'avg(nationality_ext == 1, filter=year_registered == period) * 100',

                      fname='output_immigration_2.csv')

in the init phase, and

            immigration_output():            procedurename, called in the body of the model
                - csv(period,
                      avg(nationality_ext == 0) * 100,
                      avg(nationality_ext == 1) * 100,
                      avg(nationality_ext == 0, filter=year_registered == period) * 100,
                      avg(nationality_ext == 1, filter=year_registered == period) * 100,
                      fname='output_immigration_2.csv', mode='a')
in the actual model.

This seems redundant. Would it be possible to have a command csv(a, b, fname='', mode)

            immigration_output(parameter):        procedurename, which can be called EITHER in the init phase OR in the body of the model
        - csv(argument2, argument2, fname='', mode)

with position parameter
    either  mode='w'    as currently
        mode='a'    as currently
        mode=parameter        if parameter=value1 then argument1 and argument2 are put between single brackets, meaning that the arguments themselves are written to the csv file, and mode='w' is used
                    if parameter=value2 then the result of argument1 and argument2 are written tot the csv file, and mode='a' is used

[Gaëtan de Menten]

Hi Gijs,

 

You can do this in two (and a half) different ways:

 

Option 1:

------------

The simplest way is to initialize an empty csv file (or better yet with column headers) in your "_init" function and then use both the _init function and the "normal" output function in your list of init processes (see attached output_test1.yml).

 

But I thought you already used this technique??? In fact, being able to do this (reuse some functions in both init and processes list) is the reason why I changed very early in the history of LIAM2 (before 0.1 IIRC) from a specific "init_processes" block in the entity to a list in the simulation.

 

Option 2

------------

This is almost what you are suggesting. The only trick is to realize that LIAM2 does not support passing arguments in the "simulation process list", so you either have to define intermediate functions for each different value of the argument (see output_test2.yml), or transfer the process list itself to a function (see output_test3.yml).

 

I haven't seen anybody use that later possibility yet. It is more powerful than the simple process list because you can use model inheritance, loops and pass arguments to functions, but you need to make one such "driving" function for both init and normal processes and you cannot alternate processes from multiple entities in there, so if you need alternating entities, you would need several of those functions. In short, YMMV but it could be worth it in some cases.

 

Hope it helps,

Gaëtan

 

--
You received this message because you are subscribed to the Google Groups "liam2-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liam2-users...@googlegroups.com.
To post to this group, send email to liam2...@googlegroups.com.
Visit this group at https://groups.google.com/group/liam2-users.

Disclaimer: This e-mail may contain confidential information which is intended only for the use of the recipient(s) named above.
If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.
Please note that e-mail messages cannot be considered as official information from the Federal Planning Bureau.

[Gijs Dekkers]

Dear Gaëtan,

A short reaction on your option 1. There is some confusion here; clearly I did not explain my problem correctly. I indeed want to have the output like in example1 (which we indeed use very regularly)

So "csv('period', 'avg(age)', 'sum(gender)', fname='test1.csv')" in the init phase to provide the header, and

csv(period, avg(age), sum(gender), fname='test1.csv', mode='a') in the actual model

BUT...

I would like to have LIAM2 automatically add the single brackets ' whatever' in the init-phasse (mode='w' in your example 3) but NOT in the body of the model (mode='a' in your example ").

The result would be that I would need just one line

output_test(mode):
                - csv(period, avg(age), sum(gender), fname='test3.csv', mode=mode)

And it would be interpreted as - csv('period', 'avg(age)', 'sum(gender)', fname='test3.csv', mode=w) in the init phase, and

- csv(period, avg(age), sum(gender), fname='test3.csv', mode=a) in the body of the model..

I hope this is clearer now. Thanks!

G

Op donderdag 23 augustus 2018 08:21:00 UTC+2 schreef Gijs Dekkers:

[Gaëtan de Menten]

Oops. I completely misread what you wrote. I missed the quotes in immigration_output_init in your original message. Sorry about this.

 

Now I understand what you would like to do. You would like something a bit similar to qshow() but for csv files which are generated one line at a time. I don't think there is any way to do that with the current version of LIAM2. This is a nice idea though and I could implement something like that but I have to think about it some more to determine what's the best way to provide that functionality. In any case, I don't like reusing the mode argument and whether the function is called in the init phase or not because that would prevent some useful combinations.

I would rather implement an extra argument autoheaders=True or expr_as_headers or something like that. It would be a bit more tedious than what you propose but more powerful.

 

I have opened an issue to track this: https://github.com/liam2/liam2/issues/290

 

Gaëtan

 

From: liam2...@googlegroups.com <liam2...@googlegroups.com> On Behalf Of Gijs Dekkers
Sent: Friday, August 24, 2018 15:42
To: liam2-users <liam2...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "liam2-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liam2-users...@googlegroups.com.
To post to this group, send email to liam2...@googlegroups.com.
Visit this group at https://groups.google.com/group/liam2-users.