Singleton(s) and workers

[Boris Baldassari]

Hiho dear Mojo folks,

This is my first post here, so I'd like to start with a big thank you all for the work put on mojolicious. It just rocks. :-)

I am building a full (not light) app which needs access to a set of variables (say config) stored in a dedicated module (MyApp::Model::Config). These values do change during the run, and I want all the workers to access the new values. This works very well in morbo, but as soon as I start using hypnotoad the values are not updated for all workers. As a consequence, depending on the worker serving your request you get either old or new values randomly.

I started with a singleton helper put directly in the startup method:

$app->helper( repo => sub { state $config = MyApp::Model::Config->new($app) } );
This doesn't resolve the issue.

Then I read about similar problems with db connection handles, and I decided to move them out of the startup method and I added some has attributes, still in the MyApp.pl class:

has my_config => sub { state $config = MyApp::Model::Config->new($app); };

But it still doesn't work.

So, how should I do that? Any advice on the design or code itself is welcome...

Thanks in advance,

--
Boris

[Ben van Staveren]

That won't work - under Hypnotoad every worker would end up with it's own copy, and while you would be able to update values inside a single worker, it wouldn't work across the entire worker pool. Consider using memcache or another key/value store that's convenient (redis, perhaps) to store information like that, and fetch it when required.

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at http://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Boris Baldassari]

Hi Ben,

Thank you for the quick and to-the-point answer. I guess it won't work either with prefork instead of hypnotoad.

I'll have a look at memcache and redis solutions.

Thanks again -- I've spent some time on this, so it's good to have a confirmation/statement to settle the point.

Have a great day!

--
boris

[Jan Henning Thorsen]

Could you give some more details/examples on what kind of information that is changed during run? Also, how often does this information change?

Reason for asking, is that you might come up with a far too complicated solution for this.

In some cases, it would be enough to just change the config file and then hot reload hypnotoad, but this won't work if you're changing the config too often. On the other hand, reading data from memcache/redis/something is probably a better idea than trying to synchronise information internally between your workers.

[Boris Baldassari]

Hi Jan,

The app is a kind of dashboard for sw project assessment, and it maintains a set of data on software projects (metrics, meta-information, etc.). There is an analysis (i.e. an update on this data) started every day, plus occasional changes on user's initiative. So the information is not changing *heavily*.

My point is I *want* to maintain the data on disk as json files, because I believe that a data-based process (i.e. having a process transforming data between known states) is easier to read and understand for users and maintainers.
After giving some thoughts to memcache/redis/etc. I realise I would have to synchronise the data on disk every time it changes, which is quite cumbersome and not very reliable.

Maybe hot reloading hypnotoad would be the way to go. How could I do that from the code itself, apart from executing shell commands?

Another option would be to load the data at startup, and write it down to the disk on shutdown (through a END sub). Not sure about the reliability of such a solution though.

Any idea would be welcome. Thanks!

--
Boris

[Jan Henning Thorsen]

If it's not critical that changes in the file is visible instantly, you could do something like this:

  Mojo::IOLoop->recurring(

    30 => sub {

      app->config(dashboard => decode_json(slurp app->home->rel_file("dashboard.json")));

    }

  );

This will make the application read the file every 30 second and update the "dashboard" config parameter. The code could also check if the timestamp of the json file has changed and simply not read the content if mtime is the same.

Another idea is to use AnyEvent::Filesys::Notify instead of the recurring timer. (There might be similar solutions, but that was the one that came to mind first)

About hot reloading: If you are running both the workers and manager as the same user, then the worker can do initiate hot reload using "kill USR2 => getppid".

[Boris Baldassari]

Le 23/10/2015 12:22, Jan Henning Thorsen a écrit :
> If it's not critical that changes in the file is visible instantly, you
> could do something like this:

It's not critical if the changes appears after a few seconds, so the
idea is good, thank you -- and thanks for the code, which is helpful.

> This will make the application read the file every 30 second and update
> the "dashboard" config parameter. The code could also check if the
> timestamp of the json file has changed and simply not read the content
> if mtime is the same.

I didn't like the idea of checking a mtime each time I did a request to
the data -- which would be almost each time a user displays a page.
Feels like loosing the efficiency of having everything in memory.

> Another idea is to use AnyEvent::Filesys::Notify instead of the
> recurring timer. (There might be similar solutions, but that was the one
> that came to mind first)

Oh, I didn't know of that one. And really like it, for it respects the
data/file paradigm for understandability.
It may indeed be the best solution, among other things for portability.
I will give it a try, and report back to the list next week.
Really good advice, thanks!

> About hot reloading: If you are running both the workers and manager as
> the same user, then the worker can do initiate hot reload using "kill
> USR2 => getppid".

Yes, smart! Still GTK (Good To Know (tm)).

Thanks again,


--
Boris

>> =MyApp::Model::Config->new($app)});

>> |
>> This doesn't resolve the issue.
>>
>> Then I read about similar problems with db connection
>> handles, and I decided to move them out of the startup
>> method and I added some has attributes, still in the
>> MyApp.pl class:
>> |
>> has my_config =>sub{state $config

>> =MyApp::Model::Config->new($app);};

>> |
>> But it still doesn't work.
>>
>> So, how should I do that? Any advice on the design or
>> code itself is welcome...
>>
>> Thanks in advance,
>>
>> --
>> Boris
>> --
>> You received this message because you are subscribed
>> to the Google Groups "Mojolicious" group.
>> To unsubscribe from this group and stop receiving
>> emails from it, send an email to
>> mojolicious...@googlegroups.com.
>> To post to this group, send email to
>> mojol...@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/mojolicious

>> <http://groups.google.com/group/mojolicious>.

>> For more options, visit
>> https://groups.google.com/d/optout

>> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Mojolicious" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mojolicious/T9uH9a14CHM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> mojolicious...@googlegroups.com
> <mailto:mojolicious...@googlegroups.com>.

> To post to this group, send email to mojol...@googlegroups.com

> <mailto:mojol...@googlegroups.com>.

[Boris Baldassari]

Hi all,

I finally got it working using File::ChangeNofity instead of
AnyEvent::Filesys::Notify, since it needs less dependencies and is good
enough for my needs. The idea of watching file changes was a really good
one, thanks again to Jan.

I still have one question to finish that: how could I test the
thread-safe behaviour? When running the tests (e.g. with myapp.pl test)
I make the assumption that Mojolicious is using something like morbo,
that is a single-threaded version of the application. Is there any means
to either:
* test the app in a preforked context (by launching hypno or prefork
server instead of a dev one)
* tell the test run to use an existing web site instead of the current dev?

The last option would be the best, since it would really test the app in
a production context. But I could not find anywhere a way to tell the
test agent to look for a specific url (where my dev is hosted)... Is
there one?

Thanks,

--
Boris

[Jan Henning Thorsen]

Not sure what other results you might expect in a "prefork-environment", vs a single process.

>> mojolicious+unsubscribe@googlegroups.com
>> <mailto:mojolicious+unsub...@googlegroups.com>.

>> To post to this group, send email to mojol...@googlegroups.com

>> <mailto:mojolicious@googlegroups.com>.

[Boris Baldassari]

Hi Jan,

Le 31/10/2015 23:35, Jan Henning Thorsen a écrit :
> Not sure what other results you might expect in a
> "prefork-environment", vs a single process.

Typically a lack of consistency between workers' data.

Also, in the second case (i.e. specifying an url to run the tests) it
would allow to test the product in a different context/setup, and test
things like time-outs, or make sure the product has everything needed to
work, etc.

--
boris

>
> On Monday, October 26, 2015 at 2:11:03 PM UTC+1, Boris Baldassari wrote:
>
> Hi all,
>
> I finally got it working using File::ChangeNofity instead of
> AnyEvent::Filesys::Notify, since it needs less dependencies and is
> good
> enough for my needs. The idea of watching file changes was a
> really good
> one, thanks again to Jan.
>
> I still have one question to finish that: how could I test the
> thread-safe behaviour? When running the tests (e.g. with myapp.pl

> <http://myapp.pl> test)

> <https://groups.google.com/d/topic/mojolicious/T9uH9a14CHM/unsubscribe>.

>
> >> To unsubscribe from this group and all its topics, send an
> email to

> >> mojolicious...@googlegroups.com
> <mailto:mojolicious%2Bunsu...@googlegroups.com>
> >> <mailto:mojolicious...@googlegroups.com
> <mailto:mojolicious%2Bunsu...@googlegroups.com>>.

> >> To post to this group, send email to

> mojol...@googlegroups.com <mailto:mojol...@googlegroups.com>
> >> <mailto:mojol...@googlegroups.com
> <mailto:mojol...@googlegroups.com>>.

> >> Visit this group at http://groups.google.com/group/mojolicious
> <http://groups.google.com/group/mojolicious>.
> >> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Mojolicious" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mojolicious/T9uH9a14CHM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> mojolicious...@googlegroups.com
> <mailto:mojolicious...@googlegroups.com>.

> To post to this group, send email to mojol...@googlegroups.com

> <mailto:mojol...@googlegroups.com>.