Websocket process mailbox flood issue

[me...@lanyardmarket.com]

Hi

I have a game server working with tcp connection over cowboy websocket in my cb app. For the game updates in realtime, websocket receives a message from each user per every 16 ms. Mailbox start having bottleneck and queues all messages and make the game unplayable. ( for example when you shoot, it shoots after 15 - 30 secs.

Is there anyway in my cb app to handle a single websocket to get all the messages from users without mailbox having flooded? I searched a lot of things but couldn’t find a way.

Any help would be appreciated.

Thanks

[Konstantin Gorshkov]

Hi,

Usually the mailboxes are flooded due to slow handlers. Have you made sure that your handlers are fast enough?

--
You received this message because you are subscribed to the Google Groups "ChicagoBoss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chicagoboss...@googlegroups.com.
Visit this group at https://groups.google.com/group/chicagoboss.
To view this discussion on the web visit https://groups.google.com/d/msgid/chicagoboss/c7427c72-b011-40c4-9a0d-dec505190ea0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[me...@lanyardmarket.com]

Hi Konstantin,

Thanks for your answer. Here is my firtst handler code try below;

handle_incoming(_ServiceName, WebSocketId, Message, State) -> 

      

      {ok, {Type, Data}} = parse_message(Message),

      case Type of

        1 -> member_action:member_function(WebSocketId,Data);

        2 -> match_function:set_player_match(WebSocketId,Data);

        3 -> match_function:start_match(WebSocketId,Data);

        4 -> online_action:new_match_user(WebSocketId,Data);

        5 -> match_function:update(player,WebSocketId,Data);

        6 -> match_function:update(bullet,WebSocketId,Data);

        7 -> online_action:set_player_health(WebSocketId,Data);

        8 -> match_function:update_door(WebSocketId,Data);

        9 -> online_action:respawn(WebSocketId,Data);

        51 -> ad:execute(WebSocketId,Data)

      end,

{noreply, State}.

And also tried to create a gen_server and send message there;

handle_incoming(_ServiceName, WebSocketId, Message, State) -> 

      

  gen_server:call({global, bridge}, {distribute,WebSocketId,Message}),

{noreply, State}.

Theseare not changing any bit of performance and the mailbox of websocket is still having the same issue and game can not be played.

Really stuck here. Any advice would be great.

Thanks!

23 Kasım 2018 Cuma 07:54:13 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[me...@lanyardmarket.com]

Sorry forget to show parser code;

parse_message(Msg) ->

   DecodeMsg = jsx:decode(Msg),

   {<<"t">>, Type} = lists:keyfind(<<"t">>, 1, DecodeMsg),

   {<<"m">>, Content} = lists:keyfind(<<"m">>, 1, DecodeMsg),

   {ok, {Type, Content}}.

23 Kasım 2018 Cuma 10:12:22 UTC+2 tarihinde Lanyard Market yazdı:

[Konstantin]

It is difficult to give advice. But there is some general thoughts below:

Do the functions in the case clauses use access to any external database
such as MySQL? It may create a bottleneck.

Also json parsing may create a bottleneck. A simple lightweight binary
protocol will be better in your case. Protobuffers or your own for example.

[me...@lanyardmarket.com]

I think my answer didnt show up yesterday because used a different email. You are right about parsing the json. It can be faster.
I increased incoming message delay from 16ms to 60ms and reduce json packet sizes. Now a single match server can handle only the movements of users. If they also shoot, which comes to server seperately, websocket process floods again.
When 8 users walk in 3d space (it is a browser based fps game) they now send around 50-60 bytes of data every 60ms. I can see the mailbox flood after a certain amount of user adding.(logging mailbox queue also bottlenecks the process)
I thought cb can handle much more then this with a single websocket. If there is any different approach you can advice, i would be glad to listen.

Thanks!

[me...@lanyardmarket.com]

Also shared a video here with 10 online fake players except me moving and shooting at the same time

My goal is to make it 30 players per match if i can solve the bottleneck issue.

https://www.youtube.com/watch?v=IdyQxjht1mU&feature=youtu.be



24 Kasım 2018 Cumartesi 09:08:04 UTC+2 tarihinde Lanyard Market yazdı:

[jami...@gmail.com]

Thanks Konstantin. You are right about the parsing function of the json. I use mnesia and update functions are below;

update(player,WEBSOCKETID,PACK)->

  case fgen:db_hd(boss_db:find(jamir_match_data,[{match_id,'equals',lists:nth(1,PACK)}])) of

     [] -> [];

     EXT -> PlayerData = EXT:player_data(),

            {WSID, ALLDATA} = lists:keyfind(list_to_binary(pid_to_list(WEBSOCKETID)),1,PlayerData),

            NEWDATA = setnth(3 , ALLDATA , fgen:remove(1,PACK)),

            fgen:updateDB(

                          EXT,[

                               { player_data, lists:keyreplace(list_to_binary(pid_to_list(WEBSOCKETID)), 1, PlayerData , {list_to_binary(pid_to_list(WEBSOCKETID)), NEWDATA}) },

                               { response_data, lists:sublist(EXT:response_data(),5) ++ [1] ++ lists:nthtail(6,EXT:response_data()) }

                              ]

                         )

            %randomArenaActions(EXT)

  end;

update(bullet,WEBSOCKETID,PACK)->

  case fgen:db_hd(boss_db:find(jamir_match_data,[{match_id,'equals',lists:nth(1,PACK)}])) of

     [] -> [];

     EXT -> ResponseData = EXT:response_data(),

            fgen:updateDB(

                          EXT,[

                               {bullet_data, EXT:bullet_data() ++ [{list_to_binary(pid_to_list(WEBSOCKETID)),fgen:remove(1,PACK)}]},

                               {response_data, lists:sublist(ResponseData,6) ++ [1] ++ lists:nthtail(7,ResponseData)}

                              ]

                         )

  end.

and updateDB below;

updateDB(Record, []) -> Record:save();

updateDB(Record, [H|T]) -> updateDB(Record:set([H]), T).

I think it is impossible to get rid of bottleneck if you use a single websocket for all of the game. I am now trying to use a websocket per match. I increased the 16ms to 40ms fps rate which client sends update functions. I am now able to run the game with 10 players sending 50 bit of messages 40ms each.

I have though of many things like spawning a websocket module for each player or many other things. But the cleanest thing i think is to use a single websocket per every match and make it possible to play every player with 40 - 100 ms delay.

Thanks!

23 Kasım 2018 Cuma 19:43:38 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[Konstantin Gorshkov]

Lists are very slow. State of objects should be stored in records. They are much faster. I think you can significantly gain performance by refactoring your datatypes.

--

You received this message because you are subscribed to the Google Groups "ChicagoBoss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chicagoboss...@googlegroups.com.
Visit this group at https://groups.google.com/group/chicagoboss.

To view this discussion on the web visit https://groups.google.com/d/msgid/chicagoboss/02b4c2db-acb8-4140-95af-2843ab9c602e%40googlegroups.com.

[me...@lanyardmarket.com]

I thouht records are also using mnesia. My match databased is based on mnesia. I will check what i can do. Thanks for the advice. Appreciated.

26 Kasım 2018 Pazartesi 20:56:31 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[Konstantin Gorshkov]

I think I didn't exactly wrote what I mean. I meant using erlang records instead of lists. Your code has a lot of operations on lists. Many of them provide O(n) time such as random access, keyfind, update nth, concatenation etc. Erlang records provide O(1) to access to any element, and O(n) to copy/update.

http://erlang.org/doc/reference_manual/records.html

Also Erlang has gb_trees, dicts and other data structures which can be used for improving the performance.

[me...@lanyardmarket.com]

Thanks Konstantin now i understood. What i have in mind now is a model;

A match_sup which spawns a match_worker for each match. I will store all the match data on the spawned worker gen_server. It will also act as sender which will refresh clients with

handle_info(check, OldTimer) -> 

  erlang:cancel_timer(OldTimer),

  Timer = erlang:send_after(20, self(), check),

  sender:run(),

  {noreply, Timer}.

So it will got all the data from its module record and update in the same module. Am i right?

Regards

27 Kasım 2018 Salı 12:31:38 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[me...@lanyardmarket.com]

Dear Konstantin,

Thanks for redirecting me for the issue. Now the cpu usage drop down from 30% to 3-6% with 10 players sending 80 bytes of data each 33ms.

I used gb_trees which seemed more suitable in my case in updating and fetching performance. And the problem is gone.

Regards

27 Kasım 2018 Salı 12:31:38 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[Konstantin Gorshkov]

You are welcome. I watched the video you posted on YouTube. Great work.

--

You received this message because you are subscribed to the Google Groups "ChicagoBoss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chicagoboss...@googlegroups.com.
Visit this group at https://groups.google.com/group/chicagoboss.

To view this discussion on the web visit https://groups.google.com/d/msgid/chicagoboss/99f98872-8fe7-4d5e-be61-878e7adaea84%40googlegroups.com.

[me...@lanyardmarket.com]

Thanks! It will be better and faster, my aim is to handle 60 users per match in a fps game sending 33ms datas, thanks to you.

30 Kasım 2018 Cuma 18:42:08 UTC+2 tarihinde Konstantin Gorshkov yazdı:

[Graeme Defty]

Hi,

I just upgraded to the latest CB and starting my application now gives:

<pre>

** {'module could not be loaded',
       [{elixir_compiler,file,
            [<<"/home/graeme/projects/egolf/dev/prodfix/src/lib/golf_db_system.ex">>],
            []},
        {boss_elixir_compiler,compile,2,
            [{file,"src/boss/boss_elixir_compiler.erl"},{line,31}]},
        {boss_load,compile_and_accumulate_errors,5,
            [{file,"src/boss/boss_load.erl"},{line,263}]},
        {boss_load,load_dir,4,[{file,"src/boss/boss_load.erl"},{line,224}]},
        {boss_load,load_dirs1,6,[{file,"src/boss/boss_load.erl"},{line,213}]},
        {boss_load,'-make_all_modules/3-fun-2-',3,
            [{file,"src/boss/boss_load.erl"},{line,95}]},
        {lists,map,2,[{file,"lists.erl"},{line,1239}]},
        {lists,map,2,[{file,"lists.erl"},{line,1239}]}]}

</pre>

Any thoughts?

To view this discussion on the web visit https://groups.google.com/d/msgid/chicagoboss/695fd9ad-91eb-4d0a-8d55-0b2410595779%40googlegroups.com.

[Graeme Defty]

Hi,

It seems that the elixir compiler is not being pulled into deps when I 'make' CB.

My rebar.config contains the line:

{elixir,          {git,"https://github.com/elixir-lang/elixir.git",     {tag,    "v1.4.4"}}}

Any ideas, anyone?

g

[Graeme Defty]

Ooops!  It seems I had the wrong title on this last time  :-(

______________________

Hi,

It seems that the elixir compiler is not being pulled into deps when I 'make' CB.

My rebar.config contains the line:

{elixir,          {git,"https://github.com/elixir-lang/elixir.git",     {tag,    "v1.4.4"}}}

... but after running 'make' I have no elixir.

Any ideas, anyone?

g

[Graeme Defty]

I think there are bigger problems ...

I tried manually pulling Elixir from github to the deps directory, but trying to 'make' produces an error message that "At least Erlang 17.0 is required to build Elixir".  I am running on OTP 21. This turned out to be a dumb test in the makefile, but fixing that produced:

[{'Elixir.CompileError',exception,
                 [[{description,
                       <<"internal error in lint_module;\ncrash reason: badarg\n\n  in function  erl_anno:anno_info/1\n

It would seem that we need to get onto a newer version of Elixir, but the interface to the compiler (elixir_compiler:file/1) has changed back in March (to elixir_compiler:file/2).

I need to find a solution to this as I need to support a production system which relies on it.

I am happy to help try to bring the dependencies more up to date, but I can not do it myself. Is anyone still working on this?

Is anyone using CB at all any more?

Graeme

[Graeme Defty]

OK I finally got there - OTP21 is not supported, as described in the response to this issuehttps://github.com/ChicagoBoss/ChicagoBoss/issues/686. I guess the test in the makefile was not so dumb - hahahaha. My apologies for a false alarm (and an entirely inappropriate comment!)

Is anyone working on getting us up to OTP21? Can I help in any way? (I have an update to the makefile already  :-) )

Cheers,

Graeme