[erlang-questions] Why system_flag(scheduler_bind_type) is deprecated?

[Max Lapshin]

Why system_flag(scheduler_bind_type, How) is deprecated in favor of +sbt flag?

This sbt flag is different when I have to launch via escript or via erl and this is why it is less convenient than using system_flag call.

Also system_flag call can be called according to some system configuration file and sbt needs full restart.

[Max Lapshin]

Mac OS X:

$ erl +sbt s
setting scheduler bind type 's' failed: not supported
Usage: beam.smp [flags] [ -- [init_args] ]
...
$ erl -sbt s
Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:4:4] [async-threads:
0] [hipe] [kernel-poll:false] [dtrace]
2> erlang:system_info(scheduler_bindings).
{unbound,unbound,unbound,unbound}



Linux:

$ erl +sbt s
Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0]
[hipe] [kernel-poll:false]

Eshell V5.9 (abort with ^G)
1> erlang:system_info(scheduler_bindings).
{0,1,2,3,4,5,6,7}


Setting this flag +sbt doesn't allow me to launch code under mac os.
How should I change system_flag/2 call to this flag?
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Rickard Green]

Both the 'cpu_topology' and the 'scheduler_bind_type' arguments of the system_flag/2 BIF are deprecated and will be removed since we do not want to change this configuration in runtime. With the support for this we got today the runtime configuration change isn't that problematic, but it prevents future planned improvements from being implemented.

Regards,
Rickard Green, Erlang/OTP, Ericsson AB

[Rickard Green]

On Dec 26, 2012, at 5:46 PM, Max Lapshin <max.l...@gmail.com> wrote:

> Mac OS X:
>
> $ erl +sbt s
> setting scheduler bind type 's' failed: not supported
> Usage: beam.smp [flags] [ -- [init_args] ]
> ...
> $ erl -sbt s
> Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:4:4] [async-threads:
> 0] [hipe] [kernel-poll:false] [dtrace]
> 2> erlang:system_info(scheduler_bindings).
> {unbound,unbound,unbound,unbound}
>
>
>
> Linux:
>
> $ erl +sbt s
> Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0]
> [hipe] [kernel-poll:false]
>
> Eshell V5.9 (abort with ^G)
> 1> erlang:system_info(scheduler_bindings).
> {0,1,2,3,4,5,6,7}
>
>
> Setting this flag +sbt doesn't allow me to launch code under mac os.
> How should I change system_flag/2 call to this flag?

Neither the +sbt flag nor the 'scheduler_bind_type' argument of the system_flag/2 BIF have ever been supported on MacOSX. This since the last time I tried to find an api for binding threads to processors on MacOSX I couldn't find one. This might however have changed since then. It was a long time ago I checked. If you can find an api making it possible to implement this on MacOSX, a patch implementing this is welcome.

Regards,
Rickard Green, Erlang/OTP, Ericsson AB

[Max Lapshin]

So, I need to change shell script that launches application that will check if it is Linux or not and add or not add +sbt flag?

[Max Lapshin]

There are some builds for linux that also come without this +sbt flag. How can I understand that erlang supports this flag from shell script? If I pass this +sbt flag to an erlang that doesn't support it, it just fails.



And I mention that using erlang:system_flag/2 call doesn't bring any of this problems?

[Max Lapshin]

Damn, this removal is really a problem.

I can't use erlang:system_flag/2 call anymore, I can't use +sbt option, because it is impossible to understand if this flag is supported with current erlang build, I can't work without this flag, because by default scheduler is trying to group processes on one core and udp packets are lost.

Is it possible to ignore +sbt flag, not to fail the runtime?

[Rickard Green]

Currently no, but I can add another switch that will try setting bind
type and fail silently if not supported. I'll do this at the same time
as removing the erlang:system_flags(scheduler_bind_type, _) support.

Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB.

[Max Lapshin]

Rickard, it would be great. 

[Angel J. Alvarez Miguel]

hi

I look a this :

"...since we do not want to change this configuration in runtime..."

This sounds a bit limiting

Current platforms can (and will) shutdown CPU cores in response to power management policies
and erlang deamonized software must adapt to this changes without needing a restart.

While it seems at first an unprobable scenario right now, on server side it really sounds good when
you run erlang on laptop/mobile platforms.

Even some virtualization products are planin to use hot-plug facilities to make resource management easier on current
platforms so its not really crazy today to think that cpu cores will come and leave at unexpected times on tipical cloud scenarios.

IMHO the ability to sense this on user code and facilities to control or alter the scheduler behaviour will be more important
in the near future , thus having to restart the entire VM for this seems overkill and a backward movement from the current setup.

I have recently experimiented this during the summer case when i unplug my laptop my power management script
shut down half of the cores when battery starts to run out and lowers de cpu frecuency and erlang responded
on stderrr something about scheduler failure trying to bind to the dead cores, then i managed to make some tests trying to
monitor this from erlang code and to put schedulers offline until cores reapear again.

Even i can depict scenarios where depending on licensing feaures application code can control the amount of paralelism
the VM will exhibit by tweaking schedulers affinity or just shutting them down to limit to the cores licensed...

You should think about this even if you plan to make other improvements

Regards, Angel

[Max Lapshin]

No, no Angel. You are very right: modern servers are the first candidates to change amount of CPU and RAM.

Clouds can change amount of cores and RAM on fly. It was almost impossible on commodity hardware 4 years ago, but it is already here nowadays.

Rickard, maybe OTP group should discuss this idea again?

Erlang server launched on cloud hosting should be ready for changing number of cores.

[Björn-Egil Dahlberg]

Two different things. 

1. Being able to handle loss of cores or graceful termination of schedulers at runtime.

2. Scheduler bindings at runtime.

The first is more important than the second one and something we want to be able to handle gracefully.

2013/1/4 Max Lapshin <max.l...@gmail.com>

[Max Lapshin]

Cores can not only shutdown, they can also appear.

[Zabrane Mickael]

Agree with Max on this.

Regards,

Zabrane

On Jan 4, 2013, at 1:46 PM, Max Lapshin wrote:

Cores can not only shutdown, they can also appear.

[Masklinn]

On 2012-12-27, at 00:45 , Rickard Green wrote:
>
> Neither the +sbt flag nor the 'scheduler_bind_type' argument of the system_flag/2 BIF have ever been supported on MacOSX. This since the last time I tried to find an api for binding threads to processors on MacOSX I couldn't find one. This might however have changed since then.

It has not, as far as I know. OSX only provides an affinity API (since 10.5): http://developer.apple.com/library/mac/#releasenotes/Performance/RN-AffinityAPI/_index.html

> OS X does not export interfaces that identify processors or control
> thread placement—explicit thread to processor binding is not
> supported. Instead, the kernel manages all thread placement.

[Rickard Green]

The following is the idea of how it should work some time in the future. Note that this won't happen in R16, and that it has not implemented yet and may be subject to changes.

When the runtime system boots, a fixed cpu-topology is set. The cpu-topology set is either automatically determined, or configured by the user on the command line. Schedulers are then bound to logical processors according to the bind type argument passed. This mapping between scheduler and logical processor will after this be fixed and will never change.

This does, however, *not* mean that from this point on every logical processor must remain online. It will still be possible to take processors offline as well as online. The state of the scheduler mapped to a logical processor should normally follow the state of the logical processor. That is, when a processor is taken offline, so is its scheduler... The cpu-topology used should match the actual hardware topology of the machine when it is fully equipped, and the amount of schedulers started should equal the amount of logical processors in that cpu-topology.

This way we will be able to handle reasonable changes of the cpu-topology while we will be able to have a fixed mapping between schedulers and logical processors. This fixed mapping is important since it simplifies implementations of things like better NUMA support, load-balancing that take cpu-topology into account, etc. My guess is that using this strategy we will be able to handle hot-plug, and power management scenarios on real hardware for quite some time whiteout running into trouble.

As it is today, when cpu-topology and/or scheduler bind type is changed in runtime, the mapping between schedulers and logical processors can completely transform into something that looks nothing similar to what it was before. This unnecessarily complicates things a lot. I don't think that we will see hardware that can be physically transformed this way in the near future. When that happens we will just have to deal with it. Such transformations might perhaps occur in virtualized environments, but you don't want to bind schedulers if the cpu-topology doesn't match the actual physical hardware topology. This will just get more and more important as the runtime system utilize more information about the cpu-topology.

Another scenario when you typically don't want to bind schedulers is when all schedulers aren't guaranteed to get the major part of the cpu time of the processors that they are bound to. We have in some cases seen severe performance degradation in such scenarios (which is why we do not bind schedulers by default anymore). In a virtualized environment it might also be hard to get such guarantees.

Regards,
Rickard Green, Erlang/OTP, Ericsson AB

[Max Lapshin]

Ok, Rickard. I see that you have to deal with much more complicated cases that I have.

But current situation with flags and system_flag makes almost impossible to distribute software than _needs_ spreading of tasks.

If erlyvideo is launched without +sbt spread flag, it looses packets from UDP because too many tasks are on one core.

There are some erlang distributions (OS X, ubuntu package, some others) that are just halting if you pass +sbt flag.

So I can't write proper init script that passes this flag or don't pass.

This is my problem.

[Jesper Louis Andersen]

On Jan 8, 2013, at 12:04 PM, Max Lapshin <max.l...@gmail.com> wrote:

> If erlyvideo is launched without +sbt spread flag, it looses packets from UDP because too many tasks are on one core.
> There are some erlang distributions (OS X, ubuntu package, some others) that are just halting if you pass +sbt flag.
> So I can't write proper init script that passes this flag or don't pass.

I think Rickard already said he would implement a "silent failure" flag for this, so you could do something like

erl +sbt db +sbt-ignore-failure

and have it work on, say, OSX because it would silently disable the flag. As you say, +sbt has too much of an impact in the real world to be ignored for certain types of load, so I still like the default of refusing to start if the bind type can't be satisfied.

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Max Lapshin]

I think Rickard already said he would implement a "silent failure" flag for this, so you could do something like

erl +sbt db +sbt-ignore-failure

and have it work on, say, OSX because it would silently disable the flag. As you say, +sbt has too much of an impact in the real world to be ignored for certain types of load, so I still like the default of refusing to start if the bind type can't be satisfied.

 

No, you are wrong with it.  There are problems with capturing 100 Mbit/s and more via UDP. This feature is required by 1% of users. What for to disable erlyvideo for other 99% of users?

scheduler binding is a performance setting, system should be gracefully degraded in case of lacking it.

 

No

[Angel J. Alvarez Miguel]

On Martes, 8 de Enero de 2013 12:08:41 Jesper Louis Andersen escribió:
> On Jan 8, 2013, at 12:04 PM, Max Lapshin <max.l...@gmail.com> wrote:
> > If erlyvideo is launched without +sbt spread flag, it looses packets from
> > UDP because too many tasks are on one core. There are some erlang
> > distributions (OS X, ubuntu package, some others) that are just halting
> > if you pass +sbt flag. So I can't write proper init script that passes
> > this flag or don't pass.
>
> I think Rickard already said he would implement a "silent failure" flag for
> this, so you could do something like
>
> erl +sbt db +sbt-ignore-failure

This seems to be a good falloff, here we use scritptized rebars deployments
for our projects and this fits very well to be included in the
escript_emu_args variable... provided the user cant change default vm args per
se

[Rickard Green]

On Jan 8, 2013, at 1:04 PM, "Angel J. Alvarez Miguel" <cl...@uah.es> wrote:

> On Martes, 8 de Enero de 2013 12:08:41 Jesper Louis Andersen escribió:
>> On Jan 8, 2013, at 12:04 PM, Max Lapshin <max.l...@gmail.com> wrote:
>>> If erlyvideo is launched without +sbt spread flag, it looses packets from
>>> UDP because too many tasks are on one core. There are some erlang
>>> distributions (OS X, ubuntu package, some others) that are just halting
>>> if you pass +sbt flag. So I can't write proper init script that passes
>>> this flag or don't pass.
>>
>> I think Rickard already said he would implement a "silent failure" flag for
>> this, so you could do something like
>>

>> erl +sbt db +sat-ignore-failure

>
> This seems to be a good falloff, here we use scritptized rebars deployments
> for our projects and this fits very well to be included in the
> escript_emu_args variable... provided the user cant change default vm args per
> se
>
>>
>> and have it work on, say, OSX because it would silently disable the flag.
>> As you say, +sbt has too much of an impact in the real world to be ignored
>> for certain types of load, so I still like the default of refusing to
>> start if the bind type can't be satisfied.
>>
>> Jesper Louis Andersen
>> Erlang Solutions Ltd., Copenhagen
>>

I'll add an +stbt ("try bind type") switch that will do the same thing as +sbt, but if it fails this will silently be ignored and the runtime will be started anyway. I can add the +stbt switch to r16. The change will be quite simple, so I guess that it should be quite easy to apply the change on the r15b03 source if you want to.

Regarding support for automatic detection of cpu-topology and binding of schedulers. I think this is an area really suitable for user contributions. Since the APIs that need to be used often differ between OSes and sometimes also between hardware architectures on the same OS, we will never have the time to implement support for this on all platforms users are interested in. Therefore, user contributions implementing this on various platforms are much appreciated (other contributions are of course also appreciated).

If +sbt fails on a newer Linux, the problem is most likely that no cpu-topology were automatically detected. On Linux, the runtime tries to detect this by inspecting the /sys filesystem. The content under /sys/devices/system/{cpu,node} varies quite a lot depending on hw arch. Sometimes there aren't enough info there to determine the topology, but it could also be that the information is just formatted another way than we expected.

Detection of cpu-topology and binding of threads to logical processors are implemented in $ERL_TOP/erts/lib_src/common/erl_misc_utils.c

Regards,
Rickard Green, Erlang/OTP, Ericsson AB

[Max Lapshin]

Ok. Thank you for +stbt

[Angel J. Alvarez Miguel]

Thanks for the explanations!

[Rickard Green]

Commit 209655e30a82cdf172211278e01807024eed4130 implements +stbt. The
commit is based on OTP_R15B03 and has been merged into master (to be R16A).

Regards,
Rickard

--
Rickard Green, Erlang/OTP, Ericsson AB.

[Max Lapshin]

Thank you!