Using PERCPU in application or module

[Wonsup Yoon]

Hi,

I'm trying to use PERCPU macro in application or module.

I tried using cpu::notifier, but it seems a_init is not called.

class A {

public:

A() { printf("init: %p\n", this); }

void test() { printf("test: %d\n", x); }

int x = 123;

};

PERCPU(A *, a);

static sched::cpu::notifier a_init([]() { *a = new A(); });

Also, I used constructor function to initialize As, but it aborts.

static __attribute__((constructor)) void a_init() {

for (auto c : sched::cpus) {

sched::thread *th = sched::thread::make([] { *a = new A(); },

sched::thread::attr().pin(c));

th->start();

th->join();

}

}

Any hint for this case?

[Nadav Har'El]

On Sat, May 23, 2020 at 6:35 PM Wonsup Yoon <pus...@kaist.ac.kr> wrote:

Hi,

I'm trying to use PERCPU macro in application or module.

Hi,

The PERCPU macro does not support this. What it does is to add information about this variable in a special section of the executable (".percpu"), then arch/x64/loader.ld makes sure all these entries will be together between "_percpu_start" and "_percpu_end", and finally sched.cc for every CPU creates (in the cpu::cpu(id) constructor) a copy of this data. So if a loadable module (share library) contains another per-cpu variable, it never gets added to the percpu area.

However, I believe we do have a mechanism that will suite you: dynamic_percpu<T>.

You can create (and destroy) such an object of type dynamic_percpu<T> at any time, and it does the right thing:  The variable will be allocated on all CPUs when the object is created, will be allocated on new cpus if those happen, and will be freed when the object is destroyed.

In your case you can have a global dynamic_percpu<T> variable in your loadable module. This object will be created when the module is loaded, and destroyed when the module is unloaded - which is what you want.

Nadav.

[Wonsup Yoon]

Thank you for the response.

Yes, dynamic_percpu<T> is perfect for my purpose.

However, I encountered another issue.

If I use dynamic_percpu with preempt-lock (I think it is very common pattern), it abort due to assertion failed.

It seems lazy binding prevents preemption lock.

So, I had to add -fno-plt option, and it works.

example code)

#include <stdio.h>

#include <assert.h>

#include <osv/preempt-lock.hh>

#include <osv/percpu.hh>

struct counter {

int x = 0;

void inc(){

x += 1;

}

int get(){

return x;

}

};

dynamic_percpu<counter> c;

int main(int argc, char *argv[])

{

SCOPE_LOCK(preempt_lock);

c->inc();

return 0;

}

Backtrace)

[backtrace]

0x000000004023875a <__assert_fail+26>

0x000000004035860c <elf::object::resolve_pltgot(unsigned int)+492>

0x0000000040358669 <elf_resolve_pltgot+57>

0x000000004039e2ef <???+1077535471>

0x000010000000f333 <???+62259>

0x000000004042a47c <osv::application::run_main()+60>

0x0000000040224bd0 <osv::application::main()+144>

0x000000004042a628 <???+1078109736>

0x0000000040462715 <???+1078339349>

0x00000000403fac86 <thread_main_c+38>

0x000000004039f632 <???+1077540402>

2020년 5월 24일 일요일 오후 5시 26분 17초 UTC+9, Nadav Har'El 님의 말:

[Nadav Har'El]

On Tue, May 26, 2020 at 4:22 AM Wonsup Yoon <pus...@kaist.ac.kr> wrote:

Thank you for the response.

Yes, dynamic_percpu<T> is perfect for my purpose.

However, I encountered another issue.

If I use dynamic_percpu with preempt-lock (I think it is very common pattern), it abort due to assertion failed.

It seems lazy binding prevents preemption lock.

So, I had to add -fno-plt option, and it works.

You are right about preempt lock and your workaround for lazy binding.

However, to use a per-cpu variable, you don't need full preemption locking - all you need is *migration* locking - in other words, the thread running this code should not be migrated to a different CPU (this will change the meaning of the per-cpu variable while you're using it), but it is perfectly fine for the thread to be preempted to run a different thread - as long as the original thread eventually returns to run on the same CPU it previously ran on.

So just replace your use of "preempt_lock" by "migration_lock" (include <osv/migration-lock.hh>) and everything should work, without disabling lazy binding.

Please note that if you use the per-cpu on a thread which is already bound to a specific CPU (which was the case in your original code you shared), you don't even need migration lock! A pinned thread already can't migrate to any other CPU, so it doesn't need to use this migration-avoidance mechanism at all. You can use per-cpu variables on such threads without any special protection.

 

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com.

[Wonsup Yoon]

Actually, I used preempt_lock to prevent data races. 

If two concurrent threads in a core access same per-cpu variable, I think we still need preempt lock.

example)

counter's initial value: 0

                                  CPU 0   

Thread A            A_local = counter + 1   (A_local = 1)

Thread A                 (preemption)

Thread B            B_local = counter + 1   (B_local = 1)

Thread B            counter = B_local         (counter = 1)

Thread B                   (exit)

Thread A             counter = A_local        (counter = 1)

I expect counter to be 2, but 1 returns.

2020년 5월 26일 화요일 오후 3시 4분 24초 UTC+9, Nadav Har'El 님의 말:

To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

[Nadav Har'El]

On Tue, May 26, 2020 at 9:41 AM Wonsup Yoon <pus...@kaist.ac.kr> wrote:

Actually, I used preempt_lock to prevent data races. 

If two concurrent threads in a core access same per-cpu variable, I think we still need preempt lock.

This is true - if you have two threads in the same core that access the same per-cpu variable, you need some sort of locking.

preempt lock isn't the only way, of course - you can also use a mutex, as well as std::atomic, and other solutions.

preempt lock is indeed usually the fastest method, but as you saw it comes with strings attached - the locked code really

cannot cause any preemption - which means it can't wait for any mutex, cannot do anything (including delayed symbol

resolution which might wait for a mutex).  In addition, you need to make sure the entire object is already in memory and

doesn't need to be demand-paged, or you may get a preemption in the middle of the code just to read in another page

of executable.

We have a macro OSV_ELF_MLOCK_OBJECT()  (from <osv/elf.hh>)  which marks the object with a flag (a

.note.osv-mlock section) that ensures *both* things: The object is entirely read into memory on start, and all of

its symbols are resolved on start. You can see an example of OSV_ELF_MLOCK_OBJECT() being used in a bunch

of tests in tests/. If you use this macro, you don't need to change your code's compilation.

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com.

[Wonsup Yoon]

Great! OSV_ELF_MLOCK_OBJECT seems very useful.

Thanks.

2020년 5월 26일 화요일 오후 4시 25분 57초 UTC+9, Nadav Har'El 님의 말:

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com.

[Wonsup Yoon]

I tested the dynamic_percpu and it seems mostly works fine.

However, I found using dynamic_percpu in a constructor function leads to general protection fault.

ex)

#include <stdio.h>

#include <assert.h>

#include <bsd/string.h>

#include <osv/preempt-lock.hh>

#include <osv/percpu.hh>

#include <osv/elf.hh>

struct counter {

int x = 0;

void inc(){

x += 1;

}

int get(){

return x;

}

};

dynamic_percpu<counter> c;

int main(int argc, char *argv[])

{

SCOPE_LOCK(preempt_lock);

c->inc();

return 0;

}

static __attribute__((constructor)) void test_init(void) {

SCOPE_LOCK(preempt_lock);

c->inc();

}

OSV_ELF_MLOCK_OBJECT();

output:

Cmdline: /hello   

[registers]

RIP: 0x000010000000a8f8 <counter::inc()+12>

RFL: 0x0000000000010286  CS:  0x0000000000000008  SS:  0x0000000000000010

RAX: 0xffff7fffbf81b000  RBX: 0x0000000000000001  RCX: 0x0000000000000000  RDX: 0x0000000000000000

RSI: 0xffff7fffbf81b000  RDI: 0xffff7fffbf81b000  RBP: 0x0000200000200e70  R8:  0x0000000000000001

R9:  0x0000000000000000  R10: 0x000000000000000d  R11: 0x0000000000000000  R12: 0xffffa000012bde00

R13: 0x0000000000000000  R14: 0x000000000000000d  R15: 0x0000000000000000  RSP: 0x0000200000200e70

general protection fault

[backtrace]

0x00000000403a0e84 <general_protection+116>

0x000000004039e642 <???+1077536322>

0x000010000000a74f <???+42831>

0x000000004035546c <elf::object::run_init_funcs(int, char**)+268>

0x000000004035560a <elf::program::init_library(int, char**)+362>

0x0000000040224b7a <osv::application::main()+58>

0x000000004042a688 <???+1078109832>

0x0000000040462775 <???+1078339445>

0x00000000403faca6 <thread_main_c+38>

0x000000004039f602 <???+1077540354>

2020년 5월 26일 화요일 오후 11시 44분 23초 UTC+9, Wonsup Yoon 님의 말:

[Wonsup Yoon]

Oh, it is not a OSv-specific problem.

It is because constructor function called before class initialization.

2020년 6월 2일 화요일 오후 11시 54분 26초 UTC+9, Wonsup Yoon 님의 말: