Hugo Prudente
Jul 6, 2016, 1:44:24 PM7/6/16
to Caffe Users
Hi Guys,
I'm having a really annoying problem regarding to memory leak on Cuda with Cafe.
I doesn't know the origin of the problem. Actually my environment are compose by AWS EC2 (g2.xlarge) with CUDA 7.5 + cuDNN v5 + Last Caffe.
The problem starts when I'm running the "make runtests", how we can see below, at this stage I have the leak.
The problem starts when I'm running the "make runtests", how we can see below, at this stage I have the leak.
Any help is welcome. Thanks guys;
/opt/caffe# make runtest
.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file.) type: string
default: ""
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ','. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
Cuda number of devices: 1
Setting to use device 0
Current device id: 0
Current device name: GRID K520
Note: Randomizing tests' orders with a seed of 6541 .
[==========] Running 2069 tests from 277 test cases.
[----------] Global test environment set-up.
[----------] 12 tests from ReshapeLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] ReshapeLayerTest/2.TestForwardAfterReshape
[ OK ] ReshapeLayerTest/2.TestForwardAfterReshape (459 ms)
[ RUN ] ReshapeLayerTest/2.TestInsertSingletonAxesMiddle
[ OK ] ReshapeLayerTest/2.TestInsertSingletonAxesMiddle (1 ms)
[ RUN ] ReshapeLayerTest/2.TestInferenceOfUnspecified
[ OK ] ReshapeLayerTest/2.TestInferenceOfUnspecified (1 ms)
[ RUN ] ReshapeLayerTest/2.TestInsertSingletonAxesEnd
[ OK ] ReshapeLayerTest/2.TestInsertSingletonAxesEnd (0 ms)
[ RUN ] ReshapeLayerTest/2.TestInsertSingletonAxesStart
[ OK ] ReshapeLayerTest/2.TestInsertSingletonAxesStart (1 ms)
[ RUN ] ReshapeLayerTest/2.TestForward
[ OK ] ReshapeLayerTest/2.TestForward (1 ms)
[ RUN ] ReshapeLayerTest/2.TestCopyDimensions
[ OK ] ReshapeLayerTest/2.TestCopyDimensions (1 ms)
[ RUN ] ReshapeLayerTest/2.TestGradient
[ OK ] ReshapeLayerTest/2.TestGradient (8 ms)
[ RUN ] ReshapeLayerTest/2.TestFlattenValues
[ OK ] ReshapeLayerTest/2.TestFlattenValues (0 ms)
[ RUN ] ReshapeLayerTest/2.TestInferenceOfUnspecifiedWithStartAxis
[ OK ] ReshapeLayerTest/2.TestInferenceOfUnspecifiedWithStartAxis (1 ms)
[ RUN ] ReshapeLayerTest/2.TestFlattenMiddle
[ OK ] ReshapeLayerTest/2.TestFlattenMiddle (1 ms)
[ RUN ] ReshapeLayerTest/2.TestFlattenOutputSizes
[ OK ] ReshapeLayerTest/2.TestFlattenOutputSizes (0 ms)
[----------] 12 tests from ReshapeLayerTest/2 (475 ms total)
[----------] 6 tests from FlattenLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] FlattenLayerTest/2.TestForward
[ OK ] FlattenLayerTest/2.TestForward (1 ms)
[ RUN ] FlattenLayerTest/2.TestGradient
[ OK ] FlattenLayerTest/2.TestGradient (9 ms)
[ RUN ] FlattenLayerTest/2.TestSetupWithAxis
[ OK ] FlattenLayerTest/2.TestSetupWithAxis (1 ms)
[ RUN ] FlattenLayerTest/2.TestSetupWithEndAxis
[ OK ] FlattenLayerTest/2.TestSetupWithEndAxis (0 ms)
[ RUN ] FlattenLayerTest/2.TestSetupWithStartAndEndAxis
[ OK ] FlattenLayerTest/2.TestSetupWithStartAndEndAxis (1 ms)
[ RUN ] FlattenLayerTest/2.TestSetup
[ OK ] FlattenLayerTest/2.TestSetup (1 ms)
[----------] 6 tests from FlattenLayerTest/2 (13 ms total)
[----------] 9 tests from LSTMLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] LSTMLayerTest/0.TestGradientNonZeroCont
[ OK ] LSTMLayerTest/0.TestGradientNonZeroCont (239 ms)
[ RUN ] LSTMLayerTest/0.TestLSTMUnitGradientNonZeroCont
[ OK ] LSTMLayerTest/0.TestLSTMUnitGradientNonZeroCont (46 ms)
[ RUN ] LSTMLayerTest/0.TestLSTMUnitSetUp
[ OK ] LSTMLayerTest/0.TestLSTMUnitSetUp (0 ms)
[ RUN ] LSTMLayerTest/0.TestLSTMUnitGradient
[ OK ] LSTMLayerTest/0.TestLSTMUnitGradient (46 ms)
[ RUN ] LSTMLayerTest/0.TestGradient
[ OK ] LSTMLayerTest/0.TestGradient (235 ms)
[ RUN ] LSTMLayerTest/0.TestForward
[ OK ] LSTMLayerTest/0.TestForward (3 ms)
[ RUN ] LSTMLayerTest/0.TestGradientNonZeroContBufferSize2WithStaticInput
[ OK ] LSTMLayerTest/0.TestGradientNonZeroContBufferSize2WithStaticInput (2515 ms)
[ RUN ] LSTMLayerTest/0.TestGradientNonZeroContBufferSize2
[ OK ] LSTMLayerTest/0.TestGradientNonZeroContBufferSize2 (418 ms)
[ RUN ] LSTMLayerTest/0.TestSetUp
[ OK ] LSTMLayerTest/0.TestSetUp (1 ms)
[----------] 9 tests from LSTMLayerTest/0 (3503 ms total)
[----------] 1 test from SolverFactoryTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN ] SolverFactoryTest/1.TestCreateSolver
[ OK ] SolverFactoryTest/1.TestCreateSolver (1 ms)
[----------] 1 test from SolverFactoryTest/1 (1 ms total)
[----------] 1 test from SolverFactoryTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] SolverFactoryTest/2.TestCreateSolver
This is te kern.log stack trace.
[ 1476.138072] EXT4-fs (xvdb): mounted filesystem with ordered data mode. Opts: (null)
[11024.829909] init: pollinate main process (28397) terminated with status 1
Jul 6 11:15:57 ip kernel: [ 1402.488334] RPC: Registered tcp NFSv4.1 backchannel transport module.
Jul 6 11:15:57 ip kernel: [ 1402.500512] FS-Cache: Netfs 'nfs' registered for caching
Jul 6 11:15:57 ip kernel: [ 1402.515454] Installing knfsd (copyright (C) 1996 ok...@monad.swb.de).
Jul 6 11:16:19 ip kernel: [ 1424.718138] NFS: Registering the id_resolver key type
Jul 6 11:16:19 ip kernel: [ 1424.718148] Key type id_resolver registered
Jul 6 11:16:19 ip kernel: [ 1424.718150] Key type id_legacy registered
Jul 6 11:16:19 ip kernel: [ 1424.734273] RPC: AUTH_GSS upcall failed. Please check user daemon is running.
Jul 6 11:17:05 ip kernel: [ 1470.454032] RPC: AUTH_GSS upcall failed. Please check user daemon is running.
Jul 6 11:17:11 ip kernel: [ 1476.135188] EXT4-fs (xvdb): mounting ext3 file system using the ext4 subsystem
Jul 6 11:17:11 ip kernel: [ 1476.138072] EXT4-fs (xvdb): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 17:32:38 ip kernel: [24002.820100] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.823654] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.827036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.831005] kworker/0:0 D ffff8803dfc13180 0 4 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.831103] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831107] ffff8803caa11b70 0000000000000046 ffff8803ca97c800 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.831110] ffff8803caa11fd8 0000000000013180 ffff8803ca97c800 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.831112] 7fffffffffffffff ffff8803ca97c800 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.831115] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.831121] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.831130] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.831132] [<ffffffff8172d8f4>] ? __schedule+0x384/0x7f0
Jul 6 17:32:38 ip kernel: [24002.831135] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.831138] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.831141] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.831210] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831283] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831350] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831440] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831507] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831575] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831644] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.831648] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.831651] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.831653] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.831656] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.831658] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.831661] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.831664] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.831686] INFO: task test_all.testbi:27088 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.835580] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.838992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.842977] test_all.testbi D ffff8803dfc53180 0 27088 26839 0x00000000
Jul 6 17:32:38 ip kernel: [24002.842980] ffff8803c52b9b30 0000000000000082 ffff8803c5791800 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.842985] ffff8803c52b9fd8 0000000000013180 ffff8803c5791800 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.842987] 7fffffffffffffff ffff8803c5791800 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.842990] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.842993] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.842995] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.843004] [<ffffffff8117bbb3>] ? follow_page_mask+0x173/0x290
Jul 6 17:32:38 ip kernel: [24002.843007] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.843009] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.843011] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.843087] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843154] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843221] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843287] [<ffffffffa05aeefe>] ? _nv002284rm+0xb3e/0x3990 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843354] [<ffffffffa061a811>] ? _nv000719rm+0x281/0x440 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843420] [<ffffffffa061a997>] ? _nv000719rm+0x407/0x440 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843486] [<ffffffffa061b0f8>] ? _nv000696rm+0x728/0x7d0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843554] [<ffffffffa0625153>] ? rm_ioctl+0x73/0x100 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843623] [<ffffffffa063358e>] ? nvidia_ioctl+0x13e/0x460 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843691] [<ffffffffa0631cef>] ? nvidia_frontend_ioctl+0x2f/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843758] [<ffffffffa0631d4d>] ? nvidia_frontend_unlocked_ioctl+0x1d/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.843762] [<ffffffff811d4580>] ? do_vfs_ioctl+0x2e0/0x4c0
Jul 6 17:32:38 ip kernel: [24002.843764] [<ffffffff811d47e1>] ? SyS_ioctl+0x81/0xa0
Jul 6 17:32:38 ip kernel: [24002.843766] [<ffffffff8173a49d>] ? system_call_fastpath+0x1a/0x1f
Jul 6 17:32:38 ip kernel: [24002.843768] INFO: task kworker/0:2:27092 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.847374] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.850768] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.854773] kworker/0:2 D ffff8803dfc13180 0 27092 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.854845] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.854848] ffff8800bd581b70 0000000000000046 ffff8803c7e03000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.854850] ffff8800bd581fd8 0000000000013180 ffff8803c7e03000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.854852] 7fffffffffffffff ffff8803c7e03000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.854855] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.854858] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.854864] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.854870] [<ffffffff8137247e>] ? memzero_explicit+0xe/0x10
Jul 6 17:32:38 ip kernel: [24002.854873] [<ffffffff81480bf1>] ? extract_buf+0x161/0x190
Jul 6 17:32:38 ip kernel: [24002.854876] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.854878] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.854884] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.854953] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855025] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855093] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855183] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855250] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855318] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855387] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.855390] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.855392] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.855395] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.855397] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.855399] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.855402] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.855404] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.855406] INFO: task kworker/0:3:27093 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.859015] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.862401] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.866364] kworker/0:3 D ffff8803dfc13180 0 27093 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.866435] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866438] ffff8800bd583b70 0000000000000046 ffff8803c5fc6000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.866441] ffff8800bd583fd8 0000000000013180 ffff8803c5fc6000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.866443] 7fffffffffffffff ffff8803c5fc6000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.866445] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.866448] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.866456] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.866458] [<ffffffff8172d8f4>] ? __schedule+0x384/0x7f0
Jul 6 17:32:38 ip kernel: [24002.866460] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.866462] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.866465] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.866537] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866604] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866671] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866760] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866828] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866895] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866964] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.866967] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.866970] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.866972] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.866974] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.866976] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.866979] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.866981] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.866983] INFO: task kworker/0:4:27094 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.870538] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.873958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.877977] kworker/0:4 D ffff8803dfc13180 0 27094 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.878050] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878053] ffff8800bd585b70 0000000000000046 ffff8803c79e6000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.878055] ffff8800bd585fd8 0000000000013180 ffff8803c79e6000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.878057] 7fffffffffffffff ffff8803c79e6000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.878060] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.878063] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.878071] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.878073] [<ffffffff8137247e>] ? memzero_explicit+0xe/0x10
Jul 6 17:32:38 ip kernel: [24002.878076] [<ffffffff81480bf1>] ? extract_buf+0x161/0x190
Jul 6 17:32:38 ip kernel: [24002.878078] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.878084] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.878086] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.878154] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878224] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878291] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878380] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878447] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878515] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878584] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.878587] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.878589] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.878591] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.878594] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.878596] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.878598] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.878600] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.878602] INFO: task kworker/0:5:27095 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.882175] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.885525] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.889494] kworker/0:5 D ffff8803dfc13180 0 27095 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.889565] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.889568] ffff8800bd587b70 0000000000000046 ffff8803c79e4800 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.889570] ffff8800bd587fd8 0000000000013180 ffff8803c79e4800 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.889573] 7fffffffffffffff ffff8803c79e4800 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.889575] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.889578] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.889585] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.889587] [<ffffffff8172d8f4>] ? __schedule+0x384/0x7f0
Jul 6 17:32:38 ip kernel: [24002.889590] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.889592] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.889594] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.889667] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.889734] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.889800] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.889890] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.889957] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.890025] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.890096] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.890099] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.890102] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.890104] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.890106] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.890108] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.890111] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.890113] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.890115] INFO: task kworker/0:6:27096 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.893634] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.896984] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.900950] kworker/0:6 D ffff8803dfc13180 0 27096 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.901021] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901024] ffff8800bd599b70 0000000000000046 ffff8803c79e3000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.901027] ffff8800bd599fd8 0000000000013180 ffff8803c79e3000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.901029] 7fffffffffffffff ffff8803c79e3000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.901031] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.901034] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.901042] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.901047] [<ffffffff8137247e>] ? memzero_explicit+0xe/0x10
Jul 6 17:32:38 ip kernel: [24002.901049] [<ffffffff81480bf1>] ? extract_buf+0x161/0x190
Jul 6 17:32:38 ip kernel: [24002.901051] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.901053] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.901056] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.901124] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901202] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901268] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901358] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901425] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901492] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901561] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.901564] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.901567] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.901569] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.901571] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.901574] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.901576] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.901578] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.901580] INFO: task kworker/0:7:27097 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.905206] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.908671] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.912712] kworker/0:7 D ffff8803dfc13180 0 27097 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.912782] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.912785] ffff8800bd59bb70 0000000000000046 ffff8803c79e1800 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.912788] ffff8800bd59bfd8 0000000000013180 ffff8803c79e1800 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.912790] 7fffffffffffffff ffff8803c79e1800 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.912792] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.912795] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.912797] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.912806] [<ffffffff8172d8f4>] ? __schedule+0x384/0x7f0
Jul 6 17:32:38 ip kernel: [24002.912808] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.912810] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.912813] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.912881] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.912952] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913018] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913111] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913178] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913246] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913314] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.913317] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.913320] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.913322] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.913324] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.913327] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.913329] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.913331] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.913333] INFO: task kworker/0:8:27098 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.917014] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.920428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.924542] kworker/0:8 D ffff8803dfc13180 0 27098 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.924613] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.924616] ffff8800bd59db70 0000000000000046 ffff8803c79e0000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.924618] ffff8800bd59dfd8 0000000000013180 ffff8803c79e0000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.924621] 7fffffffffffffff ffff8803c79e0000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.924623] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.924626] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.924634] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.924636] [<ffffffff8137247e>] ? memzero_explicit+0xe/0x10
Jul 6 17:32:38 ip kernel: [24002.924638] [<ffffffff81480bf1>] ? extract_buf+0x161/0x190
Jul 6 17:32:38 ip kernel: [24002.924641] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.924647] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.924649] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.924717] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.924787] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.924854] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.924943] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.925010] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.925080] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.925149] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.925152] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.925155] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.925157] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.925159] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.925161] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.925164] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.925166] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.925168] INFO: task kworker/0:9:27099 blocked for more than 120 seconds.
Jul 6 17:32:38 ip kernel: [24002.928747] Tainted: P OX 3.13.0-91-generic #138-Ubuntu
Jul 6 17:32:38 ip kernel: [24002.932124] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 6 17:32:38 ip kernel: [24002.936171] kworker/0:9 D ffff8803dfc13180 0 27099 2 0x00000000
Jul 6 17:32:38 ip kernel: [24002.936243] Workqueue: events os_execute_work_item [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936245] ffff8800bd59fb70 0000000000000046 ffff880035dc0000 0000000000013180
Jul 6 17:32:38 ip kernel: [24002.936247] ffff8800bd59ffd8 0000000000013180 ffff880035dc0000 ffff8803ca489d08
Jul 6 17:32:38 ip kernel: [24002.936250] 7fffffffffffffff ffff880035dc0000 0000000000000002 0000000000000000
Jul 6 17:32:38 ip kernel: [24002.936252] Call Trace:
Jul 6 17:32:38 ip kernel: [24002.936255] [<ffffffff8172dd89>] schedule+0x29/0x70
Jul 6 17:32:38 ip kernel: [24002.936257] [<ffffffff8172cfd9>] schedule_timeout+0x279/0x310
Jul 6 17:32:38 ip kernel: [24002.936259] [<ffffffff8172d8f4>] ? __schedule+0x384/0x7f0
Jul 6 17:32:38 ip kernel: [24002.936261] [<ffffffff81730276>] __down_common+0xf7/0x144
Jul 6 17:32:38 ip kernel: [24002.936263] [<ffffffff817302e0>] __down+0x1d/0x1f
Jul 6 17:32:38 ip kernel: [24002.936265] [<ffffffff810b36a1>] down+0x41/0x50
Jul 6 17:32:38 ip kernel: [24002.936334] [<ffffffffa063cc4f>] os_acquire_mutex+0x3f/0x50 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936401] [<ffffffffa061efd8>] _nv012536rm+0x18/0x30 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936467] [<ffffffffa05d0bbd>] ? _nv014113rm+0x3d/0x120 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936556] [<ffffffffa0487a2d>] ? _nv013080rm+0x14d/0x260 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936624] [<ffffffffa0622d39>] ? _nv000671rm+0x229/0xb10 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936691] [<ffffffffa0625d79>] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936760] [<ffffffffa063d196>] ? os_execute_work_item+0x46/0x70 [nvidia]
Jul 6 17:32:38 ip kernel: [24002.936763] [<ffffffff81086078>] ? process_one_work+0x178/0x470
Jul 6 17:32:38 ip kernel: [24002.936766] [<ffffffff81086e91>] ? worker_thread+0x121/0x410
Jul 6 17:32:38 ip kernel: [24002.936768] [<ffffffff81086d70>] ? rescuer_thread+0x430/0x430
Jul 6 17:32:38 ip kernel: [24002.936770] [<ffffffff8108dc79>] ? kthread+0xc9/0xe0
Jul 6 17:32:38 ip kernel: [24002.936772] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Jul 6 17:32:38 ip kernel: [24002.936775] [<ffffffff8173a3e8>] ? ret_from_fork+0x58/0x90
Jul 6 17:32:38 ip kernel: [24002.936777] [<ffffffff8108dbb0>] ? kthread_create_on_node+0x1c0/0x1c0
Reply all
Reply to author
Forward
0 new messages