Normally if I run out of memory it just tells me it failed to allocate memory. I tested this on both a 980ti and Titan X. I'm using Cuda 7.5, CudNN 5.1, and Nvidia driver version is 352.63.
Has anybody seen this before?
E tensorflow/stream_executor/cuda/cuda_driver.cc:1154] could not synchronize on CUDA stream: CUDA_ERROR_ILLEGAL_ADDRESS :: No stack trace available
E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:198] Unexpected Event status: 1
Aborted (core dumped)