VO SQL backend
16 views
Skip to first unread message
ffoe...@gmail.com
Jul 30, 2010, 4:52:11 PM7/30/10
to vscse-big-data-...@googlegroups.com
I'd like to know what the SQL backend is...and it would be nice if the SQL-CUDA integration would be available.
Is this a single machine?
Is this a single machine?
Michael Matthews
Jul 30, 2010, 6:11:21 PM7/30/10
to vscse-big-data-...@googlegroups.com
I am interested in the implementation details of this as well. Is this a new database written from the ground up? How generically is the GPU integration implemented, and can I create brand new functions that can be run on there? Is the source code available? I found a trac website, but it didn't have any VOTable code, it appears.
Tamas Budavari
Aug 5, 2010, 9:30:07 PM8/5/10
to VSCSE Big Data for Science 2010
Sorry for the delay, I have been out of town without proper internet
since the talk. The answer is:
In our implementation, the RDBMS is Microsoft's SQL Server 2008, which
provides easy and elegant ways of extending its functionalities. The
user-defined functions and procedures of the SQL interface are written
in C#, which communicate with an out-of-process, independent service
via standard IPC to efficiently transfer data to the workers in a
shared memory buffer. The workers are written in C++ and can use CUDA
to drive the GPUs. While the IPC is within a single box (with
potentially multiple GPUs), there is nothing to stop you from farming
out jobs from the service on the other end. The hooks for these are
not in place but at that point you are in C++ and free to use anything
you like (e.g., HPC clusters over MPI?) The results are returned in
the same shared memory buffer to the C# wrappers that present them to
the SQL user.
While the framework is deployed and being used for various projects in
house at Hopkins, it is still also under heavy development, hence the
code has not yet been released. We intend to do so, when we converge
on something stable. We will present more details on this project
later in September at the GTC 2010. I'll try to remember to post a
note here, but please do bug me about this later, if you are
interested.
> > Is this a single machine?- Hide quoted text -
>
> - Show quoted text -
since the talk. The answer is:
In our implementation, the RDBMS is Microsoft's SQL Server 2008, which
provides easy and elegant ways of extending its functionalities. The
user-defined functions and procedures of the SQL interface are written
in C#, which communicate with an out-of-process, independent service
via standard IPC to efficiently transfer data to the workers in a
shared memory buffer. The workers are written in C++ and can use CUDA
to drive the GPUs. While the IPC is within a single box (with
potentially multiple GPUs), there is nothing to stop you from farming
out jobs from the service on the other end. The hooks for these are
not in place but at that point you are in C++ and free to use anything
you like (e.g., HPC clusters over MPI?) The results are returned in
the same shared memory buffer to the C# wrappers that present them to
the SQL user.
While the framework is deployed and being used for various projects in
house at Hopkins, it is still also under heavy development, hence the
code has not yet been released. We intend to do so, when we converge
on something stable. We will present more details on this project
later in September at the GTC 2010. I'll try to remember to post a
note here, but please do bug me about this later, if you are
interested.
>
> - Show quoted text -
ffoe...@gmail.com
Sep 2, 2010, 5:13:34 PM9/2/10
to vscse-big-data-...@googlegroups.com
Tamas I'll be at GTC also.
Looking forward to your talk!
FF
Reply all
Reply to author
Forward
0 new messages