Is it possible to get the Physical memory address of a variable in python?
Ognjen Bezanov
Say I have a python variable:
a = "hello"
Is it possible for me to get the physical address of that variable (i.e.
where it is in RAM)?
I know that id(a) will give me it's memory address, but the address
given does not seem to correlate with the physical memory. Is this even
possible?
Thank you!
Ognjen
Benjamin Kaplan
When you use Python, program in Python and not C. What do you need the
memory location of a variable for? Python, like Java and .NET is a
higher level language. You're not supposed to worry about things like
the physical location in memory. There's probably some ugly hack using
ctypes, or just writing the code in C but I don't know enough about
Python's C API to know what it is.
FWIW, even the id(x) == address of x is only an implementation detail
of CPython. Other Python implementations don't use that scheme.
MRAB
> Hello all,
>
> Say I have a python variable:
>
> a = "hello"
>
> Is it possible for me to get the physical address of that variable (i.e.
> where it is in RAM)?
>
> I know that id(a) will give me it's memory address, but the address
> given does not seem to correlate with the physical memory. Is this even
> possible?
>
in a dict and the variable's value is the corresponding value for that
key (or, to be exact, a reference to the value). A particular value can
be referred to by any number of 'variables'.
Carl Banks
I'm going to guess that
A. You don't really want physical address but logical address (i.e.,
the address where you might access the memory with a C pointer), and
B. You want the address not of the object itself, but of the data
within (that is, you'd want the "pointer" you receive to point to the
ASCII string hello)
Python's way to handle cases like this is called buffer protocol, but
it only operates at the C-extension level. There is a set of
functions in the C API that look like this
PyObject_AsReadBuffer(obj,**buffer,*len)
Calling this will return the address of the string "hello" in
*buffer. See C API Reference Manual for more details. This function
can also be called from ctypes, but I don't remember the exact
procedure for modifying input arguments through a pointer.
Note: If you really want a PHYSCIAL RAM address you'll have to make
some OS-specific system call (I think) with the logical address,
probably needing superuser privileges. Obviously this call isn't
exposed in Python, because there's no use for it in Python. Unless
you're programming DMA transfers or something like that, there's no
use for it in C, either.
Carl Banks
Dave Angel
> On Mon, Nov 9, 2009 at 7:47 PM, Ognjen Bezanov <Ogn...@mailshack.com> wrote:
>
>> Hello all,
>>
>> Say I have a python variable:
>>
>> a = "hello"
>>
>> Is it possible for me to get the physical address of that variable (i.e.
>> where it is in RAM)?
>>
>> I know that id(a) will give me it's memory address, but the address given
>> does not seem to correlate with the physical memory. Is this even possible?
>>
>>
>>
>>
>> Ognjen
>>
>>
>
> When you use Python, program in Python and not C. What do you need the
> memory location of a variable for? Python, like Java and .NET is a
> higher level language. You're not supposed to worry about things like
> the physical location in memory. There's probably some ugly hack using
> ctypes, or just writing the code in C but I don't know enough about
> Python's C API to know what it is.
>
> FWIW, even the id(x) == address of x is only an implementation detail
> of CPython. Other Python implementations don't use that scheme.
>
>
>
implementation can do different things, as long as the documented
semantics are preserved.
The "variable" aaa is not at any particular location, and will quite
likely move when you define a new "variable" bbb or ccc in the same
scope. aaa is just a dictionary entry after all, in some scope.
(Although it may be optimized, for locals, and for slots)
aaa is bound to a particular object, and that object won't move. That
object has an id() value, and I believe that id value is the address.
And when you bind aaa to a different object, there'll be a different
id() and address. But unless you're writing a C extension, the actual
address is kind of irrelevant, isn't it?
DaveA
Ognjen Bezanov
Thanks for all the responses guys. In hindsight I probably should have
explained why on earth I'd need the physical address from an interpreted
language.
I'm trying to see if there is any way I can make Python share data
between two hosts using DMA transfers over a firewire connection, so
avoiding the need for another layer on top such as IPv4 + Python sockets.
Thanks to some old python bindings which I updated to python 2.6, I can
read any write to the RAM of any firewire connected host within python.
Because it uses DMA (the cpu is not involved in this at all), I can only
specify a physical address within the 4GB ram limit to read from and
write to.
Now what I've done so far is on the remote host I run python and set a
variable as so:
a = "foo"
print a
'foo'
Then on the local host I run a python script that scans the entire RAM
looking for the string "foo", and replaces it with the string "oof". I
have had success with this method. Once it's done and I do "print a" on
the remote host, I get "oof" as the variable value, so in theory it can
work.
Problem is that it's slow. Scanning 3GB of RAM every time you want to do
this is not a good method. I thought that if I could get python to
return the physical address of where the value of a variable is, then I
can just jump to that address and write the data.
From what I've been told so far, it's not possible to do this without
some OS-specific (Linux in this case) syscall. Is this correct?
Thanks!
Ognjen
Maxim Khitrov
If all you need is a memory buffer, array.array provides a
buffer_info() method that tells you the current (virtual) address. For
DMA, I think there are dma_map_* functions in linux that you may be
able to call through ctypes.
- Max
Dave Angel
but very dangerous. It could easily have been three other bytes which
just happened to match this string. And those bytes could have been
part of a disk directory entry (result disk corruption) or some other
application. Further, between the time you did the search and the time
when you updated, the relevant bytes could have been swapped to disk,
and something else put in the same physical address.
Anyway, I've been assuming all along that you meant address in the sense
that a C program uses it, which is a linear address. But as long as
you're running on an OS that has virtual memory running, there's another
translation going on invisibly to the application, which converts linear
to physical. So even after you get a memory address, you need to
convince the OS to lock such memory in place for awhile, and then tell
you where (physical address) it locked it.
Note also that if a range of addresses spans a 4k boundary (for Intel
architecture), it's possible that the physical addresses are not even
contiguous. But the system calls from Windows (and presumably also from
Linux) can hide that from you, via double-buffering.
DaveA
Carl Banks
> Hey,
>
> Thanks for all the responses guys. In hindsight I probably should have
> explained why on earth I'd need the physical address from an interpreted
> language.
>
> I'm trying to see if there is any way I can make Python share data
> between two hosts using DMA transfers over a firewire connection, so
> avoiding the need for another layer on top such as IPv4 + Python sockets.
>
> Thanks to some old python bindings which I updated to python 2.6, I can
> read any write to the RAM of any firewire connected host within python.
> Because it uses DMA (the cpu is not involved in this at all), I can only
> specify a physical address within the 4GB ram limit to read from and
> write to.
[snip]
> From what I've been told so far, it's not possible to do this without
> some OS-specific (Linux in this case) syscall. Is this correct?
What you'd normally need to do it to use system calls to allocate a
DMA-able memory buffer, then copy contents of your Python object
(which you get at using buffer protocol) to this buffer, then initiate
the DMA transfer. At this point it probably would be easier to just
write a C-extension to do that, but I see no reason it can't be done
with ctypes.
However, I'm not sure how that would interact with your pre-existing
module. I'd expect a module that does DMA to take care of physical
address mapping itself, you just pass it a logical address, or an
object that supports buffer protocol, and it does the rest.
It seems just getting a physical address and starting a DMA transfer
from it is prone to danger if memory pages are discontiguous (and they
often are), but maybe OSes these days can handle that automatically.
Carl Banks
Tim Roberts
>
>I'm trying to see if there is any way I can make Python share data
>between two hosts using DMA transfers over a firewire connection, so
>avoiding the need for another layer on top such as IPv4 + Python sockets.
>
>Thanks to some old python bindings which I updated to python 2.6, I can
>read any write to the RAM of any firewire connected host within python.
>Because it uses DMA (the cpu is not involved in this at all), I can only
>specify a physical address within the 4GB ram limit to read from and
>write to.
> From what I've been told so far, it's not possible to do this without
>some OS-specific (Linux in this case) syscall. Is this correct?
Correct. User-mode programs cannot find physical addresess, and they
cannot lock pages into memory (which is required for DMA). You need the
assistance of a driver for this. The driver will find the physical
addresses.
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.