Emacs Remote Python Interpreters

[Matt McClure]

I'd like to use Emacs as my IDE for a Python project as follows:

1. Emacs running on a Mac host machine.
2. Python running on a Linux guest VM.
3. A single copy of the code shared between the host and guest
machines via NFS or VirtualBox shared folder.

Emacs on the host machine would use the guest VM's Python interpreter
for all execution, including code completion and debugging. I imagine
using directory-local variables to configure a remote Python
interpreter and a mapping between the respective host and guest
filesystem path prefixes.

What I'm looking for is pretty similar to [PyCharm's remote python
interpreters][1]. Has anyone built something similar for Emacs, for
Python environments or for any other language? I built a little
[prototype][2], but it's just a toy so far.

[1]: http://www.jetbrains.com/pycharm/webhelp/configuring-remote-python-interpreters.html
[2]: https://github.com/matthewlmcclure/emacs-remote-python

--
Matt McClure
http://www.matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure

[Andreas Röhler]

noted at

https://blueprints.launchpad.net/python-mode/+spec/remote

thanks,

Andreas

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

Hi Matt,

> I'd like to use Emacs as my IDE for a Python project as follows:
>
> 1. Emacs running on a Mac host machine.
> 2. Python running on a Linux guest VM.
> 3. A single copy of the code shared between the host and guest
> machines via NFS or VirtualBox shared folder.
>
> Emacs on the host machine would use the guest VM's Python interpreter
> for all execution, including code completion and debugging. I imagine
> using directory-local variables to configure a remote Python
> interpreter and a mapping between the respective host and guest
> filesystem path prefixes.

I don't use Python, therefore I don't know anything about Python
debugging. However, we have been able to use gdb and perldb remotely
from inside Emacs, via Tramp. No filesystem mounting necessary.

If you could tell me a little bit about how to use the Python debugger
in Emacs, I could try to help you with the setup in the remote
case. Does there exist a similar command like "M-x gdb"?

Best regards, Michael.

[Matt McClure]

On Tue, Jun 5, 2012 at 3:45 AM, Michael Albinus <michael...@gmx.de> wrote:
> I don't use Python, therefore I don't know anything about Python
> debugging. However, we have been able to use gdb and perldb remotely
> from inside Emacs, via Tramp. No filesystem mounting necessary.
>
> If you could tell me a little bit about how to use the Python debugger
> in Emacs, I could try to help you with the setup in the remote
> case. Does there exist a similar command like "M-x gdb"?

I think TRAMP is probably my best option right now. One problem with
using Python via TRAMP is python.el's dependency on emacs.py. When I
do M-x run-python from a TRAMP buffer, I get:

>>> Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named emacs
>>>

because python.el assumes it can add a local filesystem path to
PYTHONPATH and load emacs.py from there, whereas the python process
actually runs on the remote host when I start it from a TRAMP buffer.

Emacs has `M-x pdb` for Python. As an aside, why is "pdb" the default
value of gud-pdb-command-name rather than "python -m pdb"? The former
gives "no such file or directory" whereas pdb.py is present everywhere
I've looked, so the latter seems more reliable.

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

Hi Matt,

> I think TRAMP is probably my best option right now. One problem with
> using Python via TRAMP is python.el's dependency on emacs.py. When I
> do M-x run-python from a TRAMP buffer, I get:
>
> >>> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named emacs
> >>>
>
> because python.el assumes it can add a local filesystem path to
> PYTHONPATH and load emacs.py from there, whereas the python process
> actually runs on the remote host when I start it from a TRAMP buffer.

Hmm, maybe you could put emacs.py somewhere on the remote host. Then you
must instruct Tramp the remote PYTHONPATH, like

(add-to-list 'tramp-remote-process-environment
"PYTHONPATH=/path/where/emacs.py/is/located")

> Emacs has `M-x pdb` for Python. As an aside, why is "pdb" the default
> value of gud-pdb-command-name rather than "python -m pdb"? The former
> gives "no such file or directory" whereas pdb.py is present everywhere
> I've looked, so the latter seems more reliable.

No idea why this is the default. But you can configure it, I guess.

Best regards, Michael.

[Matt McClure]

Hi Michael,

On Tue, Jun 5, 2012 at 12:21 PM, Michael Albinus <michael...@gmx.de> wrote:
> Hmm, maybe you could put emacs.py somewhere on the remote host. Then you
> must instruct Tramp the remote PYTHONPATH, like
>
> (add-to-list 'tramp-remote-process-environment
>             "PYTHONPATH=/path/where/emacs.py/is/located")

python.el has some other TRAMP-compatibility problems. I started to
hack on them and keep notes here:

https://github.com/matthewlmcclure/tramp-virtualenv

I'd be curious to get some feedback from Emacs committers to find out
if this is on a track that could be merged upstream, or if I should
take a different approach.

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

> Hi Michael,

Hi Matt,

> python.el has some other TRAMP-compatibility problems. I started to
> hack on them and keep notes here:
>
> https://github.com/matthewlmcclure/tramp-virtualenv
>
> I'd be curious to get some feedback from Emacs committers to find out
> if this is on a track that could be merged upstream, or if I should
> take a different approach.

My first comments:

| 1. python.el - a replacement for the python.el distributed with Emacs
| 24 that makes its python-send-region TRAMP-aware.

Well, that code is based on a recommendation I gave last year in
<http://thread.gmane.org/gmane.emacs.tramp/7819>.

As I have said there, this won't be the final solution. And your change
(the call of a modified Tramp function) won't be accepted by the Emacs
maintainers. I believe it would be sufficient to apply `make-temp-file'
properly. I have committed a modified version of my initial proposal to
Emacs' trunk (plus a fix in Tramp, which did appear during my tests).

| 6. tramp-dir-locals.el - a hack to make directory local variables work
| in directories opened via TRAMP.

A harsh hack :-) The problem is already reported as bug#1933 and
bug#6731 in the Emacs bugtracker. I took the opportunity to apply the
proposed patch of bug#1933 to Emacs' trunk, slightly modified. If you
set in your .emacs

(setq enable-local-variables t)

remote .dir-locals files shall be evaluated, and your hack isn't
necessary anymore.

Could you, please, check whether these patches work for you? We will
investigate your other settings afterwards.

Best regards, Michael.

[Michael Albinus]

Michael Albinus <michael...@gmx.de> writes:

> (setq enable-local-variables t)

Cut'n'waste. You must set

(setq enable-remote-dir-locals t)

Best regards, Michael.

[Matt McClure]

Hi Michael,

On Wed, Jun 6, 2012 at 9:39 AM, Michael Albinus <michael...@gmx.de> wrote:
> Matt McClure <matthew...@gmail.com> writes:
>
> | 1. python.el - a replacement for the python.el distributed with Emacs
> | 24 that makes its python-send-region TRAMP-aware.
>
> Well, that code is based on a recommendation I gave last year in
> <http://thread.gmane.org/gmane.emacs.tramp/7819>.

I wasn't aware of that thread, but good to know we had a similar idea. ;-)

> I have committed a modified version of my initial proposal to
> Emacs' trunk (plus a fix in Tramp, which did appear during my tests).

Awesome, thanks! I'll give it a try as soon as the next nightly build
appears on http://emacsformacosx.com/builds.

[Matt McClure]

Michael,

On Wed, Jun 6, 2012 at 11:50 AM, Matt McClure <matthew...@gmail.com> wrote:
> On Wed, Jun 6, 2012 at 9:39 AM, Michael Albinus <michael...@gmx.de> wrote:
>> I have committed a modified version of my initial proposal to
>> Emacs' trunk (plus a fix in Tramp, which did appear during my tests).
>
> Awesome, thanks! I'll give it a try as soon as the next nightly build
> appears on http://emacsformacosx.com/builds.

When I try python-send-buffer in the 6/7 nightly build, I get:

Debugger entered--Lisp error: (error "Variable binding depth
exceeds max-specpdl-size")
signal(error ("Variable binding depth exceeds max-specpdl-size"))
byte-code(" \203 \305 G!\203 \n\306>\203 \307\207 \203& \305 G!\203& \n\310>\203& \207\311\f@\fA\"\207"
[completion localname operation filename err zerop (file-exists-p
file-directory-p) t (expand-file-name file-name-as-directory) signal]
3)
tramp-file-name-handler(expand-file-name "py"
"/scpc:vag...@192.168.33.10:/tmp")
expand-file-name("py" "/scpc:vag...@192.168.33.10:/tmp")
byte-code("\305\306 G!\203 \307 !\202 \310 \"! \203 \n P \f\203' \311\n!\210\2022 \312\313\314\n\314\315\314\316& \210\314\207"
[prefix temporary-file-directory file suffix dir-flag make-temp-name
zerop file-name-as-directory expand-file-name make-directory
write-region "" nil silent excl] 8)
make-temp-file("py" nil ".py")
python-send-region(1 708)
python-send-buffer()
call-interactively(python-send-buffer nil nil)

Is that the failure mode that motivated the comment in python.el prior
to your change?

;; `make-temp-file' would be the natural choice for
;; implementation. But it calls `write-region' internally,
;; which also needs a temporary file - we would end in an
;; infinite loop.

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

> Michael,

Hi Matt,

> When I try python-send-buffer in the 6/7 nightly build, I get:
>
> Debugger entered--Lisp error: (error "Variable binding depth
> exceeds max-specpdl-size")
>

> Is that the failure mode that motivated the comment in python.el prior
> to your change?
>
> ;; `make-temp-file' would be the natural choice for
> ;; implementation. But it calls `write-region' internally,
> ;; which also needs a temporary file - we would end in an
> ;; infinite loop.

Nope, that has nothing to do with. Looks like my change (making
temporary-file-directory remote) has undesired side effects.

Well, a clean solution would be to give make-temp-file a file name
handler in Tramp. I'll ask on emacs-devel about, and let you know when
implemented.

Best regards, Michael.

[Michael Albinus]

Michael Albinus <michael...@gmx.de> writes:

Hi Matt,

> Well, a clean solution would be to give make-temp-file a file name
> handler in Tramp. I'll ask on emacs-devel about, and let you know when
> implemented.

I stay corrected: I could fix it in Tramp. Committed to Emacs trunk.

Best regards, Michael.

[Matt McClure]

On Fri, Jun 8, 2012 at 9:32 AM, Michael Albinus <michael...@gmx.de> wrote:
> Michael Albinus <michael...@gmx.de> writes:
>> Well, a clean solution would be to give make-temp-file a file name
>> handler in Tramp. I'll ask on emacs-devel about, and let you know when
>> implemented.
>
> I stay corrected: I could fix it in Tramp. Committed to Emacs trunk.

Thanks. I'll give it a try again tomorrow.

[Matt McClure]

> On Fri, Jun 8, 2012 at 9:32 AM, Michael Albinus <michael...@gmx.de> wrote:
>>
>> I stay corrected: I could fix it in Tramp. Committed to Emacs trunk.

I applied the patch from
http://bzr.savannah.gnu.org/lh/emacs/trunk/revision/108526 to
http://emacsformacosx.com/emacs-builds/Emacs-2012-06-08-108519-universal-10.6.8.dmg
and I still get:

Debugger entered--Lisp error: (error "Variable binding depth exceeds
max-specpdl-size")

signal(error ("Variable binding depth exceeds max-specpdl-size"))
byte-code(" \203 \305 G!\203 \n\306>\203 \307\207 \203& \305 G!\203& \n\310>\203& \207\311\f@\fA\"\207"
[completion localname operation filename err zerop (file-exists-p
file-directory-p) t (expand-file-name file-name-as-directory) signal]
3)
tramp-file-name-handler(expand-file-name "py"
"/scpc:vag...@192.168.33.10:/tmp")
expand-file-name("py" "/scpc:vag...@192.168.33.10:/tmp")
byte-code("\305\306 G!\203 \307 !\202 \310 \"! \203 \n P \f\203' \311\n!\210\2022 \312\313\314\n\314\315\314\316& \210\314\207"
[prefix temporary-file-directory file suffix dir-flag make-temp-name
zerop file-name-as-directory expand-file-name make-directory
write-region "" nil silent excl] 8)
make-temp-file("py" nil ".py")
python-send-region(1 708)
python-send-buffer()
call-interactively(python-send-buffer nil nil)

I don't see tramp-compat-temporary-file-directory in the call stack. I
wonder, does Tramp need a tramp-handle-make-temp-file?

I'll try again tomorrow with the forthcoming 6/9 build in case I
missed something important when I applied your patch ad hoc.

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

Hi Matt,

> I applied the patch from
> http://bzr.savannah.gnu.org/lh/emacs/trunk/revision/108526 to
> http://emacsformacosx.com/emacs-builds/Emacs-2012-06-08-108519-universal-10.6.8.dmg
> and I still get:
>
> Debugger entered--Lisp error: (error "Variable binding depth exceeds
> max-specpdl-size")

Doesn't happen any longer to me, with that patch.

> tramp-file-name-handler(expand-file-name "py"
> "/scpc:vag...@192.168.33.10:/tmp")
> expand-file-name("py" "/scpc:vag...@192.168.33.10:/tmp")
> byte-code("\305\306 G!\203

> \"! \203

> [prefix temporary-file-directory file suffix dir-flag make-temp-name
> zerop file-name-as-directory expand-file-name make-directory
> write-region "" nil silent excl] 8)
> make-temp-file("py" nil ".py")
> python-send-region(1 708)
> python-send-buffer()
> call-interactively(python-send-buffer nil nil)
>
> I don't see tramp-compat-temporary-file-directory in the call stack.

Yes. Tramp prints some additional traces in tramp-file-name-handler in
case of an error, which shortens the backtrace, unfortunately. I have
fixed this in Tramp 2.2.5, which I will sync with the trunk when Emacs
24.1 is out.

> I wonder, does Tramp need a tramp-handle-make-temp-file?

Nope. It was sufficient, to patch tramp-compat-temporary-file-directory.

> I'll try again tomorrow with the forthcoming 6/9 build in case I
> missed something important when I applied your patch ad hoc.

Yes, please do. I have tested today with

| detlef:~/src/emacs> bzr log | head -8
| ------------------------------------------------------------
| revno: 108542
| committer: Eli Zaretskii <el...@gnu.org>
| branch nick: trunk
| timestamp: Sat 2012-06-09 14:53:31 +0300
| message:
| Fix parallel bootstrap build in lisp/ on MS-Windows.

Best regards, Michael.

[Matt McClure]

Michael,

On Sat, Jun 9, 2012 at 9:49 AM, Michael Albinus <michael...@gmx.de> wrote:
> Yes, please do. I have tested today with
>
> | detlef:~/src/emacs> bzr log | head -8
> | ------------------------------------------------------------
> | revno: 108542

Thanks! It works as I expect in Emacs-2012-06-10-108545-universal-10.6.8.dmg.

Matt

[Michael Albinus]

Matt McClure <matthew...@gmail.com> writes:

> Michael,

Hi Matt,

> Thanks! It works as I expect in Emacs-2012-06-10-108545-universal-10.6.8.dmg.

Great!. The other problems I've seen playing with your config seem to be
python.el related. I'm not an expert in this, and I wouldn't be of much help.

However, if you believe I could assist you, just ping me.

> Matt

Best regards, Michael.