How to debug iocage in PyCharm?

[rexpo]

I'd like to mess around a bit with iocage, and to that end I have set up PyCharm 2016.3, from packages / App Café, on my TrueOS laptop.

I can successfully run iocage from the IDE, but there are 2 big problems:

- iocage needs to be run as root - how should I get it to run as root in PyCharm?

- iocage likes command line parameters - how can I easily pass parameters in? (editing the run configurations each time is no fun)

Perhaps the answer to both of these problems is to launch the Terminal in PyCharm - but there is a third problem: when I launch the terminal all I get is a blinking cursor. No shell.

So just how does one go about debugging iocage in PyCharm?

[rexpo]

I have set up PyCharm for iocage development again today, this time in a fresh FreeBSD 10.3 VM (which results in PyCharm 2017.1 being installed). I ssh -X in to the VM to run PyCharm. I used the opportunity to polish up my notes, and am posting them here in case others can benefit from it, but also to explain exactly what I have tried so far, in the hope that someone can point out the magic that I'm missing.

In addition to what is in the notes, I can add that I have also run PyCharm as root (yuck) by allowing root to log in via SSH (double-yuck, I'm sorry dead kittens), and can confirm that I can now run and debug, without it complaining that I'm not root, by passing iocage command-line parameters in by editing the configuration.

Still, I have this nagging feeling that there is a much smarter way to do this. The kittens deserve better.

# Quick & Dirty PyCharm iocage Development in FreeBSD 10.3

Start by creating a silly little 2-CPU VM with 4GB RAM and a 20GB HDD

## Install FreeBSD 10.3

- From amd64 CD

- Full disk UFS

- doc, lib32, src, ports

- sshd, ntpd, powerd

- create a user for me, with `csh` shell, and add me to the `wheel` group

Patch FreeBSD:

```

freebsd-update fetch

freebsd-update install

reboot

```

Install some packages:

```

pkg upgrade

pkg install sudo screen xclock x11auth open-vm-tools

```

Run `visudo` to allow wheel dudes to sudo. Also in /etc/ssh/sshd_config, allow X forwarding.

Activate the open VM tools - put this stuff in /etc/rc.conf:

```

vmware_guest_vmblock_enable="YES"

vmware_guest_vmhgfs_enable="YES"

vmware_guest_vmmemctl_enable="YES"

vmware_guest_vmxnet_enable="YES"

vmware_guestd_enable="YES"

```

Enable ZFS:

```

sysrc zfs_enable=YES

```

Then power off the VM, and add a second 20GB hard drive. Power it back on and create a zpool with no mountpoint (cos that's how I roll):

```

zpool create -O canmount=off -m none tank /dev/da1

```

## Install Development Tools

Do this stuff as root:

```

pkg install git python36 llvm40

python3.6 -m ensurepip

pkg install intellij-pycharm

```

## PyCharm Setup

### PyCharm First Run

Do this as ordinary user. Notice that the version of PyCharm from FreeBSD packages is 2017.1 (as opposed to 2016.3 in the TrueOS repository). Go through the first few dialogs that pop up:

- Do not import settings

- Accept defaults

- Don't bother trying to set up a command-line thing, it'll just fail cos you're not root

### Create a VirtualEnv

When the main welcome dialog box comes up, don't try to open anything, let's go straight to Configure > Settings > Project Interpreter

- Hit the little gear wheel and select Create VirtualEnv

- Name it iocage-env

- Browse to /usr/local/bin/python3.6 for Base Interpreter

Now you can activate the venv in terminal with:

```

source ~/iocage-env/bin/activate.csh

```

Install Cython in the venv:

```

pip3.6 install Cython

```

Load the Cython stuff in PyCharm (do this as root outside the venv):

```

python3.6 /usr/local/share/pycharm/helpers/pydev/setup_cython.py build_ext --inplace

```

## iocage Development

In home directory, grab the iocage source:

```

git clone --recursive https://github.com/iocage/iocage

```

Build the ZFS libraries (inside the venv):

```

cd iocage

cd py-libzfs && python3.6 setup.py build && python3.6 setup.py install

```

Now open the ~/iocage in PyCharm. Open main.py and try to run it. It works a bit, then complains that it's missing some packages. Install them. (They end up nicely in the venv site packages)

If all is well, you should be able to run main.py now, and see in the console the standard usage message that iocage always gives when run without parameters.

To run it with a parameter, for example `iocage list`, open up iocage/main.py, go to Run > Edit Configurations... Python > main. In the script parameters field you can enter `list` (or whatever command you want to run). It says we need to run it as root, of course. So how to do that? And how to run it from a command line instead of the GUI?

## Debugging Tips

:TODO: I'm hoping you guys can help me fill in this part ;)

[Brandon Schneider]

Great write up and thanks for that!

I don’t personally use PyCharm to debug, just to write the code. So I’m of no help here, but am interested to see what others do. I just test the old fashioned way and with pytest. Being able to use PyCharm would be a real step up!

Thanks again,

- Brandon

--
You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/b669a1d1-6280-4404-9dc6-fa2f7dc67ed9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rexpo]

haha I guess I just assumed that you used PyCharm for everything, after having skimmed CONTRIBUTING.md :) anyway, thanks, I'm glad you liked the note.

So what's a pytest??

ah - I need to re-read the code testing README.md ;)

hmm. I keep getting `ERROR: file not found: iocage` when I try to run the tests. At first I thought it might be due to having run the tests in my PyCharm venv, but I get the same errors when running outside the venv. Then I said doh and realised that the error might be due to never having actually installed iocage on the machine :) Strangely, I still get the error after having installed it.

Any tips?

Here is the addition to my earlier-posted notes, showing what I did while trying to play with the pytests:

## Playing with pytest

Snapshot the VM so we can roll back in case this craps out.

According to the testing readme file, I should be able to go in to the tests/unit_tests directory and just run `pytest`. It craps out:

```

[iocage-env] jamie@pycharm:~/iocage/iocage/tests/unit_tests % pytest

============================= test session starts ==============================

platform freebsd10 -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /home/jamie/iocage-env/bin/python

cachedir: ../../../.cache

rootdir: /usr/home/jamie/iocage, inifile: setup.cfg

plugins: pep8-1.0.6, cov-2.4.0

========================= no tests ran in 0.00 seconds =========================

ERROR: file not found: iocage

```

Maybe there is some problem with me running it inside the PyCharm venv?

haha maybe not! The problem could be that iocage has never actually been installed on this machine... Ok, let's install it then.

## Install iocage from source, then try pytest

Unfortunately, I can't get the unit tests to run when I install iocage to the venv either, not even to root's venv. So let's try to install to the computer, not the venv.

Logged on as root user, install pytest, py-libzfs and iocage:

```

pip3.6 install pytest-cov pytest-pep8

pip3.6 install Cython

cd ~/iocage/py-libzfs && python3.6 setup.py build && python3.6 setup.py install

cd .. && pip3.6 install .

```

See that iocage installed:

```

root@pycharm:~/iocage # which iocage

/usr/local/bin/iocage

root@pycharm:~/iocage # iocage list

+-----+------+-------+-----+---------+-----+

| JID | UUID | STATE | TAG | RELEASE | IP4 |

+=====+======+=======+=====+=========+=====+

+-----+------+-------+-----+---------+-----+

```

Good. Now try a test:

```

cd ~/iocage/iocage/tests/unit_tests

pytest

```

Nope, same thing - file not found iocage:

```

root@pycharm:~/iocage/iocage/tests/unit_tests # pytest

============================= test session starts ==============================

platform freebsd10 -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /usr/local/bin/python3.6

cachedir: ../../../.cache

rootdir: /root/iocage, inifile: setup.cfg

plugins: pep8-1.0.6, cov-2.4.0

========================= no tests ran in 0.00 seconds =========================

ERROR: file not found: iocage

```

[Brandon Schneider]

It just comes down to time :P I wish I had some to set aside and also figure that out for myself. But alas, I'll continue banging rocks together until some cataclysmic event pushes me into getting into better habits.

As for your issue, this one is simple :) You're holding it wrong! But seriously, to run the unit tests, just cd to the directory you cloned, and run pytest as not root. It will run all the unit tests for you. If you want the whole battery to run, just run pytest as root.

No need to descend into test/unit_tests. Just ~/iocage should do.

- Brandon

--
You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/3568d0f6-e987-4c15-988f-89e7fa552f5a%40googlegroups.com.

[rexpo]

You're right, I was holding it wrong! I had mis-read the code testing readme, thinking I needed to go in to those subdirectories.

It runs now! I'm getting some kind of `Python 3 was configured to use ASCII as encoding for...` when trying to run pytest --zpool=tank but I think I can sort that out.

I've rolled back my VM's snapshot to just double-check some things, and it turns out that:

- iocage *does* need to be installed for the tests to work (duh)

- I needed to `pip install mock` to get the last missing module

I guess everybody who's anybody already has mock installed but maybe you want to add that to the code testing readme anyway.

In super-cool news, I can also say that after having installed iocage and mock in root's PyCharm venv, the tests also work there. nice.

Thanks for your help!

[Brandon Schneider]

No problem! 

So the ascii encoding is likely a locale issue. Make sure you are set up to use UtF-8. 

As far as the mock requirement, you should send a PR so your name can be attached to that addition, you did find it after all ;)

Thanks!

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/cf83fc45-7b21-41bd-9c39-1dd7c1034968%40googlegroups.com.

[rexpo]

Got it, thanks Brandon. UTF-8 is set up now, and that problem is gone.

I'm on a roll here, making nice progress with getting all the pytests to work, and continuing to polish my notes on how to set this all up as I go along. I've added a local anonymous FTP server populated with FreeBSD distribution files to my little VM. (Now that I have the tests partially working, I see why you suggest the --server="custom_server" option in the readme ;)

I'll look into the PR stuff when I need a break, it would be an honour :) I have a feeling there might be another addition or two though - I've wiped my zpool clean and the tests now crap out early on, in the test for activating iocage:

...test_activate...:

zpool = 'tank'

    @require_root

    @require_zpool

    def test_activate(zpool):

        runner = CliRunner()

        result = runner.invoke(activate_cmd, [zpool])

>       assert result.exit_code == 0

E       AssertionError: assert -1 == 0

E        +  where -1 = <Result UnsupportedOperation('fileno',)>.exit_code

I'll see if I can find the cause of this (unless you beat me to it :)

And just for kicks, I'll again include the exact steps used to get this far. Might as well, I've already written them up!

  -Jamie

## Test with pytest

The last step in getting our complete development environment up and running is making sure we can run all the nice test suites. 

### Prepare the Test Environment

Logged on as an ordinary user, `source` the `iocage-env` venv, cd to `~/iocage` and try running `pytest`. You'll see that a couple of rather important modules are missing: `iocage`, of course needs to be installed! We also need the `mock` module.

```

source ./iocage-env/bin/activate.csh

cd ~/iocage

pytest

```

Install `iocage` in the venv ***(Note the use of `sudo` here! Do it!)***:

```

cd ~/iocage

sudo pip3.6 install .

```

Install `mock` in the venv *(and that we're **not** using `sudo` here)*:

```

pip3.6 install mock

```

Now pytest can be run as an ordinary to do unit tests, or sudo'd to run the full battery.

### Unit Tests

As an ordinary user, inside the venv, do this to run only the unit tests:

```

cd ~/iocage

pytest

```

Notice that we get a message at the end about having skipped `conftest.py`, needing to be root to run. That's fine, that's cool.

### Functional Tests

As an ordinary user, we'll `sudo` to run basic functional tests:

```

cd ~/iocage

sudo pytest

```

Again `conftest.py` is skipped, but this time because a zpool wasn't specified. Glorious!

*(At this point, you might want to snapshot the VM, now that everything is **FINALLY** working! It's about to mess with a lot of ZFS stuff, you see...)*

And the full-on cavalcade of tests:

```

cd ~/iocage

sudo pytest --zpool="tank" --server="localhost"

```

Oh dear, it craps out on the test_activate test, complaining about a 'fileno' thingy. Find out why!

[Brandon Schneider]

Hey Jamie!

Interesting. I hadn't seen that error before for the test and nothing off the top of my head is able to figure out why. 

I just tested here myself and activates test does still work so it must be something interesting in your environment. If you find the reason please do report back. I'm wondering if it's pycharm related. I'd try this from the shell too. 

A thing I do for the tests is have a sacrificial zpool with a small disk added to the vm. Snapshotting the vm isn't strictly necessary, as at the end of the tests it runs iocage clean -af and will wipe everything iocage related off that pool. So you can use it on a root pool too. The additional one is just for me to easily separate testing and production. Which should help speed up your iteration. 

Isn't the server option great ;) it really helps speed things along heh. 

Good luck!

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/aed2316d-596f-4308-8f10-6f4e74f3e54a%40googlegroups.com.

[rexpo]

Yeah, there is definitely something funky in this PyCharm venv when I run either the pytests or iocage itself as a standard user using sudo. If, however, I log on as root and go through the entire procedure of creating the venv, grabbing the source, installing stuff, etc. etc. then all the tests run successfully. yay!

I still like the idea of sparing the kittens, and getting all this to work as a standard user with help from sudo, but that might be too-wishful thinking. I have a few more ideas for slight improvements though, we'll see if I can come up with something better.

I see your point about not needing the snapshots - my zpool was a bit of a mess before running the big pytest, but sparkling clean (empty!) afterwards. nice.

Anyway, I'm starting to see some light at the end of the tunnel. Thanks for all your help, Brandon!

[Brandon Schneider]

Fantastic! No problem :)

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/57114e31-4c62-40d1-acb4-11ae50121898%40googlegroups.com.

[rexpo]

Holy crap Brandon, look at this - and I didn't hurt a single kitten doing it:

I almost don't want to post this - it's early days and seems too good to be true - but DAYUM just look at that screenshot!

I'm sitting at my TrueOS laptop in the kitchen, but most of these windows are running in a FreeBSD 10.3 VM in the other room - I did a `ssh -X` into the VM as an ordinary user here.

A guided tour:

• Top left, my notes for how to set this all up. Not so exciting. Yet...
• Top right, PyCharm running in the VM, as an ordinary user. (Not as root, not even with sudo!)
• Bottom right, xclock running in the VM. (an old habit ;)
• The terminal on the left, SSH'd in to the VM, showing progress of the full cavalcade of iocage pytests
• The terminal on the right, SSH'd in to the VM, was me OCD'ing `zfs list` over and over while the pytests ran

I can run and single-step in the PyCharm GUI :) And on the CLI after having sourced the venv, I can run the pytests. Initially I ran `sudo pytest --zpool="tank" --server="localhost"` which resulted in 11.0-RELEASE being downloaded. I hit CTRL-C and ran again with `sudo pytest --zpool="tank" --server="localhost" --release="10.3-RELEASE`. (I didn't think starting an 11.0-RELEASE jail on a 10.3-RELEASE host was a good idea see, so that's why I aborted it - And that's why you see remnants of the 11.0-RELEASE datasets in the zfs listing)

So this is pretty sweet... Not only am I running PyCharm as an ordinary user, but everything I run inside PyCharm uses sudo python - so debugging iocage works. Best of all, it's contained within a PyCharm VirtualEnv! So both from the GUI and from the CLI (after having sourced the venv) I can play with iocage and run the pytests, (with sudo for the full battery or without for just unit tests), and it all just works. (so far seems to, anyway - fingers crossed and knocking on wood)

I'm going to spend a couple days to test some more, and verify & polish up my notes, and I'll report back.

I'd really like to share the notes once I'm finished, what would be a good way / where would be a good place to do this? As you can see, they are already in silky smooth Markdown format, so I'm flexible.

Time for a celebratory friday beer now I think :)

  -Jamie

[Brandon Schneider]

I do believe I'll have a beer to celebrate with you! Great screenshot ;) I may do exactly this actually. 

No you do not want to run a newer RELEASE for a jail then your host. Older is fine. Never newer. Good thing you stopped that. 

Maybe another PR idea for you to have iocage prevent that even ;) 

As for the notes, the wiki on GitHub seems like a good place. Something this extensive should probably have some blurb on the readme as setting up a whole environment for a new contributor is a big deal. We could even offer a vagrant or something. 

This is very cool stuff Jamie! Thanks for keeping us all in the loop. 

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/c221c153-b412-4e72-8738-5ef3a9e6065f%40googlegroups.com.

[rexpo]

hehe yeah, as I cancelled the download of 11.0-RELEASE I was thinking to myself that I have found a little bug ;) I'll get around to looking at this PR stuff soon, I want to finish up testing these notes first!

I agree, setting up a complete development environment is a big deal, and something you shouldn't have to "waste" your time helping us with - you have important work to do! But yeah it's not obviously easy, but important that it's done right. I've spent a week on this so far, but that's mostly because I like to make good notes as I go, rip it all down, build it all up again, and refine the notes some more. Plus I've never coded in Python or used a GitHubs before. Did I forget to mention that? ;P Anyway I'm certainly glad to help out with a project like this, especially when it's for a program that I'm very much looking forward to using.

The Wiki sounds like a good place for the notes. But what's a vagrant?

ok I seriously need to go find that beer now, enough computer fun for tonight.

  -Jamie

P.S. Don't worry, I'm just teasing (kind of) about the coding stuff - I have a computer science degree, it just needs a bit of dusting off ;)

[Brandon Schneider]

Well not so much a bug, I try not to tell the user what to do. But in this case, I probably should :D

I think anything that helps another person contribute to the project is very useful time spent indeed! But having somebody who's taking great notes for the class helps way more!

Don't worry about the Python or GIthub stuff, I'll help you out. I'm also new to Python ;) I just have great work buddies helping me through some questions when I hit a wall. In this case, the kittens are certainly going to remain safe :P

In regards to a Vagrant: https://www.vagrantup.com/ sums it up better then I can in an email. It has pretty pictures after all! But the TLDR is it fits this usecase just perfectly. We would just need somewhere to host it, and that becomes the issue. But it is essentially a one command virtualbox VM that will get setup and follow exactly the steps you put into it.

Nice and reproducible dev environments. I did this for iX, and it worked great. The only catch is it's Ruby. But if you know Python, you're gonna have no trouble picking that up. I fear once you see this, it will seriously get your nerd going and you may lose a lot of time researching and playing with it, and for that I apologize. Kinda. Sorry not sorry. Heheh.

I agree, enough computer time. Enjoy your beer, I'm going to also!

- Brandon

--
You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/5a3c4707-0336-468b-8cf6-7d0d74b5baab%40googlegroups.com.

[rexpo]

ok I spent 2 minutes reading about the vagrant, and then quickly closed the window again. That's a rabbit hole I'll have to save for later, it looks like too much fun!

Well as I feared, my breakthrough on friday was too good to be true. There were some big problems and inconveniences that cropped up. So I fixed them :)

The bottom line is that I can run PyCharm as an ordinary user, and a quick change of the Project Interpreter lets me jump from running/debugging as standard user or running/debugging as root. On the command line same deal. Pytests also. It really all seems to work now. Only caveats so far that I've found is that when debugging as root in PyCharm, you have to hit the stop button twice to stop it. The web page where I got the clue about how to debug as root mentions something similar, that you might end up with a pydev debugger process that hangs, but I haven't run into this so far.

The notes really need cleaning up, especially shuffling the order a bit, but I'll just post them as they are now just for kicks. We can talk about Wiki storage and stuff another time, because I am done for the day here!! ;)

iocage Development with PyCharm in FreeBSD 10.3

This document describes how to set up a complete PyCharm development environment for iocage in a FreeBSD 10.3 VM.

Once you have everything configured, you’ll be able to do these 6 magical things:

• Edit your code in PyCharm as a normal user
• Run and debug your code in PyCharm as a normal user or as root (without hurting any kittens!)
• Run and debug your code in Python on the command line if you wish
• Run the pytest suites as a normal user or root
• Run iocage on the command line without installing it (straight from the source tree)

We use the might and power of Pycharm Virtual Environments, plus a few sneaky scripts to make this all possible.

Buckle up, let’s get started!

Install FreeBSD 10.3

Let’s get started by creating a silly little 2-CPU VM with 4GB RAM and a 20GB HDD.

• From amd64 CD
• Hostname: pycharmvm
• Full disk UFS
• doc, lib32, src, ports
• sshd, ntpd, powerd

• create a user for me, with csh shell, and add me to the wheel group

Patch FreeBSD:

freebsd-update fetch
freebsd-update install
reboot

Install some packages:

pkg upgrade
pkg install sudo xclock x11auth open-vm-tools

Run visudo to allow wheel dudes to sudo. Also in /etc/ssh/sshd_config, allow X forwarding.

Activate the open VM tools:

Put this stuff in /etc/rc.conf:

vmware_guest_vmblock_enable="YES"
vmware_guest_vmhgfs_enable="YES"
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"

Make it boot faster:

echo "autoboot_delay=3" >> /boot/loader.conf

Enable UTF-8:

Edit the /etc/login.conf file. Add to the setenv part:

LC_COLLATE=C

And add this to the end of the default: section:

        :charset=UTF-8:\
        :lang=en_US.UTF-8:

then do as root:

cap_mkdb /etc/login.conf

Optional - VIMAGE Kernel

TODO: write the VIMAGE kernel stuffs

Enable ZFS:

sysrc zfs_enable=YES

Create our zpool:

Power off the VM, and add a second 20GB hard drive. Power it back on and create a zpool with no mountpoint (cos that’s how I roll):

zpool create -O canmount=off -m none tank /dev/da1

Set up local FreeBSD mirror FTP server

TODO: Add a local, lightweight web server to the mix, for the freebsd-update process

To save massive amounts of time during testing, we should have access to an on-premises FreeBSD mirror FTP server. Let’s just quickly create an anonymous FTP server inside our silly little VM.

We will need this directory structure in the ftp user’s home directory:

9.3-RELEASE
10.3-RELEASE
11.0-RELEASE

Put these files from each release in the each release directory:

MANIFEST
base.txz
doc.txz
lib32.txz

Create the ftp user, with username ftp. Just use adduser for this as usual.

Log on as ftp and start making some directories and files:

mkdir 9.3-RELEASE 10.3-RELEASE 11.0-RELEASE

Copy the distribution files in (just grab them from ftp.freebsd.org or install DVDs or whatever)

Enable the FTP server at boot, and configure it server with some appropriate options:

• Listen only on the loopback adapter
• Run in read-only mode
• Only allow anonymous logon
• Don’t use inetd, rather spawn ftpd as a daemon

sysrc ftpd_enable=YES
sysrc ftpd_flags="-ADr -a 127.0.0.1"

Start the service:

service ftpd start

Cool beans. Now we will save a lot of time and bandwidth when testing, as we can just specify localhost as the FTP server to download FreeBSD install files from.

Install Development Tools

Do this stuff as root:

pkg install git python36 llvm40
python3.6 -m ensurepip
pkg install intellij-pycharm

PyCharm Setup

Ideally, we run PyCharm as an ordinary user, and use sudo for running & debugging iocage, pytesting, etc. I have been struggling for days to get it to all work smoothly, but have finally come up with a nice way to do it that doesn’t harm kittens.

PyCharm First Run

Do this as ordinary user. Notice that the version of PyCharm from FreeBSD packages is 2017.1 (as opposed to 2016.3 in the TrueOS repository).

SSH in to the box, start PyCharm:

ssh -X pycharmvm.lan
pycharm &

Go through the first few dialogs that pop up:

• Do not import settings
• Accept defaults

• Don’t bother trying to set up a command-line thing, it’ll just fail cos you’re not root

Create a VirtualEnv

When the main welcome dialog box comes up, don’t try to open anything, let’s go straight to Configure > Settings > Project Interpreter

• Hit the little gear wheel and select Create VirtualEnv
• Name it iocage-env

• Browse to /usr/local/bin/python3.6 for Base Interpreter

• Don’t inherit global site-packages
• Hit ok and let it update the interpreter

You can later activate the venv in terminal with:

source ~/iocage-env/bin/activate.csh

Before you do that though, let’s add the final touch of configuring this venv to allow use of sudo when running python3.6.

Prepare for Run/Debug as root in PyCharm

Configure python3.6 in our iocage venv to run without password with sudo:

sudo visudo -f /usr/local/etc/sudoers.d/iocage-env-python

Add a line of the form:

<user> <host> = (root) NOPASSWD: <full path to python>

For example:

jamie pycharmvm = (root) NOPASSWD: /usr/home/jamie/iocage-env/bin/python

We need our PYTHONPATH environment variable preserved after the sudo so put this in there as well:

Defaults	env_keep += PYTHONPATH

Now create a script for launching python with sudo. Save it as python-sudo.sh in the venv’s bin directory:

#!/bin/sh
sudo ~/iocage-env/bin/python "$@"

And make it executable:

chmod +x ~/iocage-env/bin/python-sudo.sh

Now configure PyCharm to use this script. Assuming you’re back at the PyCharm welcome screen:

• Configure > Settings > Project Interpreter
• Click the little gear icon
• Choose “More…”
• Click the green plus icon to add a new interpreter (Add Local)
• Select your new python-sudo.sh script as the interpreter

That should be it! Now things run inside PyCharm should be run as root when you select this Interpreter. If you get an error complaining that it can’t set it up, that the SDK seems invalid, then check the path in your python-sudo.sh script, and verify that it actually runs without asking for a password - source the venv and test it!

Oh, one last thing that I have found solved problems with running in the CLI that were not a problem running from the GUI. Under Edit Configurations… there are a couple of boxes checked that add content and source roots to PYTHONPATH. We need to achieve somthing similar in the CLI venv, else it’ll complain that module iocage is not found.

Edit ~/iocage-env/bin/activate.csh, and add this line to it:

setenv PYTHONPATH "/home/<username>/iocage"

Install iocage Dependencies in PyCharm

Do all this stuff logged on as a standard user, with the iocage venv sourced.

Install Cython in the venv:

pip install Cython

Load the Cython stuff in PyCharm

(do this as root outside the venv):

python3.6 /usr/local/share/pycharm/helpers/pydev/setup_cython.py build_ext --inplace

iocage Development

In home directory, grab the iocage source:

git clone --recursive https://github.com/iocage/iocage

Build the ZFS libraries (inside the venv):

cd iocage
cd py-libzfs && python3.6 setup.py build && python3.6 setup.py install

Now open the ~/iocage in PyCharm. Just relax a bit and let it do its indexing things. Now open main.py. It works a bit, then complains that it’s missing some requirements. Install them. (They end up nicely in the venv site packages)

As a final touch is to create a “fake” iocage executable. Put this in ~/iocage-env/bin/iocage:

#!/bin/sh
~/iocage-env/bin/python ~/iocage/iocage/main.py "$@"

And make it executable:

chmod +x ~/iocage-env/bin/iocage

Now we can run a fake little iocage from the CLI, but it’s actually running straight out of our source tree. Run iocage on its own to run it as a standard user, or sudo iocage to run as root. Sassy!

If all is well, you should be able to run main.py in the GUI now, and see in the console the standard usage message that iocage always gives when run without parameters.

To run it with a parameter, for example iocage list, open up iocage/main.py, go to Run > Edit Configurations… Python > main. In the script parameters field you can enter list (or whatever command you want to run).

To run as root, just go to the Project Interpreter settings, and make sure the iocage-env (1) is selected. To run as standard user, select the other one.

Test with pytest

The last step in getting our complete development environment up and running is making sure we can run all the nice test suites.

Prepare the Test Environment

Logged on as an ordinary user, source the iocage-env venv, cd to ~/iocage and try running pytest. You’ll see that a module is missing: We need to install the mock module

source ./iocage-env/bin/activate.csh
cd ~/iocage
pytest

Install mock in the venv:

pip install mock

Now pytest can be run as an ordinary to do unit tests, or sudo’d to run the full battery.

Unit Tests

As an ordinary user, inside the venv, do this to run only the unit tests:

cd ~/iocage
pytest

Notice that we get a message at the end about having skipped conftest.py, needing to be root to run. That’s fine, that’s cool.

Functional Tests

As an ordinary user, we’ll sudo to run basic functional tests:

cd ~/iocage
sudo pytest

Again conftest.py is skipped, but this time because a zpool wasn’t specified. Glorious!

And the full-on cavalcade of tests:

Note: This will destroy all iocage datasets on the specified zpool! So specify your development pool here, not production!

cd ~/iocage
sudo pytest --zpool="tank" --server="localhost" --release="10.3-RELEASE"

Other PyCharm Tweaks

Not necessary to get all this working, but needed if you actually want to start contributing ;)

Markdown support for PyCharm

TODO: Markdown is boss, install the plugin for it

Brandon’s Code Nazi Editor Settings

TODO: set this up and export the config so it can later be pasted/imported for next time this all needs to be set up

Debugging Tips

Create Multiple Debug Configurations for Different CLI parameters

One way to go about passing in CLI parameters is to set up multiple debug configurations for main.py:

Go to Run > Edit Configurations. Select main. Let’s rename this to “01 - usage” and hit Apply. Now and hit the copy button. Name it for the command you want, for example: “02 - activate” and put in the CLI parameters in the Script parameters field, for example “activate tank”.

Some handy configurations to define:

• activate
• fetch 10.3
• create

Debugging and Running on the CLI

Running from the command line is simple enough. For example run the clean command:

cd ~/iocage/iocage
python-sudo.sh main.py clean --all

Better yet, make a script for that too. Save this as ~/iocage-env/bin/ioc and make it executable:

#!/bin/sh
sudo /usr/home/jamie/iocage-env/bin/python -i /usr/home/jamie/iocage/iocage/main.py "$@"

References

http://esmithy.net/2015/05/05/rundebug-as-root-in-pycharm/

[rexpo]

what the crap? Looks like half of it is missing there. Well I'll just post the raw Markdown version here:

# iocage Development with PyCharm in FreeBSD 10.3

This document describes how to set up a complete PyCharm development environment for `iocage` in a FreeBSD 10.3 VM.

Once you have everything configured, you'll be able to do these 6 magical things:

- Edit your code in PyCharm as a normal user

- Run and debug your code in PyCharm as a normal user or as root (without hurting any kittens!)

- Run and debug your code in Python on the command line if you wish

- Run the pytest suites as a normal user or root

- Run iocage on the command line without installing it (straight from the source tree)

We use the might and power of Pycharm Virtual Environments, plus a few sneaky scripts to make this all possible.

Buckle up, let's get started!

## Install FreeBSD 10.3

Let's get started by creating a silly little 2-CPU VM with 4GB RAM and a 20GB HDD.

- From amd64 CD

- Hostname: pycharmvm

- Full disk UFS

- doc, lib32, src, ports

- sshd, ntpd, powerd

- create a user for me, with `csh` shell, and add me to the `wheel` group

### Patch FreeBSD:

```

freebsd-update fetch

freebsd-update install

reboot

```

### Install some packages:

```

pkg upgrade

pkg install sudo xclock x11auth open-vm-tools

```

Run `visudo` to allow wheel dudes to sudo. Also in /etc/ssh/sshd_config, allow X forwarding.

### Activate the open VM tools:

Put this stuff in /etc/rc.conf:

```

vmware_guest_vmblock_enable="YES"

vmware_guest_vmhgfs_enable="YES"

vmware_guest_vmmemctl_enable="YES"

vmware_guest_vmxnet_enable="YES"

vmware_guestd_enable="YES"

```

### Make it boot faster:

```

echo "autoboot_delay=3" >> /boot/loader.conf

```

### Enable UTF-8:

Edit the `/etc/login.conf` file. Add to the `setenv` part:

```

LC_COLLATE=C

```

And add this to the end of the `default:` section:

```

        :charset=UTF-8:\

        :lang=en_US.UTF-8:

```

then do as root:

```

cap_mkdb /etc/login.conf

```

### Optional - VIMAGE Kernel

***TODO: write the VIMAGE kernel stuffs***

### Enable ZFS:

```

sysrc zfs_enable=YES

```

### Create our zpool:

Power off the VM, and add a second 20GB hard drive. Power it back on and create a zpool with no mountpoint (cos that's how I roll):

```

zpool create -O canmount=off -m none tank /dev/da1

```

## Set up local FreeBSD mirror FTP server

***TODO: Add a local, lightweight web server to the mix, for the freebsd-update process***

To save massive amounts of time during testing, we should have access to an on-premises FreeBSD mirror FTP server. Let's just quickly create an anonymous FTP server inside our silly little VM.

We will need this directory structure in the `ftp` user's home directory:

```

9.3-RELEASE

10.3-RELEASE

11.0-RELEASE

```

Put these files from each release in the each release directory:

```

MANIFEST

base.txz

doc.txz

lib32.txz

```

Create the ftp user, with username `ftp`. Just use `adduser` for this as usual.

Log on as `ftp` and start making some directories and files:

```

mkdir 9.3-RELEASE 10.3-RELEASE 11.0-RELEASE

```

Copy the distribution files in (just grab them from ftp.freebsd.org or install DVDs or whatever)

Enable the FTP server at boot, and configure it server with some appropriate options:

- Listen only on the loopback adapter

- Run in read-only mode

- Only allow anonymous logon

- Don't use `inetd`, rather spawn `ftpd` as a daemon

```

sysrc ftpd_enable=YES

sysrc ftpd_flags="-ADr -a 127.0.0.1"

```

Start the service:

```

service ftpd start

```

Cool beans. Now we will save a lot of time and bandwidth when testing, as we can just specify `localhost` as the FTP server to download FreeBSD install files from.

## Install Development Tools

Do this stuff as root:

```

pkg install git python36 llvm40

python3.6 -m ensurepip

pkg install intellij-pycharm

```

## PyCharm Setup

Ideally, we run PyCharm as an ordinary user, and use `sudo` for running & debugging iocage, pytesting, etc. I have been struggling for days to get it to all work smoothly, but have finally come up with a nice way to do it that doesn't harm kittens.

### PyCharm First Run

Do this as ordinary user. Notice that the version of PyCharm from FreeBSD packages is 2017.1 (as opposed to 2016.3 in the TrueOS repository).

SSH in to the box, start PyCharm:

```

ssh -X pycharmvm.lan

pycharm &

```

Go through the first few dialogs that pop up:

- Do not import settings

- Accept defaults

- Don't bother trying to set up a command-line thing, it'll just fail cos you're not root

### Create a VirtualEnv

When the main welcome dialog box comes up, don't try to open anything, let's go straight to Configure > Settings > Project Interpreter

- Hit the little gear wheel and select Create VirtualEnv

- Name it iocage-env

- Browse to /usr/local/bin/python3.6 for Base Interpreter

- Don't inherit global site-packages

- Hit ok and let it update the interpreter

You can later activate the venv in terminal with:

```

source ~/iocage-env/bin/activate.csh

```

Before you do that though, let's add the final touch of configuring this venv to allow use of sudo when running python3.6.

### Prepare for Run/Debug as root in PyCharm

Configure python3.6 in our iocage venv to run without password with sudo:

```

sudo visudo -f /usr/local/etc/sudoers.d/iocage-env-python

```

Add a line of the form:

```

<user> <host> = (root) NOPASSWD: <full path to python>

```

For example:

```

jamie pycharmvm = (root) NOPASSWD: /usr/home/jamie/iocage-env/bin/python

```

We need our `PYTHONPATH` environment variable preserved after the `sudo` so put this in there as well:

```

Defaults env_keep += PYTHONPATH

```

Now create a script for launching python with sudo. Save it as python-sudo.sh in the venv's bin directory:

```

#!/bin/sh

sudo ~/iocage-env/bin/python "$@"

```

And make it executable:

```

chmod +x ~/iocage-env/bin/python-sudo.sh

```

Now configure PyCharm to use this script. Assuming you're back at the PyCharm welcome screen:

- Configure > Settings > Project Interpreter

- Click the little gear icon

- Choose "More..."

- Click the green plus icon to add a new interpreter (Add Local)

- Select your new python-sudo.sh script as the interpreter

That should be it! Now things run inside PyCharm should be run as root when you select this Interpreter. If you get an error complaining that it can't set it up, that the SDK seems invalid, then check the path in your python-sudo.sh script, and verify that it actually runs without asking for a password - source the venv and test it!

Oh, one last thing that I have found solved problems with running in the CLI that were not a problem running from the GUI. Under Edit Configurations... there are a couple of boxes checked that add content and source roots to `PYTHONPATH`. We need to achieve somthing similar in the CLI venv, else it'll complain that module iocage is not found.

Edit ~/iocage-env/bin/activate.csh, and add this line to it:

```

setenv PYTHONPATH "/home/<username>/iocage"

```

### Install iocage Dependencies in PyCharm

Do all this stuff logged on as a standard user, with the iocage venv sourced.

#### Install Cython in the venv:

```

pip install Cython

```

#### Load the Cython stuff in PyCharm

(do this as root outside the venv):

```

python3.6 /usr/local/share/pycharm/helpers/pydev/setup_cython.py build_ext --inplace

```

## iocage Development

In home directory, grab the iocage source:

```

git clone --recursive https://github.com/iocage/iocage

```

Build the ZFS libraries (inside the venv):

```

cd iocage

cd py-libzfs && python3.6 setup.py build && python3.6 setup.py install

```

Now open the ~/iocage in PyCharm. Just relax a bit and let it do its indexing things. Now open main.py. It works a bit, then complains that it's missing some requirements. Install them. (They end up nicely in the venv site packages)

As a final touch is to create a "fake" `iocage` executable. Put this in ~/iocage-env/bin/iocage:

```

#!/bin/sh

~/iocage-env/bin/python ~/iocage/iocage/main.py "$@"

```

And make it executable:

```

chmod +x ~/iocage-env/bin/iocage

```

Now we can run a fake little iocage from the CLI, but it's actually running straight out of our source tree. Run `iocage` on its own to run it as a standard user, or `sudo iocage` to run as root. Sassy!

If all is well, you should be able to run main.py in the GUI now, and see in the console the standard usage message that iocage always gives when run without parameters.

To run it with a parameter, for example `iocage list`, open up iocage/main.py, go to Run > Edit Configurations... Python > main. In the script parameters field you can enter `list` (or whatever command you want to run).

To run as root, just go to the Project Interpreter settings, and make sure the iocage-env (1) is selected. To run as standard user, select the other one.

## Test with pytest

The last step in getting our complete development environment up and running is making sure we can run all the nice test suites. 

### Prepare the Test Environment

Logged on as an ordinary user, `source` the `iocage-env` venv, cd to `~/iocage` and try running `pytest`. You'll see that a module is missing: We need to install the `mock` module

```

source ./iocage-env/bin/activate.csh

cd ~/iocage

pytest

```

Install `mock` in the venv:

```

pip install mock

```

Now pytest can be run as an ordinary to do unit tests, or sudo'd to run the full battery.

### Unit Tests

As an ordinary user, inside the venv, do this to run only the unit tests:

```

cd ~/iocage

pytest

```

Notice that we get a message at the end about having skipped `conftest.py`, needing to be root to run. That's fine, that's cool.

### Functional Tests

As an ordinary user, we'll `sudo` to run basic functional tests:

```

cd ~/iocage

sudo pytest

```

Again `conftest.py` is skipped, but this time because a zpool wasn't specified. Glorious!

And the full-on cavalcade of tests:

***Note: This will destroy all iocage datasets on the specified zpool! So specify your development pool here, not production!***

```

cd ~/iocage

sudo pytest --zpool="tank" --server="localhost" --release="10.3-RELEASE"

```

## Other PyCharm Tweaks

Not necessary to get all this working, but needed if you actually want to start contributing ;)

### Markdown support for PyCharm

***TODO: Markdown is boss, install the plugin for it***

### Brandon's Code Nazi Editor Settings

***TODO: set this up and export the config so it can later be pasted/imported for next time this all needs to be set up***

## Debugging Tips

### Create Multiple Debug Configurations for Different CLI parameters

One way to go about passing in CLI parameters is to set up multiple debug configurations for main.py:

Go to Run > Edit Configurations. Select `main`. Let's rename this to "01 - usage" and hit Apply. Now and hit the copy button. Name it for the command you want, for example: "02 - activate" and put in the CLI parameters in the Script parameters field, for example "activate tank".

Some handy configurations to define:

- activate

- fetch 10.3

- create 

### Debugging and Running on the CLI

Running from the command line is simple enough. For example run the `clean` command:

```

cd ~/iocage/iocage

python-sudo.sh main.py clean --all

```

Better yet, make a script for that too. Save this as `~/iocage-env/bin/ioc` and make it executable:

```

#!/bin/sh

sudo /usr/home/jamie/iocage-env/bin/python -i /usr/home/jamie/iocage/iocage/main.py "$@"

```

## References

http://esmithy.net/2015/05/05/rundebug-as-root-in-pycharm/

[Brandon Schneider]

Hahaha code nazi. Weird the whole email showed up for me. One quick note that stuck out to me is do not run reboot in FreeBSD. It skips some nice things :) 

Use shutdown -r now instead. 

Great write up though! Very thorough. 

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/9d82a55e-4b3b-48f0-b80d-c4b554cbb2a4%40googlegroups.com.

[rexpo]

hehe thanks man :) This is just how I notate, especially when I'm dealing with stuff that gets done once-a-year. It doesn't take much extra effort to write it down, and thank god for Markdown - makes it *so* easy to make it pretty at the same time. As you can see, the notes are still kinda rough and colloquial, but not bad if I say so myself.

Yeah, after I posted the second time I saw that all of it was actually there... go figure. Rather than spam the list a 3rd time I just let it be.

Yep, thorough notes, but it's not as coherent as I'd like yet. However if you can follow them and get such a rig running, I'm telling you, it's really sweet to be able to single-step the iocages in PyCharm - it's really such a nice IDE! Also fun that the same can be done on the CLI with the "fake" iocage executable - with or without sudo. So you can hop between the two of them: Activate, fetch, create on the CLI, for example, and then single-step something specific inside PyCharm. Not to mention the pytests which also just work. You gotta have the workshop properly equipped before you start building furniture in it, I always say! :)

Thanks for the tip about reboot. You know, the only time I ever use that command is after a freebsd-update, for some reason. Otherwise I always use shutdown -p/-r now. Would you please enlighten me about why reboot is bunk? In any case, I'll change that part of the note right now.

So stay tuned, I'm not finished with these notes yet. I'm actually kinda torn between writing down a coherent recipe that, if followed to the letter, results in an IDE that's ready for business, versus a guide that shows why I made the decisions I made, showing all the pitfalls and reasoning behind the end product. Give it a day or two, I'll be back with more refinements.

Anyway, thanks for this great thread - a week and half ago I was googling my arse off trying to figure out how to debug iocage (as root) in PyCharm, and just look at it now - a recipe for how to do it all, served on a pewter platter to boot! Fantastic :)

[Brandon Schneider]

Indeed it's fantastic! 

As far as I know reboot skips telling users who are logged in. Along with this:

       The reboot utility does not execute the scripts in /etc/rcnum.d or exe-
       cute shutdown actions in	inittab(4).  To	ensure a complete shutdown  of
       system  services, use shutdown(1M) or init(1M) to reboot	a Solaris system

Ignore the Solaris bit. I'm fairly certain that's true for our utility as well. I've always been told to do it that way. Perhaps another on this list may know even more reasons. 

TLDR: reboot is faster and skips a lot of things that shutdown does not. 

 - Brandon

--

You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/a535aa7d-8d26-417d-8cd5-a619e1163fbb%40googlegroups.com.