Interested in project (GTGK channel)

[Edward Hui]

Hi community, this is Edward from Hong Kong. :)
I've been browsing the 2018 ideas list of Linux XIA, I found the project with Gatekeeper is extremely attractive for me!
With my prior knowledge of network and System administration, together with my rich internship experience, I have planned to setup and implement the basic example regarding Gatekeeper and crypto library in this weekend to dig into the topics and play with the repos. Can't wait to come back and discuss more, feel free to connect me via LinkedIn if you are interested.

https://www.linkedin.com/in/edward-hui-a45795119

[Qiaobin Fu]

Hi Edward,

Glad to hear your interests in our GT-GK channel project! If you have any questions, please let us know.

Best,

Qiaobin

[Edward Hui]

Hi Qiaobin,

I do have some compilation issues regarding the gatekeeper, after trying different combination, it is suggested the Kubuntu 16.04.3 LTS (or Ubuntu...) adapts the most suitable gcc and kernel version for building the project.

Meanwhile I will be looking into some reading regarding the principle of DPDK by order as following:

The reading materials I have discovered so far:

1. Slideshare of DPDK

2. VFIO & UIO

3. (Later) cryptodev_lib

4. (Later) L2 forward sample

It would be nice if I can kindly get the answer of the following questions:

1. Should I setup two virtual network interfaces in order to simulate the connections, or should I bind the dpdk to my own actual network interfaces?

2. What if it was <empty> in Network devices using DPDK-compatible driver

I tried to record my progress on my personal blog and it's still on the stage of setting up gatekeeper. 

Link to blog post

Thank you very much the time spent on handling my enquiry, your kind reply is much appreciated :)

Many thanks,

Edward

[Nishanth Dev]

Hi Edward,

  Good to see that you’re trying to sort out issues and studying up on the material, keep at it. I would suggest a word of caution: Always try to keep the problem in mind (while you’re learning), and how you would apply what you’re learning, towards solving the problem. (Keep track of your train of thought, it makes for a more enriched learning experience).  To answer your questions: 

1. Always better to setup virtual interfaces, at least during the setup phase. I’d strongly suggest a Ubuntu VM with generous RAM and storage, and work with that. It is quite possible that you could mess things up anywhere along the way (so I also suggest saving the VM machine state every so often). I’d especially refrain from configuring my actual machine, but that's up to you.

2. Since you’re setting up on Ubuntu, this link would help: https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-config-dev   

Let us know if you have any questions.

Best,

Nishanth 

--
You received this message because you are subscribed to the Google Groups "Linux XIA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-xia+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Edward Hui]

Hi community,

When I look into the the lua part of gatekeeper, I was wondering:

1. For the lua/net.lua, I was required to manually adjusted the network ports? For instance, there are front_ports and back_ports, where point to the dummy interfaces I supposed? (i.e. enp133s0f0 && enp133s0f1).

    May I know any suggested settings that the community has been implemented?

2. For the network interface binding, I have added two virtual NIC onto the virtualbox, enp0s8 & enp0s9 with the following setting:
    Intel PRO/1000 MT Server (82545EM) as Bridged Adapter
    For testing purpose, I have binded both to dpdk as uio_pci_generic.

My messages have been deleted and I'm not sure why but anyways, I have finished my internship and I will be updating more frequently, thank you very much for the time spent on reviewing my questions and I'm looking forward to hearing from you guys soon :) !



Many thanks,
Edward

[Edward Hui]

I have attached some of the setting for current status for easier communication:

For the mapping lua (if_map.lua)

return {
    ["enp0s3"] = "0000:00:03.0",
    ["enp0s8"] = "0000:00:08.0",
    ["enp0s9"] = "0000:00:09.0",
}

For the dpdk-devbind.py provided in dpdk

Network devices using DPDK-compatible driver

============================================
0000:00:08.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=uio_pci_generic unused=e1000,igb_uio
0000:00:09.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=uio_pci_generic unused=e1000,igb_uio

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=enp0s3 drv=e1000 unused=igb_uio,uio_pci_generic

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>

[Qiaobin Fu]

Hi Edward,

Glad to hear about your progress! For your questions:

1. Yes, you are required to adjust the ports in lua/net.lua. Basically, the incoming packets to the Gatekeeper-protected servers will first arrive at front ports, then probably be forwarded to either the Grantor (GT) server or the destination via the back ports. We are working a documentation about how to test gatekeeper, and will release more information about this. One possible tool is to use pktgen to generate test packets to Gatekeeper.

2. That seems fine.

If you have any questions, please let us know.

Best,

Qiaobin

[Edward Hui]

Hi Qiaobin,

Thank you for the quick and kind reply :)

I have just tested the dpdk setting using the example application compiled and it seems like the Hugepages are correctly configured?

EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: Master core has no memory on local socket!
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   probe driver: 8086:100f net_e1000_em
hello from core 1
hello from core 2
hello from core 3
hello from core 0

Since I will be waiting for the further release information, I will now look into the libressl and cryto library of dpdk first.
I will play with these libraries for now and hopefully to test the gatekeeper with the pktgen if the the testing is ready.

Many thanks,
Edward

[Edward Hui]

Hi Community,

I have finished my proposal draft, I would like to ask should I mention the implementation and principle of cryptography in my proposal?

Many thanks,

Edward

[Cody Doucette]

Hi Edward,

I think this is a good question. You should certainly mention any theoretical concepts or implementations of cryptography that you think are relevant, both because it would show us you have an understanding of cryptography and are thinking about how to use it for the project.

However, try to keep it succinct and relevant to the project. Also, since it's a complicated subject, if you have ideas about which cryptographic tools to use but are unsure, you can include them but also mention that more investigation is necessary to figure out what is most appropriate for the project.

Best,

Cody

--

You received this message because you are subscribed to the Google Groups "Linux XIA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to linux-xia+unsubscribe@googlegroups.com.

[Edward Hui]

Hi Community, 

For the usage of libressl, should we compile the binary first and do the key pair generation in bash script?

Or we should be performing these actions in the gt/main.c and ggu/main.c with the RSA_generate_key_ex provided in openssl/rsa.h respectively?

Many thanks,

Edward Hui

[Qiaobin Fu]

Hi Edward,

Up to now, we cannot tell which is a better option yet, as the project progress, we may have a clearer picture on this. If you can compare the pros and cons of the ways you implement it in your proposal, that could be a good point.

Best,

Qiaobin

[Edward Hui]

Hi Qioabin,

Thank you for the reply.

As exchanging public key between two devices are required, in order to create exchange packet for the first connection during run-time(my approach).

1. Is it possible to reuse the method alloc_and_fill_notify_pkt from Grantor by altering data?

2. For current situation, it is expecting the packet to wait rte_eth_rx_burst and then rte_eth_tx_burst, in order to send out the packet?

Many thanks,

Edward 

[Qiaobin Fu]

Hi Edward,

1. Yes, you can change that method to add the needed data.

2. Actually, rte_eth_rx_burst and rte_eth_tx_burst are DPDK specific functions to receive and send packets via the specified device. You can check the functions in DPDK website (http://dpdk.org/doc/api/rte__ethdev_8h.html) for more information.

Best,

Qiaobin

[Edward Hui]

Hi community,

I have finished the final draft of the proposal and submitted the draft on the GSOC portal, would I ask for a kind review regarding my proposal?

I have refined several technical details and thank you for the time spent on reviewing :)

Many thanks,

Edward Hui

[Qiaobin Fu]

Hi Edward,

I just reviewed your proposal, and left a few comments. Hope this helps!

Best,

Qiaobin

[Edward Hui]

Hi Qiaobin,

Thank you very much for the comments, I have updated the draft link in the portal to the edited proposal.

To better fit in the extra information suggested to the flow with consistency:

The discussion of DPDK's cryptodev library has been added in the timeline section Week 7 - 9

The explanation of technical testing has been added in the timeline section Week 10 - 11

Many thanks,

Edward Hui

[Qiaobin Fu]

Great! Good luck! :)

Best,

Qiaobin

[Edward Hui]

Hi community,

Thank you very much for the appreciation and select my proposal for the programme this year!!

I have uploaded the detailed timeline on GSOC timeline (tech blog) and I will be cleaning and rebuilding the new workspace for the project.

And I will be summarizing the doubts regarding technical details and post here in the following week.

Many thanks,

Edward Hui

[Nishanth Dev]

Hi Edward,

Congratulations on being accepted as a GSoC student!! Feel free to reach out to the community or to me personally if you have any questions. Will send you an offline email soon with getting started and general info about the GT-GK project.

Best,

Nishanth 

--
You received this message because you are subscribed to the Google Groups "Linux XIA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to linux-xia+...@googlegroups.com.

[Edward Hui]

Hi Community,

I have forked the master repos and starting to make first commit.

https://github.com/Edwardhk/gatekeeper

May I ask if it is possible to finish the whole project within the virtual box machines or do I have access to the existing development server in the future?

For AWS setup, I will be granted a free tier usage of the instance but the EC2 type m4.16xlarge is not included, is it necessarily to use the specific type for the gatekeeper?

Many thanks,

Edward Hui

[Cody Doucette]

Hi Edward,

I have instructions for setting up a VM that can run Gatekeeper, but I need to test it and clean t up a little bit. I will get that to you ASAP.

For EC2, you can only use the machines that have the option to use the Elastic Network Adapter (ENA). I don’t know if any of them are in the free tier, but I will investigate that as well.

Best,

Cody

--

[Edward Hui]

Hi Cody,

Thank you for sharing the instructions to me and take your time, I will be looking into other settings/library first.

Btw I think the ENA enabled instances is not included in free tier.

Many thanks,

Edward

[Edward Hui]

Hi community,

First of all, thank you very much for the complete guide from Cody and I have just completed the call with Nishanth.

Problems

 

I am trying to setup the vagrant and I have encountered the following problem using vagrant up:

==> default: Waiting for machine to boot. This may take a few minutes...

    default: SSH address: 127.0.0.1:2222

    default: SSH username: vagrant

    default: SSH auth method: private key

Then I have a timeout exception from the Vagrant, so I tried the following approach:

Approach 1:

To see if it is caused by the conflicts between version of Virtual Box and Vagrant.

I have tried to download both old version of Virtual box and Vagrant and do the mix n match between new and old version.

Result: However, the problem remain unchanged.

Approach 2:

To see if it is caused by the ssh and Virtual Box port forwarding

I have tried to install a fresh and clean 16.04 image from the Vagrant (bento/ubuntu-16.04), the same source as the Vagrantfile in the guide.

Vagrant.configure("2") do |config|

  config.vm.box = "bento/ubuntu-16.04"

end

Result: The vagrant up command successfully executed and I am able to use vagrant ssh later.

As a result, I think it may be problem related to the network port setup in the Vagrantfile. Will do more investigation on that, please feel free to leave any kind comments.

Progress update:

 

As the RSA_generate_key is found insecure and become deprecated, I have completed the setup of libressl and generate private key pair using RSA_generate_key_ex and the documentation of libressl is relatively rare on the internet.

And since the libressl was actually the forked version from openssl, the method is actually coming from the original openssl module (libcrypto) built IN libressl.

Please feel free to give any suggestions :)

Many thanks,

Edward

[Edward Hui]

Hi community,

 

Update

I have figured out the problem associated with the vagrant file, it is related to the network adapter setup as expected.

The following line has been confirmed to cause the problem related to ssh in vagrant up:

config.vm.network "private_network", ip: "10.0.10.2", name: "vboxnet0", nic_type: "virtio"

config.vm.network "private_network", ip: "10.0.11.2", name: "vboxnet1", nic_type: "virtio"

config.vm.network "private_network", ip: "10.0.12.2", name: "vboxnet2", nic_type: "virtio"

config.vm.network "private_network", ip: "10.0.13.2", name: "vboxnet3", nic_type: "virtio"

After the removal of the above lines, the vagrant up can be completed and we can use vagrant ssh afterwards.

vagrant@vagrant-ubuntu-trusty-64:~$

However, during the building of the virtual machine, two errors are founded, for instance:

1. The following network error should be normal as we have removed the network-related setup. 

==> default: virtual memory exhausted: Cannot allocate memory

==> default: make[3]: *** [virtio_net.o] Error 1

==> default: make[2]: *** [librte_vhost] Error 2

==> default: make[1]: *** [lib] Error 2

==> default: make: *** [all] Error 2

==> default: modprobe: FATAL: Module uio not found.

==> default: modprobe: FATAL: Module uio_pci_generic not found.

==> default: insmod: ERROR: could not insert module /home/vagrant/gatekeeper/dependencies/dpdk/build/kmod/igb_uio.ko: Unknown symbol in module

2. The only missing header error is found as following, I can't make conclusion now, will investigate more and report again. 

==> default: In file included from /home/vagrant/gatekeeper/main/main.c:32:0:

==> default: /home/vagrant/gatekeeper//include/gatekeeper_main.h:25:26: fatal error: rte_hash_crc.h: No such file or directory

==> default:  #include <rte_hash_crc.h>

==> default:                           ^

==> default: compilation terminated.

==> default: make[1]: 

==> default: *** [main/main.o] Error 1

==> default: make: 

==> default: *** [all] Error 2

The SSH command responded with a non-zero exit status. Vagrant

assumes that this means the command failed. The output for this command

should be in the log above. Please read the output to determine what

went wrong.

Many thanks,

Edward

[Cody Doucette]

Hi Edward,

For the first issue, I think your VirtualBox settings need to be tweaked. The setup_network.sh script doesn't add the same IP addresses that the Vagrantfile is expecting (10.0.10.2, 10.0.11.2, etc). If you go into your VirtualBox network settings and change vboxnet0 to be 10.0.10.0/24, vboxnet1 to be 10.0.11.0/24, etc, then I think these lines will succeed.

In any case, you can comment them out if you'd like. Then try destroying the machine and reinitializing it:

$ vagrant halt

$ vagrant destroy

$ vagrant up

It's possible that because the machine was not booted properly with the network configuration that the provision.sh script did not properly run to setup DPDK and gatekeeper. After booting the machine with a successful network configuration, you might see the second error go away.

Best,

Cody

--

You received this message because you are subscribed to the Google Groups "Linux XIA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to linux-xia+unsubscribe@googlegroups.com.

[Daivik Dave]

Hello Everyone,

             I was going through the gatekeeper project and some previous projects related to the gatekeeper from the Ideas List and I would like to contribute to the gatekeeper project . Is there any specific reason Edward was unable to complete the project during the GSOC period ? If no one is working on Protecting the GT-GK channel, i would like to do some research on the project and post a proposal for the same , otherwise i'll find some other XIA project to contribute to.

Thanks

Daivik

[Michel Machado]

Hi Daivik,

Edward didn't meet GSoC's schedule of evaluations. We're planning to make this project available for GSoC 2019. Given your experience with our team, I recommend you to work on Gatekeeper directly, that is, working on open issues and issuing pull requests. The experience you will accumulate working with Gatekeeper directly will help you to mentor this project during next GSoC.

A good entry point to start with Gatekeeper is focusing on issues with the label "good first issue". You find them here:

https://github.com/AltraMayor/gatekeeper/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

Taking advantage of this email, please reply to my email "Google's Open Source Blog" I sent last Friday, August 31st.

--

You received this message because you are subscribed to the Google Groups "Linux XIA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to linux-xia+...@googlegroups.com.

[Michel Machado]

Hi Daivik,

I have not received your reply to my email about the GSoC blog post. Could you reply it again? I'm only missing your authorization to forward to Google.

Congratulations on ingressing in your Masters degree program.

Join the Slack group as the other GSoC students did. It'll help you to keep in touch with our group.

On Thu, Sep 6, 2018 at 12:21 AM Daivik Dave <daivik...@gmail.com> wrote:

Hello Michel,
         I will look up in the Gatekeeper Issues page and find something to contribute to . Regarding the email you sent to me for the Open source blog, I have already sent a reply previously , if it hasn’t reached you , i’ll send it again.
        For this year I have taken admission in my Masters Degree, so for some financial purposes I’ll again like to participate in GSOC as a student this year. I’ll like to be a Student for Linux XIA again this year if they allow proposals for the same organisations twice. From 2020 onwards i’ll definitely like to be a mentor for this organisation and for the next succeding years. Besides that i’ll like to keep contributing to Linux -XIA out apart from GSOC.