Klink on Linux/MIPS

[Sean McNamara]

Hi Kevin,

Would it be possible for you to provide a build of Klink (the computer side, obviously) on Linux/MIPS? Specifically I'd like to run it on a Netgear WNDR3800 router running OpenWRT. The applications of this are obvious if you think about it for a little while; the physical characteristics of routers (shape, antenna gain, wifi characteristics) are far superior to those available in handsets or PC wifi cards, so this is clearly the best way to share a connection via Klink over a range longer than a couple feet :)

All you'd really need is a MIPS cross compiler (gcc's cross-compiler is available on many Linux distros) and the ability to compile all of your dependencies for this platform. Hopefully you didn't write any x86 assembly for the Linux/x86 Klink client, otherwise the porting effort could be quite onerous.

Providing a Linux client build of Klink for MIPS (32-bit) wouldn't only help that particular hardware; there are many other routers also using MIPS. And, once you successfully build it for MIPS, adding on ARM support (for the routers that run on an ARM processor) should be even easier, because once you start targeting multiple architectures, adding "just one more" gets easier and easier.

Of course, if Klink's client-side were open source as was alluded to as a possibility a very long time ago, I wouldn't even be asking you this - I'd have completed the port myself and gladly provided you any patches needed to get it working.

I'd be willing to either pay you $150 commission, or otherwise, to do the port myself (under NDA for the source code if you wish) and contribute all sources and binaries I produce back to you. If I were to do the port myself, I don't believe I would need the Android app's source code, only the Linux/x86 client source.

My handset manufacturer and/or carrier culled the free USB tethering over RNDIS in the Jelly Bean OTA update, but Klink works magnificently -- absolutely brilliantly! -- with my Windows 8 desktop. The only severe downside is that I can't connect my Nexus 7 tablet or my Roku set-top box to the network, because I can't get enough range out of the USB WiFi dongle on my PC. I am sure that the Netgear WNDR3800 could handle the range because it's designed to do so, but I can't currently get a connection to the WNDR3800 without making it hop "through" the ethernet port of my PC, which means I have to keep my PC on for any of the other devices to have connectivity; it also adds another point of failure and adds latency.

Klink may be deceptively lightweight and might in fact contain a lot of platform-specific dependencies; if that is the case, porting it to another architecture could be cumbersome. But since you produce a static binary with few dependencies, it looks very simple from the outside. So I really have no idea what I am asking here, except that if I had the source, I would employ all of my software engineering experience (which includes porting other programs across platforms and architectures) to get it done.

By the way, making the tun/tap kernel modules available in OpenWRT on the MIPS platform will be a trivial task, so I don't consider that an obstacle. And the router does have a USB 2.0 host port with working drivers.

Thanks,

Sean

[Kevin Ko]

Hi, Sean.

I have a few devices running OpenWRT and DD-WRT, so I'll see what can be done.  My Linksys routers use the bcm47xx branch, which also targets the MIPS architecture like the chipset in your Netgear router.

Klink's client should be portable, since I've already ported it to PowerPC for legacy versions of OS X.  A barrier might be ADB.  I've been wanting to rewrite things to talk directly to USB and bypass ADB; that might need to happen in this case.  We'll see how ADB behaves on the router I guess.

Kevin

[Kevin Ko]

Hi, Sean.

I managed to cross-compile adb and klink's client program to the MIPS architecture using the OpenWRT SDK from backfire 10.03.1.  However, I haven't gotten a chance to test anything out, since my two MIPS-based routers are running DD-WRT with an old version of uClibc: adb ends up segfaulting, so I hope that it is just a version problem.  

The routers are actively being used at the moment, so I'll have to swap out DD-WRT for OpenWRT at some more convenient time.  You can play with the binaries in the meantime, but I doubt that they'll work.

The directory structure should match that of Klink's regular linux client: put or symlink adb and klink under the /usr/share/klink path; then, run klink.

The tarball containing MIPS versions of adb and klink is at http://download.faveset.com/klink/temp/klink-openwrt-mips-uClibc-0.9.30.1.tar.gz

If you'd like to try building adb yourself, I did the following:

- Grabbed Android source

- Add a ./build/core/combo/HOST_linux-mips.mk file that is just a copy of the HOST_linux-x86.mk file with some minor additions:

====

HOST_CC := $(TOOLCHAIN_PATH)/usr/bin/mipsel-openwrt-linux-gcc
HOST_CXX := $(TOOLCHAIN_PATH)/usr/bin/mipsel-openwrt-linux-g++
HOST_AR := $(TOOLCHAIN_PATH)/usr/bin/mipsel-openwrt-linux-ar

LIBGCC_A := $(wildcard $(TOOLCHAIN_PATH)/usr/lib/gcc/*/*/libgcc.a)
LIBGCC_S := $(if $(wildcard $(TOOLCHAIN_PATH)/lib/libgcc_s.so),-L$(TOOLCHAIN_PATH)/lib -lgcc_s,$(LIBGCC_A))

HOST_GLOBAL_CFLAGS += -mips32 -mtune=mips32
HOST_GLOBAL_LDFLAGS += $(LIBGCC_S) -Wl,-rpath-link,$(TOOLCHAIN_PATH)/lib

====

TOOLCHAIN_PATH points to the OpenWRT SDK path (e.g., OpenWrt-SDK-brcm47xx-for-Linux-i686-gcc-4.3.3+cs_uClibc-0.9.30.1/staging_dir/toolchain-mipsel_gcc-4.3.3+cs_uClibc-0.9.30.1)

-There will be a compile error in build/libs/host/CopyFile.c related to the HAVE_STAT_ST_MTM define. I didn't bother tracking it down, so I just commented out that part so it would follow the cygwin path. (The obvious change in system/core/include/arch/linux-mips/AndroidConfig.h didn't do anything.)

- Then, build the x86 version of acp:

$ make out/host/linux-x86/bin/acp

- Build the MIPs version of adb

$ make out/host/linux-mips/bin/adb HOST_OS=linux HOST_ARCH=mips

- It will complain about acp not being executable, so you can just drop the x86 version in its place. (You could probably just get rid of acp for adb, since it's a copy tool.)

The binary will then be in out/host/linux-mips/bin/adb.

The instructions at http://lackingrhoticity.blogspot.com/2010/02/how-to-build-adb-android-debugger.html for building adb without fetching the entire source might also help.

Kevin

[Sean McNamara]

Hi Kevin,

You are truly amazing! :-)

My WNDR800 router is set to arrive at the house soon, and I will be promptly installing OpenWRT or a close fork, CeroWRT, on it.

I really appreciate this work.  I will report my results soon!

Regards,

Sean

[Sean McNamara]

Kevin,

Just as a note: I think we can use qemu or bochs, two open source cross-platform/cross-architecture emulators, to attempt to at least execute the binaries we build on MIPS without writing out the image to hardware.

Of course, you can't test end to end network connectivity (i.e., "does it work") in a VM with likely no network hardware support, but at least we could see the dynamic loader successfully spawning the process and linking up the libraries and perhaps providing command line output for both klink and adb.

My router is in the midst of the shipping channels, so I'll be attempting a qemu MIPS32 emulator of CeroWRT tonight. I'm going to try the image at: http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-26/ for the WNDR3800.

Many thanks,

Sean

[Sean McNamara]

Hi Kevin!

Here are my results:

1. I downloaded a DD-WRT v24-sp2 build for the Netgear WNDR4500v1 from http://www.myopenrouter.com/download/43858/DD-WRT-for-WNDR4500v1/

2. I grabbed the DD-WRT mipsel toolchain used to build the above from http://www.myopenrouter.com/download/38650/DD-WRT-Toolchain/

3. The adb build you provided needs librt, which isn't provided in the firmware. The toolchain I downloaded has librt, so I loaded up your adb, klink, and the toolchain (minus the static libraries) on my ext2-formatted USB flash drive and used:
LD_LIBRARY_PATH=the_toolchain ./adb nodaemon server

This got me further, but apparently the "atexit" symbol wasn't defined by the uclibc 0.9.32 build in the toolchain, so I ended up having to recompile adb after all due to this linkage problem.

I recompiled adb using the Ubuntu 12.10 android-tools-adb package sources from http://packages.ubuntu.com/quantal/android-tools-adb (they provide a standalone Makefile, which I only had to *lightly* modify by grabbing Android's upstream zlib sources and building them and hooking up the headers and lib into the Makefile). I also had to ship libz.so to the flash drive and the new adb binary.

4. This got me further. "adb" now runs without any errors, and is able to fork its server process into the background without complaint, and it'll run just fine. Klink can spawn adb (in the same directory) without copying it into /usr/share/klink, and everything IN USERSPACE looks cool. Even ./adb nodaemon server will just sit there silently (happily).

Unfortunately, "./adb devices" fails silently, and "./adb usb" returns "error: device not found", even when I have my phone plugged into the router and the handset says "USB Debugging Connected" and shows the little bug icon in the notification area.

Figuring it was a power problem, I switched to a powered USB hub, which makes my handset say "Charging" when the battery isn't 100%, but the problem remains. I *doubt* it is an ADB problem at this point; it may actually be a kernel problem (missing feature or something not compiled in). Or something *else* is up.

5. After ** A LOT ** of "hmming" and "hawing" about this problem, I eventually made the connection between this line in Android sources, system/core/adb/usb_linux.c line 689 and the PROBLEM:

        find_usb_device("/dev/bus/usb", register_device);

Ha ha! A hard coded path! And guess what. It didn't exist on my router.

I had a mini-panic, thinking I had an incompatible kernel or the wrong kernel version.

Nope.

# ln -s /proc/bus/ /dev/bus

As soon as I did this and re-ran Klink, it immediately started and sprung to life. ADB detected my device, and we're off to the races! Klink client running AND WORKING on MIPS! I'm writing you this message connected to my extremely fragile configuration over Ethernet.

The rest, as they say, is history. I had to set up some routing rules to set the default gateway to the Klink gateway (10.0.0.1) and set a static IP for the klink0 iface. I had to tweak iptables configuration to do IP masquerading from DD-WRT' "br0" (NAT interface) to the Klink interface. But...... IT WORKS!!!!!!!! It COMPLETELY works!

So, now, here are the steps that are needed (no custom firmware needed!), assuming the adb compile is already taken care of, and a few notes:

Note 1: The "tun" device is compiled statically into the Linux kernel in the firmware I'm using, so no need to insmod/modprobe it.

Note 2: I'm assuming DD-WRT v24 or thereabouts. Very old DD-WRT, OpenWRT, CeroWRT, etc. are going to behave very differently, and some of these steps may break. This stuff is running pretty close to the operating system and the hardware, so significant version changes can have... significant impacts ;)

Note 3: Depending on how many other USB storage devices you attach, and other version/hardware-specific settings, you may need to change the value assigned to the EXTDEVPATH variable in the script in step 5. On DD-WRT v24-sp2, the operating system automatically mounts the first partition of the first USB mass storage device to /tmp/mnt/disc0_part1, so that's what I'm using.

Hardware note: most handsets draw more energy from the USB port than a low-powered router can be expected to provide. Also, if you're doing this with a different router model and it doesn't have two USB ports like mine, this is absolutely essential. Either way, you need a good, *POWERED* USB hub: either for the extra juice, for the two ports, or both.

Another hardware note: You need a USB Mass Storage device that can be detected and mounted by your router's firmware. Try not to use anything too exotic. A common USB flash drive or small hard drive should be fine. You should need less than 100 MB of space on it.

1. Obtain a working copy of adb for your router (I can provide mine and my toolchain libs if anyone wants them) -- see the email above for details.

2. Download the Klink client binary for MIPS

3. Using your workstation, shove adb, klink, and any support libraries needed by adb onto an ext2-formatted flash drive or hard drive. Use a Linux computer / virtual machine / live CD to do the format, or maybe some Windows tool exists, I don't know.

4. Connect the USB flash drive to the router. It'll need to always be in the router when the router is booted, unless you figure out how to copy its contents into a RAMdisk and remove it (I haven't gotten that far yet).

5. Enable SSH or Telnet (preferably SSH) to your router. This is important, because SSH is slow.

6. You need to run the following commands. You can create a script on the flash drive with these commands and run it when you start the router, OR you can use a startup script in NVRAM so that this launches on boot (see http://www.dd-wrt.com/wiki/index.php/Startup_Scripts ):

#!/bin/sh

export EXTDEVPATH=/tmp/mnt/disc0_part1

export LD_LIBRARY_PATH=/tmp/stuff/lib

mkdir -p /tmp/stuff

ln -s /proc/bus /dev/bus

while [ ! -f ${EXTDEVPATH}/klink ]

do

sleep 5

done

cd ${EXTDEVPATH}

cp -rf adb klink lib/ /tmp/stuff/

cd /tmp/stuff

while [ /bin/true ]

do

/tmp/stuff/adb start-server

/tmp/stuff/adb wait-for-device

/tmp/stuff/klink &

export KLINK_PID=$!

sleep 5

ifconfig klink0 up

ifconfig klink0 10.0.0.3

ifconfig klink0 netmask 255.0.0.0

echo nameserver 8.8.8.8 > /etc/resolv.conf

route add default gw 10.0.0.1 klink0

iptables -I FORWARD -j ACCEPT

if [ ! -z $KLINK_PID ]

then

wait $KLINK_PID

fi

done

7. Now, here's where I'm stuck. Here's what works and doesn't work:

 * It WORKS if I try to access the public Internet from the router itself (SSH'ed in).

 * It WORKS if I have the router do NAT, so that you have a NAT on the router (e.g. 192.168.1.0 netmask 255.255.255.0) and a NAT inside Klink (e.g. 10.0.0.0 netmask 255.0.0.0). Even my computer can surf the web.

 * It DOESN'T WORK if I put Klink, eth0, eth1, vlan1 and vlan2 in an ethernet bridge (using brctl) trying to eliminate the redundant NAT on the router and use Klink's NAT directly. I configure my client PC with IP 10.0.0.4 netmask 255.0.0.0 gateway 10.0.0.1 (also tried 10.0.0.2 with a static route), and I configure the br0 interface on the router with IP 10.0.0.2 netmask 255.0.0.0, and set the default gateway as in the script above. This SHOULD be enough to create an ethernet bridge between Klink and my computer, right? Well apparently not because it doesn't work.

So basically if I try to get the router to act as a "Gateway" that just happens to have its downstream default gateway set to Klink, it works. But if I get the router to act as a Layer 2 "Switch/Hub" (the Linux software bridge), it doesn't work. Maybe I'm missing something. Maybe it's a problem with DD-WRT. But it sure is unpleasant, and I want to get to the bottom of it, because I'm under the (perhaps false) impression that there is lower latency and overhead with an ethernet bridge than with nested NAT layers..

Sorry for the long email. I'm just happy I got the Klink side to work at all. For now I'm running my system in the inefficient NAT -> NAT mode so I have a working internet connection, but I'd like to fix bridged mode. Is this a limitation of Klink that it doesn't work?

Thanks,

Sean

[Sean McNamara]

By the way: Klink randomly says "Klink is no longer connected." and I lose my internet connection. This happens about every 5 to 20 minutes fairly regularly. I haven't figured out why, yet, but dmesg doesn't indicate that the device is being disconnected at the USB level. So, physically speaking, the handset remains electrically connected to the router and the USB stack isn't kicking it off or autosuspending it (my DD-WRT 2.6.20 kernel doesn't even *support* USB autosuspend, as it wasn't implemented in the kernel for that revision of Linux).

Instead of running my script, I ran Klink in the foreground, backgrounded it, and went off to do my thing. It hasn't disconnected yet, and I didn't pass the -a or -r flag.

I think it disconnects when doing VoIP (Skype), but not when doing other bandwidth-hungry tasks. Very odd.

I also thought that maybe the "wait" command in my script could be waking up prematurely (spurious wakeup)? I tried the following in the interactive shell where Klink is running:

while kill -0 $KLINK_PID

do

echo "Waiting..."

wait $KLINK_PID

done

So far after over an hour of steady usage it's only echoed "Waiting..." one time, so I don't think the wait command is getting woken up prematurely.

I'm kind of lost as to why it would be dying when using VoIP but not when doing other reasonably intensive tasks. :(

Anyway, thanks!!!

Sean

[Kevin Ko]

Hi, Sean.

That's pretty cool!  I'll play around with my router later to see what happens.

Regarding bridging, your experience is normal for Klink or any non-root tether app.  Since Klink assumes a non-rooted environment, it can't mangle IP headers like a regular NAT as that requires raw socket access.  Instead, Klink translates everything into socket calls like a smart proxy.  As a result, only the device running the client program can communicate with the 10.0.0.1 gateway.  Anyway, I suspect that Klink's socket translation will be the latency bottleneck and not the extra ms or so for the additional NAT layer in your router. :)  (It's about 10-20 ms for TCP with a Galaxy S3, though it should be lower with UDP.)

Also, you could try to see if /usr/sbin/udhcpc (DHCP client) works instead of hardcoding things with ifconfig if you'd like things to look cleaner.

http://lists.busybox.net/pipermail/busybox/2011-June/076059.html

I'm not sure why you're losing connection with VoIP.  Though, I wonder if it has something to do with the ported version of ADB and USB reads/writes timing out.  You could try creating a debug build and see if any USB timeout error messages occur.  You probably could also run a keepalive ping to 10.0.0.1 to see if that changes anything while using VoIP.

Kevin

[Sean McNamara]

Kevin,

Through some more testing, I discovered that, when Klink says it is "no longer connected" (on the handset's notification bar), this coincides with the data dropping out **on the device** -- so, I immediately try to bring up Google Chrome and the homepage (or something) on the device when it drops out, and almost every single time, the data has dropped out on the phone, too. The few times where the phone had data, I think I waited too long, and the phone had already re-established its data connection.

If you examine my script, you notice I play with the routing table and iptables when I restart the Klink process. Do you know if these rules would remain in effect if the klink0 interface goes away when Klink dies? That might alleviate part of my problem. I think when I change the routing table and iptables it is interrupting all connections.

When I used to run the Windows Klink client, whenever I had a data drop on the phone, it would only temporarily lag my connections on the desktop -- but it wouldn't *kill* the TCP sockets. So if I had an open TCP socket somewhere, that socket would remain open; it'd just fail to send or receive data for a short time (10 - 45 seconds) and then pick up where it left off. This worked well for most things.

But now, whenever I get a data drop, I have to do whatever application-specific thing needs to be done in order to re-establish a TCP connection with the remote endpoint. And unfortunately, I get more frequent data drops when I use VoIP applications than, say, streaming music or video, surfing the web, etc.

There also seems to be a sort of specific spite towards Skype: if I use an alternative VoIP solution other than Skype, the data drops still occur, but they are MUCH less frequent. It's every 5 - 10 minutes with Skype, every 30 - 45 minutes with other VoIP solutions, and about every 2 - 3 hours with plain jane web surfing (HTML/AJAX/CSS, but no video or audio).

The solution overall is definitely impressive -- I'm impressed that it works at all -- but I need to work out the k(l)inks. ;-) Just wanted to let you know that, yes, Klink and ADB can be shoe-horned into working on a stock DD-WRT firmware, and they'd probably run even better on an OpenWRT firmware (but my router doesn't have OpenWRT builds available due to lacking hardware support - I think the Linux CPU support code for this Broadcom SoC isn't compatible with newer kernels than 2.6.20.)

Oh, one last thing, since you may be interested in performance. My device, which is kind of slightly above average for a router with a MIPS 74k processor (about twice as fast as a WNDR3800), reports 7.5% steady CPU usage from adb when Klink is in use, and Klink itself uses about 0.2% CPU with the occasional spike to 1 or 2%. This is with light data transfers a la the web and several TCP sockets open (browser prefetch...). I'm betting that most of the CPU is in the kernel, deciding how to route all the packets at layer 3 ;-)

Thanks,

Sean

[Kevin Ko]

Hi, Sean.

Your situation seems strange.  I don't see why a cell radio dropout would have any bearing on the USB connection to the phone.

Stopping Klink's client program, and thus taking down klink0, does seem to remove any associated routing table rules.  I suspect that the behavior is similar under Windows/Linux.  However, the USB connection is usually preserved across data dropouts when connected to a regular computer.  For whatever reason, the router doesn't behave this way.

If I recall correctly, Skype's protocol is UDP-based, so that might explain the difference in behavior when compared to your streaming TCP-based applications.  I think that most real-time VoIP programs use UDP, so that doesn't explain why Skype should be any different.  Moreover, Klink's client program is very trivial in functionality, so I don't see why UDP or TCP would make a difference on the router versus desktop Linux; either way, it just encapsulates IP packets from the TAP/TUN device and shuttles it over to Klink on the phone.

Could you try running an "adb logcat" concurrently with Klink on the router?  That should help us determine which part is acting up.  If the logcat output halts when the data drops, then it's likely a USB or ADB issue.  However, if logcat continues to output things without issue, then there might be some strangeness in the MIPS build of Klink.

Kevin

--
You received this message because you are subscribed to the Google Groups "Klink" group.
To unsubscribe from this group and stop receiving emails from it, send an email to faveset-klin...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.