is tmate the right tool for my remote managment scenario, when SSH is block on my machine

699 views
Skip to first unread message

stefan.m...@gmail.com

unread,
Feb 2, 2020, 9:12:18 AM2/2/20
to tmate.io

Hello,
it took me some time to find apps to enable me to share a shell in a browser.


Me and friend managing some simple VPSs that shall be managed from a central management server to simplify workflow.
There are cases that we still need a shell and allowing us to share this session would be extremely helpful.
In addition, when I'm at work, SSH is blocked so it quite challenging to manage them.


So is tmate the tool to get around this?


The architecture is close to this one:
image

thank you for answering my humble question which answer might be obvious but due to my endless searching I cannot see the woods for the trees.

Stefan

Nicolas Viennot

unread,
Feb 2, 2020, 9:20:42 AM2/2/20
to stefan.m...@gmail.com, tmate.io
Hello Stefan,

I think tmate is a good solution for your usecase. Here's how you do it:
1) Register an API key on tmate.io (no information needed aside from an email).
2) On each machine, run the tmate service with `tmate -F -n hostname_of_machine -k api_key`
3) You'll be able to connect to a machine at `https://tmate.io/t/your_username/hostname_of_machine` any time. Provided that your company doesn't block websockets

Nico

--
You received this message because you are subscribed to the Google Groups "tmate.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tmate-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tmate-io/58ed180f-113e-496d-9418-67dc5bc0b52e%40googlegroups.com.

Stefan Müller

unread,
Feb 2, 2020, 3:13:23 PM2/2/20
to Nicolas Viennot, tmate.io

Hi Nico,

thx for the quick reply, didn't expect that :)

1) Register an API key on tmate.io (no information needed aside from an email).

that would mean I use your server as relaying service, wouldn't it? I could setup my own server on the management server, could I?


Provided that your company doesn't block websockets

Any test server around I could try to connect to? That would exclude config errors on the machine, which run the tmate service.

Stefan

Nicolas Viennot

unread,
Feb 2, 2020, 3:28:23 PM2/2/20
to Stefan Müller, tmate.io
On Sun, Feb 2, 2020 at 3:13 PM Stefan Müller <stefan.m...@gmail.com> wrote:

Hi Nico,

thx for the quick reply, didn't expect that :)

1) Register an API key on tmate.io (no information needed aside from an email).
that would mean I use your server as relaying service, wouldn't it? I could setup my own server on the management server, could I?
Yes, and Yes.
To host your own servers, follow the section "Development environment", and move to production. It's pretty involved though.
Provided that your company doesn't block websockets
Any test server around I could try to connect to? That would exclude config errors on the machine, which run the tmate service.

Just install and run "tmate -F" on your local machine. You'll get a random session URL.

stefan.m...@gmail.com

unread,
Feb 5, 2020, 11:53:31 AM2/5/20
to tmate.io

Hi Nico,
thx for the update.
You always got me saying 
Yes, and Yes.

but than you lost me again :).

that would mean I use your server as relaying service, wouldn't it? I could setup my own server on the management server, could I?

1) Register an API key on tmate.io (no information needed aside from an email).

Yes, and Yes. To host your own servers, follow the section "Development environment", and move to production. It's pretty involved though.

so I follow "Host your own tmate servers" with help given in https://groups.google.com/forum/#!topic/tmate-io/C5vhLKk1GcM.

If I want to reduce time spending on updating the container I follow "Development environment" and  https://docs.tilt.dev/docker_compose.html


Provided that your company doesn't block websockets

Any test server around I could try to connect to? That would exclude config errors on the machine, which run the tmate service.

Just install and run "tmate -F" on your local machine. You'll get a random session URL.


I do have some problems to understand the architecture, in particulate as I'm afraid that expression are mixed, so I try it to explain it in my own words, from scratch.

Linux runs through the startup process means:

  1. Power on -> System startup(Hardware )
  2. CPU jumps to BIOS
  3. BIOS runs POST etc
    • BIOS stands for Basic Input/Output System
    • POST stands for Power On Self Test
    • Performs some system integrity checks
    • Searches, loads, and executes the boot loader program.
  4. Finds first bootable device
  5. Load boot loader
    1. Boot loader Stage 1 (MBR loading)
      • MBR stands for Master Boot Record.
      • It is located in the 1st sector of the bootable disk. Typically /dev/hda, or /dev/sda
      • It contains information about GRUB (or LILO in old systems).
      • So, in simple terms MBR loads and executes the GRUB boot loader.
    2. Boot loader Stage 2 (GRUB loader)
      • GRUB stands for Grand Unified Bootloader.
      • If you have multiple kernel images installed on your system, you can choose which one to be executed.
      • GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
      • Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this). The following is sample grub.conf of CentOS.
  6. Load OS
    1. Kernel
      • This can be considered the heart of operating system responsible for handling all system processes.
      • Mounts the root file system as specified in the “root=” in grub.conf
      • Kernel executes the /sbin/init program

      • Since init was the 1st program to be executed by Linux Kernel, it has the process id (PID) of 1. Do a ps -ef | grep init and check the pid.
      • initrd stands for Initial RAM Disk.
      • initrd is used by kernel as temporary root file system until kernel is booted and the real root file system is mounted. It also contains necessary drivers compiled inside, which helps it to access the hard drive partitions, and other hardware.
    2. Init or Systemd in newer distros
      1. Init
        • Executes the system to boot into the run level as specified in /etc/inittab
        • Looks at the /etc/inittab file to decide the Linux run level. 
        • Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate programs. 
        • Execute grep initdefault /etc/inittab on your system to identify the default run level, Typically either 3 or 5.
      2. Systemd
        • Systemd is the mother of all processes and it is responsible for bringing the Linux host up to a state in which productive work can be done. 
        • First, systemd mounts the filesystems as defined by /etc/fstab, including any swap files or partitions. At this point, it can access the configuration files located in /etc, including its own.
        • It uses its configuration file, /etc/systemd/system/default.target, to determine which state or target (similar to runlevel), into which it should boot the host.
        • The default.target file is only a symbolic link to the true target file. For a desktop workstation, this is typically going to be the graphical.target, which is equivalent to runlevel 5 in the old SystemV init. For a server, the default is more likely to be the multi-user.target which is like runlevel 3 in SystemV. The emergency.target is similar to single user mode.
        • Note that targets and services are systemd units.
      3. Runlevel
        • A no. of runlevel scripts are defined inside /etc/rc.d/rcx.d
        • When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level.
        • Please note that there are also symbolic links available for these directory under /etc directly. So, /etc/rc0.d is linked to /etc/rc.d/rc0.d
        • /etc/rc.d/rc*.d/ directories, you would see programs that start with S..Startup and K.Kill
        • There are numbers right next to S and K in the program names. Those are the sequence number in which the programs should be started or killed
        • For example, S12syslog is to start the syslog deamon, which has the sequence number of 12. S80sendmail is to start the sendmail daemon, which has the sequence number of 80. So, syslog program will be started before sendmail.
  7. User
    • The expresions console, terminal and shell are understood as described in What is the difference between shell, console, and terminal? - Super User and well illustrated on Linux Terminal and Console Explained For Dummies - LinuxBabe, What's the difference between a console, a terminal, and a shell? - Scott Hanselman and Linux Virtual Console and Terminal Explained
      • Mainframe = includes devices like CPU, RAM, HDD and so on.
      • Terminal = Input device + Output device, these are standalone, means not integrate into the Mainframe 
        • Terminal runs a shell, a very simple program, without much of OS etc. in the past. It actually processes commands and returns output. So in a nutshell a monitor plus a  keyboard an little CPU which runs a simple program to understand the commands and run them against the host.
        • the old terminal concept was abstracted into software the so called Terminal Emulators 
        • Linux Terminal Emulators are the software version of physical terminal, thus a virtual terminal but the OS thinks they are real as well
      • Console = terminal attached to the host directly 
        • It is used by administrators only
        • Unlike a terminal to the terminal the console can e.g. 
          • if the operating system failed to boot up, there would be error message displayed on the console, but not on other terminals. 
          • if the operating system boot into single-user mode, you can only use console to login. Other terminals have no permission to login in single-user mode.
        • In the past there were multiple consoles dedicated to certain tasks (group of processes, and many I/O)  and / or persons
        • virtual consoles are the same as terminal emulators for terminals
        • Nowadays, by default, Linux start 6 or 7 tty, console terminals. Each has its own command history, child processes, if you start a program in this console and user session etc.The output of these is piped to standard streams, standard input (stdin), standard output (stdout) and standard error (stderr). 
      • Computer = Mainframe + Terminal
      • A shell is a program that puts up a prompt and waits for you to type commands. It executes them and then prints another prompt. So, like CMD in Windows, or Bash in Unix. It can run in a terminal or on the console. 
    1. Getty and Login
      • init or systemd start a getty process for each connected terminal
      • Getty is the program that enables you to log in through a serial device such as a virtual terminal, a text terminal, or a modem. It displays the login prompt. Once you enter your username, getty hands this over to login which asks for a password, checks it out and gives you a shell.
      • This shell is spawend and thus inherits the environment (network tack, namespace etc.) of the uses console.
    2. Bash or other shell
      • You will see you the shell what is usually Bash, what waits for input / commands. 
      • Some of them use a different default prompt for the root use but you should see something like root@computername:~$ what means I'm user root ($) and sitting in my home folder (~)
  8. ...

my sources:


Now let us continue with tmate


I'm in my console as root and install tmate as shwon on https://tmate.io/. 

I can now start tmate by either

  1. tmate
  2. tmate -F

In case 1, the client is started (not a server as said by "tmate -F...starts the server side of tmate") this creates an SSH connection to the mate server what is either either tmate.io or you own tmate/tmate-ssh-server and, if required tmate Webhooks.

This is the standard procedure as shown in many tutorials:

That triggers that there is a pseudoterminal (pty) started on the client computer, the remote machine we want to control. This pty acts a pty slave, which provides an interface that behaves exactly like a classical terminal. The tmate client attach a process to the pty , what is in most cases a shell. The input to the program  attached at the pty master end, what is on the tmate server . Anything that is written on the master end is provided to the process on the slave end as though it was input typed on a terminal, so the program on the client computer is driven by a program that is attached to the master. 

I'm repeating more or less what is written on pty(7): pseudoterminal interfaces - Linux man page. 


In case 2 the tmate client is started as foreground process. "Foreground processes refer to applications you are running that you are currently interacting with, and which applies equally to graphical user interfaces as it does to the command line" taken from Getting Started With Linux - Foreground vs Background Processes - blog.100tb.com.
I don't see any advantage of this as streams are running between the pty and shell and from the to the system and back are involved. As soon as I run a shell stdin, stdout stderr are involved anyway. It might be helpful if you run another shell, what is not initiated by tmate and want to stream the remote input to local shall. Is that what you are saying?


It is quite challenging to understand the architecture. I read "Technical Details" and the paper "tmate: A scalable UNIX terminal sharing/management tool". The process above is quite simplified but it is more or less the route isn't although you run certain process in isolated non-privileged namespaces and using UNIX-sockets for on inter-computer-communication to increase security. 

The only diagram I found is the one on Tmate - Securely Share SSH Terminal Session with Linux Users. May you copy that and create a diagram on draw.io what allows to store the diagram files on GitHub.
The folks of Mosh made a nice slide showing their concept, see Mosh: An Interactive Remote Shell for Mobile Clients - YouTube


I still do not understand why why the tmux client/server architecture is not compatible across an IP network as ssh to IP works as well. 

 

Back again to security. It is said that tmate server container needs to run with --cap-add=SYS_ADMIN what is for security reasons, but still don't like it:

Unfortunately docker-stack does not support cap-add what could me allow to bring up and down using portainer.

Does is it apply for the client as well?
So I wonder how can I activate tmate on demand without shell access?
Any safer possibility using docker? 

What I can think of is to use a Treafik revers proxy what reacts to docker label. May protainer see the tmater container when start it manually, so I can  dis-/connected from the web changing the label of the container using portainer.


Talking about docker container, are you aware of the other docker images:

  1. https://github.com/nicopace/tmate-docker
  2. https://github.com/fgimenez/fenced-tmate

any advance using them? 


Last but not least, as you mentioned TILT. Any advice when running on swarms, as TILT works only with compose and is not supported by portainer.

There is some useful read on docs.docker :https://docs.docker.com/engine/swarm/swarm-tutorial/rolling-update/ but haven't tested it yet following the examples in https://docs.tilt.dev/docker_compose.html.

Portainer is yet able to do so: https://github.com/portainer/portainer/issues/3197




Sorry for all these question but the more I read and searched the more question marks arose. 


Stefan





Message has been deleted

stefan.m...@gmail.com

unread,
Feb 11, 2020, 6:22:09 AM2/11/20
to tmate.io
Hi Nico,
was it too much I asked for or too noob-wise?
Shall I rephrase or do you need some time to compile an appropriated answer? 

Thank you

Stefan

stefan.m...@gmail.com

unread,
Feb 17, 2020, 12:38:18 PM2/17/20
to tmate.io
Hi,
I found some explanation in http://viennot.com/tmate.pdf what is mentioned in the Article on tmate, whereby the architecture diagram is not available anymore.
Unfortunately, this explanation is not available any more.

Stefan

Nicolas Viennot

unread,
Feb 17, 2020, 9:35:00 PM2/17/20
to Stefan Müller, tmate.io
I had a bit of trouble parsing your email, but I think I found your questions:

> I still do not understand why the tmux client/server architecture is not compatible across an IP network as ssh to IP works as well. 

Because it sends file descriptors though a UNIX socket. That's not possible via IP sockets.

> [cap_sys_admin] Does is it apply for the client as well?

No

> So I wonder how can I activate tmate on demand without shell access?

Yes, run it as a system daemon, like with systemd

> Any safer possibility using docker? 

Sure: docker run -it tmate/tmate tmate

> [docker containers] any advice using them? 

I wouldn't recommend them. The officials ones are fine.

Thanks,

Nico


--
You received this message because you are subscribed to the Google Groups "tmate.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tmate-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tmate-io/73a8f8fb-9f86-45f7-a3c0-4f5ad341c576%40googlegroups.com.

stefan.m...@gmail.com

unread,
Feb 23, 2020, 1:08:46 PM2/23/20
to tmate.io
Hi,
thx for the feedback again.


I had a bit of trouble parsing your email, but I think I found your questions:
yep, can imaging wasn't little :)


 I still do not understand why the tmux client/server architecture is not compatible across an IP network as ssh to IP works as well. 
Because it sends file descriptors though a UNIX socket. That's not possible via IP sockets.
?, as far as I understood, file descriptor / UNIX sockets are used for inter-host communication network sockets are for inter-process communication. 


So I wonder how can I activate tmate on demand without shell access?
Yes, run it as a system daemon, like with systemd
so, tmate would run all time or call by cronjob, I cannot think of another option

Once you close the shell, the tmate daemon would die
how is that realized?
I could think of cronjob, what start tmate every 50 minutes, wait for 10 minutes and kills it, if there isn't any incoming connection.

 

Back again to security. It is said that tmate server container needs to run with --cap-add=SYS_ADMIN what is for security reasons, but still don't like it:

... 

 Any safer possibility using docker? 
Sure: docker run -it tmate/tmate tmate

but that would now allow

Note that you need to add the SYS_ADMIN capability to the container. This is needed to create nested containers (namespaces) to secure sessions.
Any chance to that manually or handle this differently 


I can now start tmate by either
  1. tmate
  1. tmate -F

Is still don't get the difference, can you try to enlighten me? 
May you explain its interaction with stdin, stdout stderr at dummy level.

Thank you 
Stefan


Am Dienstag, 18. Februar 2020 03:35:00 UTC+1 schrieb Nicolas Viennot:
I had a bit of trouble parsing your email, but I think I found your questions:

> I still do not understand why the tmux client/server architecture is not compatible across an IP network as ssh to IP works as well. 

Because it sends file descriptors though a UNIX socket. That's not possible via IP sockets.

> [cap_sys_admin] Does is it apply for the client as well?

No

> So I wonder how can I activate tmate on demand without shell access?

Yes, run it as a system daemon, like with systemd

> Any safer possibility using docker? 

Sure: docker run -it tmate/tmate tmate

> [docker containers] any advice using them? 

I wouldn't recommend them. The officials ones are fine.

Thanks,

Nico


On Mon, Feb 17, 2020 at 12:38 PM <stefan....@gmail.com> wrote:
Hi,
I found some explanation in http://viennot.com/tmate.pdf what is mentioned in the Article on tmate, whereby the architecture diagram is not available anymore.
Unfortunately, this explanation is not available any more.

Stefan

Am Dienstag, 11. Februar 2020 12:22:09 UTC+1 schrieb stefan....@gmail.com:
Hi Nico,
was it too much I asked for or too noob-wise?
Shall I rephrase or do you need some time to compile an appropriated answer? 

Thank you

Stefan

--
You received this message because you are subscribed to the Google Groups "tmate.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tmat...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages