Multiple shells for debugging

[krisman]

I usually need more than one shell in different ptys when interactively debugging the kernel in order to, for instance, run a debug script in one session while executing a test program somewhere else. I'm assuming this is a common use-case for many people, so I'm wondering how others do it.

The obvious approach would be a ssh server inside the vm. I dislike it because I don't want to keep a dropbear/opensshd server around just for that, doing host and client side key management is annoying and going through authentication for a local debugging machine seems silly. In addition, there are cases where I want to build without network support.

In the past, with the old virtme, I used to create the vm with QMP and multiple char devices, set several ttys and then had a script to connect to the consoles on demand. Recently, I started using the vsock protocol instead

For the record, I create the VM with:

-device vhost-vsock-pci,guest-cid=3

During startup, I do:
socat VSOCK-LISTEN:1024,reuseaddr,fork EXEC:"bash -i",pty,stderr,setsid,sigint,sane,echo=1

and then I can connect with:

socat file:tty,raw,echo=0, VSOCK-CONNECT:3:1024

Is there a better/simpler way to do it? Should we get this into virtme-ng?

[matttbe]

Hello,

I guess it depends what you want to do, but maybe using tmux or screen might be enough. When I need more shells, I usually launch Byobu.

Would this work for you?

[krisman]

I considered the terminal multiplexer before, but I don't like the approach for a few reasons. The main one is that it doesn't interact well with already multiplexed workflows, either because I already use a multiplexer on the host and would have to embedded the session (very annoying). Another reason is that multiplexers don't interact nicely with vng --exec. For instance, if I want to hop on a machine doing an automated test because the test hung.

Probably there is a way to make tmux/screen exit when the inferior program dies, so I can workaround the second issue by always starting the test program embedded with the multiplexer on the vng command line.

But another advantage of being able to spawn shells on-demand is that I can easily send commands from the host to the VM programatically, through a script that writes to the connection. That wouldn't be possible with a multiplexer inside the vm.

[arighi]

@krisman maybe you can do the following to simply ssh into the guest from the host.

Start vng in network bridge mode:

$ vng -vr --net bridge

If your host is configured correctly (see also https://github.com/arighi/virtme-ng?tab=readme-ov-file#troubleshooting), the guest will function like any other host on your subnet. Then you can manually assign an IP, or with DHCP enabled, it will automatically receive an IP address within the subnet.

Then you can start sshd inside the vng guest, using your ssh key as host key (also make sure to remove the nologin file so regular users can login):

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
sudo mkdir -p /run/sshd
sudo rm -f /var/run/nologin
sudo /usr/sbin/sshd -h ~/.ssh/id_rsa

From your host:

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no VNG_GUEST_IP

In this way you can ssh into your guest from your host.

[arighi]

Maybe we could also automate these steps if you think they can be useful, like adding a new --ssh option to automatically run all the commands in the guest to setup and start ssh, wdyt?

[matttbe]

Maybe we could also automate these steps if you think they can be useful, like adding a new --ssh option to automatically run all the commands in the guest to setup and start ssh, wdyt?

If the idea is to automatically setup something, I think it would be better to setup something using AF_VSOCK, not to depend on --net, which will generate TCP traffic (not nice when you work on that :) ). Either with a proxy for SSH (similar to systemd-ssh-proxy -- but sshd + key is required) or a remote prompt similar to what @krisman is doing (but using ${SHELL} instead of bash? :) ), that's lighter.

[krisman]

@krisman maybe you can do the following to simply ssh into the guest from the host.

Start vng in network bridge mode:

$ vng -vr --net bridge

If your host is configured correctly (see also https://github.com/arighi/virtme-ng?tab=readme-ov-file#troubleshooting), the guest will function like any other host on your subnet. Then you can manually assign an IP, or with DHCP enabled, it will automatically receive an IP address within the subnet.

Hi Andrea,

I considered ssh as I mentioned in the first comment, but I think it requires too much setup for not many benefits - specially if we use tcp/ip. I really don't want to bother with dhcp or setting up a bridge if I can avoid it. VSOCK trivially solve these problems, granted, with static assignment, and seems designed specifically for this use case. Another issue with ssh itself is that the setup goes beyond what is described, with minor papercuts. For instance, I just noticed my distro (opensuse) installs sshd_config without u+r permissions, crashing sshd on start. This makes it harder to create a --ssh functionality that "Just Works" everywhere.

Also ssh doesn't buy us much - we don't care about crypto+auth. I think the only benefit of ssh in this case is that it spawns a sane pty (which is apparently a hard thing to do correctly).

I'd actually prefer my initial approach of setting up multiple chardev, so I can build with CONFIG_NET=n. But that adds complexity to virtme for not much gain.

For VSOCK , the only changes I made are(hacky, should be cleaned up):

--- a/virtme/commands/run.py
+++ b/virtme/commands/run.py
@@ -1095,6 +1095,8 @@ def do_it() -> int:
                 ]
             )

+    qemuargs.extend(["-device", "vhost-vsock-pci,guest-cid=3"])
+
--- a/src/main.rs
+++ b/src/main.rs
@@ -1018,6 +1018,11 @@ fn main() {
         handle.join().unwrap();
     }
 
+    thread::spawn(|| {
+            utils::run_cmd("socat", &["VSOCK-LISTEN:1024,reuseaddr,fork",
+               "EXEC:\"bash -i\",pty,stderr,setsid,sigint,sane,echo=0"]);
+    });
+
     // Start user session (batch or interactive).
     set_cwd();
     setup_user_session();

And then I connect with:

socat file:tty,raw,echo=1, VSOCK-CONNECT:3:1024

I have a few more quirks in the connection script to properly setup the tty (as mentioned, weirdly hard to get right).

Are you open to a PR implementing this? The only dependencies are socat/netcat and building the kernel with CONFIG_VIRTIO_VSOCKETS=y

[arighi]

@matttbe good point about the overhead of ssh, and it also doesn't provide any benefit for copying files remotely for example, because we already share the host's filesystem (or portions of it).

@krisman I really like the vsock idea, I wasn't very familiar with it, I think we should better integrate it in vng. I was trying to test your patch, but I get this:

socat file:tty,raw,echo=1 VSOCK-CONNECT:3:1024
2024/08/09 09:31:26 socat[119892] E open("tty", 02, 0666): No such file or directory

I guess I'm missing how the tty file is created and where... but please go ahead and send a PR, I'd be really happy to have this feature!

[matttbe]

@krisman I really like the vsock idea, I wasn't very familiar with it, I think we should better integrate it in vng. I was trying to test your patch, but I get this:

socat file:tty,raw,echo=1 VSOCK-CONNECT:3:1024
2024/08/09 09:31:26 socat[119892] E open("tty", 02, 0666): No such file or directory

I guess I'm missing how the tty file is created and where... but please go ahead and send a PR, I'd be really happy to have this feature!

I think the issue is that you used tty in text instead of the output of the command tty, no? Probably clearer with $():

socat file:$(tty),raw,echo=0, VSOCK-CONNECT:3:1024

Please note that for this feature to work, it requires socat, but also CONFIG_VIRTIO_VSOCKETS (which depends on CONFIG_VSOCKETS) for the VM (and CONFIG_VHOST_VSOCK on the host).

If this feature is integrated to virtme-ng, it might be better to have a random CID number to avoid issues when multiple instances are started in parallel. This issue is similar to --debug and its port 3636, but --debug is optional. One solution is then to have this vsock feature optional as well, and users can set an optional number for the CID, e.g. --vsock would use 3, and it is possible to use --vsock 4 to use another one? (same for the --debug option?)

It might be handy to have a vng --connect [CID] (or similar) to easily connect to the VM.

One last thing, the command that is executed in the VM can be something else than bash -i, e.g. zsh, ${SHELL} or even tmux, screen or byobu, etc.

[arighi]

Thanks @matttbe for looking at this, this feature has been requested multiple times. I think we can start with the simple optional way, just one CID (like port 3636 for the debugging mode) and improve later in-tree. And yes, at some point it'd be nice to have vng --connect [CID] and also have the possibility to specify a different shell/command. One step at a time. :)

[matttbe]

@arighi : Thank you for this comment! For the moment, I'm using multiple VMs in parallel, so I already added this way to set a different CID :)
Please see: #197