Cannot start plot jobs on TrueNAS Scale worker

[k2skaterii]

I've been trying to figure out this issue for a couple of days, with no luck.

Last weekend I decided to give TrueNAS Scale a try, TrueNAS Core worked but... Scale is new, and not FreeBSD. While there are only 6 applications in the official TrueNAS Scale app repo chia and machinaris are both included in that list!

When I installed machinaris VIA the TrueNAS Scale web GUI, I included the relevant environmental variables from my other plotter Docker Run files. Once the install job has completed, my new worker shows up on my workers page... I can modify the plot setting VIA the web GUI. So far so good...

Things go sideways when I try to start plotting on my new plotting worker. As soon as I try to start plotting the following error pops up.

Errors during plotman status:
 /bin/sh: 1: cannot open /dev/tty: No such device or address

I checked the obvious issue - Does /dev/tty exist? Yes it does exist!
I attached the events associated with a single failure from the apisrv.log off of the TrueNAS Scale plotter - Hopefully they are meaningful to someone!

Manually starting the plot jobs for now!

TrueNAS_Plotter_apisrv.log

[guydavis]

Hi, yes this means that the Docker container is being launched without the -t (aka tty flag) being passed on Docker launch command. Sorry, I don't know enough about Truenas launch of Docker containers to comment further. Would recommend you drop into the "truenas" channel of our Discord server with others running that OS.

[k2skaterii]

Discord... I've resisted setting an account up thus far... Maybe the time has come to do it!

I'm still troubleshooting - while I haven't figured out what is missing or what is wrong... I located your /machinaris/scripts folder a few minutes ago. Made a rookie mistake - changed multiple things without testing. Running the plotman install / stopping and starting machinaris then running the madmax install script.

I'm going to redeploy the TrueNAS image undo'ing my rookie mistake... testing after each change.

[PMiani]

@k2skaterii I reply here to #252 (reply in thread).

As mentioned by @guydavis in #255 (comment), tty is required for machinaris to work properly. By having a quick look at the deployment file in the TrueNAS Charts repo at https://github.com/truenas/charts/blob/master/charts/machinaris/1.0.2/templates/deployment.yaml I cannot find the tty flag. I am not sure if you can specify manually in the UI during setup as I do not have it configured, so no UI for me.

Considering the TrueNAS Scale release plan (https://www.ixsystems.com/blog/truenas-scale-release-plan/) states the APP part (Docker and Kubernetes) is still in preview, I decided to make a manual install using pure Docker, also allowing me to be up-to-date and not relying on someone else to update/deploy. Keep in mind though, TrueNAS is an appliance and therefore cannot be treated as a pure Linux host, therefore limitation apply (e.g. fstab get replaced at boot). Don't get me wrong. Kubernetes and Helm charts are way better than DOcker, but at the moment could not be bothered to create one of my own or rely on a Preview phase.

As far as the setup is concerned, disk choice is important.

Plotting dir (aka tmp dir)

I use a single SSD (a Samsung SSD 840, all I have since I decided to start end of July with Chia and don't feel yet the need in investing in hardware yet) as my temp dir (mapping to :/plotting to be clear). This is not part of any ZFS pool and is formatted as normal EXT4. ZFS drastically degraded the performance while plotting. If you really wanna use ZFS, RAID 0 is the best bet, especially if no SSD are there. No L2 cache (if you have an NVME or SSD for caching use directly that), LOG, etc should be added to the Pool. Sync, Aitime, Compression and Deduplication should be off!

Plots Store (aka plots dir)

To store plots - instead - RAID make no sense to me as I don't really want to waste space to have plots redundant, this means no RAID 1/5/10 etc. Also, RAID 0 make no sense either to me, as if you loose one disk, you will loose all the plots in the Pool instead of just the one stored in the failing disk. This is a decision that comes mostly from my limited hardware at my disposal and the fact that I need space, no protection. I suggest a ZFS Pool per Disk. I use NFS among Nodes, so ZFS is configured as follow:

• Sync: Disabled
• Compression LZ4
• Aitime: Off
• Quota (Warning, Critical): 99% (so will not bother you when you fill the disks)

For both tmp dir and additional plots directories I use POST-INIT Scripts to mount them via fstab. This turns out to be unstable in TrueNAS Scale, therefore at the moment AUTO_PLOT is disabled. For the time being, I manually check my disks are mapped before starting plotting. Although I rarely reboot and I am protected by UPS.

Docker

Make sure docker is enabled on boot (if you used the App part, probably it already is).
service docker enable

If you like a GUI, Portainer can be used as an alternative to the embedded UI.

docker volume create portainer-data
docker run \
   --detach \
   --name portainer \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v portainer-data:/data \
   -p 9000:9000 \
   --restart always \
   portainer/portainer-ce

Deploy the worker/plotter via either docker-compose (e.g. via Stack in Portainer) or Docker command as generated by machinaris Workers page.

Will be happy to provide more details should you need them. Also I see you use Madmax already and get really good timing in other VMs. You are lucky to have 10k SAS Disks, so I don't think it will perform poorly anyway.

[k2skaterii]

@guydavis I posted this in Discord as well!

While I don't know enough about the TrueNAS Scale container deployment to recommend a fix so it works out of the box... But I did find a work around that got things working!

I was able to get my plotting only machinatis app on my TrueNAS Scale working by connecting to the container's shell and running the following scripts that came with machinaris:
/machinaris/scripts/plotman_install.sh
/machinaris/scripts/stop_machinaris.sh
/machinaris/scripts/start_machinaris.sh

The output from the plotman_install script indicated that all dependencies were all ready installed, I don't know if anything even changed. Maybe stopping and starting machinaris from within the container is all that was needed...

Eitherway, I bet that kind of post deployment manual intervention was not intentional or expected!

[PMiani]

@k2skaterii connecting the the container with exec, thus attaching a tty on the container 👍🏻
I doubt running the commands you mentioned had any effect :)

[k2skaterii]

@k2skaterii connecting the the container with exec, thus attaching a tty on the container 👍🏻
I doubt running the commands you mentioned had any effect :)

Sounds like a learning opportunity / challenge. I haven't figured out how to bind the charts deployments to a single IP on my TrueNAS Scale, thus far both plex and machinaris are binding to all IP's on their specific ports. I can't test anything without shutting down the existing container. If I wait for the running plot jobs to finish (2x K33's) then I won't be able to redeploy / test until sometime tomorrow.

LOTS of GOOD info in your post above. I'll refer back to it when I'm testing.

[PMiani]

I'm afraid if you restart the container you will loose the tty :) Somebody should put tty: true / stdin: true in the container spec in deployment.yml in the chart I assume.

In any case, if you look at the deployment file, you cannot bind a single IP.. it uses the Host Network and thus all IPs in the Host. You really need an Ingress. Also, from the FIXME note I assume it will change at some point!

...
      # FIXME: Let's please remove hostnetwork when upstream hostport issue is sorted out with kube-router
       hostNetwork: true
...
          ports:
            - name: chia-network
              protocol: TCP
              containerPort: 8444
              hostPort: 8444
            - name: machinaris-ui
              protocol: TCP
              containerPort: 8926
            - name: machinaris-api
              protocol: TCP
              containerPort: 8927
              hostPort: 8927
...

In Docker, should you chose to go that way, you can specify (for example) in the docker compose (or command).

ports:
  - "192.168.0.1:8444:8444"

[guydavis]

This should now be fixed in v0.5.5 of Machinaris which removes the usage of stty inside Plotman for non-shell usage via the Machinaris WebUI.