URIs in the config files should be validated as URIs

[mattiapsu]

Describe the bug
I copy a test image to the aiinput folder, but it appears that processing is not happening.

Log entries
Trigger log

2020-11-06T10:01:53-05:00 [Main] ****************************************
2020-11-06T10:01:53-05:00 [Main] Starting up version 5.4.0

2020-11-06T10:01:53-05:00 [Main] Timezone offset is 300
2020-11-06T10:01:53-05:00 [Main] Current time is Fri Nov 06 2020 10:01:53 GMT-0500 (Eastern Standard Time)
2020-11-06T10:01:53-05:00 [Settings] Unable to read the configuration file: ENOENT: no such file or directory, open '/run/secrets/settings'.
2020-11-06T10:01:53-05:00 [Settings] Loaded settings from /config/settings.json

Deepstack log

/v1/vision/detection
---------------------------------------
v1/vision/addmodel
---------------------------------------
v1/vision/listmodels
---------------------------------------
v1/vision/deletemodel
---------------------------------------
---------------------------------------

p
---------------------------------------
v1/restore

Installation details

• Unraid 6.8.3 Docker setup; installed by directions/documentation using the bridge and port forward presented in Issue Document how to get node-deepstackai-trigger working on Unraid #349

Additional context
I had followed some other guides to install deepstack including from command line in Unraid. I deleted those containers and images by Unraid GUI prior to using the method above.

In order to test the system I am just copying an image into the aiinput folder and monitoring the logs for action. I'm using a cat picture posted in example setup. I have Blue Iris setup but have not automated the picture load.

If I go to my deepstack URL:port, it says deepstack is activated.

[neilenns]

Please include your docker-compose.yaml, settings.json and triggers.json file.

[mattiapsu]

I uploaded settings.json and triggers.json as .log in previous because git is saying it doesn't support .json upload, and I've also copied the code for both below. I didn't use the docker-compose.yaml, but used Unraid 'add container' with the elements defined below.

Appreciate your thoughts, thanks

settings.json

{
  // Leave this line alone. It enables Intellisense when editing this file in Visual Studio Code.
  "$schema": "https://raw.githubusercontent.com/danecreekphotography/node-deepstackai-trigger/main/src/schemas/settings.schema.json",

  // This is the default uri for Deepstack when deploying using the docker-compose.yaml file
  // provided in this sampleConfiguration folder. If you have another deployment of Deepstack
  // running elsewhere change this setting to point to the correction location.
  "deepstackUri": "http://10.0.0.1:83/",

  // Set this to true to enable annotated images for use in trigger handlers.
  // There is a performance penalty to using this, leave it off unless
  // you really want to see the annotated images.
  "enableAnnotations": false,

  // Set this to true to enable the internal web server for remote access
  // to processed and annotated images.
  "enableWebServer": false,

  // Enables verbose logging. Useful when setting up the system to ensure
  // everything is running correctly.
  "verbose": true,

  // Set this to true if your images are stored in a remote folder that's
  // mounted as a network share and then mapped to the Docker image.
  "awaitWriteFinish": true,

  // Provides the configuration details for your MQTT server. To enable
  // mqtt set the uri, username (if required), password (if required),
  // and set enabled to true.
  "mqtt": {
    "uri": "10.0.0.1:1883",
    "username": "sensormqtt",
    "password": "8nv3FWs8R6ejvB",
    "enabled": true
  }
}

triggers.json

{
  "$schema": "https://raw.githubusercontent.com/danecreekphotography/node-deepstackai-trigger/main/src/schemas/triggerConfiguration.schema.json",
  "triggers": [
    {
      "name": "Cat detector",
      "watchPattern": "/aiinput/Cat*.jpg",
      "enabled": true,
      "threshold": {
        "minimum": 20,
        "maximum": 90
      },
      "handlers": {
        "mqtt": {
          "topic": "homeassistant/BI/Basementdoor"
        }
      },
      "watchObjects": ["cat","dog"]
    },
    {
      "name": "Cat detector",
      "watchPattern": "/aiinput/Cat*.jpg",
      "enabled": false,
      "threshold": {
        "minimum": 10,
        "maximum": 90
      },
      "handlers": {
        "webRequest": {
          "triggerUris": ["http://localhost:81/admin?trigger&camera=Cat"]
        },
        "mqtt": {
          "topic": "aimotion/triggers/cat"
        },
        "telegram": {
          "chatIds": [1]
        }
      },
      "watchObjects": ["cat"]
    }
  ]
}

Unraid container setup

[neilenns]

This is almost certainly a mount/path problem with unraid. Can you connect directly to the trigger container with a console and do an ls /aiinput and see if the folder even exists? Are there files in it?

Another thing to do is put a file in the mount folder then start up the container. It will output log messages telling you how many files it finds in the mount location that are waiting to be processed.

[mattiapsu]

Running the command shows all the "cat" files in the current directory. Restarting the container doesn't change the log from what is shared above.

[neilenns]

There should be a lot more log output than what you pasted in, e.g. there's no log message showing the trigger.json file loaded (which would explain why no triggers are firing). Here's example of what it should look like:

2020-11-06T08:59:23-08:00 [Main] ****************************************
2020-11-06T08:59:23-08:00 [Main] Starting up version 5.5.1
2020-11-06T08:59:23-08:00 [Main] Timezone offset is 480
2020-11-06T08:59:23-08:00 [Main] Current time is Fri Nov 06 2020 08:59:23 GMT-0800 (Pacific Standard Time)
2020-11-06T08:59:23-08:00 [Settings] Unable to read the settings file: ENOENT: no such file or directory, open '/run/secrets/settings'.
2020-11-06T08:59:23-08:00 [Settings] Unable to load file /run/secrets/settings.
2020-11-06T08:59:23-08:00 [Settings] Unable to read the secrets file: ENOENT: no such file or directory, open '/config/secrets.json'.
2020-11-06T08:59:23-08:00 [Settings] Loaded settings from /config/settings.json
2020-11-06T08:59:24-08:00 [Local storage] Creating local storage folders in /node-deepstackai-trigger.
2020-11-06T08:59:24-08:00 [Local storage] Enabling background purge every 60 minutes for files older than 30 minutes.
2020-11-06T08:59:24-08:00 [Local storage] Running purge
2020-11-06T08:59:24-08:00 [Main] Web server enabled.
2020-11-06T08:59:24-08:00 [Triggers] Unable to read the settings file: ENOENT: no such file or directory, open '/run/secrets/triggers'.
2020-11-06T08:59:24-08:00 [Triggers] Unable to load file /run/secrets/triggers.
2020-11-06T08:59:24-08:00 [Triggers] Unable to read the secrets file: ENOENT: no such file or directory, open '/config/secrets.json'.
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration from /config/triggers.json
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Front door
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Garage
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Back door
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Back yard
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Driveway
2020-11-06T08:59:24-08:00 [Triggers] Loaded configuration for Side house
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Front door.
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Garage.
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Back door.
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Back yard.
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Driveway.
2020-11-06T08:59:24-08:00 [Trigger manager] There are 133 images waiting in /images for Side house.

Try doing an ls /config to make sure the triggers.json is there and visible.

[mattiapsu]

shows up there too... you mentioned .conf first... is there is a difference here? I did try a file extension change a while back to see what effect that would have. But it reacted the same.

it would seem it's getting stuck at local storage. Crap, I'm on kid duty, have to come back to this later. Thanks for your help.

[neilenns]

Nah I just typo'ed and corrected myself. There should be a ton more logs showing up, at the very least failure messages saying it can't read/find that file.

I'm not sure what else to suggest unfortunately. I don't have access to unraid/have never used it.

[mattiapsu]

Last question before I throw in the towel for now... the next step in the log is "creating local storage folders." I'm looking into the read/write permissions in the root appdata/deepstack-trigger folder. The /config path is read-write to the subfolder, but I'm not so sure that the root for the container is write-able. Can you tell me more about that step? Does it have a container path associated with it? What does it create?

[neilenns]

It attempts to create three folders in /node-deepstackai-trigger in the container. Do you actually see this message in your logs?

2020-11-06T08:59:24-08:00 [Local storage] Creating local storage folders in /node-deepstackai-trigger.

[neilenns]

@mattiapsu Can you update your unraid setup to point to the issue396 tagged docker image instead of latest and then try again?

I added some additional error handling and logging to the local storage folder creation step. It might be that something is going wrong there and it's just killing the whole system. If so the changes I made will at least spit out the error.

[mattiapsu]

I updated my docker image to danecreekphotography/node-deepstackai-trigger:issue396

Unfortunately, the log is still halting at the same place, right after loading settings.json

This could totally be me as I know enough to be dangerous, but by no means can do this stuff without help from people like you. I'm happy to keep trying stuff to get you more info. Also, fine if it's not worth it for you.

[neilenns]

I have an idea!

Can you try disabling MQTT? Just set enabled to false in your settings.json.

[neilenns]

And if that fixes it, try enabling it again and changing the uri to http://10.0.0.1:1883.

[mattiapsu]

It got to the next step. Here's the log now...

[neilenns]

Ok, change your Docker tagged image to latest instead of issue396 and run again.

[mattiapsu]

Success, it was simply the http reference in MQTT uri.

Thanks for troubleshooting with me, sorry for the stupid overlook. Now time to dig in.

[neilenns]

Sweet! Glad we got it figured out. I reopened this for eventual improvement to the config file parsing. There should be a check that the URI is properly formatted rather than just assuming that's the case and passing it off to the MQTT library for use (which apparently causes everything to just stop).