Make the MQTT handler still parse the message if Teltonika adds new parameters in a firmware update

[maherbm]

Hello,

I am testing Teltonika FMC003 with a new custom firmware that we received from Teltonika.

I could not make it work with fleet management as it seems MQTT handler drops out directly after coming across a parameter that was not predefined in the FMC003.json, while the Teltonika device keeps trying to resend the message.

The exact issue happens in line 135 of TeltonikaDataPayload.java

https://github.com/openremote/fleet-management/blob/b13a60df8d2f56f1108fcf640fb7cf13b5d30957/manager/src/main/java/telematics/teltonika/TeltonikaDataPayload.java#L135C13-L135C14

throw new IllegalArgumentException("Key " + entry.getValue().toString() + " not found in finalParams");
First this line shows the parameter value instead of the key. This did cost me a bit of extra time to figure out the problem from the logs ;-).

Second, IMHO, this line should not throw an exception but just log the error, and the code should continue with the parameters that have been identified. Would it be possible to perform this change to the code?

Thank you and best regards

[pankalog]

@maherbm hey, thanks for making an issue!

This was a design choice I made early-on, since I assume that people that use this integration from Teltonika would also parse the AVL IDs as instructed in the tutorial.

I'd suggest you first try to generate the AVL IDs as instructed and then either importing them or replacing the contents of FMC003.json.

For me to look into this further, I would need at least 1 payload that contains the AVL ID parameter that creates the issue.

Thanks!

[maherbm]

Dear @pankalog,

Thank you for your response. You are right, I should have first performed the AVL ID parsing :-).
I did that before the christmas, but still there are several things that needs to be improved to the code.

1. The "throw new IllegalArgumentException" above should be preceded by a logging to the User Interface, so the user can see what goes wrong in the OpenRemote UI (DOMAIN/manager/#/logs). Currently I can only see the error only via the linux shell using "docker logs". The logging to the UI can also be done of course where ever you catch this Exception, which seems currently not to be done either.
2. Currently, the Exception prints the VALUE of the AVL and not the ID as intended. Please change getValue() in (entry.getValue().toString()) to getKey(). This is useless information. For me to figure out which AVL ID was causing the problem, I had to compare the logs of the FMC003 device directly by connecting it to my PC and comparing the values in that logs with the values mentioned in the OpenRemote logs

Furthermore, regarding the AVL ID parsing code, I noticed the following issues:

3. The code did not work directly for me as it seems Teltonika website added some mechanism to block web crawling and thus the code crashes with "urllib.error.HTTPError: HTTP Error 403: Forbidden". To solve this I replaced:

dfList = pandas.read_html(url, header=1, encoding='utf-8')

with the following code that fakes a browser User-Agent

import requests

header = {
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
  "X-Requested-With": "XMLHttpRequest"
}

response = requests.get(url, headers=header)
dfList = pandas.read_html(response.text, header=1, encoding=response.encoding)

4. It seems in the new firmware update to FMC003, Teltonika added new AVL IDs that are not documented on their website, causing the above mentioned Exception to trigger. I needed to find those codes in other devices such as TFT100 and then merge the JSONs. I am still not sure it is a good strategy to ignore the whole package with all the AVLs if one AVL ID is not recognised. But that is up to you to decide.

I hope the information I provided with help to solve the mentioned issues. If you have any question, please don't hesitate to contact me :-)

Thank you and best regards,

Maher

[pankalog]

@maherbm thanks for your reply, makes debugging a bit easier. I'll try to fix this soon, I'll make some bullet points on my to-do list.

What exactly would you like the resulting attribute to be, if its corresponding AVL data is not found? The only data I have of that is just the AVL ID, so the name of it would just be the ID, and the value would be a string containing the value from the payload.

Also, for the AVL ID parser, I'd greatly appreciate a pull request! I also struggled with this parser recently with their crawler limitations, so I botched it up with hitting ctrl-s on my keyboard to literally save the platintext of the website 😆

Thanks!

Panos