Request for testing

[ldotlopez]

Hi,

I just uploaded a new branch: wip/historical-sensor

In this branch I splited the current hack for historical sensors to a new repository: https://github.com/ldotlopez/ha-historical-sensor.

But.. why? This new module is reusable for other projects and other household supplies, like water or gas (check HomeAssistant historical sensor README) and, the more important, is capable of generating statistics. (See #25)

In our case this means that we can have float-precision readings (but not in real time) and a more resilent updates. Realtime sensor (Accumulated consumption) can be disabled and relay solely on the Historical sensor (Historical consumption) which is more precise, resilent and doesn't require the "advance user".

I'm asking for some testing. Expect weird and broken stuff and data loss, backup your data.

Thanks.

[lazynoda]

Hello there!

If I understand it correctly, instead of reading real time energy data from i-de, the sensor is importing past data, right?. I'm interested on it. Just let me know the steps to follow, apart from backing up everything and download wip/historical-sensor branch.

Is it possible to have both integrations working at the same time, or I need to disable / delete the main one?

Thanks ❤️

[ldotlopez]

Hi,

Yes, this integration has two main sensors:

• "Accumulated Consumption" which takes the direct reading of the service point. It's accurate but only reports integer values for kW/h (not very precise) and can fail for hours leading to inaccurate graphs
• "Historical Consumption" which reads historical data from i-de. It's not up-to-date (it's usually some hours behind) but reports float values for kW/h and it's very reliable.

It's not possible to run both versions but you shouldn't have any problems switching between them.

After backup everything and etc you have to enable the "Historical Consumption" sensor, disabling "Accumulated Consumption" is optional.

In the energy config panel now you can use the "Historical Consumption statistics" as an energy source.

[lazynoda]

Hey @ldotlopez! I've changed the git branch, restarted the whole Home Assistant, enabled the historical consumption sensor, restarted again, but it does nothing. It's unknown and shows nothing in Energy graphs. Logs don't show anything either. Any clue? How can I debug it to ensure it does the right things?

Thx

[ldotlopez]

Hi @lazynoda

Check if sensor is working

Go to Settings → Devices and services → Entities
Search for "historical consumption"
Enable the sensor

Check if it's collecting data
Go to sidebar → History → Search for "historical consumption"
Show last week

Add sensor as energy consumption source

(Note the ' statistics' suffix)

[lazynoda]

Hi there! Sensor is enabled but not collecting data. History is empty. I can add it to Energy panel, but shows nothing on it.

Maybe chatting is faster instead of posting here in the issue?

[ldotlopez]

Hi @lazynoda

Maybe chatting is faster instead of posting here in the issue?

It's a good idea, let me setup something for the entire project.

Meanwhile, can you post debug log from hass?

[lazynoda]

Hi @ldotlopez

The logs are empty. Well not empty, but they show nothing from this integration. I just restarted the HA server, deleted the integration and add it again, just in case, enabling again the sensors. And downloaded the logs. Here they go:

Home Assistant logs

2023-02-07 12:39:19.294 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration localtuya which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-02-07 12:39:19.298 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration sonoff which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-02-07 12:39:19.299 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration ideenergy which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-02-07 12:41:25.286 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform co2signal is taking over 10 seconds.
2023-02-07 12:41:25.324 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: ideenergy, upnp, co2signal, ipp, sensor.co2signal, mqtt
2023-02-07 12:41:25.331 WARNING (MainThread) [homeassistant.config_entries] Config entry 'HP ENVY 5540 series [XXXXXX]' for ipp integration not ready yet: Invalid response from API: Timeout occurred while connecting to IPP server.; Retrying in background
2023-02-07 12:41:38.389 WARNING (MainThread) [homeassistant.components.tts] TTS base_url option is deprecated. Configure internal/external URL instead
2023-02-07 12:41:38.419 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform ideenergy is taking over 10 seconds.
2023-02-07 12:41:41.206 WARNING (MainThread) [homeassistant.components.media_player] Setup of media_player platform samsungtv is taking over 10 seconds.
2023-02-07 12:42:05.782 ERROR (MainThread) [homeassistant.components.google_assistant.http] Could not contact https://homegraph.googleapis.com/v1/devices:requestSync
2023-02-07 12:42:11.058 WARNING (Recorder) [homeassistant.components.sensor.recorder] The unit of sensor.kitchen_smoke_detector_smoke_density () can not be converted to the unit of previously compiled statistics (None). Generation of long term statistics will be suppressed unless the unit changes back to None or a compatible unit. Go to https://my.home-assistant.io/redirect/developer_statistics to fix this
2023-02-07 12:42:39.735 ERROR (MainThread) [homeassistant.components.google_assistant.http] Could not contact https://homegraph.googleapis.com/v1/devices:reportStateAndNotification

Ah, I've also updated HA to 2023.2.2, pulled latest changes (if there were any) but still nothing. Could it be because I have 2 i-DE contracts, and the first one (old home) is not active anymore?

You have a Telegram link on my profile if you wanna chat in Spanish 😛

Edit: Ok I saw you did a commit 17h ago, so I pulled and restarted. Now the sensor shows stuff.

[lazynoda]

The fact that you can add the sensor to Energy panel and it imports all past days, not just from that moment onwards... Mis dies That's amazing!

Is it possible to do it with another sensor? I would like to attach energy price, but it changed at Jan 1st so would be amazing to record both prices (2022 and 2023)

[lazynoda]

@ldotlopez any idea why I can't track energy prices using current or static price? Both options are disabled. Could it be because it has past data and no realtime one?

[ldotlopez]

@lazynoda

Currently it's not possible to "start" at any specific point but I'm working on an import feature to import data from a csv file.

I didn't try to use historical sensors+price tracking 🤔. I don't think it will work due to the hack-ish nature of these sensors but the UI should show you options

However… have you tried to add the PVPC integration? It should appear as "entity with current price"

[lazynoda]

Nah I have it already working, but the thing is, the radio buttons are disabled. I can't select them. I can only select "Do not track costs" or "Use an entity tracking the total costs". Like I said, maybe because it has no current value, just historical ones.

[lazynoda]

Hey @ldotlopez, anything special you wanna me to test? So far it's working. Every morning I have data from the day before, except for the cost issue mentioned earlier, it's amazing!

[lazynoda]

Hi @ldotlopez! I think something has changed with 2023.3.X because I have no data since March 5th. Not sure if related but I saw a DB schema update from 31 to 33. Thanks!

And if you've updated main branch, do not forget me in this branch hehe. Or merge it <3 It works on my machine

[ldotlopez]

Hi @lazynoda

Recent changes in DB are fixed in main and latest release (v1.0.4).

I need to make some changes in this branch (wip/historical-sensor) before merging it in main. My goal is still to build ha-ideenergy on top of this branch.

Sorry about the delay (busy weeks) and so many thanks for your feedback 🥇

[lazynoda]

No rush on it, just to mention in case you forgot me hahaha. I guess when you fix it and I pull the changes, it will automagically fill all the data since March 5th, right? Just to know 😝

Do you plan in the future to merge this branch into main? Maybe more testers would be great, but so far I had no issues.

[ldotlopez]

Hi @lazynoda and anyone interested

I just merged (wip/historical-sensor) into main.

You want to continue testing you have to switch to main version in HACS (or whatever you use).
Currently historical data is retrieved from 7 days ago, you can change it in [const.py](https://github.com/ldotlopez/ha-ideenergy/blob/main/custom_components/ideenergy/const.py)

Thanks!

[kikediazfernandez]

I have v1.0.4 with only the historical sensor enabled and I see data is 1 hour off compared with data collected with Shelly EM.
Also, if you look at the history, and compares the graphic with the one that Iberdrola (not I-DE) shows, data from 11:00 to 12:00 for example is shown as the 12:00-13:00 interval in HA
The API seems to return the accumulated consumption in the previous hour. My last (redacted) log for 'successful' update:

2023-05-31 14:41:32.416 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Updating data set: <DataSetType.HISTORICAL_CONSUMPTION: 2>
2023-05-31 14:41:32.416 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] HISTORICAL_CONSUMPTION update allowed
2023-05-31 14:41:33.314 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] HISTORICAL_CONSUMPTION successfully updated
2023-05-31 14:41:33.314 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Finished fetching [email protected]/xxxxxxxx coordinator data in 0.898 seconds (success: True)
2023-05-31 14:41:33.316 DEBUG (MainThread) [custom_components.ideenergy.historical_sensor] sensor.es0021xxxxxxxxxxxxxxsg_historical_consumption: Got 168 historical states from API
2023-05-31 14:41:33.389 DEBUG (DbWorker_0) [custom_components.ideenergy.historical_sensor] Deleted 0 invalid states
2023-05-31 14:41:33.399 DEBUG (DbWorker_0) [custom_components.ideenergy.historical_sensor] sensor.es0021xxxxxxxxxxxxxxsg_historical_consumption: lastest state found: 0.439 @ 2023-05-30 22:00:00+00:00
2023-05-31 14:41:33.399 DEBUG (DbWorker_0) [custom_components.ideenergy.historical_sensor] sensor.es0021xxxxxxxxxxxxxxsg_historical_consumption: no new states from API

The latest state, corresponding to 23:00-24:00, is collected as 00:00 (22:00 UTC)
"Successful" because yesterday's data still does not appear, by the way.