Xdrip+ application using a large amount of internal memory

[bsaunder2002]

I've been running Xdrip+ for a while now. (years).
Great product. Thanks to all who birthed it and helped it mature into the product it is. Wonder what the cummulative A1C drop for all users is. My guess -2 points for me! Sum all that up and think of how many billions saved in medical costs for avoided complications. Pat yourself on the back!

Ok, my issue.
Running on non-rooted ( :( ) Android 6 phone. This phone cannot move the application to the SD card. I do not have a backup solution installed on the phone right now.
The phone says I'm using something like 4.68GB of internal memory for the application.
Via the application, I see no way to free that memory up.
Another discussion that was closed due to inactivity said this may be debug logs. I tried to find that discussion again but could not.

Is there any way to purge the 4.68 w/o loosing my BG readings? Or if so, keeping the last 90 days?
In application memory management should be supported. Clean up non-essential logs, aggregate BG readings older than X (ie older than 90, I could keep just 1 reading every thirty mins).

[Navid200]

I have been using xDrip for almost 4 years now not on the same phone.
I have been uploading all my readings (one per 5-minute) to Nightscout.
The total size of my Nightscout database is 200MB.

Currently, xDrip on my phone shows App size: 30MB and user data: 60MB.
I have been using this phone for about a year now.

None of that answers your question. It's just my thoughts.

How long have you been using xDrip?
Do you have everything uploaded to Nightscout already?

I wonder if you can perform an experiment. However, it depends how much free space you have left on the phone.
If you have free space, it may be useful to perform a backup to see what the size of the backup will be.
But, if there is no space left on the phone, please don't do it. But, let me know how much space you have left.
https://navid200.github.io/xDrip/docs/Backup-Database

[Navid200]

Is this the issue?
#1730

[bsaunder2002]

Thanks for finding that.
Appears like the final solution was:
The solution consisted of generating a backup of the data for its repair, uninstalling and reinstalling the application.

I'll find/install a play store backup solution. :( and see if backing up the data, uninstalling xDrip+, reinstalling xDrip+, and restoring the data fixes it.

Details in a bit.

[Navid200]

xDrip now let's you backup to Google Drive as well:
https://navid200.github.io/xDrip/docs/GoogleDriveBackup.html

[bsaunder2002]

Good bit about backup to google drive. Sorry, I try not to cloud.

I tried my phones LG Backup to SD card, uninstall/reinstall, restoration of backup.
It didnt work. It toasted my database. Now I seem to have 7 days only. :(

I found an export that I had made a week ago'ish 10/12
-rw-r----- 1 bsaunder bsaunder 305M Oct 23 12:58 export20221012-224335.zip
unzips to
-rw-r--r-- 1 bsaunder bsaunder 4.0G Oct 12 22:43 export20221012-224335.sqlite

Which sqlite3 complains about:
bsaunder@cerberus:~/xdrip/unzipped$ sqlite3 export20221012-224335.sqlite .tables
Error: database disk image is malformed

Luckily it did .dump and I got:
-rw-r--r-- 1 bsaunder bsaunder 3.7G Oct 23 13:08 dumped

Reviewing that sqlite3 dump file I found a bunch of lines (via pgup/pgdn hold down to see scroll by):
INSERT INTO UserErrors VALUES

Sounds like a table I dont need contents in. So if I drop those from the dumped file, run the dumped file back into sqlite3 and get a .sql file, perhaps I can load it back into xdrip+ and get my history back? One would think.
So:
bsaunder@cerberus:/xdrip/unzipped$ cat dumped |grep -v "^INSERT INTO UserErrors VALUES" >dumped.2
Now:
-rw-r--r-- 1 bsaunder bsaunder 19M Oct 23 13:35 dumped.2
That cannot be. Out of a 4GB sqlite3 dump file, the UserErrors table was taking ALL but 19MB? Hun. Lets try the reverse.
bsaunder@cerberus:/xdrip/unzipped$ cat dumped |grep "^INSERT INTO UserErrors VALUES" >dumped.3
and
-rw-r--r-- 1 bsaunder bsaunder 3.7G Oct 23 13:37 dumped.3
so head shows there's stuff I probably want to keep:
INSERT INTO UserErrors VALUES(49,'BG HIGH ALERT: 200 (@00:07)',6,'Alerting',1658909242020);
but tail shows
INSERT INTO UserErrors VALUES(6765248,'We have an illegal time at parseData history 252009FCD71575401A2829E44769526678FD2740C67C9A4EC0CB7644C7A092D3C88B0571C0C2620826E184C047B1C0EEC214CF625B3AEE879F0FDD7F59F7CAA078D15E885756ADF67AB8A4FE45B5DE7045B3F34D41804BCBA4ABA16CD64A25FECC3EEC9518CF6175B43E2E7EDBF257B453E3E371E4277D0E875975952ADD4758A656B0F0B560B71739ADFD3143DE4D2B5720C0AEFFE046EBAE67624433657F3EC943A2B7EE96F646B03BA182E06A4550950EAC68F0FFBCF80E3E384967EE12CFA20B65056C9DA9E833575BDFAB9092A78CCF937B6F64D3B6B6918BA02D8862409B6F8F897E5A21EEA50C8F2888CADF0D85E9BC4F22849F2620A5D2A664E9DC5CFB765B3BB34DB87C213985778D55C21A04B37E3ECFA042BCF96AEDFF425B7C549D0FAD13D01F7A9FE53DEF1D52825ECE0202A52387736F60DDF235ED458AD426478779A10AE1E9B74A79006FF844BE51FB136EA1B195F39DDC69C',2,'NFCReaderX',1663954729214);

My guess is that table is mixing stuff I care about, and stuff I dont care about. I know you may care, but I really dont care about illegal time parse data. Im ignorant. I just want the gui to say "corporate competition makes it really hard for users to access their data." (yes, I am frustrated.)

So, I'll dig deeper to see what I can cut out of hte 3.7GB of "INSERT INTO UserErrors VALUES" to see if I can reconstruct the export and get my history back.

Feels like something should be done at the code level to handle large timeframes...maybe influxdblite3?

Any advice is greatly appreciated.

[bsaunder2002]

So removing specifically the following string seems to reduce the 3.7GB file to 292MB.
bsaunder@cerberus:~/xdrip/unzipped$ cat dumped.3 | grep -v "We have an illegal time at parseData" >dumped.4
-rw-r--r-- 1 bsaunder bsaunder 292M Oct 23 14:51 dumped.4

Are the "We have an illegal time at parseData" important or can I just strip them from the database?
Going to anyway rebuild the sql and try to import it.

[bsaunder2002]

I was unable to successfully prune the database. DB corruption as reported by sqlite3. Unable to load into xDrip+.

Ended up uninstalling which freed up the space.
Reinstalled and now it's back to 4.xGB utilization.

Must be a bug in xDrip+. I'm on Android 6.0.

Can someone look at this?

[Navid200]

DB corruption as reported by sqlite3.

Reported where? Please include a link.

[bsaunder2002]

When I tried to open the sqlite3 database using the linux sqlite3 app, it said the database was corrupted.
I then did a sqlite3 .dump which produced a large text file. As documented above, I striped out the large majority of errors from the file and then tried to import that back into the newly installed xDrip+.
The import did not recover anything older than 7 days.
I cussed a bit and went on with life.
Just last night I got a warning my phone was out of memory. xDrip+ was at 4.something GB. This is for approximately 2 weeks of usage.

There must be a bug somewhere in the code. All I can think of is that I'm android 6.

Ideas? Tests you'd like me to run?

[Navid200]

I linked a closed issue that explains reinstalling xDrip solved the problem.
You have now reinstalled xDrip.
Is xDrip still occupying more and more space since you reinstalled it?

[jonprogers]

I've got xDrip running on a Google Nexus 5 which is also Android 6, in xDrip sync follower mode and am not seeing this problem.

Is there anything I can do to compare and contrast my setup with yours to see if we can find a difference?

[bsaunder2002]

Navid200, I'm going to reinstall a second time. Perhaps attempting to import the db (at 300+MB after stripping it) it loaded a setting that caused it to fill again. Yes it has grown to 4.3GB. I didnt see it "grow" but the reinstall cant be over 2 weeks old. This time, I'll just reinstall without attempting an import and let you know.

Jon,
Thank you for the offer to help. I'm going to delete xDrip+, confirm that I get my 4.xGB back, reinstall from scratch, hook up my MiaoMiao2, and see if it recreates the large file.
As for bits...
MM2
LG G Stylo Android 6.0.
NetGuard VPN blocking most outbound stuff.

I hate having to experiment as it throws away a bunch of data. Hopefully after I figure out why it's eating space, I can patch back in the old DB and get my last couple years back on the phone.

I will update the ticket in several days.

(ps- I also wish and maybe we do, that we have some way to dump exports into a influxdb so you could setup chronograf or grafana to view reports. )

Let me know if there's something more specific. Perhaps I'm the only one having this issue?

Bill

[jonprogers]

Bill,

After years of operation the xDrip instance I have running on an android 6 phone is only using 82MB of data storage so we definitely have something different somewhere!

I also noticed your ps where you talk of reporting from the xDrip database. I think this shows we both have a different approach to data management. My approach is not to rely on a database carried on a mobile device so I upload to nightscout in realtime and report from there using tools like nightscout reporter, the nightscout built in reports and APIs and querying the backend database directly. I'm not expressing a view about right and wrong, just highlighting we have different approaches - there are many way to achieve most goals. Not relying on data stored on a device I can drop or lose just feels safer to me!
There does appear to be a facility for xDrip to upload to InfluxDB but I have no experience of using it...

[bsaunder2002]

Jon,
Thanks for the heads up about nightscout reporter. I fought getting nightscout working on a personal server. Never really got it going the way I wanted. Then COVID and I sort of just focused on my phone.
Nightscout Reporter looks really nice (demos on youtube).

I'll set it up again and play with that.

[bsaunder2002]

Ok, I got my space back.
Had to use built in File Manager to gain access to the internal storage folders. 3rd party apps would not work as I do not have root.

Uninstalled xDrip+.
Scanned with disk usage...still using 4.3GB.
Used file manager to search for xdrip and delete everything and then export.
I think the offending file was export...sqlite3. Viewer didnt show full name.
I deleted that. Viola... my disk space is freed up.
Reinstalled xDrip+ and will watch space.

(I suspect maybe the import function trying to import a huge file with the errors listed above tanked and left that .sqlite file hanging around. Really recommend xDrip+ have a feature to store all data and configs on a SD card. Really recommend Android die a painful death. Oh well.)