Don't understand BaseRegion & more

[bmason8]

Looking for some help with downloading offline maps and in particular how to create a BaseRegion for passing into the DownloadableRegion constructor.
The 'startDownload' function take a DownloadableRegion as it's second parameter and since I get the areas I want downloaded via an api so I don't have the need for the user to draw a box for an area.

So my issue is that when I go to create a BaseRegion, it's an abstract class with three function that return either a:

• DownloadableRegion
• PolygonLayerOptions
• List<LatLng>

So how am I supposed to create a DownloadableRegion if one of the parameters is a BaseRegion which itself generates a DownloadableRegion?

I'm certain this is just me not understanding what's going on but I feel like I'm in a chicken or the egg type situation.
Any help would be appreciated!

[JaffaKetchup]

Hi there, and thanks for your question! Below I have written an answer based on what I think your question is: there's a high possibility that I've misunderstood you though, so please don't hesitate to correct me if I'm wrong :)

BaseRegion is an abstract class that defines the 'conversion' functions for all regions - you should never need to construct this yourself. DownloadableRegion contains a BaseRegion and other information needed for downloading. I think you got this part: the part I don't think you got (for reasons I'll explain later) is that you should never need to construct DownloadableRegion yourself either.

To resolve your issue, you'll need to use one of RectangleRegion, CircleRegion(, or LineRegion as of v4). All of these extend BaseRegion, and represent different shapes.
Then, to construct a DownloadableRegion, take your ...Region and use .toDownloadable() on it. You can then pass this to the download functions.

I think the probable reason you got stuck here is because the documentation for this library makes it hard to setup. I am gradually working on this for the next update.
Having said that however, here is another reason): I think you may be using v3. Please upgrade to v4.0.0-dev.6 from pub.dev: it is 99% stable & production-ready, and is unlikely to have more breaking changes.

The major thing about v4 for you though, is that DownloadableRegion's constructor is marked as @internal, which would've alerted you to the problem in your editor, thus allowing you to try another way.
DownloadableRegion can't be abstract or private because I need to use it internally and elsewhere in the library (not in the same file).

I hope you found this easy enough to understand, and I hope it resolved your issue!

[JaffaKetchup]

Thanks for your answers!

Don't worry (if you were) about ignoring the warning, it happens to the best of us and I certainly understand that I need to improve documentation on things like this!

About the example just being a big block of code, I plan to actually create a whole example package at some point sooner or later, which would make it much easier to understand and take only the necessary parts.

I provide the pre-download checks functionality because it provides information about the battery as well, and should further reduce the code needed for many developers.

Performance issues are unfortunately to be expected until I can implement my own ImageProvider. At the moment tile provider's (like mine) override the getImage synchronous function (synchronous because it returns an ImageProvider), meaning that all of my file handling and checks have to be done synchronously before loading the tile to the screen. Likewise to the example, I plan to implement a custom ImageProvider (which will be asynchronous) at some point - I've experimented before, however, and it was quite complicated for me, so it may or may not end up in v4.
The example has further performance issues because I built it badly and rushed with getting everything to sort of work: this has led to many variables in many states and the widget being re-setStated almost constantly. Again, the example rewrite ahead should fix these issues at the expense of a bigger example package.

That network call is sort of a 'half-bug', ideally it wouldn't be logged, but I'm not sure how to silence it. Besides, using the caching rules (can't remember what they're called right now, but they are a enumerable property of StorageCachingTileProvider) should allow you to select whether a network call is made in the first place.

However, your last paragraph about the storeName functionality also seems to be a 'half-bug'. New stores are automatically generated when bulk-downloading; but thinking about it now, not necessarily when browse-caching. I don't know if you knew, but you can use createStore() on a MapCachingManager instance to force the creation of a new cache. Anyway, I'll try to fix this before release.

Again, thanks for your answers! If you need anymore help, feel free to ask here or in a new issue/discussion.

[bmason8]

That's great news about an example package. I'm sure it'll be a lot of work so it's appreciated!

No problem regarding the network call logging (I didn't properly explain that I'm getting more than just that log output). This is no problem because it's actually just a log message:
flutter: SocketException: Failed host lookup: 'api.mapbox.com' (OS Error: nodename nor servname provided, or not known, errno = 8).
However, I'm getting and actual error that looks similar but includes a bunch of other stuff: I'll post it below but will need to alter it to remove my access token just fyi): Sorry for the formatting I didn't know how to fix it lol.

======== Exception caught by image resource service ========
The following SocketException was thrown resolving an image codec:
Failed host lookup: 'api.mapbox.com' (OS Error: nodename nor servname provided, or not known, errno = 8)

When the exception was thrown, this was the stack: 
#0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:677:35)
#1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1680:26)
#2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)
#3      RawSecureSocket.startConnect (dart:io/secure_socket.dart:237:22)
#4      SecureSocket.startConnect (dart:io/secure_socket.dart:60:28)
#5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2437:24)
#6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2805:12)
#7      _HttpClient._getConnection (dart:_http/http_impl.dart:2810:12)
#8      _HttpClient._openUrl (dart:_http/http_impl.dart:2694:12)
#9      _HttpClient.getUrl (dart:_http/http_impl.dart:2573:48)
#10     NetworkToFileImage._downloadFromTheNetworkAndSaveToTheLocalFile (package:network_to_file_image/src/network_to_file_image.dart:204:57)
#11     NetworkToFileImage._loadAsync (package:network_to_file_image/src/network_to_file_image.dart:164:23)
#12     NetworkToFileImage.load (package:network_to_file_image/src/network_to_file_image.dart:120:16)
#13     ImageProvider.resolveStreamForKey.<anonymous closure> (package:flutter/src/painting/image_provider.dart:505:13)
#14     ImageCache.putIfAbsent (package:flutter/src/painting/image_cache.dart:380:22)
#15     ImageProvider.resolveStreamForKey (package:flutter/src/painting/image_provider.dart:503:83)
#16     ImageProvider.resolve.<anonymous closure> (package:flutter/src/painting/image_provider.dart:333:9)
#17     ImageProvider._createErrorHandlerAndKey.<anonymous closure>.<anonymous closure> (package:flutter/src/painting/image_provider.dart:464:26)
#18     SynchronousFuture.then (package:flutter/src/foundation/synchronous_future.dart:41:35)
#19     ImageProvider._createErrorHandlerAndKey.<anonymous closure> (package:flutter/src/painting/image_provider.dart:461:11)
#23     ImageProvider._createErrorHandlerAndKey (package:flutter/src/painting/image_provider.dart:453:16)
#24     ImageProvider.resolve (package:flutter/src/painting/image_provider.dart:330:5)
#25     Tile.loadTileImage (package:flutter_map/src/layer/tile_layer.dart:1267:36)
#26     _TileLayerState._addTile (package:flutter_map/src/layer/tile_layer.dart:1072:10)
#27     _TileLayerState._update (package:flutter_map/src/layer/tile_layer.dart:1014:7)
#35     throttleStreamTransformerWithTrailingCall.<anonymous closure> (package:flutter_map/src/core/util.dart:46:12)
#36     throttleStreamTransformerWithTrailingCall.<anonymous closure>.<anonymous closure> (package:flutter_map/src/core/util.dart:52:26)
(elided 21 frames from class _RawReceivePortImpl, class _Timer, dart:async, and dart:async-patch)
Image provider: NetworkToFileImage("/var/mobile/Containers/Data/Application/E669647B-8BC4-46D8-86B7-5738C87DD91B/Documents/mapCache/outdoors/apimapboxcomstylesv1mapboxoutdoors-v11tiles1425745667?access_token=MY_TOKEN", "https://api.mapbox.com/styles/v1/mapbox/outdoors-v11/tiles/14/2574/5667?access_token=MY_TOKEN", scale: 1.0)
File: /var/mobile/Containers/Data/Application/E669647B-8BC4-46D8-86B7-5738C87DD91B/Documents/mapCache/outdoors/apimapboxcomstylesv1mapboxoutdoors-v11tiles1425745667?access_token=MY_TOKEN
Url: https://api.mapbox.com/styles/v1/mapbox/outdoors-v11/tiles/14/2574/5667?access_token=MY_TOKEN

• this is @JaffaKetchup fixing this formatting -

I've done some more testing and either something more is going on or I'm just doing things incorrectly but this is what I've found:

1. If I use the StorageCachingTileProvider with only a parentDirectory it works great (same performance as I'm used to on v_2.. versions). Doesn't matter if I'm online or offline/airplane mode, it works great.
2. If I add the storeName parameter it causes the performance issues I mentioned before. I thought it was only if I was offline but it actually ruins performance when online as well. I've also tried with behavior: CacheBehavior.cacheOnly but it made no difference.

So am I doing something wrong with how I'm using the storeName parameter or could it be some type of bug? I didn't know you could use createStore() to create a new cache or that they are automatically generated when bulk downloading. I'm hoping to use the stores and bulk downloading in the following way:

• User has 2 map styles available to them. Anytime they switch styles, I switch the storeName so it caches the different styles (even for the same area) when panning around.
• If they enable 'Offline Maps' -> it requires them to choose which map style to have downloaded (ie storeName sub folder) and then downloads the tiles into that storeName subfolder. Are there any issues you see with that approach? Would duplicate tiles that are already in that store/sub folder just get overwritten or would that cause an issue?

Below is a bit of an example of how I'm using it.

This runs in a setup function before the map gets loaded.
mapTilesParentDirectory = await MapCachingManager.normalCache; if (!mapTilesParentDirectory.existsSync()) { mapTilesParentDirectory.create(); print('mapTilesParentDirectory created'); }

Then I use the tile provider like this:

TileProvider tileProvider;
    // To avoid a blank map -> use a NonCachingNetworkTileProvider until the setup is done (required because 'mapTilesParentDirectory' needs to be asynchronously fetched or created in setup)
    if (setupInProgress) {
      tileProvider = NonCachingNetworkTileProvider();
      print('Using NonCachingNetworkTileProvider()');
    } else {
      print('Using StorageCachingTileProvider(): storeName: $storeName');
      tileProvider = StorageCachingTileProvider(
        cachedValidDuration: Duration(days: 60),
        parentDirectory: mapTilesParentDirectory,
        storeName: storeName,
        // behavior: CacheBehavior.cacheOnly,
      );
    }

return TileLayerOptions(
        tileProvider: tileProvider,
        urlTemplate: "https://api.mapbox.com/styles/v1/mapbox/$type/tiles/{z}/{x}/{y}?access_token={accessToken}",
        additionalOptions: {'accessToken': mapboxApiToken},
        backgroundColor: Colors.blueGrey);

I also had the thought that maybe there was an issue because I wasn't creating the subfolder for the storeName so I added some code to create the folder at the end of the parentDirectory as well but it made no difference to performance either.
Thanks for your help and hopefully I'm just doing something stupid!

[bmason8]

After testing some more, I'm now seeing the same choppy map performance whenever (and only when) after bulk downloading tiles. Even if I never use a storeName when downloading and just use a parentDirectory. So if I just let the StorageCachingTileProvider cache on it's own, then go offline, it's great.
So I'm thinking I must be doing something wrong when downloading tiles.

[JaffaKetchup]

Hi again,

I fixed your formatting, so don't worry about that 😂. That error does seem to be because of a combination of my library, my dependencies, and flutter_map. I could potentially wrap the calls in try{} catch{} to silence that potential error: that error can be ignored if using this library safely, and I can't really prevent it easily.
If possible, could you try the same code and specify CacheBehavior.cacheOnly and see if the error still appears when offline?

However, this path from the log really worries me:

/var/mobile/Containers/Data/Application/E669647B-8BC4-46D8-86B7-5738C87DD91B/Documents/mapCache/outdoors/apimapboxcomstylesv1mapboxoutdoors-v11tiles1425745667?access_token=MY_TOKEN

I can see my library's proper structure here: <cache>/mapCache/<storeName>/<processedTileName>, but that question mark character is an invalid path character on Android systems according to multiple sources. Have you tested your app on Android yet? If you can, please check if it works, if you can't, I'll check it on my side. If it does work, then that's a lucky miracle and needs to be urgently fixed.

---

If I use the StorageCachingTileProvider with only a parentDirectory it works great (same performance as I'm used to on v_2.. versions). Doesn't matter if I'm online or offline/airplane mode, it works great.
If I add the storeName parameter it causes the performance issues I mentioned before. I thought it was only if I was offline but it actually ruins performance when online as well. I've also tried with behavior: CacheBehavior.cacheOnly but it made no difference.

This is extremely interesting to me, as I thought that there should theoretically be no difference, because omitting storeName just defaults to a store called 'mainStore'. If possible, could you create another store, maybe called 'secondStore' (or something else) and then delete the 'mainStore' (using the built-in .deleteStore() method), to see if the performance becomes ok again? If so, there may be performance problems when using multiple stores, potentially indicating a memory leak or running unnecessary code. I am sure you aren't doing anything wrong, this does seem to be my library.

---

mapTilesParentDirectory = await MapCachingManager.normalCache; if (!mapTilesParentDirectory.existsSync()) {mapTilesParentDirectory.create(); print('mapTilesParentDirectory created'); }

I understand why you need this, but I'm not sure why I didn't spot this in testing. Hopefully my fix to auto create the store directories will also solve this recursively.

---

User has 2 map styles available to them. Anytime they switch styles, I switch the storeName so it caches the different styles (even for the same area) when panning around.
If they enable 'Offline Maps' -> it requires them to choose which map style to have downloaded (ie storeName sub folder) and then downloads the tiles into that storeName subfolder. Are there any issues you see with that approach? Would duplicate tiles that are already in that store/sub folder just get overwritten or would that cause an issue?

This is a fairly common use-case, one I tried - and failed - to simplify back in v3 development. This should just work the way you described it. In answer to the first question, therefore, I can't see many issues except one: see #28. In answer to your second question, that depends on the preventRedownload setting when you create your DownloadableRegion from .toDownloadable(): if this is true, existing tiles (for example from browse caching) won't be overwritten, and vice-versa. Just make sure your internal variable and the MapCachingManager never go out of sync and start writing across stores; that would be quite messy.

Your code also looks good to me.

You appear to be doing everything right, so it's probably me 'at fault' :D

[JaffaKetchup]

Ooo, only just saw your latest response after I posted my long one!

I have also seen this performance issue in my example app.
Out of interest, what happens when you have only a hundred tiles to bulk download; ie. at what number of tiles does it start getting choppy?
Also, do you have any regular setStates in your app that might reload a MapCachingManager().storeSize?
One last question: what happens if you disable the recovery mode when bulk downloading? Does the lag go away?

Please make sure you only test as much as you can support whilst paying for your MapBox tiles (they aren't exactly cheap): if you can't test anymore, don't worry, just let me know :).

[bmason8]

I tried downloading tiles again with preventRedownload and it didn't make a difference.

I tried with CacheBehavior.cacheOnly and it made no difference unfortunately. I do get this message though while panning around with it on:
LateInitializationError: Local 'bytes' has not been initialized.

I can see my library's proper structure here: /mapCache//, but that question mark character is an invalid path character on Android systems according to multiple sources. Have you tested your app on Android yet? If you can, please check if it works, if you can't, I'll check it on my side. If it does work, then that's a lucky miracle and needs to be urgently fixed.
For all this last round of testing (I've using Android for this round of tests) after bulk downloading tiles, it does load them while offline (with jank) but only shows that error (that I posted the log of) when NOT setting behavior: CacheBehavior.cacheOnly. If behavior: CacheBehavior.cacheOnly is set, then I only get the harmless SocketException: Failed host lookup: 'api.mapbox.com' (OS Error: No address associated with hostname, errno = 7) message and as mentioned above LateInitializationError: Local 'bytes' has not been initialized.

Regarding the path that worries you with the '?'. Is that because you're just saving the tile with same name as the network call made to retrieve it? If so, that is probably not ideal if avoidable because that means the secret access token is stored in every single tile name. I'm not sure how the tiles need to be named for getting retrieved though so ignore that concern if it's required or not viewable by other apps.

Out of interest, what happens when you have only a hundred tiles to bulk download; ie. at what number of tiles does it start getting choppy?
Also, do you have any regular setStates in your app that might reload a MapCachingManager().storeSize?
One last question: what happens if you disable the recovery mode when bulk downloading? Does the lag go away?
Please make sure you only test as much as you can support whilst paying for your MapBox tiles (they aren't exactly cheap): if you can't test anymore, don't worry, just let me know :).

Firstly, thanks for the reminder about downloading tiles! I totally wasn't thinking about the cost so I'm sure my company wouldn't want me racking up tile download costs! lol.
So during all my initial testing I was downloading about 10,000 tiles so I dropped that to about 1,000 and it made a noticeable difference. Still not as smooth as when there are no tiles downloaded via bulk download but smoother for sure. It's very interesting to me that after I delete any tiles that were bulk downloaded, the map is instantly back to being fully responsive.
CacheDirectory directory = await MapCachingManager.normalCache; MapCachingManager(directory).deleteStore();

Maybe it's a bug or maybe I'm downloading things incorrectly. I actually download each area at the same time so I'm going to do another round of tests where I only download a single large-ish area to see if that makes a difference.

This is extremely interesting to me, as I thought that there should theoretically be no difference, because omitting storeName just defaults to a store called 'mainStore'. If possible, could you create another store, maybe called 'secondStore' (or something else) and then delete the 'mainStore' (using the built-in .deleteStore() method), to see if the performance becomes ok again? If so, there may be performance problems when using multiple stores, potentially indicating a memory leak or running unnecessary code. I am sure you aren't doing anything wrong, this does seem to be my library.

I haven't tried this but in the last two rounds of testing I haven't been using a store at all because the issue is present regardless. Seems like downloading tiles in bulk is messing things up (at least while saving them to the same main mainStore (ie not passing in a value and letting the default get used).
Let me know if there's anything more I can test!

[JaffaKetchup]

@bmason8 Hi,

I've been writing some more of the new example app, this time properly implementing custom sources per store, and I can still confirm that this is an issue.
I'm pretty sure it's not crossing stores (which is a good thing), so I think the issue is with the the way I've implemented my ImageProvider to solve those performance issues? I have to call PaintingBinding.instance?.imageCache?.evict(this); if the call for a tile fails (otherwise the tile will never load in, even if the problem resolves itself), and I have a feeling I need a similar thing to completely clear the image cache after a store switch.
Hopefully I can implement this fairly easily.

In other news, there's two new features:

• coverImage() in MapCachingManager, generates a cover image for a store.
• MapMetadataManager, a class 'interface' to persist map caching related data across restarts, separate to the main tile stores - for example to store the last used store or to store a source URL per cache store.

These will be pushed to GitHub, along with the work on the example app, soon.

[bmason8]

Awesome, that's good to hear. I tried so many ways to confirm what I was doing wasn't the issue but was still unsure so now I feel a little less crazy haha. Those additions sounds great as well.
Thanks again!

[JaffaKetchup]

Hi @bmason8,
Sorry, this is taking a really long time!

It turns out this is not going to be as easy fix as I thought, so for the time being, you may want to do one of these things:

• Rollback to a previous version where this works for you
• Find a workaround (perhaps using persistent storage to remember the last used store, like in the example, then using flutter_phoenix)
• Put up with the issue until I can fix it

I'm really sorry for the inconvenience.

[bmason8]

@JaffaKetchup No worries! I'm about to push out an update using the dev-7 version and will be using the NonCachingNetworkTileProvider for the satellite tiles until a fix is ready. My users are used to that behaviour anyways so it's no big deal. Thanks for all your work on this!

[JaffaKetchup]

@bmason8 Please see #15 (comment) and #31. #31 will be the new issue for all further discussion on this topic.
Thanks!