OpenStreetMap
Recently, Maxar has seen sharp usage increases through automated requests coming from a few areas of the world that are of concern. These are having detrimental impacts to our service and we are taking measures mitigate. Unfortunately this will mean an interruption of our imagery services for OpenStreetMap. Soon we will suspend our imagery service, and we are working with the developers of OSM editing tools to implement a more secure service and restore full access to the Maxar Standard and Premium layers as soon as possible.
Thank you for your understanding and we look forward to solving this problem together.
Best, Kevin
Абмеркаванне
Камэнтар Stereo ад 19 Снежань 2019 у 18:01
What a pity. Maxar’s imagery is a crucial resource in many places, and the OSM community is very thankful for it. I hope that a solution is found soon. I understand you’re in touch with Bryan Housel, but can you please keep the maintainers of the other editors in the loop too? Shoot me a message if I can help connect everyone.
Камэнтар SimonPoole ад 19 Снежань 2019 у 18:01
See https://github.com/osmlab/editor-layer-index/pull/548
Камэнтар Stereo ад 19 Снежань 2019 у 18:03
@SimonPoole you’d ask every user to sign up for a Maxar API key? Do a bit of oauth magic to grab one automatically? Hmm.
Камэнтар @kevin_bullock ад 19 Снежань 2019 у 18:18
Thank you @Stereo and @SimonPoole - yes we are working with Bryan Housel and user authentication is definitely interesting.
Камэнтар SimonPoole ад 19 Снежань 2019 у 18:31
@Stereo we have over a decade with “per editor app” keys (bing) and that has worked reasonably well (at least for those that have bothered to get their own keys).
Longer term probably per user keys are the way to go. Writing a small web app that gets an api key and stores it in the user prefs shouldn’t be a big issue allowing all editors to access it (I’m not holding my breath on that actually happening though).
Камэнтар SimonPoole ад 19 Снежань 2019 у 18:35
And the other part of the discussion https://github.com/MarcusWolschon/osmeditor4android/issues/775
Камэнтар mmd ад 19 Снежань 2019 у 20:35
Just adding a pointer to previous work by Roland on API keys: https://github.com/openstreetmap/openstreetmap-website/pull/2145
Камэнтар tgertin ад 19 Снежань 2019 у 21:04
Is it possible to add oAuth to the imagery?
Камэнтар tgertin ад 19 Снежань 2019 у 21:07
because then users would automatically be able to see the imagery when they log-in with their OSM account only
Камэнтар RebeccaF ад 19 Снежань 2019 у 23:09
Hi Kevin, thanks for sharing this. Please keep us updated on any timelines you are aware of so that we can better communicate to people who are currently using the imagery on what to expect! When the service is taken offline, do you have any info on how it will appear to mappers (especially thinking about mappers which are mapping on the Tasking Manager on a project that has imagery default set to Maxar)? Thanks so much for taking time to communicate on this, I realise it must be a very busy time, so really appreciate it! Rebecca
Камэнтар Stereo ад 20 Снежань 2019 у 00:04
https://josm.openstreetmap.de/ticket/18440 is the ticket on the JOSM side.
https://github.com/bryceco/GoMap/issues/242 is the ticket for GoMap.
Камэнтар Warin61 ад 20 Снежань 2019 у 08:55
In my area of the world Maxar tends to have the most up to date imagery, so it is a useful check that things still exist or not. I do tend to use other imagery for detail as some of them have more resolution.
Камэнтар Jorieke V ад 20 Снежань 2019 у 10:03
Hi Kevin, Really a pity to hear! Any chance you know if the imagery will be back online soon? We do have currently some tasks on the HOT Tasking Manager to support our MSF teams on the ground where Maxar Imagery is the only imagery that is of a reasonable date for the support that our teams need. If it will take a while, we might search for other solutions. Thanks! Jorieke
Камэнтар Verdy_p ад 20 Снежань 2019 у 10:43
Probably if Maxar can’t support the increase of usage on its own servers a caching server hosted by OSM-supporting sponsors will act to relay the usage while enforcing a low usage on the Maxar’s resources.
I hope that this will not avoid the access to Maxar’s resources for specific areas in emergency (e.g. for HOT projects for a limited time, until these resources are cached by OSM for longer term usage, e.g. when improving or correcting some areas or checking at the history, which can occur for years but in spreaded areas but at much lower frequency).
We should alreayd have an infrastructure allowing to host multiple caches for backup/recovery and permanence of the service, while limiting the direct interaction between tens of thousands of users over very large areas at all scales (notably for images in high resolution, which can easily escape the capacity of caches). The cache servers we need for OSM need to have large caching capacity and long refresh time, and should be compliant with “modern” HTTP/1.1 requests (“If-modified-since:” headers or similar) that can save a lot of bandwidth, and possibly should support modern compression algorithm and modern transport layers (e.g. HTTP2, i.e. the Google’s prototol), including streaming and queued requests on the same shared socket, while also preserving the necessary metadata (notably those for copyright and attribution, and notices for the usage policy).
If there were abusive uses recently, may be this can be traced to specific clients not complying to the rules or having implementation bugs. If not, we could trace back to the users refusing to endorse the tile usage policy, or could come from users using small devices with insufficient local storage, or users trying to use some advanced tools (not normal browsers) like Wget to perform automated queries bypassing all caches with a high frequency of requests: systermatically refusing to honor the caching date limits should be considered an abuse (note: users may want to clear their caches occasionnally.
Studying the serverside logs to identify the sources of these “abusive” requests should be interesting. If there are bugs in some of our applications for some situations (e.g. for users conencted via some proxies), users should be informed and there should be a way to notify them that their local proxy does not comply the rules, and that they should select a more appropriate proxy or compliant VPN if possible.
This may be a problem for users in some countries that cannot escape the mandatory proxy set to them by their ISP and government: for these users, we should have a solution to propose to them, by proposing them to apply for an account on a compliant proxy we can support (such usage should ne nominative, and should have its own policy : for personal use only, and only for use in an OSM editor, not for use in any web site or dedicated mobile application
(e.g. for a merchant site: merchants should have to implement their own caches for their own users, and ensure their storage has a cumfortable size and long enough conservation respecting the caching time limits for at least 85% of requests or better; and they should have monitoring statistics for the efficiency of their cache, i.e. hits/misses vs. requests and some history of these statistics to detect possible caveats and ensure correct and rapid adaptation of their own policies, or allowing their operators to consider increasing their storage and possibly update/fix their proxy softwares, firewalls, routings and so on.).
I hope that Maxar’s decision was not based on very temporary spikes that may have been caused by some recent updates in client softwares or changes in protocols (e.g. a major update in some popular web browser like Chrome, Firefox or Safari causing this temporary caveat for many users over a short time, or a major update of the OS enforcing the reinstallation of the browser and clearing of its local cache; normally major OS updates are scheduled in the world over a period of a few days so that not too many people will upgrade to the same time, causing major usage spikes also on OS-provider’s sites, and there should be a large tolerance for updates needed so that not all updates need to be made to continue a service, even if it’s not the best tuned for future uses, and users should have a resonnably long delay to upgrade, except for very serious security issues that should be fixed by strictly minimal changes on updated systems)
Камэнтар Warin61 ад 20 Снежань 2019 у 21:26
Could HOT projects cache the tiles it uses - reducing Maxar loads?
Камэнтар dekatherm ад 21 Снежань 2019 у 02:27
Which areas of the world are stealing your imagery?
Камэнтар Verdy_p ад 21 Снежань 2019 у 04:59
May be this is related to a recent Mapathon (announced after fact) in South East Asia. If this is the cause, such events should have been prepared by making sure the imageries proposed were cached.
I suggest that large mapthon events instruct their participants to use a local caching proxy for their web configuration, rather than a direct Internet access.
Small mapathons may not need that, but large mapathons organized online should negaciate their imagery source with a dedicated proxy (at least for the time of the main event, such instruction may be removed later to use usual imagery).
If this is caused by some HOT project, HOT project admins should corectly configure their projet and monitor the activity on the HOT tasks before removing the specifically proxied imagery source.
Камэнтар Joseph E ад 21 Снежань 2019 у 07:41
Bummer! In my area (Indonesia) a large number of roads were added to the database using Digitalglobe / Maxar imagery. Now it is temporarily not possible to check if this date is correct, because Maxar usually has the most recent imagery available here in New Guinea. I hope this problem is resolved soon for JOSM
Камэнтар Verdy_p ад 21 Снежань 2019 у 08:31
So the huge spike seen by Maxar was in Indonesia, this may be related to a recent large mapathon that occured there (see the recent OSM Newsletters that speaks about it, but not the fact it blocked Maxar now for all OSM contributors).
OSM Indonesia should really try to negotiate a safer hosting solution for the imagery, or should ask for help from the community to find this.
It’s true that Digital Globe imagery is extremely useful for many developing countries and most HOT projects.
I think that multiple OSM-related organizations could discuss about implementing a large CDN for caching and delivering imagery tiles (Maxar could also participate itself to a part of the CDN on its own caching servers, and the OSM Foundation or HOT could work on coordinating them, using dynamic DNS to distribute and monitor the work load of participating servers). Other participants that have servers/storage/bandwidth capacities could help (e.g. the Wikimedia Foundation, OSM Germany, OSM France, and their own sponsors).
Камэнтар Boggedy ад 22 Снежань 2019 у 15:49
Caching does not remedy outright misuse, it can increase it if anything.
I would be more inclined to go the oauth route so that individual users who abuse the imagery permish can be blocked fast.
Камэнтар Verdy_p ад 22 Снежань 2019 у 16:09
Of course caching helps. Notably for the issue above which was clearly related to a large Mapathon event in Indonesia, and not related to misuses by individual users (unless demonstrated, but Maxar does not detail the cause publicly, and may be already discussing about the iccue privately with OSM application maintainers or the OSM Foundation to avoid disclosing personal data publicly).
OAuth may eventually help against a few OSM users but this can already be tracked by OSM subscriptions and accounts, as msot of this usage seems related to active edits, that OSM servers may already track back to the same IP sources as those detected by Maxar of their imagery server (we cannot compare these usage logs, both are protected for good privacy reasons).
Maintaining an imagery source normally jsut requires creating a good caching proxy (or several ones). And OSM itself has its own tiles usage policy for the same purpose on its own tile servers, meant to be used by contributors and not by third party sites that should create their own cache for their web services (they are not all required to create map rendering servers).
Clearly this is noit a question of copyright, licence or authentication and prior authorization, but about usage policy. What the above discussion demonstrates is that Maxar provided good imagery sources for us, and they did not anticipate the success. Instead of blockiong everything, they should have implented some quota checks and reasonnable delays for delivering updated images, and should have asked the community for help to mirror their content on additional proxies.
This is a serious issue that we we should consider because we really need these imagery sources to work with, and such event may be something breaking any tentative applications for granting us more imagery data. We should come with a solution based on a generic CDN that could mirror all the imagery we want to use. And many OSM participating organizations can help for that or can provide additional means to create it, by motivating their own users to donate or help maintaining such CDN. For now only the base OSM rendered tiles map has a CDN. This should continue for other imageries. And we could say to providers that we can mirror this to help reduce the storage/bandwidth.
As well the usage policy can be enforced a bit more in OSM editors (iD or JOSM mainly) using better caching strategies, but many users don’t necessarily contribute with a device with high storage capabilities (notably for mobile devices which are the most widely used in developing countries for which Maxar currently provides some of the best imagery data).
Камэнтар Warin61 ад 22 Снежань 2019 у 20:56
It is an issue that OSM will need to deal with. If OSM does not deal with it then other imagery providers may well fall to the same over use problem.
Камэнтар Verdy_p ад 23 Снежань 2019 у 00:23
Than ks Warin61. At least someone that sees that this is a serious problem for sustainable development of OSM. Otherwise we would have to severely limit our objectives and many OSM projects would have to be abandonned, and possibly lot of data would have to be simply erased because becoming obsolete and unmaintainable in many development countries.
There’s already been several signs of this before, including with the partial retreat of Mapbox (building its own proprietary infrastructure on top the OSM base layer) and other providers stopping to grant us access to their sources.
It’s the whole credibility of OSM that becomes in danger (and a new possibility for Google or major commercial providers for once again increasing its presence, its prices, and its right to limit/filter data as they want).
We need a common CDN for sharing all granted imageries. And implement a repository of sources that can scale better for the many new goals we have developed.
Otherwise OSM will no longer be open to any contributors but will only be usable by import bots maintained by a few, still using opendata, but without review and we’ll become under influence of only governmental efforts (or lack of).
Камэнтар Vincent de Phily ад 23 Снежань 2019 у 13:44
The technical side of creating a proxy shouldn’t be too hard: get a server with a big disk and a lot of bandwidth, restrict access to OSM accounts via Oauth, and make sure different actors (not just OSM Foundation) can setup and federate caching for the same sources, and make it easy to update the software and sources list.
That’d (hopefully) solve Maxar’s issue, but would also be a clean ready-made solution for things like Strava or one-off HOT imagery.
The trickyer part of the problem is financing the bandwidth and worldwide servers. Bing is big enough that OSM takes just a small fraction of their resources. Mapbox has OSM in its DNA. For smaller and less engaged players, we’ll need to foot the bill, wether OSMF directly, using a big sponsor, or using lots of small sponsors and great federation..
Камэнтар SimonPoole ад 23 Снежань 2019 у 13:58
Could please everybody stop feeding the troll?
Maxar has neither indicated any specific region or technical issues as the reason for their issues, and further it is clear that proxying/caching would not solve the issues of bad actors overusing services (see the problems we have with the cache network for the standard style tiles).
Камэнтар mmd ад 23 Снежань 2019 у 14:05
They indicated elsewhere (obviously without disclosing any details) that bad actors outside of OSM are scraping imagery. impacting their service.
This HOT mapathon stuff mentioned earlier on is pretty much fake news, after all.
Камэнтар @kevin_bullock ад 23 Снежань 2019 у 21:08
Hi everyone, thanks for all your feedback. I can assure you that the requests were machine made, and automated in nature. Per the comment above, having nothing to do with OSM activity, whether that be a mapathon or something similar in nature. We’ve hosted and served imagery for OSM for years now and are very familiar with mapathon and other OSM patterns. Again, thank you for your feedback. Kevin
Камэнтар Darren B123 ад 24 Снежань 2019 у 10:58
Hello
Maxar Premium was my favourite imagery for mapping Nepal. It’s significantly more up-to-date than Bing, and clearer! Back to Bing I go …..
I look forward to the return of Maxar!
Камэнтар geow ад 25 Снежань 2019 у 12:05
Hello, as a workaround you might use the iD-fork RapiD. There the Maxar data is still available as background layer “Facebook’s Map with AI - Maxar imagery”.
But beware, I noticed for example in Nepal that Facebook’s AI wrongly classifies quite a few streams and ravines as suggestions for „Roads”. It appears that only optical pattern recognition is used and no DEM for plausibility checks, see e.g. https://mapwith.ai/rapid#background=fb-mapwithai-maxar&disable_features=boundaries&map=16.60/29.39309/82.93539
Камэнтар kirsanov ад 26 Снежань 2019 у 15:48
Hi geow, thanks for the feedback – you are right, there are some false-positives in RapiD (dry river beds, canals, narrow beaches, glacier cracks); this is why we built it to make sure ML data is only a suggestion and the user is in charge to make the final decision. We are working on improving the models, but it’s a pretty challenging task!
Камэнтар Darren B123 ад 27 Снежань 2019 у 12:15
Thanks for the info, Geow
Камэнтар DaCor ад 30 Снежань 2019 у 20:18
Hi kevin, I understand the issue and that its being worked on. Given the time of year I don’t expect a quick resolution but would you have any idea of an ETA for a fix?
Камэнтар jbergmann91 ад 2 Студзень 2020 у 15:00
Echoing the sentiment above! Is there any information you can provide on an updated timeline for when Maxar might become available again? It’s been so critical for a lot of community mapping as it’s been the most up to date and often of highest quality, so it would be great to understand what we can communicate to communities and organizations creating (or already mapping) tasks!
Камэнтар NunoCaldeira ад 3 Студзень 2020 у 10:14
FIY those that miss Maxar imagery to map on OpenStreetMap, it’s still available on RapiD.
Камэнтар Wulf4096 ад 5 Студзень 2020 у 10:39
Doesn’t work on https://mapwith.ai/rapid either.
Камэнтар geow ад 5 Студзень 2020 у 11:22
@Wulf4096, it still works, in RapiD simply select the background layer “Facebook’s Map with AI - Maxar imagery”, see my link above.
Камэнтар Wulf4096 ад 5 Студзень 2020 у 11:31
Works in Chromium, but not Firefox…
Камэнтар geow ад 5 Студзень 2020 у 11:41
no problem with me neither in Safari nor in Firefox 71.0 (macOS 10.13.6)
Камэнтар XnL ад 8 Студзень 2020 у 13:35
It works, but not same. For example, remote places like this: osm.org/#map=13/66.9712/179.1161 with Maxar Premium Imagery showed details of river, but not Facebook Maxar, not ESRI Imagery
Камэнтар _PG_ ад 8 Студзень 2020 у 14:51
“Maxar via Rapid” TMS-layer now works only in Rapid site (mapwith.ai). Seems referer filtering enabled.
So it is no way to use this layer directly in JOSM now.
Камэнтар sjdvda ад 20 Студзень 2020 у 05:08
Unfortunately, it appears that the Maxar imagery in RapID is much older (>6 months) than the one previously available on OSM for my location - a lot of new highway interchanges and a new train line are missing. Does anyone have an update on when we might get the updated Maxar imagery back in ID?
Камэнтар @kevin_bullock ад 20 Студзень 2020 у 05:23
Hi all - we are close to having the Maxar services back up and available. Thanks to everyone for their patience and input. We are testing the new services with the HOT team to make sure it’s ready for the broader OSM community. Sorry for the delay in updating this post, our team at Maxar has been working on a new, secure implementation. As I have more news/info regarding the testing I will post here, Kevin.
Камэнтар Verdy_p ад 20 Студзень 2020 у 06:16
note that 6 months is not exceptionnaly old in imagery sources; most of them are only refreshed once in about every 3-10 years. They may be yearly updates, but only to integrate the coverage of some subareas. Some of them are refreshed more often because there are collectivities paying for them (and I’m not speakingabout satellite imagery, whose resolution is quite poor, but really about aerial imagery, made from planes, or very high resolution imagery made with drones that cannot cover a very large area and generally cover an area not larger than a typical village or small town). this requires people, time of work from them on the field, or paying the cost or plane flights and the personal driving them and maintaining them. This require time to process the images, qualify those that are usable (e.g. dismiss cases where images are blurred by clouds or sun flares on the objectives, or data losses and incorrect encodings, or cases where images are highly deformed by the elevation and there’s still no consistant data on the elevation at a sufficient resolution, or because the precise location of the images were lost with reception errors from GPS mesurements during the capture). Imagery is not free (as a beer), there’s necessarily someone paying for them (even if it’s a non-profit NGO which also has limtied resources and depends on donations and free participation of volunteers).
For a newer infrastucture that is still not on the imagery, you can still estimate its location by comparing with local snapshots you take on the field or that your find in services like Mapillary. This is generally enough to make a draft but quite good estimation with low margin of errors for the positioning. you can still draw that in OSM and place a “fixme” tag indicating that the estimation may be reviewed later for better precision if needed, you can easily measure the margin of imprecision from the existing imagery.
The only case where this is problematic is after a major castrophic event that radically changes the field (like cyclone, volcanic eruptions, large fires, and landslides) so that it will never be the same again. But for these events, there are NGOs helping and taking imagery captures rapidly, plus the participation of aerial imagery organization to provide a recent source of data in those most affected areas, because they are needed to manage the emergency and plan the recovery with lot of local participants and many volunteers from around the world.
So 6 months is not old. We are much better interested in getting higher resolution images that allow feeling the gaps generally easily, with the additional local knowledge (and in your case, a train line is a large object which has a very smooth geometry, it’s quite easy to be very precise by locating precisely only very few points. You’ll need the imagery only for details (e.g. the exact position of side tracks and crossings from one track to another, or some local equipement along the line, like the position of poles suppring the electric feed or the exact position of a bridge (but note that bridgers for trains take months to be built and consolidated: in 6 months, you may already see the work in progress or the bridges already there, even if there’s still no rail on it and the line is still not fully connnected to the rest of the network.
Камэнтар Jorieke V ад 20 Студзень 2020 у 11:37
Thanks a lot Kevin!
Камэнтар mirror176 ад 10 Люты 2020 у 10:23
Lost the higher resolution (at least in my area) digitalglobe premium then this was a major saddening moment. Still patiently looking forward to the return though glad you guys fight the bad actors. Any chance work has progressed in trying to get a way we can find approximate imagery age too? Newer lowres + older hires imagery combinations are helpful for mapping. Always wondered if it was possible to get imagery for my handheld garmin gps I use for my on the go osm notes at the moment but gotta say that lazyness kicks in for me as I usually doubt the effort would be justified. Thanks again!
Камэнтар @kevin_bullock ад 14 Люты 2020 у 21:38
The Maxar imagery layers have been restored in ID Editor! Huge thanks to Bryan and Quincey from the ID team. And thanks to all OSMers who have been patient and provided a wealth of feedback and gratitude. Let’s get editing!!
Камэнтар DaCor ад 14 Люты 2020 у 22:38
Thats fantastic news, thank you!
Any info on this imagery for Josm and others?
Камэнтар RebeccaF ад 14 Люты 2020 у 22:49
Thank you so much Kevin, Bryan and Quincey!! Your efforts are SO appreciated!
Камэнтар @kevin_bullock ад 15 Люты 2020 у 15:41
@DaCor / All - JOSM has just pushed a new release and the Maxar layers are now available. Huge thanks to Vincent and the JOSM team!!
Камэнтар Theo Asselman ад 16 Люты 2020 у 13:48
Thanks for all the effort and bringing Maxar back to us!
However, there’s maybe still some waitingtime before JOSM picks it up again? I updated to 15806 but the Maxar tiles don’t showup. (I can see Bing of Esri.)
And Googling on the error I got, well, even Google has no idea.
https://drive.google.com/file/d/1Gh4noXxxmo3WqzZyL6JE2D0Y6dIsroZj/view?usp=sharing
Anything I missed or I should do?
Камэнтар DaCor ад 16 Люты 2020 у 13:52
Kevin, thank you for that, it’s really appreciated
Камэнтар XnL ад 16 Люты 2020 у 14:20
@Theo Asselman JOSM restored access to Maxar Imagery in version 15855, so you can wait for next stable release or temporary use Development version
Камэнтар Theo Asselman ад 16 Люты 2020 у 14:30
Ah, thanks XnL! I switched to 15859 and there they are! :-D
Камэнтар Darren B123 ад 16 Люты 2020 у 14:41
Great news! Thanks :-)
Камэнтар imagico ад 18 Люты 2020 у 14:38
Do i see it right that there is only one Maxar image layer now although both iD and JOSM still provide separate entries for ‘Standard’ and ‘Premium’? iD seems to use different API keys for them but they are still the same, JOSM adds a ‘foo’ parameter but it does not seem to have an effect.
Камэнтар CENTRALHUB ад 20 Люты 2020 у 21:51
Thanks for the update and we are glad that the Maxar Imagery service comes back.
I am wondering is there any way that we can report the quality problem of Maxar imagery. Sometime I found that in some are Maxar is elder than the images provided by other services (like ESRI or bing). In some places the Maxar imagery is serving worse quality than before thought its data is newer. Major issues can be foggy images or serious distortion. Besides, in some cases Maxar is missing data for populous city or town for many years - it will be nice if we can see the data in this area in the future.
Thanks for everyone’s efforts.
Камэнтар Kovirii ад 10 Люты 2025 у 10:21
Still nothing?