OpenStreetMap
Experimental publishing of Sentinel 2 satellite data
BushmanK 于 2016年七月13日 以 English 发布 最后一次更新于2016年七月14日。Recently, I’ve published several tiles of fresh Sentinel 2 satellite imagery using Nextgis.com free spatial data hosting service. All tiles are available here. Two tiles were published by request, others are covering two major populated territories in Russia - Moscow region and Saint Petersburg region, one tile was randomly picked, it covers western part of Republic Mordovia, famous for endless forests (and one of my goals was to test it as a source of information about logging) and high concentration of prisons, located there. For some tiles, both visible wavelength and visible+NIR composites were made. Workflow of making those composites was quite simple:
- Make a dump of georeferencing data from any 10m/pix resolution channel image
- Use
convert -combinefrom ImageMagick to merge single channels into 16-bit per channel RGB image. - Use
convert -contrast-stretchto manipulate image histogram - Put georeferencing information back, save resulting file as 8-bit per channel RGB image, compress it with Deflate or LZW
- Upload it to Nextgis.com, set up web map and WMS service.
(Steps 1 and 4 were made using GlobalMapper, but could be done using GDAL and Python script.)
I asked people to give me some feedback, however, only one person (not counting those who asked me to make two of those tiles) informed me that he used this information to update forest boundaries, changed by logging and wildfires. I’m obviously doing it not for any kind of reward or acknowledgement, however, I really don’t like to do anything nobody is going to use by whatever reason.
It also works as a kind of social experiment. Many times, when I mentioned Landsat 8 data in context of someone’s complains regarding of outdated/missing Bing/MapBox imagery, people were saying something like: “Oh, I’m not a programmer, it’s so hard to make those composites by myself and I don’t know how to use it”. To be precise, a couple of people managed to learn how to do that by their own. But now I gave everybody easily available data (with ready to use WMS definition strings for JOSM) and it doesn’t seem like anybody wants to use it (or, by some crazy reason, they don’t want to acknowledge, that they used it). So, lack of imagery or outdated imagery is just an excuse to do nothing. I’m not blaming anyone - OSM is a volunteer project, nobody has any obligations, everybody probably have own life and other things to do. But it would be nice if people just stop lying to themselves and to others that only obstacle for keeping something up to date is lack of fresh imagery.
Just in case if someone is interested, according to this, we can legally use Sentinel 2 data. Changeset or particular objects traced from it should have “Copernicus Sentinel data 2016” (or appropriate date) in source tag, as a requirement of informing data users about its source.
讨论
imagico 于 2016年07月13日 19:30 的评论
Yes, it is indeed important to show and demonstrate that up-to-date imagery these days is not a big issue any more and that there is no good reason to waste time mapping based on 15 year old images as they still can be found in Bing/Mapbox for large parts of the Earth. It is fairly frustrating if you see mappers spending a lot of time and energy on tracing from these when you know significantly better data is available. It would be really good if editors (JOSM, iD) would start actively suggesting to use other sources in areas where the standard image sources are poor. At least for Bing this is easy to determine from the tile metadata.
There are also a few Sentinel-2 images on the OSM images for mapping - like the Panama Canal, Glaciers in Peru and Antarctic islands.
In general i would recommend not using false color combination or unrealistically stretched contrast functions unless you have specific reasons for that for mapping certain stuff since it makes recognizing things in the images more difficult - especially if mappers have no or very limited on-the-ground knowledge of the area.
For Landsat by the way the USGS offers on demand WMS services for individual images but only in false color and without pansharpening - this can be useful if you quickly want an up-to-date image for verification but is not really useful if you want to get most out of the data.
Regarding attribution - what is written on the contributors page should be sufficient - for the changeset source tag it would be important to specify the scene ID or the image date.
BushmanK 于 2016年07月13日 20:24 的评论
@imagico,
Unfortunately for OSM, general level of proficiency even in basic remote sensing technologies is quite low, despite certain efforts of MapBox to change it by their blog posts on Landsat, for example.
But since there is certain significant difference between active usage of “alternative” data sources in HOT mapping and quite lazy attitude in case of regular mapping, it seems like it’s all about motivation. Just like in that saying: “One who wants to do something, looks for an opportunity and method. One who doesn’t want to, looks for an obstacle”. (By the way, that’s my reason why I don’t want to spend time on making detailed step-by-step tutorials how to process remote sensing data - if someone says he needs one, he’s probably uses it as an excuse too, because there are tons of information on this topic on the Web.)
Regarding of contrast stretching - if I wouldn’t do any, some details will be completely lost, since original dynamic range is huge and linear min-max stretch will give no real difference between forest, bushes, forested wetlands and other visually similar features. It’s also hard to differentiate this year, last year and older logging parcels without NIR data, since young trees and tall grass look similar in visible light.
imagico 于 2016年07月13日 21:22 的评论
Yes, there are good reasons for using infrared image data, especially if water and wetlands are concerned. You really need to know how to interpret what you see then though.
Color processing can be difficult, producing a color rendering that shows well articulated bright areas but also shadows and that at the same time is consistent with the color impression you get on the ground is a challenge.
BushmanK 于 2016年07月14日 00:07 的评论
@imagico,
Since we usually do have some high resolution imagery within an area, covered by large Sentinel 2 tile or full scene, it helps to establish certain level of understanding. However, without even basic remote sensing knowledge and skills, it’s barely possible to extract and interpret any significant amount of information from medium-resolution imagery.
To retain low contrast details on some tiles, I used CLAHE in form of ImageJ/Fiji plugin module. It really helps, and it’s okay to use it with relatively large window (256px or something), since nobody is going to use automated classification on this imagery. Combining slightly less aggressive contrast stretch with CLAHE gives nice results, as you can see.
imagico 于 2016年07月14日 10:13 的评论
It all depends on what you want to use the images for. CLAHE and similar techniques will work out details at the cost of large scale consistency, what color a certain area appears in depends not only on this area itself but also on its surrounding.
By stretching contrast the way you do you also inevitable loose information in the darkest and brightest parts (like mountain slopes facing away from the sun and cloud shadows) - which can for example be a problem at high latitudes where the sun is low even in summer.
In
http://bushman.nextgis.com/resource/4/display?base=osm-mapnik&lon=41.1627&lat=43.3709&angle=0&zoom=12&styles=3
for example you can very well see the limits of the forested areas but nuances in the dark forests and in the bright parts (like snow-clouds and rock-ice distinction) are fairly difficult to recognize.
PlaneMad 于 2016年07月14日 12:29 的评论
bushman, its great that you are sharing this for others. Its amazing what we can do now with Sentinel and Landsat, but too much is hidden behind complex workflows and cryptic commands. I think more work is needed to simplify these instructions for the common man, right now it still feels like most documentation is targeted at a developer who is comfortable with the terminal or had background theory of satellites.
In my spare time while playing around with such data, I document my journey so that it could be useful for others, and while writing i’m constantly thinking how to appeal to someone who is 12 years old to do this. If it helps, here’s some of my material.
BushmanK 于 2016年07月14日 14:52 的评论
@imagico,
I’m completely aware of these drawbacks, however, to utilize full extent of dynamic range in case of mountain areas, usage of non-smooth adjustment curves is required, or two (high key and low key) images should be produced by histogram manipulation and then - blended together with “if darker/if brighter” blending mode. It definitely takes time.
@PlaneMad, That work you are doing is important, but reading information you’ve provided, I see no way anyone can perform more or less effective image processing without any knowledge. Image processing requires certain level of understanding, because there is no universal method - processing of every scene depends on types of land use, types of vegetation, urban/rural populated land, climatic zone, season etc. Theoretically, you can make tens of instructions and make people trying different ones without any understanding until they get desired result. But even to see, if result is desired, one need knowledge and understanding. Basically, trained monkey can’t do image processing, only thinking human can. That’s why education is more important than training.
And, as I said, my experiment shows that even having ready-to-use data, many people refusing to use it. Probably, because their mind is inert, and they can’t just easily start using something new. Same thing happens sometimes, when Bing updates imagery. People complaining: “oh, that new image, it’s different, I can’t see familiar things”.
Zverik 于 2016年07月16日 21:31 的评论
How much of this Landsat 8 tutorial can be used for processing Sentinel 2 images?
BushmanK 于 2016年07月16日 23:41 的评论
@Zverik,
I’d say close to nothing.
convert -contrast-stretchand preserves georeferencing, but there is no tutorial, which will give you universally usable black point and white point parameters (ones in-percentile-range 0.05 0.95) - it depends on specific circumstances, where and when particular image was taken. User should understand, how to manipulate them or to evaluate image statistics to pick them initially.listgeoandgeotifcptrick - yes, sometimes - in case of data in GeoTIFF format (L1C data I’m using are in GeoJPEG2000, and I don’t know, if there is a source of GeoTIFFs, however, it’s possible to convert them with GDAL, but that’s a kind of awful workaround)So, bottom line is: it’s irrelevant. And I have to add, that certain decisions anyone has to make when cooking composites, require understanding instead of just careful following predefined steps. But you don’t really believe in things like that, as far as I remember. (And why people spending five years in universities, learning fundamentals, instead of just grabbing some step-by-step instructions and starting working like a pro?)
There supposed to be special software for Sentinel (and many others) data processing - STEP, but I tried it twice with no positive results - it spits out some error message, like it’s unable to read this particular format, but it happens all the time.
Another Landsat 8 turorial is way closer to practical solution (everything before NDVI calculation), but again, it doesn’t give any universal solution for choice of bands and for contrast stretching. It’s easy to play with different bands and parameters in QGIS, because it’s interactive. But again - certain understanding is required, because otherwise it would be endless trial-and-error. Oh, and you need pretty large RAM to load all channels together, including ones you don’t really need.
Did I answer your question?
Zverik 于 2016年07月17日 09:58 的评论
So basically anyone in the OSM community who wants to have the latest imagery from Sentinel 2 has to go through you? And if they are too shy to ask, they will be met with “you failed my social experiment” blame?
BushmanK 于 2016年07月17日 14:35 的评论
@Zverik,
I knew you are skillful in turning meaning of someone’s words upside down and reading things between the lines (which aren’t usually there), but your last comment is a masterpiece.
If anyone wants to have Sentinel 2 data, this person has huge amount of options. Including learning some very simple things. It is required to effectively make own composites. Composites, produced by any step-by-step instruction will often (not every time) be close to useless.
These required things are: basics of color correction (histogram manipulation, in particular), basics of QGIS or GDAL (understanding of about five simple operations), basics of graphic formats (concept of bits per pixel, resolutions, image channels, compression). All these things are covered by many different sources of information and could be learned in one day. Examples usually can’t serve as a proof, but back in nineties I learned how to do that having no idea about programming or GIS, just with certain skills of image editing in Photoshop and being able to use Windows command line.
Getting back to asking me to make those composites - the most important result of my experiment was not the fact that people don’t want to ask me to make composites, but the fact they (except a couple of them) don’t want to use ones I already made for them, covering two areas with largest density of mappers in Russia. But I’m still making those (for just a couple of people).
Are you serious about someone being super-shy, which prevents him from asking me? Because if yes, you just said, that people in OSM community have social development level of kindergarten kids. (Which is sometimes true, but in form of self-esteem, primitive demagogy and passive-aggressive behavior.) In that case, they probably have to work on their social skills (for their own good, to live normal life), not on remote sensing skills. In other words, I’m okay with giving food to one, who can hold a spoon, but I don’t want to spoon feed any lazy adult people like you want me to do. If someone is afraid of me because I don’t hesitate to call dumb people “idiots”, that’s not my business either, because I explicitly told it many times: there is nothing bad in lack of knowledge, but it’s really bad if someone doesn’t want to learn anything. “Learn” means “getting some understanding”, not “learning how to repeat steps like a monkey”.
It’s easy to see that I always been supporting meaningful discussions on topic of remote sensing data on Russian forum and answering questions “how it works”, while community was usually quite cold towards this topic. Don’t say that it was because no tutorials were available - there are tons of information and examples on topic of Landsat data, more or less useful, and a couple of people successfully demonstrated that it’s easy to learn how to use it.
SimonPoole 于 2016年07月18日 21:10 的评论
@imagico iD has been using the “best” imagery for an area as default since early this year, and has been recommending it in the layer selection since it exists AFAIK.
imagico 于 2016年07月20日 08:41 的评论
@SimonPoole - the “best” classification is not well suited since it is a binary onedimensional and whole source classification relative to other constantly changing sources. For a real improvement editors would need to detect where Bing/Mapbox have poor imagery (as said easy with Bing based on the date metadata) and suggest other imagery there.
@Zverik - in my experience the problem with processing satellite images is not the technical side. GDAL meanwhile has also support to directly read from Sentinel-2 packages so this is relatively easy. The problem is more the difficulties of the matter itself - choosing a well suited image and processing it for a consistent and well readable appearance. This requires experience which you can not substitute through clever tools and documented workflows. Still there are a lot of things that make Sentinel-2 more difficult than Landsat - i have written about those in my blog quite extensively.
Since i have started the OSMIM i have always said requests for images are welcome but there have been very few and i don’t think this is because mappers are shy but because of lack of awareness and lack of appreciation for relatively low resolution but very up-to-date imagery. Many mappers these days have never learned how to extract meaningful information from lower resolution imagery.
BushmanK 于 2016年07月20日 14:23 的评论
@imagico,
That’s basically what I’m trying to say (besides other things) regarding of Sentinel processing. Technically, it’s easy; anyone can learn it with experience; any decent result can’t be achieved by following simple “brainless” workflow. However, Zverik only wants to hear what he likes and prefers to blame “rude smart ass professionals” for not willing to write step-by-step instructions and keeping information secret. That’s hilarious, because where we, “professionals” (I’m just total amateur, by the way) have learned it from? Obviously, from fundamental sources (not tutorials) and own experience.
imagico 于 2016年07月20日 18:56 的评论
There is nothing wrong with looking for and asking for instructions but of course for a faceted topic like satellite image processing there is not one single way to do things and what works well for someone processing satellite data professionally rarely works equally well for beginners. So when i write specific instructions (like here) i tend to do this based on a workflow specifically designed for the task in question which is usually not what i use myself routinely. You can try to follow such instructions step by step to do very specific things or you can try to learn more generic matters from them but you cannot expect develop a fundamental understanding that enables you to do new things and make qualified decisions just by reading and following a specific tutorial.
And the accusation of “rude smart ass professionals” is not that far fetched with regards to Sentinel-2 - there is a strong tendency within ESA and related companies towards elitist, non-transparent work and a distinct culture of non-openness, the SNAP/Sentinel toolboxes system is just one example for this. The open data nature of the Sentinel program is very new to most of the people working in that environment.