Your Europe & Asia maps are broken. This is the error message for one of them (HTTP/1.1 404 Not Found, reported by https://upload.wikimedia.org/wikipedia/en/3/31/1916-Asia-political-map.jpg):

File not found: /v1/AUTH_mw/wikipedia-en-local-public.31/3/31/1916-Asia-political-map.jpg

Is it asking for authorisation? It is UPLOAD.wikimedia, after all.

Updating the map with the shop is perfect and you seem to have done a good job there. No problem with you adding email address + website URL into the shop address, either. However, putting the shop URL into this Diary is SPAM and you may well get banned as well as your diary entry removed.

I suggest that you edit your diary to remove the spammy link asap (swap it for the shop name).

Hi giggls.
What were the specifics that caused you such a big problem?

Here is the first result on osm:

osm.wiki/GPX

Look for yourself & you are bound to find better references than that.

I’m impressed. Well written, coherent and, in my view, exactly what these diaries are designed for. The argument you set out makes good sense to me.

@Steube
Thank you. It is a relief to find a balanced view within these violent eddies & storms of disagreement. If only I could manage that myself all the time.

Hi @Firefishy

The sysadmin team are successfully mitigating spam via non user-privacy invading methods.

You shot the patient (Diaries) in order to kill the disease. I agree that that does not invade the privacy of those that now cannot find anything within any Diary. Well done, and a Silver Star for creative writing.

Another good method to save bandwidth is to pull all power cords from all servers. Please send my usual commission of $1.2m / day should you decide to make use of this energy-efficient optimal solution.

We receive 1,000s of requests an hour from Googlebot and other search bots. We do not wildly deny content from being index.

This is the Sat, 22 Jun 2019 23:38:58 GMT (and current) update of the OSM robots.txt:

$ cat robots.txt
User-agent: *
Disallow: /user/*/diary
Disallow: /user/*/traces/
Allow: /user/
Disallow: /traces/tag/
Disallow: /traces/page/
Disallow: /api/
Disallow: /edit
Disallow: /browse
Disallow: /diary
Disallow: /login
Disallow: /geocoder
Disallow: /history
Disallow: /message
Disallow: /trace/
Disallow: /*lat=
Disallow: /*node=
Disallow: /*way=
Disallow: /*relation=

Host: www.openstreetmap.org

I count one Allow (redundant, since SEs assume Allow: *) & seventeen Disallow. If that is NOT wildly denying content indexing I tremble at what you may cook up in the future.

It is my understand that Alex is involved with the running of stopforumspam.com and he is very aggressively promoting the service with us.

For the record, I’m one of 2 moderators at StopForumSpam (SFS). The only other staff is pedigree (admin) and none of us are paid. SFS is a community supported, 14 year (I think) operation that acts as saviour to millions of forum/blog websites.

I first got involved in 2009-12-02 at a time when I was blocking all registrations from *@gmail.com in my utter desperation to stop the torrents of spam coming in with accounts using that email address. SFS saved my sanity at a time when I was heading for a physical & emotional breakdown after weeks & months of 24/7 desperation to keep my business, built with the last of my money, online. When I say that it works I know whereof I speak.

I believe that I may have directly promoted SFS perhaps 5 times. There are 160 mentions of ‘spam’ within my current diary listing (including today) and 6 mentions of ‘stopforumspam’. If that is “aggressive promotion” then so be it. I spend large amounts of my time trying to promote OSM and that is because I think it the Bees Knees; similar with SFS.

Alex contacting our hosting company to complain about piwiki is not a cool move on his part.

Explain your evidence for “complain about piwiki” to UCL.

I went through channels to report to OSM that (what appeared to be) 2 x static piwik.osm.org files were missing the response headers that could otherwise reduce the site bandwidth. The response was “nothing to do with us, guv”. So, I traced the operator of that IP & went through channels to ask about a strange file that I had discovered on my computer coming from his location. He agreed that it was from one of their hosted servers and said that he would email me when he had sourced it. Later that day that is exactly what he did.

The man at UCL NoC was a professional, and that is how he acted. Helpful, human, and professional. All I got from OSM were amateurs with overblown emotions. Brats that put the boot in on the first opportunity. If you got any blowback from UCL then stop telling lies & stop sending webbugs to every client. Then all those problems will evaporate.

@Mateusz:
You obviously never read all the section, else you would have realised my intention:

I won’t belong to a club that accepts me as a member
- Oscar Wilde

I have abandoned all hope of receiving Justice at your hands. It is a foreigner at your table.

@Firefishy:
Well I have read & replied to your email.

For the benefit of others, this hyperbolic rapscallion writes:

If you want to remain involved in the project, cut out the abuse and stick to mapping.

Any further unacceptable language or actions from you and I will seek that you are politely asked to leave our project for good.

Politely? Love it.

Apparently, saying that these diaries are being deluged with spam is “unacceptable language”, and be damned with reality. You have not addressed at any point whether my statements are truthful, nor engaged in any discussion with me to understand my motives.

I’m a 20-year professional in these matters & have given a practical, simple step-wise method to deal with the situation which has been proven effective by others, but you have not discussed any of that. In fact, you have not discussed anything at all.

It seems that if it happens inside your head then it is, Ipso Facto, reality without any external checks required. “Always happy to help.” You are not helping.

PS Firefishy:
I notice that you did not answer a single question from my earlier reply. Seems I have touched a nerve.

If there was a simple way to deal with the low level spam issue, while ensuring the extremely high bar of user privacy we maintain, we’d have done it already.

Sorry Firefishy, but that is simply not true.

I respect TomH’s hard-working credentials & devotion & ability & likewise with everyone involved. The setup is a work of art & does well. But it is not fit for purpose, as the last month has shown.

Either you will change to meet tomorrow or you will go under.

People are attempting to help and you are so self-absorbed that you cannot perceive it. Well done, son.

@Firefishy:

I state in my post:

That means that technically it is not spam!

So, since it is not spam, please explain why it was removed.

If I was admin/mod here I would have removed it, no question. But since you are upset that I called it ‘spam’, please explain what you would want me to call it. I shall then edit the post to match what you want me to call it rather than ‘spam’.

Please explain the way in which I am “perpetually insulting our voluntee admin team”.

• Is it by explaining that the site is being spammed/abused? You may think that an insult to the spammers/abusers but it can hardly be an insult to the Admin.

• Is it by saying our admin “appear to be on holiday” because 24 hours later the spam/abuse has not been removed? In that case, either stop taking holidays or alter arrangements so that removing spam does not rely on just one person.

• Is it because I pointed out that this recent spam began because “TomH unilaterally removed his former unilateral block on these Diary pages”? In that case, please ask him to stop taking unilateral actions.

• Is it because I am shining a light upon some very unfortunate actions/omissions/attitudes? Well, in that case stop acting like that/put in the methods that will work/change your attitude.

• Is because abusing me because you feel uncomfortable is far easier than considering my knowledgeable suggestions for both making your admin team life effective and reducing visible spam & abuse to nothing?

@michalfabik:
Something I’ve recalled about CAPTCHAs just as I was about to go and get some food:

One of the ways that the spam-masters broke CAPTCHAs in the early days before XEvil came on the scene was to employ humans at pennies / CAPTCHA. A routine was placed into XRumer so that when it met a CAPTCHA it would post it to a human who would solve it & post the answer back. The whole thing done automatically within a minute or so.

So what is the point? Put in routines to stop bots and you have zero need for a CAPTCHA. And that is the point where SFS becomes the saviour, for it has an API & thus can give an automated routine to stop known bots. The attitude is that you cannot stop all bots, so reduce the number to a level that humans can handle via moderation & report to SFS (thus then auto-blocked) and the site is kept clean.

@freebeer:

so i can only guess when the flood starts and ends.

10,935 posts x 1 post-per-8-secs = 87,480 secs / 3,600 = 24.3 hours.

Best guess: it started with 1 person, then another one (or more) joined in.

@michalfabik:
Re: CAPTCHAs: My personal experience dates back to the beginning of CAPTCHAs & they were then an utter pain to use & unreliable at their designated task of keeping spambots out. After a few months I got my site to the point that I could keep the bots out without using a CAPTCHA, so at that point I dropped them entirely as not to annoy my users unnecessarily. You have to accept that you will always get human spammers, and those were manually sent to SFS. SFS was also used early in the sequence to keep the spam out but I think that I am rubbing some folks up the wrong way by continually mentioning it, so kept it out of the diary post.

I agree that CAPTCHAs have improved to the point where they are only a modest pain rather than an utter pain, but you may notice that the word ‘pain’ is still in there. Even so, SFS uses a CAPTCHA to contain the worst of spambots, but that was not my call.

Look back to my XRumer post and search for “XEvil” & you will see that the spammers have been routinely auto-breaking CAPTCHAs since 2017.

Finally, look at my Trillions post and follow the link to Neil at the BBC and you will see that he is sat at his desk behind a “min. 1Gb/s link”. That is normal for folks at a modern NoC with a direct link to the Internet backbone and my experience near the end of my time as a Webmaster/Network engineer was that some of them will use an un-throttled bot across that link to try to download your entire site (>1,000 accesses/second - fricking unbelievable).

@TomH
I see that you have, indeed, re-opened these Diaries for search by the Search Engines (SEs). Excellent, thank you. Sunlight at last.

That action clearly needs a second action to prevent a re-occurrence of the spam that drove you (and everyone else) crazy before. What changes have you made to stop the spam?

Oh come on, man. What a foolish response. Who on earth wants a ‘grovelling apology’, certainly not me.

If you really have done that & not taken other changes to stop the spam then you really would be incompetent.

I’ve said it before & I’m happy to say it again (I just wish that you would accept it): I see you as trustworthy & competent, hard-working & reliable. You are also a complete prat in certain areas, but then arn’t we all?

@ndrw6:

I’d be OK with making (diaries) accessible to logged in users only.

Why on earth would you want that?

Going back 20 years most forums allowed anonymous posts. Then spam began to bloom. In the current era not a single internet forum allows anonymous posts. However, almost every forum allows anonymous viewing & searching for most of their site, which is completely different.

Modern forum admin have had to learn how to deal with spam, or go out of business. What has become obvious is that none of the HSM admin have any experience of dealing with spam. Now, ignorance is not a fault, but dumb-arse reactions to spam definitely are.

The HSM equivalent to anonymous posts is allowing non-editors to post. For some reason, some influential members have historically taken against enforcing map-edits on posters. That has led directly to our current spam problems. It is now time to decide - do you want an OpenStreetMap or a HiddenStreetMap? And remember, most of HSM is already hidden from common view. Do you want that to continue?

To try to be clear, these are the proposals:

1. In general, allow anonymous GET visits
2. Restrict POST privilege to users that have made xx map edits

There would in addition be other suggestions related to SE access & bots generally + technical aspects on HTTP/1.0 Content Negotiation, which is missing, but I want to consult on that first before I put any suggestions forward.

Hi ndrw6

I don’t mind having diary pages delisted from search engines if that helps reduce spam.

You write as if the two are necessarily connected. If that were true then the Internet world would be entirely composed of gulags that cannot connect. Now I accept that the totalitarian remnants of this world (China, Russia, North Korea, etc.) are like that, but that is not how the Internet was designed, nor how the rest of the world carries on, nor what is necessary to prevent spam, nor what I want for this world’s future.

It doesn’t make the project any less open

If you truly believe that then you are in a state of denial. Seriously, if you think that the best way of fixing problems is by killing the patient then you had better drink your final Kool-Aid.

it matters that the community communication tools are kept functional (this includes them not being overwhelmed by spam)

Yup, completely agree with that one, ndrw6.

I’ve been fighting spam for 20+ years and know for a certainty that HSM has been deploying the wrong methods. In fact, it has been deploying zero methods as best as I can tell. Any organisation that says “Let’s shut it all down as to stop the spam” has lost their tiny minds and actually hates what they are doing. No mother would ask a doctor to kill her child as to fix a disfiguring disease unless she hated her child, or had gone insane, or both.

OSM data is searchable, just not using the common web search engines

Yeah, I got that, Warin61. The point is that they can NOT be searched from outside of HSM, only from inside. That fact means that they are hidden for the greatest part of the population.

I like being part of an open-source movement, and do not actually want to be part of some Hidden Society.