I also forgot to mention:

For the endpoint to be marked unavailable, two consecutive checks must fail.

So single connection drops are unlikely to be registered.

@Andy Allan, Hey! I already thought of that, and there are additional connectivity checks to my other server in Poland. I exclude any downtime that is also present on that server. :-)

I also verify connectivity with non-OSM services (to prevent false positives)

By the way, do you by chance know anything about the official uptime OSM configuration? I have noticed it’s more optimistic, which can indicate a higher timeout limit (assuming 60 seconds?). I would love to see if it’s possible to reduce the timeout on official checks, as they seem to be not fully indicative of the average user experience (applications won’t usually wait for a response for a minute).

Btw, I tried to make mapping of bus relations a little bit more fun: https://github.com/Zaczero/osm-relatify 😉

Thank you for your insight. It’s really interesting to learn. Have a happy new year 🦀!

@mmd That’s interesting! Thank you for sharing. :-)

That’s a good suggestion! However, I don’t think solving this mystery is my priority or of significant importance. I just personally found it interesting :-)

@TrickyFoxy Thank You! I wasn’t aware of that. I quickly compared the downtime on 12-03 for “Website HTTPS”, with the official number being approximately 3 minutes and 30 seconds of downtime. However, during my observation, I noted about 27 minutes of unavailability. Perhaps, the official uptime checks are hitting some form of cache, or the downtime was more concentrated in the European region. The next month I will do a more in-depth comparison :-)

@Firefishy

First of all, thank you for your valuable feedback. That’s quite a lot of information to analyze. Let me address it and provide you with my general thoughts about the current situation.

Transparency

First of all, it’s truly awesome to see OWG taking steps towards making information transparent and publicly available. It’s important that we don’t forget the core values of OSM.

Cloud comparison

I am willing to create the comparison, with one caveat. The data that I prepare would require supervision prior to its final publication. Given that this task isn’t something I do regularly, there’s a significant chance of errors. However, I think it would be more advantageous to first address the topics I’ll be presenting shortly before diving into the comparison. In my view, preparing the comparison is of lower priority, especially considering the current free availability of AWS for us.

Now that we’ve addressed that matter, I can shift my attention to expressing my thoughts and making specific recommendations.

1. Backup deduplication

I want to strongly emphasize the importance of prioritizing data duplication. The sooner we implement this, the more streamlined our storage maintenance will be, facilitating smoother transfers—whether to other vendors or onto self-owned hardware. This proactive approach will undoubtedly enhance the overall efficiency and management of our data storage system.

2. openstreetmap-storage-backups - 112.4 TB - $120/month

Backups including some historical. Backups are not de-duped by design (heavy admin / risk burden). Some opportunity to manually cleanup, but very low priority. No automatic cleanup.

I cannot provide specific commentary on this just yet. It would be helpful to have a more detailed breakdown first. For instance, what types of backups are being discussed, and what are their sizes? Perhaps you could direct me to a resource that would allow me to deduce this information on my own.

3. openstreetmap-planet - 71.1 TB - $100/month

Historical and current copies of published planet files. Deep-Archive, for future restore to AWS hosted planet service with full back catalog. No automatic cleanups.

I have analyzed https://planet.openstreetmap.org/planet/, but I couldn’t comprehend the need for 70TB. The most recent full-history dump weighs 200GB. Let’s assume we’ll store 5 of those, which would sum up to around 1TB. Historical dumps can be downloaded via torrent as they are currently, or reconstituted from the latest versions when unavailable. If we incorporate non-full-history dumps into the calculation, we can conservatively estimate the total storage requirement to be around 2TB.

4. openstreetmap-tile-aggregated-logs - 32.1 TB - $125/month

Archival of processed tile CDN usage logs. Historical reference for Ops to work out tends and usage patterns. More data here than provided by public logs: Index of /tile_logs 4 @pnorman can clarify.

Shouldn’t we employ a log sampling method? If the core idea is to analyze usage patterns and detect abuse, even sampling as much as 1% would significantly reduce our storage requirements.

5. openstreetmap-wal - 28.7 TB - $400/month

Live streaming “Write Ahead Log” copies of the OpenStreetMap core Postgres database. The WAL files are used for syncing follower instances of the core Postgres database server. Vital asset to our data recovery plans. Can be used for recovery between full weekly database backup or corruption. For clarity this database is private and not published via planet data (eg: messages, users etc). Automatic cleanup after 1 year.

If we are already conducting weekly database backups, retaining WAL files for a duration of 1 year seems wasteful in my opinion. Automatic cleanup should take place within 1 month at most, preferably within 2 weeks. Allocating 29TB of storage solely for WAL files appears wasteful to me.

6. openstreetmap-imagery-backups - 18.2 TB - $35/month

Backups of imagery provided to OpenStreetMap. Deep archival. Primarily backups of imagery hosted on kessie 3. No automatic cleanups.

I believe deduplication would be a perfect match in this scenario. Once imagery is added, it rarely changes, so any incremental backups would be of negligible size. Employing complete backups every time is not a suitable approach for addressing this kind of problem.

7. openstreetmap-fastly-logs - 5.3 TB - $125/month

Inbound fastly CDN logs for processing. Key to us finding and managing abuse, source for publish tile log analysis: Index of /tile_logs 4 Automatic Cleanup after 31 days.

The same argument as for 4 applies here.

8. openstreetmap-gps-traces - 2.8 TB - $80 to $225/month

The GPS traces that are uploaded to OpenStreetMap.org, the storage backend for website: Public GPS Traces OpenStreetMap 3 Formerly provided by NFS service, moved to S3 to simply admin burden and to seamlessly work across our hosting data centres. No automatic cleanup, but opportunity to improve costs with S3 “tier” lifecycle rules.

I can observe that there are approximately 10,000,000 traces uploaded at the moment, averaging 0.3 MB per trace. When I download a trace from the website, I notice that it is uncompressed, which is consistent with the 0.3 MB estimate. Given that traces are essentially text files, applying basic compression can reduce their size by a factor of 20.

9. openstreetmap-fastly-processed-logs - 1.9 TB - $50/month

Archival of processed tile CDN view logs. Historical reference for Ops to work out tends and usage patterns. More data than provided by public logs: Index of /tile_logs 4 @pnorman can clarify.

The same argument as for 4 applies here

10. openstreetmap-user-avatars - 113.1 GB - $5/month

The user “avatar” images as uploaded by users. No automatic cleanup, but opportunity to improve costs with S3 “tier” lifecycle rules. Formerly provided by NFS service, moved to S3 to simply admin burden and to seamlessly work across our hosting data centres.

I don’t have any suggestions; everything appears to be fine.

11. openstreetmap-aws-cloudtrail - 76.0 GB - $2/month

Storage backend for AWS Cloudtrail API access logging service. Security monitoring. No automatic cleanup.

I don’t have any suggestions; everything appears to be fine.

12. openstreetmap-gps-images - 62.7 GB - $10/month

The processed display images used by OpenStreetMap.org on Public GPS Traces OpenStreetMap 4 Formerly provided by NFS service, moved to S3 to simply admin burden and to seamlessly work across our hosting data centres.

This also appears to be acceptable, but the cost seems somewhat excessive. I’m not sure about the reason for this.

13. openstreetmap-backups - 21.1 GB $0.03/month

Historical database backups from OSM in first few years. No automatic cleanup.

This is not worth discussing. I can only suggest deleting this data completely, as it appears to be entirely redundant and only complicates maintenance.

Summary

Taking into account the suggestions outlined in points 3, 4, 5, 7, 8, and 9, it becomes apparent that these proposed enhancements are relatively straightforward to integrate and would significantly enhance the efficiency of OSM operations. While I wholeheartedly endorse the concept of data deduplication, I will exclude it from my summary calculation for the sake of simplicity.

Based on my calculations, the current monthly expenditure of $100 + $125 + $400 + $125 + $225 + $50 equals $1025. However, by implementing the suggested changes, the projected monthly cost would reduce substantially to $3 + $1.25 + $16 + $1.25 + $10 + $0.5, amounting to $32. This would signify an impressive ~30x reduction in costs. I assume a linear scaling of costs here, which is not entirely accurate, but it’s the best I have been able to come up with given the limited information.

I have observed an emerging trend in published information where, rather than optimizing things, we are increasingly opting to pour more and more money into them (theoretically; it’s free now but I do not see it as an excuse). While I comprehend that volunteer work is often limited, I must highlight that the suggestions I have outlined are neither difficult nor time-consuming to implement.

I trust that you will view this feedback as constructive. Kindly take a moment to reflect upon it, and respond at your convenience.

-K

@spatialia

I appreciate the clarification, and you’re absolutely right. Upon revisiting, I realized I confused the standard S3 Glacier rates with the S3 Deep Archive costs. The distinction, when properly examined, does align with the numbers mentioned for deep archive storage, which is roughly $1 per TB.

However, it’d be worthwhile to have a comprehensive comparison between various cloud services, especially considering other potential costs. For example, expenses like “openstreetmap-wal - 28.7 TB - $400/month” still seem quite substantial.

Thanks for pointing it out and helping me see the bigger picture.

@spatialia

That’s indeed a good observation. However, I did my due diligence by checking the S3 Glacier pricing at https://aws.amazon.com/s3/glacier/pricing/, and based on my findings, the cheapest rate I identified was approximately $3.5 per TB. This calculation doesn’t align with the numbers mentioned, so additional insights that could clarify this discrepancy, would be welcome.

@MxxCon

I’m sorry to say, but the idea of Amazon being evil is not a personal belief; it’s a fact. Please conduct your own research, as my discussing it now is not feasible and would demand significant effort from my end. All I ask is for individuals making key commentary to provide justifications for their statements.

Here are some videos you may want to consider watching:

• https://www.youtube.com/watch?v=Ag1AKIl_2GM - this explains well what good, human-centric software should be like, gives Amazon as an evil example (4:41)
• https://www.youtube.com/watch?v=61KcJPFCcWU - good example about Amazon not carrying about its customers
• https://www.youtube.com/watch?v=pBffKzWECUQ - about Amazon being a monopoly in general
• https://www.youtube.com/watch?v=jMuRavcHCDs - about Amazon spying on customers via Ring cameras
• https://www.youtube.com/watch?v=SyEgD-5GK9c - about Amazon’s unfounded cancellation of one man’s services

@apm-wa

I understand how such an offering would be hard to refuse given the cost-saving implications for the OSM project. However, my concern is rooted in the timeline of events. From the data I gathered, OSM’s reliance on AWS dates back to at least 2022, whereas the free AWS credits began just ~6 months ago.

Furthermore, I’d like to express concerns about placing substantial reliance on a corporation like Amazon. Even if services are currently free, Amazon has a track record of making sudden and significant changes to its policies. To my knowledge, there’s no assurance that Amazon’s sponsorship will be perpetual, and transitioning away later could come with considerable costs and complexities. It’s free until the day it isn’t.

@Friendly_Ghost

I see your viewpoint; allow me to clarify. Here’s a full sentence quote:

I’m not willing to continue the discussion in such an environment, where there’s an overemphasis on subjective views and a lack of grounded argumentation.

This sentence highlights that numerous individuals (not exclusively Firefishy) fail to present well-founded arguments alongside their comments. Continuing a conversation within such an environment proves challenging for me. I am open to engaging with individuals who offer substantiated facts rather than solely relying on their personal beliefs.

And about thanking to Firefishy, I would appreciate the opportunity to do so personally. However, I am still awaiting a response from him.

@apm-wa

Thank you for the comprehensive overview and directing me to those resources. It’s clear from the 2021 OSM community survey and the Strategic Plan Outline that there was a strong consensus within the community to prioritize the stability of core infrastructure. This sheds more light on the overall situation, and I appreciate the effort to provide this context.

The strategic decision to prioritize infrastructure stability by opting for cloud services is understood and commendable, especially considering the rapid growth OSM has experienced. I can see that OSM’s choices were driven by its commitment to the project’s core values and its community.

However, one aspect still nudges at me: the choice of Amazon as the cloud service provider. With the wide selection of cloud service providers available, each with its own pricing models and philosophical underpinnings, why was Amazon — a corporation known for its controversial business practices — the chosen one? There are other providers like Backblaze and Hetzner, which, in my research, offer competitive, if not more affordable pricing, and do not have a reputation of commodifying its users to the extent Amazon does.

While the community’s overarching goals are clear, I think it’s crucial for the OSMF to consider not just the technical and financial aspects but also the ethical dimensions when partnering with third-party entities. It’s noteworthy to mention that Amazon’s reputation, especially concerning privacy and human rights, has reached an almost “meme” status in certain circles. Aligning with such an entity could potentially raise eyebrows, and it’s essential to be conscious of the broader implications of such associations.

@Friendly_Ghost

Firefishy … only to hear you complaining about “a lack of grounded argumentation” later on without even a “thank you”

Sorry, but what? This sounds like a false accusation, so please provide some evidence that I did indeed point out to Firefishy the lack of grounded argumentation in his detailed budget report. I believe that my only response was that I will not engage in further discussion for reasons mentioned.

I appreciate the references you’ve provided. Focusing on issue #682, which you’ve specifically mentioned: “Getting credits from AWS” doesn’t explicitly confirm a free sponsorship from AWS. The information I’ve gathered originally, already indicates financial investment of OSMF into Amazon, so the point about “getting credits” brings no new information.

It’s important to differentiate between accessibility and public resource. Yes, anyone can create an account on Slack, but that doesn’t classify it as an open public resource. Just as signing up for a newsletter doesn’t make the contents of that newsletter universally accessible.

Transparency in a project like OSM is of most importance. I’ve voiced concerns about the project’s direction and transparency, not to attack OSM but to emphasize its importance.

Lastly, I’d appreciate it if you could ensure your arguments are well-considered before presenting them. Continually addressing ambiguities is becoming quite time-consuming for me.

@MxxCon

This will likely be my final comment on this matter directed to you. Firstly, the Slack link you’ve shared isn’t openly accessible without an account; it can’t be classified as a public resource. As for the GitHub issue, it doesn’t reference any ongoing AWS sponsorship. It seems there’s a disconnect in our conversation: my concerns center on OSM’s reliance on Amazon at all, not on refusing any sponsorship (or any other deals) from AWS.

@MxxCon

Thank you for taking the time to share your perspective with me.

Amazon’s sponsorship

Can you point me to where the details about the ongoing Amazon’s sponsorship were made public? I based my findings on the OWG official website. Most people would consider that a reliable primary source. Saying it was discussed “possibly elsewhere” seems a bit vague. All the information I gathered suggests that the sponsorship is still in the planning stages.

Amazon’s contributions

I acknowledge and respect Amazon’s efforts to support the OSM. Their commitment, in terms of resources, to this open-source project is evident. My intent wasn’t to overlook their contributions, but rather, to express my concerns about OSM’s philosophical direction and values.

GitHub & Microsoft

I’m aware of GitHub’s ties to Microsoft and the ethical considerations that surround this. However, as it stands today, there’s no alternative platform with such a vibrant and active community as GitHub. This is a sentiment that can be mirrored with platforms like YouTube & Google. While I use these services, I consciously limit my exposure by adopting alternative frontends, and ensuring that any information I share is intended for public viewing. It’s a balancing act of leveraging the benefits of mainstream platforms while upholding my privacy values. On that note, I’d like to highlight that all my private projects reside on the Gitea platform.

Spending vs Budgeting

At the time of my initial comments, I was working off the information available to me. As discussions evolved, new details came to light. It’s essential to remember the context of initial inquiries and the state of knowledge at that moment. I have since acknowledged the newly released information, and any assumption to the contrary would be inaccurate.

@Firefishy

First and foremost, thank you for your thoughtful response and willingness to open up a dialogue about OSMF’s infrastructure choices. I genuinely appreciate it.

I want to clarify something upfront: When I first brought up this concern, there was no public knowledge about OSMF’s free AWS credits. The information available at that time suggested that we had not only budgeted for, but were also spending, a significant amount on S3. Now, while I wholly understand OSMF’s monetary constraints, and I respect the pragmatic choices that are often required in such situations, there are core principles I stand by that deeply inform my perspective.

One of my core beliefs is the importance of creating and utilizing software that respects its users. I have a profound aversion to the practices of many big-tech companies. In my eyes, they commodify human beings and often operate in ways I find ethically objectionable. OSM has always been a sanctuary for me in this regard – an embodiment of what the tech world can be when driven by pure intent and community spirit. To discover the organization’s ties to Amazon deeply affected my perception of OSM’s independence.

I’m not suggesting that OSM should immediately invest in its own hardware, though it would be ideal. My primary concern revolves around our reliance on Amazon. I believe OSM should be cautious of anchoring itself too deeply to a company whose ethos might not resonate with ours. While AWS, with its current credits, might seem cost-effective now, there are numerous other solutions available that may better embody OSM’s guiding principles. -Did you hear about the free TVs? Don’t dance with the devil.

Regarding backup deduplication: While I acknowledge the complexities of setting it up properly, I don’t see it as prohibitively difficult. When we’re discussing potential savings of tens of thousands of Euros – especially considering how limited some OSM groups’ budgets are – it seems well worth the effort. Plus, having a deduplicated backup would significantly ease any future transitions, potentially even to self-owned hardware, given the reduced storage demands.

Your dedication and professionalism are truly commendable. I would be grateful for an opportunity to converse further on this topic. Whether it’s a call or another method, I believe dialogue can pave the way to mutual understanding. After all, our shared objective is the betterment of OSM.

-K

@RobJN, and that’s a sign of OSM changing. I have always assumed that OSM does not want to release half-baked features that harm the user experience. The alternative you mentioned is not the only approach. As you can see now, the issue has been resolved; it only required a little more time to be refined.