When the OSMF board started their survey i posted here some critical commentary on the design of the survey. Now that the results have been published (and the board should be commended for doing so in a fairly comprehensive fashion) there has been analysis and interpretations presented from various sides. I also already made a few comments as part of my report from the advisory board and although i don’t want to do any more elaborate numbers processing i wanted to share my overall impressions of the survey taking into account the responses as well. As before with my comments on the questions and also in contrast to most of the other analysis that has been presented so far this is focusing less on technical analysis and more of overall qualitative considerations and methodological critique.

Retrospective questions (F1-F5)

The first five questions of the survey were asking for retrospective approval or disapproval on some specific decisions of the OSMF board. I already commented on the issues of these questions in detail while the survey was running. My primary impression from the results on these is that the approval in the resposes is actually weaker than i anticipated considering the leading character and selection of the questions. In other words: Despite the questions specifically not being of a nature that is likely to evoke strong negative reactions the answers show a remarkably broad spectrum of views.

What should be clear though - and i hinted at that in my comments on the design of the questions already - is that the reduction of complex multi-dimensional matters to a simple one-dimensional approval/disapproval scale inevitably aggregates very different attitudes to identical answers. In question F1 for example (the diversity topic) the strong disapproval voices probably contain both those who consider the whole diversity topic to be esotheric touchy-feely snowflake mumbo-jumbo and want the board to ignore the whole matter as well as those who think the board is doing way too little with creating committees and should simply ban all white European males from OSM for life (yes, i am exaggerating here - but most will get my point). Equally the ‘neutral answers’ will consist of those who have no opinion on the matter and those who have a strong opinion which however is fully orthogonal to what the board asked specifically. And finally the approvals most certainly consist of people who think tasking some committee is a good way to deal with the problem without changing anything of substance as well as those who project their desire of what meaningful measures such committee should propose in their opinion into their answer to the question.

So as i said a long time ago - if the purpose of these questions was for the board to pat themselves on the back and reassure themselves they did good that would be a very wasteful use of the instrument of a survey. If the goal is to critically evaluate past decisions with the aim to potentially adjust or revise them and to make better decisions in the future those questions and their answers are decidedly non-helpful.

Future oriented questions (S1-S3)

The other three topical questions of the survey were targeted at future decisions of the OSMF and the results show a noticably stronger differentiation across demographics than with F1-F5, which is noteworthy because the board concentrated on F1-F5 when they tried to argue for the lack of bias and for the poll allegedly being representative for the OSM community.

Like with the retrospective questions i already presented my critique of the question design while the survey was running. The analysis of the results the board has now presented allows me to illustrate that critique more clearly. The first question was asking respondents to prioritize a number of tasks (or more accurately: topics, because they are too unspecific to be actually called tasks) and the board in their analysis averaged these prioritizations. Such determination of an average of ratings of a set of items is inherently based on the assumption that the set is complete or in other words in this case: That these are the only significant topics to consider for the OSMF board and that there are no other topics of importance. To illustrate why that is the case here a short abstract example:

Imagine you have a set of five topics/tasks given to be rated (A,B,C,D,E) and another three topics/tasks (X,Y,Z) considered by the respondents to be worthy of inclusion in the set. And we have seven respondents in the survey who would want to rate the topics like this:

AXYZBCDE
AXYZBCDE
AXYZBCDE
XYZBACDE
XYZBACDE
XYZBACDE
XYZBACDE

Now if you calculate the average using the method the board used you get to

A: (3*8 + 4*4)/7 = 5.714
X: (3*7 + 4*8)/7 = 7.571
Y: (3*6 + 4*7)/7 = 6.571
Z: (3*5 + 4*6)/7 = 5.571
B: (3*4 + 4*5)/7 = 4.571
C: (3*3 + 4*3)/7 = 3.0
D: (3*2 + 4*2)/7 = 2.0
E: (3*1 + 4*1)/7 = 1.0

so topic ‘A’ is rated higher on average than topic ‘B’. However if you now would not include topics X,Y,Z in the survey you would get the responses

ABCDE
ABCDE
ABCDE
BACDE
BACDE
BACDE
BACDE

A: (3*5 + 4*4)/7 = 4.429
B: (3*4 + 4*5)/7 = 4.571
C: (3*3 + 4*3)/7 = 3.0
D: (3*2 + 4*2)/7 = 2.0
E: (3*1 + 4*1)/7 = 1.0

so topic ‘B’ in this case would appear to be rated higher than topic ‘A’. Apart from that of course also here the vagueness of the options is an issue. For example with the topic “Attribution guidelines” this is quite evident. So far the OSMF has indicated the only kind of guidelines they are considering is something very lenient and corporate friendly. Most knowledgable respondents will have likely assumed that giving this topic priority would mean giving a corporate friendly attribution guidelines priority. Hence the repondents giving this topic low priority probably consist of those who think such guidelines are not necessary or not desirable as well as those who would like a meaningful guideline reflecting the views of the OSM community on attribution but who do not believe the OSMF would create such guidelines so they prefer the OSMF not to pursue this topic at all to pursuing it primarily in the interests of corporate OSM data users.

The main technical issue with the other questions was as mentioned the lack of a ‘none of the above’ option so when you look at the responses it is important to consider what answers will likely have been chosen in lieu of a ‘none of the above’ option.

In case of the AI mapping question that choices would probably have been the ‘continue to take no position’ option (39.6 percent) - except for those who explicitly would want the board to get active but in a way that is neither approving not banning these methods who would have possibly chosen the ‘no opinion’ option (9.7 percent) despite them having an opinion on the matter.

In retrospect you can say the question could not have been designed better to accomodate the interests of the large corporate interests in the OSMF. The main interest of those is to prevent meaningful regulation of such activities. And the logical option ‘adopt a policy to regulate use of ai/ml tools to ensure the primacy of local knowledge and the local community’ is explicitly missing. It is almost certain that such an option would have gained more support than either of the radical options and potentially even more than the moderate options offered. So independent of the question if this selective set of possible answers was a conscious choice of the board to engineer a certain outcome or not (i would assume not - Hanlon’s razor) this is a great example how you can design a survey in a way that ensures a certain overall outcome (no support for meaningful regulation).

And in that regard the third of these three question is actually the most interesting one because despite all the deficits i have mentioned about it, it is asked in a fairly open form. And it is also the question yielding the most surprising results. I would have expected there to be more support for the ‘paying developers’ options, especially on a topic that is of practical interest for many. But i have probably underestimated how well many in the OSM community understand the dynamics of paid development and that this is not so likely to develop in a direction beneficial for the larger community and how many, especially also newcomers, value the volunteer work in the community.

What i wonder is if that sentiment is specific to map development or if a similar view is prevalent regarding paid development in general. One really interesting question for the survey would have been: Do you overall prefer OSM related software being developed exclusively by volunteers (like in case of the JOSM editor) or do you prefer development to be led by paid developers (like in case of the iD editor).

As i have said before the prudent thing to do in this field (new map rendering technologies) would be to look into strategic investment and cooperation with free software development projects (for example through organizations like OSGeo) to support development of tools suitable for community map design projects with a long term future perspective. But i am not sure if the OSMF at the moment has the capacity to act strategically on that time scale. So the choice not to rush doing something concrete at the moment, that seems to be the current approach of the board on this, is not a bad initial choice despite this specific option only being chosen by 8.7 percent of the respondents.

Free form answers

Unfortunately so far the board has only published English language translations mixed with original English language comments without indicating which of the comments are just approximate representations of a comment and which are actually genuine articulations. This makes more detailed analysis impossible unfortunately (you would analyse the design of the translation tool just as much as the articulation of the respondents). What the free form comments however allow is getting an anecdotal impression of the mindsets with which people have participated in the survey - which was helpful in better understanding the answers and this way contributed to my analysis above.

Conclusion

To conclude i want to go back to the starting question of the survey and the matter of diversity.

There is an irony in that on multiple levels: The key to substantially supporting diversity in a society is to protect minorities and provide them the space they need to flourish, even against the articulated interests of a majority. Yet the subject of diversity is covered in the survey in form of a single question asking for retroactive one-dimensional approval or disapproval on a decision of the board that has been made without broader input from the OSM community in all its diversity, that has no immediate effect on the matter of diversity at all and so far has also not led to any decision of substance in that field either. And in interpreting the answer to that question the board is looking exclusively at the average, in other words: The majority opinion. It is at the very core of the idea of doing a statistical survey to look primarily at the statistical average and not at the outliers.

But the irony goes a step further: The justification for supporting minorities even against the articulated interests of a majority is evidently an ethical argument. And ethics is obviously a domain rooted in and depending on reason - the same reason the board seems to reject when countering reason based critique of their actions not with arguments but by presenting survey results that seems to indicate majority support for their actions.

The bottom line is genuine bottom-up diversity in the OSM community is fueled by the openness, tolerance, generosity and reason of the individual OSM community members giving each other the space and freedom to express themselves according to their individual preferences and values - no matter now much away they are from the statistical average determined by some survey. If a survey helps people to realize this, that is great but the OSM community cannot depend on survey results telling wannabe community managers how exactly they need to engineer the community to achieve the kind of pseudo-diversity on paper someone somewhere has thought up to be desirable.