Let's talk about new data quality features

[emi420]

Hi everyone, as part of the HOT efforts to improve the quality of the mapping, we're working on new tools for validation.

New feature: validation before uploading changes to the map

Starting on Q4 2022, we started to have meetings and to write down ideas for an end to end solution. The Data Team wrote the documents that are the stepping stone for definitions about data quality. At the Tech Team we wrote the technical documents and developed product ideas.

One of the ideas that caught the attention of many people is measuring data quality just before uploading the changes to the map. As we can process a set of changes, we can show warnings to the mappers about the Semantic and Positional Accuracy aspects.

This new proposed feature will analyze the changes when the user is on the "Upload" screen of the iD Editor, enabling automated data quality validation:

You can check the technical detail of this proposal on the PDF below:

https://drive.google.com/file/d/1FvSmjyDNFzXI53cVTnZB1F-SKP5siljC/view?usp=sharing

Validation

There are two different validation results that we'll show on the first alpha version that will be available for some users to test.

Bad geometry buildings

4-wall buildings that have not a squared geometry.

Wrong tagging

Data model validation using these YAML configuration files

https://github.com/hotosm/underpass/tree/master/validate

For example, if you see “building=no” is wrong tagging because there’s a file building.yaml and “building=no” is not listed on that file

Discussion: wording

This feature is in-progress, we're still developing all the software parts to enable it and we'll have several stages for testing it before making it available for all users into the Tasking Manager.

In the meanwhile, we can have discussions about it to make sure it fits the needs of the majority of Tasking Manager users, including mappers, project managers and validators.

What we want to discuss first is the wording of the messages that will be displayed to the user, for example "Building is not a square" or "Bad value for roof:material" , as some mappers can be discouraged to map or put any value for a tag just to get ride of these messages.

• What do you think?
• What are your concerns about a feature like this?
• Do you have any ideas to improve it?
• Do you think it's a bad idea or that it can be implemented different?
• What's the wording that could be used to help users to improve their mapping?
• Could be good to show this feature to beginners only? or to all users? or different wording depending on user's experience?

Let's start discussing it and make the map a better map!

Thanks :)

[nicolelaine]

I am very excited about this idea! Some thoughts on wording:

"Name is required for a place" might lead to novice mappers adding random names to features to be able to upload their data. We often see novice mappers add names to features (i.e they will add their personal name or their company name to a building.) If they feel like they must add a name to move forward, they might add something completely arbitrary.

"Building is not a square" might also lead novice mappers to think that their building MUST be a square regardless of what the imagery looks like.

[emi420]

Yes, we must definitely be careful to not dis-motivate users to map, as the goal is to motivate and help mappers to map better.

Thanks @nicolelaine !

[petya-kangalova]

Some questions and discussion points from the Tasking Manager Meet Up:

1. Would this result override the idEditor? Or will be additional?
2. Would resolving these issues block posting? People had concerns this will be deter novice mappers from mapping..
3. Suggestion to link to Wiki for each data quality issue to explain to the user what it means.

[emi420]

Hi @petya-kangalova !

1. This is an additional layer of information to be displayed over the editor, it would not replace the iD Editor
2. This tool only analyze the changes to be uploaded and show information about Data Quality issues. It would not block posting, users will be able to upload changes anyway, except there's a decision to block posting in certain cases (ex: users flagged for having a lot of data quality issues)
3. Sounds good!

[RAytoun]

There should NEVER be an automated block because the machine is unable to identify false errors from real errors. If needed a mapper with a lot of data issues should be referred or flagged to someone for help (for example = validator)

[SColchester]

Thanks for sharing, pdf feedback:

Wording

'Building is not a square' should be reworded. Buildings are rarely 'square' they are more often 'rectangular'. The wording may confuse mappers into thinking they have to trace perfect 'squares' with equal length sides. I would recommend 'Building has un-squared corners' (which would align with the iD 'squaring' tool) or 'Building does not have right-angled corners'.

Overlapping and duplicate buildings - how are these different? I'd argue in the visual example both are 'overlapping' buildings. Is there any need for a naming difference? The only instance where 'duplicate' term might be more relevant is if there is a building which shares the exact same footprint as another building.

Implementation

Can we define how this new check will be enabled and at which level? I'd argue that this should be enabled/disabled at the Tasking Manager project level (with it set to default on). Most projects will likely benefit from these checks, but for some more complex projects (e.g. a place naming project) it will be unsuitable.

The checks

The two semantic checks are not suitable for beginners or for remote mapping, I'd leave them disabled for now. I'd argue that places (e.g. settlement names) and roof material types are only added by experienced mappers, ideally on the ground.

An impactful semantic check should focus on checking all of the fields for a building and flagging an issue if there is anything beyond 'building=yes'. This will catch a common error whereby new mappers fill building fields (e.g. name, or address) with meaningless text - typically tracking hashtags.

Overlapping/duplicate buildings - there is already a check for this in iD, it is prompted as soon as the user traces the objects --> 'building crosses building':

Will this be disabled and superseded? It is more powerful than your check because it recognizes overlap with all features immediately. But, the problem with this existing 'issue' check is that it prompts the users to change the building layer, which is almost never the correct thing to do. Will the new overlapping/duplicate building check ignore layer differences? (Which would be a good thing).

Additional questions

More info about your checks will be great. What happens after they are displayed - if you click on them does it jump to them? Does it block upload? How will they interact with the 'Warnings' and 'Errors' that already show in the save panel to the left?

[emi420]

Hi @SColchester !

1. 'Building has un-squared corners' sounds good.
2. Duplicate building is defined as the one that overlaps with an existing building more than a certain threshold (ex: 90%). This can be adjusted.
3. Yes, this can be implemented that way by the TM developers, as an optional feature for the project. One we have this in place we plan to add more validation checks that can be also enabled/disabled or customized depending on the project.
4. I agree that roof:material should be added by experienced mappers ideally on the field, but if a beginner remote mapper add this field anyway, it could be good to show the warning?
5. We can add a feature to detect hashtags (ex: strings starting with the '#' character) into fields like name or address.
6. It will not disable existing features on the iD editor, it will only show information about data quality issues found. The good thing about our tool is that it can be customized for different needs and also used on the full process from before user uploads changes to the map, to just after (near-real time) and also with historical data.
7. Yes! I'll use the feedback from these discussions and meetings to progress into the idea and then show more information about each use case.

Thanks!

[SColchester]

Thanks @emi420 for reponse and clarifications! On 4. Yes - and this should be extended for any building fields where they have added values - it should automatically assume that any new values are incorrect

[martien-176]

Because the "Data quality checks warnings" do not indicate where the affected features are located, beginner mappers could possibly get confused.

Besides from that, it would maybe enough to just limit the message to a warning for the mapper that iD Editor found issues. Show the pictures and text purely as examples of what the mapper should look for, and as an explanation on how to deal with these errors. But do not give detailed info about what and how many possible errors there are found. Refer the mapper to the iD Editor issues list instead for further actions.

Then there would also be no problems with the wording.

It would also prevent the situation that the iD Editor issues list and the "Data quality checks list" are not synchronized.

[emi420]

Hey @martien-176 !

1. I think we'll be able to show where the feature is located.
2. We can definitely add that, @petya-kangalova suggested to show a link to a wiki.
3. Yes, maybe a good start could be to show information about data quality related to the issues that were found, but not to show the exact issue and its location. I was also thinking into add related content explaining why data quality is important and which is the impact that it could have, like this:

So the feature could become more like a motivation/helper tool to map better. Detecting issues and showing related content, both explaining how to map good and why this is important.

For the precise listing of issues we'll still have a reporting tool that is intended to be used by validators, project managers and data teams.

[adiatmad]

This is super, thanks for sharing this, Emi!
When it comes to beginner mappers, I believe that the description of wording is more instructive so that everyone has the same understanding. It would be more helpful if we could get more information by hovering over the warning. Hovering over "Duplicate buildings," for example, will display a pop-up description of duplicate buildings.

[RAytoun]

I am still not understanding what this is all about!

Why are we reinventing the wheel?

iD Editor already has the Issues function that warns mappers that they are making an error.
JOSM already has the Validation function and multiple plug ins and Paint Styles that help the mapper identify issues.

We also have other tools like OSM Inspector, KeepRight, ImproveOSM, Osmose, to name a few.

Would it not be better to concentrate on a tool that quantifies and reports the number of errors occurring on a project/campaign or area so that we can make better judgements on the quality of the mapping rather than trying to replace or overwrite the existing editor functions.

Let's look at helping beginners to understand how to use the functions to understand how to fix errors they are making.

[emi420]

Hi @RAytoun !

We're looking for options on what kind of tools could be useful to add into the Tasking Manager for improving the quality of the mapping, this is not a final feature, it's just a proposal so we can have discussions over it.

Your opinions are really important and I'm taking notes in order to make the next proposal, that it will be more aligned to some ideas that were discussed here, like not showing the exact list of issues but use the results of the evaluation as a way to measure what kind of things the mappers could do in order to improve their mapping and help them with related suggestions and content. For example, if the tool detects that a set of changes have a lot of possible issues with tags combined with "building", we can show content explaining the impact of incorrect tagging on buildings and provide related links.

Regarding the tool that do quantifying and reporting, we're working on that also, expect news soon. But I wanted to have discussions on this topic first, as it's a more complex topic.

Thanks a lot.

[RAytoun]

Thank you for those kind words Emi.
My thoughts here are:
What is the issue - data quality
What are we trying to do - improve the quality of the initial mapping, not having to chase on afterwards and try and clean it up.
When is the best time to achieve that - to check the mapping of beginners and help them to improve as early as possible to avoid a trail of errors that needs cleaning up later.
What is the best solution - To teach the mapper how to map correctly before they start uploading data onto OpenStreetMap.
How do we achieve that - with an link in the Tasking Manager for beginners - Introduction to mapping on OSM that is a programmed assisted sandpit for beginners to try mapping before going live.
History of sandpit - JOSM used to have one where the mappers could experiment and try things out without going live with uploads. TeachOSM is developing one for students to do mapping projects that the teacher can assess without it uploading onto OpenStreetMap.

I have raised this concept on numerous occasions and it does not seem to be getting through. This will weed out those mappers who are not really interested in mapping on OSM and improve the mapping and give confidence to those that are interested in donating their time to OpenStreetMap. I have been at the forfront of validation for 8 years now and can tell you that the standard of mapping of beginners is getting worse, not better. So no matter what initial teaching is being given it is not achieving the desired acceptable quality that we need.

Picture this introduction.
They open up the sandpit and it starts off either with an iD Editor interface or a JOSM interface. The first thing that you see is the background imagery. A number of arrows indicate what a building looks like in the imagery, then it draws one of the buildings, tags it and squares it. Then the mapper is invited to draw another 6 buildings. The programme has a predrawn example of what is to be expected. the shape the mapper has drawn is close then great. If it bears no resemblance then the programme will show the mapper what that building should have looked like and then ask the mapper to draw it again.

Then it can move to a highway and repeat the process

Then it can move to a waterway and repeat the process

If the mapper shows competence in achieving the acceptable level of mapping they can move on to the live editor and start mapping there.

All that we are doing is replacing the training video with a proper training process. The we will start to improve the initial mapping of mappers and they will also be more comfortable in knowing that they got through the checked training steps.

That would be my first choice for improving the quality of beginner mapping.

[RAytoun]

We can refine this process to give examples of incorrect buildings and how to fix them, like overlapping buildings, like building shares a node with a highway or waterway or landuse. And it can explain what are (Issues in iD are and how to fix them), (Validator function in JOSM is and how to fix issues). Mappers may create errors but may not know how to correct them once they are created.

[RAytoun]

It would ideally have a link to a help channel where they can ask questions that will give them any extra help they may need.

[Ichchhie]

A similar kind of discussion happened a few months back in 2022 in one of the collective meetups which got initiated from a PR: The PR at first might look irrelevant but if scrolled down, some relevant inputs can be seen.

Sharing to bring perspectives from the PR as well!

[SColchester]

@RAytoun Main added value I see for this is that it notifies contributors at a very early stage about common quality issues that are currently not picked up in iD:

• Buildings without right-angled corners
• Buildings with meaningless values in fields (typically an incorrectly added hashtag string)

These two common issues are not picked up in the iD editor, and if HOT's version of iD had this check integrated it would be a value-add. It may lighten the load down-the-line for validation by prompting beginners to fix these issues themselves.

[SColchester]

I should clarify, these common issues can be picked up in the iD editor, but it is not currently an intuitive path to follow for beginner mappers to run a check - this enhancement should push these error checks to contributors without them having to perform any additional actions.

[emi420]

Maybe what we can do is: if the tool detects those common issues, it can suggest the user to use the iD checks before upload the changes.

[martien-176]

Presenting the mapper a list of errors is like duplicating the iD Editor issues list (reinventing the wheel). I have the impression beginner mappers often are not aware of the iD Editor issues list though, so a notification that there are issues would imo be useful (see also my previous comment).

I agree with Ralph that focusing on a real-time tool to check mapping quality would be worthwhile. I hope this can happen on a task level: it would be great to have a sort of heat-map overlay over the project map so that it is immediatly visible which tasks have the most issues and need extra and/or immediate attention.

But i realize that is belonging to another topic ("quality measurement - While uploading (live)").

[emi420]

Yes, I agree that the list of errors is duplicating the iD editor issues list in this case, maybe not in other cases as we have plans to add more checks, not only "wrong tagging" and "unsquared building".

The notification that there are issues and the suggestion to use the iD editor check in order to list and fix those issues could be a nice feature instead of showing the list.

And yes, we'll have another discussion specifically for real-time reporting.

Thanks @martien-176

[Ichchhie]

A lot of things have been shared already but I just wanted to add a few things where I see this new data quality module, can be useful in TM ecosystem:

1. Redefine Mapper Level Milestones #3713
   Since there is already a concern that the current criteria of counting only changesets for the mapper level upgrade might not be the right way, I see, this DQ measurement can be additional criteria in redefining the mapper level

2. Likewise there can be the addition of a Featured weekly/monthly Leaderboard in TM to show the top users based on the data quality

Both the above points I believe will directly/indirectly contribute to people looking for improving their mapping and maintaining the data quality.

3. As said in the above comments, definitely having a sandbox would make a good addition for user onboarding but if that's technically complex and takes time, I think its better if we get started with simple implementations first. This will also help with testing of this new component and probably improvise when we move onto sandbox kinds.

[martien-176]

Hi @Ichchhie, i agree mostly with what you are saying.

I would be careful though with the leaderboard. Some mappers could (will?) possibly be tempted to game the system, eg by only looking for parts of the project containing less features. And leaving the more difficult parts to others.

[Ichchhie]

Yeah, that might be the case, but looking at the positive side, having leaderboards in mapping tools can serve as a motivator for people to participate and receive recognition for their efforts. Crowdsourced mapping always involves a tradeoff between quality and quantity. About your example, the positive aspect is that even if people eventually leave, they will have mapped the easier features correctly, which may aid in mapping the difficult areas later on. Also, since we are considering redefining mapper milestones if it is possible to incorporate this mix of difficult and easy features that may encourage mappers to learn how to map the difficult ones as well?

However, just a thought, need to hear what others think as well. Thank you for your input.