Context:

The context of this work is Wikimedia's ORES, a ML-powered service that provides scoring information for wiki contributions. The necessity for such a system resulted of a growing community where anyone can edit and where it would be unrealistic to manually control the quality of every edit.

Problem:

The problem here is that the adequate usage of ORES is heavily context-dependent. We will be focusing on the binary damaging classifier, that for a given edit tells us if it is harmful or not. ORES lacks comprehensive documentation of many aspects and it is quite difficult to optimize the system for one's personal needs - even more so without extensive knowledge in the machine learning field.

Objectives:

With PreCall we propose a system that is intended to help users - experts and non experts - make the right optimization decisions by proposing different visualization of ORES data, updated in instantly upon interaction.

Procedure:

With help of the later mentioned literature we plan on developing PreCall in an iterative approach - function and design wise. The system will, upon being started, query the ORES API once, save the results and then present them under the form of visualizations that might still be susceptible to changes. As of now, we plan on showing a radar graph illustrating different model statistics, a threshold slider bar, both of which should be manipulable by the user, as well as a visualization that represents current values of the confusion matrix in an understandable and intuitive form, listing the percentage of edits that were (1) correctly detected as good, (2) wrongly detected as damaging, (3) correctly detected as damaging and (4) wrongly detected as good. The full procedure would include:

1) Extensive research on and documentation of ORES and its functionalities

2) Construction of basic version of PreCall with working API queries

3) Research on existing systems

4) Analysis of insights gained from 3)

5) Application of 4) to PreCall in an iterative development/evaluation cycle

6) Completing the thesis' written part

References

(1) ORES: Facilitating re-mediation of Wikipedia’s socio-technical problems (Halfaker et al.)

(2) A Review of User Interface Design for Interactive Machine Learning https://dl.acm.org/citation.cfm?id=3185517

(3) Diagnostic Visualization for Non-expert Machine Learning Practitioners: A Design Study http://tomeri.org/diagnosticVisualization-VLHCC-2016.pdf and http://www.felienne.com/archives/5191

(4) Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models https://eng.uber.com/manifold/

(5) Visualizations for model tracking and predictions in machine learning https://dspace.mit.edu/handle/1721.1/113133

Other Links:

https://www.mediawiki.org/wiki/ORES/Applications

https://www.mediawiki.org/wiki/ORES/Thresholds