Usage

evOWLuator can be invoked through the evowluate binary. It is possible to get an overview of available subcommands by invoking it with the -h flag:

  • classification: runs the evaluation for the ontology classification task.

  • consistency: runs the evaluation for the ontology consistency task.

  • matchmaking: runs the evaluation for the matchmaking task.

  • info: prints information about configured reasoners and datasets.

  • convert: converts a dataset into the specified OWL syntax.

  • visualize: generates high-level statistics and plots for a previous evaluation.

All subcommands support the following help flags:

  • -h, --help: print help for the specified subcommand.

  • --debug: halt execution and print additional information on error.

Subcommand-specific flags are detailed in the following.

classification, consistency, matchmaking

Required arguments:

  • -m, --mode <MODE>: evaluation mode. Possible values:

    • correctness: check the validity of reasoners’ output, using one of them as an oracle. The test oracle is the first reasoner specified in the list following the -r flag.

    • performance: collect statistics about performance, in terms of turnaround time and maximum memory usage.

    • energy: compute a relative estimate of the energy consumed by the inference task. This mode requires specifying the class name of the energy probe the framework should use via the -e flag.

  • -e, --energy-probe <PROBE>: class name of the energy probe the framework should use, only required with -m energy. Built-in probes: powertop, powermetrics.

Optional arguments:

  • -d, --dataset <DATASET>: target dataset. Default: first dataset in the data directory.

  • -r, --reasoners <REASONER> [<REASONER> ...]: list of reasoners to use. Default: all configured reasoners.

  • -n, --num-iterations <NUM>: number of iterations for each test. Default: 5.

  • -t, --timeout <TIMEOUT>: timeout imposed on each reasoner for a single inference task, in seconds. Default: 1200.

  • -s, --syntax <SYNTAX>: reference OWL syntax. Values: dl, functional, krss, krss2, manchester, obo, owlxml, rdfxml, turtle. Default: preferred syntax for each reasoner.

  • --resume-after <ONTOLOGY>: resume the evaluation after the specified ontology.

info

Optional arguments:

  • -d, --dataset <DATASET>: target dataset. Default: first dataset in the data directory.

  • -r, --reasoners <REASONER> [<REASONER> ...]: list of reasoners to use. Default: all configured reasoners.

convert

Required arguments:

  • -d, --dataset <DATASET>: dataset to convert.

  • -s, --syntax <SYNTAX>: target syntax. Values: dl, functional, krss, krss2, manchester, obo, owlxml, rdfxml, turtle

visualize

Required arguments:

  • path: path to the directory containing the evaluation results to visualize.

Optional arguments:

  • -s, --size <WIDTH> <HEIGHT>: width and height of the figure in inches.

  • -p, --plots <PLOT> [<PLOT> ...]: subplots to show (default: all).

  • -r, --reasoners <REASONER> [<REASONER ...]: reasoners whose results should be plotted. Default: all configured reasoners.

  • --no-gui: do not show the interactive GUI.

  • --no-titles: omit titles for figures and axes.

  • --no-labels: omit value labels.

  • --label-fmt <FORMAT>: float format of value labels.

  • --label-rot: rotation of value labels in degrees.

  • --xtick-rot: rotation of labels on the x axis in degrees.

  • --ytick-rot: rotation of labels on the y axis in degrees.

  • --legend-loc <LOCATION>: location of the legend. Values:

    • none: do not plot the legend.

    • best: let the matplotlib engine decide the position.

    • [upper, lower, center] right, left, center: explicit position.

  • --legend-cols: number of columns of the legend.

  • --legend-only: only plot the legend.

  • --colors: colors to use for each reasoner.

  • --markers: markers to use for each reasoner.

  • --marker-size: marker size in points.