gonito/templates/writing-papers.hamlet

<p>If you write a paper, Gonito can make your life a little bit easier: you won't need to copy&paste evaluation results manually and all your results will referenced in a proper manner.

<p>Just add the following piece of LaTeX code to your paper:

<pre>
  \\usepackage{hyperref}
  \\usepackage{xstring}
  % Format a reference to a Gonito submission
  \\newcommand{\gonitoref}[1]{\{\href{#{rootAddress}/q/#1}{\StrMid{#1}{1}{6}}\}}
  % A bare score from Gonito
  \\newcommand{\gonitobarescore}[1]{\minput{scores/#1.txt}}
  % A score from Gonito along with a reference
  \\newcommand{\gonitoscore}[1]{\gonitobarescore{#1} \gonitoref{#1}}
  % A reference and a score as two cells in a table
  \\newcommand{\gonitoentry}[1]{\gonitoref{#1} & \minput{scores/#1.txt}}

<p>Now you will be able to reference your Gonito submissions using its git commit hash, e.g.: <tt>\gonitoref{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5}</tt>. The hash will be printed in a shorter form (just first 6 digits) and it will be clickable leading to the Gonito entry describing the submission (information how to get the data will be presented there).

<p>You might explain the idea like this:

<pre>
  \\gonitoscore{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5}%
  \\footnote{Reference codes to repositories stored at
  Gonito.net~\cite{gonito2016} are given in curly brackets. Such a repository may be also accessed by going
  to \url{http://gonito.net/q} and entering the code there.}

<p>Here is the BiBTeX entry referenced in the above snippet:

<pre>
  @incollection { gonito2016,
  \  title = {Gonito.net -- Open Platform for Research Competition, Cooperation and Reproducibility},
  \  author = "Grali{\'n}ski, Filip and Jaworski, Rafa{\l} and Borchmann, {\L}ukasz and Wierzcho{\'n}, Piotr",
  \  editor = "Branco, António and Calzolari , Nicoletta and Choukri, Khalid",
  \  booktitle = {Proceedings of the 4REAL Workshop: Workshop on Research Results Reproducibility and Resources Citation in Science and Technology of Language},
  \  year = "2016",
  \  pages = "13-20"
  }

<h3>Hashes

<p>Two kinds of (SHA1) hashes can be used here:

<ul>
  <li>submission hashes, i.e. the commit hash with the submission; the problem is that they can contain multiple output files (so-called variants), in such a case…
  <li>… output hashes can be used (i.e. the SHA1 digest of a specific output file)

<p>In general, Gonito tries to guess the right metric in case of ambiguity. It is, however, a good idea to give the metric name along the reference codes, e.g.:

<pre>
  \\gonitoscore{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5-Accuracy}

<h3>Actually getting the scores

<p>But, wait, with Gonito you can give evaluation scores without manual copy&paste. The <tt>\gonitoscore</tt> command gives a score and a reference. The score is taken from a file <tt>scores/HASH.txt</tt>. You need to get it from Gonito, but it's not difficult to set it up in such a way that the scores could be downloaded automatically. For instance, if you use Makefile for building your papers, you could use the following snippet:

<pre>
  SCOREFILES=$(shell ./extract-score-files.pl main.tex)
  \
  paper.pdf: paper.tex main.tex $(SCOREFILES)
  #{tab}... building instructions ...
  \
  scores/%.txt:
  #{tab}mkdir -p scores
  #{tab}wget "https://gonito.net/api/txt/score/"$* -O $@

<p>The script <tt>extract-score-files.pl</tt> is like this:

<pre>
  #!/usr/bin/perl
  \
  use strict;
  \
  open(my $ih, '<', $ARGV[0]);
  binmode($ih, ':utf8');
  \
  my %found = ();
  \
  while (my $line=<$ih>) {
  \    while ($line =~ m<\\gonito(?:barescore|score|entry)\{([^\}]+)\}>g) {
  \        $found{$1} = 1;
  \    }
  }
  \
  print join(" ", map { "scores/${_}.txt" } sort keys %found);

<p>The <tt>\gonitoentry</tt> command is very similar to <tt>\gonitoscore</tt>, it just formats the infromation as two cells in a table (i.e. <tt>\gonitoentry</tt> can be used <strong>only</strong> in tables).

<p>Note that you can commit score files to your repository, so that everything is OK when your paper is edited at Overleaf or a similar service.