2015-11-11 22:10:41 +01:00
<h1>Challenge How To
2017-09-28 20:44:44 +02:00
<h2>Install GEval (recommended, but not obligatory)
2015-11-11 22:10:41 +01:00
2019-11-25 22:50:34 +01:00
<p>In order to evaluate your work locally, you need the GEval tool. There are two ways to get it.
2018-04-09 11:38:34 +02:00
2019-11-25 22:50:34 +01:00
<h4>option A — just download the GEval fully static binary (simpler if you have a standard 64-bit Linux)
<pre>
wget <a href="https://gonito.net/get/bin/geval">https://gonito.net/get/bin/geval</a>
chmod u+x geval
./geval --help
<h4>option B — install GEval with Haskell Stack (needs some time and a couple of gigabytes of hard disk space)
<p>First, install <a href="https://docs.haskellstack.org/en/stable/README/">Haskell Stack</a>:
2017-09-23 15:45:14 +02:00
<pre>
curl -sSL https://get.haskellstack.org/ | sh
<p>and then install GEval itself:
2015-11-11 22:10:41 +01:00
<pre>
git clone git://gonito.net/geval
cd geval
stack setup
stack install
2018-04-09 11:38:34 +02:00
~/.local/bin/geval --help
2015-11-11 22:10:41 +01:00
2019-11-25 22:50:34 +01:00
<p>For troubleshooting, see <a href="https://gitlab.com/filipg/geval#installing">GEval readme</a>.
2017-09-28 20:44:44 +02:00
<p>(GEval will make your life easier, but is not obligatory, so if you have any trouble with installing/running GEval, you can skip this step, for the time being.)
2017-09-23 15:45:14 +02:00
2015-11-11 22:10:41 +01:00
<h2>Get your repo
2018-06-06 10:30:53 +02:00
$case (appRepoScheme settings)
$of SelfHosted
2020-02-22 09:12:13 +01:00
$if (not isIDSet || not isSSHUploaded) && isNothing mAltRepoScheme
2018-06-06 10:30:53 +02:00
<p>You need to
$if not isIDSet
\ set up your ID
$if not isSSHUploaded
\ and
$if not isSSHUploaded
\ upload your SSH public key
\ (see <a href="@{YourAccountR}">your account</a>) if you want to have a repo hosted on Gonito.net, then:
2015-11-11 22:10:41 +01:00
2020-02-22 09:12:13 +01:00
$maybe altRepoScheme <- mAltRepoScheme
2021-08-21 12:11:40 +02:00
<p>(Depending on your setup, you might need to create the repo at your git server first; remember to initiate it without any commits!)
2020-02-22 09:12:13 +01:00
<pre>
git clone #{altRepoScheme}#{challengeName challenge}
$nothing
<pre>
git clone --single-branch #{appRepoHost settings}#{shownId}/#{challengeName challenge}
2017-10-03 22:19:18 +02:00
2020-02-22 19:12:18 +01:00
<p>(Warning about empty repository is expected, don't worry about it.)
2017-10-03 22:19:18 +02:00
2018-06-06 10:30:53 +02:00
<pre>
cd #{challengeName challenge}
2020-02-22 19:12:18 +01:00
git pull #{repoUrl repo}
2018-06-06 10:30:53 +02:00
2019-11-25 23:09:09 +01:00
<h4>Using an external repository
$maybe serverPubKey <- (appServerSSHPublicKey settings)
2020-02-22 09:12:13 +01:00
$if isNothing mAltRepoScheme
<p>Alternatively, you can use any other Git repo, e.g. GitLab, GitHub or your own repo, make sure Gonito.net has access to your repo, either by making it public or giving read access to the SSH public key:
$else
<pre>
#{serverPubKey}
2019-11-25 23:09:09 +01:00
$nothing
2020-02-22 09:12:13 +01:00
$if isNothing mAltRepoScheme
<p>Alternatively, you can use any other Git repo, e.g. GitLab, GitHub or your own repo, make sure Gonito.net has access to your repo.
2019-11-25 23:09:09 +01:00
2018-06-06 10:30:53 +02:00
$of Branches
<p>Clone the repo:
<pre>
2018-10-15 14:19:20 +02:00
git clone --single-branch #{repoUrl repo}
2015-11-11 22:10:41 +01:00
2018-06-06 10:30:53 +02:00
$maybe gitAnnexRemote <- (repoGitAnnexRemote repo)
<pre>
git annex init
git annex initremote storage #{gitAnnexRemote}
git annex get --from storage
2015-11-11 22:10:41 +01:00
<h2>Work on your solution
2020-02-22 09:12:13 +01:00
<p>You need to generate your solution for the test set as <tt>test-A/out.tsv</tt>. It is also recommended to generate the output for the dev set (<tt>dev-0/out.tsv</tt>).
2015-11-11 22:10:41 +01:00
<p>You can evaluate results for the dev set locally:
<pre>
cd #{challengeName challenge}
geval --test-name dev-0
2017-09-28 16:51:10 +02:00
<h2>Push your solution to the git repo
2015-11-11 22:10:41 +01:00
2017-09-28 16:51:10 +02:00
<p>Commit and push <tt>out.tsv</tt> files to your repo. It is also recommended to push your source codes along with <tt>out.tsv</tt> files.
2015-11-11 22:10:41 +01:00
<pre>
cd #{challengeName challenge}
2020-02-22 09:12:13 +01:00
git checkout -b #{myBranch} # switch to some other branch
2018-06-06 10:30:53 +02:00
git add foo.py build.sh # add your source codes
2020-02-22 09:12:13 +01:00
git add gonito.yaml # it's a good practice to add metadata file, see below
2018-06-06 10:30:53 +02:00
2018-07-28 21:36:45 +02:00
$if isJust (repoGitAnnexRemote repo)
2018-06-06 10:30:53 +02:00
<pre>
# if your output files are large or if they should not pushed to the regular repo (e.g. contain sensitive information):
git annex add dev-0/out.tsv test-A/out.tsv
git annex copy --to storage
# otherwise (e.g. they are just class labels), add the output files in a regular manner:
git add dev-0/out.tsv test-A/out.tsv
2018-07-28 21:36:45 +02:00
$else
2018-06-06 10:30:53 +02:00
<pre>
git add dev-0/out.tsv test-A/out.tsv # add your output files
<pre>
2015-11-11 22:10:41 +01:00
git commit -m 'my brilliant solution'
2018-06-06 10:30:53 +02:00
$case (appRepoScheme settings)
$of SelfHosted
<pre>
2020-02-22 09:12:13 +01:00
git push origin #{myBranch}
2018-09-01 11:06:42 +02:00
<h3>Repos hosted on Gonito.net
<p>If you use a repo hosted here, a submission and evaluation is triggered automatically. You'll see the evaluation results in your console while pushing.
2018-06-06 10:30:53 +02:00
$of Branches
<pre>
2020-02-22 09:12:13 +01:00
git push origin #{myBranch}
2015-11-11 22:10:41 +01:00
2017-09-28 16:51:10 +02:00
<h3>External repos
2018-09-01 11:06:42 +02:00
<p>If you use an external repo (e.g. at your own of Gitolite or at GitLab/GitHub), you can configure a webhook.
2017-09-28 16:51:10 +02:00
$maybe token <- mToken
<p>Your webook is:
<pre>
2020-02-22 09:12:13 +01:00
wget --timeout=0 --quiet -O - '#{appRoot settings}/trigger-remotely' --post-data 'token=#{token}&branch=#{myBranch}&challenge=#{challengeName challenge}&url=#{urlToYourRepo}'
2020-02-21 22:56:39 +01:00
<p>or you can just use the following URL:
<pre>
2020-02-22 09:12:13 +01:00
#{appRoot settings}/trigger-remotely-simple/#{token}/#{challengeName challenge}/#{urlToYourRepo}/#{myBranch}
2017-09-28 16:51:10 +02:00
<h3>Manual submission
<p>In case other methods fail, you can submit your solution manually — go to the <a href="@{ChallengeSubmissionR $ challengeName challenge}">submit form</a>.
2018-10-06 23:30:12 +02:00
<h2>Submission metadata
2018-11-03 18:02:27 +01:00
<p>Gonito can take the metadata (description, tags, parameters) of a submission from a number of sources (in order of precedence):
2018-10-06 23:30:12 +02:00
<ol>
2018-11-03 18:02:27 +01:00
<li>the YAML files specified in the <tt>param-files</tt> field of the <tt>gonito.yaml</tt> file
<ul>
<li>only applicable for parameters,
<li><tt>gonito.yaml</tt> will be always skipped even if matches a mask given in the <tt>param-files</tt> field,
<li>parameters blacklisted in the <tt>unwanted-params</tt> field of the <tt>gonito.yaml</tt> file will be discarded;
<li><tt>gonito.yaml</tt> file committed to the repository
<ul>
<li>description given in the <tt>description</tt> field,
<li>tags given in <tt>tags</tt> field,
<li>parameters given in <tt>params</tt> field,
2018-10-06 23:30:12 +02:00
<li>names of output files (only for parameters)
2018-11-03 18:02:27 +01:00
<ul>
<li>e.g. if the output file is <tt>out-epochs=10,learning-rate=0.01.tsv</tt>, then parameters <i>epochs</i>=10 and <i>learning-rare</i>=0.01 will be extracted;
<li>submission form (when submitting manually);
<li>git commit message
<ul>
<li>description taken from the first paragraph
<li>tags taken from a line starting with <tt>tags:</tt>
2021-07-28 21:00:02 +02:00
<p>Here is an example of <tt>gonito.yaml</tt>, in which all metadata could be given (note that you can also add links to external resources using the `links` section):
2018-11-03 18:02:27 +01:00
<pre>
description: This my brilliant solution
tags:
\ - neural-network
\ - left-to-right
params:
\ epochs: 10
\ learning-rate: 0.01
unwanted-params:
\ - model-file
\ - vocab-file
param-files:
\ - "*.yaml"
\ - config/*.yaml
2021-07-28 21:00:02 +02:00
links:
\ - title: "Some external link"
\ url: "https://example.com/foo-bar-baz-123"
\ - title: "Yet another link"
\ url: "https://example.org/xyz"
\ - url: "https://example.net/bare-link-without-text"
2018-10-06 23:30:12 +02:00
<p>It might seem a little bit complicated, but you could simply use the method which is the most convenient for you.