From 9e60fe05280c660a25f1a6e0c04dd16a9f4e1d98 Mon Sep 17 00:00:00 2001 From: cartesinus Date: Fri, 31 Mar 2023 14:54:48 +0200 Subject: [PATCH 1/2] Update README with baseline --- README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/README.md b/README.md index 275fe6f..59dadf8 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,21 @@ The test set prepared for the final evaluation will be released on May 17, 2023, |Intents | 193 | 193 | |Slots types | 80 | 79 | +## Baseline + +We use XLM-RoBERTa Base as a baseline model for intent detection and slot-filling. XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. Each model was trained independently on the entire training set and optimized on the evaluation set. Results of the evaluation of the final epoch checkpoint on the test set are presented in the table below: + +| Language | Intent Accuracy (%) | Slot F1 (%) | +|----------|---------------------|-------------| +| en-US | 90.40 | 78.77 | +| es-ES | 97.07 | 96.91 | +| pl-PL | 95.94 | 96.02 | +| all | 95.04 | - | + +All models are available at huggingface where all details of model training and how to execute them are described: +- intent: [en-US](cartesinus/fedcsis-intent_baseline-xlm_r-en), [es-ES](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-es), [pl-PL](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-pl), [all](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-all) +- slot: [en-US](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-en), [es-ES](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-es), [pl-PL](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-pl) + ## Submissions - The solutions for the task are to be submitted via the Gonito platform challenge available at . From b6c1a7f6d2492a4b385167ddbc9eb0cc5aa610bf Mon Sep 17 00:00:00 2001 From: cartesinus Date: Fri, 31 Mar 2023 14:59:04 +0200 Subject: [PATCH 2/2] Fix link to baseline model in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 59dadf8..755b80f 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ We use XLM-RoBERTa Base as a baseline model for intent detection and slot-fillin | all | 95.04 | - | All models are available at huggingface where all details of model training and how to execute them are described: -- intent: [en-US](cartesinus/fedcsis-intent_baseline-xlm_r-en), [es-ES](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-es), [pl-PL](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-pl), [all](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-all) +- intent: [en-US](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-en), [es-ES](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-es), [pl-PL](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-pl), [all](https://huggingface.co/cartesinus/fedcsis-intent_baseline-xlm_r-all) - slot: [en-US](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-en), [es-ES](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-es), [pl-PL](https://huggingface.co/cartesinus/fedcsis-slot_baseline-xlm_r-pl) ## Submissions