s464905/en-ner-conll-2003
1
0
Fork 0
forked from kubapok/en-ner-conll-2003
Code Pull Requests Activity
transformer
en-ner-conll-2003/Transformer.ipynb

710 lines
31 KiB
Plaintext
Raw Permalink Normal View History

Transformer implementation
2024-06-03 12:24:54 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Transformer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Użyj transformeroewgo pipeline'u (https://huggingface.co/docs/transformers/main_classes/pipelines) do implementacji zadania rozpoznawania jednostek nazewniczych (NER) na zbiorze danych https://git.wmi.amu.edu.pl/kubapok/en-ner-conll-2003. \\\n",
"Dokonaj ewaluacji za pomocą narzędzia GEval."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import bibliotek"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer\n",
"from tqdm.notebook import tqdm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Wczytanie danych"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"test_A_data = pd.read_csv(\"test-A/in.tsv\", sep=\"\\t\", header=None, names=[\"x\"])\n",
"dev_0_data = pd.read_csv(\"dev-0/in.tsv\", sep=\"\\t\", header=None, names=[\"x\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ustawienie modelu, tokenizatora oraz pipeline'u"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at dbmdz/bert-large-cased-finetuned-conll03-english were not used when initializing BertForTokenClassification: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
"- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
]
}
],
"source": [
"model = AutoModelForTokenClassification.from_pretrained(\"dbmdz/bert-large-cased-finetuned-conll03-english\")\n",
"tokenizer = AutoTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n",
"recognizer = pipeline(\"ner\", model=model, tokenizer=tokenizer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Funkcja naprawiająca przewidziane tagi"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def correct_labels(data):\n",
" corrected_lines = []\n",
"\n",
" for line in data:\n",
" corrected_line = []\n",
" previous_token = \"O\"\n",
"\n",
" for token in line:\n",
" if (\n",
" token == \"I-ORG\"\n",
" and previous_token != \"B-ORG\"\n",
" and previous_token != \"I-ORG\"\n",
" ):\n",
" corrected_line.append(\"B-ORG\")\n",
" elif (\n",
" token == \"I-PER\"\n",
" and previous_token != \"B-PER\"\n",
" and previous_token != \"I-PER\"\n",
" ):\n",
" corrected_line.append(\"B-PER\")\n",
" elif (\n",
" token == \"I-LOC\"\n",
" and previous_token != \"B-LOC\"\n",
" and previous_token != \"I-LOC\"\n",
" ):\n",
" corrected_line.append(\"B-LOC\")\n",
" elif (\n",
" token == \"I-MISC\"\n",
" and previous_token != \"B-MISC\"\n",
" and previous_token != \"I-MISC\"\n",
" ):\n",
" corrected_line.append(\"B-MISC\")\n",
" else:\n",
" corrected_line.append(token)\n",
"\n",
" previous_token = token\n",
"\n",
" corrected_lines.append(corrected_line)\n",
"\n",
" return corrected_lines"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Funkcja przewidująca tagi NER"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def predict_ner_tags(data):\n",
" predictions = []\n",
" counter = 1\n",
" for line in data:\n",
" print(f'Predicting NER tags for line {counter}/{len(data)}... ', end='')\n",
" word_positions = []\n",
" position = 0\n",
" result = recognizer(line)\n",
" entity_dict = {res['start']: res['entity'] for res in result}\n",
"\n",
" for word in line.split():\n",
" word_positions.append(position)\n",
" position += len(word) + 1\n",
" classified_words = []\n",
"\n",
" for checked_position in word_positions:\n",
" entity = entity_dict.get(checked_position, \"O\")\n",
" classified_words.append(entity)\n",
"\n",
" predictions.append(classified_words)\n",
" print('Done')\n",
" counter += 1\n",
" return correct_labels(predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Funkcja zapisująca wyniki"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def save_predictions(predictions, filename):\n",
" with open(filename, \"w\") as f:\n",
" for line in predictions:\n",
" f.write(\" \".join(line) + \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Wyznaczenie tagów NER"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prediction for dev-0 data\n",
"Predicting NER tags for line 1/215... Done\n",
"Predicting NER tags for line 2/215... Done\n",
"Predicting NER tags for line 3/215... Done\n",
"Predicting NER tags for line 4/215... Done\n",
"Predicting NER tags for line 5/215... Done\n",
"Predicting NER tags for line 6/215... Done\n",
"Predicting NER tags for line 7/215... Done\n",
"Predicting NER tags for line 8/215... Done\n",
"Predicting NER tags for line 9/215... Done\n",
"Predicting NER tags for line 10/215... Done\n",
"Predicting NER tags for line 11/215... Done\n",
"Predicting NER tags for line 12/215... Done\n",
"Predicting NER tags for line 13/215... Done\n",
"Predicting NER tags for line 14/215... Done\n",
"Predicting NER tags for line 15/215... Done\n",
"Predicting NER tags for line 16/215... Done\n",
"Predicting NER tags for line 17/215... Done\n",
"Predicting NER tags for line 18/215... Done\n",
"Predicting NER tags for line 19/215... Done\n",
"Predicting NER tags for line 20/215... Done\n",
"Predicting NER tags for line 21/215... Done\n",
"Predicting NER tags for line 22/215... Done\n",
"Predicting NER tags for line 23/215... Done\n",
"Predicting NER tags for line 24/215... Done\n",
"Predicting NER tags for line 25/215... Done\n",
"Predicting NER tags for line 26/215... Done\n",
"Predicting NER tags for line 27/215... Done\n",
"Predicting NER tags for line 28/215... Done\n",
"Predicting NER tags for line 29/215... Done\n",
"Predicting NER tags for line 30/215... Done\n",
"Predicting NER tags for line 31/215... Done\n",
"Predicting NER tags for line 32/215... Done\n",
"Predicting NER tags for line 33/215... Done\n",
"Predicting NER tags for line 34/215... Done\n",
"Predicting NER tags for line 35/215... Done\n",
"Predicting NER tags for line 36/215... Done\n",
"Predicting NER tags for line 37/215... Done\n",
"Predicting NER tags for line 38/215... Done\n",
"Predicting NER tags for line 39/215... Done\n",
"Predicting NER tags for line 40/215... Done\n",
"Predicting NER tags for line 41/215... Done\n",
"Predicting NER tags for line 42/215... Done\n",
"Predicting NER tags for line 43/215... Done\n",
"Predicting NER tags for line 44/215... Done\n",
"Predicting NER tags for line 45/215... Done\n",
"Predicting NER tags for line 46/215... Done\n",
"Predicting NER tags for line 47/215... Done\n",
"Predicting NER tags for line 48/215... Done\n",
"Predicting NER tags for line 49/215... Done\n",
"Predicting NER tags for line 50/215... Done\n",
"Predicting NER tags for line 51/215... Done\n",
"Predicting NER tags for line 52/215... Done\n",
"Predicting NER tags for line 53/215... Done\n",
"Predicting NER tags for line 54/215... Done\n",
"Predicting NER tags for line 55/215... Done\n",
"Predicting NER tags for line 56/215... Done\n",
"Predicting NER tags for line 57/215... Done\n",
"Predicting NER tags for line 58/215... Done\n",
"Predicting NER tags for line 59/215... Done\n",
"Predicting NER tags for line 60/215... Done\n",
"Predicting NER tags for line 61/215... Done\n",
"Predicting NER tags for line 62/215... Done\n",
"Predicting NER tags for line 63/215... Done\n",
"Predicting NER tags for line 64/215... Done\n",
"Predicting NER tags for line 65/215... Done\n",
"Predicting NER tags for line 66/215... Done\n",
"Predicting NER tags for line 67/215... Done\n",
"Predicting NER tags for line 68/215... Done\n",
"Predicting NER tags for line 69/215... Done\n",
"Predicting NER tags for line 70/215... Done\n",
"Predicting NER tags for line 71/215... Done\n",
"Predicting NER tags for line 72/215... Done\n",
"Predicting NER tags for line 73/215... Done\n",
"Predicting NER tags for line 74/215... Done\n",
"Predicting NER tags for line 75/215... Done\n",
"Predicting NER tags for line 76/215... Done\n",
"Predicting NER tags for line 77/215... Done\n",
"Predicting NER tags for line 78/215... Done\n",
"Predicting NER tags for line 79/215... Done\n",
"Predicting NER tags for line 80/215... Done\n",
"Predicting NER tags for line 81/215... Done\n",
"Predicting NER tags for line 82/215... Done\n",
"Predicting NER tags for line 83/215... Done\n",
"Predicting NER tags for line 84/215... Done\n",
"Predicting NER tags for line 85/215... Done\n",
"Predicting NER tags for line 86/215... Done\n",
"Predicting NER tags for line 87/215... Done\n",
"Predicting NER tags for line 88/215... Done\n",
"Predicting NER tags for line 89/215... Done\n",
"Predicting NER tags for line 90/215... Done\n",
"Predicting NER tags for line 91/215... Done\n",
"Predicting NER tags for line 92/215... Done\n",
"Predicting NER tags for line 93/215... Done\n",
"Predicting NER tags for line 94/215... Done\n",
"Predicting NER tags for line 95/215... Done\n",
"Predicting NER tags for line 96/215... Done\n",
"Predicting NER tags for line 97/215... Done\n",
"Predicting NER tags for line 98/215... Done\n",
"Predicting NER tags for line 99/215... Done\n",
"Predicting NER tags for line 100/215... Done\n",
"Predicting NER tags for line 101/215... Done\n",
"Predicting NER tags for line 102/215... Done\n",
"Predicting NER tags for line 103/215... Done\n",
"Predicting NER tags for line 104/215... Done\n",
"Predicting NER tags for line 105/215... Done\n",
"Predicting NER tags for line 106/215... Done\n",
"Predicting NER tags for line 107/215... Done\n",
"Predicting NER tags for line 108/215... Done\n",
"Predicting NER tags for line 109/215... Done\n",
"Predicting NER tags for line 110/215... Done\n",
"Predicting NER tags for line 111/215... Done\n",
"Predicting NER tags for line 112/215... Done\n",
"Predicting NER tags for line 113/215... Done\n",
"Predicting NER tags for line 114/215... Done\n",
"Predicting NER tags for line 115/215... Done\n",
"Predicting NER tags for line 116/215... Done\n",
"Predicting NER tags for line 117/215... Done\n",
"Predicting NER tags for line 118/215... Done\n",
"Predicting NER tags for line 119/215... Done\n",
"Predicting NER tags for line 120/215... Done\n",
"Predicting NER tags for line 121/215... Done\n",
"Predicting NER tags for line 122/215... Done\n",
"Predicting NER tags for line 123/215... Done\n",
"Predicting NER tags for line 124/215... Done\n",
"Predicting NER tags for line 125/215... Done\n",
"Predicting NER tags for line 126/215... Done\n",
"Predicting NER tags for line 127/215... Done\n",
"Predicting NER tags for line 128/215... Done\n",
"Predicting NER tags for line 129/215... Done\n",
"Predicting NER tags for line 130/215... Done\n",
"Predicting NER tags for line 131/215... Done\n",
"Predicting NER tags for line 132/215... Done\n",
"Predicting NER tags for line 133/215... Done\n",
"Predicting NER tags for line 134/215... Done\n",
"Predicting NER tags for line 135/215... Done\n",
"Predicting NER tags for line 136/215... Done\n",
"Predicting NER tags for line 137/215... Done\n",
"Predicting NER tags for line 138/215... Done\n",
"Predicting NER tags for line 139/215... Done\n",
"Predicting NER tags for line 140/215... Done\n",
"Predicting NER tags for line 141/215... Done\n",
"Predicting NER tags for line 142/215... Done\n",
"Predicting NER tags for line 143/215... Done\n",
"Predicting NER tags for line 144/215... Done\n",
"Predicting NER tags for line 145/215... Done\n",
"Predicting NER tags for line 146/215... Done\n",
"Predicting NER tags for line 147/215... Done\n",
"Predicting NER tags for line 148/215... Done\n",
"Predicting NER tags for line 149/215... Done\n",
"Predicting NER tags for line 150/215... Done\n",
"Predicting NER tags for line 151/215... Done\n",
"Predicting NER tags for line 152/215... Done\n",
"Predicting NER tags for line 153/215... Done\n",
"Predicting NER tags for line 154/215... Done\n",
"Predicting NER tags for line 155/215... Done\n",
"Predicting NER tags for line 156/215... Done\n",
"Predicting NER tags for line 157/215... Done\n",
"Predicting NER tags for line 158/215... Done\n",
"Predicting NER tags for line 159/215... Done\n",
"Predicting NER tags for line 160/215... Done\n",
"Predicting NER tags for line 161/215... Done\n",
"Predicting NER tags for line 162/215... Done\n",
"Predicting NER tags for line 163/215... Done\n",
"Predicting NER tags for line 164/215... Done\n",
"Predicting NER tags for line 165/215... Done\n",
"Predicting NER tags for line 166/215... Done\n",
"Predicting NER tags for line 167/215... Done\n",
"Predicting NER tags for line 168/215... Done\n",
"Predicting NER tags for line 169/215... Done\n",
"Predicting NER tags for line 170/215... Done\n",
"Predicting NER tags for line 171/215... Done\n",
"Predicting NER tags for line 172/215... Done\n",
"Predicting NER tags for line 173/215... Done\n",
"Predicting NER tags for line 174/215... Done\n",
"Predicting NER tags for line 175/215... Done\n",
"Predicting NER tags for line 176/215... Done\n",
"Predicting NER tags for line 177/215... Done\n",
"Predicting NER tags for line 178/215... Done\n",
"Predicting NER tags for line 179/215... Done\n",
"Predicting NER tags for line 180/215... Done\n",
"Predicting NER tags for line 181/215... Done\n",
"Predicting NER tags for line 182/215... Done\n",
"Predicting NER tags for line 183/215... Done\n",
"Predicting NER tags for line 184/215... Done\n",
"Predicting NER tags for line 185/215... Done\n",
"Predicting NER tags for line 186/215... Done\n",
"Predicting NER tags for line 187/215... Done\n",
"Predicting NER tags for line 188/215... Done\n",
"Predicting NER tags for line 189/215... Done\n",
"Predicting NER tags for line 190/215... Done\n",
"Predicting NER tags for line 191/215... Done\n",
"Predicting NER tags for line 192/215... Done\n",
"Predicting NER tags for line 193/215... Done\n",
"Predicting NER tags for line 194/215... Done\n",
"Predicting NER tags for line 195/215... Done\n",
"Predicting NER tags for line 196/215... Done\n",
"Predicting NER tags for line 197/215... Done\n",
"Predicting NER tags for line 198/215... Done\n",
"Predicting NER tags for line 199/215... Done\n",
"Predicting NER tags for line 200/215... Done\n",
"Predicting NER tags for line 201/215... Done\n",
"Predicting NER tags for line 202/215... Done\n",
"Predicting NER tags for line 203/215... Done\n",
"Predicting NER tags for line 204/215... Done\n",
"Predicting NER tags for line 205/215... Done\n",
"Predicting NER tags for line 206/215... Done\n",
"Predicting NER tags for line 207/215... Done\n",
"Predicting NER tags for line 208/215... Done\n",
"Predicting NER tags for line 209/215... Done\n",
"Predicting NER tags for line 210/215... Done\n",
"Predicting NER tags for line 211/215... Done\n",
"Predicting NER tags for line 212/215... Done\n",
"Predicting NER tags for line 213/215... Done\n",
"Predicting NER tags for line 214/215... Done\n",
"Predicting NER tags for line 215/215... Done\n",
"\n",
"Prediction for test-A data\n",
"Predicting NER tags for line 1/230... Done\n",
"Predicting NER tags for line 2/230... Done\n",
"Predicting NER tags for line 3/230... Done\n",
"Predicting NER tags for line 4/230... Done\n",
"Predicting NER tags for line 5/230... Done\n",
"Predicting NER tags for line 6/230... Done\n",
"Predicting NER tags for line 7/230... Done\n",
"Predicting NER tags for line 8/230... Done\n",
"Predicting NER tags for line 9/230... Done\n",
"Predicting NER tags for line 10/230... Done\n",
"Predicting NER tags for line 11/230... Done\n",
"Predicting NER tags for line 12/230... Done\n",
"Predicting NER tags for line 13/230... Done\n",
"Predicting NER tags for line 14/230... Done\n",
"Predicting NER tags for line 15/230... Done\n",
"Predicting NER tags for line 16/230... Done\n",
"Predicting NER tags for line 17/230... Done\n",
"Predicting NER tags for line 18/230... Done\n",
"Predicting NER tags for line 19/230... Done\n",
"Predicting NER tags for line 20/230... Done\n",
"Predicting NER tags for line 21/230... Done\n",
"Predicting NER tags for line 22/230... Done\n",
"Predicting NER tags for line 23/230... Done\n",
"Predicting NER tags for line 24/230... Done\n",
"Predicting NER tags for line 25/230... Done\n",
"Predicting NER tags for line 26/230... Done\n",
"Predicting NER tags for line 27/230... Done\n",
"Predicting NER tags for line 28/230... Done\n",
"Predicting NER tags for line 29/230... Done\n",
"Predicting NER tags for line 30/230... Done\n",
"Predicting NER tags for line 31/230... Done\n",
"Predicting NER tags for line 32/230... Done\n",
"Predicting NER tags for line 33/230... Done\n",
"Predicting NER tags for line 34/230... Done\n",
"Predicting NER tags for line 35/230... Done\n",
"Predicting NER tags for line 36/230... Done\n",
"Predicting NER tags for line 37/230... Done\n",
"Predicting NER tags for line 38/230... Done\n",
"Predicting NER tags for line 39/230... Done\n",
"Predicting NER tags for line 40/230... Done\n",
"Predicting NER tags for line 41/230... Done\n",
"Predicting NER tags for line 42/230... Done\n",
"Predicting NER tags for line 43/230... Done\n",
"Predicting NER tags for line 44/230... Done\n",
"Predicting NER tags for line 45/230... Done\n",
"Predicting NER tags for line 46/230... Done\n",
"Predicting NER tags for line 47/230... Done\n",
"Predicting NER tags for line 48/230... Done\n",
"Predicting NER tags for line 49/230... Done\n",
"Predicting NER tags for line 50/230... Done\n",
"Predicting NER tags for line 51/230... Done\n",
"Predicting NER tags for line 52/230... Done\n",
"Predicting NER tags for line 53/230... Done\n",
"Predicting NER tags for line 54/230... Done\n",
"Predicting NER tags for line 55/230... Done\n",
"Predicting NER tags for line 56/230... Done\n",
"Predicting NER tags for line 57/230... Done\n",
"Predicting NER tags for line 58/230... Done\n",
"Predicting NER tags for line 59/230... Done\n",
"Predicting NER tags for line 60/230... Done\n",
"Predicting NER tags for line 61/230... Done\n",
"Predicting NER tags for line 62/230... Done\n",
"Predicting NER tags for line 63/230... Done\n",
"Predicting NER tags for line 64/230... Done\n",
"Predicting NER tags for line 65/230... Done\n",
"Predicting NER tags for line 66/230... Done\n",
"Predicting NER tags for line 67/230... Done\n",
"Predicting NER tags for line 68/230... Done\n",
"Predicting NER tags for line 69/230... Done\n",
"Predicting NER tags for line 70/230... Done\n",
"Predicting NER tags for line 71/230... Done\n",
"Predicting NER tags for line 72/230... Done\n",
"Predicting NER tags for line 73/230... Done\n",
"Predicting NER tags for line 74/230... Done\n",
"Predicting NER tags for line 75/230... Done\n",
"Predicting NER tags for line 76/230... Done\n",
"Predicting NER tags for line 77/230... Done\n",
"Predicting NER tags for line 78/230... Done\n",
"Predicting NER tags for line 79/230... Done\n",
"Predicting NER tags for line 80/230... Done\n",
"Predicting NER tags for line 81/230... Done\n",
"Predicting NER tags for line 82/230... Done\n",
"Predicting NER tags for line 83/230... Done\n",
"Predicting NER tags for line 84/230... Done\n",
"Predicting NER tags for line 85/230... Done\n",
"Predicting NER tags for line 86/230... Done\n",
"Predicting NER tags for line 87/230... Done\n",
"Predicting NER tags for line 88/230... Done\n",
"Predicting NER tags for line 89/230... Done\n",
"Predicting NER tags for line 90/230... Done\n",
"Predicting NER tags for line 91/230... Done\n",
"Predicting NER tags for line 92/230... Done\n",
"Predicting NER tags for line 93/230... Done\n",
"Predicting NER tags for line 94/230... Done\n",
"Predicting NER tags for line 95/230... Done\n",
"Predicting NER tags for line 96/230... Done\n",
"Predicting NER tags for line 97/230... Done\n",
"Predicting NER tags for line 98/230... Done\n",
"Predicting NER tags for line 99/230... Done\n",
"Predicting NER tags for line 100/230... Done\n",
"Predicting NER tags for line 101/230... Done\n",
"Predicting NER tags for line 102/230... Done\n",
"Predicting NER tags for line 103/230... Done\n",
"Predicting NER tags for line 104/230... Done\n",
"Predicting NER tags for line 105/230... Done\n",
"Predicting NER tags for line 106/230... Done\n",
"Predicting NER tags for line 107/230... Done\n",
"Predicting NER tags for line 108/230... Done\n",
"Predicting NER tags for line 109/230... Done\n",
"Predicting NER tags for line 110/230... Done\n",
"Predicting NER tags for line 111/230... Done\n",
"Predicting NER tags for line 112/230... Done\n",
"Predicting NER tags for line 113/230... Done\n",
"Predicting NER tags for line 114/230... Done\n",
"Predicting NER tags for line 115/230... Done\n",
"Predicting NER tags for line 116/230... Done\n",
"Predicting NER tags for line 117/230... Done\n",
"Predicting NER tags for line 118/230... Done\n",
"Predicting NER tags for line 119/230... Done\n",
"Predicting NER tags for line 120/230... Done\n",
"Predicting NER tags for line 121/230... Done\n",
"Predicting NER tags for line 122/230... Done\n",
"Predicting NER tags for line 123/230... Done\n",
"Predicting NER tags for line 124/230... Done\n",
"Predicting NER tags for line 125/230... Done\n",
"Predicting NER tags for line 126/230... Done\n",
"Predicting NER tags for line 127/230... Done\n",
"Predicting NER tags for line 128/230... Done\n",
"Predicting NER tags for line 129/230... Done\n",
"Predicting NER tags for line 130/230... Done\n",
"Predicting NER tags for line 131/230... Done\n",
"Predicting NER tags for line 132/230... Done\n",
"Predicting NER tags for line 133/230... Done\n",
"Predicting NER tags for line 134/230... Done\n",
"Predicting NER tags for line 135/230... Done\n",
"Predicting NER tags for line 136/230... Done\n",
"Predicting NER tags for line 137/230... Done\n",
"Predicting NER tags for line 138/230... Done\n",
"Predicting NER tags for line 139/230... Done\n",
"Predicting NER tags for line 140/230... Done\n",
"Predicting NER tags for line 141/230... Done\n",
"Predicting NER tags for line 142/230... Done\n",
"Predicting NER tags for line 143/230... Done\n",
"Predicting NER tags for line 144/230... Done\n",
"Predicting NER tags for line 145/230... Done\n",
"Predicting NER tags for line 146/230... Done\n",
"Predicting NER tags for line 147/230... Done\n",
"Predicting NER tags for line 148/230... Done\n",
"Predicting NER tags for line 149/230... Done\n",
"Predicting NER tags for line 150/230... Done\n",
"Predicting NER tags for line 151/230... Done\n",
"Predicting NER tags for line 152/230... Done\n",
"Predicting NER tags for line 153/230... Done\n",
"Predicting NER tags for line 154/230... Done\n",
"Predicting NER tags for line 155/230... Done\n",
"Predicting NER tags for line 156/230... Done\n",
"Predicting NER tags for line 157/230... Done\n",
"Predicting NER tags for line 158/230... Done\n",
"Predicting NER tags for line 159/230... Done\n",
"Predicting NER tags for line 160/230... Done\n",
"Predicting NER tags for line 161/230... Done\n",
"Predicting NER tags for line 162/230... Done\n",
"Predicting NER tags for line 163/230... Done\n",
"Predicting NER tags for line 164/230... Done\n",
"Predicting NER tags for line 165/230... Done\n",
"Predicting NER tags for line 166/230... Done\n",
"Predicting NER tags for line 167/230... Done\n",
"Predicting NER tags for line 168/230... Done\n",
"Predicting NER tags for line 169/230... Done\n",
"Predicting NER tags for line 170/230... Done\n",
"Predicting NER tags for line 171/230... Done\n",
"Predicting NER tags for line 172/230... Done\n",
"Predicting NER tags for line 173/230... Done\n",
"Predicting NER tags for line 174/230... Done\n",
"Predicting NER tags for line 175/230... Done\n",
"Predicting NER tags for line 176/230... Done\n",
"Predicting NER tags for line 177/230... Done\n",
"Predicting NER tags for line 178/230... Done\n",
"Predicting NER tags for line 179/230... Done\n",
"Predicting NER tags for line 180/230... Done\n",
"Predicting NER tags for line 181/230... Done\n",
"Predicting NER tags for line 182/230... Done\n",
"Predicting NER tags for line 183/230... Done\n",
"Predicting NER tags for line 184/230... Done\n",
"Predicting NER tags for line 185/230... Done\n",
"Predicting NER tags for line 186/230... Done\n",
"Predicting NER tags for line 187/230... Done\n",
"Predicting NER tags for line 188/230... Done\n",
"Predicting NER tags for line 189/230... Done\n",
"Predicting NER tags for line 190/230... Done\n",
"Predicting NER tags for line 191/230... Done\n",
"Predicting NER tags for line 192/230... Done\n",
"Predicting NER tags for line 193/230... Done\n",
"Predicting NER tags for line 194/230... Done\n",
"Predicting NER tags for line 195/230... Done\n",
"Predicting NER tags for line 196/230... Done\n",
"Predicting NER tags for line 197/230... Done\n",
"Predicting NER tags for line 198/230... Done\n",
"Predicting NER tags for line 199/230... Done\n",
"Predicting NER tags for line 200/230... Done\n",
"Predicting NER tags for line 201/230... Done\n",
"Predicting NER tags for line 202/230... Done\n",
"Predicting NER tags for line 203/230... Done\n",
"Predicting NER tags for line 204/230... Done\n",
"Predicting NER tags for line 205/230... Done\n",
"Predicting NER tags for line 206/230... Done\n",
"Predicting NER tags for line 207/230... Done\n",
"Predicting NER tags for line 208/230... Done\n",
"Predicting NER tags for line 209/230... Done\n",
"Predicting NER tags for line 210/230... Done\n",
"Predicting NER tags for line 211/230... Done\n",
"Predicting NER tags for line 212/230... Done\n",
"Predicting NER tags for line 213/230... Done\n",
"Predicting NER tags for line 214/230... Done\n",
"Predicting NER tags for line 215/230... Done\n",
"Predicting NER tags for line 216/230... Done\n",
"Predicting NER tags for line 217/230... Done\n",
"Predicting NER tags for line 218/230... Done\n",
"Predicting NER tags for line 219/230... Done\n",
"Predicting NER tags for line 220/230... Done\n",
"Predicting NER tags for line 221/230... Done\n",
"Predicting NER tags for line 222/230... Done\n",
"Predicting NER tags for line 223/230... Done\n",
"Predicting NER tags for line 224/230... Done\n",
"Predicting NER tags for line 225/230... Done\n",
"Predicting NER tags for line 226/230... Done\n",
"Predicting NER tags for line 227/230... Done\n",
"Predicting NER tags for line 228/230... Done\n",
"Predicting NER tags for line 229/230... Done\n",
"Predicting NER tags for line 230/230... Done\n"
]
}
],
"source": [
"print(\"Prediction for dev-0 data\")\n",
"dev_0_labels = predict_ner_tags(dev_0_data[\"x\"])\n",
"\n",
"print()\n",
"\n",
"print(\"Prediction for test-A data\")\n",
"test_A_labels = predict_ner_tags(test_A_data[\"x\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Zapis wyników do plików"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"save_predictions(dev_0_labels, \"dev-0/out.tsv\")\n",
"save_predictions(test_A_labels, \"test-A/out.tsv\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Copy Permalink