自然语言处理评价指标 - PARENT

# PARENT: Precision And Recall of Entailed N-grams from the Table

来源：Google Research

GITHUB: language/language/table_text_eval at master · google-research/language (github.com)

ARIXV: https://arxiv.org/abs/1906.01081

前言：PARENT 基于 n-gram 模型

n 元语法 - 维基百科，自由的百科全书 (wikipedia.org)

文本连续出现的 n 个语词，通过 n 个语词出现的概率来推断语句的结构

当 n 分别为 1、2、3 时，又分别称为一元语法（unigram）、二元语法（bigram）与三元语法（trigram）

# PARENT 指标

We develop a new metric, PARENT (Precision And Recall of Entailed N-grams from the Table) (§3). When computing precision, PARENT effectively uses a union of the reference and the table, to reward correct information missing from the reference.

PARENT 依赖于 reference 以及 table

Figure 1: A table from the WikiBio dataset (right), its reference description and three hypothetical generated texts with scores assigned to them by automatic evaluation metrics. Text which cannot be inferred from the table is in red, and text which can be inferred but isn’t present in the reference is in green. PARENT is our proposed metric

对比传统的 BLEU、ROUGE 评分，PARENT 的优势在于考虑了 TABLE（上图右侧的评分），使得信息错误的评分不会高于信息正确的评分。

Table 的表达形式

Tables can be expressed as set of records T = ${r_k}^K_k$=1, where each record is a tuple (entity, attribute, value). When all the records are about the same entity, we can truncate the records to (attribute, value) pairs.

形式化描述

text $G$, reference $R$, Tables $T$

let $D_M$ = ${(T^i,R^i,G^i)}^N_{i=1}$ denote an evaluation set of tables, references and texts generated from a model $M$, and $R^i_n$ , $G^i_n$ denote the collection of n-grams of order $n$ in $R^i$ and $G^i$ respectively.

We use $\#_{R^i_n} (g)$ to denote the count of n-gram $g$ in $R^i_n$ , and $\#_{G^i_ n,R^i_n}(g)$ to denote the minimum of its counts in $R^i_n$ and $G^i_n$. Our goal is to assign a score to the model, which correlates highly with human judgments of the quality of that model.