Re: Can't evaluate the BLEU score for single sentence pairs

Philipp Koehn <pkoehn <at> ...> writes:

> 
> Hi,
> 
> since the BLEU score is based on the geometric mean between 1 to 4-gram
> precision, this often fails for single sentences. (no 4-gram match means
> multiplying with 0).
> 
> One remedy is to add 1 to all precision counts, but this fix is not 
implemented
> in any of the standard BLEU scorers.
> 
> -phi
> 
> On Mon, Jul 9, 2012 at 10:27 PM, Loki Cheng <moonloki <at> ...> wrote:
> > Hi, all
> > I can't evaluate the BLEU score for single sentence pairs with
> > multi-bleu.perl. I don't know why? example just like below:
> >
> > Original Chinese sentence: 安理會 還 請 建設 和平 委員會 向 安理會 提供 咨詢
> 意見 。
> > English translation result: 安理會 還 請 建設 和平 委員會 提供 咨詢 意見 的
> 委員會 。
> >
> > error message: Use of uninitialized value in division (/) at
> > ./multi-bleu.perl line 116, <STDIN> line 1.
> >
> > So it's impossible to evaluate it?
> > Any suggestion will be appreciated.
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support <at> ...
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> 
> 
> 
> _______________________________________________
> Moses-support mailing list
> Moses-support <at> ...
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 

There is also a simple bug with this error message.  For single sentence pairs, 
it doesn't make sense to calculate the confidence interval by looking at 
samples.  

my $ecart=0;
my $sample=20;
my $s=($#HYP+1);
if ($s >= $sample) { # Don't bother calculating ecart for small files.
  for(my $n=0;$n<$sample;$n++) {
      my $min = ($n*$s)/$sample;
      my $max = (($n+1)*$s)/$sample;
      my $sc=calculate_bleu($min, $max);
      $sc=~/BLEU = (.*?),/;
      if (abs($1-$BLEU)>$ecart) { $ecart=abs($1-$BLEU); }
  }
  $ecart=sprintf("%.2f",$ecart);
  $SCORE=~s/(BLEU = .*?),/$1 +\/- $ecart,/;
}

For the two Chinese sentences,

BLEU = 45.65, 83.3/54.5/40.0/33.3 (BP=0.920, ratio=0.923, hyp_len=12, 
ref_len=13)

  John

_______________________________________________
Moses-support mailing list
Moses-support <at> mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support