DST labeling issue (discussion)

[zqwerty]

Since many participants concern the labeling criteria of the DST task, we open this issue to facilitate discussion.

1. Please consult the original papers first (MultiWOZ, MultiWOZ 2.1, CrossWOZ). And provided ontology based on data ontology-data.json.
2. You can correct the wrong annotations if you want.
3. I will post some FAQs below

[zqwerty]

Q: In slot “hotel-type”, whether label value “hotel” is labeled as soon as the word “hotel” appears in utterance? What’s label value about slot “hotel-type” in the following cases:

I need a hotel starting Thursday for 3 nights for 2 people . “hotel-type-?”
I am looking for a 3 star hotel in Cambridge with free Wifi. “hotel-type-?”
I need a hotel with Wifi. “hotel-type-?”

A: hotel is a special type of hotel. see database

Q: when to add the name of the hotel/restaurant/attraction to the dialogue state?

A: When we collect and annotate the test set, we use the following the annotation criteria in general:

1. If the user side specifically mentions the hotel name, it will be annotated in the dialog state.
2. If the agent side says the hotel's name, we also annotate it except for the following cases:
   • The agent provides multiple choices, and the user is not sure about the choice.
   • The user adds constraints that make the current hotel invalid, then remove the hotel name annotation from the dialog state.

However, due to the complexity of data collection and annotation, you still might see some noise in the test set. We are trying our best to make it as consistent as possible.

Following cases:

1. If all the attributes, except name, only exist in candidates provided by system cannot be added into Dialog State?

Yes. dialog state mostly logs user constraints.

1. Should the candidate’s name be added into Dialog State when user say yes to one of candidates provided by system？
2. Should the candidate’s name be added into Dialog State when user wants more information(pricerange, with wifi,etc.) about one of candidates provided by system？

Yes

1. User clearly stated that he did not want to book the hotel.
2. User has a clear intention to book the hotel.
3. System recommends a hotel, and user asks the detail infomation of it further, but doesn't have a clear intention to book the hotel.
4. System recommends a hotel, and user appreciates the hotel, but doesn't have a clear intention to book the hotel.
5. System clearly stated that he has booked the hotel for user.

book is not necessarily required for annotating ‘hotel-name’. For these five cases, most likely, ‘hotel-name’ should be annotated in the test set.

Q: In some samples, some slots are labeled by values about the booked entity which mentioned only by system-role after ‘book is done’(MUL2337) and in other samples are not(MUL1334).

A: It should not be annotated in the dialog state.

Q: In CrossWOZ, the price-range may have multiple valid labels.

Example:

dialogue id : 10929
turn id : 2
usr: "In the evening when colleagues get together, some colleagues want to eat Small sesame cake and Stir-fried fermented mung bean juice dried. I recommend a restaurant with a per capita consumption of 100-150 yuan."
sys: "No one has such a high per capita, I recommend you to go to Dongxing Shun Burst Belly Zhang, only 57 yuan per capita."
In current turn, the label sys_state_init is "100-150 yuan". But in the next turn, the label is "150 yuan or less"

A: The labels are created by humans to query the database. Some of the labels may not fully reflect the user constraints, since the system side annotator may relax some constraints. You can also correct the label if you want.

Q: We find that the label in current turn is deleted in the next turn

Examples are as follows:
Case1：
In this case the label ‘旅馆-互联网-不在意’ should be add to MUL1003-5.

    MUL1003-4:
      SYS: 艾伦拜尔旅馆怎么样？是4星级的，价格便宜。
      USER：位置是在东部吗？我改主意了，可以的话给我找一家带wifi的酒店吧。
      label： [
                "旅馆-互联网-有",
                "旅馆-价格范围-便宜的",
                "旅馆-区域-东方",
                "景点-名称-剑桥艺术品"
                ]
   MUL1003-5:
      SYS:  这边显示东部地区有3个宾馆类型的地方有空房，全部都带有WIFI。需要我帮您预订一家吗？
      USER: 不好意思，我需要是便宜的酒店而不是宾馆。在东部的，不需要互联网。
      label:  [
                "旅馆-价格范围-便宜的",
                "旅馆-区域-东方",
                "景点-名称-剑桥艺术品"
                 ]

CASE2:
In this case the label ‘旅馆-类型-酒店’ should be add to MUL1003-12.

   MUL1003-11
      SYS：预订成功了，参考编号是Z1WWH4AH。
      USER: 太糟糕了。你可以取消吗？我需要酒店。
            label : [
                "旅馆-互联网-有",
                "旅馆-价格范围-便宜的",
                "旅馆-区域-东方",
                "旅馆-类型-酒店”,
                "旅馆-预订人数-1",
                "旅馆-预订停留天数-2",
                "旅馆-预订日期-星期三",
                "景点-名称-剑桥艺术品"
            ]
   MUL1003-12
      SYS: 您还是要在剑桥镇的同一片区域找酒店吗？
      USER: 我们重新开始哈 。我想要的地方是这样的：便宜，在东边，有wifi。从周三起预订4晚，一个人住。
       label:[
                "旅馆-价格范围-便宜的",
                "旅馆-区域-东方",
                "旅馆-预订人数-1",
                "旅馆-预订停留天数-4",
                "旅馆-预订日期-星期三",
                "景点-名称-剑桥艺术品"
            ]

CASE3:
In the case the label ‘旅馆-星级-4’ should be add to PMUL2975-5, since user not change this label directly.

    PMUL2975-3
    SYS: 剑桥艺术品位于格林路5号。
    USER: 我还想找一家贵点儿的4星级宾馆入住。
            [
                "旅馆-价格范围-昂贵的",
                "旅馆-星级-4",
                "景点-区域-东方",
                "景点-类型-博物馆"
            ]

     PMUL2975-5
     SYS: 东部没有可为您推荐的贵点儿的四星级酒店。您是否可以考虑换个区域或者价位呢？
     USER: 那么，有哪些便宜的酒店呢？
     [
                "旅馆-价格范围-便宜的",
                "旅馆-区域-东方",
                "景点-区域-东方",
                "景点-类型-博物馆"
            ]

A: I agree with your labeling criteria. MultiWOZ dataset has a noisy annotation, which results in MultiWOZ 2.1, 2.2(You can also use this data for training). When collecting the testset, we try to avoid these noisy annotations(some constraints are wrongly deleted). However, these cases are hard to detect automatically and some of them come from misunderstandings of users. Thus these errors could not be eliminated totally.

[zqwerty]

the name label in CrossWOZ may be absent (e.g. 10909 dialog in test.json in CrossWOZ), so we will also accept if your prediction in one of selectedResults. We will update the evaluation script to 1) try to match the prediction with names in selectedResults; 2) match some values that have the same meaning like "有" and "是的" ASAP