Relational Strategies in Customer Service (RSiCS) Dataset

Human-computer data from three live customer service Intelligent Virtual Agents (IVAs) in the domains of travel and telecommunications were collected, and annotators marked all text that was deemed unnecessary to the determination of user intention. After merging the selections of multiple annotators to create highlighted texts, a second round of annotation was performed to determine the classes of language present in the highlighted sections such as the presence of Greetings, Backstory, Justification, Gratitude, Rants, or Emotions. This resulting corpus is a valuable resource for improving the quality and relational abilities of IVAs.

Presentation

This research was presented at the 2018 O'Reilly Artificial Intelligence conference in NYC. A video of the full presentation is available on YouTube here

This research was published in AAAI 2020 and made available here

Data

Data was collected from four sources. The conversation logs of three commercial customer service IVAs and the Airline forums on TripAdvisor.com during August 2016.

Dataset numbering used in files:

  1. TripAdvisor.com airline forum
  2. Train travel IVA
  3. Airline travel IVA
  4. Telecommunications support IVA

Sanitation

The commercial data was sanitized to protect the identity of the companies and their customers. No sanitation was needed for the TripAdvisor data as it is already publicly viewable on the forum.

Personal Identifiable Information

Personal Identifiable Information (PII) present in the commercial datasets have been removed by a manual two-pass review. All numerical characters contained in PII such as telephone numbers, account numbers, confirmation or case numbers, etc have been replaced with #‘s. Similarly, all other characters contained in PII such as names and addresses were replaced with -‘s. The original length of the requests is therefore preserved as this may be a useful feature in analysis.

Example:

I cant sign into my account 1234 with my user name johndoe

becomes

I cant sign into my account #### with my user name -------

Company Origin

In order to publicly release the commercial data we agreed to anonymize it’s source. Company names were replaced with cname-* and product names with pname-*.

Example:

Can I sign up for an Airline clubname perks card?

becomes

Can I sign up for an cname-- pname--- perks card?

Re-identification

While it may be possible to deduce PII or the company of origin for a dataset, doing so violates the terms of use for this data. By downloading or using this dataset for any reason you agree to not attempt any type of re-identification.

Data Annotation

The instructions given to the eight annotators and details of the annotation process is available here

Annotation process

File Contents and Formatting

x_y_align.csv

Alignment of annotator x to all other annotators in their group for dataset y.

Columns:

all_data_by_threshold.csv

All requests with selections merged by threshold. Each request is repeated 4 times, once for each merging threshold.

Columns:

tagged_selections_by_sentence.csv

Second annotation pass tagging relational language present in selections made by first pass of annotation. Only contains requests in all_data_by_threshold.csv not marked as MultiIntent.

Columns:

all_multi_intent.csv

All requests flagged as containing multiple intentions by at least one annotator. Useful for developing multiple intent detection strategies.

Columns:

Terms of Use

This RSiCS dataset is made freely and publicly available under the Open Database License with the additional conditions:

Download

By downloading this dataset you agree to the Terms of Use declared above.

rsics_dataset.tar.gz