You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* remove WIP
* re-structure the README + re-center it on usage rather prompts creation
* remove link to api documentation
* pass on CONTRIBUITING
* api doc
* grammarly
* `get_fixed_answer_choices_list` is redundant
* fix unused import
* adress comments
* citation
PromptSource implements 4 classes to store, manipulate and use prompts and their metadata: `Template`, `Metadata`, `DatasetTemplates` and `TemplateCollection`. All of them are implemented in [`templates.py`](promptsource/templates.py)
3
+
4
+
## Class `Template` and `Metadata`
5
+
`Template` is a class that wraps a prompt, its associated metadata, and implements the helper functions to use the prompt.
6
+
7
+
Instances of `Template` have the following main methods that will come handy:
8
+
*`apply(example, truncate=True, highlight_variables=False)`: Create a prompted example by applying the template to the given example
9
+
-`example` (Dict): the dataset example to create a prompt for
10
+
-`truncate` (Bool, default to `True`): if True, example fields will be truncated to `TEXT_VAR_LENGTH` chars
11
+
-`highlight_variables`(Bool, default to `False`): highlight the added variables (internal use for the app rendering)
12
+
*`get_id()`: Get the uuid of the prompt
13
+
*`get_name()`: Get the name of the prompt
14
+
*`get_reference()`: Get any additional information about the prompt (such as bibliographic reference)
15
+
*`get_answer_choices_list(example)`: If applicable, returns a list of answer choices for a given example.
16
+
17
+
Each `Template` also has a `metadata` attribute, an instance of the class `Metadata` that encapsulates the following 3 attributes:
18
+
*`original_task`: If True, this prompt asks a model to perform the original task designed for this dataset.
19
+
*`choices_in_prompt`: If True, the answer choices are included in the templates such that models see those choices in the input. Only applicable to classification tasks.
20
+
*`metrics`: List of strings denoting metrics to use for evaluation
21
+
22
+
## Class `DatasetTemplates`
23
+
`DatasetTemplates` is a class that wraps all the prompts (each of them are instances of `Template`) for a specific dataset/subset and implements all the helper functions necessary to read/write to the YAML file in which the prompts are saved.
24
+
25
+
You will likely mainly be interested in getting the existing prompts and their names for a given dataset. You can do that with the following instantiation:
>>>len(prompts) # Returns the number of prompts for the given dataset
30
+
>>> prompts.all_template_names # Returns a sorted list of all templates names for this dataset
31
+
```
32
+
33
+
## Class `TemplateCollection`
34
+
`TemplateCollection` is a class that encapsulates all the prompts available under PromptSource by wrapping the `DatasetTemplates` class. It initializes the `DatasetTemplates` for all existing template folders, gives access to each `DatasetTemplates`, and provides aggregated counts overall `DatasetTemplates`.
35
+
36
+
The main methods are:
37
+
*`get_dataset(dataset_name, subset_name)`: Return the DatasetTemplates object corresponding to the dataset name
38
+
-`dataset_name` (Str): name of the dataset to get
39
+
-`subset_name` (Str, default to None): name of the subset
40
+
*`get_templates_count()`: Return the overall number count over all datasets. NB: we don't breakdown datasets into subsets for the count, i.e subsets count are included into the dataset count
Copy file name to clipboardExpand all lines: CONTRIBUTING.md
+32-28Lines changed: 32 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
# Contributing
2
2
3
-
One of the best ways to contribute is by writing prompts!
3
+
The best way to contribute growing P3 is by writing prompts for new datasets!
4
4
5
5
### What are Prompts?
6
6
7
-
A prompt consists of a template(input template and target template, along with collection of associated metadata. A template is a piece of code written in a templating language called
7
+
A prompt consists of a template: input template and target template, along with collection of associated metadata. A template is a piece of code written in a templating language called
8
8
[Jinja](https://jinja.palletsprojects.com/en/3.0.x/). A template defines
9
9
a function that maps an example from a dataset in the
10
10
[Hugging Face datasets library](https://huggingface.co/datasets) to two strings of
@@ -17,7 +17,7 @@ prompt.
17
17
18
18
1.**Set up the app.** Fork the app and set up using the
For datasets that have splits with no labels (for instance test split without ground truth labels), you can wrap the conditional statement on the target side.
186
+
For instance for `super_glue/boolq`, the following prompt would return an empty target on the test split, but not an empty prompted example:
187
+
```jinja2
188
+
{{ passage }}
189
+
Question: {{ question }}
190
+
Answer:
191
+
|||
192
+
{% if label != -1 %}
193
+
{{ answer_choices[label] }}
194
+
{% endif %}
195
+
```
183
196
***Conditional generation format.** Always specify the target and separate it from the prompt
184
197
by indicating the vertical bars `|||`. The target will be generated by a generative model
185
198
conditioned on the input you wrote. You can always transform an "infix" prompt format
0 commit comments