Skip to content

Commit 9f0e742

Browse files
ML Fairness Inframl-fairness-infra-github
ML Fairness Infra
authored and
ml-fairness-infra-github
committed
internal
PiperOrigin-RevId: 348511463
1 parent f3d1769 commit 9f0e742

File tree

1 file changed

+328
-0
lines changed

1 file changed

+328
-0
lines changed

‎fairness_indicators/documentation/tutorials/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "_E4uORykIpG4"
7+
},
8+
"source": [
9+
"##### Copyright 2020 The TensorFlow Authors."
10+
]
11+
},
12+
{
13+
"cell_type": "code",
14+
"execution_count": null,
15+
"metadata": {
16+
"cellView": "form",
17+
"id": "aBT221yVIujn"
18+
},
19+
"outputs": [],
20+
"source": [
21+
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
22+
"# you may not use this file except in compliance with the License.\n",
23+
"# You may obtain a copy of the License at\n",
24+
"#\n",
25+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
26+
"#\n",
27+
"# Unless required by applicable law or agreed to in writing, software\n",
28+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
29+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
30+
"# See the License for the specific language governing permissions and\n",
31+
"# limitations under the License."
32+
]
33+
},
34+
{
35+
"cell_type": "markdown",
36+
"metadata": {
37+
"id": "aalPefrUUplk"
38+
},
39+
"source": [
40+
"# Fairness Indicators TensorBoard Plugin Example Colab"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {
46+
"id": "fFTJpyFlI-uI"
47+
},
48+
"source": [
49+
"\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
50+
" \u003ctd\u003e\n",
51+
" \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/responsible_ai/fairness_indicators/tutorials/Fairness_indicators_TensorBoard_Plugin_Example_Colab\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
52+
" \u003c/td\u003e\n",
53+
" \u003ctd\u003e\n",
54+
" \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/fairness-indicators/blob/master/fairness_indicators/documentation/examples/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
55+
" \u003c/td\u003e\n",
56+
" \u003ctd\u003e\n",
57+
" \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/fairness-indicators/blob/master/fairness_indicators/documentation/examples/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
58+
" \u003c/td\u003e\n",
59+
" \u003ctd\u003e\n",
60+
" \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/fairness-indicators/fairness_indicators/documentation/examples/Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
61+
" \u003c/td\u003e\n",
62+
"\u003c/table\u003e"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {
68+
"id": "UZ48WFLwbCL6"
69+
},
70+
"source": [
71+
"##Overview\n",
72+
"\n",
73+
"In this activity, you'll use [Fairness Indicators for TensorBoard](https://github.com/tensorflow/tensorboard/tree/master/docs/fairness-indicators.md). With the plugin, you can visualize fairness evaluations for your runs and easily compare performance across groups.\n"
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"metadata": {
79+
"id": "u33JXdluZ2lG"
80+
},
81+
"source": [
82+
"# Importing\n",
83+
"\n",
84+
"Run the following code to install the required libraries."
85+
]
86+
},
87+
{
88+
"cell_type": "code",
89+
"execution_count": null,
90+
"metadata": {
91+
"id": "EoRNffG599XP"
92+
},
93+
"outputs": [],
94+
"source": [
95+
"!pip install fairness_indicators 'absl-py\u003c0.9,\u003e=0.7'\n",
96+
"!pip install google-api-python-client==1.8.3\n",
97+
"!pip install tensorboard-plugin-fairness-indicators\n",
98+
"!pip install tensorflow-serving-api==2.2.0c2"
99+
]
100+
},
101+
{
102+
"cell_type": "markdown",
103+
"metadata": {
104+
"id": "mglfaM4_mtIk"
105+
},
106+
"source": [
107+
"**Restart the runtime.** After the runtime is restarted, continue with following cells without running previous cell again."
108+
]
109+
},
110+
{
111+
"cell_type": "code",
112+
"execution_count": null,
113+
"metadata": {
114+
"id": "sFZJ8f_M7mlc"
115+
},
116+
"outputs": [],
117+
"source": [
118+
"# %tf.disable_v2_behavior()\t# Uncomment this line if running in Google Colab."
119+
]
120+
},
121+
{
122+
"cell_type": "code",
123+
"execution_count": null,
124+
"metadata": {
125+
"id": "B8dlyTyiTe-9"
126+
},
127+
"outputs": [],
128+
"source": [
129+
"import datetime\n",
130+
"import os\n",
131+
"import tempfile\n",
132+
"from tensorboard_plugin_fairness_indicators import summary_v2\n",
133+
"import tensorflow.compat.v1 as tf\n",
134+
"\n",
135+
"# example_model.py is provided in fairness_indicators package to train and\n",
136+
"# evaluate an example model. \n",
137+
"from fairness_indicators import example_model\n",
138+
"\n",
139+
"tf.compat.v1.enable_eager_execution()"
140+
]
141+
},
142+
{
143+
"cell_type": "markdown",
144+
"metadata": {
145+
"id": "TsplOJGqWCf5"
146+
},
147+
"source": [
148+
"# Data and Constants"
149+
]
150+
},
151+
{
152+
"cell_type": "code",
153+
"execution_count": null,
154+
"metadata": {
155+
"id": "NdLBi6tN5i7I"
156+
},
157+
"outputs": [],
158+
"source": [
159+
"# To know about dataset, check Fairness Indicators Example Colab at:\n",
160+
"# https://github.com/tensorflow/fairness-indicators/blob/master/fairness_indicators/documentation/examples/Fairness_Indicators_Example_Colab.ipynb\n",
161+
"\n",
162+
"train_tf_file = tf.keras.utils.get_file('train.tf', 'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')\n",
163+
"validate_tf_file = tf.keras.utils.get_file('validate.tf', 'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')\n",
164+
"\n",
165+
"BASE_DIR = tempfile.gettempdir()\n",
166+
"TEXT_FEATURE = 'comment_text'\n",
167+
"LABEL = 'toxicity'\n",
168+
"FEATURE_MAP = {\n",
169+
" # Label:\n",
170+
" LABEL: tf.io.FixedLenFeature([], tf.float32),\n",
171+
" # Text:\n",
172+
" TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),\n",
173+
"\n",
174+
" # Identities:\n",
175+
" 'sexual_orientation': tf.io.VarLenFeature(tf.string),\n",
176+
" 'gender': tf.io.VarLenFeature(tf.string),\n",
177+
" 'religion': tf.io.VarLenFeature(tf.string),\n",
178+
" 'race': tf.io.VarLenFeature(tf.string),\n",
179+
" 'disability': tf.io.VarLenFeature(tf.string),\n",
180+
"}"
181+
]
182+
},
183+
{
184+
"cell_type": "markdown",
185+
"metadata": {
186+
"id": "mfbgerCsEOmN"
187+
},
188+
"source": [
189+
"# Train the Model"
190+
]
191+
},
192+
{
193+
"cell_type": "code",
194+
"execution_count": null,
195+
"metadata": {
196+
"id": "YwoC-dzEDid3"
197+
},
198+
"outputs": [],
199+
"source": [
200+
"model_dir = os.path.join(BASE_DIR, 'train',\n",
201+
" datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))\n",
202+
"\n",
203+
"classifier = example_model.train_model(model_dir,\n",
204+
" train_tf_file,\n",
205+
" LABEL,\n",
206+
" TEXT_FEATURE,\n",
207+
" FEATURE_MAP)"
208+
]
209+
},
210+
{
211+
"cell_type": "markdown",
212+
"metadata": {
213+
"id": "jTPqije9Eg5b"
214+
},
215+
"source": [
216+
"# Run TensorFlow Model Analysis with Fairness Indicators\n",
217+
"This step might take 2 to 5 minutes."
218+
]
219+
},
220+
{
221+
"cell_type": "code",
222+
"execution_count": null,
223+
"metadata": {
224+
"id": "QLjiy5VCzlRw"
225+
},
226+
"outputs": [],
227+
"source": [
228+
"tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')\n",
229+
"\n",
230+
"example_model.evaluate_model(classifier,\n",
231+
" validate_tf_file,\n",
232+
" tfma_eval_result_path,\n",
233+
" 'gender',\n",
234+
" LABEL,\n",
235+
" FEATURE_MAP)"
236+
]
237+
},
238+
{
239+
"cell_type": "markdown",
240+
"metadata": {
241+
"id": "U1ROnulYc8Ub"
242+
},
243+
"source": [
244+
"# Visualize Fairness Indicators in TensorBoard\n",
245+
"\n",
246+
"\n",
247+
"Below you will visualize Fairness Indicators in Tensorboard and compare performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization. You can also select different evaluation runs using the drop down menu at the top-left corner."
248+
]
249+
},
250+
{
251+
"cell_type": "markdown",
252+
"metadata": {
253+
"id": "zCV-Jo0xda6g"
254+
},
255+
"source": [
256+
"## Write Fairness Indicators Summary\n",
257+
"Write summary file containing all required information to visualize Fairness Indicators in TensorBoard."
258+
]
259+
},
260+
{
261+
"cell_type": "code",
262+
"execution_count": null,
263+
"metadata": {
264+
"id": "JNaNhTCTAMHm"
265+
},
266+
"outputs": [],
267+
"source": [
268+
"import tensorflow.compat.v2 as tf2\n",
269+
"\n",
270+
"writer = tf2.summary.create_file_writer(\n",
271+
" os.path.join(model_dir, 'fairness_indicators'))\n",
272+
"with writer.as_default():\n",
273+
" summary_v2.FairnessIndicators(tfma_eval_result_path, step=1)\n",
274+
"writer.close()"
275+
]
276+
},
277+
{
278+
"cell_type": "markdown",
279+
"metadata": {
280+
"id": "MB2Gfm9BdXVY"
281+
},
282+
"source": [
283+
"## Launch TensorBoard\n",
284+
"Navigate to \"Fairness Indicators\" tab to visualize Fairness Indicators."
285+
]
286+
},
287+
{
288+
"cell_type": "code",
289+
"execution_count": null,
290+
"metadata": {
291+
"id": "UiHhDWu8tyEI"
292+
},
293+
"outputs": [],
294+
"source": [
295+
296+
"%load_ext tensorboard"
297+
]
298+
},
299+
{
300+
"cell_type": "code",
301+
"execution_count": null,
302+
"metadata": {
303+
"id": "ix6d718udWsK"
304+
},
305+
"outputs": [],
306+
"source": [
307+
308+
"%tensorboard --logdir=$model_dir"
309+
]
310+
}
311+
],
312+
"metadata": {
313+
"accelerator": "GPU",
314+
"colab": {
315+
"collapsed_sections": [],
316+
"name": "Fairness Indicators TensorBoard Plugin Example Colab.ipynb",
317+
"private_outputs": true,
318+
"provenance": [],
319+
"toc_visible": true
320+
},
321+
"kernelspec": {
322+
"display_name": "Python 3",
323+
"name": "python3"
324+
}
325+
},
326+
"nbformat": 4,
327+
"nbformat_minor": 0
328+
}

0 commit comments

Comments
 (0)