frameworks/tensorflow2/TF2_AddDatasets_Splits.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"TF2_AddDatasets_Splits.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"code","metadata":{"id":"ERLUqyk08nvq","colab_type":"code","colab":{}},"source":["!pip install tensorflow-gpu==2.0.0alpha tensorflow_datasets"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"8tblShDJ7TQz","colab_type":"code","colab":{}},"source":["import tensorflow as tf\n","import tensorflow_datasets as tfds"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"nd8FQpLK7XLo","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":35},"outputId":"3e7cf9a9-6888-4d94-a817-50a5bf3ec4e6","executionInfo":{"status":"ok","timestamp":1558937708951,"user_tz":-480,"elapsed":828,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"}}},"source":["print(\"Tensorflow version: {}\".format(tf.__version__))"],"execution_count":4,"outputs":[{"output_type":"stream","text":["Tensorflow version: 2.0.0-alpha0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"qnl-f6xBZHuK","colab_type":"text"},"source":["# TensorFlow Datasets Reference\n","\n","* https://www.tensorflow.org/datasets/overview"]},{"cell_type":"markdown","metadata":{"id":"Cum6uMfa7Bcj","colab_type":"text"},"source":["# Splits"]},{"cell_type":"markdown","metadata":{"id":"hTLHjTpD7Eki","colab_type":"text"},"source":["All `DatasetBuilder`s expose various data subsets defined as `tfds.Splits` (e.g. `tfds.Split.TRAIN` or `tfds.Split.TEST`). A given dataset's splits are defined in `tfds.DatasetBuilder.info.splits`. They are accessiable via `tfds.load` and `tfds.DatasetBuilder.as_dataset`. Both of them take the parameter (`split=`) as the input."]},{"cell_type":"markdown","metadata":{"id":"mLfy0UYBBavo","colab_type":"text"},"source":["## Using splits"]},{"cell_type":"code","metadata":{"id":"DURbIB_s7DEC","colab_type":"code","colab":{}},"source":["combined_split = tfds.Split.TRAIN + tfds.Split.TEST"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"gbPYY0qs8UkS","colab_type":"code","colab":{}},"source":["combined_ds = tfds.load(\"mnist\", split=combined_split)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"4Bgb3dUu_oTu","colab_type":"text"},"source":["You can also merge all splits and load the whole dataset."]},{"cell_type":"code","metadata":{"id":"4ePoGSaG_n6k","colab_type":"code","colab":{}},"source":["combined_ds_all = tfds.load(\"mnist\", split=tfds.Split.ALL)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zt_6lJXPAOD9","colab_type":"text"},"source":["One thing reminds that options for the argument (`split=`) lead the dataset merged together and being returned. It is necessary to split the dataset during training."]},{"cell_type":"code","metadata":{"id":"QZWe2LxXBVr1","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":35},"outputId":"f7eef9ab-0d20-4ed1-d013-10f9a3d653b5","executionInfo":{"status":"ok","timestamp":1558939265125,"user_tz":-480,"elapsed":877,"user":{"displayName":"王DevOps","photoUrl":"https://lh6.googleusercontent.com/-DzgWpvOQ_c0/AAAAAAAAAAI/AAAAAAAAA8I/fX9gQd_TAjU/s64/photo.jpg","userId":"04300517850278510646"}}},"source":["combined_ds"],"execution_count":12,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<_OptionsDataset shapes: {image: (28, 28, 1), label: ()}, types: {image: tf.uint8, label: tf.int64}>"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"markdown","metadata":{"id":"0wsVBJ8YBfOW","colab_type":"text"},"source":["## Using subsplits"]},{"cell_type":"markdown","metadata":{"id":"LsTGisgKBvJT","colab_type":"text"},"source":["You can further split the dataset into customized splits via `tfds.Split.subsplit`. \n","\n","There are three options for you to split the dataset.\n","* specify number of subsplits\n","* specify a percentage slice\n","* specify weights\n","\n","There are sveral **warnings** for you while using the api `tfds.Split.subsplit`.\n","* TFDS does not currently guarantee the order of the data on the disk. If you regenerate the data, the subsplits may no longer be the same.\n","* If the total number of examples can't be devided by 100, the remainder examples may not be evenly distributed among subsplits.\n","\n","```\n","# 此内容为代码格式\n","```\n","\n"]},{"cell_type":"markdown","metadata":{"id":"9mrYGGZjDAIe","colab_type":"text"},"source":["### Specify number of subsplits"]},{"cell_type":"code","metadata":{"id":"5BJUCUMTBjlN","colab_type":"code","colab":{}},"source":["train_half_1, train_half_2 = tfds.Split.TRAIN.subsplit(2)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"nW0ZbDfHDPLS","colab_type":"code","colab":{}},"source":["train_half_dataset = tfds.load(\"mnist\", split=train_half_1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"F7hPVr1aET-x","colab_type":"text"},"source":["### Specify a percentage slice\n","\n","Here we use an another api `tfds.percent[<beginning in percentage>:<ending in percentage>]` to split the dataset."]},{"cell_type":"code","metadata":{"id":"PewC2BrSDY8s","colab_type":"code","colab":{}},"source":["first_10_percent = tfds.Split.TRAIN.subsplit(tfds.percent[:10])\n","last_2_percent = tfds.Split.TRAIN.subsplit(tfds.percent[-2:])\n","middle_50_percent = tfds.Split.TRAIN.subsplit(tfds.percent[25:75])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"31SEBBMvE9nF","colab_type":"code","colab":{}},"source":["middle_50_dataset = tfds.load(\"mnist\", split=middle_50_percent)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"UH6kaovPFGe5","colab_type":"text"},"source":["### Specify weights"]},{"cell_type":"code","metadata":{"id":"5vLZNWSEFHCG","colab_type":"code","colab":{}},"source":["half, quarter1, quarter2 = tfds.Split.TRAIN.subsplit([2, 1, 1])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"aQ1uCWJbFRR7","colab_type":"code","colab":{}},"source":["half_dataset = tfds.load(\"mnist\", split=half)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nIOtnKgDGIAm","colab_type":"text"},"source":["## Composing split, adding, and subsplitting"]},{"cell_type":"markdown","metadata":{"id":"zFes_MpAGRgF","colab_type":"text"},"source":["It is possible to compose the above operations."]},{"cell_type":"code","metadata":{"id":"lNns6EE_GRCI","colab_type":"code","colab":{}},"source":["# merge the half of TRAIN split and the TEST\n","split = tfds.Split.TRAIN.subsplit(tfds.percent[25:75]) + tfds.Split.TEST\n","\n","\n","# merge TRAIN and TEST Split and then split into 2 parts\n","first_half, second_half = (tfds.Split.TRAIN + tfds.Split.TEST).subsplit(2)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"agsxjGhyGHlh","colab_type":"code","colab":{}},"source":["first_half_ds = tfds.load(\"mnist\", split=first_half)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qD8E3J2fHKQm","colab_type":"text"},"source":["Notice that a split can not be added twice, the operation `subsplit` can only happen once.\n","\n","```python\n","#\n","# WRONG\n","# Overlap between splits. Split {'train'} has been added with itself.\n","#\n","wrong_split = tfds.Split.TRAIN.subsplit(tfds.percent[:25]) + tfds.Split.TRAIN\n","tfds.load(\"mnist\", split=wrong_split)\n","\n","#\n","# WRONG\n","# Trying to slice Split train which has already been sliced\n","#\n","wrong_split1, worng_split2 = tfds.Split.TRAIN.subsplit(tfds.percent[:25]).subsplit(2)\n","tfds.load(\"mnist\", split=wrong_split1)\n","\n","#\n","# WRONG\n","# Trying to slice Split train which has already been sliced\n","#\n","wrong_split = (tfds.Split.TRAIN.subsplit(tfds.percent[:25]) + \\\n","               tfds.Split.TEST.subsplit(tfds.percent[:25])).subsplit(tfds.percent[:50])\n","tfds.load(\"mnist\", split=wrong_split)\n","```"]},{"cell_type":"markdown","metadata":{"id":"abP7RWtAMYOd","colab_type":"text"},"source":["## Dataset using non-conventional named split\n","\n","For dataset using splits not in conventional names (`tfds.Split.{TRAIN, TEST, VALIDATION}`), you can still use the split api given the customized name that is like `tfds.Split('customized_split')`."]},{"cell_type":"code","metadata":{"id":"OakEhleXM8BX","colab_type":"code","colab":{}},"source":["coco2014_split = tfds.Split(\"test2015\") + tfds.Split.TEST\n","ds = tfds.load(\"coco2014\", split=coco2014_split)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-kbaEBkT6xra","colab_type":"text"},"source":["# Dataset Builder"]},{"cell_type":"markdown","metadata":{"id":"5ZHC7tnq6tmz","colab_type":"text"},"source":["\n","You can build your own data as a Tensorflow dataset via `DatasetBuilder`. There are several sequential steps:\n","* Writing `my_dataset.py`\n","* Specify `DatasetInfo`\n","* Downloading and extracting source data\n","* Specify dataset splits\n","* Writing an example generator\n","* Dataset configuration\n","* Create your own `FeatureConnector`\n","* Adding the dataset to `tensorflow/datasets` (optional)\n","* Large datasets and distributed generation\n","* Testing `MyDataset`\n","\n","TDFS provides a way to transform datasets into a standard format, do the preprocessing to make they ready for a machine learning pipeline, and more importantly provides a standard input pipeline using `tf.data`. In order to enable the above, each dataset must implement a class of `DataBuilder`. The implementation specifies\n","* where the data is coming from (e.g. URL)\n","* what the dataset looks like (e.g. features)\n","* how the dataset should be splitted (e.g. TRAIN or TEST)"]},{"cell_type":"markdown","metadata":{"id":"czo9WxY4PYz5","colab_type":"text"},"source":["## Writing `my_dataset.py`"]},{"cell_type":"markdown","metadata":{"id":"ncC4EyvFPjib","colab_type":"text"},"source":["### Using the default template\n","\n","If you want to contribute to official tensorflow dataset repo and add a new dataset, it is recommaned for you to start from auto generating the required python scripts. After that, search `TODO(my_dataset)` to do some modifications."]},{"cell_type":"code","metadata":{"id":"MWOaLyEhXw_H","colab_type":"code","colab":{}},"source":["!git clone https://github.com/tensorflow/datasets.git\n","!mv datasets tensorflow_dataset\n","\n","!python ../tensorflow_dataset/tensorflow_datasets/scripts/create_new_dataset.py \\\n","  --dataset my_dataset \\\n","  --type image   # text, audio, translation, ..."],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hxSOIFMYXGPt","colab_type":"text"},"source":["A alternative way."]},{"cell_type":"code","metadata":{"id":"SkurDAEdXLb0","colab_type":"code","colab":{}},"source":["!pip install tensorflow_datasets\n","!python -m tensorflow_datasets.scripts.create_new_dataset --dataset dataset_name --type dataset_type"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"CcJ28jJAXrRe","colab_type":"text"},"source":["By default, two files would be generated (thay are `my_dataset.py` and `my_dataset_test.py`)."]},{"cell_type":"markdown","metadata":{"id":"N_wreddcPmjw","colab_type":"text"},"source":["### Starting from `DatasetBuilder`\n","\n","Each dataset is defained as a subclass (inherited) of `tfds.core.DataBuilder` implementing the necessary methods:\n","* `_info`\n","* `_download_and_prepare`\n","* `_as_dataset`"]},{"cell_type":"code","metadata":{"id":"nEC7Oy0bPr8H","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}