Skip to content

Commit

Permalink
Merge branch 'data-consistance-refine' of https://github.com/TomNong/…
Browse files Browse the repository at this point in the history 
…texar into data-consistance-refine
  • Loading branch information
Shibiao Nong committed Apr 5, 2019
2 parents 362599b + 79f9d86 commit 5cc5126
Showing 1 changed file with 4 additions and 14 deletions.
18 changes: 4 additions & 14 deletions texar/data/data/tfrecord_data.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,11 @@ class TFRecordData(DataBase):
#
# # 'image_raw' is a list of image data bytes in this
# # example.
# 'image_raw': ['...'],
# 'image_raw': [...],
# }
# }
...
# Read image data from TFRecord file and do resizing
hparams={
Expand Down Expand Up @@ -211,13 +211,11 @@ def default_hparams():
.. code-block:: python
...
feature_original_types = {
"input_ids": ["tf.int64", "FixedLenFeature", 128],
"label_ids": ["tf.int64", "FixedLenFeature"],
"name_lists": ["tf.string", "VarLenFeature"],
}
...
"feature_convert_types" : dict, optional
Specifies dtype converting after reading the data files. This
Expand All @@ -238,12 +236,10 @@ def default_hparams():
.. code-block:: python
...
feature_convert_types = {
"input_ids": "tf.int32",
"label_ids": "tf.int32",
}
...
"image_options" : dict, optional
Specifies the image feature name and performs image resizing,
Expand Down Expand Up @@ -277,27 +273,21 @@ def default_hparams():
.. code-block:: python
...
dataset: {
...
"num_shards": 2,
"shard_id": 0,
...
"shard_id": 0
}
...
For gpu 1:
.. code-block:: python
...
dataset: {
...
"num_shards": 2,
"shard_id": 1,
...
"shard_id": 1
}
...
Also refer to `examples/bert` for a use case.
Expand Down

0 comments on commit 5cc5126

Please sign in to comment.