Merge pull request #83 from NervanaSystems/peter/documentation

Update documentation to reflect 0.5

.gitignore

@@ -8,14 +8,12 @@
 .styleenv
 .coverage
 flake8.txt
-/build/
+**/build/*
 generated
 /dist/
 *.hdf5
 *.h5
-#!nlp_architect/server/angular-ui/dist/angular-ui/*.html
-#*.html
-docs-sources/build/**
+docs-source/_build/**
 !nlp_architect/solutions/set_expansion/ui/templates/*.html
 .vscode
 !tests/fixtures/data/server/*.gz

README.md

@@ -41,24 +41,33 @@ Features:
 * Core NLP models used in many NLP tasks and useful in many NLP applications
 * Novel NLU models showcasing novel topologies and techniques
 * Optimized NLP/NLU models showcasing different optimization algorithms on neural NLP/NLU models
-* Simple REST API server ([doc](http://nlp_architect.nervanasys.com/service.html)):
-  * serving trained models (for inference)
-  * plug-in system for adding your own model
-* 4 Demos of models (pre-trained by us) showcasing NLP Architect (Dependency parser, NER, Intent Extraction, Q&A)
+* Model-oriented design:
+  * Train and run models from command-line.
+  * API for using models for inference in python.
+  * Procedures to define custom processes for training,    inference or anything related to processing.
+  * CLI sub-system for running procedures
 * Based on optimized Deep Learning frameworks:
 
   * [TensorFlow]
   * [PyTorch]
   * [Intel-Optimized TensorFlow with MKL-DNN]
   * [Dynet]
 
-* Documentation [website](http://nlp_architect.nervanasys.com/) and [tutorials](http://nlp_architect.nervanasys.com/tutorials.html)
 * Essential utilities for working with NLP models - Text/String pre-processing, IO, data-manipulation, metrics, embeddings.
+* Plug-able REST API server to serve models via REST API
 
 ## Installing NLP Architect
 
 We recommend to install NLP Architect in a new python environment, to use python 3.6+ with up-to-date `pip`, `setuptools` and `h5py`.
 
+### Install using `pip`
+
+Includes only core library (without `examples/` directory)
+
+```sh
+pip install nlp-architect
+```
+
 ### Install from source (Github)
 
 Includes core library and all content (example scripts, datasets, tutorials)
@@ -76,14 +85,6 @@ Install (in develop mode)
 pip install -e .
 ```
 
-### Install from pypi (using `pip install`)
-
-Includes only core library
-
-```sh
-pip install nlp-architect
-```
-
 ### Further installation options
 
 Refer to our full [installation instructions](http://nlp_architect.nervanasys.com/installation.html) page on our website for complete details on how to install NLP Architect and other backend installations such as MKL-DNN or GPU backends.
@@ -93,40 +94,30 @@ Users can install any deep learning backends manually before/after they install
 
 NLP models that provide best (or near) in class performance:
 
-* [Word chunking](http://nlp_architect.nervanasys.com/chunker.html)
-* [Named Entity Recognition](http://nlp_architect.nervanasys.com/ner_crf.html)
+* [Word chunking](http://nlp_architect.nervanasys.com/tagging/sequence_tagging.html#word-chunker)
+* [Named Entity Recognition](http://nlp_architect.nervanasys.com/tagging/sequence_tagging.html#named-entity-recognition)
 * [Dependency parsing](http://nlp_architect.nervanasys.com/bist_parser.html)
 * [Intent Extraction](http://nlp_architect.nervanasys.com/intent.html)
-* [Sentiment classification](http://nlp_architect.nervanasys.com/supervised_sentiment.html)
-* [Language models](http://nlp_architect.nervanasys.com/tcn.html)
-* [Transformers](http://nlp_architect.nervanasys.com/) (for most NLP tasks)
+* [Sentiment classification](http://nlp_architect.nervanasys.com/sentiment.html#supervised-sentiment)
+* [Language models](http://nlp_architect.nervanasys.com/lm.html#language-modeling-with-tcn)
+* [Transformers](http://nlp_architect.nervanasys.com/transformers.html) (for NLP tasks)
 
 Natural Language Understanding (NLU) models that address semantic understanding:
 
 * [Aspect Based Sentiment Analysis (ABSA)](http://nlp_architect.nervanasys.com/absa.html)
+* [Joint intent detection and slot tagging](http://nlp_architect.nervanasys.com/intent.html)
 * [Noun phrase embedding representation (NP2Vec)](http://nlp_architect.nervanasys.com/np2vec.html)
 * [Most common word sense detection](http://nlp_architect.nervanasys.com/word_sense.html)
 * [Relation identification](http://nlp_architect.nervanasys.com/identifying_semantic_relation.html)
 * [Cross document coreference](http://nlp_architect.nervanasys.com/cross_doc_coref.html)
 * [Noun phrase semantic segmentation](http://nlp_architect.nervanasys.com/np_segmentation.html)
 
-Components instrumental for conversational AI:
-
-* [Joint intent detection and slot tagging](http://nlp_architect.nervanasys.com/intent.html)
-* [Memory Networks for goal oriented dialog](http://nlp_architect.nervanasys.com/memn2n.html)
-
 Optimizing NLP/NLU models and misc. optimization techniques:
 
-* [Quantized BERT (8bit)](http://nlp_architect.nervanasys.com/)
-* [Knowledge Distillation using BERT](http://nlp_architect.nervanasys.com/)
+* [Quantized BERT (8bit)](http://nlp_architect.nervanasys.com/quantized_bert.html)
+* [Knowledge Distillation using Transformers](http://nlp_architect.nervanasys.com/transformers_distillation.html)
 * [Sparse and Quantized Neural Machine Translation (GNMT)](http://nlp_architect.nervanasys.com/sparse_gnmt.html)
 
-End-to-end Deep Learning-based NLP models:
-
-* [Reading comprehension](http://nlp_architect.nervanasys.com/reading_comprehension.html)
-* [Language Modeling using Temporal Convolution Network (TCN)](http://nlp_architect.nervanasys.com/tcn.html)
-* [Unsupervised Cross-lingual embeddings](http://nlp_architect.nervanasys.com/crosslingual_emb.html)
-
 Solutions (End-to-end applications) using one or more models:
 
 * [Term Set expansion](http://nlp_architect.nervanasys.com/term_set_expansion.html) - uses the included word chunker as a noun phrase extractor and NP2Vec to create semantic term sets
@@ -155,34 +146,8 @@ The main design guidelines are:
 * REST API servers with ability to serve trained models via HTTP
 * Extensive model documentation and tutorials
 
-## Demo UI examples
-
-Dependency parser
-<p>
-  <img src="https://raw.githubusercontent.com/NervanaSystems/nlp-architect/master/assets/bist-demo-small.png" height="375"/>
-</p>
-Intent Extraction
-<p>
-  <img src="https://raw.githubusercontent.com/NervanaSystems/nlp-architect/master/assets/ie-demo-small.png" height="375"/>
-<p>
-
-## Packages
-
-| Package                 	| Description                                          	|
-|-------------------------	|------------------------------------------------------	|
-| `nlp_architect.api`      	| Model API interfaces                                 	|
-| `nlp_architect.common`   	| Common packages                                      	|
-| `nlp_architect.cli`      	| Command line module                                  	|
-| `nlp_architect.data`     	| Datasets, loaders and data processors                	|
-| `nlp_architect.models`   	| NLP, NLU and End-to-End models                       	|
-| `nlp_architect.nn`      	| Topology related models and additions (per framework)	|
-| `nlp_architect.pipelines`	| End-to-end NLP apps                                  	|
-| `nlp_architect.procedures`| Procedure scripts                                    	|
-| `nlp_architect.server`   	| API Server and demos UI                              	|
-| `nlp_architect.solutions` | Solution applications                                	|
-| `nlp_architect.utils`    	| Misc. I/O, metric, pre-processing and text utilities 	|
-
 ### Note
+
 NLP Architect is an active space of research and development; Throughout future
 releases new models, solutions, topologies and framework additions and changes
 will be made. We aim to make sure all models run with Python 3.6+. We
@@ -191,17 +156,17 @@ encourage researchers and developers to contribute their work into the library.
 ## Citing
 
 If you use NLP Architect in your research, please use the following citation:
-```
+
 @misc{izsak_peter_2018_1477518,
   title        = {NLP Architect by Intel AI Lab},
   month        = nov,
   year         = 2018,
   doi          = {10.5281/zenodo.1477518},
   url          = {https://doi.org/10.5281/zenodo.1477518}
 }
-```
 
 ## Disclaimer
+
 The NLP Architect is released as reference code for research purposes. It is
 not an official Intel product, and the level of quality and support may not be
 as expected from an official product. NLP Architect is intended to be used

dev-requirements.txt

@@ -1,4 +1,4 @@
-sphinx
+sphinx==1.8.5
 sphinx_rtd_theme
 flake8-html
 pep8

docs-source/source/CONTRIBUTING.rst

@@ -22,8 +22,9 @@ Contribution Process
    * Create an issue on GitHub:
      https://github.com/NervanaSystems/nlp-architect/issues
 
-2. Clone and/or update your checked out copy of nlp-architect to ensure you have the
-   most recent commits from the master branch:
+2. Fork NLP Architect and/or update your checked out copy of 
+   nlp-architect to ensure you have the
+   most recent commits from the master branch, for example:
 
   .. code-block:: bash
 
@@ -48,17 +49,16 @@ Contribution Process
 
   .. code-block:: bash
 
-      nlp_architect test   # ensure all are OK
-      nlp_architect style  # ensure there are no style related issues
+      ./scripts/run_tests.sh    # ensure all are OK
+      ./scripts/check_style.sh  # ensure there are no style related issues
 
 5. If necessary you may want to update and/or rebuild the documentation.
    This all exists under docs-source/source and is in
    `Sphinx reStructuredText format <http://sphinx-doc.org/rest.html>`_:
 
   .. code-block:: bash
 
-      cd scripts/
-      sh create_docs.sh   # builds the doc and starts a local server directly
+      ./scripts/create_docs.sh   # builds the doc and starts a local server directly
 
 6. Commit your changes and push your feature branch to your GitHub fork.  Be
    sure to add a descriptive message and reference the GitHub issue associated
@@ -71,23 +71,10 @@ Contribution Process
       git commit -m "Added new awesome functionality.  Closes issue #1"
       git push origin my_new_feature_branch
 
-7. Create a new pull request to get your feature branch merged into master for
-   others to use.  You'll first need to ensure your feature branch contains the
-   latest changes from master.  Furthermore, internal devs will need to assign
-   the request to someone else for a code review.  You must also ensure there
-   are no errors when run through the items defined in step 4.
-
-  .. code-block:: bash
-
-      # (external contribs): make a new pull request:
-      https://github.com/NervanaSystems/nlp-architect/pulls
-
-      # merge latest master changes into your feature branch
-      git fetch origin
-      git checkout master
-      git pull origin master
-      git checkout my_new_feature_branch
-      git merge master  # you may need to manually resolve any merge conflicts
+7. Create a new `pull request <https://github.com/NervanaSystems/nlp-architect/pulls>`_
+   to get your feature branch merged into master for others to use.
+   You'll first need to ensure your feature branch contains the latest changes from
+   master.
 
 8. If there are issues you can continue to push commits to your feature branch
    by following step 6.  They will automatically be added to this same merge