Initial commit

Author: vdt

.DS_Store

@@ -0,0 +1,124 @@
+# CarND Behavioral Cloning Project
+
+The goals / steps of this project are the following:
+* Use the simulator to collect data of good driving behavior
+* Build, a convolution neural network in Keras that predicts steering angles from images
+* Train and validate the model with a training and validation set
+* Test that the model successfully drives around track one without leaving the road
+* Summarize the results with a written report
+
+
+### Files Submitted & Code Quality
+
+1. Submission includes all required files and can be used to run the simulator in autonomous mode
+
+My project includes the following files:
+* [model.py](model.py)  containing the script to create and train the model
+* [drive.py](drive.py) for driving the car in autonomous mode
+* [model.h5](model-004.h5) containing a trained convolution neural network (this model.h5 file is model-004.h5(which is one of many model-xxx.h5 files produced by tweaking the model parameters and running ```python model.py```))
+* [output text file 1 while running ```python model.py``` on aws EC2 GPU instance](output-text-file1)
+* [output text file 2 while running ```python model.py```](output-text-file2)
+* [output text file 3 while running ```python model.py```](output-text-file3)
+* [video.py](video.py) for converting the image files to video
+* [README.md](README.md) summarizing the results
+
+2. Submission includes functional code
+Using the Udacity provided simulator and my drive.py file, the car can be driven autonomously around the track by executing 
+```python drive.py model.h5```
+
+3. Submission code is usable and readable
+
+The [model.py](model.py) file contains the code for training and saving the convolution neural network. The file shows the pipeline I used for training and validating the model, and it contains comments to explain how the code works.
+
+### Model Architecture and Training Strategy
+
+#### 1. An appropriate model architecture has been employed
+
+[NVIDIA's End-to-End Deep Learning Model for Self-Driving Cars](https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/) is used. It consists of a convolution neural network with 5x5 and 3x3 filter sizes and depths between 24 and 64 ([model.py](model.py) code lines 46-75) 
+
+The data is normalized in the model using a Keras lambda layer ([model.py](model.py) code line 67)
+
+The model includes ELU layers to introduce nonlinearity ([model.py](model.py) code lines 70-80)
+
+#### 2. Attempts to reduce overfitting in the model
+
+The model contains dropout layers in order to reduce overfitting ([model.py](model.py) code line 76). 
+
+The model was trained and validated on different data sets to ensure that the model was not overfitting ([model.py](model.py) code line 142).
+
+The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track.
+
+#### 3. Model parameter tuning
+
+The model used an adam optimizer with the default learning rate (0.0001) ([model.py](model.py) code line 110 and line 148).
+
+#### 4. Appropriate training data
+
+I used [Udacity's SDC-ND Sample Training Data](https://d17h27t6h515a5.cloudfront.net/topher/2016/December/584f6edd_data/data.zip) for Training [The Model](model.py)
+
+### Model Architecture and Training Strategy
+
+#### 1. Solution Design Approach
+
+I started of with a simple CNN Model
+
+In order to gauge how well the model was working, I split my image and steering angle data into a training and validation set. I found that my first model had a low mean squared error on the training set but a high mean squared error on the validation set. This implied that the model was overfitting. 
+
+To combat the overfitting, I modified the model so that there is less overfitting.
+
+The final step was to run the simulator to see how well the car was driving around track one. There were a few spots where the vehicle fell off the track or slammed the bridge and stopped or hit the tree and stopped. To improve the driving behavior in these cases, I tried to implement [NVIDIA's End-to-End Deep Learning Model for Self-Driving Cars](https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/)
+
+After lots of trial and error , and tweaking the training set(from 80% to 90%) and test set (from 20% to 10%) and changing the batch size and learning rate.
+
+#### 2. Final Model Architecture
+
+Finally , [NVIDIA's End-to-End Deep Learning Model for Self-Driving Cars](https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/) was used. 
+
+The Final Model Architecture consisted of convolution neural network with 5x5 and 3x3 filter sizes and depths between 24 and 64 ([model.py](model.py) code lines 46-75) 
+
+The data is normalized in the model using a Keras lambda layer ([model.py](model.py) code line 67)
+
+The model includes ELU layers to introduce nonlinearity ([model.py](model.py) code lines 70-80)
+
+The model contains dropout layers in order to reduce overfitting ([model.py](model.py) code line 76). 
+
+The model was trained and validated on different data sets to ensure that the model was not overfitting ([model.py](model.py) code line 142).
+
+Here is a visualization of the architecture (note: visualizing the architecture is optional according to the project rubric)
+
+| Layer (type) | Output Shape | Param #      | Connected to |
+|--------------|--------------|--------------|--------------|
+|lambda_1 (Lambda)| (None, 66, 200, 3) | 0 | lambda_input_1[0][0] |
+|convolution2d_1 (Convolution2D)|  (None, 31, 98, 24) |   1824 |       lambda_1[0][0] |
+|convolution2d_2 (Convolution2D) | (None, 14, 47, 36)  |  21636   |    convolution2d_1[0][0] |
+|convolution2d_3 (Convolution2D) | (None, 5, 22, 48)  |   43248   |    convolution2d_2[0][0] |
+|convolution2d_4 (Convolution2D) |(None, 3, 20, 64) |    27712    |   convolution2d_3[0][0] |
+|convolution2d_5 (Convolution2D) | (None, 1, 18, 64)  |   36928   |    convolution2d_4[0][0] |
+|dropout_1 (Dropout)    |          (None, 1, 18, 64)    | 0      |     convolution2d_5[0][0] |
+|flatten_1 (Flatten)     |         (None, 1152)    |      0       |    dropout_1[0][0] |
+|dense_1 (Dense)   |               (None, 100)     |      115300    |  flatten_1[0][0] |
+|dense_2 (Dense)    |              (None, 50)     |       5050     |   dense_1[0][0] |
+|dense_3 (Dense)    |              (None, 10)      |      510      |   dense_2[0][0] |
+|dense_4 (Dense)     |             (None, 1)       |      11        |  dense_3[0][0] |
+
+#### 3. Creation of the Training Set & Training Process
+
+#### 4. Final Video
+
+for recording or saving the images for video in folder [rn1](rn1)
+
+```python drive.py model-004.h5 rn1```
+
+
+
+for taking the recorded or saved images and making the video rn1.mp4 at 60 frames per second (default)
+
+```python video.py rn1 ```  outputs [video(youtube)](https://youtu.be/gvwRCXzHGGs) / [video in the repo](Video-60fps.mp4)
+
+
+
+
+for taking the recorded or saved images and making the video rn1.mp4 at 40 frames per second
+
+```python video.py rn1 --fps 40``` outputs [video(youtube)](https://youtu.be/lEZAF99rWQI) / [video in the repo](Video-40fps.mp4)
+

README.md

@@ -0,0 +1,142 @@
+#parsing command line arguments
+import argparse
+#decoding camera images
+import base64
+#for frametimestamp saving
+from datetime import datetime
+#reading and writing files
+import os
+#high level file operations
+import shutil
+#matrix math
+import numpy as np
+#real-time server
+import socketio
+#concurrent networking 
+import eventlet
+#web server gateway interface
+import eventlet.wsgi
+#image manipulation
+from PIL import Image
+#web framework
+from flask import Flask
+#input output
+from io import BytesIO
+
+#load our saved model
+from keras.models import load_model
+
+#helper class
+import utils
+
+#initialize our server
+sio = socketio.Server()
+#our flask (web) app
+app = Flask(__name__)
+#init our model and image array as empty
+model = None
+prev_image_array = None
+
+#set min/max speed for our autonomous car
+MAX_SPEED = 25
+MIN_SPEED = 10
+
+#and a speed limit
+speed_limit = MAX_SPEED
+
+#registering event handler for the server
+@sio.on('telemetry')
+def telemetry(sid, data):
+    if data:
+        # The current steering angle of the car
+        steering_angle = float(data["steering_angle"])
+        # The current throttle of the car, how hard to push peddle
+        throttle = float(data["throttle"])
+        # The current speed of the car
+        speed = float(data["speed"])
+        # The current image from the center camera of the car
+        image = Image.open(BytesIO(base64.b64decode(data["image"])))
+        try:
+            image2 = np.asarray(image)       # from PIL image to numpy array
+            image2 = utils.preprocess(image2) # apply the preprocessing
+            #image = np.array([image2])       # the model expects 4D array
+            image2 = image2[None, :, :, :]
+
+            # predict the steering angle for the image
+            steering_angle = float(model.predict(image2, batch_size=1))
+            # lower the throttle as the speed increases
+            # if the speed is above the current speed limit, we are on a downhill.
+            # make sure we slow down first and then go back to the original max speed.
+            global speed_limit
+            if speed > speed_limit:
+                speed_limit = MIN_SPEED  # slow down
+            else:
+                speed_limit = MAX_SPEED
+            throttle = 1.0 - steering_angle**2 - (speed/speed_limit)**2
+
+            print('{} {} {}'.format(steering_angle, throttle, speed))
+            send_control(steering_angle, throttle)
+        except Exception as e:
+            print(e)
+
+        # save frame
+        if args.image_folder != '':
+            timestamp = datetime.utcnow().strftime('%Y_%m_%d_%H_%M_%S_%f')[:-3]
+            image_filename = os.path.join(args.image_folder, timestamp)
+            image.save('{}.jpg'.format(image_filename))
+    else:
+        
+        sio.emit('manual', data={}, skip_sid=True)
+
+
+@sio.on('connect')
+def connect(sid, environ):
+    print("connect ", sid)
+    send_control(0, 0)
+
+
+def send_control(steering_angle, throttle):
+    sio.emit(
+        "steer",
+        data={
+            'steering_angle': steering_angle.__str__(),
+            'throttle': throttle.__str__()
+        },
+        skip_sid=True)
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Remote Driving')
+    parser.add_argument(
+        'model',
+        type=str,
+        help='Path to model h5 file. Model should be on the same path.'
+    )
+    parser.add_argument(
+        'image_folder',
+        type=str,
+        nargs='?',
+        default='',
+        help='Path to image folder. This is where the images from the run will be saved.'
+    )
+    args = parser.parse_args()
+
+    #load model
+    model = load_model(args.model)
+
+    if args.image_folder != '':
+        print("Creating image folder at {}".format(args.image_folder))
+        if not os.path.exists(args.image_folder):
+            os.makedirs(args.image_folder)
+        else:
+            shutil.rmtree(args.image_folder)
+            os.makedirs(args.image_folder)
+        print("RECORDING THIS RUN ...")
+    else:
+        print("NOT RECORDING THIS RUN ...")
+
+    # wrap Flask application with engineio's middleware
+    app = socketio.Middleware(sio, app)
+
+    # deploy as an eventlet WSGI server
+    eventlet.wsgi.server(eventlet.listen(('', 4567)), app)