Session 14: HTTP python library
- Time: 2h
- Date: Tuesday, March-17th-2020
-
Goals:
- Learn how to implement web servers using the HTTP python module
- Debugging the HTTP protocol with the Browser developer's tools
- Introduction
- HTTP module web server
- Web Server 1: Using the httpserver handler
- Web Server 2: Implementing our own handler
- Debugging the HTTP protocol
- TODO
- Exercises
- End of the session
- Author
- Credits
- License
Python includes an HTTP module for implementing both HTTP clients and servers very easily. In the previous week we learnt how to implement them from sockets. We did it this way for you to learn how the socket mechanism worked. Now that you know how they work, we will focus on programming the server using the HTTP module. This is the module that we will use from now on.
We will follow a top-down approach: starting with the fully-working server provided by the http module. Then we will focused on the internal details for implementing our own servers
The HTTP module includes a fully working web server. Let's test it
- Create a new folder in your 2019-2020-PNE-practices project. Call it: Session-14
- Download these html example files and store them into the Session-14 folder:
- Select the Session-14 folder
- Right-click on it and chose the Open in terminal entry
- From that terminal execute the following command:
python -m http.server 8080
- It will launch the web server. Go to the browser and open this page: http://localhost:8080. This other URLs are equivalent: http://0.0.0.0:8080/, http://your-IP:8080 and http://127.0.0.1:8080. You should see something like this:
The server is showing the index.html (that is our main web page). You should be able to navigate to the other pages, just clicking on the links
This is our first version of the web server using the http.server module. Here you can find all the documentation of the http.server module
The Handler is the name given to an object that is called when a client is connected and need to be attended. In this example we are using the Handler provided by the http.server module
- Create a new python file called webserver1.py and type this code
import http.server
import socketserver
# Define the Server's port
PORT = 8080
# -- This is for preventing the error: "Port already in use"
socketserver.TCPServer.allow_reuse_address = True
# -- Use the http.server Handler
Handler = http.server.SimpleHTTPRequestHandler
# -- Open the socket server
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("Serving at PORT", PORT)
# -- Main loop: Attend the client. Whenever there is a new
# -- clint, the handler is called
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("Server Stopped!")
httpd.server_close()Initially a server socket is created, bound to our IP and port and changed to the listen mode. Then, serve_forever() method is called. It makes the server wait until a client is connected. When it happens, the handler function is called.
- Execute it from pycharm. It should work exactly the same as the previous example
Let's define our own Handler for attending the clients. As always, we will start with something very simple and we will be adding more functionality on every example
This example just prints a message on the console telling us that we have received a GET request message
import http.server
import socketserver
# Define the Server's port
PORT = 8080
# -- This is for preventing the error: "Port already in use"
socketserver.TCPServer.allow_reuse_address = True
# Class with our Handler. It is a called derived from BaseHTTPRequestHandler
# It means that our class inheritates all his methods and properties
class TestHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
"""This method is called whenever the client invokes the GET method
in the HTTP protocol request"""
# We just print a message
print("GET received!")
# IN this simple server version:
# We are NOT processing the client's request
# We are NOT generating any response message
return
# ------------------------
# - Server MAIN program
# ------------------------
# -- Set the new handler
Handler = TestHandler
# -- Open the socket server
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("Serving at PORT", PORT)
# -- Main loop: Attend the client. Whenever there is a new
# -- clint, the handler is called
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("")
print("Stopped by the user")
httpd.server_close()We should create a Class derived from the http.server.BaseHTTPRequestHandler base Class. For attending the GET request message, we should implement the do_GET method. This method will be called whenever there is a GET request from the client. In this example we are just printing a message on the console
Notice that we are not processing the client requests. And we are not generating any response message. For the later reason the browser will show us an error message
We have also improved how the server is stopped: whenever the control-c is detected (or the server is stopped within pycharm) the server_close() method is called and a message is printed on the script
- Type the code in the webserver2-1.py file and execute it
- In the server you should see an error like this:
In our console there should appear the message: GET received! (many times, because as the browser is not getting any response from the server, it is sending the request message many times)
As our TestHandler is derived from the Base class BaseHTTPRequestHandler, it inheritates all its properties and methods. You can find all the properties and method in the http.server documentation (Check it!!)
In this example we will print the following properties:
- self.requestline: It contatins the full request line (remember that the request line is the first line of the HTTP request messages)
- self.command: The command that the clients is invoking. It should be always GET as we are inside the do_GET() method
- self.path: It contains the resource that the client is asking for
This is the new web server:
import http.server
import socketserver
import termcolor
# Define the Server's port
PORT = 8080
# -- This is for preventing the error: "Port already in use"
socketserver.TCPServer.allow_reuse_address = True
# Class with our Handler. It is a called derived from BaseHTTPRequestHandler
# It means that our class inheritates all his methods and properties
class TestHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
"""This method is called whenever the client invokes the GET method
in the HTTP protocol request"""
# We just print a message
print("GET received! Request line:")
# Print the request line
termcolor.cprint(" " + self.requestline, 'green')
# Print the command received (should be GET)
print(" Command: " + self.command)
# Print the resource requested (the path)
print(" Path: " + self.path)
# IN this simple server version:
# We are NOT processing the client's request
# We are NOT generating any response message
return
# ------------------------
# - Server MAIN program
# ------------------------
# -- Set the new handler
Handler = TestHandler
# -- Open the socket server
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("Serving at PORT", PORT)
# -- Main loop: Attend the client. Whenever there is a new
# -- clint, the handler is called
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("")
print("Stoped by the user")
httpd.server_close()- Execute the new web server and connects from the browser. Check your console messages:
Now we will add a response message to our handler so that the browser will show some contents instead of an error message. Our server will always send the same message, no matter what resources is the client requesting: is a happy server :-)
In our TestHandler Class we can use the following methods for generating the response very easily:
- self.send_response(code) : Creates a response header with a status line with the error CODE passed as an argumento. The response is not really sent yet, but stored into a buffer. The status codes can be found in this link
- self.send_header(): Add a header to the response message (Ex. Content-Type, Content-Length...)
- self.end_headers(): Add a blank line to the response message (indicating that the header is finished)
The message body is sent using the self.wfile.write() method
import http.server
import socketserver
import termcolor
# Define the Server's port
PORT = 8080
# -- This is for preventing the error: "Port already in use"
socketserver.TCPServer.allow_reuse_address = True
# Class with our Handler. It is a called derived from BaseHTTPRequestHandler
# It means that our class inheritates all his methods and properties
class TestHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
"""This method is called whenever the client invokes the GET method
in the HTTP protocol request"""
# Print the request line
termcolor.cprint(self.requestline, 'green')
# IN this simple server version:
# We are NOT processing the client's request
# It is a happy server: It always returns a message saying
# that everything is ok
# Message to send back to the clinet
contents = "I am the happy server! :-)"
# Generating the response message
self.send_response(200) # -- Status line: OK!
# Define the content-type header:
self.send_header('Content-Type', 'text/plain')
self.send_header('Content-Length', len(contents.encode()))
# The header is finished
self.end_headers()
# Send the response message
self.wfile.write(contents.encode())
return
# ------------------------
# - Server MAIN program
# ------------------------
# -- Set the new handler
Handler = TestHandler
# -- Open the socket server
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("Serving at PORT", PORT)
# -- Main loop: Attend the client. Whenever there is a new
# -- clint, the handler is called
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("")
print("Stoped by the user")
httpd.server_close()- Run the server. Now the browser should display the following message:
And this is what you get in the pycharm console:
The servers are sometimes hard to debug. Maybe one of our server is not working fine, but we do not know why. The browsers have tools for helping the developers to understand what is going wrong. We will learn how to use the Firefox's developer toools. Although I will use Firefox in the examples, the Chrome browser has also very similar tools
- Run the previous web server
- Open a new tab
- click on the top-right menu and select the entry webdeveloper/network
- Connect to the server: localhost:8000 (or the port you have configured)
- You will see all the requests done by the browser, and their status codes
- Clicking on the first request will give you more information about the response and request messages interchanged:
Modify the previous server for send back the index.html file when the / resource is requested. It should send the error.html page (and the correspoding error code) if any other resource is requested
All the exercises and experiments performed during this session should be stored in the Session-12 folder
- Filename: Session-12/exercise-1.txt
- Description: A Text file in which you should write down you answers to the exercise 1
Run the web-server-2. Open the browser and connect to the URL: http://127.0.0.1:8080/hello. Answer the following questions:
- Which is the request line?
- Which is the resource name that the client is asking for? (Path)
Repeat the exercise for this URL: http://127.0.0.1:8080/file.html
Repeat the exercise for this URL: http://127.0.0.1:8080/hi/there?name=virus&type=corona
What should be the URL that we have to write in the Browser for accessing the /dna/u5 resource?
- Filename: Session-12/we-server-Ex2.py
- Description: It is the web-server-2.py server, modify so that the Content-Type header has the value: text/plain
Run the server and use the curl tool to confirm that the response message has the new header:
< Content-Type: text/plain
Try to connect from the Browser. What happens? Could you see the web page? (is the page green?)
- Filename: Session-12/we-server-Ex3.py
- Description: It is the web-server-2.py server, modified so that the Content-Length header has the value: 5
When you connect from the browser without any change in the Content-Length you will see the Green server... but What happens when you modify the Content-Length?
Test it with the curl command. Could you see the HTML message?
- Filename: Session-12/we-server-Ex4.py
- Description: It is the web-server-2.py server, modified so that it reads the index.html file and sends it as a response to the client
- Filename: Session12/index.html
- Description: The HTML page of the Green server, in a file
<!DOCTYPE html>
<html lang="en" dir="ltr">
<head>
<meta charset="utf-8">
<title>Green server</title>
</head>
<body style="background-color: lightgreen;">
<h1>GREEN SERVER</h1>
<p>I am the Green Server! :-)</p>
</body>
</html>When you connect from the browser, you will se the Green server web page
Once it is working, modify the file index.html so that the message that appears is: "I am the Green Server! :-) (MODIFIED BY ME!!!!)"
The session is finished. Make sure, during this week, that everything in this list is checked!
-
You have all the items of the session 11 checked!
-
Your working repo contains the Session-12 Folder with the following files:
- echo-server.py
- web-server-1.py
- web-server-2.py
- exercise-1.txt
- web-server-Ex2.py
- web-server-Ex3.py
- web-server-Ex4.py
- index.html
-
All the previous files have been pushed to your remote Github repo
- Juan González-Gómez (Obijuan)
- Alvaro del Castillo. He designed and created the original content of this subject. Thanks a lot :-)
