For example, for a chat room, since there are multiple connections that need to be processed at the same time, it is obvious that blocking or synchronizing methods are inappropriate, just like buying a ticket with only one window, queuing up multiple people, etc. So how do we solve this problem? There are three main methods: forking, threading, and asynchronous I/O.
The methods of Forking and threading are very simple, and can be implemented by using the min-in class of the SocketServer service class. forking is only suitable for Unix-like platforms; threading needs to pay attention to memory sharing issues.
Asynchronous I/O is a bit difficult if the underlying method is implemented. To be simple, we can consider using the framework from the standard library or Twisted (Twisted is a very powerful framework for asynchronous network programming).
1. Use ScoketServer to implement Forking and threading
Below we use two examples to create a forking server and a threading server respectively.
Forking server:
from SocketServer import TCPServer, ForkingMixIn, StreamRequestHandler
class Server(ForkingMixIn, TCPServer): pass
class Handler(StreamRequestHandler):
def handle(self):
addr = ()
print 'Got connection from', addr
('Thank you for connecting')
server = Server(('', 1234), Handler)
server.serve_forever()
threading server:
from SocketServer import TCPServer, ThreadingMixIn, StreamRequestHandler
class Server(ThreadingMixIn, TCPServer): pass
class Handler(StreamRequestHandler):
def handle(self):
addr = ()
print 'Got connection from', addr
('Thank you for connecting')
server = Server(('', 1234), Handler)
server.serve_forever()
2. Use select to implement asynchronous I/O
Asynchronous I/O, for example, if a large group of people want you to listen to them, then you give each of them one minute to talk, everyone takes turns to talk, and continue to talk later if they haven’t finished speaking. That is, a time slice method.
To implement asynchronous I/O, we can use the framework asynccore/asynchat or Twisted, which are based on select functions or poll functions (poll is only suitable for Unix-like systems). Both the select and poll functions come from the select module.
The select function requires three must sequences as parameters and an optional timeout value in seconds. In the sequence are integer values representing file descriptors, and they are the connections we want to wait for. These three sequences are about input, output, and exception conditions. If the timeout value is not given, select will be in a blocking state (that is, waiting) until there is a file descriptor ready for action. If the timeout value is given, select only blocks the given time. If the timeout value is 0, then it will not block. The value returned by select is a tuple of three sequences that represent a subset of the activities of the corresponding parameters. For example, the first sequence returns a sequence composed of input file descriptors for reading.
A sequence can contain file objects (not suitable for Windows) or sockets. The following example creates a server that uses select to serve several connections (note: the server socket itself is also provided to the select so that it can signal when a new connection is ready to be accepted). This server simply prints data received from the client. You can use telnet (or write a simple socket-based client) to connect to test it.
select server
import socket, select
s = ()
host = ()
port = 1234
((host, port))
(5)
inputs = [s]
while True:
rs, ws, es = (inputs, [], [])
for r in rs:
if r is s:
c, addr = ()
print 'Got connection from', addr
(c)
else:
try:
data = (1024)
disconnected = not data
except :
disconnected = True
if disconnected:
print (), 'disconnected'
(r)
else:
print data
3. Twisted
Twisted is an event-driven network framework for Python. It was originally developed for online games, but is now used in various network software. With Twisted, you can implement event processors, which are very similar to using GUI toolkits (Tk, GTK, Qt, wxWidgets). In this section I will introduce some basic concepts and demonstrate how to use Twisted to do some relatively simple network programming. Twisted is a very powerful framework and provides a lot of support, such as: Web server and client, SSH2, SMTP, POP3, IMAP4, AIM, ICQ, IRC, MSN, Jabber, NNTP, DNS, etc.
The socket-based servers we wrote earlier, they all have a displayed event loop: looking for new connections and new data; the SocketServer-based server has an implicit loop: looking for connections and creating processors for connections. But when the processor still displays the read data.
And Twisted uses more event-based approaches. To write a basic server, you need to implement an event processor that handles situations such as a new client connection, new data arrival, and client connection interruption. In Twisted, your event handler is defined in a protocol; you also need a factory, which can construct this protocol object when a new connection arrives, but if you just want to create an instance of a custom Protocol class, you can use the factory from Twisted, the Factory class in the module. When you write your protocol, use Protocol in the module as your parent class. When you get a connection, the event handler connectionMade is called; when you lose a connection, the connectionLost is called. Accept data from the client using the processor dataReceived. But you cannot use event processing policy to send data to the client; to send data to the client, you can use it, it has a write method. It also has a client property that contains the client's address (host name and port).
The following example is a Twisted version of the server. There is instantiated Factory and its protocol property is set so that it knows which protocol to use to communicate with the client (this is called your custom protocol). Then you use factory to start listening to the specified port, and the factory handles the connection through the instantiated protocol object. Listen to the listenTCP function in the reactor module. Finally, you start the server by calling the run function in the reactor module.
from import reactor
from import Protocol, Factory
# Define your Protocol class
class SimpleLogger(Protocol):
def connectionMade(self):
print 'Got connection from',
def connectionLost(self, reason):
print , 'disconnected'
def dataReceived(self, data):
print data
# Instantiate Factory
factory = Factory()
# Set the protocol property of the factory so that it knows which protocol to use to communicate with the client (this is called your customization
# protocol)
= SimpleLogger
# Listen to the specified port
(1234, factory)
# Start running the main program
()
It is easy to write a custom protocol for your processing purposes. The module contains several useful existing protocols, where LineReceiver executes dataReceived and calls the event handler lineReceived when it receives a complete line. If you have to do something else when accepting data, in addition to using lineReceived, you can use the event handler named rawDataReceived defined by LineReceiver. Here is an example of a server using LineReceiver:
from import reactor
from import Factory
from import LineReceiver
class SimpleLogger(LineReceiver):
def connectionMade(self):
print 'Got connection from',
def connectionLost(self, reason):
print , 'disconnected'
def lineReceived(self, line):
print line
factory = Factory()
= SimpleLogger
(1234, factory)
()
urllib and urllib2
urllib and urllib2 work similarly, they allow you to access files over the network, just like accessing your own computer. Through simple function calls, the resources located by the URL can be used as input into your program. If you are equipped with the re module, you can download web pages, extract information, and automatically create reports of what you are looking for.
urllib2 is more popular. For simple download tasks, urllib is better. If you need HTTP verification or cookies, or if you want to write some extensions to handle your own protocol, then urllib2 is the right choice.
1. Open remote file
The operation of opening a remote file is similar to that of local files. The difference is that you can only use read mode and use the urlopen of the urllib module:
>>> from urllib import urlopen
>>> webpage=urlopen('')
If you are online, the variable webpage now contains a file-class object associated with the web page:.
Note: If you are not currently connected to the Internet and you want to practice urllib, you can access local files in the following form:
localpage=urlopen(r'file:c:\')
The file class object returned by urlopen supports close, read, readline, readlines and other methods.
The following code extracts the URL of the "Documentation" link in the official Python homepage:
>>> import re
>>> text = ()
>>> m = ('<a href="([^"]+)">Documentation</a>', text, )
>>> (1)
'/'
2. Obtain remote files
The urlopen function gives you a file class object that you can read. If you only care about downloading files and storing a copy to the local file when using urlib, you can use urlretrieve instead. urlretrieve returns a tuple (filename, headers), filename is the name of the local file (replica) (it is automatically created by urllib). Headers contain some information about the remote file.
If you want to specify a name for the replica, you can provide the second parameter:
urlretrieve('', 'C:\\python_webpage.html')
This will get the Python official homepage and store it in local C:\python_webpage.html. If you do not specify the file name of the replica, the file will be placed in a temporary place, you can open it using the open function. If you want to clear these temporary replicas, you can call the urlcleanup function without any parameters, and it will do the cleaning work for you.
1. Sockets
Sockets are objects that provide current portable standards for network application providers on a suite pair on a specific network protocol (such as TCP/IP, ICMP/IP, UDP/IP, etc.). They allow programs to accept and connect, such as sending and accepting data. In order to establish a communication channel, it is extremely important that each endpoint of a network communication has a socket object.
Sockets are part of the BSD UNIX system core, and they are also adopted by many other UNIX-like operating systems including Linux. Many non-BSD UNIX systems (such as ms-dos, windows, os/2, mac os and most host environments) provide socket support in the form of libraries.
The three most popular socket types are: stream, datagram and raw. Stream and datagram sockets can directly interface with the TCP protocol, while raw sockets interface to the IP protocol. But sockets are not limited to TCP/IP.
2. Socket module
The socket module is a very simple object-based interface that provides access to low-level BSD socket-style networks. This module allows you to implement client and server sockets. To build a simple server with TCP and stream sockets in python, you need to use the socket module. Using the functions and class definitions contained in this module, programs that communicate over the network can be generated. Generally speaking, establishing a server connection requires six steps.
Step 1 is to create the socket object. Call the socket constructor.
socket=(familly,type)
The value of family can be AF_UNIX (Unix domain, used for inter-process communication on the same machine), or AF_INET (TCP and UDP for IPV4 protocols), as for type parameters, SOCK_STREAM (stream socket) or SOCK_DGRAM (data packet socket), SOCK_RAW (raw socket).
The second step is to bind (assign) the socket to the specified address, (address)
The address must be a two-element tuple, ((host,port)), host name or ip address + port number. An exception is raised if the port number is being used or retained, or the host name or IP address is incorrect.
Step 3: After binding, the socket must be prepared to accept the connection request.
(backlog)
The backlog specifies the maximum number of connections, at least 1. After receiving the connection request, these requests must be queued, and if the queue is full, the request will be rejected.
In step 4, the server socket waits for the client to request a connection through the socket's accept method:
connection,address=()
When the accept method is called, the socket enters the 'waiting' (or blocking) state. When a client requests a connection, the method establishes the connection and returns to the server. The accept method returns a tuple containing two elements, such as (connection, address). The first element (connection) is a new socket object through which the server communicates with the client; the second element (address) is the client's internet address.
Step 5 is the processing stage, where the server and the client communicate (transmit data) through send and recv methods. The server calls send and sends information to the customer in the form of a string. The send method returns the number of characters that have been sent. The server uses the recv method to receive information from the client. When calling recv, an integer must be specified to control the maximum amount of data accepted by this call. When accepting data, the recv method will enter the 'blocket' state, and finally return a string, which is used to represent the received data. If the amount sent exceeds the allowed by recv, the data will be truncated. The excess data will be buffered on the receiver side. When recv is called later, the excess data will be deleted from the buffer.
Step 6: When the transmission is over, the server calls the socket's close method to close the connection.
Establishing a simple customer connection requires 4 steps.
Step 1, create a socket to connect to the server socket=(family,type)
Step 2: Use socket's connect method to connect to the server ((host,port))
In step 3, the client and the server communicate through the send and recv methods.
Step 4, after the end, the client closes the connection by calling the socket's close method.
3. A simple example of server-client communication
server:
import socket
s=()
(('',xxxx)) #ip address and port number
(5)
cs,address = ()
print 'got connected from',address
('byebye')
ra=(512)
print ra
()
Client:
import socket
s=()
(('',xxxx)) #The same IP address and port number as the server program
data=(512)
('hihi')
()
print 'the data received is',data
run:
In native test (in Windows environment, you can change the IP address to native IP, the port number is above 1024, and Windows will be reserved below 1024), run --CMD--enter the command line mode
First, the python server program, then the python client program.
Or after starting the server program, use the telnet ip address port number to get the same result.
--------------------------------------------------------------------------------
Make server continuously accept connections
import socket
s=()
(('192.168.43.137',2000))
(5)
while 1:
cs,address = ()
print 'got connected from',address
('hello I am server,welcome')
ra=(512)
print ra
()
Test whether it is feasible to coexist with two sockets in two programs
import socket
s=()
(('192.168.43.137',2000))
data=(512)
print 'the data received is\n ',data
('hihi I am client')
sock2 = ()
(('192.168.43.137',2000))
data2=(512)
print 'the data received from server is\n ',data2
('client send use sock2')
()
()
Network Programming Framework Sunday, April 12, 2009 at 10:39 am twisted is a very awesome network programming framework recognized in Python. If those who learn python network programming do not learn twisted, they can only be considered to understand python network programming. Just like using django to develop websites, both are famous frameworks under python. twisted is a single-threaded event-driven network engine. There are relatively fewer learning materials about it, and there are even fewer Chinese, so you must bite the bullet and read the English document when learning twisted, that is, its twisted documentation, where you can basically find all the basic knowledge you need. In particular, there are many examples in core documentation and example. If all of these examples are run, then your twisted can be considered a beginner.
I mainly wrote a Tracker server for an internal content distribution network using twisted factory and protocol framework. It is not based on the standard BT protocol. If you want to learn, it is best to follow the standard BT protocol. The URL was also given before. As for how to use twisted, I will introduce it in detail in the subsequent article.
This article first introduces two ways of twisted work, reactor and application.
The reactor is the core of the event loop within Twisted -- the loop which drives applications using Twisted. The reactor provides basic interfaces to a number of services, including network communications, threading, and event dispatching.
Reactor is the core of the twisted event loop, which provides basic interfaces to some services, such as network communication, threading and event distribution.
For a detailed introduction to reactor, see the first section of the Low-Level Twisted chapter in the twisted core documentation. The installation and use of various reactors are introduced in detail.
The reactors I know have the following
reactor platform Usage
IOCPReactor win32 from import iocpreactor ()
from import reactor
selectReactor win32, posix from import reactor
pollReactor posix from import pollreactor
()
from import reactor
epollReactor linux2.6 from import epollreactor
()
from import reactor
kqueueReactor BSD series from import kqreactor
()
from import reactor
The above are the most used reactors. Except for kqueueReactor I have never used, I have used the others. All work normally. It is recommended to choose the best reactor according to different platforms when programming.
The system uses selectreactor by default.
Here is a small example:
from import Protocol, Factory
from import reactor
### Protocol Implementation
# This is just about the simplest possible protocol
class Echo(Protocol):
def dataReceived(self, data):
"""As soon as any data is received, write it back."""
(data)
def main():
f = Factory()
= Echo
(8000, f)
()
if __name__ == '__main__':
main()