Model Containers and AlexNet Construction
The article and code have been archived to [Github repository:/timerring/dive-into-AI 】
In addition to the modules mentioned above, another important concept is the model containers (Containers), there are three commonly used containers, these containers are inherited from the。
- : Packing multiple network layers sequentially
- : wraps multiple network layers like python's lists and can be iterated on
- : Wraps multiple network layers like python's dict, assigning each layer a name by means of (key, value).
In deep learning, the two steps of feature extraction and classifier are fused into a single neural network. In a convolutional neural network, the front convolutional layer and the pooling layer can be considered as the feature extraction part, while the back fully connected layer can be considered as the classifier part. For example, LeNet can be divided intofeature extractionrespond in singingsorterBoth parts can be used separately. to pack.
The code is as follows:
class LeNetSequetial():
def __init__(self, classes):
super(LeNet2, self).__init__()
= (
nn.Conv2d(3, 6, 5),
(),
nn.MaxPool2d(2, 2),
nn.Conv2d(6, 16, 5),
(),
nn.MaxPool2d(2, 2)
)
= (
(16*5*5, 120),
(),
(120, 84),
(),
(84, classes)
)
def forward(self, x):
x = (x)
x = (()[0], -1)
x = (x)
return xAt initialization time, thewill invoke__init__()method, which adds each submodule to its own_modulesattribute. As you can see here, the parameter we pass in can be either a list, or an OrderDict. If it's an OrderDict, then we use the key from the OrderDict, otherwise we use a number as the key.
def __init__(self, *args):
super(Sequential, self).__init__()
if len(args) == 1 and isinstance(args[0], OrderedDict):
for key, module in args[0].items():
self.add_module(key, module)
else:
for idx, module in enumerate(args):
self.add_module(str(idx), module)After the network initialization is complete there are two submodule：featuresrespond in singingclassifier。
(indicates contrast)featuresThe sub-modules in the following, each network layer is keyed by a serial number:
When performing forward propagation, it enters LeNet'sforward()function, which first calls the firstSequetialContainer:As a result ofis also a module, so it will call__call__()function, which calls the
result = (*input, **kwargs)Enter(used form a nominal expression)forward()function, where all modules are called in turn, and the output of the previous module is the input of the next.
def forward(self, input):
for module in self:
input = module(input)
return inputAs you can see above in theIn this case, each subnetwork layer module is indexed using a serial number, i.e., a number is used as the key.
Once the number of network layers increases and it becomes difficult to find a specific network layer, OrderDict can be used in this case. This can be compared to the code above
class LeNetSequentialOrderDict():
def __init__(self, classes):
super(LeNetSequentialOrderDict, self).__init__()
= (OrderedDict({
'conv1': nn.Conv2d(3, 6, 5),
'relu1': (inplace=True),
'pool1': nn.MaxPool2d(kernel_size=2, stride=2),
'conv2': nn.Conv2d(6, 16, 5),
'relu2': (inplace=True),
'pool2': nn.MaxPool2d(kernel_size=2, stride=2),
}))
= (OrderedDict({
'fc1': (16*5*5, 120),
'relu3': (),
'fc2': (120, 84),
'relu4': (inplace=True),
'fc3': (84, classes),
}))
...
...
...summarize
beThe container for sequentially wrapping a set of network layers has the following two properties.
- Sequentiality: each network layer is constructed in strict order among themselves, and we must pay attention to whether the shapes between the input and output data match between the front and back network layers when constructing the network
- bring one's own
forward()Functions: in the(used form a nominal expression)forward()The function reads each network layer in turn through a for loop and performs the forward propagation operation. This makes the model we are building much more concise.
becontainer for wrapping a set of network layers to invoke them in an iterative manner, with the following 3 main methods:
- append(): add network layer after ModuleList
- extend(): splice the two ModuleLists.
- insert(): inserts the network layer in the specified position in the ModuleList.
The following code iterates through the list generator to create 20 fully-connected layers in a loop, which is very convenient, except for the fact that theforward()function needs to be called manually for each network layer.
class ModuleList():
def __init__(self):
super(ModuleList, self).__init__()
= ([(10, 10) for i in range(20)])
def forward(self, x):
for i, linear in enumerate():
x = linear(x)
return x
net = ModuleList()
print(net)
fake_data = ((10, 10))
output = net(fake_data)
print(output)becontainer for wrapping a set of network layers to be called in an indexed way, with the following 5 main methods:
- clear(): clear ModuleDict
- items(): return iterable key-value pairs (key, value)
- keys(): return all keys of the dictionary.
- values(): returns all values of the dictionary.
- pop(): returns a pair of keys and removes them from the dictionary
The following model creates twoModuleDict：respond in singingThe network layer is executed by passing in the corresponding key during forward propagation.
class ModuleDict():
def __init__(self):
super(ModuleDict, self).__init__()
= ({
'conv': nn.Conv2d(10, 10, 3),
'pool': nn.MaxPool2d(3)
})
= ({
'relu': (),
'prelu': ()
})
def forward(self, x, choice, act):
x = [choice](x)
x = [act](x)
return x
net = ModuleDict()
fake_img = ((4, 10, 32, 32))
output = net(fake_img, 'conv', 'relu')
# output = net(fake_img, 'conv', 'prelu')
print(output)Container Summary
- : Sequential, where each network layer is executed in strict order, often used in block construction, where code calls during forward propagation become concise
- : iterative rows, often used for massive repetitive network construction, repeated construction via for loops
- : Indexed, commonly used for optional network layers
AlexNet implementation
AlexNet features the following:
- Replacing saturated activation functions with ReLU to mitigate gradient vanishing
- LRN (Local Response Normalization) is used to locally normalize the data and mitigate gradient vanishing.
- Improve network robustness and increase generalization with Dropout
- Using Data Augmentation, including TenCrop and some color modifications
The network structure of AlexNet can be divided into two parts: features and classifier.
It can be found in the Computer Vision LibraryI found the code for AlexNet in theto encapsulate the network layer.
class AlexNet():
def __init__(self, num_classes=1000):
super(AlexNet, self).__init__()
= (
nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
nn.Conv2d(64, 192, kernel_size=5, padding=2),
(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
nn.Conv2d(192, 384, kernel_size=3, padding=1),
(inplace=True),
nn.Conv2d(384, 256, kernel_size=3, padding=1),
(inplace=True),
nn.Conv2d(256, 256, kernel_size=3, padding=1),
(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
)
= nn.AdaptiveAvgPool2d((6, 6))
= (
(),
(256 * 6 * 6, 4096),
(inplace=True),
(),
(4096, 4096),
(inplace=True),
(4096, num_classes),
)
def forward(self, x):
x = (x)
x = (x)
x = (x, 1)
x = (x)
return xAbove is the detailed content of PyTorch model containers and AlexNet build examples, for more information about PyTorch AlexNet build please follow my other related articles!