"As we all know, videos can be P", today we will learn how to add filters to videos.
In iOS, there are generally two ways to process videos: GPUImage andAVFoundation 。
1. GPUImage
In previous articles, we already have a certain understanding of GPUImage. It was generally used to process the image data collected by the camera before, but it was just as convenient to process local videos.
Look directly at the code:
// movie NSString *path = [[NSBundle mainBundle] pathForResource:@"sample" ofType:@"mp4"]; NSURL *url = [NSURL fileURLWithPath:path]; GPUImageMovie *movie = [[GPUImageMovie alloc] initWithURL:url]; // filter GPUImageSmoothToonFilter *filter = [[GPUImageSmoothToonFilter alloc] init]; // view GPUImageView *imageView = [[GPUImageView alloc] initWithFrame:CGRectMake(0, 80, , )]; [ addSubview:imageView]; // chain [movie addTarget:filter]; [filter addTarget:imageView]; // processing [movie startProcessing];
There are only a few lines of the core code. GPUImageMovie is responsible for reading video files, GPUImageSmoothToonFilter is responsible for processing filter effects, and GPUImageView is responsible for displaying the final image.
String the three together through the filter chain, and then call the startProcessing method of GPUImageMovie to start processing.
Although GPUImage is simple to use, it has many disadvantages such as no sound, calling the UI on non-main threads, trouble exporting files, and inability to playback control.
Summary: Although GPUImage is very convenient to use, it has many shortcomings and does not meet the needs of the production environment.
2. AV Foundation
1. Use of AVPlayer
First, let’s review the easiest way to use AVPlayer:
NSURL *url = [[NSBundle mainBundle] URLForResource:@"sample" withExtension:@"mp4"]; AVURLAsset *asset = [AVURLAsset assetWithURL:url]; AVPlayerItem *playerItem = [[AVPlayerItem alloc] initWithAsset:asset]; AVPlayer *player = [[AVPlayer alloc] initWithPlayerItem:playerItem]; AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:player];
The first step is to build the AVPlayerItem, then create the AVPlayer through the AVPlayerItem, and finally create the AVPlayerLayer through the AVPlayer.
AVPlayerLayer is a subclass of CALayer and can be added to any layer. When AVPlayer calls the play method, the image can be rendered on AVPlayerLayer.
The way AVPlayer is used is very simple. However, in the above way, the most original image can only be rendered on AVPlayerLayer. If we want to process the original image while playing, we need to modify the rendering process of AVPlayer.
2. Modify the rendering process of AVPlayer
To modify the rendering process of AVPlayer, we need to start with AVPlayerItem, which is mainly divided into four steps:
Step 1: Customize the AVVideoCompositing class
AVVideoCompositing is a protocol that our custom class wants to implement. In this custom class, the original image of each frame can be obtained, processed and output.
In this protocol, the most important thing is to implement the startVideoCompositionRequest method:
//
- (void)startVideoCompositionRequest:(AVAsynchronousVideoCompositionRequest *)asyncVideoCompositionRequest {
dispatch_async(, ^{
@autoreleasepool {
if () {
[asyncVideoCompositionRequest finishCancelledRequest];
} else {
CVPixelBufferRef resultPixels = [self newRenderdPixelBufferForRequest:asyncVideoCompositionRequest];
if (resultPixels) {
[asyncVideoCompositionRequest finishWithComposedVideoFrame:resultPixels];
CVPixelBufferRelease(resultPixels);
} else {
// print error
}
}
}
});
}
The processed CVPixelBufferRef is obtained from the AVAsynchronousVideoCompositionRequest method and the output is obtained. Let’s take a look at the implementation of this method:
//
- (CVPixelBufferRef)newRenderdPixelBufferForRequest:(AVAsynchronousVideoCompositionRequest *)request {
CustomVideoCompositionInstruction *videoCompositionInstruction = (CustomVideoCompositionInstruction *);
NSArray<AVVideoCompositionLayerInstruction *> *layerInstructions = ;
CMPersistentTrackID trackID = ;
CVPixelBufferRef sourcePixelBuffer = [request sourceFrameByTrackID:trackID];
CVPixelBufferRef resultPixelBuffer = [videoCompositionInstruction applyPixelBuffer:sourcePixelBuffer];
if (!resultPixelBuffer) {
CVPixelBufferRef emptyPixelBuffer = [self createEmptyPixelBuffer];
return emptyPixelBuffer;
} else {
return resultPixelBuffer;
}
}
In this method, we get sourcePixelBuffer from AVAsynchronousVideoCompositionRequest via trackID, which is the original image of the current frame.
Then call the applyPixelBuffer method of videoCompositionInstruction, take sourcePixelBuffer as input to get the processed resultPixelBuffer. In other words, our processing operations on images all happen in the applyPixelBuffer method.
In the newRenderdPixelBufferForRequest method, we have obtained the original image of the current frame sourcePixelBuffer, and in fact, we can also directly process the image in this method.
Then why do you still need to put processing operations in CustomVideoCompositionInstruction?
Because when actually rendering, the creation of the instance of the custom AVVideoCompositing class is done internally. That is, we cannot access the final AVVideoCompositing object. Therefore, some dynamic modifications of rendering parameters cannot be performed. From the AVAsynchronousVideoCompositionRequest, the AVVideoCompositionInstruction object can be obtained, so we need to customize the AVVideoCompositionInstruction, so we can indirectly modify the rendering parameters by modifying the properties of the AVVideoCompositionInstruction.
Step 2: Customize AVVideoCompositionInstruction
The key point of this class is the implementation of the applyPixelBuffer method:
//
- (CVPixelBufferRef)applyPixelBuffer:(CVPixelBufferRef)pixelBuffer {
= pixelBuffer;
CVPixelBufferRef outputPixelBuffer = ;
CVPixelBufferRetain(outputPixelBuffer);
return outputPixelBuffer;
}
Here, the processing details of OpenGL ES are encapsulated into filter. The implementation details of this class can be ignored first, just know that it accepts the original CVPixelBufferRef and returns the processed CVPixelBufferRef.
Step 3: Build AVMutableVideoComposition
The built code is as follows:
= [self createVideoCompositionWithAsset:];
= [CustomVideoCompositing class];
- (AVMutableVideoComposition *)createVideoCompositionWithAsset:(AVAsset *)asset {
AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoCompositionWithPropertiesOfAsset:asset];
NSArray *instructions = ;
NSMutableArray *newInstructions = [NSMutableArray array];
for (AVVideoCompositionInstruction *instruction in instructions) {
NSArray *layerInstructions = ;
// TrackIDs
NSMutableArray *trackIDs = [NSMutableArray array];
for (AVVideoCompositionLayerInstruction *layerInstruction in layerInstructions) {
[trackIDs addObject:@()];
}
CustomVideoCompositionInstruction *newInstruction = [[CustomVideoCompositionInstruction alloc] initWithSourceTrackIDs:trackIDs timeRange:];
= ;
[newInstructions addObject:newInstruction];
}
= newInstructions;
return videoComposition;
}
The process of building AVMutableVideoComposition mainly does two things.
The first thing is to set the customVideoCompositorClass property of videoComposition to our customized CustomVideoCompositing .
The second thing is to first build the AVMutableVideoComposition object through the system-provided method videoCompositionWithPropertiesOfAsset, and then modify its instructions property to a custom CustomVideoCompositionInstruction type. (As mentioned in the "first step", you can get the CustomVideoCompositionInstruction object in CustomVideoCompositing in the future.)
Note: Here you can save CustomVideoCompositionInstruction, and then modify its properties to modify the rendering parameters.
Step 4: Build AVPlayerItem
With AVMutableVideoComposition, the following things are much simpler.
You only need to assign an additional videoComposition property when creating an AVPlayerItem.
= [[AVPlayerItem alloc] initWithAsset:]; = ;
In this way, the entire link is stringed together, and when AVPlayer is played, it can receive the CVPixelBufferRef of the original image in the applyPixelBuffer method of CustomVideoCompositionInstruction.
3. Apply filter effects
What you need to do in this step is: Add a filter effect on the CVPixelBufferRef and output the processed CVPixelBufferRef.
There are many ways to do this. Including but not limited to: OpenGL ES, CIImage, Metal, GPUImage, etc.
In order to use the GPUImageSmoothToonFilter used earlier, here is a way to introduce the GPUImage.
The key code is as follows:
- (CVPixelBufferRef)renderByGPUImage:(CVPixelBufferRef)pixelBuffer {
CVPixelBufferRetain(pixelBuffer);
__block CVPixelBufferRef output = nil;
runSynchronouslyOnVideoProcessingQueue(^{
[GPUImageContext useImageProcessingContext];
// (1)
GLuint textureID = [ convertYUVPixelBufferToTexture:pixelBuffer];
CGSize size = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
CVPixelBufferGetHeight(pixelBuffer));
[GPUImageContext setActiveShaderProgram:nil];
// (2)
GPUImageTextureInput *textureInput = [[GPUImageTextureInput alloc] initWithTexture:textureID size:size];
GPUImageSmoothToonFilter *filter = [[GPUImageSmoothToonFilter alloc] init];
[textureInput addTarget:filter];
GPUImageTextureOutput *textureOutput = [[GPUImageTextureOutput alloc] init];
[filter addTarget:textureOutput];
[textureInput processTextureWithFrameTime:kCMTimeZero];
// (3)
output = [ convertTextureToPixelBuffer:
textureSize:size];
[textureOutput doneWithTexture];
glDeleteTextures(1, &textureID);
});
CVPixelBufferRelease(pixelBuffer);
return output;
}
(1) The video frames read in at the beginning are in YUV format. First, convert the CVPixelBufferRef in YUV format to OpenGL texture.
(2) Use GPUImageTextureInput to construct the starting point of the filter chain, GPUImageSmoothToonFilter to add the filter effect, GPUImageTextureOutput to construct the end point of the filter chain, and ultimately output the OpenGL texture.
(3) Convert the processed OpenGL texture into CVPixelBufferRef.
In addition, since CIImage is easy to use, I will also mention the usage by the way.
The key code is as follows:
- (CVPixelBufferRef)renderByCIImage:(CVPixelBufferRef)pixelBuffer {
CVPixelBufferRetain(pixelBuffer);
CGSize size = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
CVPixelBufferGetHeight(pixelBuffer));
// (1)
CIImage *image = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer];
// (2)
CIImage *filterImage = [CIImage imageWithColor:[CIColor colorWithRed:255.0 / 255
green:245.0 / 255
blue:215.0 / 255
alpha:0.1]];
// (3)
image = [filterImage imageByCompositingOverImage:image];
// (4)
CVPixelBufferRef output = [ createPixelBufferWithSize:size];
[ render:image toCVPixelBuffer:output];
CVPixelBufferRelease(pixelBuffer);
return output;
}
(1) Convert CVPixelBufferRef to CIImage.
(2) Create a CIImage with transparency.
(3) Use system methods to superimpose CIImage.
(4) Convert the superimposed CIImage into CVPixelBufferRef.
4. Export the processed video
After the video is processed, you will eventually hope to export and save.
The exported code is also very simple:
= [[AVAssetExportSession alloc] initWithAsset: presetName:AVAssetExportPresetHighestQuality];
= ;
= AVFileTypeMPEG4;
= [NSURL fileURLWithPath:];
[ exportAsynchronouslyWithCompletionHandler:^{
// Save to album // ...
}];
The key point here is to set videoComposition to the AVMutableVideoComposition object constructed earlier, and then set the output path and file format to start exporting. After the export is successful, you can transfer the video file to the album.
Summary: Although AVFoundation is cumbersome, it has powerful functions and can easily export the results of video processing. It is the best choice for video processing.
Source code
PleaseGitHub View the full code on.
This is the end of this article about adding filters to videos in iOS. For more related content on adding filters to iOS, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!