I have been doing all kinds of things before, but I found that I haven't updated the article for a long time. As the end of the year approaches, I will sort out the content related to the video processing I have done recently~
Recently, we are doing video editing and processing related development, one of which is audio and video synthesis. The requirement is that users can choose to add videos in the album, then add a music clip, and adjust the original soundtrack and the added music volume in real time, and finally synthesize them into a video.
analyze
First of all, for video processing, universal ffmpeg can definitely be implemented, but relying on ffmpeg and using a magic-like statement to maintain and extend it is very limited. If you are not familiar with ffmpeg structure, a large number of C APIs will not be able to start. It is suitable for those who are familiar with ffmpeg and are familiar with AV Foundation.
The second best solution is AVFoundation, which is very powerful as an Apple audio and video editing tool. The official demo uses AVAudioEngine to achieve audio mixing, and can even edit pcm data, but the disadvantages are also obvious: 1 It has nothing to do with video, and you also have to start an AVAudioPlayerNode for playing (that's better to use AVAudioPlayer alone) 2 There is no need for audio such as "Bel Canto, Voice Change". So it is not considered, but it is still very powerful to achieve some special sound effects. If you are interested, you can go to the official website.demo-Using AVAudioEngine for Playback, Mixing and Recording (AVAEMixerSample) Check it out.
The last solution I chose is AVAudioMix. If you are familiar with AVPlayer and AVPlayerItem, you may notice that AVAudioMix exists as an attribute in the AVPlayerItem category.
/*! @property audioMix @abstract Indicates the audio mix parameters to be applied during playback @discussion The inputParameters of the AVAudioMix must have trackIDs that correspond to a track of the receiver's asset. Otherwise they will be ignored. (See for the declaration of AVAudioMixInputParameters and AVPlayerItem's asset property.) */ @property (nonatomic, copy, nullable) AVAudioMix *audioMix;
"Indicates the audio mix parameters to be applied during playback" indicates that audioMix can be set during playback. It should be noted that the trackID needs to correspond.
Added: Some people may think that the easiest thing is to create an AVPlayer to play videos and an AVAudioPlayer to play music at the same time; of course, this method can achieve basic needs, but perfectly synchronizing the status of these two players will be a problem, and in the end, it will have to go through the process of mixing and writing files, which is logically very bad.
Playback implementation
In order to clearly express the statement, the code that is not very relevant to AVPlayer and other codes are omitted, and you can also download my demo to view all the content.
The process is as follows:
1 Create an AVURLAsset for video and audio
AVURLAsset *videoAsset = [AVURLAsset assetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"demo" ofType:@"mp4"]]]; AVURLAsset *musicAsset = [AVURLAsset assetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"music" ofType:@"mp3"]]];
2 Declare and instantiate the core classes of audio and video processing
@property (nonatomic, readwrite, strong) AVMutableComposition *composition; @property (nonatomic, readwrite, strong) AVMutableVideoComposition *videoComposition; @property (nonatomic, readwrite, strong) AVMutableAudioMix *audioMix; AVMutableComposition *composition = [AVMutableComposition composition]; AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition]; AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
3 Create 1 video processing track and 2 audio processing tracks (video soundtrack + added music tracks)
AVMutableCompositionTrack *compositionVideoTracks = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid]; AVMutableCompositionTrack *compositionAudioTracks[2]; compositionAudioTracks[0] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; compositionAudioTracks[1] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
4 Instantiate a video track and 2 audio tracks based on the previously created video and audio resources (AVURLAsset)
AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
[compositionVideoTracks insertTimeRange: ofTrack:videoTrack atTime:kCMTimeZero error:&error];
AVAssetTrack *audioTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
[_comTrack1 insertTimeRange: ofTrack:audioTrack atTime:kCMTimeZero error:&error];
AVAssetTrack *musicTrack = [[ tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
[_comTrack2 insertTimeRange: ofTrack:musicTrack atTime:kCMTimeZero error:&error];
5 Configure the AVMutableAudioMix parameter, note that the trackID here must be the trackID corresponding to the AVMutableCompositionTrack created above, not the trackID in the AVAssetTrack. I had a very strange problem using AVAssetTrack before, and then*Found this solution on
NSMutableArray<AVAudioMixInputParameters *> *trackMixArray = [NSMutableArray<AVAudioMixInputParameters *> array];
{
AVMutableAudioMixInputParameters *trackMix1 = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:_comTrack1];
= _comTrack1.trackID;
[trackMix1 setVolume:_videoVolume atTime:kCMTimeZero];
[trackMixArray addObject:trackMix1];
AVMutableAudioMixInputParameters *trackMix2 = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:_comTrack2];
= _comTrack2.trackID;
[trackMix2 setVolume:_musicVolume atTime:kCMTimeZero];
[trackMixArray addObject:trackMix2];
}
= trackMixArray;
6 Build AVPlayerItem, set the asset and most important audioMix, and then hand it over to AVPlayer to play videos and music at the same time!
- (AVPlayerItem *)playerItem {
if (!_currentItem) {
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:];
= ;
= ;
_currentItem = playerItem;
}
return _currentItem;
}
7 Adjust the volume during playback. This is actually the same as in step 5. Reconfigure the AVMutableAudioMix parameter and assign the value to AVPlayerItem. The same applies to setting the music volume.
- (void)setVideoVolume:(CGFloat)volume {
NSMutableArray *allAudioParams = [NSMutableArray array];
AVMutableAudioMixInputParameters *audioInputParams =
[AVMutableAudioMixInputParameters audioMixInputParameters];
[audioInputParams setTrackID:_comTrack1.trackID];
_videoVolume = volume;
[audioInputParams setVolume:_videoVolume atTime:kCMTimeZero];
[allAudioParams addObject:audioInputParams];
AVMutableAudioMixInputParameters *audioInputParams2 =
[AVMutableAudioMixInputParameters audioMixInputParameters];
[audioInputParams2 setTrackID:_comTrack2.trackID];
[audioInputParams2 setVolume:_musicVolume atTime:kCMTimeZero];
[allAudioParams addObject:audioInputParams2];
AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
[audioMix setInputParameters:allAudioParams];
[_currentItem setAudioMix:audioMix];
}
Export implementation
Here, use AVAssetExportSession directly to export videos. It is the same as setting the audioMix attribute of AVPlayerItem. You can export mixed videos by setting audioMix to the AVAssetExportSession instance.
NSURL *outputFileUrl = [NSURL fileURLWithPath:outputPath];
AVAssetExportSession *_assetExport =[[AVAssetExportSession alloc]initWithAsset: presetName:AVAssetExportPreset1280x720];
_assetExport.outputFileType = AVFileTypeMPEG4;
_assetExport.audioMix = _currentItem.audioMix;
_assetExport.outputURL = outputFileUrl;
_assetExport.shouldOptimizeForNetworkUse = YES;
[_assetExport exportAsynchronouslyWithCompletionHandler:^{
//
}];
Finally posted the demo address/lucifron1994/VideoMixAudioDemo
The above is all the content of this article. I hope it will be helpful to everyone's study and I hope everyone will support me more.