This article shares the development process of iOS live APP for your reference. The specific content is as follows
1. General process of audio and video processing:
Data acquisition → Data encoding → Data transmission (streaming media server) → Decode data → Play and display
1. Data collection:
The camera and the pickup collect video and audio data, and the original data is obtained at this time
Techniques or agreements involved:
Camera: CCD, CMOS
Pickup: acousto-electric conversion device (micron head), audio amplifier circuit
2. Data encoding:
Use relevant hardware or software to encode (digitize) and process the original audio and video data (such as audio and video mixing, packaging, etc.) to obtain available audio and video data
Techniques or agreements involved:
Coding method: CBR, VBR
Encoding format
Video: H.265, H.264, MPEG-4, etc. Packaging containers include TS, MKV, AVI, MP4, etc.
Audio: G.711μ, AAC, Opus, etc., packaged with MP3, OGG, AAC, etc.
3. Data transmission:
The encoded audio and video data is transmitted, and early audio and video are transmitted through cables such as coaxial cables. After the development of IP network, it is better to use IP network to transmit
Techniques or agreements involved:
Transport protocols: RTP and RTCP, RTSP, RTMP, HTTP, HLS (HTTP Live Streaming), etc.
Control signaling: SIP and SDP, SNMP, etc.
4. Decode the data:
Use relevant hardware or software to decode the received encoded audio and video data to obtain images/sounds that can be directly displayed
Techniques or agreements involved:
Generally, the corresponding encoder will have corresponding decoders, and there are also some third-party decoding plug-ins.
5. Playback display:
Display the corresponding image or sound in the monitor (TV, surveillance screen, etc.) or speakers (headphones, speakers, etc.).
Techniques or agreements involved:
Monitors, speakers, 3D glasses, etc.
(II) Analysis of the working process of video streaming and video streaming:
1. Video streaming terminal
Pushing streaming means sending the collected audio and video data to the streaming media server through the streaming media protocol.
1. Choose a streaming protocol
Most of the live broadcast applications now use the RTMP protocol, and some use the HLS protocol.
To adopt the RTMP protocol, we need to look at the process of interacting with the streaming media server. The default port of the RTMP protocol is 1935, and TCP protocol is used. And you need to understand the packaging format of FLV.
The HLS protocol is used, because it involves slicing, the delay will be relatively large, and you need to understand the TS flow.
2. Collect audio and video data
When doing live broadcasts, the source of data is indispensable, which is to collect data from cameras and microphones.
The acquisition of audio and video data on iOS platform requires the use of a framework to obtain audio and video data from the callback of the captureSession session.
3. Hard-coded, soft-coded audio and video data
Soft encoding is to use CPU resources to compress audio and video data, and hard encoding is the opposite of it.
For soft encoding, the FFmpeg library is widely used in combination with the encoding library to implement it. FFmpeg+X624 is used to encode video data YUV/RGB output H264 data.
FFmpeg+fdk_aac to encode audio data PCM outputs AAC data.
4. Package audio and video data according to the selected streaming protocol
Pack the audio and video into packets.
5. Interact with the server to send packet data
According to the selected streaming media protocol, send the corresponding command to connect to the server. After the connection to the server is successful, you can send packet data.
Part 2. Pulling end
Pulling stream means obtaining audio and video data from the streaming media server.
1) Analytical protocol
The streaming media protocol (RTMP, HLS) used to parse the player side according to the URL.
2) Depacket
Decapsulation is the process of demux, separating audio and video data from container format (FLV, TS).
3) Decoding
Decoding means decompressing the obtained data and restoring it into original data. Decoding is to turn H264 into YUV and AAC into PCM.
Decoding can be done using soft decoding and hard decoding.
Soft decoding is to use CPU resources to decompress data, and the method used is FFmpeg decoding.
Hard decoding, for iOS platform, can be used (this framework can only be used on iOS 8.0 and above systems)
Hard decode video data. On the Android platform, MediaCodec can be used to hard decode video data.
4) Rendering data
OpenGL is used to render YUV data to present the video screen. The PCM is sent to the hardware resources of the device to play, producing sound.
iOS plays streaming audio, using Audio Queue, that is, using the framework.
3. iOS development of iOS live broadcast platform useful third-party resource sharing:
1. Real-time beauty reference website:/blog/real-time-filter/
2. Live test address:
HLS：http://live./live/hongkong.m3u8Phoenix TV *
SMTP: rtmp:///live/hks * TV
RSTP: rtsp:///tvblive/ Interactive News Station 1
3. HLS streaming tool download, used to process m3u8 audio and video sources HLS Streaming Tool Download
4. Live broadcast related SDK
1). VideoCore has documentation on Sina Cloud, and you can read the iOS streaming SDK usage documentation
2). NetEase Cloud Xin Live SDK seems to be charged, and the price is not cheap. Generally, companies may not know how to use it, but you can try it and learn it, which will help you learn!
3). Storm Cloud Live/apisdk/sdk/live/
4). Rongyun Live/live
5). Tencent Live SDK/solution/
5. Live broadcast related open source library
1). BeautifyFaceDemo image processing based on GPUImage, you can refer to it for beauty treatment
2). The open source RTMP Swift version can be learned! It seems that the author is from Japan or works in Japan and has written a lot of Japanese!
3). PLPlayerKit A player that does not rely on FFmpeg, supports RTMP
4）. HLS-Demo /yangchao0033/HLS-DemoHLS play demo
5）. SmarterStreaming /daniulive/SmarterStreamingLive SDK (supports private protocols and RTMP streaming, such as Windows streaming/android streaming/iOS streaming/windows player/android player/iOS player)
4. iOS development iOS live broadcast platform Detailed explanation of common video live broadcast protocols
1. RTMP (Real Time Messaging Protocol, real-time messaging protocol)
RTMP is an open protocol developed by Adobe Systems for audio, video and data transmission between Flash players and servers. It has three variants:
1) Plain text protocol working on TCP, using port 1935;
2) RTMPT is encapsulated in HTTP requests and can pass through the firewall;
3) RTMPS is similar to RTMPT, but uses HTTPS connection;
The RTMP protocol is used by Flash for the transmission of objects, videos and audio. This protocol is built on the TCP protocol or the polling HTTP protocol. The RTMP protocol is like a container used to hold data packets. These data can be either AMF format data or audio and video data in FLV. A single connection can transmit multiple network streams over different channels, and packets in these channels are transmitted in fixed-sized packets.
2. RTSP (Real Time Streaming Protocol, real-time streaming protocol)
RTSP defines how one-to-many applications can effectively transmit multimedia data over an IP network. RTSP provides an extensible framework where data sources can include real-time data and existing stored data. The purpose of this protocol is to control multiple data transmission connections, provide a way to select transmission channels such as UDP, multicast UDP and TCP, and provide a method for selecting a transmission mechanism based on RTP.
The RTSP syntax and operation are similar to HTTP/1.1, but do not emphasize time synchronization in particular, so network latency is more tolerant. The caching function of the proxy server is also applicable to RTSP, and because RTSP has a redirection function, the server that provides service can be switched according to the actual load situation to avoid delays caused by excessive load concentration on the same server.
3. RTP (Real-time Transport Protocol, real-time transmission protocol)
RTP is a transport layer protocol for multimedia data streams that details the standard packet formats for delivering audio and video over the Internet. The RTP protocol is often used in streaming media systems (with RTCP protocol), video conferencing and one-button communication systems (with H.323 or SIP), making it the technical foundation of the IP telephone industry.
RTP is built on the UDP protocol and is often used with RTCP. It does not provide on-time sending mechanisms or other quality of service (QoS) guarantees. It relies on low-level services to implement this process.
RTP does not guarantee or prevent unordered transmission, nor does it determine the reliability of the underlying network, but only sends, regardless of whether the transmission is lost or not, whether the receiver has received the packet. RTP implements orderly transmission. The sequence number in RTP allows the receiver to reorganize the sender's packet sequence. At the same time, the sequence number can also be used to determine the appropriate packet location. For example, in video decoding, sequential decoding is not required.
4. RTCP (Real-time Transport Control Protocol, real-time transmission control protocol)
RTCP is a supporting protocol for RTP, providing out-of-channel control for RTP media streams. RTCP and RTP work together to package and send multimedia data, regularly transmitting control data between multimedia streaming session participants.
The main function of RTCP is to provide feedback on the quality of service (QoS) provided by RTP, collect statistical information about related media connections, such as the number of transmitted bytes, the number of transmitted packets, the number of lost packets, one-way and two-way network delays, etc. Web applications can use the information provided by RTCP to improve service quality, such as limiting traffic or switching to codecs with smaller compression ratios.
The above is all the content of this article. I hope it will be helpful to everyone's study and I hope everyone will support me more.