目录
- 1.视频流编码器（VideoStreamEncoder）
- 1.1 视频流编码器接口（VideoStreamEncoderInterface）
- 1.2 资源类（Resource）
- 1.3 已编码图像的回调（EncodedImageCallback）
- 1.4 视频资源监听器（VideoSourceRestrictionsListener）
1.视频流编码器（VideoStreamEncoder）
VideoStreamEncoder是编码流程当中至关重要的一个类，它实现的功能包括：
（1）开始编码器，暂停编码器，停止编码器
（2）设置编码器（根据config）
（3）调控编码参数，如码控参数，帧率等等
（4）调控前向纠错器
（5）视频帧的传输和接收
（6）根据网络环境，调控丢帧
（7）根据编码情况，调控屯帧
（8）根据情况，调控编码质量（因为网络环境，编码器情况，可能会调整编码策略，影响编码质量）
（9）对已编码图像进行后处理
（10）统计编码信息
可以说，在视频编码流程中，VideoStreamEncoder是获取了视频帧之后，第一个会使用到的类。这个类十分综合，大多数的编码过程都与这个类相关，所以其声明比较长，这里做简单记录，深入到每个模块之后再分析。
// VideoStreamEncoder represent a video encoder that accepts raw video frames as
// input and produces an encoded bit stream.
// Usage:
// Instantiate.
// Call SetSink.
// Call SetSource.
// Call ConfigureEncoder with the codec settings.
// Call Stop() when done.
/*VideoStreamEncoder表示了一个视频编码器，它接收原始视频帧作为输入并且产生编码后的比特流用法:(1) 创建实例(2) SetSink，设置接收器(3) SetSource，设置资源(4) ConfigureEncoder，配置编码器参数(5) Stop()，结束流程
*/
class VideoStreamEncoder : public VideoStreamEncoderInterface,private EncodedImageCallback,public VideoSourceRestrictionsListener {public:// TODO(bugs.webrtc.org/12000): Reporting of VideoBitrateAllocation is being// deprecated. Instead VideoLayersAllocation should be reported.enum class BitrateAllocationCallbackType {kVideoBitrateAllocation,kVideoBitrateAllocationWhenScreenSharing,kVideoLayersAllocation};VideoStreamEncoder(const Environment& env,uint32_t number_of_cores,VideoStreamEncoderObserver* encoder_stats_observer,const VideoStreamEncoderSettings& settings,std::unique_ptr<OveruseFrameDetector> overuse_detector,std::unique_ptr<FrameCadenceAdapterInterface> frame_cadence_adapter,std::unique_ptr<webrtc::TaskQueueBase, webrtc::TaskQueueDeleter>encoder_queue,BitrateAllocationCallbackType allocation_cb_type,webrtc::VideoEncoderFactory::EncoderSelectorInterface* encoder_selector =nullptr);~VideoStreamEncoder() override;VideoStreamEncoder(const VideoStreamEncoder&) = delete;VideoStreamEncoder& operator=(const VideoStreamEncoder&) = delete;// 添加自适应资源void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) override;// 获取自适应资源std::vector<rtc::scoped_refptr<Resource>> GetAdaptationResources() override;// 设置资源void SetSource(rtc::VideoSourceInterface<VideoFrame>* source,const DegradationPreference& degradation_preference) override;// 设置接收器void SetSink(EncoderSink* sink, bool rotation_applied) override;// TODO(perkj): Can we remove VideoCodec.startBitrate ?// 设置起始码率void SetStartBitrate(int start_bitrate_bps) override;// 设置前向纠错控制器void SetFecControllerOverride(FecControllerOverride* fec_controller_override) override;// 配置编码器void ConfigureEncoder(VideoEncoderConfig config,size_t max_data_payload_length) override;void ConfigureEncoder(VideoEncoderConfig config,size_t max_data_payload_length,SetParametersCallback callback) override;// Permanently stop encoding. After this method has returned, it is// guaranteed that no encoded frames will be delivered to the sink.// 永久性停止编码，此方法返回后，保证不会有编码后的帧传递给接收器void Stop() override;// 传递关键帧void SendKeyFrame(const std::vector<VideoFrameType>& layers = {}) override;// 检测到丢包，需要调整编码策略，例如降低分辨率或帧率void OnLossNotification(const VideoEncoder::LossNotification& loss_notification) override;// 检测到码率变化，需要调整编码参数void OnBitrateUpdated(DataRate target_bitrate,DataRate stable_target_bitrate,DataRate target_headroom,uint8_t fraction_lost,int64_t round_trip_time_ms,double cwnd_reduce_ratio) override;// 根据拥塞窗口减少比例，来调整目标比特率，避免丢包和网络拥塞DataRate UpdateTargetBitrate(DataRate target_bitrate,double cwnd_reduce_ratio);protected:friend class VideoStreamEncoderFrameCadenceRestrictionTest;// Used for testing. For example the `ScalingObserverInterface` methods must// be called on `encoder_queue_`.TaskQueueBase* encoder_queue() { return encoder_queue_.get(); }// 检测到视频限制性资源发生变化void OnVideoSourceRestrictionsUpdated(VideoSourceRestrictions restrictions,const VideoAdaptationCounters& adaptation_counters,rtc::scoped_refptr<Resource> reason,const VideoSourceRestrictions& unfiltered_restrictions) override;// Used for injected test resources.// TODO(eshr): Move all adaptation tests out of VideoStreamEncoder tests.// 注入自适应资源，这里与AddAdaptationResource不同，还需要入参reason，即更加// 精准的调控void InjectAdaptationResource(rtc::scoped_refptr<Resource> resource,VideoAdaptationReason reason);// 注入自适应限制void InjectAdaptationConstraint(AdaptationConstraint* adaptation_constraint);void AddRestrictionsListenerForTesting(VideoSourceRestrictionsListener* restrictions_listener);void RemoveRestrictionsListenerForTesting(VideoSourceRestrictionsListener* restrictions_listener);private:class CadenceCallback : public FrameCadenceAdapterInterface::Callback {public:explicit CadenceCallback(VideoStreamEncoder& video_stream_encoder): video_stream_encoder_(video_stream_encoder) {}// FrameCadenceAdapterInterface::Callback overrides.// 下面是对FrameCadenceAdapterInterface::Callback当中函数的重载void OnFrame(Timestamp post_time,bool queue_overload,const VideoFrame& frame) override {// 使用VideoStreamEncoder当中的OnFrame()函数重载video_stream_encoder_.OnFrame(post_time, queue_overload, frame);}void OnDiscardedFrame() override {video_stream_encoder_.OnDiscardedFrame();}void RequestRefreshFrame() override {video_stream_encoder_.RequestRefreshFrame();}private:VideoStreamEncoder& video_stream_encoder_;};// VideoFrameInfo记录长宽，是否是纹理等信息class VideoFrameInfo {public:VideoFrameInfo(int width, int height, bool is_texture): width(width), height(height), is_texture(is_texture) {}int width;int height;bool is_texture;int pixel_count() const { return width * height; }};// 编码速率设定struct EncoderRateSettings {EncoderRateSettings();EncoderRateSettings(const VideoBitrateAllocation& bitrate,double framerate_fps,DataRate bandwidth_allocation,DataRate encoder_target,DataRate stable_encoder_target);bool operator==(const EncoderRateSettings& rhs) const;bool operator!=(const EncoderRateSettings& rhs) const;// 码率控制参数VideoEncoder::RateControlParameters rate_control;// This is the scalar target bitrate before the VideoBitrateAllocator, i.e.// the `target_bitrate` argument of the OnBitrateUpdated() method. This is// needed because the bitrate allocator may truncate the total bitrate and a// later call to the same allocator instance, e.g.// |using last_encoder_rate_setings_->bitrate.get_sum_bps()|, may trick it// into thinking the available bitrate has decreased since the last call./*这是`VideoBitrateAllocator`之前的标量目标比特率，即`OnBitrateUpdated()`方法的`target_bitrate`参数。这是因为比特率分配器可能会截断总比特率，而在稍后对同一分配器实例的调用中，例如`|using last_encoder_rate_setings_->bitrate.get_sum_bps()|`，可能会误导它认为自上次调用以来可用的比特率已经减少。*/DataRate encoder_target;DataRate stable_encoder_target;};class DegradationPreferenceManager;// 重新配置编码器void ReconfigureEncoder() RTC_RUN_ON(encoder_queue_);// 检测到编码器配置发生变化void OnEncoderSettingsChanged() RTC_RUN_ON(encoder_queue_);// 检测到帧void OnFrame(Timestamp post_time,bool queue_overload,const VideoFrame& video_frame);// 检测到丢弃帧void OnDiscardedFrame();// 需要刷新帧void RequestRefreshFrame();// 可能会编码视频帧void MaybeEncodeVideoFrame(const VideoFrame& frame,int64_t time_when_posted_in_ms);// 编码视频帧void EncodeVideoFrame(const VideoFrame& frame,int64_t time_when_posted_in_ms);// Indicates whether frame should be dropped because the pixel count is too// large for the current bitrate configuration.// 在当前码率配置下，根据pixle count来检查当前帧是否需要被丢弃bool DropDueToSize(uint32_t pixel_count) const RTC_RUN_ON(encoder_queue_);// Implements EncodedImageCallback.// 接收编码器编码之后的图像EncodedImageCallback::Result OnEncodedImage(const EncodedImage& encoded_image,const CodecSpecificInfo* codec_specific_info) override;// 丢帧的回调函数，需要给出reasonvoid OnDroppedFrame(EncodedImageCallback::DropReason reason) override;// 暂停编码器bool EncoderPaused() const;// 标记丢帧开始void TraceFrameDropStart();// 标记丢帧结束void TraceFrameDropEnd();// Returns a copy of `rate_settings` with the `bitrate` field updated using// the current VideoBitrateAllocator.// 返回一个`rate_settings`的副本，其中`bitrate`字段已使用当前的`VideoBitrateAllocator`更新。EncoderRateSettings UpdateBitrateAllocation(const EncoderRateSettings& rate_settings) RTC_RUN_ON(encoder_queue_);// 获取输入帧率Fpsuint32_t GetInputFramerateFps() RTC_RUN_ON(encoder_queue_);// 设置编码器速率void SetEncoderRates(const EncoderRateSettings& rate_settings)RTC_RUN_ON(encoder_queue_);// 进行编码后处理void RunPostEncode(const EncodedImage& encoded_image,int64_t time_sent_us,int temporal_index,DataSize frame_size);// 释放编码器void ReleaseEncoder() RTC_RUN_ON(encoder_queue_);// After calling this function `resource_adaptation_processor_` will be null.// 关闭资源自适应队列，resource_adaptation_processor会设置为nullvoid ShutdownResourceAdaptationQueue();// 申请更换编码器void RequestEncoderSwitch() RTC_RUN_ON(encoder_queue_);// Augments an EncodedImage received from an encoder with parsable// information.// 修改或增强已编码图像EncodedImage AugmentEncodedImage(const EncodedImage& encoded_image,const CodecSpecificInfo* codec_specific_info);// 处理已丢弃的帧void ProcessDroppedFrame(const VideoFrame& frame,VideoStreamEncoderObserver::DropReason reason)RTC_RUN_ON(encoder_queue_);const Environment env_;TaskQueueBase* const worker_queue_;const int number_of_cores_;// sink是接收器EncoderSink* sink_ = nullptr;// VideoStreamEncoder配置const VideoStreamEncoderSettings settings_;// 比特分配回调的类型const BitrateAllocationCallbackType allocation_cb_type_;// 码率控制配置const RateControlSettings rate_control_settings_;webrtc::VideoEncoderFactory::EncoderSelectorInterface* constencoder_selector_from_constructor_;std::unique_ptr<VideoEncoderFactory::EncoderSelectorInterface> constencoder_selector_from_factory_;// Pointing to either encoder_selector_from_constructor_ or// encoder_selector_from_factory_ but can be nullptr.// 编码器选择器VideoEncoderFactory::EncoderSelectorInterface* const encoder_selector_;// 视频流编码器观察者的行为VideoStreamEncoderObserver* const encoder_stats_observer_;// Adapter that avoids public inheritance of the cadence adapter's callback// interface.// 适配器，避免了公开继承节拍适配器的回调接口。CadenceCallback cadence_callback_{*this};// Frame cadence encoder adapter. Frames enter this adapter first, and it then// forwards them to our OnFrame method.// 帧节拍编码器适配器。帧首先进入这个适配器，然后它将帧转发到我们的OnFrame方法。std::unique_ptr<FrameCadenceAdapterInterface> frame_cadence_adapter_RTC_GUARDED_BY(encoder_queue_) RTC_PT_GUARDED_BY(encoder_queue_);// 编码器配置项VideoEncoderConfig encoder_config_ RTC_GUARDED_BY(encoder_queue_);// 编码器std::unique_ptr<VideoEncoder> encoder_ RTC_GUARDED_BY(encoder_queue_)RTC_PT_GUARDED_BY(encoder_queue_);bool encoder_initialized_ = false;// 码率分配器std::unique_ptr<VideoBitrateAllocator> rate_allocator_RTC_GUARDED_BY(encoder_queue_) RTC_PT_GUARDED_BY(encoder_queue_);int max_framerate_ RTC_GUARDED_BY(encoder_queue_) = -1;// Set when ConfigureEncoder has been called in order to lazy reconfigure the// encoder on the next frame.// 当调用了ConfigureEncoder时设置，以便在下一帧上延迟重新配置编码器。bool pending_encoder_reconfiguration_ RTC_GUARDED_BY(encoder_queue_) = false;// Set when configuration must create a new encoder object, e.g.,// because of a codec change.// 当配置必须创建一个新的编码器对象时设置，例如，由于编解码器变更。bool pending_encoder_creation_ RTC_GUARDED_BY(encoder_queue_) = false;// 编码器配置的回调函数absl::InlinedVector<SetParametersCallback, 2> encoder_configuration_callbacks_RTC_GUARDED_BY(encoder_queue_);// 上一帧的信息std::optional<VideoFrameInfo> last_frame_info_ RTC_GUARDED_BY(encoder_queue_);// 边界裁剪的width和heightint crop_width_ RTC_GUARDED_BY(encoder_queue_) = 0;int crop_height_ RTC_GUARDED_BY(encoder_queue_) = 0;// 编码器目标比特率std::optional<uint32_t> encoder_target_bitrate_bps_RTC_GUARDED_BY(encoder_queue_);// 最大数据负载长度size_t max_data_payload_length_ RTC_GUARDED_BY(encoder_queue_) = 0;// 上一个编码器的码率设定std::optional<EncoderRateSettings> last_encoder_rate_settings_RTC_GUARDED_BY(encoder_queue_);// 编码器是否停止并丢帧bool encoder_paused_and_dropped_frame_ RTC_GUARDED_BY(encoder_queue_) = false;// Set to true if at least one frame was sent to encoder since last encoder// initialization.// 如果自上次编码器初始化以来至少有一帧已发送到编码器，则设置为truebool was_encode_called_since_last_initialization_RTC_GUARDED_BY(encoder_queue_) = false;// 编码器是否失败bool encoder_failed_ RTC_GUARDED_BY(encoder_queue_) = false;// Used to make sure incoming time stamp is increasing for every frame.// 用于确保每个帧的到达时间戳都在增加。int64_t last_captured_timestamp_ RTC_GUARDED_BY(encoder_queue_) = 0;// Delta used for translating between NTP and internal timestamps.// 用于在NTP时间戳和内部时间戳之间转换的差值。const int64_t delta_ntp_internal_ms_ RTC_GUARDED_BY(encoder_queue_);// 最后一次记录帧日志的时间戳int64_t last_frame_log_ms_ RTC_GUARDED_BY(encoder_queue_);// 已捕捉帧的数量int captured_frame_count_ RTC_GUARDED_BY(encoder_queue_) = 0;// 由于网络阻塞丢弃的帧数量int dropped_frame_cwnd_pushback_count_ RTC_GUARDED_BY(encoder_queue_) = 0;// 由于编码器阻塞丢弃的帧数量int dropped_frame_encoder_block_count_ RTC_GUARDED_BY(encoder_queue_) = 0;// 屯帧的数量（待编码帧数量）std::optional<VideoFrame> pending_frame_ RTC_GUARDED_BY(encoder_queue_);// 屯帧的时间int64_t pending_frame_post_time_us_ RTC_GUARDED_BY(encoder_queue_) = 0;// 存储了视频帧中自上次更新以来发生变化的区域VideoFrame::UpdateRect accumulated_update_rect_RTC_GUARDED_BY(encoder_queue_);// 指示accumulated_update_rect_是否包含有效的更新区域信息bool accumulated_update_rect_is_valid_ RTC_GUARDED_BY(encoder_queue_) = true;// 前向纠错控制FecControllerOverride* fec_controller_override_RTC_GUARDED_BY(encoder_queue_) = nullptr;// 记录最近一次参数更新的时间std::optional<int64_t> last_parameters_update_ms_RTC_GUARDED_BY(encoder_queue_);// 记录最近一次编码操作的时间戳std::optional<int64_t> last_encode_info_ms_ RTC_GUARDED_BY(encoder_queue_);// 编码器信息VideoEncoder::EncoderInfo encoder_info_ RTC_GUARDED_BY(encoder_queue_);// 编码器，存储视频编解码器的相关配置信息，包括编解码器类型、编码设置、分辨率、帧率、比特率等参数VideoCodec send_codec_ RTC_GUARDED_BY(encoder_queue_);// 帧丢弃器FrameDropper frame_dropper_ RTC_GUARDED_BY(encoder_queue_);// If frame dropper is not force disabled, frame dropping might still be// disabled if VideoEncoder::GetEncoderInfo() indicates that the encoder has a// trusted rate controller. This is determined on a per-frame basis, as the// encoder behavior might dynamically change./*如果帧丢弃器没有被强制禁用，那么即使 VideoEncoder::GetEncoderInfo() 表示编码器拥有一个可信的速率控制器，帧丢弃可能仍然被禁用。这是逐帧决定的，因为编码器的行为可能会动态变化。*/// 强制禁用frameDropperbool force_disable_frame_dropper_ RTC_GUARDED_BY(encoder_queue_) = false;// Incremented on worker thread whenever `frame_dropper_` determines that a// frame should be dropped. Decremented on whichever thread runs// OnEncodedImage(), which is only called by one thread but not necessarily// the worker thread./*每当 `frame_dropper_` 确定应该丢弃一帧时，在工作线程上增加。在运行 `OnEncodedImage()` 的线程上减少，这个函数只由一个线程调用，但不一定是工作线程。*/std::atomic<int> pending_frame_drops_{0};// Congestion window frame drop ratio (drop 1 in every// cwnd_frame_drop_interval_ frames).// 拥塞窗口帧丢弃比例（每 cwnd_frame_drop_interval_ 帧丢弃 1 帧）。std::optional<int> cwnd_frame_drop_interval_ RTC_GUARDED_BY(encoder_queue_);// Frame counter for congestion window frame drop.// 用于拥塞窗口帧丢弃的帧计数器int cwnd_frame_counter_ RTC_GUARDED_BY(encoder_queue_) = 0;// 码率调节器std::unique_ptr<EncoderBitrateAdjuster> bitrate_adjuster_RTC_GUARDED_BY(encoder_queue_);// TODO(sprang): Change actually support keyframe per simulcast stream, or// turn this into a simple bool `pending_keyframe_request_`.// 下一帧的类型std::vector<VideoFrameType> next_frame_types_ RTC_GUARDED_BY(encoder_queue_);// 用于处理和写入与视频帧编码相关的元数据FrameEncodeMetadataWriter frame_encode_metadata_writer_{this};// Provides video stream input states: current resolution and frame rate.// 处理视频流输入状态VideoStreamInputStateProvider input_state_provider_;// 视频流适应器，用于适配不同的视频流需求，包括分辨率、帧率和比特率const std::unique_ptr<VideoStreamAdapter> video_stream_adapter_RTC_GUARDED_BY(encoder_queue_);// Responsible for adapting input resolution or frame rate to ensure resources// (e.g. CPU or bandwidth) are not overused. Adding resources can occur on any// thread.// 负责调整输入分辨率或帧率，以确保资源（如CPU或带宽）不被过度使用。增加资源可以在任何线程上进行。// 资源自适应处理器std::unique_ptr<ResourceAdaptationProcessorInterface>resource_adaptation_processor_ RTC_GUARDED_BY(encoder_queue_);// 管理质量下降的偏好设置，因为在调控编码策略时，很有可能会带来视频质量的下降，例如快速算法等std::unique_ptr<DegradationPreferenceManager> degradation_preference_manager_RTC_GUARDED_BY(encoder_queue_);// 自适应限制std::vector<AdaptationConstraint*> adaptation_constraints_RTC_GUARDED_BY(encoder_queue_);// Handles input, output and stats reporting related to VideoStreamEncoder// specific resources, such as "encode usage percent" measurements and "QP// scaling". Also involved with various mitigations such as initial frame// dropping.// The manager primarily operates on the `encoder_queue_` but its lifetime is// tied to the VideoStreamEncoder (which is destroyed off the encoder queue)// and its resource list is accessible from any thread./*处理与VideoStreamEncoder相关的输入、输出和统计报告，特别是“编码使用百分比”测量和“QP缩放”等资源。还涉及各种缓解措施，如初始帧丢弃。该管理器主要在`encoder_queue_`上操作，但其生命周期与VideoStreamEncoder（在编码器队列外销毁）相关联，其资源列表可以从任何线程访问。*/VideoStreamEncoderResourceManager stream_resource_manager_RTC_GUARDED_BY(encoder_queue_);// 额外的资源std::vector<rtc::scoped_refptr<Resource>> additional_resources_RTC_GUARDED_BY(encoder_queue_);// Carries out the VideoSourceRestrictions provided by the// ResourceAdaptationProcessor, i.e. reconfigures the source of video frames// to provide us with different resolution or frame rate.// This class is thread-safe./*执行由ResourceAdaptationProcessor提供的VideoSourceRestrictions，即重新配置视频帧的源，为我们提供不同的分辨率或帧率。这个类是线程安全的。*/VideoSourceSinkController video_source_sink_controller_RTC_GUARDED_BY(worker_queue_);// Default bitrate limits in EncoderInfoSettings allowed.// 默认的比特率限制const bool default_limits_allowed_;// QP parser is used to extract QP value from encoded frame when that is not// provided by encoder.// QP解析器用于在编码器未提供QP值时，从编码帧中提取QP值。QpParser qp_parser_;const bool qp_parsing_allowed_;// The quality convergence controller is used to determine if a codec has// reached its target quality. This is used for screenshare to determine when// there's no need to continue encoding the same repeated frame./*质量收敛控制器用于确定一个编解码器是否已经达到了其目标质量。这在屏幕共享中用于判断何时不需要继续编码相同的重复帧。*/QualityConvergenceController quality_convergence_controller_RTC_GUARDED_BY(encoder_queue_);// Enables encoder switching on initialization failures.// 在初始化失败时启用编码器切换。bool switch_encoder_on_init_failures_;const std::optional<int> vp9_low_tier_core_threshold_;const std::optional<int> experimental_encoder_thread_limit_;// This is a copy of restrictions (glorified max_pixel_count) set by// OnVideoSourceRestrictionsUpdated. It is used to scale down encoding// resolution if needed when using requested_resolution.//// TODO(webrtc:14451) Split video_source_sink_controller_// so that ownership on restrictions/wants is kept on &encoder_queue_, that// these extra copies would not be needed./*这是由OnVideoSourceRestrictionsUpdated设置的限制（实际上是最大像素计数）的副本。当使用requested_resolution时，如果需要，它被用来降低编码分辨率。TODO(webrtc:14451) 分离video_source_sink_controller_，以便将限制/需求的所有权保留在&encoder_queue_上，这样就不需要这些额外的副本了。*/std::optional<VideoSourceRestrictions> latest_restrictions_RTC_GUARDED_BY(encoder_queue_);// Used to cancel any potentially pending tasks to the worker thread.// Refrenced by tasks running on `encoder_queue_` so need to be destroyed// after stopping that queue. Must be created and destroyed on// `worker_queue_`./*用于取消任何可能挂起的工作任务线程任务。被在`encoder_queue_`上运行的任务引用，因此需要在停止该队列后销毁。必须在`worker_queue_`上创建和销毁。*/ScopedTaskSafety task_safety_;// 编码队列std::unique_ptr<TaskQueueBase, TaskQueueDeleter> encoder_queue_;// Required for automatic corruption detection.// 自动检测数据损坏std::unique_ptr<FrameInstrumentationGenerator>frame_instrumentation_generator_;
};
1.1 视频流编码器接口（VideoStreamEncoderInterface）
VideoStreamEncoder的父类是VideoStreamEncoderInterface，这里定义了一些基础的函数。类的声明位于video/video_stream_encoder_interface.h中
// This interface represents a class responsible for creating and driving the
// encoder(s) for a single video stream. It is also responsible for adaptation
// decisions related to video quality, requesting reduced frame rate or
// resolution from the VideoSource when needed.
// TODO(bugs.webrtc.org/8830): This interface is under development. Changes
// under consideration include:
//
// 1. Taking out responsibility for adaptation decisions, instead only reporting
// per-frame measurements to the decision maker.
//
// 2. Moving responsibility for simulcast and for software fallback into this
// class.
/*这个接口代表一个负责创建和驱动单个视频流的编码器（encoder）的类。它还负责与视频质量相关的适应性决策，例如在需要时从 VideoSource 请求降低帧率或分辨率。TODO(bugs.webrtc.org/8830): 此接口正在开发中。正在考虑的变更包括：1. 移除适应性决策的责任，改为仅向决策者报告每帧的测量数据。2. 将simulcast（同时向多个接收者发送不同质量的视频流）和软件回退的责任转移到这个类中。
*/
class VideoStreamEncoderInterface {public:// Interface for receiving encoded video frames and notifications about// configuration changes.// 接收编码后的视频帧以及关于配置变更通知的接口class EncoderSink : public EncodedImageCallback {public:// 检测到编码器配置发生变化virtual void OnEncoderConfigurationChanged(std::vector<VideoStream> streams,bool is_svc,VideoEncoderConfig::ContentType content_type,int min_transmit_bitrate_bps) = 0;// 检测到码率分配发生变化virtual void OnBitrateAllocationUpdated(const VideoBitrateAllocation& allocation) = 0;// 检测到视频层级分配发生变化virtual void OnVideoLayersAllocationUpdated(VideoLayersAllocation allocation) = 0;};virtual ~VideoStreamEncoderInterface() = default;// If the resource is overusing, the VideoStreamEncoder will try to reduce// resolution or frame rate until no resource is overusing.// TODO(https://crbug.com/webrtc/11565): When the ResourceAdaptationProcessor// is moved to Call this method could be deleted altogether in favor of// Call-level APIs only// 如果资源使用过量，VideoStreamEncoder 将尝试降低分辨率或帧率，直到没有资源使用过量为止。// TODO(https://crbug.com/webrtc/11565): 当 ResourceAdaptationProcessor 被移动到 // Call 时，这个方法可能会被完全删除，转而只使用 Call 级别的 API。virtual void AddAdaptationResource(rtc::scoped_refptr<Resource> resource) = 0;virtual std::vector<rtc::scoped_refptr<Resource>>GetAdaptationResources() = 0;// Sets the source that will provide video frames to the VideoStreamEncoder's// OnFrame method. `degradation_preference` control whether or not resolution// or frame rate may be reduced. The VideoStreamEncoder registers itself with// `source`, and signals adaptation decisions to the source in the form of// VideoSinkWants.// TODO(bugs.webrtc.org/14246): When adaptation logic is extracted from this// class, it no longer needs to know the source./*设置提供视频帧给 VideoStreamEncoder 的 OnFrame 方法的源。`degradation_preference` 控制是否允许降低分辨率或帧率。VideoStreamEncoder 在 `source` 中注册自己，并以 VideoSinkWants 的形式向源发出适应性决策信号。TODO(bugs.webrtc.org/14246): 当适应性逻辑从这个类中提取出来时，它就不再需要知道源。*/virtual void SetSource(rtc::VideoSourceInterface<VideoFrame>* source,const DegradationPreference& degradation_preference) = 0;// Sets the `sink` that gets the encoded frames. `rotation_applied` means// that the source must support rotation. Only set `rotation_applied` if the// remote side does not support the rotation extension./*设置接收编码帧的 `sink`。`rotation_applied` 表示源必须支持旋转。只有在远端不支持旋转扩展时才设置 `rotation_applied`。*/virtual void SetSink(EncoderSink* sink, bool rotation_applied) = 0;// Sets an initial bitrate, later overriden by OnBitrateUpdated. Mainly// affects the resolution of the initial key frame: If incoming frames are// larger than reasonable for the start bitrate, and scaling is enabled,// VideoStreamEncoder asks the source to scale down and drops a few initial// frames.// TODO(nisse): This is a poor interface, and mixes bandwidth estimation and// codec configuration in an undesired way. For the actual send bandwidth, we// should always be somewhat conservative, but we may nevertheless want to let// the application configure a more optimistic quality for the initial// resolution. Should be replaced by a construction time setting./*设置初始比特率，后续会被 OnBitrateUpdated 方法覆盖。主要影响初始关键帧的分辨率：如果传入的帧比初始比特率合理的大小要大，并且缩放被启用，VideoStreamEncoder 会请求源缩小尺寸并丢弃一些初始帧。TODO(nisse): 这是一个设计不佳的接口，它以不期望的方式混合了带宽估计和编解码器配置。对于实际的发送带宽，我们应该始终保持一定的保守性，但我们可能仍然希望允许应用程序为初始分辨率配置一个更乐观的质量。应该被构造时的设置所取代。*/virtual void SetStartBitrate(int start_bitrate_bps) = 0;// Request a key frame. Used for signalling from the remote receiver with// no arguments and for RTCRtpSender.generateKeyFrame with a list of// rids/layers./*请求一个关键帧。用于从远程接收方发出信号，无需参数，以及用于 RTCRtpSender.generateKeyFrame 方法，该方法带有一系列 rids/layers。*/virtual void SendKeyFrame(const std::vector<VideoFrameType>& layers = {}) = 0;// Inform the encoder that a loss has occurred.// 通知编码器发生了丢包virtual void OnLossNotification(const VideoEncoder::LossNotification& loss_notification) = 0;// Set the currently estimated network properties. A `target_bitrate`// of zero pauses the encoder.// `stable_target_bitrate` is a filtered version of `target_bitrate`. It is// always less or equal to it. It can be used to avoid rapid changes of// expensive encoding settings, such as resolution.// `link_allocation` is the bandwidth available for this video stream on the// network link. It is always at least `target_bitrate` but may be higher// if we are not network constrained./*设置当前估计的网络属性。`target_bitrate` 为零时会暂停编码器。 `stable_target_bitrate` 是 `target_bitrate` 的一个过滤版本。它总是小于或等于 `target_bitrate`。它可以用来避免昂贵编码设置（如分辨率）的快速变化。`link_allocation` 是网络链路上可用于此视频流的带宽。它总是至少为 `target_bitrate`，但如果我们不受网络限制，可能会更高。*/virtual void OnBitrateUpdated(DataRate target_bitrate,DataRate stable_target_bitrate,DataRate link_allocation,uint8_t fraction_lost,int64_t round_trip_time_ms,double cwnd_reduce_ratio) = 0;// Set a FecControllerOverride, through which the encoder may override// decisions made by FecController./*设置一个 FecControllerOverride，通过它编码器可以覆盖 FecController 所做的决策*/virtual void SetFecControllerOverride(FecControllerOverride* fec_controller_override) = 0;// Creates and configures an encoder with the given `config`. The// `max_data_payload_length` is used to support single NAL unit// packetization for H.264./*使用给定的 `config` 创建并配置一个编码器。`max_data_payload_length` 用于支持 H.264 的单个 NAL 单元打包。*/ virtual void ConfigureEncoder(VideoEncoderConfig config,size_t max_data_payload_length) = 0;virtual void ConfigureEncoder(VideoEncoderConfig config,size_t max_data_payload_length,SetParametersCallback callback) = 0;// Permanently stop encoding. After this method has returned, it is// guaranteed that no encoded frames will be delivered to the sink.virtual void Stop() = 0;
};
1.2 资源类（Resource）
上面关于Resource类的声明位于api/adaptation/resource.h中，这里的可以表示帧率，码率等编码信息，这个类存在的意义是对这些信息进行监控，如果某个资源超出了预期范围，需要将这个资源进行调控。
// A Resource monitors an implementation-specific resource. It may report
// kOveruse or kUnderuse when resource usage is high or low enough that we
// should perform some sort of mitigation to fulfil the resource's constraints.
//
// The methods on this interface are invoked on the adaptation task queue.
// Resource usage measurements may be performed on an any task queue.
//
// The Resource is reference counted to prevent use-after-free when posting
// between task queues. As such, the implementation MUST NOT make any
// assumptions about which task queue Resource is destructed on.
/*1.一个资源监控特定实现的资源。当资源使用量足够高或足够低，以至于我们应该执行某种缓解措施以满足资源的约束时，它可能会报告kOveruse（过载）或kUnderuse（欠载）。2.这个接口上的方法在适应性任务队列上被调用。资源使用量的测量可以在任何任务队列上执行。3.资源被引用计数，以防止在任务队列之间发布时出现使用后释放的问题。因此，实现必须不对资源被销毁的任务队列做任何假设。
*/
class RTC_EXPORT Resource : public RefCountInterface {public:Resource();// Destruction may happen on any task queue.~Resource() override;virtual std::string Name() const = 0;// The `listener` may be informed of resource usage measurements on any task// queue, but not after this method is invoked with the null argument.virtual void SetResourceListener(ResourceListener* listener) = 0;
};// ResourceListenter和ResourceUsageState的声明如下
enum class ResourceUsageState {// Action is needed to minimze the load on this resource.// 过载，需要降低这个资源kOveruse,// Increasing the load on this resource is desired, if possible.// 欠载，需要增加这个资源kUnderuse,
};RTC_EXPORT const char* ResourceUsageStateToString(ResourceUsageState usage_state);class RTC_EXPORT ResourceListener {public:virtual ~ResourceListener();// 评估当前资源的状态virtual void OnResourceUsageStateMeasured(rtc::scoped_refptr<Resource> resource,ResourceUsageState usage_state) = 0;
};
1.3 已编码图像的回调（EncodedImageCallback）
EncodedImageCallback用于图像已经编码之后的回调，声明位于api/video_codecs/video_encoder.h中
class RTC_EXPORT EncodedImageCallback {public:virtual ~EncodedImageCallback() {}struct Result {enum Error {OK,// Failed to send the packet.ERROR_SEND_FAILED,};explicit Result(Error error) : error(error) {}Result(Error error, uint32_t frame_id) : error(error), frame_id(frame_id) {}Error error;// Frame ID assigned to the frame. The frame ID should be the same as the ID// seen by the receiver for this frame. RTP timestamp of the frame is used// as frame ID when RTP is used to send video. Must be used only when// error=OK./*分配给帧的帧ID。该帧ID应与接收方看到的该帧的ID相同。当使用RTP发送视频时，帧的RTP时间戳被用作帧ID。仅当错误为OK时必须使用。*/uint32_t frame_id = 0;// Tells the encoder that the next frame is should be dropped.// 告诉编码器下一个帧应该被丢弃。bool drop_next_frame = false;};// Used to signal the encoder about reason a frame is dropped.// kDroppedByMediaOptimizations - dropped by MediaOptimizations (for rate// limiting purposes).// kDroppedByEncoder - dropped by encoder's internal rate limiter.// TODO(bugs.webrtc.org/10164): Delete this enum? It duplicates the more// general VideoStreamEncoderObserver::DropReason. Also,// kDroppedByMediaOptimizations is not produced by any encoder, but by// VideoStreamEncoder./*用于向编码器发出信号，说明帧被丢弃的原因。kDroppedByMediaOptimizations: 由 MediaOptimizations 丢弃（出于速率限制的目的）。kDroppedByEncoder: 由编码器的内部速率限制器丢弃。TODO(bugs.webrtc.org/10164): 删除这个枚举？它与更通用的 VideoStreamEncoderObserver::DropReason 重复。此外，kDroppedByMediaOptimizations 不是由任何编码器产生的，而是由 VideoStreamEncoder 产生的。*/enum class DropReason : uint8_t {kDroppedByMediaOptimizations,kDroppedByEncoder};// Callback function which is called when an image has been encoded.// 在一帧被编码之后会被调用virtual Result OnEncodedImage(const EncodedImage& encoded_image,const CodecSpecificInfo* codec_specific_info) = 0;// 丢弃一帧virtual void OnDroppedFrame(DropReason /* reason */) {}
};
1.4 视频资源监听器（VideoSourceRestrictionsListener）
// The listener is responsible for carrying out the reconfiguration of the video
// source such that the VideoSourceRestrictions are fulfilled.
// 监听器负责执行视频源的重新配置，以满足 VideoSourceRestrictions 的要求。
class VideoSourceRestrictionsListener {public:virtual ~VideoSourceRestrictionsListener();// The `restrictions` are filtered by degradation preference but not the// `adaptation_counters`, which are currently only reported for legacy stats// calculation purposes./*`restrictions` 会根据降级偏好进行过滤，但 `adaptation_counters` 不会，它们目前仅用于传统统计计算目的的报告。*/virtual void OnVideoSourceRestrictionsUpdated(VideoSourceRestrictions restrictions,const VideoAdaptationCounters& adaptation_counters,rtc::scoped_refptr<Resource> reason,const VideoSourceRestrictions& unfiltered_restrictions) = 0;
};
总体来讲，VideoStreamEncoder完成了进行视频编码的第一道工序，包括获取视频帧，根据编码情况进行码率调整，根据网络情况丢帧等等，由于涵盖内容太广泛，一些其他很重要的类没有分析，例如DegradationPreferenceManager、QpParser等，后续单独记录