写点什么

OpenHarmony 3.2 Beta Audio——音频渲染

  • 2023-03-02
    上海
  • 本文字数:7961 字

    阅读完需:约 26 分钟

一、简介

Audio 是多媒体子系统中的一个重要模块,其涉及的内容比较多,有音频的渲染、音频的采集、音频的策略管理等。本文主要针对音频渲染功能进行详细地分析,并通过源码中提供的例子,对音频渲染进行流程的梳理。


二、目录

foundation/multimedia/audio_framework

audio_framework├── frameworks│   ├── js                          #js 接口│   │   └── napi│   │       └── audio_renderer      #audio_renderer NAPI接口│   │           ├── include│   │           │   ├── audio_renderer_callback_napi.h│   │           │   ├── renderer_data_request_callback_napi.h│   │           │   ├── renderer_period_position_callback_napi.h│   │           │   └── renderer_position_callback_napi.h│   │           └── src│   │               ├── audio_renderer_callback_napi.cpp│   │               ├── audio_renderer_napi.cpp│   │               ├── renderer_data_request_callback_napi.cpp│   │               ├── renderer_period_position_callback_napi.cpp│   │               └── renderer_position_callback_napi.cpp│   └── native                      #native 接口│       └── audiorenderer│           ├── BUILD.gn│           ├── include│           │   ├── audio_renderer_private.h│           │   └── audio_renderer_proxy_obj.h│           ├── src│           │   ├── audio_renderer.cpp│           │   └── audio_renderer_proxy_obj.cpp│           └── test│               └── example│                   └── audio_renderer_test.cpp├── interfaces│   ├── inner_api                   #native实现的接口│   │   └── native│   │       └── audiorenderer       #audio渲染本地实现的接口定义│   │           └── include│   │               └── audio_renderer.h│   └── kits                        #js调用的接口│       └── js│           └── audio_renderer      #audio渲染NAPI接口的定义│               └── include│                   └── audio_renderer_napi.h└── services                        #服务端    └── audio_service        ├── BUILD.gn        ├── client                  #IPC调用中的proxy端        │   ├── include        │   │   ├── audio_manager_proxy.h        │   │   ├── audio_service_client.h        │   └── src        │       ├── audio_manager_proxy.cpp        │       ├── audio_service_client.cpp        └── server                  #IPC调用中的server端            ├── include            │   └── audio_server.h            └── src                ├── audio_manager_stub.cpp                └── audio_server.cpp
复制代码


三、音频渲染总体流程


四、Native 接口使用

在 OpenAtom OpenHarmony(以下简称“OpenHarmony”)系统中,音频模块提供了功能测试代码,本文选取了其中的音频渲染例子作为切入点来进行介绍,例子采用的是对 wav 格式的音频文件进行渲染。wav 格式的音频文件是 wav 头文件和音频的原始数据,不需要进行数据解码,所以音频渲染直接对原始数据进行操作,文件路径为:foundation/multimedia/audio_framework/frameworks/native/audiorenderer/test/example/audio_renderer_test.cpp



bool TestPlayback(int argc, char *argv[]) const{ FILE* wavFile = fopen(path, "rb"); //读取wav文件头信息 size_t bytesRead = fread(&wavHeader, 1, headerSize, wavFile);
//设置AudioRenderer参数 AudioRendererOptions rendererOptions = {}; rendererOptions.streamInfo.encoding = AudioEncodingType::ENCODING_PCM; rendererOptions.streamInfo.samplingRate = static_cast<AudioSamplingRate>(wavHeader.SamplesPerSec); rendererOptions.streamInfo.format = GetSampleFormat(wavHeader.bitsPerSample); rendererOptions.streamInfo.channels = static_cast<AudioChannel>(wavHeader.NumOfChan); rendererOptions.rendererInfo.contentType = contentType; rendererOptions.rendererInfo.streamUsage = streamUsage; rendererOptions.rendererInfo.rendererFlags = 0;
//创建AudioRender实例 unique_ptr<AudioRenderer> audioRenderer = AudioRenderer::Create(rendererOptions);
shared_ptr<AudioRendererCallback> cb1 = make_shared<AudioRendererCallbackTestImpl>(); //设置音频渲染回调 ret = audioRenderer->SetRendererCallback(cb1);
//InitRender方法主要调用了audioRenderer实例的Start方法,启动音频渲染 if (!InitRender(audioRenderer)) { AUDIO_ERR_LOG("AudioRendererTest: Init render failed"); fclose(wavFile); return false; }
//StartRender方法主要是读取wavFile文件的数据,然后通过调用audioRenderer实例的Write方法进行播放 if (!StartRender(audioRenderer, wavFile)) { AUDIO_ERR_LOG("AudioRendererTest: Start render failed"); fclose(wavFile); return false; }
//停止渲染 if (!audioRenderer->Stop()) { AUDIO_ERR_LOG("AudioRendererTest: Stop failed"); }
//释放渲染 if (!audioRenderer->Release()) { AUDIO_ERR_LOG("AudioRendererTest: Release failed"); }
//关闭wavFile fclose(wavFile); return true; }
复制代码


首先读取 wav 文件,通过读取到 wav 文件的头信息对 AudioRendererOptions 相关的参数进行设置,包括编码格式、采样率、采样格式、通道数等。根据 AudioRendererOptions 设置的参数来创建 AudioRenderer 实例(实际上是 AudioRendererPrivate),后续的音频渲染主要是通过 AudioRenderer 实例进行。创建完成后,调用 AudioRenderer 的 Start 方法,启动音频渲染。启动后,通过 AudioRenderer 实例的 Write 方法,将数据写入,音频数据会被播放。


五、调用流程


  1. 创建 AudioRenderer

std::unique_ptr<AudioRenderer> AudioRenderer::Create(const std::string cachePath,    const AudioRendererOptions &rendererOptions, const AppInfo &appInfo){    ContentType contentType = rendererOptions.rendererInfo.contentType;        StreamUsage streamUsage = rendererOptions.rendererInfo.streamUsage;       AudioStreamType audioStreamType = AudioStream::GetStreamType(contentType, streamUsage);    auto audioRenderer = std::make_unique<AudioRendererPrivate>(audioStreamType, appInfo);    if (!cachePath.empty()) {        AUDIO_DEBUG_LOG("Set application cache path");        audioRenderer->SetApplicationCachePath(cachePath);    }
audioRenderer->rendererInfo_.contentType = contentType; audioRenderer->rendererInfo_.streamUsage = streamUsage; audioRenderer->rendererInfo_.rendererFlags = rendererOptions.rendererInfo.rendererFlags;
AudioRendererParams params; params.sampleFormat = rendererOptions.streamInfo.format; params.sampleRate = rendererOptions.streamInfo.samplingRate; params.channelCount = rendererOptions.streamInfo.channels; params.encodingType = rendererOptions.streamInfo.encoding;
if (audioRenderer->SetParams(params) != SUCCESS) { AUDIO_ERR_LOG("SetParams failed in renderer"); audioRenderer = nullptr; return nullptr; }
return audioRenderer;}
复制代码


首先通过 AudioStream 的 GetStreamType 方法获取音频流的类型,根据音频流类型创建 AudioRendererPrivate 对象,AudioRendererPrivate 是 AudioRenderer 的子类。紧接着对 audioRenderer 进行参数设置,其中包括采样格式、采样率、通道数、编码格式。设置完成后返回创建的 AudioRendererPrivate 实例。


2. 设置回调


int32_t AudioRendererPrivate::SetRendererCallback(const std::shared_ptr<AudioRendererCallback> &callback){    RendererState state = GetStatus();    if (state == RENDERER_NEW || state == RENDERER_RELEASED) {        return ERR_ILLEGAL_STATE;    }    if (callback == nullptr) {        return ERR_INVALID_PARAM;    }
// Save reference for interrupt callback if (audioInterruptCallback_ == nullptr) { return ERROR; } std::shared_ptr<AudioInterruptCallbackImpl> cbInterrupt = std::static_pointer_cast<AudioInterruptCallbackImpl>(audioInterruptCallback_); cbInterrupt->SaveCallback(callback);
// Save and Set reference for stream callback. Order is important here. if (audioStreamCallback_ == nullptr) { audioStreamCallback_ = std::make_shared<AudioStreamCallbackRenderer>(); if (audioStreamCallback_ == nullptr) { return ERROR; } } std::shared_ptr<AudioStreamCallbackRenderer> cbStream =std::static_pointer_cast<AudioStreamCallbackRenderer>(audioStreamCallback_); cbStream->SaveCallback(callback); (void)audioStream_->SetStreamCallback(audioStreamCallback_);
return SUCCESS;}
复制代码


参数传入的回调主要涉及到两个方面:一方面是 AudioInterruptCallbackImpl 中设置了我们传入的渲染回调,另一方面是 AudioStreamCallbackRenderer 中也设置了渲染回调。


3. 启动渲染

bool AudioRendererPrivate::Start(StateChangeCmdType cmdType) const{    AUDIO_INFO_LOG("AudioRenderer::Start");    RendererState state = GetStatus();
AudioInterrupt audioInterrupt; switch (mode_) { case InterruptMode::SHARE_MODE: audioInterrupt = sharedInterrupt_; break; case InterruptMode::INDEPENDENT_MODE: audioInterrupt = audioInterrupt_; break; default: break; } AUDIO_INFO_LOG("AudioRenderer::Start::interruptMode: %{public}d, streamType: %{public}d, sessionID: %{public}d", mode_, audioInterrupt.streamType, audioInterrupt.sessionID);
if (audioInterrupt.streamType == STREAM_DEFAULT || audioInterrupt.sessionID == INVALID_SESSION_ID) { return false; }
int32_t ret = AudioPolicyManager::GetInstance().ActivateAudioInterrupt(audioInterrupt); if (ret != 0) { AUDIO_ERR_LOG("AudioRendererPrivate::ActivateAudioInterrupt Failed"); return false; }
return audioStream_->StartAudioStream(cmdType);}

复制代码


AudioPolicyManager::GetInstance().ActivateAudioInterrupt 这个操作主要是根据 AudioInterrupt 来进行音频中断的激活,这里涉及了音频策略相关的内容,后续会专门出关于音频策略的文章进行分析。这个方法的核心是通过调用 AudioStream 的 StartAudioStream 方法来启动音频流。


bool AudioStream::StartAudioStream(StateChangeCmdType cmdType){    int32_t ret = StartStream(cmdType);
resetTime_ = true; int32_t retCode = clock_gettime(CLOCK_MONOTONIC, &baseTimestamp_);
if (renderMode_ == RENDER_MODE_CALLBACK) { isReadyToWrite_ = true; writeThread_ = std::make_unique<std::thread>(&AudioStream::WriteCbTheadLoop, this); } else if (captureMode_ == CAPTURE_MODE_CALLBACK) { isReadyToRead_ = true; readThread_ = std::make_unique<std::thread>(&AudioStream::ReadCbThreadLoop, this); }
isFirstRead_ = true; isFirstWrite_ = true; state_ = RUNNING; AUDIO_INFO_LOG("StartAudioStream SUCCESS");
if (audioStreamTracker_) { AUDIO_DEBUG_LOG("AudioStream:Calling Update tracker for Running"); audioStreamTracker_->UpdateTracker(sessionId_, state_, rendererInfo_, capturerInfo_); } return true;}
复制代码


AudioStream 的 StartAudioStream 主要的工作是调用 StartStream 方法,StartStream 方法是 AudioServiceClient 类中的方法。AudioServiceClient 类是 AudioStream 的父类。接下来看一下 AudioServiceClient 的 StartStream 方法。

int32_t AudioServiceClient::StartStream(StateChangeCmdType cmdType){    int error;    lock_guard<mutex> lockdata(dataMutex);    pa_operation *operation = nullptr;
pa_threaded_mainloop_lock(mainLoop);
pa_stream_state_t state = pa_stream_get_state(paStream);
streamCmdStatus = 0; stateChangeCmdType_ = cmdType; operation = pa_stream_cork(paStream, 0, PAStreamStartSuccessCb, (void *)this);
while (pa_operation_get_state(operation) == PA_OPERATION_RUNNING) { pa_threaded_mainloop_wait(mainLoop); } pa_operation_unref(operation); pa_threaded_mainloop_unlock(mainLoop);
if (!streamCmdStatus) { AUDIO_ERR_LOG("Stream Start Failed"); ResetPAAudioClient(); return AUDIO_CLIENT_START_STREAM_ERR; } else { AUDIO_INFO_LOG("Stream Started Successfully"); return AUDIO_CLIENT_SUCCESS; }}
复制代码


StartStream 方法中主要是调用了 pulseaudio 库的 pa_stream_cork 方法进行流启动,后续就调用到了 pulseaudio 库中了。pulseaudio 库我们暂且不分析。


4. 写入数据

int32_t AudioRendererPrivate::Write(uint8_t *buffer, size_t bufferSize){    return audioStream_->Write(buffer, bufferSize);}
复制代码


通过调用 AudioStream 的 Write 方式实现功能,接下来看一下 AudioStream 的 Write 方法。


size_t AudioStream::Write(uint8_t *buffer, size_t buffer_size){    int32_t writeError;    StreamBuffer stream;    stream.buffer = buffer;    stream.bufferLen = buffer_size;    isWriteInProgress_ = true;
if (isFirstWrite_) { if (RenderPrebuf(stream.bufferLen)) { return ERR_WRITE_FAILED; } isFirstWrite_ = false; }
size_t bytesWritten = WriteStream(stream, writeError); isWriteInProgress_ = false; if (writeError != 0) { AUDIO_ERR_LOG("WriteStream fail,writeError:%{public}d", writeError); return ERR_WRITE_FAILED; } return bytesWritten;}
复制代码


Write 方法中分成两个阶段,首次写数据,先调用 RenderPrebuf 方法,将 preBuf_的数据写入后再调用 WriteStream 进行音频数据的写入。


size_t AudioServiceClient::WriteStream(const StreamBuffer &stream, int32_t &pError){       size_t cachedLen = WriteToAudioCache(stream);    if (!acache.isFull) {        pError = error;        return cachedLen;    }
pa_threaded_mainloop_lock(mainLoop);

const uint8_t *buffer = acache.buffer.get(); size_t length = acache.totalCacheSize;
error = PaWriteStream(buffer, length); acache.readIndex += acache.totalCacheSize; acache.isFull = false;
if (!error && (length >= 0) && !acache.isFull) { uint8_t *cacheBuffer = acache.buffer.get(); uint32_t offset = acache.readIndex; uint32_t size = (acache.writeIndex - acache.readIndex); if (size > 0) { if (memcpy_s(cacheBuffer, acache.totalCacheSize, cacheBuffer + offset, size)) { AUDIO_ERR_LOG("Update cache failed"); pa_threaded_mainloop_unlock(mainLoop); pError = AUDIO_CLIENT_WRITE_STREAM_ERR; return cachedLen; } AUDIO_INFO_LOG("rearranging the audio cache"); } acache.readIndex = 0; acache.writeIndex = 0;
if (cachedLen < stream.bufferLen) { StreamBuffer str; str.buffer = stream.buffer + cachedLen; str.bufferLen = stream.bufferLen - cachedLen; AUDIO_DEBUG_LOG("writing pending data to audio cache: %{public}d", str.bufferLen); cachedLen += WriteToAudioCache(str); } }
pa_threaded_mainloop_unlock(mainLoop); pError = error; return cachedLen;}
复制代码


WriteStream 方法不是直接调用 pulseaudio 库的写入方法,而是通过 WriteToAudioCache 方法将数据写入缓存中,如果缓存没有写满则直接返回,不会进入下面的流程,只有当缓存写满后,才会调用下面的 PaWriteStream 方法。该方法涉及对 pulseaudio 库写入操作的调用,所以缓存的目的是避免对 pulseaudio 库频繁地做 IO 操作,提高了效率。


六、总结

本文主要对 OpenHarmony 3.2 Beta 多媒体子系统的音频渲染模块进行介绍,首先梳理了 Audio Render 的整体流程,然后对几个核心的方法进行代码的分析。整体的流程主要通过 pulseaudio 库启动流,然后通过 pulseaudio 库的 pa_stream_write 方法进行数据的写入,最后播放出音频数据。


音频渲染主要分为以下几个层次:

(1)AudioRenderer 的创建,实际创建的是它的子类 AudioRendererPrivate 实例。

(2)通过 AudioRendererPrivate 设置渲染的回调。

(3)启动渲染,这一部分代码最终会调用到 pulseaudio 库中,相当于启动了 pulseaudio 的流。

(4)通过 pulseaudio 库的 pa_stream_write 方法将数据写入设备,进行播放。


对 OpenHarmony 3.2 Beta 多媒体系列开发感兴趣的读者,也可以阅读我之前写过几篇文章:

https://mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247501603&idx=1&sn=3a53ff64d9d1ad3c62365b72f6c4b95a&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247499613&idx=1&sn=73af29d87717531243f070e7e69dfbd3&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247500816&idx=1&sn=953310bfa91f64f3b922bde5ef99c374&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247500837&idx=1&sn=c0098dd2475aee364314428b7a1ed7e6&scene=21#wechat_redirect》。

用户头像

OpenHarmony开发者官方账号 2021-12-15 加入

OpenHarmony是由开放原子开源基金会(OpenAtom Foundation)孵化及运营的开源项目,目标是面向全场景、全连接、全智能时代,基于开源的方式,搭建一个智能终端设备操作系统的框架和平台,促进万物互联产业的繁荣发展

评论

发布
暂无评论
OpenHarmony 3.2 Beta Audio——音频渲染_OpenHarmony_OpenHarmony开发者_InfoQ写作社区