How do I call CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer?

I’m trying to figure out how to call this AVFoundation function in Swift. I’ve spent a ton of time fiddling with declarations and syntax, and got this far. The compiler is mostly happy, but I’m left with one last quandary.

public func captureOutput(
    captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
    fromConnection connection: AVCaptureConnection!
) {
    let samplesInBuffer = CMSampleBufferGetNumSamples(sampleBuffer)
    var audioBufferList: AudioBufferList

    var buffer: Unmanaged<CMBlockBuffer>? = nil

    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        nil,
        &audioBufferList,
        UInt(sizeof(audioBufferList.dynamicType)),
        nil,
        nil,
        UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
        &buffer
    )

    // do stuff
}

The compiler complains for the 3rd and 4th arguments:

  • AVCaptureSession cancels background audio
  • iOS AVAudioSession interruption notification not working as expected
  • Crash when toggling access to Camera and Photos
  • Compositing 2 videos on top of each other with alpha
  • YouTube live on iOS?
  • Rotating Video w/ AVMutableVideoCompositionLayerInstruction
  • Address of variable ‘audioBufferList’ taken before it is initialized

    and

    Variable ‘audioBufferList’ used before being initialized

    So what am I supposed to do here?

    I’m working off of this StackOverflow answer but it’s Objective-C. I’m trying to translate it into Swift, but run into this problem.

    Or is there possibly a better approach? I need to read the data from the buffer, one sample at a time, so I’m basically trying to get an array of the samples that I can iterate over.

    4 Solutions Collect From Internet About “How do I call CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer?”

    Disclaimer: I have just tried to translate the code from Reading audio samples via AVAssetReader to Swift, and verified that it compiles. I have not
    tested if it really works.

    // Needs to be initialized somehow, even if we take only the address
    var audioBufferList = AudioBufferList(mNumberBuffers: 1,
          mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
    
    var buffer: Unmanaged<CMBlockBuffer>? = nil
    
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        nil,
        &audioBufferList,
        UInt(sizeof(audioBufferList.dynamicType)),
        nil,
        nil,
        UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
        &buffer
    )
    
    // Ensure that the buffer is released automatically.
    let buf = buffer!.takeRetainedValue() 
    
    // Create UnsafeBufferPointer from the variable length array starting at audioBufferList.mBuffers
    let audioBuffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers,
        count: Int(audioBufferList.mNumberBuffers))
    
    for audioBuffer in audioBuffers {
        // Create UnsafeBufferPointer<Int16> from the buffer data pointer
        var samples = UnsafeMutableBufferPointer<Int16>(start: UnsafeMutablePointer(audioBuffer.mData),
            count: Int(audioBuffer.mDataByteSize)/sizeof(Int16))
    
        for sample in samples {
            // ....
        }
    }
    

    Swift3 solution:

    func loopAmplitudes(audioFileUrl: URL) {
    
        let asset = AVAsset(url: audioFileUrl)
    
        let reader = try! AVAssetReader(asset: asset)
    
        let track = asset.tracks(withMediaType: AVMediaTypeAudio)[0]
    
        let settings = [
            AVFormatIDKey : kAudioFormatLinearPCM
        ]
    
        let readerOutput = AVAssetReaderTrackOutput(track: track, outputSettings: settings)
        reader.add(readerOutput)
        reader.startReading()
    
        while let buffer = readerOutput.copyNextSampleBuffer() {
    
            var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
            var blockBuffer: CMBlockBuffer?
    
            CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
                buffer,
                nil,
                &audioBufferList,
                MemoryLayout<AudioBufferList>.size,
                nil,
                nil,
                kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                &blockBuffer
            );
    
            let buffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))
    
            for buffer in buffers {
    
                let samplesCount = Int(buffer.mDataByteSize) / MemoryLayout<Int16>.size
                let samplesPointer = audioBufferList.mBuffers.mData!.bindMemory(to: Int16.self, capacity: samplesCount)
                let samples = UnsafeMutableBufferPointer<Int16>(start: samplesPointer, count: samplesCount)
    
                for sample in samples {
    
                    //do something with you sample (which is Int16 amplitude value)
    
                }
            }
        }
    }
    

    Martin’s answer works and does exactly what I asked in the question, however, after posting the question and spending more time with the problem (and before seeing Martin’s answer), I came up with this:

    public func captureOutput(
        captureOutput: AVCaptureOutput!,
        didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
        fromConnection connection: AVCaptureConnection!
    ) {
        let samplesInBuffer = CMSampleBufferGetNumSamples(sampleBuffer)
        self.currentZ = Double(samplesInBuffer)
    
        let buffer: CMBlockBufferRef = CMSampleBufferGetDataBuffer(sampleBuffer)
    
        var lengthAtOffset: size_t = 0
        var totalLength: size_t = 0
        var data: UnsafeMutablePointer<Int8> = nil
    
        if( CMBlockBufferGetDataPointer( buffer, 0, &lengthAtOffset, &totalLength, &data ) != noErr ) {
            println("some sort of error happened")
        } else {
            for i in stride(from: 0, to: totalLength, by: 2) {
                // do stuff
            }
        }
    }
    

    This is a slightly different approach, and probably still has room for improvement, but the main point here is that at least on an iPad Mini (and probably other devices), each time this method is called, we get 1,024 samples. But those samples come in an array of 2,048 Int8 values. Every other one is the left/right byte that needs to be combined into to make an Int16 to turn the 2,048 half-samples into 1,024 whole samples.

    it works for me. try it:

    let musicUrl: NSURL = mediaItemCollection.items[0].valueForProperty(MPMediaItemPropertyAssetURL) as! NSURL
    let asset: AVURLAsset = AVURLAsset(URL: musicUrl, options: nil)
    let assetOutput = AVAssetReaderTrackOutput(track: asset.tracks[0] as! AVAssetTrack, outputSettings: nil)
    
    var error : NSError?
    
    let assetReader: AVAssetReader = AVAssetReader(asset: asset, error: &error)
    
    if error != nil {
        print("Error asset Reader: \(error?.localizedDescription)")
    }
    
    assetReader.addOutput(assetOutput)
    assetReader.startReading()
    
    let sampleBuffer: CMSampleBufferRef = assetOutput.copyNextSampleBuffer()
    
    var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
    var blockBuffer: Unmanaged<CMBlockBuffer>? = nil
    
    
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        nil,
        &audioBufferList,
        sizeof(audioBufferList.dynamicType), // instead of UInt(sizeof(audioBufferList.dynamicType))
        nil,
        nil,
        UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
        &blockBuffer
    )