Finn Voorhees' Blog

Platform-Specific Release Notes with Xcode Cloud

Tue, 09 Jan 2024 00:00:00 GMT

I've been using Xcode Cloud as a CI/CD for all my apps since it came out. It has streamlined the release cycle, and since the addition of support for "What to Test" notes in June 2023, you can build and distribute an app to external testers with the click of a button, a git commit, or a terminal command.

When adding Xcode Cloud release workflows to a cross-platform iOS and macOS app, however, I ran into an issue. While you can specify release notes for the build by editing TestFlight/WhatToTest.<LOCALE>.txt, these release notes will be used for all workflows in your project. Because the iOS app was built using SwiftUI and the macOS app was built using AppKit, each had different features to test and bugs that I was fixing, so it didn't make sense to write the same release notes for each platform. I didn't really want to have to make separate commits to make builds with unique release notes, so I just submitted an enhancement request (FB13501281), prefixed each line of my release notes with (iOS) or (macOS), and moved on.

To my surprise, however, a few days later I received a response in the feedback app from someone letting me know my suggestion had been passed along to the responsible team, offering a workaround in the meantime:

#  ci_pre_xcodebuild.sh
if [[ "$CI_XCODEBUILD_ACTION" == "archive" ]]; then
    if [[ "$CI_PRODUCT_PLATFORM" == "iOS" ]]; then
        echo "Set WhatToTest notes for iOS"
        mv $CI_PROJECT_FILE_PATH/../TestFlight/iOS-WhatToTest.en-US.txt $CI_PROJECT_FILE_PATH/../TestFlight/WhatToTest.en-US.txt
    elif [[ "$CI_PRODUCT_PLATFORM" == "macOS" ]]; then
        echo "Set WhatToTest notes for macOS"
        mv $CI_PROJECT_FILE_PATH/../TestFlight/macOS-WhatToTest.en-US.txt $CI_PROJECT_FILE_PATH/../TestFlight/WhatToTest.en-US.txt
    fi
fi

The solution is a custom build script that automatically replaces the contents of WhatToTest.<LOCALE>.txt with the contents of a platform-specific release notes file depending on which platform the running workflow is targeting.

While it would be great to specify a custom path to release notes in each workflow, this script has been working nicely for me and isn't too much additional setup.

Subscribing to SwiftData changes outside SwiftUI

Sun, 28 Apr 2024 00:00:00 GMT

While there are plenty of articles about using SwiftData outside of SwiftUI, many fail to mention how you might observe insertions and deletions similarly to how SwiftUI’s Query macro works.

Because SwiftData is backed by CoreData, changes can be detected using the .NSPersistentStoreRemoteChange notification. A simple Database wrapper class can be created that vends AsyncStream's of SwiftData models given a FetchDescriptor.

This AsyncStream can then be used to, for example, update a diffable data source to display your data in a UICollectionView.

Database.swift

import Foundation
import SwiftData

@MainActor final class Database {
    // MARK: Lifecycle

    init(isStoredInMemoryOnly: Bool = false) {
        do {
            let configuration = ModelConfiguration(isStoredInMemoryOnly: isStoredInMemoryOnly)
            container = try ModelContainer(for: Link.self, configurations: configuration)
        } catch {
            fatalError("\(error)")
        }
    }

    // MARK: Public

    public var context: ModelContext { container.mainContext }

    // MARK: Internal

    func models<T: PersistentModel>(
        filter: Predicate<T>? = nil,
        sort keyPath: KeyPath<T, some Comparable>,
        order: SortOrder = .forward
    ) -> AsyncStream<[T]> {
        let fetchDescriptor = FetchDescriptor(
            predicate: filter,
            sortBy: [SortDescriptor(keyPath, order: order)]
        )
        return models(matching: fetchDescriptor)
    }

    func models<T: PersistentModel>(matching fetchDescriptor: FetchDescriptor<T>) -> AsyncStream<[T]> {
        AsyncStream { continuation in
            let task = Task {
                for await _ in NotificationCenter.default.notifications(
                    named: .NSPersistentStoreRemoteChange
                ).map({ _ in () }) {
                    do {
                        let models = try container.mainContext.fetch(fetchDescriptor)
                        continuation.yield(models)
                    } catch {
                        // log/ignore the error, or return an AsyncThrowingStream
                    }
                }
            }
            continuation.onTermination = { _ in
                task.cancel()
            }
            do {
                let models = try container.mainContext.fetch(fetchDescriptor)
                continuation.yield(models)
            } catch {
                // log/ignore the error, or return an AsyncThrowingStream
            }
        }
    }

    // MARK: Private

    private let container: ModelContainer
}

Usage

for await items in database.models(sort: \TodoItem.creationTime, order: .reverse) {
    var snapshot = NSDiffableDataSourceSnapshot<Int, TodoItem>()
    snapshot.appendSections([0])
    snapshot.appendItems(items, toSection: 0)
    dataSource.apply(snapshot)
}

Note: You might want to throw a .removeDuplicates() from swift-async-algorithms on the stream to avoid unnecessary updates.

Reading and Writing Spatial Photos with Image I/O

Sun, 21 Jan 2024 00:00:00 GMT

This is a follow up to my previous post on Reading and Writing Spatial Video with AVFoundation.

Writing Spatial Photos
Reading Spatial Photos

Spatial photos are even less documented than spatial video, however it's not too difficult to create your own spatial photos once you know the basics. While spatial videos are contained in MV-HEVC video files, spatial photos are stored in a HEIC file. HEIC files can contain multiple images and metadata, and in the case of spatial photo, each spatial photo file contains two images: one for the left eye and one for the right. Once you know what metadata you need to add, spatial photos become quite easy to create using the Image I/O framework.

Writing Spatial Photos

To start out, you will need two CGImages, one representing each eye. In this case we are using a solid red image for the left eye and a solid blue image for the left eye, but in practice you will most likely render a scene with a camera offset or capture two images using a stereo camera system:

let imageSize = CGRect(x: 0, y: 0, width: 3072, height: 3072)
let leftImage = CIContext().createCGImage(.red, from: imageSize)!
let rightImage = CIContext().createCGImage(.blue, from: imageSize)!

Create a new image destination pointing towards an output URL, specifying "public.heic" for the type identifier and "2" for the image count:

let destination = CGImageDestinationCreateWithURL(url as CFURL, UTType.heic.identifier as CFString, 2, nil)!

Create a properties dictionary. This will contain information on the left and right image index as well as a matrix that relates a camera’s internal properties to an ideal pinhole-camera model. The camera intrinsics may be specific to the technique you are using to generate stereo images:

let properties = [
    kCGImagePropertyGroups: [
        kCGImagePropertyGroupIndex: 0,
        kCGImagePropertyGroupType: kCGImagePropertyGroupTypeStereoPair,
        kCGImagePropertyGroupImageIndexLeft: 0,
        kCGImagePropertyGroupImageIndexRight: 1,
    ],
    kCGImagePropertyHEIFDictionary: [
        kIIOMetadata_CameraModelKey: [
            kIIOCameraModel_Intrinsics: [
                1676.249856948853, 0, 1536,
                0, 1676.249856948853, 1536,
                0, 0, 1
            ] as CFArray
        ]
    ]
]

Add the left and right image to you image destination, passing the properties to each call:

CGImageDestinationAddImage(destination, leftImage, properties as CFDictionary)
CGImageDestinationAddImage(destination, rightImage, properties as CFDictionary)

Finalize the image:

CGImageDestinationFinalize(destination)

You should now have a spatial photo stored at url that you can view in the Apple Vision Pro simulator. It is worth noting there are other property values you may specify, however the ones above are the bare minimum for getting a spatial photo to display in the simulator. The example spatial photo from Apple contains values for camera extrinsics (with a position offset indicating it was taken with a two-camera system) as well as a horizontal disparity adjustment.

Reading Spatial Photos

Reading a spatial photo is as easy as creating a CGImageSource pointing to the spatial photo URL and copying the left and right image from it:

let source = CGImageSourceCreateWithURL(url as CFURL, nil)!
let leftImage: CGImage = CGImageSourceCreateImageAtIndex(source, 0, nil)!
let rightImage: CGImage = CGImageSourceCreateImageAtIndex(source, 1, nil)!

Reading and Writing Spatial Video with AVFoundation

Mon, 01 Jan 2024 00:00:00 GMT

Reading Spatial Video using AVAssetReader
Reading Spatial Video using AVPlayer
Writing Spatial Video using AVAssetWriter

If you've worked with AVFoundation's APIs, you'll be familiar with CVPixelBuffer, an object which represents a single video frame. AVFoundation manages the tasks of reading, writing, and playing video frames, but the process changes when dealing with spatial video (aka MV-HEVC), which features video from two separate angles.

Loading a spatial video into an AVPlayer or AVAssetReader on iOS appears similar to loading a standard video. By default, however, the frames you receive only show one perspective (the "hero" eye view), while the alternate angle, part of the MV-HEVC file, remains uncompressed.

With iOS 17.2 and macOS 14.2, new AVFoundation APIs were introduced for handling MV-HEVC files. They make it easy to get both angles of a spatial video, but are lacking in documentation[^1]. Here's a few tips for working with them:

Reading Spatial Video using AVAssetReader

AVAssetReader can read media data faster than realtime, and is good for reading a video file, applying some transform, and writing back to a file using AVAssetWriter (for example, converting a spatial video to an SBS video playable on a Meta Quest / XREAL Air). Reading spatial video requires informing VideoToolbox that we want both angles decompressed from the video, instead of just the "hero" eye.

Create an AVAssetReader:

let asset = AVAsset(url: <path-to-spatial-video>)
let assetReader = try AVAssetReader(asset: asset)

Create an AVAssetReaderTrackOuptut, specifying that we want both MV-HEVC video layers decompressed:

let output = try await AVAssetReaderTrackOutput(
    track: asset.loadTracks(withMediaType: .video).first!,
    outputSettings: [
        AVVideoDecompressionPropertiesKey: [
            kVTDecompressionPropertyKey_RequestedMVHEVCVideoLayerIDs: [0, 1] as CFArray,
        ],
    ]
)
assetReader.add(output)

Start copying sample buffers containing both angles:

assetReader.startReading()

while let nextSampleBuffer = output.copyNextSampleBuffer() {
    guard let taggedBuffers = nextSampleBuffer.taggedBuffers else { return }

    let leftEyeBuffer = taggedBuffers.first(where: {
        $0.tags.first(matchingCategory: .stereoView) == .stereoView(.leftEye)
    })?.buffer
    let rightEyeBuffer = taggedBuffers.first(where: {
        $0.tags.first(matchingCategory: .stereoView) == .stereoView(.rightEye)
    })?.buffer

    if let leftEyeBuffer,
       let rightEyeBuffer,
       case let .pixelBuffer(leftEyePixelBuffer) = leftEyeBuffer,
       case let .pixelBuffer(rightEyePixelBuffer) = rightEyeBuffer {
        // do something cool
    }
}

Reading Spatial Video using AVPlayer

When dealing with real-time playback, you'll often want to use AVPlayer, which manages the playback and timing of a video automatically. AVPlayerVideoOutput is the new API added for reading spatial video in real time, and is straightforward to set up.

Create an AVPlayer:

let asset = AVAsset(url: <path-to-spatial-video>)
let player = AVPlayer(playerItem: AVPlayerItem(asset: asset))

Create an AVPlayerVideoOutput for outputting stereoscopic video:

let outputSpecification = AVVideoOutputSpecification(
    tagCollections: [.stereoscopicForVideoOutput()]
)
let videoOutput = AVPlayerVideoOutput(specification: outputSpecification)
player.videoOutput = videoOutput

Add a periodic time observer for reading frames at a specified interval:

player.addPeriodicTimeObserver(
    forInterval: CMTime(value: 1, timescale: 30),
    queue: .main
) { _ in
    guard let taggedBuffers = videoOutput.taggedBuffers(
        forHostTime: CMClockGetTime(.hostTimeClock)
    )?.taggedBufferGroup else { return }

    let leftEyeBuffer = taggedBuffers.first(where: {
        $0.tags.first(matchingCategory: .stereoView) == .stereoView(.leftEye)
    })?.buffer
    let rightEyeBuffer = taggedBuffers.first(where: {
        $0.tags.first(matchingCategory: .stereoView) == .stereoView(.rightEye)
    })?.buffer

    if let leftEyeBuffer,
       let rightEyeBuffer,
       case let .pixelBuffer(leftEyePixelBuffer) = leftEyeBuffer,
       case let .pixelBuffer(rightEyePixelBuffer) = rightEyeBuffer {
        // do something cool
    }
}

Play!

player.play()

Writing Spatial Video using AVAssetWriter

Following the advice in Q&A: Building apps for visionOS, the steps for creating a spatial video from two stereo videos are:

Create an AVAssetWriter:

let assetWriter = try! AVAssetWriter(
    outputURL: <path-to-spatial-video>,
    fileType: .mov
)

Create and add a video input for the spatial video. It is important to specify kVTCompressionPropertyKey_MVHEVCVideoLayerIDs, kCMFormatDescriptionExtension_HorizontalFieldOfView, and kVTCompressionPropertyKey_HorizontalDisparityAdjustment in the compression properties. Without these, your video will not be read as a spatial video on visionOS:

let input = AVAssetWriterInput(
    mediaType: .video,
    outputSettings: [
        AVVideoWidthKey: 1920,
        AVVideoHeightKey: 1080,
        AVVideoCompressionPropertiesKey: [
            kVTCompressionPropertyKey_MVHEVCVideoLayerIDs: [0, 1] as CFArray,
            kCMFormatDescriptionExtension_HorizontalFieldOfView: 90_000, // asset-specific, in thousandths of a degree
            kVTCompressionPropertyKey_HorizontalDisparityAdjustment: 200, // asset-specific
        ],
        AVVideoCodecKey: AVVideoCodecType.hevc,
    ]
)
assetWriter.add(input)

Create an AVAssetWriterInputTaggedPixelBufferGroupAdaptor for the video input:

let adaptor = AVAssetWriterInputTaggedPixelBufferGroupAdaptor(assetWriterInput: input)

Start writing:

assetWriter.startWriting()
assetWriter.startSession(atSourceTime: .zero)

Start appending frames. Each frame consists of two CMTaggedBuffers:

let left = CMTaggedBuffer(tags: [.stereoView(.leftEye), .videoLayerID(0)], pixelBuffer: leftPixelBuffer)
let right = CMTaggedBuffer(tags: [.stereoView(.rightEye), .videoLayerID(1)], pixelBuffer: rightPixelBuffer)
adaptor.appendTaggedBuffers([left, right], withPresentationTime: <presentation-timestamp>)

Finish writing:

input.markAsFinished()
assetWriter.endSession(atSourceTime: <end-time>)
assetWriter.finishWriting {
    // share assetWriter.outputURL
}

[^1]: Since publishing this article, Apple has added sample code for both reading multiview 3D video files and writing multiview HEVC.