Implementing SensitiveContentAnalysis in SwiftUI

With the release of iOS 17 came with it a new framework, SensitiveContentAnalysis, which allows developers, on device, to determine if visual content being presented to the user contains nudity.

When the framework flags user-provided media as sensitive, intervene by adjusting the user interface in a way that explains the situation and gives the user options to proceed. Avoid displaying flagged content until the user makes a decision.

With the private messaging app that I’m developing I wanted to leverage this functionality to keep privacy at the forefront, in addition, to the end-to-end encryption.

Setup

Initialization

The first task is to add the Sensitive Content Analysis entitlement: Without this the analyzer will fail.

Per Apple:
The SensitiveContentAnalysis entitlement is not available for Enterprise development or for people with free accounts.

Detection

For my app I have a model object called YAIAttachment. The object contains the realitve properties for the type, i.e. Video or Image and the encrypted URL / Media for the resource. I wrote a convience property to determine if the media “isSensitive”.

var isSensitive: Bool {
        get async throws {
            let analyzer: SCSensitivityAnalyzer = .init()
            
            /// You have to download the asset to the device locally. You can't load it from a remote URL
            /// or the validation won't work
            
            let mediaType: MediaTypeCategory = MediaTypeCategory(rawValue: self.mediaType) ?? .standard
            let remoteURL: URL = self.isAssetRemote ? self.viewAssetDecryptedURL! : self.asset!.fileURL!
            let (data, _) = try await URLSession.shared.data(from: remoteURL)
            
            if mediaType == .video {
                let videoURL: URL = data.yai_fileURL("mp4") ?? URL(fileURLWithPath: "")
                let response = analyzer.videoAnalysis(forFileAt: videoURL)
                let isSensitive = try await response.hasSensitiveContent().isSensitive
                return isSensitive
            }
            
            let imageURL: URL = data.yai_fileURL("png") ?? URL(fileURLWithPath: "")
            let response = try await analyzer.analyzeImage(at: imageURL)
            return response.isSensitive
        }
    }

There are two important caveats to using the analyzer:

1. This is a concurrent operation
2. The analyzers for images and videos won’t work if you pass in a remote URL

As my example shows above, writing an async computed property is pretty straightforward, but it took me a bit to realize the analyzer was failing because the urls I was passing in were remote. To “work around” this was to download the asset, store it locally and pass that temp url to the analyzer.

What Next

Now that we have everything setup to determine if the asset is sensitive or not, what do we do about it. The answer is…well…whatever you want. Referring back to Apple’s doc:

The SensitiveContentAnalysis framework doesn’t dictate your user interface. You can tailor your app’s experience according to the examples in apps such as Messages.

At first I wasn’t a fan of this and had hoped Apple had provided an OOTB UI, but being able to use a branded UI that fits with my layout and UX made much more sense.

    @ViewBuilder
    private func sensitiveWarningView(_ attachment: YAIAttachment) -> some View {
        if isSensitiveAttachment(attachment) {
            VStack(spacing: 10){
                Text("Sensitive Content")
                    .bold()
                    .font(.title)
                    .fontDesign(.rounded)
                HStack(spacing: 5) {
                    Image(systemName: "lock.trianglebadge.exclamationmark.fill")
                        .font(.title)
                    Text("Tap to View")
                }
            }
            .unredacted()
            .onTapGesture {
                if let recordIndex: Int = self.viewModel.sensitiveAttachments.firstIndex(of: attachment.record.recordID) {
                    self.viewModel.sensitiveAttachments.remove(at: recordIndex)
                    self.viewModel.visibleSensitiveAttachments.append(attachment.record.recordID)
                }
            }
        }
    }
        @ViewBuilder
    private func standardImageView(_ attachment: YAIAttachment) ->some View {
        AsyncImage(url: attachment.viewAssetDecryptedURL!) { phase in
            if let image = phase.image {
                /// if the image is valid
                image
                    .resizable()
                    .aspectRatio(contentMode: .fit)
                    .clipped()
                    .overlay(
                        sensitiveWarningView(attachment)
                    )
            } else if phase.error != nil {
                /// some kind of error appears
                Text("404! \n No image available 😢")
                    .bold()
                    .font(.title)
                    .multilineTextAlignment(.center)
                
            } else {
                /// showing progress view as placeholder
                ProgressView()
                    .font(.largeTitle)
            }
        }
    }

standardImageView(attachment)
    .redacted(reason: isSensitiveAttachment(attachment) ? .placeholder : [])
    .animation(.easeInOut, value: self.viewModel.sensitiveAttachments.count)
    .onTapGesture {
  // toggle to remove the warning overlay
    }

References
* SCSensitivityAnalyzer
* Testing your app’s response to sensitive media