cupertino-docs/docs/visualintelligence/documentation_visualintelligence_integrating-your-app-with-visual-intelligence.json

{
  "abstract" : "Enable people to find app content that matches their surroundings or objects onscreen with visual intelligence.",
  "codeExamples" : [

  ],
  "contentHash" : "4fee816751fc5e36e959622ee84976e74024e4b7243ba45e9e6cf7a3625b6ab7",
  "crawlDepth" : 1,
  "crawledAt" : "2026-05-08T23:27:07Z",
  "declaration" : {
    "code" : "struct LandmarkEntity: IndexedEntity {\n    static var typeDisplayRepresentation: TypeDisplayRepresentation {\n        return TypeDisplayRepresentation(\n            name: LocalizedStringResource(\"Landmark\", table: \"AppIntents\", comment: \"The type name for the landmark entity\"),\n            numericFormat: \"\\(placeholder: .int) landmarks\"\n        )\n    }\n\n\n    var displayRepresentation: DisplayRepresentation {\n        DisplayRepresentation(\n            title: \"\\(name)\",\n            subtitle: \"\\(continent)\",\n            image: .init(data: try! self.thumbnailRepresentationData)\n        )\n    }\n\n\n    \/\/ ...\n}",
    "language" : "swift"
  },
  "id" : "41FF8BB1-77A4-B43E-3D8D-BC5564253A13",
  "kind" : "protocol",
  "overview" : "Visual Intelligence  Integrating your app with visual intelligence  Integrating your app with visual intelligence ArticleIntegrating your app with visual intelligenceEnable people to find app content that matches their surroundings or objects onscreen with visual intelligence.OverviewWith visual intelligence, people can visually search for information and content that matches their surroundings, or an onscreen object. Integrating your app with visual intelligence allows people to view your matching content quickly and launch your app for more detailed information or additional search results, giving it additional visibility.\n\nTo integrate your app with visual intelligence, the Visual Intelligence framework provides information about objects it detects in the visual intelligence camera or a screenshot. To exchange information with your app, the system uses the App Intents framework and its concepts of app intents and app entities.\n\nWhen a person performs visual search on the visual intelligence camera or a screenshot, the system forwards the information captured to an App Intents query you implement. In your query code, search your app’s content for matching items, and return them to visual intelligence as app entities. Visual intelligence then uses the app entities to display your content in the search results view, right where a person needs it.",
  "rawMarkdown" : "---\nsource: https:\/\/developer.apple.com\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence\ncrawled: 2026-05-08T23:27:07Z\n---\n\n# Integrating your app with visual intelligence | Apple Developer Documentation\n\n- [ Visual Intelligence ](\/documentation\/visualintelligence)\n\n- [ Integrating your app with visual intelligence ](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence)\n\n-  Integrating your app with visual intelligence \n\nArticle# Integrating your app with visual intelligence\n\nEnable people to find app content that matches their surroundings or objects onscreen with visual intelligence.## [Overview](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Overview)\n\nWith visual intelligence, people can visually search for information and content that matches their surroundings, or an onscreen object. Integrating your app with visual intelligence allows people to view your matching content quickly and launch your app for more detailed information or additional search results, giving it additional visibility.\n\n### [Explore the role of the App Intents framework](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Explore-the-role-of-the-App-Intents-framework)\n\nTo integrate your app with visual intelligence, the Visual Intelligence framework provides information about objects it detects in the visual intelligence camera or a screenshot. To exchange information with your app, the system uses the [App Intents](\/documentation\/AppIntents) framework and its concepts of app intents and app entities.\n\nWhen a person performs visual search on the visual intelligence camera or a screenshot, the system forwards the information captured to an App Intents query you implement. In your query code, search your app’s content for matching items, and return them to visual intelligence as app entities. Visual intelligence then uses the app entities to display your content in the search results view, right where a person needs it.\n\nTo learn more about a displayed item, someone can tap it to open the item in your app and view information and functionality. For example, an app that allows people to view information about landmarks might show detailed information like hours, a map, or community reviews for the item a person taps in visual search.\n\n### [Provide a display representation](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Provide-a-display-representation)\n\nVisual Intelligence uses the [`DisplayRepresentation`](\/documentation\/AppIntents\/DisplayRepresentation) of your [`AppEntity`](\/documentation\/AppIntents\/AppEntity) to organize and present your content in the visual intelligence search experience. Make sure to provide localized, concise, and high-quality display representations that consist of a title, subtitle, and an image. The following code from the [Adopting App Intents to support system experiences](\/documentation\/AppIntents\/adopting-app-intents-to-support-system-experiences) sample code project shows the display representation of an `AppEntity` for a landmark. It uses strings from the model object for simplicity. In your code, make sure to provide a localized display representation.\n\n\n\n```\nstruct LandmarkEntity: IndexedEntity {\n    static var typeDisplayRepresentation: TypeDisplayRepresentation {\n        return TypeDisplayRepresentation(\n            name: LocalizedStringResource(\"Landmark\", table: \"AppIntents\", comment: \"The type name for the landmark entity\"),\n            numericFormat: \"(placeholder: .int) landmarks\"\n        )\n    }\n\n\n    var displayRepresentation: DisplayRepresentation {\n        DisplayRepresentation(\n            title: \"(name)\",\n            subtitle: \"(continent)\",\n            image: .init(data: try! self.thumbnailRepresentationData)\n        )\n    }\n\n\n    \/\/ ...\n}\n\n```\n\nFor additional information about display representations, refer to [Integrating custom data types into your intents](\/documentation\/AppIntents\/Integrating-custom-types-into-your-intents#Provide-a-visual-representation-for-your-entity).\n\n### [Provide search results](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Provide-search-results)\n\nTo integrate your app with visual search, provide visual intelligence with content that matches a person’s surroundings or onscreen object, as described in the steps below and illustrated in the following image:\n\n- In your Xcode project, adopt the [`IntentValueQuery`](\/documentation\/AppIntents\/IntentValueQuery) protocol and implement its [`values(for:)`](\/documentation\/AppIntents\/IntentValueQuery\/values(for:)) requirement.\n\n- Change the [`values(for:)`](\/documentation\/AppIntents\/IntentValueQuery\/values(for:)) function to receive a [`SemanticContentDescriptor`](\/documentation\/visualintelligence\/semanticcontentdescriptor) as its `input`. The [`SemanticContentDescriptor`](\/documentation\/visualintelligence\/semanticcontentdescriptor) makes visual intelligence information available to your app.\n\n- Use the descriptor’s [`labels`](\/documentation\/visualintelligence\/semanticcontentdescriptor\/labels) to access a list of labels that visual intelligence creates or the [`pixelBuffer`](\/documentation\/visualintelligence\/semanticcontentdescriptor\/pixelbuffer) of the camera capture.\n\n- Search your app’s content using the labels and perform an image search with an image you create from the `pixelBuffer`.\n\n- Describe your search results as [`AppEntity`](\/documentation\/AppIntents\/AppEntity) objects and return them as the result of the query.\n\nNote\n\nLabels are general, high-level terms in the `en_US` locale and might change over time. Visual Intelligence doesn’t translate them or include synonyms. For example, `SemanticContentDescriptor` might provide the labels `tower` or `building` for a well-known building. It won’t provide the building’s actual name as a label.\n\nThe following example code from the [Adopting App Intents to support system experiences](\/documentation\/AppIntents\/adopting-app-intents-to-support-system-experiences) sample code project demonstrates how an app that enables people to view information about points of interest and landmarks might access the `pixelBuffer` for its search:\n\n\n\n```\nstruct LandmarkIntentValueQuery: IntentValueQuery {\n\n\n    @Dependency var modelData: ModelData\n\n\n    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {\n\n\n        guard let pixelBuffer: CVReadOnlyPixelBuffer = input.pixelBuffer else {\n            return []\n        }\n\n\n        let landmarks = try await modelData.search(matching: pixelBuffer)\n\n\n        return landmarks\n    }\n}\n\n```\n\nThe `search(matching:)` function asynchronously returns a list of app entities that represent landmarks. Returning results quickly makes for a good search experience, so make sure to limit the list of returned items, if needed. If your app finds a large number of matches — for example, several hundred items — you might return the first hundred results, and give people the opportunity to view the full list in your app as described in [Link to additional results in your app](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Link-to-additional-results-in-your-app).\n\nThe process for matching the provided pixel buffer to app entities depends on your app. A common case is to convert the pixel buffer into an image, then use the image in an image search. The following code snippet shows how you might implement this conversion:\n\n\n\n```\nprivate func createImage(_ pixelBuffer: CVReadOnlyPixelBuffer) -> CGImage? {\n    let context = CIContext()\n    let image = CIImage(cvPixelBuffer: pixelBuffer)\n    return context.createCGImage(image, from: image.extent)\n}\n\n```\n\n### [Open an item in your app](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Open-an-item-in-your-app)\n\nTo allow someone to open your app and view additional information or access additional actions for a visual search, create an [`OpenIntent`](\/documentation\/AppIntents\/OpenIntent). In the intent’s `perform()` method, open your app to match the app entity that visual intelligence passes to the method, as illustrated in the image below.\n\nContinuing the example that shows information about points of interest or landmarks, the `OpenIntent` might look like this:\n\n\n\n```\nstruct OpenLandmarkIntent: OpenIntent {\n    static let title: LocalizedStringResource = \"Open Landmark\"\n\n\n    @Parameter(title: \"Landmark\", requestValueDialog: \"Which landmark?\")\n    var target: LandmarkEntity\n}\n\n```\n\nNote\n\nIf your query returns more than one app entity type using `@UnionValue`, create an `OpenIntent` for each app entity type that’s part of the union value.\n\nAdopting the `OpenIntent` protocol isn’t specific to integrating your app with visual intelligence. Adopting App Intents, including one or more `OpenIntent` implementations, is a best practice for modern apps that offer additional integration with system experiences. If you’ve already adopted App Intents, you might be able to reuse existing code to open an item in your app with an `OpenIntent`.\n\nFor more information about adopting App Intents in your app, refer to [App Intents](\/documentation\/AppIntents) and [Making actions and content discoverable and widely available](\/documentation\/AppIntents\/Making-actions-and-content-discoverable-and-widely-available).\n\n### [Return different values in one query](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Return-different-values-in-one-query)\n\nYour app can’t contain more than one [`IntentValueQuery`](\/documentation\/AppIntents\/IntentValueQuery) that takes a [`SemanticContentDescriptor`](\/documentation\/visualintelligence\/semanticcontentdescriptor). To return more than one [`AppEntity`](\/documentation\/AppIntents\/AppEntity) type from a single intent value query, use the [`UnionValue()`](\/documentation\/AppIntents\/UnionValue()) Swift macro to return multiple app entity types. The following example uses a union value for its result — indicated by the `@UnionValue` annotation — to return a list of individual landmarks and collections of landmarks:\n\n\n\n```\n@UnionValue\nenum VisualSearchResult {\n    case landmark(LandmarkEntity)\n    case collection(CollectionEntity)\n}\n\n\nstruct LandmarkIntentValueQuery: IntentValueQuery {\n\n\n    @Dependency var modelData: ModelData\n\n\n    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {\n        \/\/ ...\n\n\n        \/\/ Returned search results are either landmarks or a collection.\n        let landmarks = try await modelData.search(matching: pixelBuffer)\n\n\n        return landmarks\n    }\n}\n\n```\n\n### [Link to additional results in your app](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Link-to-additional-results-in-your-app)\n\nReturning visual search results quickly and limiting the number of items ensures a quick and enjoyable experience for people using your app. However, your app might offer a lot — possibly hundreds — of results, or browsing long lists of items might be part of your app’s core experience. If you need to provide many results, display a limited amount and allow people to open your app from the “More results” button to view more visual search results.\n\nFirst, create a new app intent that conforms to the [`semanticContentSearch`](\/documentation\/AppIntents\/AssistantSchemas\/VisualIntelligenceIntent\/semanticContentSearch) schema. With App Intents domains and schemas, you can quickly create app intents that follow a predefined form to enable specific functionality, such as opening a content search experience or list of results.\n\nTip\n\nType `visualintelligence_`, choose the suggested semantic content search schema, and let Xcode code completion create the conforming app intent for you.\n\nIn the semantic content search intent’s `perform()` method, navigate to your app’s search experience and pass information that the [`SemanticContentDescriptor`](\/documentation\/visualintelligence\/semanticcontentdescriptor) object provides to perform a search and show the full list of results.\n\n## [See Also](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#see-also)\n\n### [Essentials](\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence#Essentials)\n\n[Adopting App Intents to support system experiences](\/documentation\/AppIntents\/adopting-app-intents-to-support-system-experiences)Create app intents and entities to incorporate system experiences such as Spotlight, visual intelligence, and Shortcuts.[`struct SemanticContentDescriptor`](\/documentation\/visualintelligence\/semanticcontentdescriptor)A type that represents a scene that visual intelligence captures, like a screenshot, photo, or photo and video stream.",
  "sections" : [
    {
      "content" : "",
      "title" : "Overview"
    },
    {
      "content" : "",
      "title" : "See Also"
    }
  ],
  "source" : "appleWebKit",
  "title" : "Integrating your app with visual intelligence | Apple Developer Documentation",
  "url" : "https:\/\/developer.apple.com\/documentation\/visualintelligence\/integrating-your-app-with-visual-intelligence"
}