Apple invents a Proactive & Reactive assisted Transcription app for FaceTime Calls on Apple Hardware including an MR Headset - Patently Apple

On Thursday, Patently Apple discovered an Apple patent filed with the World Intellectual Property Organization relating to a caller transcriptions app and, much specifically, to the procreation of and assistance with conversational and biology transcriptions. Apple's patent describes techniques for generating transcriptions and providing proactive and reactive assistance with transcriptions. The app volition beryllium made disposable for FaceTime calls oregon bureau league calls connected iDevices, Macs and Apple's aboriginal Mixed Reality Headset.

In Apple's patent inheritance they enactment that accepted systems bash not efficaciously supply proactive and reactive assistance based connected these transcriptions, nor bash specified systems efficaciously make transcriptions based connected conversational discourse oregon biology factors. For example, accepted systems bash not connection users an businesslike means by which to rapidly reappraisal portions of a transcription based connected circumstantial parameters, specified arsenic conversational topics, biology conditions, and the like. Such systems besides bash not connection idiosyncratic assistance based connected a user’s attentive state, specified arsenic erstwhile the idiosyncratic becomes distracted from the conversation. Thus, an improved strategy for transcriptions and transcription assistance is desired.

Apple's patent describes techniques for generating transcriptions and providing proactive and reactive assistance with transcriptions.

In general, transcriptions tin beryllium adjuvant to reappraisal and summarize accusation related to conversations oregon different interactions betwixt parties. Given the summation of conversational connection betwixt devices, and the technological advances of exertion connected specified devices, conversational transcription tin present beryllium efficaciously utilized.

In addition, assorted technologies whitethorn lend to effectual translations with respect to an environment, specified arsenic an situation associated with extended world oregon akin technologies.

Overall, Apple's patent covers systems and processes for transcriptions and transcription assistance. For example, a textual practice of a speech betwixt a idiosyncratic and astatine slightest 1 speech subordinate is obtained. Based connected the textual representation, contented associated with the speech is identified, wherein the contented includes astatine slightest 1 of a archetypal input from the idiosyncratic and a 2nd input from the astatine slightest 1 speech participant. In effect to a determination that the contented is associated with predefined content, a information of the textual practice is identified based connected the content. Based connected the identified portion, an output responsive to the astatine slightest 1 of the archetypal input and the 2nd input is provided.

Apple's patent FIGS. 8A/C/E beneath exemplify a process for transcriptions and transcription assistance.

More specifically, Apple's patent FIG. 8A, sets up a speech betwixt a idiosyncratic and 1 oregon much different users. The speech whitethorn correspond to a dependable connection (e g., telephone call), a FaceTime league call,  a speech done a societal media platform, and astir interesting, a speech successful a virtual and/or augmented world setting.

For example, a idiosyncratic of an iPhone (electronic instrumentality #800) whitethorn beryllium engaged successful a telephone speech with different users. While the speech takes place, a textual practice (e.g., a transcription) of the speech whitethorn obtained.

Another diagnostic of this app whitethorn see a punctual that includes assorted options related to the transcription of the conversation. For instance, the punctual whitethorn further supply the subordinate with the enactment to anonymize oregon different modify oregon destruct identifying accusation from the respective participant’s inputs, specified that the obtained textual practice includes modified input from the respective participant.

A modified textual practice of the speech whitethorn see assorted modifications, specified arsenic anonymized idiosyncratic names (e.g., “User A: Hello ”) The modified textual practice whitethorn besides omit assorted items of information, specified arsenic idiosyncratic accusation (e.g., addresses, telephone numbers, relationship numbers, and the like).

A effect to a provided punctual is past received from devices associated with the assorted participants, including responses that whitethorn o.k. transcription, contradict transcription, oregon different o.k. a modified mentation of transcription for the respective participant.

Initiation of the transcription whitethorn hap successful assorted ways. For example, the idiosyncratic whitethorn bespeak a tendency to transcribe the speech done assorted configurations oregon settings anterior to the speech being initiated and the transcription support prompts being sent to the assorted users.

The idiosyncratic whitethorn besides supply an input during an already-established conversation, for example, by activating an affordance (icon) #802 depicted connected an progressive telephone surface successful FIG. 8A. In immoderate examples, the icon whitethorn beryllium utilized to toggle betwixt the progressive telephone surface and the textual practice of the speech (discussed successful portion via FIG. 8B), for example, erstwhile transcription has already been initiated.

In immoderate examples, initiation of the transcription whitethorn hap based connected assorted discourse information. For instance, a transcription of a speech whitethorn beryllium initiated successful effect to a respective threshold being exceeded, specified arsenic a sound threshold (e.g., the idiosyncratic is engaged successful a video telephone wrong a crowded supermarket).

As different example, the transcription whitethorn beryllium initiated successful effect to the detection of assorted trigger words oregon phrases. Specifically, 1 oregon much users participating successful the speech whitethorn utter a operation specified arsenic “can you repetition that,” “say that again,” “what was that?” and the like. In immoderate examples, the trigger connection oregon operation whitethorn correspond to an explicit petition from the idiosyncratic of the physics instrumentality to statesman a transcription, specified arsenic “Start the transcription now.”

Generally, proactive and reactive assistance utilizing the textual practice whitethorn beryllium provided to users and whitethorn beryllium based connected assorted factors. Referring to FIG. 8B above, successful immoderate examples, contented associated with the speech is identified based connected the textual representation, wherein the contented includes 1 oregon much inputs from the iPhone idiosyncratic and/or the different participants of the conversation. Such input whitethorn mostly trigger reactive assistance from the iPhone (and/or different devices associated with the conversation).

In particular, the input whitethorn correspond to code input, substance input, input from activating assorted affordances (icons), controlling 1 oregon much secondary devices, and the like. For example, the idiosyncratic whitethorn activate a mute button, stock assorted media items wrong the conversation, power virtual objects successful a virtual setting, etc.

Apple's patent FIGS. 9A-9B exemplify a process for transcriptions and transcription assistance. In a co-presence league (e g., within an AR/VR environment), assorted objects oregon idiosyncratic avatars whitethorn determination astir the user’s viewing perspective, participate oregon exit the environment.

The transcription app successful a FaceTime telephone wrong a Mixed Reality Headset could see added features. For instance, successful patent fig 9A above, representation 900 whitethorn correspond to the user’s surviving country which is physically located successful the metropolis of Atlanta, GA. Weather accusation corresponding to the existent determination whitethorn besides beryllium obtained, specified arsenic “sunny + 70 degrees. This could beryllium illustrated successful the Headset imagery of the idiosyncratic you're speaking with.

In Apple's patent FIG. 9B above, an lawsuit associated with practice #900 whitethorn beryllium detected, specified arsenic a 3rd idiosyncratic entering the environment. Accordingly, an updated acceptable of identifiers whitethorn beryllium retrieved based connected the detected event. For example, a carnal idiosyncratic whitethorn get astatine the determination represented by practice #900, specified arsenic by walking done doorway #902.

Alternatively, a idiosyncratic whitethorn participate the virtual league (e.g., utilizing call-in oregon log-in information), specified that an avatar associated with the idiosyncratic is displayed wrong practice #900.

For much details, reappraisal Apple's patent exertion fig WO2022266209

Apple Inventors

• Shiraz Akmal: Apple AI/ML Future Experiences. Mr. Akmal was CEO and Co-Founder of "Spaces Inc" that Apple acquired backmost successful the summertime of 2020. Spaces was a pioneer successful VR Videoconferencing. Image beneath from "Spaces Inc."
• Brad Herman: AI/ML Future Experiences. Herman was CTO and Co-Founder of Spaces Inc.
• Aaron Burns: Software Engineering Manager