Introduction to AI-Assisted Search
A patent recently filed by Google outlines how an AI assistant may use at least five real-world contextual signals, including identifying related intents, to influence answers and generate natural dialog. It’s an example of how AI-assisted search modifies responses to engage users with contextually relevant questions and dialog, expanding beyond keyword-based systems.
How AI-Assisted Search Works
The patent describes a system that generates relevant dialog and answers using signals such as environmental context, dialog intent, user data, and conversation history. These factors go beyond using the semantic data in the user’s query and show how AI-assisted search is moving toward more natural, human-like interactions.
Purpose of Filing a Patent
In general, the purpose of filing a patent is to obtain legal protection and exclusivity for an invention, and the act of filing doesn’t indicate that Google is actually using it.
Key Points of the Patent
The patent uses examples of spoken dialog but it also states the invention is not limited to audio input:
“Notably, during a given dialog session, a user can interact with the automated assistant using various input modalities, including, but not limited to, spoken input, typed input, and/or touch input.”
The name of the patent is, Using Large Language Model(s) In Generating Automated Assistant response(s). The patent applies to a wide range of AI assistants that receive inputs via the context of typed, touch, and speech.
Five Factors that Influence LLM Modified Responses
There are five factors that influence the LLM modified responses:
- Time, Location, And Environmental Context
- User-Specific Context
- Dialog Intent & Prior Interactions
- Inputs (text, touch, and speech)
- System & Device Context
The first four factors influence the answers that the automated assistant provides, and the fifth one determines whether to turn off the LLM-assisted part and revert to standard AI answers.
Time, Location, And Environmental Context
There are three contextual factors: time, location, and environmental that provide contexts that are not existent in keywords and influence how the AI assistant responds. While these contextual factors, as described in the patent, aren’t strictly related to AI Overviews or AI Mode, they do show how AI-assisted interactions with data can change.
The patent uses the example of a person who tells their assistant they’re going surfing. A standard AI response would be a boilerplate comment to have fun or to enjoy the day. The LLM-assisted response described in the patent would generate a response based on the geographic location and time to generate a comment about the weather like the potential for rain. These are called modified assistant outputs.
The patent describes it like this:
“…the assistant outputs included in the set of modified assistant outputs include assistant outputs that do drive the dialog session in manner that further engages the user of the client device in the dialog session by asking contextually relevant questions (e.g., “how long have you been surfing?”), that provide contextually relevant information (e.g., “but if you’re going to Example Beach again, be prepared for some light showers”), and/or that otherwise resonate with the user of the client device within the context of the dialog session.”
User-Specific Context
The patent describes multiple user-specific contexts that the LLM may use to generate a modified output:
- User profile data, such as preferences (like food or types of activity).
- Software application data (such as apps currently or recently in use).
- Dialog history of the ongoing and/or previous assistant sessions.
Here’s a snippet that talks about various user profile related contextual signals:
“Moreover, the context of the dialog session can be determined based on one or more contextual signals that include, for example, ambient noise detected in an environment of the client device, user profile data, software application data, ….dialog history of the dialog session between the user and the automated assistant, and/or other contextual signals.”
Related Intents
An interesting part of the patent describes how a user’s food preference can be used to determine a related intent to a query.
For example, …one or more of the LLMs can determine an intent associated with the given assistant query… Further, the one or more of the LLMs can identify, based on the intent associated with the given assistant query, at least one related intent that is related to the intent associated with the given assistant query… Moreover, the one or more of the LLMs can generate the additional assistant query based on the at least one related intent.
The patent illustrates this with the example of a user saying that they’re hungry. The LLM will then identify related contexts such as what type of cuisine the user enjoys and the intent of eating at a restaurant.
The patent explains:
“In this example, the additional assistant query can correspond to, for example, “what types of cuisine has the user indicated he/she prefers?” (e.g., reflecting a related cuisine type intent associated with the intent of the user indicating he/she would like to eat), “what restaurants nearby are open?” (e.g., reflecting a related restaurant lookup intent associated with the intent of the user indicating he/she would like to eat)… In these implementations, additional assistant output can be determined based on processing the additional assistant query.”
System & Device Context
The system and device context part of the patent is interesting because it enables the AI to detect if the context of the device is that it’s low on batteries, and if so, it will turn off the LLM-modified responses. There are other factors such as whether the user is walking away from the device, computational costs, etc.
Takeaways
- AI Query Responses Use Contextual Signals: Google’s patent describes how automated assistants can use real-world context to generate more relevant and human-like answers and dialog.
- Contextual Factors Influence Responses: These include time/location/environment, user-specific data, dialog history and intent, system/device conditions, and input type (text, speech, or touch).
- LLM-Modified Responses Enhance Engagement: Large language models (LLMs) use these contexts to create personalized responses or follow-up questions, like referencing weather or past interactions.
- Examples Show Practical Impact: Scenarios like recommending food based on user preferences or commenting on local weather during outdoor plans demonstrates how real-world contexts can influence how AI responds to user queries.
Conclusion
This patent is important because millions of people are increasingly engaging with AI assistants, thus it’s relevant to publishers, ecommerce stores, local businesses, and SEOs. It outlines how Google’s AI-assisted systems can generate personalized, context-aware responses by using real-world signals. This enables assistants to go beyond keyword-based answers and respond with relevant information or follow-up questions, such as suggesting restaurants a user might like or commenting on weather conditions before a planned activity.
The patent can be read here.