The topic DeepMind details Googlebook ‘Magic Pointer’ with demos you can try, also coming to… is currently the subject of lively discussion — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
The Magic Pointer on Googlebook was built with Google DeepMind. The research team behind this underlying capability shared more about the premise of AI-enabled pointers.
DeepMind wants to use AI “to help the pointer not only understand what it’s pointing at, but also why it matters to the user.”
Our goal is to address a common frustration: because a typical AI tool lives in its own window, users need to drag their world into it. We want the opposite: intuitive AI that meets users across all the tools they use, without interrupting their flow. for example, imagine pointing to an image of a building, and requesting “Show me directions”. Nothing more is needed when the AI system already understands the context.
The idea is to replace “text-heavy prompts with simpler, more intuitive interactions.”
An AI-enabled pointer would streamline this process by smoothly capturing the visual and semantic context around the pointer, letting the computer “see” and understand what’s important to the user.
Similarly, an “AI system that understands this combination of context, pointing and speech would allow users to make complex requests in natural shorthand.”
In the example below, a “paused frame in a travel video becomes a booking link for that cool-looking restaurant.”
Additionally, you will soon have the ability to “use your pointer to ask Gemini in Chrome about the part of the webpage you care about.” This is in the process of rolling out.
for example, you can select a few products on a page and ask to compare, or point to where you want to visualize a new couch in your living room.