Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

Okay, good morning, folks. Thank you for showing up. Uh, I'm Nidhi, I'm a product person at Google DeepMind, and today I'll be talking about multimodal collaborative agents. Typically these are agents that work with fuzzy intent, intent when the intent when it's not clear, the user hasn't gotten theright keywords to specify.

And when they come in for a shopping intent, how can you have the agents still guide the user towards their goal with high agency execution, proactive elicitation, and a lot of handholding. The frameworks that we'll be discussing today, uh, are grounded in shopping or commerce because I wanted to show you a few patterns that are much easier to see in commerce, but they're quite applicable in other consumer verticals as well: finance, education, or whatever you guys work on.

Um, yeah, so before we even start into the frameworks, I wanted to discuss about why this is an existing problem. Currently, a lot of the agents that we have act more like a wrapper to the search bar. They assume that the user has a well-defined intent, has theright keywords, already knows what they're looking for, and so when they come in, they just have theright vocabulary to interact with the agent.

However, there is quite a huge articulation gap. When the users come in, rarely they have their intent well-formed; rather, they kind of have a fuzzy feeling or a vibe when they are kind of looking to shop. So the agent needs to play quite a huge role in handholding them.

They need to work with the user in first understanding their preferences or even eliciting these preferences more proactively, and then helping the user, uh, kind of show them different possibilities of what could be possible that they can shop for, and then eventually move towards, uh, kind of recommendations that work with these constraints that they have in mind.

So what we're going to be discussing is this kind of a flywheel or a loop that goes from very, very fuzzy intent all the way towards, uh, kind of achieving the user's goal. So this is how the shopping loop looks likeright now.

The very first thing that the agent prepares for when the user comes in is what we call the discovery phase. This is where the u where the agent takes all the different, uh, contextual information that they have about the user.

This could be past conversations, what the user has specified in their query. It could be information present in their personal context or even the references that the user has provided. And then comes up with a collaborative strategy. And we'll be going into the details of this, but it comes up with a collaborative strategy on what more does the agent need to elicitate back from the user in order to help get the intent in a better shape, to get clarity on what exactly the user is looking for.

And then it moves to the second phase, which is what we call the research phase, which is, again, a two-step process. First is where it learns what is the best way to elicitate this preference. So, for example, a lot of times the text the text-based, uh, elicitation may not be the best way to get the preferences from the user because sometimes the user don't know what they're looking for.

So then how can the agent start being more creative in terms of elicitating these preferences? Can they start using some kind of visual references or visual visual inspiration boards for grounding and speaking a common language with the user?

And then the second phase to this is also coming up, uh, going into the background and doing the heavy lifting for the user. So taking the burden away from the user to describing what they want and rather going into the background and doing all the comparisons, trade-offs, uh, summarization of all the information that they're looking for, and coming back with theright set of, uh, potential options for the user.

And then the third phase, once, um, the user and the agent is ready to go into some kind of, uh, uh, is ready to go into the last phase, which is where, um, the agent is ready to give out a response.

This is where the agent needs to start adapting the response in a way that is most useful for the user. So typically a lot of systems fail here where they just give out a text-heavy response. What the agent should rather be doing is adapting to the query that the user had in mind.

So should it be using, uh, some kind of a bulleted list? Should it be using comparison tables? Should it be using, um, visual boards? So this is where the there's adaptive response happening as well, where the, uh, agent starts to develop, uh, more, more of a, uh, smarter, uh, presentation skills, uh, for the user to really find the answer that they're looking for.

So we'll be diving into all these topics as I, uh, kind of work through the presentation. So first and foremost, we're looking at discovery. Like I was mentioning, this is where, uh, the, the, the agent needs to remember what exactly matters.

Um, you the agent starts to look at the context across a bunch of different signals. It starts to look at past conversations. It looks at some of the reference images or reference links that the user might have provided.

It also starts to look at personal context and starts to build out a working state. So if you look at the sample code that we have here, like, there is there is a goal. Uh, one of the let's, let's work with this query where the user is trying to redo their living room with a certain budget in mind.

And some of the, um, some of the things that the agent develops as part of the working state is the session history. It has a user context. It also kind of extracts out the hard constraint that the user might have provided in the query.

But things start get getting interesting when we come to the softer constraints. So this is where the user may not be able to describe what they're looking for and may might have provided, like, a reference image on, you know, an inspiration that they had in mind or some kind of a layout design that they really liked and is trying to get to the agent in terms of, uh, this is what speaks more to me.

So this is where the agent starts to be more proactive and pulls out some of the salient signals from the from the reference images and starts to develop a mental model of what the user might really be looking to get at.

So here the user is, uh, so the agent has specified, uh, has identified that maybe the style that is working out well for them is of a certain kind. It also, uh, is also is also starting to work towards a confidence score on, like, how confident it is in terms of pulling out some of this information present in the multimodal inputs provided.

And then the last thing, uh, that happens as part of developing this working state is also figuring out what are the variables that the agent needs to pull out almost in real time because these variables, uh, depend this variables do affect how the results will be displayed back to the user.

This could be variables that need to be refreshed in real time, like, uh, inventory, because if, if what you're providing back to the user is stale information, then it's kind of a moot point. So these are the, uh, these are the variables that you want to refresh in real time and it's, it's, it's and becomes a part of the agent's working state.

Um, few ways that you can evaluate this state, uh, the way we have developed our autoraters, we, we do make sure that all facts are retained, meaning that whatever was mentioned in the part of, uh, whatever was mentioned in part of the context is properly represented in the working state.

We also capture if the confidence collaboration was within a certain error bound because, uh, the agent needs to be able to confidently pull out these signals from the input. We also focus on getting out, um, the counterfactual sensitivity, uh, counterfactual sensitivity.

So we do this by flipping some parts of the queries and making sure that when the query changes, the underlying constraints pulled out by the agent, the those also change, and the ones that are not relevant stay the same.

So we kind of want to measure the sensitivity both ways. And then the second part that happens in the discovery phase as well is once the agent knows what they have what they already know about the user, what more do they need to know about the user?

So what we call is this what we call this is kind of discovering the intent gap. Um, there's a lot of times that there are a lot of unknown variables before the agent can provide the best answer. And amongst all these unknown variables, the agent doesn't need to go and find all the unknown variables up front.

So what I mean by that is in this our unknown in, in our working example, some of the unknown variables that the agent said they need to find out more about is maybe knowing what the room width would be for the user or even kind of working on improving the confidence for the style before they can recommend back the results.

Now, uh, once these unknown variables are figured out, the agent also needs to work on a collaborative strategy, compare all the different moves possible, and then figure out what is that one unknown variable that it should prioritize such that it has the maximal information gain at that point.

So in this case, um, I mean, one could argue that maybe finding out the room width is the best next move for the agent because if the products that the, the agent is recommending doesn't fit into the room, then again, it's a moot point.

And that is one variable that is going to meaningfully change the, uh, the direction of the conversation. And that's what the agent works on in this step, kind of figuring out what's the best next thing to ask to the user and what's, uh, why is that the best next thing to ask as well.

Once it has, uh, okay, so I'll get into the autoraters section on, like, why how do we evaluate this collaborative strategy. We first focus on making sure that the agent is able to identify all the different blockers that are needed to be answered before the agent can come back with, uh, meaningful responses.

We also work towards making sure that the agent is, um, optimal in trying to get some of these responses. So we also don't want to have the agent constantly going into the loop and continually continuously asking these questions.

So overasking is definitely something we flag. We also work on, uh, question utility. So again, how optimally is the question being asked? Is the question indeed useful to get theright or elicit theright preference from the user? And many more.

Okay, uh, moving on. The, the second part that I was mentioning is the multimodal elicitation. This is where the research phase happens. So play along, but, like, let's say the, the agent has asked about the room width. The user has provided the response.

And then the agent goes to the next step, which is where the agent is asking about, uh, the next constraint that it needs to know about is what was the style preference of the user. Now, the first and the first thing that the agent needs to do is kind of form this temporary bridge between the constraint itself and how that maps back to the, um, the, the product catalog and the ontology and your knowledge knowledge database.

And this is going to be important because when you start retrieving these products, you want to have a way to map these constraints back to your knowledge base. So this almost happens in real time where we map the, uh, known constraint or the constraint that the agent is exploring back to the knowledge database.

We also work towards, um, figuring out we also work on, like, the agent knowing what is the best way to get the response for this constraint as well. So in this case, the agent has decided that maybe since this is this is kind of a subjective constraint, the textual textual elicitation is not the best way to do this.

So one of the ways that the agent thinks this could, uh, this could be, uh, elicited back from the user is using some kind of a visual preference, um, board. So the agent then goes back to determining what is the best form of options to show to the user.

This could be a combination of figuring out from the existing constraints, the past conversation, and then the temporary mapping that you have created from the constraints back to your product ontology. So in this case, the, the agent thinks that maybe coming up with a few styles that are most similar to what the user had provided as a reference image could be a good way to start thinking or guiding the user towards a common language on what could be, uh, something that the user is interested in.

And then the, the, uh, then the agent also goes into kind of, um, observing the space of what kind of reactions the user is giving. So it could start looking at these micro signals of if there was a hover or a click in a certain direction and starts updating his con its confidence model on what kind of signals, uh, what kind of signals can be used to im improve the confidence and, like, what could be the style preference for the user.

Um, some of the autoraters we use here, we do look at how efficient the ag how efficient the agent is in discovering hidden preferences. So typically we would use a user simulator, feed it with some constraints, and then we'll see how, how efficient the agent was in kind of eliciting some of these constraints.

We also focus on turn efficiency. So how efficient was the user in, uh, how many turns did it take for the user to be able to elicit to be able to uncover all these hidden preferences. Ideally, we don't want the user to go into this loop and keep asking the same questions and again, again and again.

Or also we don't want to go into the loop of asking some, um, some questions which may not give you the best, uh, which may not give you the best information required to proceed the conversation. And then we also look at format format selection accuracy.

So we typically also look for if the agent is asking theright,right question in theright format. So for example, if the if the question was something that was easily statable, theright format could be a textual elicitation. But if, if this was more of a fuzzy question where the user is clearly having an articulation gap and is not able to describe the preference, maybe the best way to do this is speak a common language and come up with some visual, uh, anchor points for the user to say what speaks more to them.

Um, yeah. So then the last step in this process, once the so to recap, like, basically the now, now the agent knows exactly what they what the user is looking for. The agent has identified a collaboration strategy, has figured out all the preferences for the user, and knows what is going to be most optimal in terms of, um, the different priorities for the user.

The last step in this process is to also use the model intelligence to figure out what is the best way to provide this response back to the user. So in this working example, we found out that the agent knows what the style preference is, has figured out that the user is looking to buy products under a certain budget, and knows what are the different dimensions along which it needs to find, uh, these products for because through the conversation, they figured out some of the different, um, different metadata information that is going to be relevant to surface when they're giving out this product information back to the user.

So one, one important step that happens at this, this stage is figuring out what's the best way to surface back this response. So for example, like, if the user was looking for a particular policy or review information about a particular product, maybe the best way to do this is to go give out a summary or a bulleted list.

But if the if the user was looking more towards comparing two different products, maybe the best response is to give out a trade-off table or a comparison table across the different axes that the user cares about. Um, and in case that in our example where the user was looking more towards kind of style inspiration or ideas on how they can kind of redo their room, then maybe the best way is to give out some visual references and product inspiration photos on, like, how what are the different options and possibilities, uh, for the user to uncover.

So some of the ways that we focus on, uh, evaluating, uh, this stage is focusing on the format accuracy. So we want to make sure that the response format is kind of optimal for the user query so that they can find out exactly what they're looking for.

Uh, ideally, the information that they're looking for should not be buried in the response, but should be easy for the user to spot so they can commit to the next stage in the intent journey, which is to basically buy the product.

Uh, we also look at, uh, data fidelity, which is to make sure that the model is not hallucinating and it's really capturing the information in the correct format in the, you know, the, the information is just accurate and is captured across, um, the response.

We also look at user actionability. So this is ensuring that the response format is such that, uh, the user is very confident and com and commits to the next action, which is, like I said, the action to purchase the product.

So just to recap so far, what we have is, um, you want to design the product. Um, you want to you want to design the product such that you're prepared to accept wives, like I say. So that is to say that users will come with fuzzy intent.

Users will not have a well-defined goal. So you want to make sure that your system is able to work through queries that are not clean. Um, the second takeaway is you want to focus on showing and asking rather than always asking with textual pre uh, text with textual elicitations.

Again, like, visuals and comparisons do reveal preferences much, much faster. It allows you, the agent and the user to speak a common language. Third one I would say is, uh, shape the answer. So do focus on making sure that the presentation format is ideal for the user being able to find theright information.

Um, the, the way you have the model response structure is also very much part of the, uh, intelligence. And then the last one is make sure that you grade the loop. You have theright autoraters set up on every step of the process.

Um, and honestly, developing these autoraters, uh, is a is, is almost like an evolving system. It starts very simple, but as in when you start, the, the system starts evolving, you want the autoraters to kind of gradually grow with your system and start, um, um, yeah, it should just gradually grow with your system.

So that's all I had. Um, I can take a few questions, uh, but hopefully the learnings we shared are useful for whatever you folks are building. Yeah.

Yeah. Uh, so this vision is from the version from the, uh, shopping agent perspective.

Yeah.

While you ship in Gemini or ChatGPT. But what about the, uh, the merchant agent merchant side, for example? How is soft soft policies, hard policies, also other products can be surfaced? Should the merchants have some certain ontology structure, policy structure, uh, shaped in the form of, let's say, UCP protocol?

And also within the retrieval side from the merchant product, should it be also structured in a certain way so the agent can, can basically retrieve them in a high fidelity and also be fair to the merchant, uh, uh.

Yeah.

In regards to.

Yeah, great question. So the question is about what should the how should the ontology be structured on the merchant side so that it's fair both for the agent and the merchants. So yes, we do take a lot of advantage on the, uh, domain expertise of the merchant on as to what they're trying to sell.

And, like, we do work towards creating so you remember how I was mentioning about the bridge between, uh, the constraints that the user might have specified and then what the agent, uh, kind of understands. That is where we do expect a lot of intelligence to flow from the merchant side where the ontology on how that constraint could map to the different metadata that the agent has sorry, the merchant has maps in.

So yes, we do partner a lot. And then there's also, like, the UCP stuff that we launched recently, which allows all the merchants to kind of start speaking the common language with the agent as well.

So when you mentioned the visual aspects, so there you, you basically expect to have some certain, uh, like, uh, structured web. The one is, like, uh, like, the MCP apps, for example, to, to have, like, a certain way how to have, uh, good visuals from the merchant.

Or, like, you want to let the agents design the generative UI?

Yeah. So ideally, I mean, honestly,right now we focus on making sure all of this flows back to the agent and the agent makes the decisions because you want to build, like, a horizontal common layer across all the different merchants.

So and also, like, it should be a seamless experience for the, uh, for the user who's interacting with our apps. Soright now the response format is very much part of, uh, the agent's intelligence. It is not something that the merchant gets to, uh, decide.

Okay.

Yeah. Yes.

Um, just curious your opinion, what happens when your user is not a user anymore? It's an agent.

That's a great question. Uh, in the early stages of building this out, but I think I mean, there could be a case where, um.

Where an MCP would come in and.

Yeah.

Propose.

Yeah. Yeah, exactly. So yeah, I would I would expect, like, an MCP to be the interface between the two for sure. Uh, honestly, we haven't gotten to a point where we have agents interacting with our agents just yet.

Also, like, what we've realized, at least from our user studies, is, uh, users really like to be more involved in the process of choosing or even exploring the different possibilities. So during the upper funnel journeys where users is looking more towards discovery, inspiration, that is where they would rather be interacting with the system than in their agent.

I think where the agent typically comes in or even where, uh, what we've heard is, like, towards the lower end of the journey where they're just looking to compare or negotiate or compare prices across different merchants, but very much upper in the funnel, it's the users who kind of interact more with our systems.

So yeah.

Yeah. I think, um, yeah, I can take questions outside, but thank you folks for coming.