I have been thinking a lot about the product manager interview in the AI-first world. I’ve been following how some companies are experimenting with the product manager interview and how that role is changing. I’m less interested in the actual interview formats themselves, but rather in whether or not we are asking the right questions on the most important capabilities.
I feel like there is a growing interest in asking candidates to build prototypes, have some familiarity with the architectures of AI, design evaluations, or have some reasoning around model performance and cost of inference. These are all interesting things to explore, but I wonder if we risk shifting from one set of interview conventions to another without truly appreciating what has fundamentally changed about the role.
Is the most important change really about technical fluency, or is it that AI is making something that has always been a core part of product management much more consequential? The ability to define what good looks like for a product experience, particularly when the underlying technology is not deterministic.
Changing Interviews
I recently read about a product management interview that started with a fairly typical product-sense exercise and then transitioned into a live prototyping session with an AI tool. After the candidate got something working, the interview moved into the area of latency, token consumption, retrieval, and what would be needed to make this experience production ready.
This is just one report, so I wouldn’t put too much stock into a specific company’s interview format. But it’s interesting to see where things might be going. As prototyping gets easier, the line between imagining a product and showing a product is starting to blur.
And that opens up an opportunity to understand how people think, rather than just how they describe how they think. But it also raises the question: what are we actually assessing? Is it the ability to create a functional prototype in 30 minutes, or is it the judgment on what to build, what doesn’t work, and what is needed to take something that is an interesting demo and make it a reliable product?
I would guess that the latter is a lot more revealing.
Prototyping is easier. Evaluating what you have built is harder.
One of the more interesting side effects of AI assisted development is how quickly an idea can become something tangible. As a product manager, you can now put together a pretty convincing experience without having to do the engineering that you might have had to do in the past.
This changes the world of prototyping, but it doesn’t necessarily make product development easier in the most important ways.
Imagine you ask a candidate to build an AI assistant to help managers prepare for difficult conversations with employees. In a short period of time, they could have a working interface that asks questions, makes recommendations, suggests language.
Now, the interesting thing is when you look at what it’s producing.
Is the assistant understanding the context well enough to be helpful? Is it able to distinguish between helping someone prepare for a conversation and potentially giving inappropriate advice around employment? What happens when you have incomplete information? What if the model confidently suggests something that shouldn’t be suggested?
I’d be interested to know if the candidate is seeing these problems, how they reason about the severity of the problems, what they would change?
As the cost of generating software decreases, I wonder if the ability to critically evaluate what is generated becomes an even more important differentiator. It’s easier to build something that seems to work. But knowing if it’s actually solving the right problem and is reliable enough to be something that a customer can rely on in their workflow is a different kind of judgment.
What is Good?
This is where I find AI product management interesting.
Product management has always been about understanding customer problems, prioritizing opportunities, and defining success. But with AI, there is another dimension to it. A product can seem successful based on traditional metrics, but it can be failing in ways that aren’t captured by those metrics.
Imagine an AI assistant that reviews contracts for small businesses. We might define success as being able to find important clauses and explain them clearly. But there are a lot of important questions left unanswered with that definition of success.
What if the assistant misses an auto-renewal clause? What if it incorrectly interprets a termination provision? What if it confidently explains something but that explanation isn’t supported by the document? Is that equally bad? Should the product react differently based on the potential consequences? When should it acknowledge uncertainty or recommend human review?
If you have something that is right 95% of the time, that can be incredibly useful in one situation, but completely unacceptable in another. The aggregate number doesn’t tell you much unless you understand what the other failures are.
So, I’ve been getting more interested in the relationship between product judgment and evaluations.
In recent engineering guidance on AI agents, they make an important point: when you design evaluations, you’re forced to be explicit about what successful behavior looks like. It’s not something you can just leave up to the people building the underlying models. Because the definition of success depends on the customer, the workflow, and the consequences of getting something wrong.
And I wonder if for AI products, the traditional product requirements document is going to be more and more complemented by something that is more like an executable definition of acceptable behavior.
Imagine a customer service agent that can modify a reservation. It’s pretty straightforward to say customers should be able to change their reservations. But what happens if they have multiple reservations? What if they want to make a change that violates a policy? What if they change their mind in the middle of a transaction? What if one thing succeeds and the next thing fails?
These aren’t just technical edge cases. These are part of the product experience. They are a reflection of how much thought you’ve put into defining what the system should and shouldn’t do.
And I think an interview where you explore how someone reasons about these situations may be more revealing than one that is more about familiarity with AI terminology.
Technical judgment without PMs turning into engineers
This isn’t to say that technical understanding is less important. If anything, the AI product manager is going to need to understand the trade-offs between things like quality of the model, latency, cost, reliability, retrieval, different approaches to system design.
The difference, in my view, is between understanding a technology and having judgment on how to use that technology.
Imagine you have two models that can do the same task. One is more accurate but is substantially slower and more expensive. The other is faster, cheaper, but makes more mistakes. Which one do you want? Well, it depends on the customer experience, the consequences of failure, the economics of the product.
In fact, you might not even have a single right model. It might be different for different requests. And the answer might change as models get better or costs go down.
I’d want to know how someone approaches that decision. What assumptions are they challenging? What would make them reconsider?
I also think it’s interesting to explore if you even need AI in the first place. Some workflows might benefit from a combination of deterministic software and AI rather than an entirely agent-driven experience. That’s as much of a product decision as which model you choose.
Obviously, the technical depth will vary depending on the role. But I think the ability to tie technical decisions to customer outcomes is going to be important in almost any AI product work.
Understanding Failure
Another area that I find more and more important is how product managers think about failure. Especially as AI moves from generating information to taking actions.
Imagine you have an agent that can make purchases for a customer. The experience can seem pretty simple when everything works as expected. But the product decisions get a lot more interesting when the agent misunderstands intent, has stale information, only completes part of a transaction.
Questions about authorization, reversibility, confirmation, recovery become inseparable from the customer experience.
And the same is true for enterprise agents that might be working across multiple systems. If you take the wrong action, there could be consequences that are hard to undo.
I’m interested in whether or not candidates naturally see these dimensions of the problem. Or are they seeing reliability and safety as something that someone else is going to worry about later?
I’d also be curious to see how their thinking changes as new information comes along. What if you have an evaluation that gets better, but customer satisfaction gets worse? What if you have a cheaper model that becomes available, but it behaves differently in important situations? How does that change their recommendation?
With AI capabilities changing so quickly, the ability to re-visit assumptions might be just as important as the initial point of view.
Are we testing the right things?
I don’t think there is a single ideal interview format for AI product managers. I think the answer is going to be very different depending on whether you’re building foundational model capabilities, enterprise agents, or AI-enabled consumer products.
I also don’t think we need to reinvent the whole product management interview. Customers, meaningful problems, trade-offs, judgment, those have always been important parts of the role.
But maybe what is changing is the directness with which we can observe those capabilities.
With a prototype, you can see the gap between an idea and an experience. With an evaluation exercise, you can see how someone defines success. With a discussion on model trade-offs, you can see if technical knowledge is translating into product decisions. With exploring failure scenarios, you can see how thoughtfully someone understands the consequences of what they’re building.
These seem like interesting directions, though I’m still figuring out how they should come together.
Maybe the more interesting question isn’t how much AI knowledge a product manager should have. It’s more about how do we identify someone that can take increasingly powerful, imperfect technology and turn it into products that customers can truly rely on.
With AI making it easier to build, maybe it also makes the quality of our decisions about what we build and what we consider good enough much more visible.
I’m curious how others are thinking about this. Are AI product interviews fundamentally changing? Or are we just finding new ways to assess the things that have always distinguished great product managers?