Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

Hello everyone.

So, my name is Ayush Bhardwaj, and I did applied AI for a hedge fund, and now I do everything tech plus applied AI for a pharma tech startup, because you know the way startups are, you have to do everything, we have multiple hats.

So, before I start the session, I would like to do a small survey. Can I get a raise of hands for all the engineers in the room? Okay, that's a tough room. Now, can I get a raise of hands for managers?

Okay, just to be clear, managing an agent does not count. You have to manage people. Okay, we have few managers as well. Interesting. So this will help me, like, fine-tune my talk a bit. So today my aim is to take you through the journey of how do you actually build and iterate in applied vertical AI, and my experience is from the hedge fund and the pharma tech company.

So before delving deep into the recipe, I'll just, like, take you through what do I even mean by applied vertical AI, because I don't know, it sounds like a very weird term. It's like the vertical word is kind of forced.

I won't lie, it is. I coined this term probably. So, applied AI is, like, built for, so what applied vertical AI is essentially applied AI but built for one very specific industry. Its aim is to simulate a job of a person in that particular industry, in a sense.

So an example of applied AI is Google Translate, which is like general purpose, helps you translate. It could be used in education tech, and it can have, like, tons and various sorts of usages. Whereas Allos, which is my employer, the pharma tech company, we specifically build drugs with AI, so that's a very specific use case.

Another example of applied vertical AI field could be the legal tech firms that are now coming up with. You must, I'm sure you must have heard about them. So those are, like, another good examples of applied vertical AI.

So,

when I left the hedge fund,right, so I was expecting that the world would change for me, because, you know, hedge funds are, like, really fast and really pressure sensitive, whereas pharma is like, okay, we're going to take 15 years, but we're going to do itright.

Hedge fund was all about, like, you need to do it fast and mostlyright. It does not matter if we lose at one paradigm as long as we are overall winning. Whereas a pharma firm is like, we have to be absolutelyright, you can take a week more.

And it was true, it's a completely different world. But to your surprise and to mine as well, nothing changed, actually. My job increased, but the core part of my job, applied AI, remained the exact same, and I cannot express how surprised I was, because I thought that it'll be a complete different thing, but apparently it was not.

So, then I spoke to other people as well across legal AI and the people coming up with the prop tech firms, which is essentially the real estate tech firms, and I realized that everyone is kind of building the applied vertical AI in a very similar way.

I could see some steps that could be essentially abstracted out, and that's what we'll do today. So, before again delving deep into that, I received a few reach-outs saying, are people actually putting agents into production? And I was like, this is such a wrong question to ask.

Everyone is putting agents into production, even like 15-year-old kids these days. The question to ask is whether they actually work, whether they actually make or save money, whether they justify their ROI, whether they are making way more than the amount we are investing into it, like end to end.

And I can say from my anecdotal experience, yes, at the both places I worked, the agent either saved the money or made more money. So with that, let's get started. So, the recipe, I'll take you through a series of seven steps, roughly, and try to, like, make this process as simple as possible.

So, the first step is formulate the problem. So, this sounds like very trivial, but a lot of people, specifically startups, get this wrong. We just try to do too much at once. Whereas from what I have learned and what I think a lot of colleagues would agree, you need to pick a very narrow task.

You just cannot ask it to do everything. A good example for this could be, let's say if you build something in finance, you won't ask it to, like, hey, can you fetch me top three market opportunities that I could invest in.

No, that won't work. You have to be, like, very specific. Like, you pick a market, you say, let's take the US equities, then you pick an industry, let's take IT. And then you ask it to, like, rank stocks based on some parameters, like capital expenditure, or let's say the AI investments.

So you pick, like, very specific things, and then you sort of formulate a very narrow job for the AI agent to do. And you can build, like, n number of AI agents. Last I checked, there was no text on building more AI agents.

So why do you want your single agent to do everything? So this is important, and this in the same, in the pharma context is the exact same. We just break down the process into steps and then ask really pointed questions with the agent.

We model our agent for a task. So once we have our problemright off the way, we know what we're trying to solve. The next step is identify the data. And I cannot stress this enough, this is a really, really, really important step, because everyone has news data.

Everyone has, like, seller-side reports from JP Morgan, Morgan Stanley. Everyone has the ARCZ preprint server or PubCam or your research papers,right? But what actually makes your application better than, let's say, ChatGPT or Claude, it is your proprietary data.

So the thing with proprietary data is, it's really expensive to buy, and most people won't sell it to you. So you need to curate it by yourself. Imagine your organization has been working for three years,right? They already have a lot of data.

It's just unstructured. And in the age of LLMs, I think this is a very fairly easy task to make unstructured data into structured data. Like an LLM workflow could do it overnight. So, to give you a great example of the proprietary data that finance industry has, it's the trade thesis, which is like what trade worked and why it worked.

And in pharma, it is the data for failed experiments. Because successful experiments data, yes, you can get it, but failed experiments, that's relatively hard to get. So now we have the problem, we have the data. What's the third step?

That is to model the problem, like, write the prompt. So while writing prompt, you, like, what we should aim is to model it after the person who you are trying to replace. I mean, that's the hypothesis, but yeah, no offense, we're not trying to replace anyone with AI.

But that's the ideology behind writing prompts. Encode how a person would solve this job into multiple steps. So it's just like a, like a mental model. So this is, again, fairly simple. Next thing, observability. I'm sure you have been to this conference at three years, and this word, I think, I don't know, you'll be hearing about, like, a thousandth time.

There are tons of observability providers. If you can see it, you can fix it. So you need observability to see the traces, understand what your AI application is doing, and debug it. So, sorry, but all of this was the easy part, to be honest.

All of this fits one screen. The mythical 10x engineers can do this stuff in minutes. Like, literally, this is the code you precisely need to build an AI agent. So that's why it's not the moat, of course, except your proprietary data.

So what do you do now? What do you do after doing the first four steps, which is observability and prompts and, like, getting the dataright and everything? You iterate. Now, the thing with iteration is, like, when I joined the hedge fund, I thought, how hard it can be.

I mean, everyone can iterate. I mean, we have been iterating our whole life for each of the tasks. But to be honest, I could build it, but I just could not tell if it worked, because I'm not a trader.

I'm not someone who has a PhD in biology or chemistry. I just don't understand what the model is saying, what is the output of my AI agent is. And since most of you are engineers, you would relate. You can instantly tell that Sonnet 5 sucks, because you have your own training.

You understand, okay, this code is not great code. Whereas some X model, let's say Fable 5, you see, okay, this is great, but not as great as the high pays. Because you have been trained for this for life.

You have a mental model to judge these things. But you just do not have the same kind of mental model when it comes to, like, predicting trade theses or doing, like, really specific tasks that vertical AI industry does.

And this is also the place where, like, a lot of vertical AI projects quietly die. Because on the surface, it looks like you have made it, you have built it, let's put this into production and start selling it.

But no one would buy it. The same way you won't use an inferior coding model. So, as an engineer, when I ran into this, I just couldn't accept, honestly. I thought, no, there's certainly more that I can do.

We don't need other people. So I thought I could, LLM as a judge, my way out of it.

And this was a really, really stupid mistake, to be honest, because what LLM is essentially doing, it's predicting the next probable word. So if you see, it's just, like, jargoning its way out. It does not understand what alpha means.

It does not understand how to actually create value unless you have, like, taught it somewhere. And whereas a human can just tell it instantly what's bullshit and what's not. So, I'll just try to delve a bit more, deeper on why you can just iterate.

So, first thing is that model cannot verify itself specifically in these fields, because reinforcement learning via verifiable rewards is really good at math and code. Because you have, like, answer keys, you can verify your code is compiling or not, and there are tons of stuff you can just model the complete thing around this.

But in these fields, there is just no way to model it. And let's say if any error gets in, it just compounds with every step. And that's what Lincoln seems to think as well. And now, the more important part that we touched upon previously, the data.

So the interesting thing with pharma and finance is, the data was never there. And I'll explain to you why.

So, any institutional manager holding over $100 million in qualifying US equities are forced to publicly file their holdings, long position holdings, every quarter. And once a hedge fund does this, this is the percentage decrease in their returns. Because everyone just sees those reverse engineers and takes away their moat.

And when it comes to pharma,right, so this is the number of, so by law, you are, like, required to disclose every clinical trial, pass or fail, you have done. But 30% of the firms, which is, like, nearly one-third of firms, never do.

And in, like, 2026, FDA had to, like, publicly remind over, I don't know, what, 2,000 sponsors that they are, I mean, doing injustice by not releasing unfavorable results. Because this is the exact data which helps the model think, which helps your LLM actually reason through these complex and niche industries.

And they hide it because for them, it's like a chicken laying golden eggs. Why would they sell their chicken? So naturally, neither OpenAI nor Anthropic has this data because it's, like, gatecapped. You just cannot hire a trader for $100 an hour and have them annotate that stuff because they're, like, lots of NDAs and they definitely earn more.

So, okay, now I have told you about tens of problems,right? Now, you naturally think, okay, yeah,right, then what do we do? How do we build a startup in, like, a vertical space? So,

very self-explanatory. You hire the person who you want to sell it to. Because there is, to be honest, no other way around. I have tried a lot of stuff. You just need to hire the user. In finance, in a hedge fund, this was very easy because the user was kind of like my boss, the trader.

We worked together. But in the pharma tech startup, it was very weird. We were, like, a bunch of young engineers and we were, like, oh, we need a 20-year-old scientist in our company to tell us what to do.

Yeah, I guess we do. And then we hired someone,right? And that someone actually changed the trajectory of our tools. Our tools started making sense when we pitched to the other pharma companies, the big ones, the big pharma, they started liking our tools because it kind of spoke their language versus the normal jargon-ish LLM language.

So, once you have hired the user, let's say, then what would you make that user do? You try to build a learning loop out of it. The domain expert can start at, like, a very, very low level, the ground level, where they just think about prompts.

Okay, I mean, let's not ask LLM to do this. Let's ask a very specific query again. They'll help you curate data. Just like engineers know which conferences are bullshit, which are not, which research paper sites are great, which are not, which are, like, top leaders in engineering, which are just, like, influencers.

Similarly, a pharma expert or, let's say, a trader knows which sources are more reliable than the other. So they help you curate their data. They help you, like, refine your prompts better. And they try to create, like, thinking models of how they would think about a problem.

Because, I mean, let's say if you follow five steps to solve a problem,right? You just cannot do it in any random order. There has to be a logical flow. There has to be a natural flow. So that's what they try to curate.

Like, decompose a problem, gradually refine, and then finally judge. So, the person who sort of has lived through the complete of the industry that they're trying to revolutionize, their judgment is now, like, turning into agents. So that's what's happening behind the loop.

So, to do this, there are, like, again, multiple ways. I mean, each of these could have been an hour-long session on its own, and I wish I could take. But these are, like, few ways that I identified. I'll just, like, take you through them, like, really quickly in the interest of time.

So, supervised fine-tuning, I think most of you would know, where, like, model mimics human demonstrations. Reinforcement learning from human feedback is, like, a kind of a very efficient way where human preferences train a reward model. Then rubrics as a reward is, I like to call it, reinforcement learning from AI feedback.

This is because that you can, human can just create a rubric, and then AI will just, like, grade itself based on that rubric and improve its own processes. But again, there is a slight chance that you might run into an echo chamber with rubrics as a rubric.

And the best of all, and I think ROI is there. Whereas the observability part that you set up earlier, you just analyze the logs plain and simple. You understand where the model is going wrong, and then you just try to correct it.

So, this is where you have, like, don't have to touch any weights and the most highest ROI way to get the impact from, like, start on. And once you understand, like, what more you could do or if error analysis is solving or not, you can just gradually climb up the ladder and probably later on go to the ultimate reinforcement learning from human feedback.

Because that's, I think, in our industry, kind of the golden standard these days that you need to do RLHF to actually get some edge. But certainly, there are some pitfalls of it. Like, for example, now there's GLM 5.2,right?

You fine-tuned it,right? Alibaba Cloud or, let's say, DeepSeek will release a newer model. Then you have to fine-tune that too as well. So there is a cost. It's not cheap. So once you have done all this, you just create a loop and you just, like, go on to that loop.

You hired one user, you hire more users, they ask more queries, the scope increases, the data increases. At this point, you are kind of generating your own data. The exercise you have been doing in loop,right? That exercise itself is generating a very, I would say, a crazy data set of what works and what does not work.

And this loop never stops. Once you feel confident enough in your application, you just ship it, provide it to the external paying users, and then you see the magic of it, that it actually works. So, I just pulled this step from Stanford AI Index report, because it's a really nice report that gives you an idea of what the state of AI is.

And this says, like, 80%, 89% of enterprise AI agents never reach production. Again, I disagree. Every AI reaches production, but it just fails to work or, like, justify its own cost. So that's the real thing. You can just build and ship AI agents whenever you want, but you need to justify ROI.

And finance and pharma are two such industries where if it does not make money, it's shown the door. Simple. They won't, like, wait and say, okay, maybe it'll work in two years. Maybe the cost will be lowered by the third year.

No. It has to instantly make money. It has to, like, hit the ground running. And if it does not, shown the door instantly. So just to summarize, the seven steps that I feel are, like, good enough to give you an abstraction of how the vertical AI industry moves.

You formulate the problem statement, you source your data sources, you prompt it well, you refine those prompts, you observe how your tool is performing, you don't iterate yet, you hire the user. And this user or users now play with the tool as much as possible.

They're, like, kind of form a learning loop, an endless learning loop that goes on. And at a point when you feel, yeah, it's really delivering that alpha over, let's say, Claude and ChatGPT, you just ship it, you start earning money.

So, one more interesting thing. So, HITL is, like, kind of a thing. Everyone is like, yeah, let's add human in the loop. I would say not yet. Finance and pharma are still those two industries where it's AITL, AI in the loop.

Because everything is, like, done by the expert, but the AI assistant really helps save time. Like, for example, it may take an X amount for a trader to form different trade theses, and AI can just give him five candidate trade theses.

But which one would actually work in the market and which won't, the discussion still lies with the trader. And same for pharma. When you are, like, picking drug candidates, which one to pick, the expert still does it, but you just, like, reduce the time of expert by a lot, lot.

So, and it will stay this way for really long. So, for the models to actually make good decisions, they don't need to do correlation. They need to do causation. And as Yann Lacan puts it, these are, like, tech statistics, not real-world models.

You cannot just pattern match with past and use future to predict to it. And so, we are, like, kind of not there yet. That's what I call as the AGI line. Once we are there, yeah, probably then models will just, like, make drugs.

You will have vibe-coded drugs. Someone would be vibe-coding market. But yeah, not yet. So, a final takeaway that I would call, if there's one thing you are taking away from this talk, this is it. Model, infra, ecosystem, everyone selling you tons of stuff at this conference is just commodity.

Everyone has it. If you have it, everyone has it. Everyone can pay X number of dollars for a subscription. But what is moat and no one will come and sell it to you, you won't have to curate it on your own, is the domain expertise.

You need your data, you need other people's data that is just not out there on the internet, and that's what will form your moat. So, thank you for your time. I think you enjoyed the talk. And yeah, let me know if you have any questions.

We can meet outside. Thank you.