Source: Hard AI

The following section is an interview with Microsoft CEO Satya Nadella, abridged

Topics: Microsoft's Adjustments | Copilot+ PC | Artificial Intelligence Platform | OpenAI Partnerships | Integration and Modularization of Artificial Intelligence | Capital Expenses and the Future

Microsoft's AI transformation

Ben Thompson: I used to work at Microsoft. In fact, I attended the first Build Conference here in 2012, and it was a very unique event. I'm not sure if you remember when it competed with another consumer-centric event hosted by the Windows team in New York. Today, Microsoft seems to be a more united company. What cultural and organizational changes do you think have taken place during this time?

Satya Nadella: OK, let me just say that, Ben. For me, my memories go back to PDC (Professional Developers Conference) in '91, when we first discussed Win32. In fact, I was still working at Sun at the time, and I hadn't even joined Microsoft; I was very clear about the future; the PC and what it wanted to do, and the server architecture were already very clear to me in '91.

I think this is super important for any tech company, which is to have what I call a complete mindset. For me, complete thinking starts with “What is systems innovation?” ——Whether it's a chip, operating system, or application platform. So why is this desirable for any consumer or any app developer? So for me, whether on Azure or Windows, or even on this morning's Windows Copilot+ PC, we need this kind of innovation. We haven't had such a complete idea for a long time; we've had it — Arm, and I think we've got it. We already have an app platform. Microsoft is developing its own apps, and third parties are also developing apps. So for me, culturally speaking, something that allows you to build a complete product is something I think I must work towards.

Ben Thompson: You mentioned consumers and developers. You didn't mention the word enterprise. Do you think complete thinking is currently working on these three levels?

Satya Nadella: That's a great question. In my opinion, businesses are end users too. In fact, I think at Microsoft, as long as we're at our best, we can — if you remember, we've always been a company of knowledge workers.

Ben Thompson: Exactly. But I think the split between a company's buyers and users is sometimes a tricky problem

Satya Nadella: Honestly, the company I joined in the early 90s wasn't like that.

Ben Thompson: Because back then there were only developers.

Satya Nadella: There are also end users. So, one thing I always think about is that we're really thinking about the end user. For me, Excel is a product for end users.

Ben Thompson: Definitely in the 80s.

Satya Nadella: Long before it became IT. So looking back, that doesn't mean I don't want to do something that actually meets IT needs right now. In fact, Paul Maritz once said: The magic is in end users, developers, and IT. Incidentally, to be a good corporate company, you must be good at coordinating the relationships between the three. That means if you equate developers with end users, you can be a great consumer company, and they're all different categories, right? The consumer is now a very broad concept. It could mean many, many things. We're not going to do a good Hollywood movie or many other things, but when it comes to games, I call it productivity, we want to do great things.

Ben Thompson: Yes, people have forgotten that Microsoft is disruptive not only in the consumer field and PC field, but also in the server field and product hardware. Is there a shift between which comes first and which comes later? I think there may be a time when Microsoft had high expectations for the consumer sector, but maybe people use PCs at work and then they want to use PCs at home, and phones are clearly a bit different in this regard, but is that still Microsoft's sweet spot?

Satya Nadella: Yes, I think that's it. Let's take Windows, and I want us to build great Windows PCs for both work and home users, even in terms of form factor. If the wave of artificial intelligence is already here, let's redesign the operating system and hardware. So that's what I hope we can actually do well, and today is a good step in that direction.

Copilot+PC

Ben Thompson: Yes, I'm really impressed; I think it's very persuasive; it really makes sense to understand why you should buy Windows, and that feeling didn't seem to exist since the browser came along and the app was removed from the desktop. How did you take advantage of this and move closer to it? Will there be a major shift in your market positioning? Do you just rely on the original equipment manufacturer; do you think it will sell it itself? Are you investing a lot in ways you didn't before, or are you investing all the time?

Satya Nadella: We're always investing, but in tech, timing is everything, right? We've been working on Arm and we've been talking about NPU for a long time.

Ben Thompson: Yes, 10 years ago, Microsoft introduced the Arm PC, and I was there.

Satya Nadella: So, the key point is that it's being integrated. Think about what just happened. With all of these models, and the ability we can have, whether from a privacy perspective, a latency perspective, or a COGS (cost) perspective, to have built-in models because when you use...

Ben Thompson: Honestly, you're speaking on stage a bit like Apple. First, you mentioned “MacBook” more times than the Apple of the Jony Ive (Jony Ive) era, which is clearly a comparison. There's a lot of talk about local privacy, but the local models you mentioned are critical — COGS, from your point of view, if you use the customer's energy, it's actually free. How far do you think you can go in this regard? For example, AI PCs, I've been waiting for specifications. You did mention that 16GB of RAM is not bad for Windows, but it's still too small for AI.

Satya Nadella: Now with 45 TOPS, I think this Copilot PC has taken its first steps, I love our first steps on the AI PC, but with this Copilot+ PC, I think we've succeeded. BTW, I'm pretty sure distributed computing will still be distributed, so it's actually consistent. Let's take Recall; I think this is a killer feature, and we've been working on it for a long time.

Ben Thompson: I've always used the “rewind” feature on my Mac, which is definitely a superpower.

Satya Nadella: Yes, the point is that now that I have a semantic index (Semantic Index), I can type in natural language queries and then recall, and even facts we've forgotten. I memorize things through sight, and I remember things through associations, and now I can recall without learning to search, just by typing in my intentions. However, the interesting thing about Recall — if you notice it, not only does it return to the previous page, but because of the semantic index, I can call up the content of the moment, so this ability requires a lot of edge computing.

Another interesting point is that one of my favorite demos is that you can use the NPU (45 TOPS in all) while playing Call of Duty without damaging the battery. This ability lets the operating system know how to properly use all chips and systems, and I think this will be a real breakthrough for us.

Ben Thompson: It's easy to demonstrate this; the drawing and drawing enhancements you've demonstrated on your Surface PC are really cool. An eternal question is that Android has the same issues as before Windows — yes, you've defined the specs for these AI PCs, but how do you deliver a consistent experience?

Satya Nadella: That's a great question, Ben. This has always been one of the challenges of our ecosystem. Frankly, I think we're all learning how to actually do a good operating system — frankly, it's a chip issue.

For example, when we work with Qualcomm, it takes a lot of work to ensure that their chips are in the best condition. Right now, Intel and AMD are doing similar things, which is great — if I use the cloud as an analogy, I'm happy to see some of the top people who know a lot about chips put their energy into building great products.

Ben Thompson: When you're in the position of a challenger rather than a leader, do you find it easier to push all parties in the right direction?

Satya Nadella: I think so. These are all the results of competition. In competition, we are stronger and more disciplined, and when you have something to win, we enforce it more strictly, so this is great, so we have the best chip innovations. Not sure if you've noticed OEM innovations...

Ben Thompson: I'll check it out later.

Satya Nadella: You should check it out. Dell, HP, Samsung, Acer, and Lenovo are all involved. Incidentally, Surface set the tone, but that's not to say that no one followed after Surface set the tone. Frankly speaking, we can integrate everything together. This is actually a proof of the ecosystem and a proof of leadership. Other hardware vendors will think, let's try it. This is it. Once every ten years, once a generation.

But I think even if it goes back 30 years, it's in this land that we launched Windows '95. You could say, “Oh, that was the heyday of Windows” — but did you know, even in the heyday of Windows, we forgot one thing: the internet. That December, we launched the SR-1, or browser. However, in the AI era, I feel that structurally we are more perfect, we have more opportunities to win, and the entire ecosystem is innovating with us.

Artificial intelligence platform

Ben Thompson: There's a problem. You've always called artificial intelligence a platform opportunity. I have a question, and I was even preparing for your introduction, that is, to what extent can platform opportunities be hardware independent, without a paradigm shift, whether entering the market or fully revealing these features?

In this regard, this talk was interesting because it was very specific, “Look, this makes Windows better, and you can get these features by buying a device.” I once wrote an article saying that I think one of your great wins is essentially removing Windows from the center of Microsoft. Sure, it's important to you, but it won't be the center of everything. Are you capable of achieving comprehensive development in an unprecedented way? How important is Windows to drive your future? Is it critical to realizing this platform opportunity, or can you still get this opportunity on iOS or Android?

Satya Nadella: First, my approach is based on the current state of the world, not out of imagination. Second, I want us to simultaneously integrate every layer of our stuff into a cohesive architecture that serves the interests of developers and end users.

Ben Thompson: When I left Microsoft, I wanted a memo on Windows, and the timing was right, so I wrote this post. I thought this was crazy at the time, but now it seems even more meaningful.

Satya Nadella: Because to some extent, I honestly think we have to really make sure we do the best work for the 200 million devices that have been sold. But that doesn't mean the other 1 billion devices sold aren't important. For another billion devices, we need great innovation, and I'll talk about that later, but first let's say to 200 million Windows users, what amazing things can we do with this platform transformation? That's from the chip to the experience to the third-party developer — by the way, this isn't isolated; Windows itself isn't independent.

I don't know if you've noticed, but one thing is super critical today: adopting artificial intelligence at the edge. There are two challenges — how do you ensure privacy and how do you ensure security? If your classifier doesn't continuously learn from all adversarial attacks that have occurred in the past hour, it won't be able to provide security on the cutting edge or latest model, and this will be done in the cloud. So I want to be able to call cloud services. It's kind of like Windows Defender, how can you have Windows Defender if you're not connected to the cloud? The same goes for artificial intelligence security. So, you want the cloud to do what it's good at, and you want the client to do what it's good at, and I think that's the key.

Another interesting thing is that I'm really excited about Copilot Runtime. For me, I wanted a real namespace - by the way, the WebNN thing is so cool that I can write some JavaScript and use WebNN to get a model and then run the NPU locally. I can go to Gap.com or any website, and now I can start adding AI features and uninstalling AI locally. I think the combination of cloud computing, networking, and edge technology is a cohesive idea.

In fact, it gave us a head start when building systems for Android. In fact, at tomorrow's Build conference, you'll hear us talk about one thing that I'm excited about, and that's using Phi, right? Now, as a developer, you can use PHI as a hosting model service on the Azure AI cloud. You can use existing chip technology, basically Project Silica, which already exists on Windows, or you can package it into your app and then use it on Android and iOS. This is the direction we intend to move forward.

Ben Thompson: We weren't going to talk too much about Windows. I have some bigger ideas, but this demo was really compelling enough to rethink this field from a fresh perspective.

Satya Nadella: We'll get you back on Windows, Ben.

Ben Thompson: I don't have any barriers to Windows, I just hate changes!

OpenAI partnerships

Ben Thompson: Microsoft seems to be a more united company, and I've mentioned this before. When you visit customers, how important is it to know that they have the support of a united company behind them?

Satya Nadella: You mean within Microsoft?

Ben Thompson: No, target external customers, such as big companies, big companies.

Satya Nadella: I think what our customers expect of us is, first of all, one thing I'm very, very concerned about. Because we're a company, all of these parts are together, and integration is important, but each layer must also exist independently.

So for me, my opinion of Microsoft is, yeah, we're not like a corporate group in the end, we have to have a real argument, which is the cohesion of the architecture. Our customers care about us and the integrated value we bring, but they also care about whether everything is competitive. So yes, customers care about that, and we have to stick to it internally. In fact, our best state isn't just consolidation; it must be integration plus the competitiveness of every layer of the stack.

Ben Thompson: So when you talk about “one Microsoft” integration, how do you solve the problem of partnering with OpenAI? Have concerns about this increased? For example, “Microsoft is great, you're all moving in the right direction, but there seems to be a dependency here, and we're not sure if you can control it, which means we can't control it either”, how are these conversations going?

Satya Nadella: For us, I think the partnership with OpenAI was like the partnership with Intel back then, or the partnership with SAP when we built SQL, because it defined both the industry and Microsoft, so we're very involved in this partnership. This is simple logic.

Ben Thompson: Who has computing power can rule the world?

Satya Nadella: Exactly. Back in 2019, we made an unconventional bet. At the time, we thought we should probably invest a lot of computing power because OpenAI was more confident than any other company (even Microsoft insiders), and that bet has been working for the past five years, and I've been focusing on securing a partnership for the next five years, as you know — in fact, this is a critical period of success for both parties. At least that's what I did.

I think for them, we provide infrastructure, they are model builders. They build apps, we build apps, third parties build apps, and this cycle goes back and forth. There will be competition, and there will also be some competition that is completely vertically integrated. The vertical integration works very well. You have to keep an open mind, and sometimes collaboration is the only way out.

Artificial intelligence integration and modularity

Ben Thompson: You mentioned that OpenAI has beliefs in computing, and this is something Microsoft will definitely rely on. Considering Google's leading edge in models, especially infrastructure, is there or should there be an anti-Google alliance in the field of artificial intelligence? Are we seeing this kind of alliance emerge, not just Microsoft and OpenAI, but possibly Apple?

Satya Nadella: I looked at it and said, listen, I think there are always people who can vertically integrate. I always look back. There's the Gates/Grove model, the Apple model, and maybe the new Google model. This is the vertical integration model. I think both models have merit.

In the long run, I believe more in horizontal specialization. In the case of chips, Nvidia CEO Jensen Huang (Jensen Huang) is sitting there and is very active in implementing the incredible roadmap. Today, guess what? He needed to ensure that leading artificial intelligence models were trained on Nvidia chips.

Guess what? Google was not trained on Nvidia chips. Google sells Nvidia chip-driven computing power, but Google's model was trained on TPU. I think Hwang In-hoon will notice this too. AMD CEO Su Zifeng is innovating. We are making our own chips.

Therefore, everyone will want to innovate chips and innovate models. OpenAI, Meta, Llama, Mistral, all of these small language models are all doing a great deal of work.

In any case, any of our applications, such as Copilot, will definitely use GPT-4O and mix it with Phi and others. So I think any business is most interested in model-as-a-service (MAAS, model-as-a-service). So I think it's going to be a more diverse market; at least my historical lesson is that winner-take-all situations are very rare, and we need to be very aware of this and make sure you're prepared in this situation. But in other cases, we need to adopt a broad platform strategy.

Ben Thompson: Of course it makes sense; you're talking about the idea of commercializing models. Microsoft has hired a lot of talent from Inflection AI, and it seems like you want to make sure the models you provide are diverse. But if the model were to be commercialized, why would the dynamics of cloud computing be any different from the past 12 or 15 years? Will this change anything new?

Satya Nadella: I think that's a very good point. I think hyperscale enterprises have a fundamental structural advantage in this regard. In a sense, if you were to say what the world needs more in five years, I'd say hyperscale computing utilities are everywhere. If you think about it carefully, I think the new formula for economic growth is as clear as ever: you need more renewable energy, better grids, and better computing power. If you have these, then every other area of the economy can really benefit from these two things. Any country, any community, if it can reach the best cutting edge in terms of efficiency, can take the lead in economic growth and gain momentum. So if you consider this high level premise, then this is definitely the case.

Ben Thompson: But what about the competitive landscape? Because Amazon was the first to enter the cloud computing field, they basically included customers of all SaaS companies. Microsoft is moving to the cloud with its enterprise customer base. Google, on the other hand, is saying, “Come on, our cloud is the best, try our service,” but they are far in 3rd place. Will the competitive landscape evolve in a similar way? Will data gravity (data gravity) dominate? Maybe AI is a new thing, but is the actual competitive landscape still...

Satya Nadella: I don't think I've met at least one enterprise customer using a single cloud service. I remember when I first got involved with cloud computing, everyone talked about it, as if it were a winner-take-all market, and I always thought we were just leading in the server field, and when people even said Microsoft won, I didn't really understand it. Every type of server, whether it's an operating system, database, web server, or all that middle tier stuff, has two or three players.

So, fundamentally speaking, I think this market definitely has room for two or even three companies, and revenue share, which former Microsoft CEO Steve Ballmer often told me — revenue share and market share are two different things in a multi-player market.

However, I still think that all three companies have room for development. Amazon AWS experienced a period of six or seven years without competitors. Now, competitors have appeared, and we have reached this stage. I'm very, very optimistic about the next phase. We didn't start from scratch. In fact, if anything, we had a starting point, and that changed everything. Take B2C customers, whether it's Shopify, Spotify, or any other company. Thanks to the OpenAI API, none of these customers were users of Azure before. Now they're not only Azure users, but loyal Azure users, which is a huge turning point for us.

Capital expenditure and future

Ben Thompson: You've talked about your visibility into revenue and expenses, so do you have any advantage in this competitive landscape? There's no doubt that you have to invest in artificial intelligence, but over the past 7 years, your capital expenditure ratio has risen from 13% to gross profit, which is a huge increase — what makes you believe this will pay off, or is it unimportant because competition dictates that you invest no matter what?

Satya Nadella: I think you're pointing out the laws of the economy correctly; we're an entity with a high capital expenditure. Most people are concerned about our capital expenses simply because of artificial intelligence. But please, even without artificial intelligence, we are a knowledge-intensive and capital-intensive enterprise, which is what hyperscale enterprises need. If you can't invest $50-60 billion in capital expenses each year, you can't get tickets to the hyperscale market, which is a necessary condition for entering the market.

But at the same time, this will always be subject to market conditions. Your expenses should not far exceed the growth of your income. Therefore, there is an absolute control factor, that is, when the step function assigned to the training calculation changes, the training block changes accordingly, but the inference is ultimately driven by demand. Therefore, if the two are combined, I don't think it's very difficult to adjust even if there are cyclical changes. This is a pure business management issue, I'm not managing it for a quarter,

Ben Thompson: You're not as worried as Wall Street. Just a quick question because I love this one. Bill Gates said that we overestimated what happened in two years and underestimated what happened in ten years. Because it feels like a lot has happened in two years, are these units still correct?

Satya Nadella: I think these are probably all the correct units. Take Moore's Law as an example. There's a beautiful chart on the Epoch AI website. I really like it. They talked about machine learning, and since 1950, the number of runs of machine learning algorithms has followed Moore's Law. It doubled every 15 or 16 months, then tripled by 2010, and I think its slope is actually steeper, probably doubling every six months, or even less, which makes it hard to keep your head clear. Everyone says, “Oh, I am until what is exponential growth”, believe me, live in that world --

Ben Thompson: When that exponential growth started, things became completely different.

Satya Nadella: Yes, it's hard. That's why I think AI security is a super important thing, and we have to keep that in mind, but we also have to keep in mind that new innovations will emerge. So how do you take advantage of new innovations without compromising safety? It's a very different game.

----

The following section is an interview with Microsoft CTO Kevin Scott, abridged

Topics: Artificial Intelligence Platforms | Artificial Intelligence Expansion | Artificial Intelligence as a Tool | Partnering with OpenAI | Law of Scale | Capital Expenses

Artificial intelligence platform

Ben Thompson: Let's take a decade or so back in time and take me back to Microsoft's journey from artificial intelligence, high performance computing, or artificial intelligence computing to forming a partnership with OpenAI. Did anything make you realise that this is the path you need to follow?

Kevin Scott: Yes, of course. What's interesting is that I've only been working at Microsoft for a little over seven and a half years. Ten years ago, when I was in charge of the engineering and operations team on LinkedIn, we already saw very clearly that artificial intelligence was on a very interesting new curve. Artificial intelligence back then wasn't the generative AI it is today, but what people did with really complex statistical machine learning, and the benefits we had already reaped from the expansion of these systems 10 years ago, was unexpectedly fast.

I've been working on this for a relatively long time, so before and after Google went public, I built many machine learning systems at Google, including large-scale machine learning systems that ran ad auctions at the time. At the time, the importance of scale was already evident. However, a relatively recent update six years ago was that this increase in scale caused the AI model to start behaving like a platform.

As a result, we're beginning to see that the extended nature of these large language models allows these large language models to be reused for many, many different things, rather than building a model specifically for one thing, and then you apply a large number of extensions, and that specific thing, such as ad click-through rate (CTR) prediction, becomes very good.

Ben Thompson: Actually, this is a question I wanted to ask because you and Satya have been talking about the transformation of platforms, and the word “platform” keeps popping up.

Kevin Scott: Yes.

Ben Thompson: I wanted to ask you what this meant, but I heard from your answers that you meant the platform for promotion.

Kevin Scott: Exactly. It's a component that can be combined in a very flexible and versatile way with the software system you're building. Therefore, in the world of artificial intelligence, companies like Microsoft may have hundreds of different teams, from top to bottom, responsible for data, machine learning algorithms, systems for training machine learning, how to deploy systems, and how to get feedback from the deployment process and actual usage and incorporate it into models to continuously improve everything over time.

There are hundreds of small flywheels rotating throughout the process, not the training work of a central model. What you get from this is very useful in many of the applications you already have, and it also provides the possibility to build a lot of new things that were impossible before.

Ben Thompson: So I'd like to emphasize the concept of a platform again. I've heard it from you, maybe I haven't heard it from you, I had this in mind before I talked to you, but there are platforms, such as Windows, that's what we think about platforms. You have an application interface and a network effect. This is a two-sided network with developers on one side and users on the other, but also platforms like x86.

Kevin Scott: Yes.

Ben Thompson: I think you're using the term platform closer to x86 than Windows, is that right?

Kevin Scott: Yes.

Ben Thompson: Perhaps the more appropriate example is general-purpose processing, because when you talk about the transition from dedicated to general purpose, it sounds like a shift from a dedicated processor that only does one thing to a general-purpose logic chip that can be widely programmed.

Kevin Scott: Yes, I think x86 is probably a very appropriate comparison, because the interesting thing about x86 is that it is a general infrastructure that allows writing a large amount of software, and the functionality of the system and platform continues to increase over time because it becomes cheaper and more powerful every 18 months or so. Therefore, many people can develop based on it, so as to achieve rapid improvement in capabilities.

There are clear boundaries between x86 and operating systems, PC manufacturers, and people building applications based on them. Sometimes, Microsoft develops apps and operating systems at the same time, so there's a bit of both, but the possibilities for people to do things on the Wintel platform are limitless. This isn't about all the useful stuff Microsoft predicts. People can believe this is a fun platform because you have an index called Moore's Law, which will eventually cause this thing to be completely everywhere.

Ben Thompson: We're going to be discussing Moore's Law, and I know this is something you and Satya often discuss, and it's something you want to discuss, but you mentioned Wintel. The result of x86 is that later, you have Windows, you have Linux, and eventually even Mac or something, so you do have tiers, but overall, from a developer's point of view, they're concerned about the OS layer.

For AI models, my question is, where do real opportunities arise? Let's go backwards. I think one interesting thing about Nvidia now is that, obviously, we have a lot of reasons to be optimistic about Nvidia for secular reasons, but I think one structural reason is worth paying attention to, and that is CUDA. This is where we all specialize. We frame this, we frame that. LLM generalized it, and now, something higher level is actually happening where you don't need to know CUDA to build artificial intelligence applications. But is this an actual layer, or will there be an operating system on top of it?

Kevin Scott: Hard to say. Probably not in the sense of an operating system --

Ben Thompson: It's not a traditional operating system, but it also has this background.

Kevin Scott: Yes. I think this is the history of the development of computers; people are always improving their level of abstraction.

Ben Thompson: We're rearranging it because of the new model.

Kevin Scott: Yes, 100%, I think that's completely true. So, I don't know what the level of abstraction would be, but it's very different now. We now have timely engineers who can trick these systems into doing very complicated things by simply issuing instructions in natural language, such as what you want the system to do or not do. We're developing a variety of tools to find out that you have to cram a bunch of things into the context window of a large language model to make it do what you want it to do. The GraphRag system we built at Microsoft Research is one such tool. It can perform graph-structured contextual synthesis, so you can use context very efficiently without sending unnecessary tags to the model. This is a good thing, because the more tags you send, the higher the cost and latency, and you only need to get the information it needs to answer the questions you need to answer or complete the tasks you need to complete.

So, I don't know what the abstraction is all about, so why are we talking about the Copilot stack, and we haven't even figured out what everything in the Copilot stack actually is. Over the past few years, we've figured out many of the elements required to deploy modern apps, but even as the models get more powerful, the level of abstraction will increase. Going back to your Windows analogy, the first version of Windows didn't have DirectX, because graphics processing wasn't powerful enough at the time to consider using shaders at all.

Ben Thompson: It's not that nobody created it, it's that nobody thought of it yet.

Kevin Scott is right, so there's still a lot of work to be done. But I think the least you'll see from us is what components you'll need to have in addition to a powerful cutting-edge model to build these really rich and exciting new apps.

Ben Thompson: Is it a Copilot stack? In the long run, in my opinion, if you were a developer, if you were a developer considering building a computer application in 1975 and 1985 and 1995, the scope of your decisions was completely different. In some ways, what you're going to do is obvious, so what's your opinion on how this evolution will happen?

Kevin Scott: Hmm, so I think our point now is that when all these platforms come out, these abstractions will be layered. So at the bottom of the abstraction stack, you have a large underlying model, and then you have data sets and retrieval mechanisms to make sure the model can access the information it needs to do its job. On top of that, you'll also have a bunch of things being coordinated, so you might need to operate across multiple models to complete the tasks you want to accomplish, and you might need to do this for cost, quality, or data privacy reasons.

One of the things we really want to see in the year ahead is a breakdown of where the reasoning takes place. If you can do this, I think you'll want as much inference as possible, and as much artificial intelligence as possible on the device, and only when the device runs out of power or capacity, you'll need to call on something more powerful and complex in the cloud.

Kevin Scott: I think the consensus we have now reached is that as you, various platforms will experience a series of layers of abstraction as they emerge. At the bottom of the abstraction stack, there is a large underlying model, followed by data sets and retrieval mechanisms to ensure that the model can access the information it needs to complete the task. Based on this, we have a range of tools to orchestrate, and may require operation across multiple models to meet cost, quality, or data privacy requirements.

Over the next year, we're looking forward to seeing more diverse places where reasoning takes place. Some inference can be done on a device, such as a PC or mobile phone, and if possible, we should reason on the device as much as possible. Only when the device's capabilities or capacity are insufficient will it be necessary to call on more powerful and complex resources in the cloud.

Ben Thompson: In this process, the most important thing is the orchestration agent, which first decides whether to execute locally or in the cloud. If we choose cloud execution, we also need to consider how to optimize the prompts or requests to reduce the number of tags and increase efficiency. Like the Phi model on Windows you mentioned before, the most important part is not only collaborative drawing on Windows, but also reducing costs for Copilot through the cloud.

Kevin Scott: I think this is an important question, and there are more and more levels of abstraction. The point is, if you're building a product that's really useful and has a very wide range of target users, then you need to distribute it to as many people as possible. So cost (COGS) is definitely a factor to consider. So, it would be great if there were some ways for you to offer them in the form of a high-quality product, such as providing that app by uninstalling it from the cloud to a small model. It should definitely be done.

Ben Thompson: I remember you mentioned before that we don't have to worry too much about cost (COGS) because everything is going to be really cheap.

Kevin Scott: Yes.

Artificial intelligence expansion

Kevin Scott: I think the interesting thing about the whole field right now is that the capabilities of cutting-edge models are really growing exponentially, and I don't think we're close to declining marginal returns.

Ben Thompson: If we were to get in the way of scaling up, what would it be? Is it data or what?

Kevin Scott: I think the data is already difficult. I think everyone will run into this kind of problem in terms of the scale of some cutting-edge models, and it's a challenge to have enough data to support it. I mean, one of the major innovations in the Phi model is that you're actually using a cutting-edge model to generate...

Ben Thompson: Synthetic data.

Kevin Scott: We've been doing this for years, and we've done this for non-generative models. Therefore, if you want to build a computer vision classifier model and want to ensure that it does not reflect deviations from the underlying training data set after training, you can use the generated model to generate a large set of synthetic data to obtain a fair distribution of training data, so as to obtain the desired model performance. So I think generating synthetic data is an increasingly powerful way for people to build small and large models. For reinforcement learning in particular, I think this is really valuable.

Ben Thompson: What's driving this? You mentioned the basic model, and you compared Sam Altman (Sam Altman) on stage, just like we started with smaller animals, what is a smaller animal?

Kevin Scott: Shark. First a shark, then a killer whale, and now the blue whale is an unnamed training model that will apparently be released at some point. So whether the answer is, a lot of efficiency and a lot of scalability are in the smaller models, because these big models can generate all the synthetic data and provide everything — you can optimize, but that doesn't answer the basic model's question. How do you think they can scale?

Kevin Scott: It's a bit difficult to fully answer this question without revealing a bunch of things I don't want to reveal.

Ben Thompson: I think that's enough.

Kevin Scott: But I think synthetic data is also useful for training large basic models, so imagine if you want to train a basic model and make it very, very, very good at coding, there are lots of ways to generate synthetic programs with specific characteristics, because programs are these deterministic entities, you can synthesize something and then run it through something like a model checker to prove “is it compilable?” , “Did it produce a set of outputs?” , “Is it a valid input and can you put it in the training process?” By generating synthetic data in these fields, you can design a training course, or at least part of a training course, for any model. In these fields, generating well-formed training inputs is very direct, allowing the model to perform better in the specific course you are trying to train. But that doesn't mean all of the data can be synthetic.

Ben Thompson: You gave an example of a model for click-through rate prediction and ad targeting; you were talking about coding. The benefit of all of this is that even though you're making a probabilistic model, the data you're using is definitive, right?

Kevin Scott: Exactly.

Ben Thompson: So when you reach this kind of versatile functionality, what makes you confident that this versatility can be extended to fields, where it's almost like an extreme, where you have pure creativity, where there are no wrong answers, where you work very well, and at the other extreme, you work in a field with verification capabilities, so you can actually make artificial intelligence work in parallel and get the best answers because you can rate it. But there's also a middle ground, and I think people — I call it a “lazy feature” — want artificial intelligence to do the work for them, but the real problem is that they don't necessarily have a grader. So, can artificial intelligence be used universally in this field?

Kevin Scott: Yes, I think we'll be able to generalize a lot of things. One of the things we mentioned in the Phi paper was probably “Textbooks are all you need”, which also made me very confident. We have the ability to train human experts in fairly limited courses to enable them to do very specialized things.

Just like the way I was trained as a computer scientist, I read lots of computer science papers and textbooks, and did lots of problem sets to practice, practice, and practice again and again. After a while, I'm competent enough to do something useful in the world. So, it gives me confidence that we'll be able to figure out how to generate enough courses for these models and find a learning function that allows us to build something with pretty strong cognitive abilities.

Now, what I don't know — this is going to be a matter of concern we'll figure out soon — I'm betting with some people that I imagine a computer would prove the “Riemann conjecture” before mathematicians. The Riemann conjecture is one of the century-old problems in mathematics. It was proposed by David Hilbert (David Hilbert) in the late 19th or early 20th century, and everyone has been trying to solve this problem. The Riemann conjecture is basically a statement about the distribution of prime numbers; this is a very difficult question. This is an easy question to state, but there have been some extremely smart people who have been trying to solve it for a long time. So I actually believe this is a potentially very complicated question, and the proof will be incredible. My prediction is that a computer will solve this problem before humans, and that it may involve human assistance. It won't be completely autonomous.

Artificial intelligence as a tool

Ben Thompson: You said in today's keynote that you've loved tools your whole life.

Kevin Scott: Yes.

Ben Thompson: Will artificial intelligence still be a tool?

Kevin Scott: Yes, I think so.

Ben Thompson: Why is this happening? Why wouldn't it be something more autonomous?

Kevin Scott: We don't know, but I think we've got a lot of clues about what humans want. Since “Deep Blue” (Deep Blue) defeated Gary Kasparov in chess in 1997, no human has been superior to a computer in chess, yet people don't care about the game between two computers in chess; everyone is concerned about the game between humans at chess.

Ben Thompson: So, is there an idea that maybe AI will take over everything, but we don't care at all because we only care about other humans?

Kevin Scott: I don't think artificial intelligence will replace anything; I think it will continue to be the tool we use to make things for each other, serve each other, and do valuable things for each other.

I think what we're looking for is meaning and connection, we want to do things for each other, and I think we have a huge opportunity to use these tools to do more of these things in slightly different ways. But I'm not worried that we'll lose our sense of place or purpose because of this.

Ben Thompson: How will artificial intelligence improve lives in Virginia where you grew up?

Kevin Scott: I told my mom's story on stage this morning, and I think she had a lot of trouble with her health last fall.

The population is shrinking in many places. The population of China, Italy, Japan, Germany is shrinking. You can go see when the population peaks in many places. I think France will reach its peak sometime in the early 2030s,

Therefore, we will live in a world with a declining population in the future, so in order to maintain our standard of living, we need fewer people to complete jobs. Over time, for a better standard of living, we must increase our productivity a lot. There must be a way to do everything that needs to be done with fewer people.

In some parts of rural America, where there are canaries in coal mines, we will all face this problem at some point where there are no doctors lining up to care for the rapidly aging population there. Therefore, I think the way artificial intelligence is applied in these places is that it allows people to get everything they need fairly, so they can have dignity and live a beautiful life. I know all of this sounds very abstract, and these are far off questions.

Ben Thompson: I'm from a small town in Wisconsin, and I know what you're talking about.

Kevin Scott: Yes, this is a real thing. I thought about the medical crisis my mom was in. I think everyone in the system is doing their best, but trying their best is still not good enough. I think her results might have been very different if I hadn't intervened. I think of old ladies who don't have sons to intervene. If artificial intelligence can play some role in the intervention process, give people more autonomy over their own health care, more autonomy over their own education, and more autonomy over their own entrepreneurial opportunities, I think this must be a good thing. Of course, that doesn't mean we can completely ignore its risks and drawbacks.

Ben Thompson: I think in general, and in our circle in particular, unlike many other technological revolutions, we haven't fully considered the potential benefits. When you talk about all these good things, people usually say, “Oh, I'd better mention security issues too.” But if you ask people outside of this field, they'll say, “Oh, we know the benefits” — but that part is often overlooked. It's like saying, “No, wait, can we just stop talking? Can we really talk more about what these benefits are?” So, I appreciate your thoughts on this.

Kevin Scott: I do think there was at least one technological revolution of this nature, and that was the printing revolution, when there was a printing press.

Ben Thompson: Church—it took 10, 15 years, but they quickly caught up with the tide of time.

Kevin Scott: Yes, actually, it's been longer than that, and it's been about a century of turbulence and upheaval, and the end result is today's world. Humans today cannot imagine a world without words, books, and the free flow of information.

Ben Thompson: We also ended up with a completely reorganized Westphalian system. We have been through years of war and many things have happened. We've been through the breakdown of the entire Reformation, and many things have happened?

Kevin Scott: Yes, my wife is a trained early modern European historian who is now a philanthropist. The printing revolution is part of our family conversation.

Collaboration with OpenAI

Ben Thompson: Is it because you're an outsider, you've worked at Google and LinkedIn, so you can come to Microsoft and say, “I'm not sure you realise how far behind Google, you need to do something very aggressive here”?

Kevin Scott: Maybe. I think Microsoft actually realized at the time that it was left far behind.

Ben Thompson: Generally, you don't need to convince anyone?

Kevin Scott: Yes. The question is what to do? I'm always drawn to questions.

Ben Thompson: You need a place where you can use talents.

Kevin Scott: Yes, I'm an engineer, and you'll notice that people solve problems in different ways. Some are good at starting, some are good at finishing, and few are good at both. Most of the jobs I chose were about — like the movie “The Magical Nanny McPhee” (Nanny McPhee), I don't know if you've watched it.

McPhee is a fictional babysitter character, and I'm not saying I have magic, but her character is that when kids need her but don't want her, she has to stay; and when they don't need her anymore but want her to stay, she has to leave. That's what drew me to do it. It's like, “Okay, this is a tricky situation. I think I can help with this particular problem.”

Ben Thompson: Since Microsoft is lagging in some areas, do you think this is a challenge and an opportunity?

Kevin Scott: That's not to say the whole company is lagging—Microsoft is good, cloud computing and other businesses are strong. It's about falling behind in terms of artificial intelligence, which wasn't so clearly important in 2017.

Ben Thompson: Does this have anything to do with their product portfolio? For example, if Bing were larger and had more advertising business, they would have more ad revenue — or was it due to negligence? What's driving them?

Kevin Scott: Hard to say. I think one thing is very clear about investing in artificial intelligence now, and that is that you have to be strict about how you invest. This isn't an area where you want all your flowers to flourish; you have to spread your resources across different bets, all of which will ultimately bring huge returns.

Therefore, I think Microsoft has invested a lot of money in artificial intelligence and has a large number of people doing related work, but artificial intelligence is indeed scattered into many different fields, and in enterprises, it is really too expensive and complicated to spread artificial intelligence into many different fields. I think this is an issue that people are still struggling with.

Ben Thompson: So, how did you convince Microsoft: “Look, you spent this much money, we get it. You have Microsoft Research and you have XYZ. Actually, all you need to do is that your core competency, Microsoft, is the ability to spend money, and the OpenAI organization doesn't have money, but it has the ability to build what needs to be done; we have to work together”?

Kevin Scott: I'd like to challenge that assertion. I don't think our core competency is spending money. If you look at the company's DNA, I think our core competency is to build platforms on which others can build their businesses and products.

Ben Thompson: Fair enough. That way, you'll spend a lot less money.

Kevin Scott: Yes, I think that's true when it comes to success. So our arguments are essentially the same as those we have made so far. It's like, the technology trend we're seeing now is that it acts like a platform, and the platform itself will really benefit from focus and have its own views on what you want to invest in”. It's not just your capital, but all the opportunity costs you've spent developing this new platform.

Ben Thompson: Although Microsoft has always defaulted to “build rather than buy.” In this case, the issue isn't “buying” but “building and collaborating,” which is a more precarious position. What evidence or moment convinced the board to say “we don't have time to catch up”?

Kevin Scott: I think it was around 2019 when we reached our first partnership with OpenAI, when we had a pretty clear understanding of the laws of scale. We are aware of the need to act now, and there are two or three options. In my opinion, and in Satya's judgment, this was the fastest way for us to quickly launch and get to market.

Ben Thompson: But the risk is that you're handing over a lot of control to an entity you can't fully control. As a major supporter of this proposition, how much pressure did you feel in November 2013?

Kevin Scott: That was really stressful. But I want to repeat that I think Microsoft, as a platform provider, has done a pretty good job of building complex things with partners over the years. It's not like the PC revolution was entirely completed by Microsoft, but rather the result of Microsoft's collaboration with Intel, Nvidia, and the entire OEM ecosystem. Even Azure's success is inseparable from our partnerships with other infrastructure providers such as Databricks and Snowflake, as well as numerous services running on our cloud and other clouds. So in this day and age, if you're really talking about these hyperscale platforms, we have to be really good at collaborating. Don't think like this: I have to take care of everything myself; it's too difficult.

Ben Thompson: Back to the abstract question, in this case, are you confident that, broadly speaking, it keeps you up at night, so apart from the fact that whoever has computing power can rule the world, will eventually be commercialized? If you get to a critical point, of course, you'll have to do some work, but an Office application can run any model, not necessarily an OpenAI model.

Kevin Scott: I don't think this has much to do with commercialization; it's more related to the two steps we're taking right now. It's like you have a frontier field that is rapidly developing. I think if you want to become a modern AI cloud, or if you want to build modern AI applications, you'd better have access to cutting-edge models. This is already a no-brainer. I think OpenAI has done a great job building these cutting-edge models and making good use of computing resources, and as the cutting-edge models advance, you'll have an ecosystem of very, very smart people who are looking for ways to optimize every part of it.

The laws of scaling (Scaling Laws)

Ben Thompson: Do you think Phi is a real test of your strategy? It basically only took a few years for you to go from being able to do nothing to building the best small model?

Kevin Scott: Yes, I think the interesting thing about Phi isn't about what it replaced, but rather that it combines so well with what we already have because you can do so many things with cutting-edge models, and I don't want anyone to be confused. I think half of the message I conveyed at the construction conference today was this: “You really need to consider how fast this cutting-edge model is developing; one real mistake is getting too addicted to linear optimization that everyone is doing”.

Ben Thompson: Has the tech world forgotten what it feels like to build on Moore's Law? If you go back to the 80s and 90s, you'll need to take a while to understand — you need to build an inefficient application because you want to optimize the front end of the user experience, and you just trust Intel to solve all of your problems.

Kevin Scott: Yes, you're right.

Ben Thompson: Have some aspects been forgotten?

Kevin Scott: I think so.

Ben Thompson: Because everyone complains about the program being bloated, but actually sometimes you need it because performance issues will be solved.

Kevin Scott: Yes, you can fix the bloat later.

Ben Thompson: Exactly.

Kevin Scott: You don't want meaningless bloat, but you don't want...

Ben Thompson: You don't want to over optimize to get rid of it.

Kevin Scott: When I think back to the early days of my career, people used to be proud of it. You write these programs, then go deep into the internal loop of your critical path functions and write lots of...

Ben Thompson: Like, OK, you've optimized from 0.0002 ms to 0.0001 ms.

Kevin Scott: It's been a while. It's really important; it's the key to distinguishing useful things from waste. But because you have Moore's Law, this exponential process of improvement, and if you don't realize it, all you're doing is probably just writing an assembly language for the internal loop of something and missing every chance to write more powerful software.

I was a compiler optimizer when I was in grad school. I have a friend, Todd Proebsting (Todd Proebsting), a professor at the University of Arizona who worked at Microsoft Research for a while. He proposed Proebsting' Law (Proebsting' Law), which is a joke on Moore's Law. Probustin's Law says that compiler optimization researchers work to double the performance of a computer program every 18 years.

He was right. This is one of the reasons I decided not to work as a compiler optimizer because you can spend six months working on something very, very complex and raise the benchmark by 4%, and in the same time, materials scientists and architects will double the speed of this stuff. So what are you doing? Instead of trying to optimize the old slow system, find a way to take advantage of the new fast system that is about to come.

Ben Thompson: What's driving this? You mentioned the new model that is about to be launched, but you also mentioned the case of the GPT-4, GPT-4o, which is 12 times less expensive and 6 times faster. Now, I think if you really dig deep, GPT-4O isn't as good as GPT-4 in some ways, but it's good in others, and it's optimized. Is this just an optimization of the model? It's a solution and an inference, can you figure out a new way to solve this? What are the drivers behind this?

Kevin Scott: Yes, so I think you have two basic points. One is that hardware is getting better. Nvidia has done a lot of work, AMD is doing a good job now, Microsoft's self-developed chips are also underway, and many companies are building their own chips in their own environments. Although smaller transistors are getting cheaper and provide more energy for general computing, at least for now, we still have enough room for innovation in how to use these chips for embarrassing parallel applications such as artificial intelligence.

Ben Thompson: Well, there's always room for more innovation. Even if it is not completely obtainable from the transistor side, it is possible to innovate in other areas such as networks.

Kevin Scott: Yes.

Ben Thompson: “We're going to recreate it in its entirety” is a phrase from “Moneyball” (Moneyball).

Kevin Scott: We got a great price/performance advantage from the hardware, but more importantly, we put a lot of effort into innovation. This includes how to optimize the entire system software stack and how to take advantage of new data formats. Currently, many jobs are moving to faster parallel data processing, such as using FP8 instead of doing all 32-bit arithmetic operations on the model, which helps to use memory more efficiently and complete more operations.

Ben Thompson: So there's a counterintuitive way of saying that you're using less accurate and somewhat dumb, and that's actually a better answer because price and speed are more important?

Kevin Scott: So far, the reduction in accuracy hasn't made anything dumb; you just look at all the activations in the neural network, they're all very, very sparse. The neural network is large, but there aren't many activation signals per one.

Ben Thompson: What impressed me was that the greatest characteristic of this parallel approach was that it was abstracted everywhere. In my opinion, the most attractive applications are those that take parallelism to the extreme; this is also an application that doesn't have to worry about any kind of computational accuracy. If the cost is reduced parallelism, you need more, more.

Kevin Scott: Right, so the hardware is getting better, and our technology is getting better. For example, training techniques and techniques for building inference engines are getting better faster than hardware.

Microsoft's expenses

Ben Thompson: How do you gain confidence in the money invested? This is probably a question that people are questioning. For example, oh, you'd say, “OK, we're confident in our revenue,” what is that confidence? Is it Office Copilot's revenue? Is it API usage? Is it good to overinvest, or is it good to not have enough computational and reasoning skills?

Kevin Scott: What we're seeing now is that the downside of excessive computing power is relatively small. This is just theoretical, because the reality is that we don't have excess computing power. Now that people are in high demand for all of these artificial intelligence products and services, we're just doing crazy things to make sure we have enough computing power and optimize the entire system enough to meet the needs we're seeing.

Looking ahead, there are plenty of economic opportunities here. In my opinion, the API business has gone from scratch, from small to large, faster than ever before. The Copilot business is hugely appealing. Copilot's user engagement is the highest we've ever seen in any new Microsoft 365 office product, and probably the highest ever. So, many times, you're going to sell a new corporate product, and then it takes quite a while for the product to spread to the organization.

Ben Thompson: Sounds like if you could spend more on capital expenses, you would definitely do it; are you just limited by supply?

Kevin Scott: Yes, but if I could invest more in CapEx...

Ben Thompson: Data center energy is also a consideration.

Ben Thompson: Aren't you concerned about demand?

Kevin Scott: No, not right now.

Editor/jayden