And why should it remain at 110mm2 despite the IMC being single channel for each die if we are to follow your speculation..?
And there s still the need for a 9th chip...?
At this point your estimation is self contradictory...
Just for fun: On a small 7nm square die, arrange 4 x 4-core CCXs in a fully-connected topology (6 links). That gives you a 16-core building block, a super-CCX with no uncore logic. Mount that super-CCX die on top of a just slightly bigger 12nm die containing all the uncore logic (IO, security processor, memory controllers, etc.). Now you have a 16-core CPU. For 64-core "Rome", mount 4 of those on a passive 28nm interposer in a fully-connected topology.
That gives you a 9-die CPU, albeit in a very implausible way.
You should read the whole thread before commenting. Here, I was discussing the "stacked die" suggested by Vattila (duplicated below for your convenience). Also, please look at my diagram again; there is no memory controller on each of the CPU dies; all the 8 memory controllers are moved to the System Controller die.
Well, if i were aware that you said that there s no IMC at all then i would have found your estimation of the uncore being unchanged at 110mm2 even less plausible...
Assuming that there s no IMC induce that there s more room to increase the core count, basicaly they could host 24 cores for each 210mm2 die.
Okay, time for a reality check.
Let's lay down a base context first:
* AMD's R&D budget has been constrained a lot in past 10 years
* AMD started working on znver2 probably in 2013 or 2014
* Naples was the first iteration of the Zen server line - its roots are back to 10h Magny Cours
* Naples use Zeppelin die which is reused in client Ryzen and Threadripper; Zeppelin CCX is reused in Raven APU
* back in summer 2017 there was no Rome but a Starship instead - 48c/96t znver2
* later in 2017 the reliable Canard PC specified "EPYC 2" as 64c, 256MB L3 (4MB per core), PCIe4
* nowadays Charlie@SA has been happy with current Rome config, AMD is confident, etc.
So Rome seems to be a 64c chip or a bit less likely a 48c one.
Now, let's introduce the current "rumor mill" favorite plan aka chiplets. According to a youtuber, the Rome top SKU consists of 9 chips - 1 IO and 8 compute. Details are sparse, but it seems the IO chip would be manufactured at an older process than the compute ones. This idea was further detailed in the diagram posted by OP.
== Naples scaled ==
* double L3 per core - Keep the traffic levels down.
* 8 cores in a CCX - The core interconnect can't probably be a Nehalemish xbar but for instance a SandyBridge's ring bus or whatever. It adds complexity (as in Sandy in 2011) and requires a special CCX for APUs.
* 2 CCXs on a die - This opens up possibilities for a nice TR and scaled down Ryzens. At the same time it keeps the level of complexity down - identical CCXs. Uniform intercore latency for ubiquitous 8c is a nice bonus.
* 4 dies on a package - Simply keep the socket, NUMA mapping, etc. the same.
=> Major investments are: new CCX for APUs, redone intra CCX interconnect and cutting-down Ryzens.
== The chiplets ==
* 8 cores in a CCX - The same issues as above. 8c intercore latency also the same.
* New type of "low latency" interconnect - low latency, a super-high power efficiency (all traffic past L3 goes out of the chip, back to the IO chip, then to RAM) => R&D
* The IO ccMaster - dealing with traffic from all 64c at low latency => R&D
* L4 - R&D
* IO chip itself - can it be reused for ordinary Ryzens - 1x IO + 1x compute? Wasting server-grade IO and L4 for desktop? A different die?
=> Major investments: ???
Now, it's time to lay the Occam's razor. The chiplet solution vs an ordinary one.
Does it make sense to throw away the Magny-Naples know-how given the budget? Mind you, this was really a decision made back in ~2014 (the times when Kaveri struggled with its crippled fw).
Does it make sense to reject znver1 and go to a super radical design which nobody has ever tried in x86 world with an evolution arch revision (znver2)?
Are you sure you can justify the power when going in/out to NB all the time? The same for minimal latency. Can you scale the IO ccMaster, etc.?
Are the benefits worth it? UMA, yields, etc.
8. AMD aren't going to be in anyway die size constrained nor is 200 mm sq is going to be a problem for yields.
You have completely contracted yourself, the uncore way is significantly more complex, requires significantly more R&D and only removes components that are low risk of defects (physical interfaces) or things where AMD only needs to get one die with them working (USB, SATA etc) .I thought 4x4 was the way AMD was going to go too, but 7 nm yields are likely not great and the wafers very expensive, not to mention the design costs.... It may have been cheaper to do the 12 nm Uncore die method.
Plus there's the factor of having to continue to deal with the 12 nm WSA (even if the 7 nm is no more).
My friend, please read things in context. The 110mm^2 pertains to Vatilla's suggested architecture, not mine.
. The "uncore" part remains at 110mm^2..
Well, if i were aware that you said that there s no IMC at all then i would have found your estimation of the uncore being unchanged at 110mm2 even less plausible...
Assuming that there s no IMC induce that there s more room to increase the core count, basicaly they could host 24 cores for each 210mm2 die.
Still think these uncore die people are more crazy then birthers.
1. scaling IO with core count makes sense ( more dies , more IO)....
2. There are still vast improvements to cache coherency AMD can make that will have a big impact on inter CCX latency, they can make distributed home Agents, Move home Agents from memory controller to CCX, run the interfaces at fast speeds etc
3. R&D , where exactly is the massive increase in funding to match this, AMD is only just returning form their crippling lack of R&D, 7nm costs are much higher then 14/16nm and they have console/GPU to worry about as well as they have neglected GPU R&D for a good 3 years.
4.This uncore die strategy means a 2 die a year plan becomes minimum 4 , APU, performance desktop, uncore die, server die.
5. Executing data is cheap, moving data is expensive, the original exascale white papers where around making movement of data cheaper, GPU chiplets, CPU chiplets, stacked memory, active interposer . big reductions in power to transmit, this uncore die does none of that. Most of all its HPC targeted not general compute.
6. Big Fat L4's like people are talking about aren't free, you either waste more power or add latency to memory access (check at the same time as checking memory or wait for L4 result and then check) , if big caches just cost silicon are where mega awesome for performance you would see far more then the "failed" crystalwell
7. This plan Hurts worst case performance a lot
........for almost no gain.
.....it also increases power per memory request.
8. AMD aren't going to be in anyway die size constrained nor is 200 mm sq is going to be a problem for yields
200mm a die, 4x4 ccx a die, 16mb per L3 , 32 lanes of 25/50ghz SERDES a die , improved coherency mechanisms, simple drop in compatibility to all the existing Sockets.
1. scaling IO with core count makes sense ( more dies , more IO), having a one size fits all uncore does not.
2. There are still vast improvements to cache coherency AMD can make that will have a big impact on inter CCX latency, they can make distributed home Agents, Move home Agents from memory controller to CCX, run the interfaces at fast speeds etc
3. R&D , where exactly is the massive increase in funding to match this, AMD is only just returning form their crippling lack of R&D, 7nm costs are much higher then 14/16nm and they have console/GPU to worry about as well as they have neglected GPU R&D for a good 3 years.
4.This uncore die strategy means a 2 die a year plan becomes minimum 4 , APU, performance desktop, uncore die, server die.
5. Executing data is cheap, moving data is expensive, the original exascale white papers where around making movement of data cheaper, GPU chiplets, CPU chiplets, stacked memory, active interposer . big reductions in power to transmit, this uncore die does none of that. Most of all its HPC targeted not general compute.
6. Big Fat L4's like people are talking about aren't free, you either waste more power or add latency to memory access (check at the same time as checking memory or wait for L4 result and then check) , if big caches just cost silicon are where mega awesome for performance you would see far more then the "failed" crystalwell
7. This plan Hurts worst case performance a lot for almost no gain, it also increases power per memory request.
8. AMD aren't going to be in anyway die size constrained nor is 200 mm sq is going to be a problem for yields
200mm a die, 4x4 ccx a die, 16mb per L3 , 32 lanes of 25/50ghz SERDES a die , improved coherency mechanisms, simple drop in compatibility to all the existing Sockets.
You just cant add more cores to a socket willy nilly anyway, you need to have both the pins to deliver power to it and the power delivery already in the motherboard design.How exactly? Move to 16 memory channels and 256 PCIe lanes? What about socket compatibility? It would be nice if you can draw a diagram to illustrate what you mean, like I did.
Currently the home agents for tracking memory requests are the memory controller the data resides on, this means any data that needs to transfer between CCX's needs to check where it is from the memory controller that data ultimately resides on, if CCX A in on die A and CCX B on die B annd memory resides on die C you can see the problem. Distributed home agents means generally that the home agent closest to the data is responsible for tracking that data.How does cache coherency affect inter-CCX latency? What do you mean by "(distributed) home agents"? And doesn't running interfaces at fast speeds burn more power too?
It all comes down to cost, remember you have the built and entire SKU stack, whats your costs look like on a salvage 4 or 6 core part.Read my earlier comment (post #118, pg. 5). AMD only needs to design 3 unique dies for a complete CPU product stack that will deliver a knock-out to Intel from top to bottom in 2019. Is that so prohibitively expensive?
I've read it, doesn't change my point, your making things cost more power and more latency all memory requests get an extra hop in each direction. If you can make it go faster (architectural design around memory access) you can do the same thing with out the uncore die and be even faster again.Read my explanation of how I arrived at the diagram again (post #55, pg. 3). I did not rule out the use of silicon interposers; only that it might not be necessary in this case. Interposers, stacked memory, active interposers, etc. are expensive.
Its not wishfull its just generally not worth it otherwise we would be seeing dirt cheap 45/32nm SOI edram caches packaged with lots of CPU's but we dont.You must have missed it; I repeated several times that the L4 cache in my diagram is just wishful thinking. I don't agree that L4 caches are useless, especially for server workloads. Your opinion as as good as mine though, so I'll leave it there.
Worse case performance is always a cache miss/dirty line and reread from memory, you just increased access latency.How does it hurt worst case performance? Care to elaborate?
yeah i dont see it. AMD think they can address 80% of the market with naples, they will be able to address even more with a 64 core 256mb L3 , improved inter CCX latency SKU. How much more of the market does your design on up and at what cost?I listed a number of advantages of this architecture. Please see post #129, pg. 6
Yes but now you have made every request require its use there are many common scenarios were the GMI interfaces will see little use , how does that play out on a NUMA aware hypervisor where 90% + of vms are 4 threads and below and almost all 8 threads and below. Or a NUMA aware renderer etcI acknowledge this point. But it may not be as bad as you think. Remember this architecture removes the need for 6 criss-crossing IF links needed for interconnecting the 4 dies in NAPLES architecture. These links use (relatively) high power SERDES's to drive (relatively) long traces. For ROME, these links are also expected to run at 25MT's vs 10.7 MT's in NAPLES, so there's some more power saved here.
This is wrong, AMD themselves have said otherwise, it was designed to compete with a projected intel 10nm Server SKU.[/QUOTE]. Remember, ROME was designed to compete with Sky Lake, not Cascade Lake or the hack-job Cooper Lake. Prior to the 64C/9-die rumor, I also thought ROME would be a small evolutionary step from NAPLES.
This is wrong, AMD themselves have said otherwise, it was designed to compete with a projected intel 10nm Server SKU.
An 8-core Ryzen 3000 ("Matisse") engineering sample has supposedly arrived at RTG.
Now the questions are: Is it an APU?
Apparently, there has been some changes to the "interconnect" (wherever that is) that requires RTG to make changes to the video drivers and that why RTG is getting the sample.
Nope, you didn't leave out anything important....
Am I correct? Did I leave out anything?
Nope, you didn't leave out anything important.
Evolutionary, not revolutionary. Cheap, not expensive. Common sense, not over-engineered.
Btw this chiplet idea comes from a single source - the youtuber... That makes it really super-credible.
Please, could you name the other sources?Don't be too quick to judge. AdoredTV is not the first or the only one who said this. Besides, as I said, I no longer doubt that ROME will be will 9 dies not just because the rumor sources are credible (in my opinion), but also because it makes sense.
Looking forward to this.It is neither expensive or over-engineered. Nor is it nonsensical. I will address your other points once I am free. Give me a bit of time.
Sorry to crush your image of Zen, but in fact, the 17h is not revolutionary in the x86 landscape. It's more an evolution of Intel Core processor architecture with AMD's traditional features (including MCM). OTOH, 15h/Bulldozer was clearly a revolutionary x86 design...Clearly, people have difficulty imagining AMD being bold enough to do something revolutionary. They did that with Zen, didn't they?
Intel will not be a sitting duck. They seem to plan stitching two 28c(???) in a MCM manner aka Cascade Lake-AP in a monstrous 5903pin socket. Sure, the TDP will go through the roof but benchmarks will probably look OK as usually.kokhua said:...CPU product stack that will deliver a knock-out to Intel from top to bottom in 2019.
Sorry to crush your image of Zen, but in fact, the 17h is not revolutionary in the x86 landscape. It's more an evolution of Intel Core processor architecture with AMD's traditional features (including MCM). OTOH, 15h/Bulldozer was clearly a revolutionary x86 design...
.
Please, could you name the other sources?.
Looking forward to this.
Sorry to crush your image of Zen, but in fact, the 17h is not revolutionary in the x86 landscape. It's more an evolution of Intel Core processor architecture with AMD's traditional features (including MCM). OTOH, 15h/Bulldozer was clearly a revolutionary x86 design...
When you focus purely on AMD, then sure the 17h is a revolution coming from 15h. But still, this completely new AMD family/architecture just expanded Magny's MCM.
I can't see AMD pulling off a very dubious and radical architecture departure during a current architecture refresh.
Intel will not be a sitting duck. They seem to plan stitching two 28c(???) in a MCM manner aka Cascade Lake-AP in a monstrous 5903pin socket. Sure, the TDP will go through the roof but benchmarks will probably look OK as usually.
Frankly, when the first "new thing" rumors surfaced, I thought AMD would slap 16c of more or less enhanced 16h/Jaguar cores together. At that time the cat cores did OKish and 'micro-servers', web serving and ARM were the main buzzwords.I don't know about you. I didn't foresee AMD coming up with something like EPYC, a modular MCM with 4 Zeppelin dies, each a standalone processor, re-useable for Ryzen. Or a 32C Threadripper. Or that Zen would actually deliver a huge increase in IPC. I suspect Intel did not either. Hell, many thought AMD will go bankrupt not so long ago!
Well, given Intel's marketing/relationship/lineage/powers, it wouldn't be that surprising. Objectivity can go aside. There are people who remember eg. Intel Dunnington era.Of course not. But seriously, you don't think stitching 2 huge dies together with EMIB, and a 350W TDP is going to stave off AMD, do you?
Ryzen Desktop and Notebooks: Different 8C/16T APU, i.e. bring back integrated GPU to mainstream desktop CPUs. A competitive CPU paired with a superior GPU would be a win for AMD. Fuse off features for product segmentation.
15h, 16h, 17h, Interlagos, Magny, Dunnington, ......
Yes, of course, every link on higher levels needs to carry more traffic from the multitude of cores at lower levels (e.g. socket to socket). But you can approach this in many ways, i.e. with different topologies. I discuss this to some degree in my topology thread. In particular, the fatter links on higher hierarchical levels need not go between just two end-points (routers) — and in fact, in the Zen design, in some cases they do not. Instead, they use a router-less approach with multiple sublinks.
For example, the fat link between two EPYC sockets comprises of four sublinks, each between the corresponding dies in each socket (die 0 in socket 0 links to die 0 in socket 1, etc.). This obviates the need for a central router in each socket, which may have become a bottleneck.
On the other hand, I expect that each CCX has an Infinity Fabric controller that routes the traffic in and out of the CCX. I am not sure how the dies in the MCM package are interconnected, i.e. whether each die has a router, or there are direct sublinks between corresponding CCXs in each die (CCX 0 in die 0 links to CCX 0 in die 1, etc.).
However, in my topology thread, I speculate and calculate how many links and ports are required to use the router-less approach at every level, even for the CCXs (i.e. core 0 in CCX 0 is direct-connected to core 0 in CCX 1, etc.).
If you draw out the topology for this router-less approach it will look like a sparsely connected hypercube, I guess. But now we are in territory were I am out of my depth. I may miscategorise these things. This is why I started the topology thread in the first place. I was hoping for an expert or two to chime in.
Now, back to your architecture: How do you handle the choke point in your System Controller chip? Are there connections between the 8-core chiplets, or do all connect only to a central router (crossbar)? If the latter, how much bandwidth does it need to handle, and is that feasible?