The high bandwidth memory (HBM) sector stands out as a lucrative opportunity within the competitive storage industry landscapeRecently, Taiwan Semiconductor Manufacturing Company (TSMC) announced its collaboration in producing the foundational dies for HBM4, utilizing both the N12FFC+ and N5 process technologiesThis announcement aligns with its ongoing expansions of CoWoS (Chip on Wafer on Substrate) advanced packaging capacity, primed to satisfy the burgeoning demand for HBM in various applicationsThe continuous capacity constraints experienced by leading memory manufacturers such as SK Hynix, Samsung, and Micron echo this burgeoning need; they have collectively indicated that their HBM production lines are currently sold outIn response to this insatiable demand, both Samsung and SK Hynix have shifted over 20% of their DRAM production lines to HBM lines

As the industry progresses further into HBM3E and HBM4 technologies, it appears that these crucial players in the memory sector are becoming increasingly intertwined with TSMC, signaling a notable transformation in the industry ecosystem.

I. Unstoppable Performance of HBM Underpins Its Growth

Within the industry, HBM is often categorized under advanced packaging technologies; however, it more accurately fits within the realm of next-generation memory solutionsShort for High Bandwidth Memory, HBM fundamentally refers to a type of memory that utilizes 2.5D or 3D advanced packaging technology to stack multiple DRAM dies akin to a multi-layer cake

• What is the Endpoint of the Competition Among AI Large Models?
• Launch of the First Free Cash Flow ETFs
• The Dollar Will Soar Even Higher
• The Fed's Liquidity Crisis Resurfaces
• $15 Billion Milestone for ETF in Three Months

The classification of HBM under advanced packaging primarily stems from its reliance on TSMC's CoWoS technology, which is pivotal in nearly all HBM systems available today.

By marrying HBM with AI computing chips utilizing the 2.5D CoWoS packaging, the computational performance is harnessed to its fullestAside from this configuration, innovations within the industry are ongoing, with several advanced packaging techniques aimed at enhancing HBM capabilitiesFor instance, TSMC's next-generation wafer system platform—CoW-SoW, SK Hynix's TSV (Through Silicon Via) technology for HBM, and Samsung's non-conductive film thermal compression (TC-NCF) stand at the forefront of these advancementsA word regarding the significance of packaging technology will follow in subsequent sections.

HBM is constructed through the stacking of several DRAM dies, interconnected using TSVs and micro bumps, ultimately linking multilayer DRAM to a base die

This intricate assembly enables high-speed interconnections with the GPU, CPU, or ASIC all sitting on the silicon interposer, which links through a series of bumps to the packaging substrate that connects via solder balls to the PCBThe innovative design notably minimizes the footprint of the device while enhancing bandwidth, reducing latency, and ensuring energy efficiency.

As the demand for advanced GPUs, memory, and related technologies surges in the AI era, deficiencies in supply remain problematicContemporary GPUs are increasingly compensating for CPU functionalities while simultaneously enhancing their processing capabilitiesNevertheless, while processor performance rises approximately 55% year-on-year, memory performance only improves around 10%. Traditional memory solutions like GDDR5 face challenges encompassing lower bandwidth and higher power consumption, rendering them inadequate for current GPU and CPU needs.

In graphics memory design, options typically revolve around GDDR and HBM

Industry examinations consistently point to HBM's advantages over GDDRFor instance, comparing AMD’s specs, GDDR5 features a 32-bit memory interface compared to HBM's astonishing 1024-bit interfaceAlthough HBM operates at a lower clock rate of 500MHz compared to GDDR5's 1750MHz, the bandwidth statistics present an illuminating disparity—HBM achieves a stack bandwidth exceeding 100GB/s, whereas GDDR5 attains merely 25GB/sFrom a spatial efficiency perspective, the integration of HBM directly onto the GPU entails significantly reducing PCB space consumption, while GDDR5 takes up threefold the area of an HBM chip.

While HBM's advantages are evident, it does carry inherent disadvantages when juxtaposed with alternatives like GDDR5 and DDR5. Reports from TrendForce indicate that HBM dimensions can be 35% to 45% greater at equal process nodes

Furthermore, yield rates (including TSV yields) remain about 20% to 30% lower than DDR5 yieldsAdditionally, HBM production lead times are significantly longer, required roughly 1.5 to 2 months more than DDR5, resulting in a timeline exceeding two quarters from wafer fabrication to finished packagingIn the long-term context of the AI surge, the priority for major companies to seize HBM capabilities is completely justifiable.

According to the latest findings by TrendForce, the three leading players in the storage industry began ramping up their investments in advanced process technologies this yearFollowing the sharp rise in memory contract prices, improvements in production capability have commenced, with expectations that the share of DRAM production incorporating 1-alpha nm processes will reach approximately 40% by the end of the year

HBM forms the top priority due to its profitability potential and the persistent growth in demandHowever, yields, hovering around 50-60%, combined with the considerably larger die size compared to other DRAM products (above 60%), occupy a significant degree of wafer allocation.

HBM has already secured its spot as a standard in high-performance computing fields like AI servers, data centers, and autonomous drivingProjections from TrendForce indicate that HBM's contribution to total DRAM production capacities will be 2% in 2023, scaling up to 5% in 2024, with expectations to surpass 10% by 2025. Moreover, beginning in 2024, HBM is forecasted to account for over 20% of total DRAM value, potentially exceeding 30% in 2025.

II. HBM3E Sets the Stage, HBM4 is On the Horizon

Since SK Hynix debuted the first-generation HBM in 2013, the fierce competition among major players has birthed the second (HBM2), third (HBM2E), fourth (HBM3), and now the fifth generation (HBM3E) products.

This year, HBM3E is set to dominate the market, with major shipments expected in the second half

Additionally, reports indicate that the sixth version (HBM4) is under development with substantial changes including a shift from the traditional 1024-bit memory interface to a broader 2048-bit interface, potentially doubling the transmission speedsOur attention will pivot towards HBM3E and HBM4 today.

1) HBM3E Dominates This Year

In the latter half of 2023, Micron, SK Hynix, and Samsung are systematically delivering samples of 24GB HBM3E featuring an eight-layer vertical stack for NVIDIA's validationEarlier this year, both Micron's and SK Hynix's HBM3E variants were approved by NVIDIA, leading to formal orders from the tech giant

Recent industry chatter suggests a delay with Samsung's HBM3E acceptance as TSMC employed standards based on SK Hynix while Samsung's employed TC-NCF, requiring further verification from the tech giant.

According to TrendForce’s analysis, HBM3E is poised to become the market standard in the latter half of 2023. Currently, SK Hynix retains its role as the leading supplier, alongside MicronBoth companies utilize the 1-beta nm process, with announced shipments to NVIDIA already initiatedMeanwhile, Samsung is slated to finalize its validation by Q2 before commencing deliveries shortly afterward.

August 2023 marked the successful completion of SK Hynix's HBM3E development, having initiated mass production in March

Theoretical peak bandwidth for SK Hynix's HBM3E memory stack reaches an impressive 1.18TB/s at a 1024-bit interface, transferring data at 9.2GT/sAccording to Kim Kwi-wook, head of SK Hynix's HBM technical team, the product cycle has shifted to a yearly launch beginning with HBM3E.

On February 26, 2023, Micron confirmed it had started mass production of HBM3E for NVIDIA’s upcoming H200 Tensor Core GPUThe HBM3E model boasts a capacity of 24GB with an eight-layer stack offering a data transfer speed of 9.2GT/s, surpassing a peak bandwidth of 1.2TB/sRelative to HBM3, HBM3E represents a substantial bandwidth increase of 44%. Furthermore, announcements surfaced about plans to launch a high-capacity 36GB 12-Hi HBM3E stack within the same year.

In a recent investor meeting, Micron conveyed that negotiations for HBM storage supply for 2025 had made significant progress, with orders settled regarding volume and pricing

Micron anticipates generating hundreds of millions in revenue from HBM storage noted in its ending fiscal reporting period as of September 2024, with expectations for a substantial rise to billions in 2025 fiscal year revenues.

On October 2023, Samsung announced the introduction of its next-generation HBM3E, codenamed Shinebolt, revealing the industry's first-ever 36GB HBM3E 12H DRAM on February 27. Claimed to achieve bandwidths of up to 1280GB/s and data rates of 9.8GT/s, it significantly surpasses its competitors—SK Hynix at 9GHz and Micron at 9.2GHzThe 36GB capacity represents a 50% enhancement over the previous generation's eight-layer stacksReports affirm that Samsung secured a new $3 billion deal with AMD to deliver HBM3E 12H DRAM for the upcoming second-generation AMD Instinct MI350 AI chip slated for release later this year.

Reviewing parameters for HBM3E across these three pivotal manufacturers illustrates a commitment toward innovations surrounding chip density, bandwidth, height, and capacity improvements

As a benchmark, NVIDIA's GH200 Grace Hopper Superchip has become the world’s first chip equipped with HBM3E memory, boasting a memory capacity three and a half times greater and bandwidth three times higher than its predecessorNVIDIA asserts that this infusion of HBM3E will expedite AI model execution by approximately 3.5 times compared to existing models.

Discerning trends affecting HBM specifications and GPU adaptation over the next few years, TrendForce anticipates three significant developments post-2024.

First, the transition from HBM3 to HBM3E is expected; with NVIDIA gearing up to enhance shipments of H200 equipped with HBM3e in the latter half of the year, transitioning H100 to become the mainstream model

Additionally, products like GB200 and B100 will also utilize HBM3EMeanwhile, AMD's roadmap anticipates launching the MI350 before the year's end, potentially introducing intermediate products like MI32x to compete with the H200, both leveraging HBM3E.

Second, HBM capacities will continue to expand, with a shift anticipated in AI server ecosystems toward capacities ranging from 192GB to 288GB by the end of 2024, moving away from the current NVIDIA H100 capacity of 80GBAMD's range, previously limited to a 128GB MI300A, is forecasted to also gravitate toward much larger HBM capacities.

Third, advancements will see an elevation from 8-high to 12-high configurations in HBM3e GPU product lines

NVIDIA's B100 and GB200 primarily using 8-high HBM3e will achieve 192GB capabilities, while the upcoming B200 in 2025 could feature 12-high HBM3e configurations, doubling to 288GB capacitySimilarly, AMD's anticipated MI350 and MI375 will likely embrace this leap in HBM technology.

Despite the advantages, HBM chips do carry a higher cost structure relative to conventional storage chips—and this pricing is on the ascendanceTrendForce cites that current TSV yields for HBM3e hover only around 40% to 60%, indicating needs for improvementFurthermore, since not all major manufacturers have completed HBM3e customer validations, HBM purchasers are inclined to accept price hikes in securing reliable quality sourcesGiven the variability in average sales pricing driven by DRAM suppliers' reliability and supplies, fluctuations in profit margins are likely to steer market dynamics ahead.

III. HBM4 to Bring Latest Transformations to the Industry

HBM4 marks a critical juncture in HBM technology, emphasizing advancements in packaging technologies and significant changes across three key dimensions in the industry

Primarily, leading memory manufacturers plan to implement substantial revisions to high bandwidth memory technology, transitioning from the existing 1024-bit interfaces toward wider 2048-bit memory interfacesSecond, the HBM ecosystem is reshaping, fostering enhanced collaboration among major manufacturers—SK Hynix, Samsung, Micron—with TSMCHistorically, these companies primarily utilized their proprietary process technologies for chip production; however, synergy is shifting the landscape, and thirdly, the trend of HBM chip customization gaining prominence is clear.

To gain insights into the trajectories for HBM4, let’s turn to the developmental plans and advanced packaging technologies of key players in the sector.

1) Progress among Major Players on HBM4

On April 19, 2024, SK Hynix announced strategic advancements in producing its next-generation HBM, focusing on integrating advanced packaging technologies with logic layers in close collaboration with TSMC

Following this, on May 2, SK Hynix stated its production timeline for HBM4 had been advanced from 2026 to 2025. Specifically, the first products featuring 12-layer DRAM stacks will launch in the latter half of 2025, with 16-layer variants to follow in 2026. With the production timeline still in the early stages, detailed specifications are not yet available.

Taking a lead in the HBM sphere, SK Hynix is noted for its pioneering TSV and MR-MUF (Mass Reflow-Molded Underfill) technologies, which have garnered recognition in the industryAs stack heights increase for HBM chips, the MR-MUF technology faces critical scrutiny due to warping, causing wafer ends to bend and creating voids, a distribution irregularity in protective materialsIn acknowledgment, SK Hynix has been working on reducing these warping phenomena while advancing TSV and MR-MUF tech developments; the focus is now shifting toward addressing the void issues.

Furthermore, SK Hynix delves into next-generation advanced packaging innovations, including chiplet and hybrid bonding technologies, to foster heterogeneous integration between memory storage and logic chips, promoting the development of novel semiconductor solutions

Among these, hybrid bonding emerges as a prospective new option for HBM packagingYet, contrary to previous plans, SK Hynix intends to continue leveraging leading MR-MUF technologies within HBM4, with hybrid bonding additions introduced progressively in tandem with relaxed HBM standards.

Samsung incorporates its own optimization route within HBM4, intending to use NCF assembly technology designed for high-temperature applications, anticipated to integrate 16H technologies into its next-generation HBM4. Samsung envisions having HBM4 samples by 2025. Reports surfaced that Samsung is contemplating the use of hybrid bonding or NCF within this next iteration, with a prevailing sentiment favoring hybrid bonding due to its capacity to assemble stacks compactly without requiring silicon vias.

Noteworthy is Samsung's application of the TC-NCF (Thermal Compression Non-Conductive Film) advanced packaging technology, differing slightly from SK Hynix's MR-MUF techniques

Each time a chip stack is added, a layer of non-conductive adhesive film is positioned between the layersThis polymer film insulates the chips from one another, safeguarding connection points from impactAs time has progressed, Samsung has incrementally reduced the thickness of the NCF material, bringing the fifth-generation HBM3E down to a remarkable 7 micrometers (μm). The company asserts that this approach substantially mitigates warping issues, fitting for building taller stacks.

Micron surfaces as a formidable player in the HBM competition, swiftly shifting focus toward HBM3. The roadmap detailed in recent announcements highlights plans for HBM4 to debut by 2026, followed by HBM4E in 2028.

The anticipated HBMNext, as a prospective next-generation HBM memory by Micron, hints at being HBM4. Projections indicate that HBMNext will yield capacities of both 36GB and 64GB, featuring configurations such as 12-Hi with 24Gb stacks (36GB) or 16-Hi with 32Gb stacks (64GB). Performance claims assert that bandwidth per stack will reach figures ranging from 1.5TB/s to over 2TB/s, heralding data transfer speeds exceeding 11.5GT/s per pin.

2) Industry-Wide Changes Spark Discussions

The advent of the HBM4 era brings forth notable transformations across the industry that are garnering considerable attention.

Firstly, there is a significant shift in memory interfaces: from the current stack bandwidth of 1024 bits to potentially doubling this to 2048 bits per stack.

Various industry insiders assert that widening the sixth generation of HBM4 interfaces from 1024 bits to 2048 bits per stack is under consideration

Each of the HBM stacks since 2015 has utilized a 1024-bit interfaceTraditionally, the current memory interfaces have been designed to be wide but slower; however, as clock speeds rise, it presents potential risks related to signal integrity and energy efficiency dynamics akin to GDDR tech as HBM4 starts to arrive on the stage.

To achieve this without augmenting the footprint taken by HBM memory stacks, the general aim is to virtually double the interconnect density of the forthcoming HBM memory, establishing a memory technology that indeed allows vendors to boost bandwidth without necessarily increasing clock speeds, marking a considerable technical leap in HBM4.

Theoretically, adopting a 2048-bit memory interface would effectively double transmission speeds

For example, NVIDIA’s flagship Hopper H100 GPU currently utilizes six HBM3 chips achieving a 6144-bit bandwidthShould the memory interface transitions to 2048 bits, NVIDIA can theoretically halve the chip count to three while maintaining equivalent performance metrics.

However, all of this necessitates increasingly tight collaboration between chip manufacturers, memory makers, and packaging firms to ensure smooth progressionDan Kochpatcharin, TSMC's design infrastructure management director, articulated at the TSMC OIP 2023 conference in Amsterdam, stating that it is crucial to foster working relationships with all three partners to use their advanced packaging techniques and ensure compatibility with HBM4 layouts and speed demands.

Secondly, TSMC is engaged in close collaboration with the three leading manufacturers, as future specialty processes may necessitate the construction of new foundries to accommodate growth.

As previously indicated, with the onset of HBM4, these three memory juggernauts—SK Hynix, Samsung, Micron—along with TSMC, are forming tighter relationships than ever before

TSMC's senior directors acknowledged their collaboration with primary HBM memory partners to realize the full stack integration of HBM4 at advanced nodes.

Recently, TSMC highlighted its plans to incorporate logic processes for HBM4 manufacturing during the European Technology Conference in 2024, employing N12FFC+ and N5 process technologies to achieve these manufacturing goalsAdditionally, advancements are continually made to the CoWoS protocols, focused on integrating wafer-level processors with HBM4 memory using cutting-edge packaging techniques like CoW-SoW.

In further discussions, TSMC supplied fresh insights on new details concerning the foundational chips that will manufacture HBM4. TSMC's announcement suggests that these chips will embrace logic processes, utilizing two models—the N12FFC+ and N5 technologies

Importantly, the N12FFC+ processes are predicted to yield cost-efficient foundational chips that’ll meet HBM functionality, whilst the N5 foundational chips can deliver lower power consumption at HBM4 speeds while escalating logic capabilities.

Published records indicate that TSMC will implement N12FFC+ manufacturing techniques (12nm FinFet Compact Plus); these will be invoked for embedding HBM4 memory stacks adjacent to system-on-chip (SoC) installations on substratesTSMC claims that their process will greatly enhance the performance performance of HBM4, allowing memory vendors to construct 12-Hi (48GB) and 16-Hi stacks (64GB) achieving bandwidths exceeding 2TB/s.

To accommodate future HBM4 production needs, TSMC plans to expand its specified specialty technology capacity by 50% by 2027. TSMC disclosed during the recent European Technology Conference that it anticipates needing to not only convert existing capacity to meet specialty technology demands but may face the necessity to build new greenfield wafer fabrication spaces

One primary push for this demand will be attributed to TSMC's forthcoming specialized nodal technology—N4e, positioned at the 4nm ultra-low-power processing node.

Additionally, any mention of HBM that arises must recognize TSMC's CoWoS.

CoWoS technology is articulated as being divided into two phases: CoW (Chip-on-Wafer) and WoS (Wafer-on-Substrate). In simple terms, CoW focusing on chip stacking, integrates a variety of logic ICs (like CPUs, GPUs, AISC) alongside HBM memoryWoS, on the other hand, processes these chips adhering to substrates, which encompasses these CoW applications bonded onto substrates before compiling into Printed Circuit Board Assemblies (PCBAs) that constitute the computational units of AI server structures involving other components like network, storage, power supply units (PSUs), and additional I/O peripherals.

Currently, CoWoS supports both 2.5D and 3D packaging variants, enabling enterprises to select varied cost-efficient manufacturing techniques to suit differing application needs

Presently advancing its capabilities, TSMC is modifying its CoWoS-L and CoWoS-R technologies for optimal integration within HBM4. CoWoS-L and CoWoS-R deploy beyond eight layers, accommodating superior routing capabilities to meet HBM4's requirements, ensuring high signal integrity amid an expanding network of over 2000 connections.

TSMC's foundational chips for HBM4 will facilitate the use of its CoWoS-L or CoWoS-R advanced packaging techniques to build system-in-package (SiP) frameworks, capable of accommodating multiple HBM4 memory stacks—potentially totaling 12—providing high bandwidth capabilities with an approximate throughput of 6GT/s at a current of 14mA, according to TSMC's dataFurthermore, TSMC is engaged in collaborations with EDA partners such as Cadence, Synopsys, and Ansys to assess channel signal integrity and thermal precision in HBM4 setups.

In addition, TSMC is also laying groundwork for its newest packaging technology, CoW-SoW

Released during a recent North American technology conference, CoW-SoW’s architectural framework will integrate with wafer-level design in creating 3D integrationsThis new methodology builds upon TSMC's InFO_SoW (Integrated Fan-Out Wafer-Level System-on-Chip) technology—previously unveiled in 2020. To date, only Tesla has adopted CoW-SoW for its Dojo supercomputer, which is now operational, as articulated by TSMC

With the forthcoming CoW-SoW platform, TSMC aims to amalgamate two distinct packaging approaches—InFO_SoW and System-on-Chip integration (SoIC). By employing chip-on-wafer (CoW) techniques, this methodology will allow for the direct stacking of memory or logic above the wafer systemFurthermore, this enables the plausible integration of HBM4 directly above chips while also optimizing cost through adding additional logic atop the chip-level system

The expected timeline for mass production of CoW-SoW technologies extends to 2027; however, the specifics surrounding precise product availability remain elusive.

The licenses surrounding HBM4 have been discussed frequently, with TSMC, SK Hynix, Samsung, and Micron pronouncing the need for proliferating customized solutionsCurrently, procurement agreements aimed at HBM with NVIDIA, AMD, and Microsoft have begun to include customized implementations.

SK Hynix has pointed out the market's shifting inclination toward specialized and customized products in response to evolving customer requirements, emphasizing that top-tier performance in next-generation HBM remains essential while distinctly exceeding conventional memory performance benchmarks.

Samsung underscored an industry-wide recognition that the isolated endeavors of processor and memory companies have historically failed to capture the innovative potential vital for the AI generation

Hence, “custom HBM” is emerging as a trend, representing an initial stride towards collaborative optimization between processors and memory systems amid accelerating innovations.

Recent technology conversations from Micron revealed a significant alignment toward HBM customization in light of looming capacity shortages within the sectorThis stems from the increasingly tailored demands presented by downstream clientsTo satisfy varying performance, functionality, size, forms, and efficacious demands, the marketplace demonstrates heightened inclinations towards specialized and customized HBM products.

There has been some apprehension regarding potential oversupply resulting from the HBM capacity expansions across the industry