Facebook/Meta Tackling Transparent Page Placement For Tiered-Memory Linux Systems

Again throughout the Linux 5.15 cycle Intel contributed an enchancment for tiered reminiscence methods the place much less used reminiscence pages may very well be demoted to slower tiers of reminiscence storage. However as soon as demoted that kernel infrastructure did not have a method of selling these demoted pages again to the sooner reminiscence tiers ought to they change into scorching once more, although now Fb/Meta engineers have been engaged on such performance.

Previous to the Linux 5.15 kernel, throughout the reminiscence reclaim course of when the system RAM was beneath reminiscence strain was to easily toss out chilly pages. Nevertheless, with Linux 5.15 got here the power to shift these chilly pages to any slower reminiscence tiers. Particularly, trendy and forthcoming servers with Optane DC persistent reminiscence or CXL-enabled reminiscence, and so on. Subsequently the pages are nonetheless accessible if wanted however not occupying valuable system DRAM if they don’t seem to be getting used and to keep away from simply flushing them out or swapping to disk.

As famous in that article from September, nonetheless, there wasn’t a method of selling pages again into DRAM when capability is offered or the pages change into scorching. Fb (now Meta) has been engaged on that promotion dealing with and this previous week despatched out their newest patches.

The Linux kernel has been working to higher cope with scorching/chilly pages throughout multi-tiered reminiscence methods. After being demoted, this patch sequence permits scorching pages to return to the highest tier.

Transparent Page Placement for Tiered-Memory offers that help for leveraging AutoNUMA and selling pages from sluggish tier nodes to top-tier nodes.

From the patch sequence:

We examined this patchset on methods with CXL-enabled DRAM and PMEM tiers. We discover this patchset can deliver hotter pages to the toptier node whereas transferring the colder pages to the slow-tier nodes for vary of Meta manufacturing workloads with dwell site visitors. In consequence, toptier nodes serve extra scorching pages and the appliance efficiency improves.

…

With default web page placement coverage, file caches fills up the toptier node and anons get trapped within the slowtier node. Solely 14% of the full anons reside in toptier node. Distant NUMA learn bandwidth is 80%. Throughput regression is 18% in comparison with all reminiscence being served from toptier node.

This patchset brings 80% of the anons to the toptier node. Anons on the slowtier reminiscence is generally chilly anons. Because the toptier node can’t host all the new reminiscence, some scorching information nonetheless stay on the slowtier node. Though, distant NUMA learn bandwidth reduces from 80% to 40%. With this patchset, throughput regression is simply 5% in comparison with the baseline of toptier node serving the entire working set.

With tiered reminiscence servers to change into extra prevalent with CXL, it is nice this tiered-memory dealing with is being sorted out now and shortly sufficient ought to work its strategy to the mainline kernel.