NXP's (formerly Freescale) QorIQ LS1024A SoC (formerly known as Mindspeed
Comcerto 2000) provides fast hardware forwarding between its three Ethernet
MACs. For forwarding between an Ethernet MAC and WLAN, it depends on the ARM
CPU to pass frames between the forwarding engine called PPFE (programmable
packet forwarding engine) and the WLAN hardware connected via PCIe. Most
forwarding decisions as well as NAT are handled inside the PPFE hardware. The
role of Linux is reduced to forwarding descriptors between PPFE and the WLAN
hardware.  Linux rarely accesses the data inside the Ethernet frames.

Performance can be improved by using DMA coherent memory for data buffers
because they do not require L1/L2 cache flushing, invalidation, etc. which
turned out to be expensive on the LS1024A.  The PPFE driver allocates buffers
using dma_alloc_coherent() and builds an skb around it. A few changes to the
kernel were necessary to support DMA coherent skbs. We added the field
dma_coherent to struct sk_buff to indicate that the skb head is in DMA coherent
memory. skb_free_head() in skbuff.c checks for this flag and calls
dma_free_coherent() instead of put_page() or kfree() on skb->head.

dma_alloc_coherent() returns two values: (1) the CPU address which is a regular
pointer i.e. a virtual address and (2) a dma handle which is basically a
physical address that can be communicated to the hardware. For our fast
forwarding approach to work, the virtual address must be in lowmem which means
that there is a simple mapping from virtual address space to physical address
space. In many situation, dma_alloc_coherent() returns an address from the
vmalloc space which does not work for our purposes because dma_map_single()
uses virt_to_page() to map a pointer (virtual address) to a page, and
virt_to_page() does not work with addresses from the vmalloc space. For
dma_alloc_coherent() to return an address in lowmem, a few conditions need to
be met: (1) The caller needs to pass in GFP_ATOMIC for arm_dma_alloc() to
allocate memory from its atomic pool. (2) The atomic pool needs to be allocated
from CMA (CONFIG_CMA and CONFIG_DMA_CMA need to be set)

dma_free_coherent() requires the cpu address (virtual address) and the dma
handle (physical address) to be passed in. However, in our approach, the dma
handle gets lost, because we do not store it in the skb. This turns out not to
be a problem because the dma handle is not required if the memory came from the
atomic pool. Although it does work in practice, the DMA-API debug feature
complains about the incorrect dma handle passed into dma_free_coherent().
Examples of these error messages can be found below. They can be ignored.

December 9, 2015
Daniel Mentz <danielmentz@google.com>


[   10.468709] ------------[ cut here ]------------
[   10.473383] WARNING: CPU: 0 PID: 0 at lib/dma-debug.c:1093 check_unmap+0x695/0x84c()
[   10.481163] NULL NULL: DMA-API: device driver tries to free DMA memory it has not allocated [device address=0x0000000000000000] [size=2048 bytes]
[   10.494246] Modules linked in: pfe(O)
[   10.497952] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.1.0-gfiber0+ #298
[   10.505988] Hardware name: Generic Freescale LS1024A (Flattened Device Tree)
[   10.513095] [<800142a5>] (unwind_backtrace) from [<80011441>] (show_stack+0x11/0x14)
[   10.520885] [<80011441>] (show_stack) from [<803dd59b>] (dump_stack+0x6f/0x98)
[   10.528139] [<803dd59b>] (dump_stack) from [<80020f87>] (warn_slowpath_common+0x6b/0x94)
[   10.536270] [<80020f87>] (warn_slowpath_common) from [<80020fd3>] (warn_slowpath_fmt+0x23/0x2c)
[   10.545009] [<80020fd3>] (warn_slowpath_fmt) from [<802097d5>] (check_unmap+0x695/0x84c)
[   10.553139] [<802097d5>] (check_unmap) from [<80209a41>] (debug_dma_free_coherent+0x61/0x68)
[   10.561692] [<80209a41>] (debug_dma_free_coherent) from [<7f8042eb>] (pfe_eth_poll+0x3f6/0x644 [pfe])
[   10.571001] [<7f8042eb>] (pfe_eth_poll [pfe]) from [<7f8045e7>] (pfe_eth_low_poll+0x36/0x3c [pfe])
[   10.580027] [<7f8045e7>] (pfe_eth_low_poll [pfe]) from [<8031be17>] (net_rx_action+0x153/0x240)
[   10.588759] [<8031be17>] (net_rx_action) from [<800233d1>] (__do_softirq+0xd5/0x294)
[   10.596541] [<800233d1>] (__do_softirq) from [<80023813>] (irq_exit+0x8f/0xd8)
[   10.603803] [<80023813>] (irq_exit) from [<8004b4a1>] (__handle_domain_irq+0x49/0x84)
[   10.611670] [<8004b4a1>] (__handle_domain_irq) from [<800092ef>] (gic_handle_irq+0x27/0x50)
[   10.620069] [<800092ef>] (gic_handle_irq) from [<803e1b3f>] (__irq_svc+0x3f/0x88)
[   10.627573] Exception stack(0x80631f58 to 0x80631fa0)
[   10.632649] 1f40:                                                       00000000 80633350
[   10.640861] 1f60: 00000000 00000000 80630000 806324e8 80632400 8069b6b4 806324a0 8062a304
[   10.649064] 1f80: 803e6a00 8069b6b4 8063003c 80631fa0 8000f14b 8000f14c 40070033 ffffffff
[   10.657285] [<803e1b3f>] (__irq_svc) from [<8000f14c>] (arch_cpu_idle+0x30/0x34)
[   10.664728] [<8000f14c>] (arch_cpu_idle) from [<80045fdb>] (cpu_startup_entry+0x11f/0x204)
[   10.673035] [<80045fdb>] (cpu_startup_entry) from [<805dba31>] (start_kernel+0x309/0x314)
[   10.681244] ---[ end trace 62a3b00cc739c0de ]---

[   11.212979] ------------[ cut here ]------------
[   11.217643] WARNING: CPU: 0 PID: 0 at lib/dma-debug.c:509 add_dma_entry+0xb1/0xd4()
[   11.225323] DMA-API: exceeded 7 overlapping mappings of cacheline 0x03e00000
[   11.232390] Modules linked in: pfe(O)
[   11.236094] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.1.0-gfiber0+ #298
[   11.244121] Hardware name: Generic Freescale LS1024A (Flattened Device Tree)
[   11.251216] [<800142a5>] (unwind_backtrace) from [<80011441>] (show_stack+0x11/0x14)
[   11.258994] [<80011441>] (show_stack) from [<803dd59b>] (dump_stack+0x6f/0x98)
[   11.266247] [<803dd59b>] (dump_stack) from [<80020f87>] (warn_slowpath_common+0x6b/0x94)
[   11.274366] [<80020f87>] (warn_slowpath_common) from [<80020fd3>] (warn_slowpath_fmt+0x23/0x2c)
[   11.283094] [<80020fd3>] (warn_slowpath_fmt) from [<80208efd>] (add_dma_entry+0xb1/0xd4)
[   11.291283] [<80208efd>] (add_dma_entry) from [<7f801861>] (pfe_hif_rx_poll+0x1c8/0x594 [pfe])
[   11.299949] [<7f801861>] (pfe_hif_rx_poll [pfe]) from [<8031be17>] (net_rx_action+0x153/0x240)
[   11.308593] [<8031be17>] (net_rx_action) from [<800233d1>] (__do_softirq+0xd5/0x294)
[   11.316363] [<800233d1>] (__do_softirq) from [<80023813>] (irq_exit+0x8f/0xd8)
[   11.323616] [<80023813>] (irq_exit) from [<8004b4a1>] (__handle_domain_irq+0x49/0x84)
[   11.331474] [<8004b4a1>] (__handle_domain_irq) from [<800092ef>] (gic_handle_irq+0x27/0x50)
[   11.339863] [<800092ef>] (gic_handle_irq) from [<803e1b3f>] (__irq_svc+0x3f/0x88)
[   11.347366] Exception stack(0x80631f58 to 0x80631fa0)
[   11.352433] 1f40:                                                       00000000 80633350
[   11.360636] 1f60: 00000000 00000000 80630000 806324e8 80632400 8069b6b4 806324a0 8062a304
[   11.368838] 1f80: 803e6a00 8069b6b4 8063003c 80631fa0 8000f14b 8000f14c 400f0033 ffffffff
[   11.377049] [<803e1b3f>] (__irq_svc) from [<8000f14c>] (arch_cpu_idle+0x30/0x34)
[   11.384480] [<8000f14c>] (arch_cpu_idle) from [<80045fdb>] (cpu_startup_entry+0x11f/0x204)
[   11.392776] [<80045fdb>] (cpu_startup_entry) from [<805dba31>] (start_kernel+0x309/0x314)
[   11.400978] ---[ end trace 62a3b00cc739c0df ]---