A network dump might be very useful for collecting kernel crash dumps from embedded machines and machines with a larger amount of RAM then available swap partition size. Besides net dumps we can also try to compress the core dump. However, often this may still not be enough swap to keep whole core dump. In such situation using network dump is a convenient and reliable way for collecting kernel dump.

So, first, let’s talk a little bit about history. The first implementation of the network dumps was implemented around 2000 for the FreeBSD 4.x as a kernel module. The code was implemented in 2010 with the intention of being part of FreeBSD 9.0. However, the code never landed in FreeBSD. Finally, in 2018 with the commit r333283 by Mark Johnston the netdump client code landed in the FreeBSD. Subsequently, many other commitments were then implemented to add support for the different drivers (for example r333289). The first official release of FreeBSD, which support netdump is FreeBSD 12.0.

Now, let’s get back to the main topic. How to configure the network dump? Two machines are needed. One machine is to collect core dump, let’s call it server. We will use the second one to send us the core dump - the client. So, to configure the client we have to use dumpon command:
dumpon [-i index] [-r] [-v] [-k <pubkey>] [-Zz] [-g <gateway>] -s <server> -c <client> <iface>

It allows us to configure the IP address of the server (‘s’ option), the IP which should be used to send the packages (‘c’ option). If it is necessary we can also provide the getaway. From some time we can also encrypt our core dumps which may be very useful if we don’t want to send unencrypted kernel memory in the network (which is a smart approach). For basic configuration we will configure the server, client and the interface from which we want to send our core dump.
dumpon -s 192.168.67.1 -c 192.168.67.2 vtnet0

It’s very straightforward and that's all that we need to do on the client’s side. On the server-side, we have to install additional port netdumpd.
# pkg install netdumpd
[...]

After that, we can run netdumpd with no additional privileges (we can use our user or create special user responsible only for collecting core dumps. Just please remember to keep core dumps in safe place that not all users can have access to). We can use simple command to run netdump:
$ netdumpd -D -d ./network/dumps

This command will run netdump in the debug mode (option D). The option d describes the directory where the new core dumps should be stored. So, you just need to make sure that the netdumpd can store a PID file. It is stored as a PID file under /var/run/netdump.PID by default. This can be changed with the ‘P’ option. We can also use an ‘a’ option to bind to only one given address or the ‘i’ option to provide the script which should be executed when the core dump will be received. Now we can just wait for a core dump. It’s FreeBSD it may take a while… Ok, so instead we can simulate the core dump of machine. On our client we can run command to force core dump:
# sysctl debug.kdb.panic=1
netdump: overwriting mbuf zone pointers
netdump in progress. searching for server...
netdumping to 192.168.67.1 (02:72:50:15:b2:00)
Dumping 150 out of 985 MB:..11%..22%..32%..43%..54%..64%..75%..86%..96% netdump finished.

The debug.kdb.panic allows us to enforce panic on the machine. In the console, we see that the netdump was sent. On our server we can see that the core dump was received ass well. 
Empty bounds file for client w1 [192.168.67.2]
New dump from client w1 [192.168.67.2] (to ./vmcore.w1.0)
.........(KDH from w1 [192.168.67.2])
Completed dump from client w1 [192.168.67.2]

The core dump was stored as vmcore.w1.0. The w1 is the hostname of the client machine. Now we can use kgdb(1) to analyze the situation:
$ cd ./network/dumps
$ gdb /path/to/kernel/kernel ./vmcore.w1.0
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
     <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bhyve/w1/boot/kernel/kernel... (No debugging symbols found in /path/to/kernel/kernel)
0xffffffff80bfbde3 in sched_switch ()
(kgdb)

The net dumps are very powerful and straightforward to configure the feature, which allows us to collect a full memory dump over the network. At the end of the day it is worth noticing that currently FreeBSD netdumps only work with IPv4 and use the UDP protocol. The developers decided to use UDP protocol because it much simpler than TCP/IP. Besides it is also easier to implement, as it does not require any allocations. In case of the panic we don’t want to manipulate with virtual memory or the subsystem may be not available. In the future this may be changed.

Bibliography
The History and Future of Core Dumps in FreeBSD, Sam W. Gwydir, AsiaBSDCon 2017
Improving netdump hardware support and performance with iflib, Sam W. Gwydir, AsiaBSDCon 2018