release44

DragonFly BSD 4.4

Version 4.4.0 was never officially released

Version 4.4.1 released 07 Dec 2015.

Version 4.4.2 released 16 Feb 2016.

Version 4.4.3 released 19 April 2016.

DragonFly version 4.4 bring further updates to accelerated video for both i915 and radeon users, a new locale system, and a new default linker. Significant behind-the-scenes work has also been done, with symbol versioning, Hammer1 improvements, and other changes. Version 4.4.1 was the first release due to the late inclusion of OpenSSL update 1.0.1q. Version 4.4.2 was released to include OpenSSL 1.0.1r, plus updates to iwm(4) and other miscellaneous bugfixes. Version 4.4.3 includes OpenSSL 1.0.1s, I219 (Skylake) network support, some HAMMER1 bugfixes, and other updates.

The details of all commits between the 4.2 and 4.4 branches are available in the associated commit messages for 4.4RC, 4.4.0, 4.4.1, 4.4.2, and 4.4.3.

Big-ticket items

Improved graphics support

The radeon and i915 drivers are now updated to match the versions found with the Linux 3.18 kernel.

Collation support

DragonFly is the first BSD to properly support collation for named locales. This means data is sorted per the rules of the language and territory selected. Before this release, data would be sorted by the POSIX standard, basically in order of the character set. The lack of collation support manifested in various ways, e.g. French and German sorting would be completely wrong, but people would most often notice because collated databases would return query result sets in the wrong order. A well-known example is PostgreSQL databases with collation fields. The only way to get results in the proper order was to graft ICU to PostgreSQL, which was a non-default ports options that was not approved by PostgreSQL project. Now PostgreSQL and other databases that use libc provide proper results out of the box.

The collation defintions from CLDR (Unicode Common Locale Data Repository), and DragonFly has imported Release 27.01. The collation format, generation tool and libc handling was derived from the Illumos project.

Since this was a long-standing goal of FreeBSD, the collation code from DragonFly has already been imported into FreeBSD CURRENT and will be available for Release 11.0, so the boast of "DragonFly is the only BSD to support named locale collation" will only be for a limited time. There was significant collaboration between FreeBSD and DragonFly (bug discovery/fixing/RB Tree) and patches were fed back to Illumos, so this was a very positive FOSS story resulting in a common locale solution for three platforms.

Locales

The locale support on DragonFly has occasionally been synchronized with FreeBSD over the years, but FreeBSD's support itself had been static. This release brings a complete overhaul to the locale system. The data for all six locale categories (LC_CTYPE, LC_COLLATE, LC_TIME, LC_NUMERIC, LC_MONETARY, LC_MESSAGES) come from a single authoritarian source: the Unicode CLDR Project (http://cldr.unicode.org/). Releases of CDLR and Unicode are fairly frequent and are used to generate the POSIX-formatted input files. Keeping the locale data up to date in the future is intended to be regular maintenance.

While LC_COLLATE support was the most impactful change, locale users will immediately notice improvements in numeric, time and monetary string handling. For usability, the CTYPE defintions were merged into one map, including a very signicant hand-created data file to provide a comprehensive UTF-8 encoding that covers all supported (and many unsupported) languages rather than use the strict and narrow per-locale CLDR CTYPE definition.

Three-component locale names have been introduced here. Common examples are sr_cyrl_RS, sr_latn_RS, zh_Hans_CN, zh_Hant_TW, etc. We introduced "short codes", so now codes like "de_DE", "fr_FR", "en_US", "el_GR", etc. These short-codes are generally mapped to 8-bit character sets such ia ISO-8859-x, but sometimes they are mapped to UTF-8 if the traditional single-byte encoding doesn't adequately cover the locale anymore (e.g. the currency is not supported).

We took the opportunity to intelligently limit the locales supported. For example, a locale will support either ISO8859-1 or ISO8859-15 but not both. Generally this means ISO8859-15 is used in Western Europe and the rest of the locales remain with the older IS08859-1. Several older encodings were eliminated. Note that it was necessary to keep the zh_CN prefix due to compatibility issues with nvi and other 3rd party software.

Regex library replaced with TRE

As a consequence of the locale upgrades, the original regex library had to be forced into POSIX (single-byte) mode always. The support for multi-byte characters just wasn't there. We turned to the TRE Regex library written by Ville Laurikari. It was already being used by the MUSL library and the latest MacOS. The latter had been significantly modified to keep FreeBSD legacy support with specific tokens, and also added bracket pattern matching. The DragonFly regex library matches the Apple version in features and performance, and scores extremely high on the AT&T regex testsuite (much higher than the original library). The only regression is the recent addition of "\<" and ">/" token support, but the bracket equivalents of "?:<:" and "?:>:" work fine. TRE is faster, more capable, and supports multibyte characters, so it's a nice addition to this release.

Change in linker default.

DragonFly now uses the gold linker by default. The legacy "ld.bfd" linker is still available and can be permanently reselected through key definitions in the make.conf file.

Changes since DragonFly 4.2

Kernel

Improved CPU power saving settings

Reduced file allocation/free contention

Reduce kqueue contention

Implemented lwp_setname(2) system call

Fixed dsp(4) nonblocking operation support

Add aperf(4) driver to display effective CPU frequency

Numerous cleanups and fixes to HAMMER

Sync ACPICA to 20151124

Mostly finished removing i386 (32-bit) bits

Adjust boot loader heap to handle larger MD images

Cleanup pass on dead kernel code (syslink remains, etc)

dsched removed (issues with bugs and doesn't work well with SSDs)

Recode pager algorithms for low-memory and out-of-memory pkill

Graphics

drm/i915 and drm/radeon drivers now match Linux kernel 3.18

i915 supports ValleyView/Baytrail and Cherryview Atom SOCs

Broadwell GPUs are now fully accelerated

Preparatory work for Skylake support

The system console now supports drm graphics by default; virtual terminals don't just show a black screen anymore once Xorg has been started and one of the kms kernel modules loaded.

Improved power management. Panel Self Refresh available on i915 hardware

Temperature sensors now supported on Radeon hardware

Networking

Realtek 8168H support in re(4)

iwm(4) driver added (updated in 4.4.2)

rtadvd updated, rtadvctl added

Asynchronous UDP connections, for handling much greater load

New larger TCP starting window, for high-latency connections

Kernel nmbcluster values are live-adjustable, for extreme traffic Networking:

Stabilize UNIX domain socket

New GC code for fd-passing over unix domain sockets

Misc IPv6 sync with FreeBSD

Improved socket(2) performance for TCP and UDP

Improved TCP connect(2) local port selection

Added accept(4) system call

Added support for SOCK_CLOEXEC and SOCK_NONBLOCK socket(2) and accept4(2) flags

Make HW flow control features available to ifconfig

Add tunables to allow NFSROOT iosize and readahead to be set

Bring in expanded ipfw from FreeBSD (called ipfw3 in DFly)

Other drivers

MIDI support has been readded

Many bugfixes for the device mapper

dm-delay and dm-flakey have been added to the device mapper

USB modems work better (or at least don't panic the kernel)

Improved access to ram ECC features and status

wlan updated from FreeBSD (to just prior to the split-device removal)

Userland

the regex library has been replaced with the multi-byte and generally much more capable TRE regex library. It matches OSX in features. (DF is the first BSD to move to TRE after MacOS)

libm replaced with OpenBSD version (this is a collaborative work in progress)

libc now features symbol versioning which can enable binaries created on Release 4.4 to execute on DragonFly for years into the future.

Complete overhaul and update of locales, including the implementation of collation.

malloc.h removed (DragonFly is the first BSD to remove this header)

gcc50 libstdc++ modified to enable full usage of C99 functions on clang

As a consequence of locales update, the output of ls(1) long format and -T format has changed when named locales are in use. Notably, the long format months are always abbreviated in English (using the POSIX definitions thus guaranteed to be both fast and three letters wide) and the year, hour and second information is always displayed. Under the C/POSIX locale there is no change in ls(1) output other than future timestamp handling was corrected to match the POSIX standard requirements.

If WPA Supplicant from base is used, there will be a 10-second delay during boot as a message is displayed that strongly recommends that the DPorts version (security/wpa_supplicant) be used instead.

Improved powerd(8)

Improved top(1) and ps(1) output for LWPs and kernel threads

Fixed the support for pthread_set_name_np(3)

Ported tcpdrop(8)

Addition of many new locales including six Arabic locales (UAE, Saudi Arabia, Egypt, Jordan, Morocco, Qatar), additional Spanish locales (Mexico, Argentina, Costa Rica), additional English locales (Phillipines, Singapore, Hong Kong), corrected Norwegian locales (nb and nn only), extended Swedish (Finland), Sami (Finland, Norway), Serbian presented in both Cyrillic and Latin forms now. ('locale -a' provides a complete list.)

rtadvc imported from FreeBSD

rtsold updated (in 4.4.2)

Various tools have been upgraded in the base system:

nvi2 updated to version 2.1.3

libexecinfo added (synced from FreeBSD)

iconv synchronized with FreeBSD

openssl updated to 1.0.1q (1.0.1r in 4.4.2)

xz updated to 5.2.2

libedit updated to version 2015-03-25

binutils updated to 2.25.1

grep updated to 2.22

tcsh updated to 6.19.00

libdialog updated to v1.2-20150920

(tn)ftp updated to '10 OCT 2015'

gcc updated to 5.2

acpica updated to 20150717

sort(1), which had come from NetBSD, was replaced by FreeBSD version

localedef(1), internal tool with origins on Illumos

cldr2def, internal tool with origins in abandoned FreeBSD project, but extended for DF

tzdata updated to 2016a. (4.4.2)

Removed from the base system:

hostapd (latest version available via dports: net/hostapd)

mklocale (replaced by localedef)

colldef (replaced by localedef)

HAMMER improvements

Many, many cleanups and fixes to Hammer1, thanks to Tomohiro Kusumi

Other improvements

DPorts count hovering around 22,800 ports. Many previously broken ports have been fixed by regular users due to contributions offered through GitHub's Pull Request mechanism. (Thanks!)

Six "official" sets of DragonFly wallpaper is available at x11-themes/dragonfly-wallpapers (pkg install dragonfly-wallpapers). These are automatically installed and preselected for KDE users, and automatically installed for XFCE4 users but the wallpapers still have be manually selected with that desktop (at least for now). They get installed at share/wallpapers which is symlinked to share/backgrounds/dragonfly.

Hammer2 Status

New in this release: Hammer2 supports root mounts, and live dedup. Hammer2 filesystems can be set up, for single-host testing. Bulkfree scans are currently not automated and out of space conditions are currently not recoverable (since everything is copy-on-write, additional mechanisms are needed to allow an out-of-space condition to be rectified). The WANT_HAMMER2 option has to be added to build HAMMER2 pieces. Do not experiment with it unless you have data you can lose! H2 is not ready for prime-time and the media format will continue to change in incompatible ways.

Work has generally progressed with the frontend essentially stable and passing most filesystem stress tests. A solution to the hardlink problem (related to renames of directories along the path to a hardlink breaking the hardlink) has been put in place. The physical hardlink target was previously stored in the most common parent directory vs all of the links. Now the hardlink target will is placed even higher in the directory tree, at or above the common directory at a directory with the 'xlink' chflag set. Renames across xlink points are not allowed.

Work continues on the backend, which is really where all the sophistication is going to be. The backend is now threaded, with one or more threads per physical drive. The frontend replicates and dispatches requests, collects results, and performs cluster operations such as quorum validation and is capable of detaching and aborting the remainder of a request once sufficient progress has been made. The idea is that a stuck physical drive does not cascade back to the frontend and cause the frontend to get stuck. This work is preliminary but the design feels solid so far.

Note that clustering functions are not yet ready for testing in any manner.

Clang Status

The Clang dport mostly works in DragonFly but is not officially supported for kernel or boot-loader builds until we track down and correct a number of issues with the clang-built boot loader. Ultimately the intention is to make Clang one of DragonFlys built-in compilers (DragonFly always ships with two which right now are gcc-4.7.* and gcc-5.2.1, with gcc-5.2.1 as the default).