apnscp (a/k/a "ApisCP") 3.1 has been released! Codenamed "Business as Usual", 3.1 shifts focus back from widescale adoption to innovation. This release covers several postponed items that were intended in 3.0, but required significant buildup of other areas to satisfactorily implement.

Upgrading

apnscp will upgrade as part of normal nightly updates (see UPGRADING.md). A manual upgrade may be initiated via upcp . upcp will always adhere to your upgrade policy, so ensure the policy is set to edge, major, or all. cpcmd config:get apnscp.update-policy will confirm the policy and cpcmd config:set apnscp.update-policy major will set the default update policy to include all patch and minor updates.

PHP-FPM

PHP-FPM is implemented as a socket-activated service in systemd. Socket activation only spawns a worker pool on page activity, which mitigates a Thundering herd problem in mass deployments. PHP-FPM has been tested exhaustively on 500+ account platforms with minimal impact of service on startup.

This implementation makes use of systemd's excellent dependency tracking to only start once MySQL and PostgreSQL further enhancing startup abilities of both database servers. Upon activation, PHP-FPM worker is chroot'd to the respective account either as an unprivileged system user or - as a cPanel behavior - the account user; however, this practice is highly discouraged. Documentation is covered in PHP-FPM.md.

TimescaleDB

Massive time-series data aggregation poses a significant challenge as platforms accumulate more data with age. Bandwidth data is binned every 3 minutes (controlled via [bandwidth] => resolution in config.ini). Over the span of 12 months with over 500 sites the number of records balloons to 87.6 million records; a lot to sum when looking at historical records. Worse yet record lookups become expensive with simple algorithms. For example, average case runtime to retrieve a record requires 27 steps (big O log2). We can do better by partitioning data into windows. Knowing that bandwidth cycles every month, data can be separated into smaller chunks, 30 day 7.3 million records per segment thus limiting the amount of searches required by 15%. These chunks are unified into a single virtual table called a hypertable thus ensuring transparent storage mechanism for time-series data.

Improved retrieval performance isn't the best feature, continuous aggregates ("cagg") are! A cagg works in the background, automatically, to summarize data from individual data points. Going back to the bandwidth example, if we know bandwidth updates every 5 minutes, TimescaleDB recalculates the total, storing in cache, every 5 minutes in revised totals.

Bandwidth lookups fly now! Prior to caggs, the previous bandwidth overage query took over 20 seconds to run; now it completes in 1/100th the time, 200 ms.

As part of 3.1, TimescaleDB will provide real-time reporting on account resource usage via cgroups allowing the panel to quickly revoke access on sites that exceed their 24-hour quotas (or 30-day). Windows are flexible and extensions simple to use. For example to get the bandwidth used by all domains yesterday, in 5 minute resolutions filling in omissions in data:

SELECT site_id, TIME_BUCKET_GAPFILL('5 minutes', ts) AS bucket, SUM(in_bytes+out_bytes) AS sum FROM bandwidth_log WHERE ts >= NOW() - INTERVAL '1 day' AND ts < NOW() GROUP BY site_id, bucket ORDER BY site_id, bucket;

Bandwidth limits

Following Timescale implementation, bandwidth enforcement is active in 3.1. Thresholds may be configured in config.ini within [bandwidth] that control notification and suspension thresholds. Bandwidth may be optionally forgiven using the bandwidth:amnesty API method for appliance admin:

# Bandwidth is now forgiven for the domain, which ends on the bandwidth cycle day cpcmd bandwidth:amnesty domain.com

ACME v2

apnscp now supports Let's Encrypt ACME v2 protocol. Upon upgrading to apnscp 3.1, an automated migration will update all v1 certificates, which begins its sunset November 1. Wildcard certificates are supported now supported as well. Provided apnscp has control over the DNS for the domain (see DNS.md), this challenge attempt is preferred extending the theoretically maximum number of SSL-protected domains to 50 (wildcard + base domain, 100 hostname limit in SNI). Let's Encrypt issuance is covered in SSL.md.

cPanel/apnscp imports

Restoring from both cPanel and apnscp backups are now supported in apnscp. cPanel restores cover all aspects except for PostgreSQL backups. apnscp restore support is in preview and will be further developed in 3.1. Backups are always strongly encouraged. A drop-in solution with Bacula exists as an addin for apnscp. See Migrations.md for additional information.

Bootstrapper addin facility

Addins are drop-in packages that alter the platform in a meaningful way. As with Bootstrapper, any package that alters system state must do so using Ansible to ensure platform durability. Unleashing a shell script with an assortment of fallible sed commands (that can't even be bothered to set -euo pipefail !) are so 1996. Running a couple commands that automate changes and only change what needs to be changed, plus give you a digest of these changes, that's now.

A few packages of varying complexity were released during 3.1 development to provide a framework for using the Addin system:

PowerDNS courtesy Lithium Hosting

Bacula

rspamd DQS

While not strictly enforced yet, we ask that third-party developers who release modules bundle these modules with plays so that an administrator may run the plays to heal the platform. If you have any questions with writing plays, stop by our developer chat!

Delegated whitelisting

Rampart provides generalized protection to all facets of platform: MySQL, IMAP, POP3, SMTP, SSH, panel access, HTTP, and so on. Any service accessible is guarded against brute-force attacks by Rampart, which results in some interesting scenarios with SOHO businesses. Delegated whitelisting allows account administrators to declare up to n IPv4/IPv6 addresses immune from brute-force deterrence.

When the address is in a delegated whitelist (Account > Whitelist), an address is immune from brute-force blocks. A user that logs into the panel with the blocked IP address is still presented with a popup explaining the service that triggered a block.

SSO subordinate domains

Bridging the gap between reseller and typical hosting accounts, apnscp now supports login to child domains by the parent. For the first domain, set the service parameter billing,invoice=IDENTIFIER . For each child domain parented to this domain, set billing,parent_invoice=IDENTIFIER . Child domains may not login to the parent unless transitioned into by the parent and only within the session transitioned from which the parent transitioned.

Domain transitioning is a simple process within the user card dropdown. If no known domains are on the same server as the parent, the domain is presented normally.

IMAP/POP3/SMTP SNI

IMAP, POP3, and SMTP now support SNI via implicit SSL. Any SSL certificate installed on an account is also available for use with email. Note that explicit (opportunistic SSL via "STARTTLS") does not support SNI. IMAPS (993), POP3S (995), and SMTPS (465) now utilize SNI via haproxy as an SSL terminator. Further work will explore using haproxy to terminate HTTPS traffic as well greatly simplifying the HTTP stack, providing a lightweight DoS sink, and providing zero downtime rolling restarts for all SSL certificates changes. haproxy may be enabled (or disabled) using the cp.bootstrapper Scope.

cpcmd scope:set cp.bootstrapper haproxy_enabled true upcp -sb

IPv6 support

IPv6 support is here! All components are covered (via PR#1).

NAT + hairpin network checks

apnscp now detects if your machine is on a private IP address and configures itself accordingly. Moreover, hairpinning is also detected when performing self-referential accessibility checks prior to SSL issuance.

PowerDNS support

As a contribution from Lithium Hosting, apnscp now includes support for PowerDNS. When migrating from a cPanel server that already uses PowerDNS, apnscp will work in tandem with the DNS cluster to change DNS.

cpcmd scope:set cp.bootstrapper powerdns_enabled true upcp -sb software/powerdns cpcmd scope:set dns.default-provider powerdns

Additional configuration will be necessary in auth.yaml if the same server is not hosting the DNS master. README.md as part of the PowerDNS distribution covers configuration in depth!

WHMCS compatibility

A few changes were introduced to improve compatibility with account tracking in WHMCS, including the separate WHMCS module provided by Lithium Hosting. Disallow username changes via [auth] => allow_username_change. Users may no longer change their username thus allowing WHMCS to operate correctly.

cpcmd scope:set cp.config auth allow_username_change false

cpcmd yaml/json output

cpcmd now supports a variety of output specifiers including Yaml and JSON:

cpcmd -o yaml admin:list-plans cpcmd -o json admin:list-plans cpcmd -o cli admin:list-plans cpcmd -o var_dump admin:list-plans # And for nostalgia... cpcmd -o print admin:list-plans

IO + resource throttling

To set a 2 MB/s write throttle on all PHP-FPM tasks use blkio,writebw or throttle IOPS use the "iops" equivalent, blkio,writeiops :

EditDomain -c cgroup,writebw=2 domain.com # Apply the min of blkio,writ.ebw/blkio,writeiops # Both are equivalent assuming 4 KB blocks EditDomain -c cgroup,writebw=2 -c blkio,writeiops=512 domain.com

Memory ceilings likewise may be set via cgroup,memory .

# Set ceiling of 512 MB for all processes EditDomain -c cgroup,memory=512 domain.com

IO and CPU weighting may be set via ioweight and cpuweight respectively. ioweight requires usage of the CFQ/BFQ IO elevators.

# Default weight is 100 # Halve IO priority, double CPU priority EditDomain -c cgroup,ioweight=50 -c cgroup,cpuweight=200 domain.com

IO throttles also affect tasks spawned from the terminal including Node, Ruby, and Python processes in addition to mail services (last mile delivery via Maildrop + Dovecot IMAP/POP3 access).

Web App blacklists

Disallow web apps for your site via [webapps] => blacklist. For example to disable all web apps but WordPress:

cpcmd scope:set cp.config webapps blacklist '*,!wordpress'

Security improvements

Following an excellent Rack911 audit, further adjustments have been introduced in 3.1 to reinforce the principle of least privilege:

mysql:export-pipe(), pgsql:export-pipe() drop permissions prior to export

Job runner drops permissions voluntarily unless a job requests to elevate. Certain tasks such as Bootstrapper that require elevation will continue to run without occupying a worker slot.

Process calls that drop via suid/sgid settings drop UID/GID in all components of the pipeline

Any process spawned with an effective UID will continue to retain this effective UID for successive spawns. Forked processes can optionally discard the privileged UID (normally "root") by setting the "suid" option prior to execution.

unshare() syscall is experimental. Spawning a session via SSH, crond, or login will create a new PID namespace. PID namespacing detaches the system process tree from the active session replacing it with a limited process tree. This deflects chroot breakage via /proc/1/root traversal.

IP-restricted access

Panel access may be restricted to an IP address or range of IPs via auth , iprestrict service setting. The default value is inherited from [auth] => ip_restriction_limit in config.ini. Access lists may be setup via Account > Settings > Security within the panel. Set to 0 to disable this server-wide except for the appliance administrator. Accounts that SSO ("Login As" feature) do not honor IP-restrictions.

Altering API sensitivity

API calls follow a consistent flow with the UI: only unhandled exceptions or fatal() macro calls terminate flow. Errors may be fatal at the API callee's discretion. Sensitivity may be adjusted in API calls by packaging an "Abort-On:" header that corresponds to the Error Reporter types. For example, if using the Util_API client:

$client = \Util_API::create_client( $key, null, null, [ 'stream_context' => stream_context_create([ 'http' => [ 'header' => 'Abort-On: info' ] ]) ] );

SOAP API calls will terminate whenever an info() macro is used. This works great with error() calls that may arise when adding a side. These would otherwise be encoded in SOAP headers that can be difficult to extract.

Feature renames

Scopes are now accessible via the scope module. In 3.0 Scopes resided in a confusing "config" module. apnscp Scope namespace has been shorted to "cp" as well.

# On 3.0: cpcmd config:get apnscp.update-policy # On 3.1: cpcmd scope:get cp.update-policy

FLARE service

FLARE is a beacon service built to push out multiple, same-day updates in response to zero-day threats. All 3.1 servers participate in FLARE. A FLARE signal forces upcp , which respects the update policy of the server.

That's the major features for 3.1. Now onto 3.2 and beyond! Thank you for everyone for your feedback thus far. Feedback is what makes a product great, so hop by our Discord server if you haven't yet.

Changelog