Elasticsearch

Ingest node lookup processor

We are beginning to think about what an ingest node lookup processor may look like.

Snapshot support in Index Lifecycle Management

We are starting efforts to bring snapshot support to ILM.

Replicated closed indices

Last week, we merged the core replicated closed indices feature into master. We have since added the capability for replicated closed indices to be snapshotted. This also changes snapshot requests in v8.0.0 to automatically expand wildcards to closed indices and hence start snapshotting closed indices by default. For v7.1.0 and above, wildcards are by default only expanded to open indices, which can be changed by explicitly setting the expand_wildcards option either to all or closed. Note that indices are always restored as open indices, even if they have been snapshotted as closed replicated indices.

Cross Cluster Replication

We are adding a forget follower API so that follower shard history retention leases can be manually removed from the primary in case that unfollowing was unable to do so (e.g., if the remote cluster was offline when the unfollow API was executed); while the shard history retention leases would eventually expire, 12 hours is a long time to wait, which would cause history to be kept around much longer than necessary, and would also prevent ILM from performing lifecycle actions such as shrink and delete.

Rolling upgrades from non-secured to secured clusters

A rolling upgrade of a non-secured cluster to a secured cluster is currently not possible due to the need to enable TLS. We have opened a meta-issue to track work on this and, as a first step, opened a PR which adds a setting, which when enabled, will allow the netty-transport to accept incoming plaintext connections even if TLS is enabled.

Resiliency improvements

We backported a long list of recent snapshot resiliency fixes to 6.7, which fixes issues with snapshots hanging.

We fixed a data replication bug where an acknowledged write can be lost. If a write request fails to execute on a replica after completing on the primary, the primary will reach out to the master to remove that failing shard from the in-sync set. If the node with primary is shutting down while reaching out to the master to fail the replica, this can trigger an exception that might unfortunately be ignored and leaves a very short window for the write request to mistakenly be acknowledged to the client. The acknowledged write will be lost if the failing shard is subsequently promoted to primary.

We adapted the cluster state recovery logic in 7.0+ not to close bad indices as this turned out to be problematic for both Zen2 as well as replicated closed indices.

Security token service

We merged the work to support concurrent refresh of tokens, which will avoid some issues in Kibana when a token needs to be refreshed and concurrent requests try to do so.

We also opened a PR to refactor the token service in preparation for splitting the tokens into a separate index since this is data that users probably do not want to backup.

Lucene

Lucene 8.0.0

The release process is underway, but a couple of Solrbugs were discovered that will require a re-spin of the release candidates.

Other

We have proposed that we use a new TermDeletedAttribute rather than removing tokens to address issues with stop words that occur at articulation points.

We are iterating on improving detection of how a line crosses a polygon, which is especially important when crossing the dateline.

We are iterating on fixing a bug with forced merges.

We are looking into only decoding blocks of term frequencies when at least on frequency from the block is requested.

Changes

Changes in Elasticsearch

Changes in 8.0:

Bug fix for AnnotatedTextHighlighter - port of 39525 #39747

BREAKING: Add lower bound on poll_interval #39593

BREAKING: Remove deprecated _time and _term sort orders #39450

Handle total hits equal to tracktotalhits #37907

Changes in 7.1:

Combine overriddenOps and skippedOps in translog #39771

Simulate BWC tests that start with the current version #39102

Chunk + Throttle Netty Writes #39286

Bug fix for AnnotatedTextHighlighter - port of 39525 #39749

Allow snapshotting replicated closed indices #39644

Optimize Bulk Message Parsing and Message Length Parsing #39634

Tie-break completion suggestions with same score and surface form #39564

Allow inclusion of unloaded segments in stats #39512

Add stopword support to IntervalBuilder #39637

Shortcut counts on exists queries #39570

Clearer error message - installing windows service #33804

Fix Token Service retry mechanism #39639

Support concurrent refresh of refresh tokens #39631

Add support for S3 intelligent tiering (#38836) #39376

Enhancements to IndicesQueryCache. #39099

Support concurrent refresh of refresh tokens #38382

Changes in 7.0:

Update logstash-management.json to use typeless template #38653

Change licence expiration date pattern #39681

Bug fix for AnnotatedTextHighlighter - port of 39525 #39750

Add option to force load term dict into memory #39741

Drop node if asymmetrically partitioned from master #39598

Avoid serialising state if it was already serialised #39179

Types removal security index template #39705

Bundle java in distributions #38013

Index on rollup.rollup_search.json is a list #39097

ilm.explain_lifecycle documents human again #39113

metric on watcher stats is a list not an enum #39114

Update spec files that erroneously documented parts as optional #39122

SQL: Enable accurate hit tracking on demand #39527

Changes in 6.7:

Ingest ingest then create index #39607

Introduce forget follower API #39718

Bug fix for AnnotatedTextHighlighter #39525

Add Docker build type #39378

Use any index specified by .watches for Watcher #39541

Fixing the custom object serialization bug in diffable utils. #39544

Fix security index auto-create and state recovery race #39582

Check for .watches that wasn ' t upgraded properly #39609

SQL: Don ' t allow inexact fields for MIN/MAX #39563

SQL: fix COUNT DISTINCT column name #39537

Make _doc work as an alias of the actual type of an index. #39505

Changes in 6.6:

Don ’ t ack if unable to remove failing replica #39584

Use cause to determine if node with primary is closing #39723

Fix Fuzziness#asDistance(String) #39643

SQL: Fix merging of incompatible multi-fields #39560

SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in ' SYS COLUMNS ' #39518

Changes in Elasticsearch Management UI

Changes in 7.1:

Change common router errors to be thrown instead of returned #32670

Fix common router request handler issue #32275

Fix API endpoint typo indices/clearcaches -> indices/clearcache. #32582

Add user_action app for gathering telemetry on user actions #31543

fixing issue with request body not being sent for console DELETE requ … #32407

Changes in 6.7:

CCR Fix form bug + add missing i18n #32544

Changes in Elasticsearch SQL ODBC Driver

Changes in 7.1:

Test: add Kibana ' s sample data to the tests #123

Support for local timezone #118

Changes in Rally

Changes in 1.1.0: