8.0.11 (20 March 2024)

Announcement

• With the advent of es_client==8.12.5, environment variables can now be used to automatically
  populate command-line options. The ESCLIENT_ prefix just needs to prepend the capitalized
  option name, and any hyphens need to be replaced by underscores. --http-compress True is
  automatically settable by having ESCLIENT_HTTP_COMPRESS=1. Boolean values are 1, 0, True,
  or False (case-insensitive). Options like hosts which can have multiple values just need to
  have whitespace between the values, e.g.
  ESCLIENT_HOSTS='http://127.0.0.1:9200 http://localhost:9200'. It splits perfectly. This is
  tremendous news for the containerization/k8s community. You won't have to have all of the
  options spelled out any more. Just have the environment variables assigned.
• Also, log blacklisting has made it to the command-line as well. It similarly can be set via
  environment variable, e.g. ESCLIENT_BLACKLIST='elastic_transport urllib3', or by multiple
  --blacklist entries at the command line.
• es_client has simplified things such that I can clean up arg sprawl in the command line
  scripts.

Changes

Lots of pending pull requests have been merged. Thank you to the community
members who took the time to contribute to Curator's code.

• DOCFIX - Update date math section to use y instead of Y (#1510)
• DOCFIX - Update period filtertype description (#1550)
• add .dockerignore to increase build speed (#1604)
• DOCFIX - clarification on prefix and suffix kinds (#1558)
  The provided documentation was adapted and edited.
• Use builtin unittest.mock (#1695)
  • Had to also update helpers.testers.verify_client_object.
• Display proper error when mapping incorrect (#1526) - @namreg
  Also assisting with this is @alexhornblake in #1537
  Apologies for needing to adapt the code manually since it's been so long.
• Version bumps:
  • es_client==8.12.6

8.0.10 (1 February 2024)

Changes

The upstream dependency, es_client, needed to be patched to address a
Docker logging permission issue. This release only version bumps that
dependency:

• es_client==8.12.4

8.0.9 (31 January 2024)

Announcements

Curator is improving command-line options using new defaults and helpers from
from the es_client module. This will make things appear a bit cleaner at
the command-line as well as normalize command-line structure between projects
using es_client. No more reimplementing the same code in 5 different
projects!

Changes

• Fix Docker logging per #1694. It should detect whether that path exists and
  that the process has write permissions before blindly attempting to use it.
• If --config is not specified Curator will now assume you either mean to
  use CLI options exclusively or look for a config in the default location.
  Curator will not halt on the absence of --config any more, per #1698
• Increment Dockerfile settings to python:3.11.7-alpine3.18
• Some command-line options are hidden by default now but remain usable. The
  help output explains how to see the full list, if needed.
• Dependency bumps
  • As es_client covers all of the same upstream dependencies that were
    necessary in previous releases, all local dependencies have been erased
    in favor of that one. For this release, that is es_client==8.12.3

7.0.1 (16 August 2023)

Changes

• This release brings some Python modules up to current releases, namely:
  • certifi>=2023.5.7
  • click==8.1.4
  • elasticsearch7==7.17.9
  • pyyaml==6.0.1
• The PyYAML change is notable as the Cython 3.0 upgrade caused it to no longer work.
  Version 6.0.1 fixes that, enabling Curator to build again.
• cx_Freeze changed some configurations to be more Pythonic. #1681 fixes that.

8.0.8 (21 July 2023)

Announcements

Small change to further reduce memory usage by not creating unused data
structures.

This revealed a glitch with dry-runs that would eventually have been reported.

Changes

• Don't populate IndexList.index_info until required by a filter. In other
  words, stop populating the zero values as part of instantiation.
• This uncovered an oversight with the 8.0.7 release. Certain actions, if
  taken with no filters, or only a pattern filter, would never ever populate
  the index_info. This wasn't a huge problem, unless you were using the
  dry-run flag. The dry-run output requires the index_info to be present for
  each index. In 8.0.7, where the empty structure was already in place, a
  dry-run wouldn't fail, but it also wouldn't show any data. This is fixed.
• A few tests also needed updating. They were using contrived scenarios to
  test certain conditions. Now these tests are manually grabbing necessary
  metadata so they can pass.

8.0.7 (21 July 2023)

Announcements

There are no changes in this release that change how Curator will appear to behave to the user. However...

This release ends the practice of collecting all stats and metadata at IndexList initiation. This should make execution much faster for users with enormous clusters with hundreds to thousands of indices. In the past, this was handled at IndexList instantiation by making a cluster state API call. This is rather heavy, and can be so much data as to slow down Curator for minutes on clusters with hundreds to thousands of shards. This is all changed in this release.

For example, the pattern filter requires no index metadata, as it only works against the index name. If you use a pattern filter first, the actionable list of indices is reduced. Then if you need to filter based on age using the creation_date, the age filter will call get_index_settings to pull the necessary data for that filter to complete. Some filters will not work against closed indices. Those filters will automatically call get_index_state to get the open/close status of the indices in the actionable list. The disk space filter will require the index state as it won't work on closed indices, and will call get_index_stats to pull the size_in_bytes stats.

Additionally, the cat API is used to get index state (open/close), now, as it is the only API call besides the cluster state which can inform on this matter. Not calling for a huge dump of the entire cluster state should drastically reduce memory requirements, though that may vary for some users still after all of the index data is polled, depending on what filters are used.

There is a potential caveat to all this rejoicing, however. Searchable snapshot behavior with ILM policies usually keeps indices out of Curator's vision. You need to manually tell Curator to allow it to work on ILM enabled indices. But for some users who need to restore a snapshot to remove PII or other data from an index, it can't be in ILM anymore. This has caused some headaches. For example, if you are tracking an index in the hot tier named 'index1' and it is in process of being migrated to the cold tier as a searchable snapshot, it may suddenly disappear from the system as 'index1' and suddenly re-appear as 'restored-index1'. The original index may now be an alias that points to the newly mounted cold-tier index. Before this version, Curator would choke if it encountered this scenario. In fact, one user saw it repeatedly. See the last comment of issue 1682 in the GitHub repository for more information.

To combat this, many repeated checks for index integrity have become necessary. This also involves verifying that indices in the IndexList are not actually aliases. Elasticsearch provides exists tests for these, but they cannot be performed in bulk. They are, however, very lightweight. But network turnaround times could make large clusters slower. For this reason, it is highly recommended that regex filters be used first, early, and often, before using any other filters. This will reduce the number of indices Curator has to check and/or verify during execution, which will speed things up drastically.

Other Changes

Most of the unit tests needed updating due to these changes (adding a few more Mock responses), so I took the time to refactor them to use a builder method instead of repeating the same few lines of code in each test. This represents the vast majority of code deletions (2,085 lines!). All of the tests are passing, both integration and unit tests.

8.0.6 (18 July 2023)

Breakfix Release

• Small breakfix change to catch a similar rare race condition patched in
  8.0.5 covering the get_index_stats() method of IndexList. This patch
  covers the get_metadata() method and closes #1682.

8.0.5 (13 July 2023)

Announcements

Release for Elasticsearch 8.8.2

Changes

• Small PEP formatting changes that were found editing code.
• Bump Python version in Dockerfile to 3.11.4
• Bump Python dependency versions.
• Change targetName to target_name in setup.py for newest version
  of cx_Freeze. Hat tip to @rene-dekker in #1681 who made these changes
  to 5.x and 7.x.
• Fix command-line behavior to not fail if the default config file is not
  present. The newer CLI-based configuration should allow for no config file
  at all, and now that's fixed.
• Initial work done to prevent a race condition where an index is present at IndexList
  initialization, but is missing by the time index stats collection begins. The resultant
  404s were causing Curator to shut down and not complete steps.
• When running in a Docker container, make Curator log to /proc/1/fd/1 by
  default, if no value is provided for logfile (otherwise, use that).

8.0.4 (28 April 2023)

Announcements

Allow single-string, base64 API Key tokens in Curator.

To use a base64 API Key token in YAML configuration:

  elasticsearch:
    client:
      hosts: https://host.example.tld:9243
    other_args:
      api_key:
        token: '<base64 token goes here>'

To use a base64 API Key token at the command-line:

  curator --hosts https://host.example.tld:9243 --api_token <base64 token goes here> [OTHER ARGS/OPTIONS]

NOTE: In neither of the above examples are the alligator clips necessary (the < and > characters).

Changes

• Update es_client to 8.7.0, which enables the use of the base64 encoded API Key token.
  This also fixes #1671 via untergeek/es_client#33

8.0.3 (22 February 2023)

Announcements

A new action called cold2frozen has been added to Curator. It is not going to be of much use to
the vast majority of Elasticsearch users as it serves a very narrow use-case. That is, it migrates
searchable snapshot indices from the cold tier to the frozen tier, but only if they are not
associated with ILM (Index Lifecycle Management) policies. As escalation into the cold and frozen
tiers is usually handled by ILM, this is indeed a rare use case.

Changes

• Fixed instruction display for delete repository action of es_repo_mgr
• Fix unit tests to import more specifically/cleanly
• Fixed Hatch build includes (this was speed-released to PyPI as 8.0.2.post1) as Curator did not
  function after a pip install.
• Added cold2frozen action, and tests.