Add support the testing mysql and postgres databases locally and in GitHub Actions.
Fix Docker and Kubernetes run commands.
Fix name of Docker image in GitHub Actions CI workflow build job.
Fixes the “lsstdoc” tracking mode, where version strings with double-digit numbers weren’t being parsed correctly.
This release includes many improvements to LTD Keeper’s build and testing infrastructure:
Python packaging is now defined in
Tests, linting, and other build processes are now run through tox
Testing and Docker builds are now run through GitHub Actions.
The Docker image is now based on Python 3.7 and uses a multi-stage approach to the build to ensure the image is as small as possible.
The codebase is now formatted with black and isort; these linters are run automatically through pre-commit.
The codebase uses type annotations, which are checked though mypy.
LTD_KEEPER_PROXY_FIXenvironment lets you toggle the Werkzeug ProxyFix middleware trust the
X-Forwarded-headers from proxy servers (default is “0”, or “off”). The
LTD_KEEPER_TRUST_X_PREFIXenvironment variable settings specify the number of proxies to trust for the given the header when
This version adds initial support for LTD Events, which acts as an adapter between LSST the Docs and the SQuaRE Events (Kafka) messaging system in the Roundtable platform. Now, when an edition is updated, LTD Keeper sends an
edition.updatedpayload to the LTD Events webhook endpoint.
More event types will be added in the future.
The LTD Events webhook endpoint is configured through the
LTD_EVENTS_URLenvironment variable. If this environment variable is left unset, a webhook request isn’t sent.
Introduced a new Kustomize-based deployment for LTD Keeper in the
/manifests/repository directory. This deployment is designed to work with the Roundtable application platform. See LSST the Docs’s deployment on Roundtable at https://github.com/lsst-sqre/roundtable/tree/master/deployments/lsst-the-docs.
Dropped the Nginx pod from the
keeperpod. Now we assume that LTD Keeper is being deployed behind a solid reverse proxy, such as
nginx-ingress, and that we don’t need to introduce yet another webserver into the stack.
Since nginx is no longer in the application pod, we switch uWSGI to use the
http-socketmode instead of
Fixed a bug that prevented logging configuration and authorization tests with the “v2” endpoint for
POST products/<product>/builds/. [DM-20768]
Added additional tests to ensure that editions tracking
masterbranches were automatically being created for documents using the
lsst_doctracking mode for the main edition. No application fixes were required. [DM-20487]
This version introduces a new “v2” endpoint for
POST products/<product>/builds/that returns two new fields:
post_dir_urls. These fields provide a mapping of presigned POST URLs and associated fields for different prefixes/directories in the S3 bucket associated with the registered build. The benefit of using presigned POST URLs is that clients no longer need their own AWS S3 credentials. LTD Keeper exclusively maintains control over S3 credentials and restricts access to S3 resources through these presigned URLs.
The LTD Conveyor client, version 0.5.0, now uses this new version of the endpoint.
Version 2 endpoints are accessible through a
application/vnd.ltdkeeper.v2+jsonAccept header. Existing clients are unaffected by this change as the default endpoint will continue to operate for the foreseeable future.
Updates to most dependencies:
Flask 0.12.2 to 1.0.3
uWSGI 2.0.17 to 2.0.18
Flask-SQLAlchemy 2.3.2 to 2.4.0
SQLAlchemy 2.3.2 to 1.3.4
PyMySQL 0.8.0 to 0.9.3
Flask-HTTPAuth 2.3.2 to 3.3.0
Flask-Migrate 2.1.1 to 2.5.2
boto3 1.7.54 to 1.9.168
requests 2.18.4 to 2.22.0
Updates to developer and test dependencies:
pytest 3.5.0 to 3.6.3
pytest-cov to 2.5.1 to 2.7.1
pytest-flake8 1.0.0 to 1.0.4
responses 0.9.0 to 0.10.6
pytest-mock 1.9.0 to 1.10.4
mock 2.0.0 to 3.0.5
Fix a bug during product creation (
POST /products) where the product object needs to be flushed in the SQLAlchemy session before creating the default edition.
Minor PEP 8 fixes for regex strings and string comparisons.
Fixes a problem with the new
keeper.s3.delete_directoryimplementation when the S3 prefix has no corresponding objects.
Fixes a bug in
keeper.s3.delete_directoryrelated to “directories” that have 1000 or more objects. The S3 and Boto APIs for deleting objects cannot handle more than 1000 object keys at once. Now this function internally paginates over objects to bypass this limitation.
Adds an experimental Kubernetes deployment of Flower to help monitor the Celery task queue.
Removed unneeded code from the
new_buildroute that was throwing an error if the build corresponded to an edition with a manual tracking mode (as opposed to Git refs).
autoincrementfeature for Edition slugs. Now an edition can be created with
autoincrement=True. Instead of passing a known slug, this features computes the next available integer slug. This feature is designed for the notebook-based report system to create report instances with monotonically increasing instance numbers.
manualtracking mode. This mode ensures that an edition is not updated automatically with a new build. The edition can only be updated with a manual PATCH request that modifies the build URL.
Make an Edition’s
Nonewhen its tracking mode is not
Do not require a
tracked_refswhen creating an Edition that does not use the
Update to Python 3.6.6 (in Docker base image and Travis).
Update boto to 1.7.54 (for Python 3.6.6 compatibility).
Update Celery to 4.2.0 (to fix a compatibility issue with Kombu 4.2’s release).
This release improves and expands the system of edition tracking modes.
There are three new tracking modes:
eups_major_releasetracks an EUPS major release tag (
vX_Y) and its Git variant (
eups_weekly_releasetracks an EUPS weekly release tag (
w_YYYY_WW) and its Git variant (
eups_daily_releasetracks an EUPS daily release tag (
d_YYYY_MM_DD) and its Git variant (
In addition, the code for determining whether an edition should rebuild or not given the tracking mode has been refactored out of the
Edition.should_rebuild model method and into a new
Each tracking mode is now built around a uniform interface.
_ characters can now appear in edition slugs.
Previously these characters were automatically converted to
- characters in edition names, but this prevented editions from being named after semantic version tags or EUPS tags.
This release includes the celery task queuing system and major internal updates to the application structure and dependencies.
Endpoints that launch asynchronous queue tasks now provide a
queue_urlfield. This is a URL to an endpoint that provides status information on the queued task. For example, after
PATCHing an edition with a new build, you can watch the
queue_urlto see when the rebuild is complete. The
queue_urls are provided by the new
We don’t yet provide a way to query the queue in general — you can only get URLs by being the user that triggered the task.
PATCH /editions/(id), should no longer timeout (500 error) for large documentation projects.
/editions/(id)resource includes a new
pending_rebuildfield. This field acts as a semaphore and is set to
trueif there is a pending rebuild task. You can’t
true. If necessary, an operator can
falseif the Celery task that rebuilds the edition failed.
keeper-redis. This deployment consists of a single Redis container (official
redis:4-alpineimage). There is no persistent storage or high-availability at this time (this was judged a fair trade off since the Celery queue is inherently transient).
keeper-redis. This service fronts the
keeper-worker-deployment. This deployment mirrors
keeper-deployment, except that the run
commandstarts a Celery worker for the LTD Keeper application. This deployment can be scaled up to provide additional workers. The
keeper-worker-deploymentis not fronted by a service since the Celery workers pull tasks from
Switched from Flask-Script to
flask.cli. The Makefile now fronts most of the Flask commands for convience during development. Run
make helpto learn more.
Application architecture improvements:
Moved the Flask application factory out of
get_auth_tokenroute to the
Moved DB connection object to
Edition.from_url()methods for consistency with
Now we specifically set up the
keeperlogger instead of the root logger. This keeps things manageable when turning on debug-level logging.
New app configuration for logging level. Debug-level logging is used in the development and testing profiles, while info-level logging is used in production.
New celery app factory in
New Celery task queuing infrastructure in
keeper.taskrunner. In a request context, application code can add an asynchronous task by calling
append_task_to_chain()with a Celery task signature. These task signatures are persisted, within the request context, in
flask.g.tasks. Just before a route handler returns it should call
launch_task_chain(), which launches the task chain asynchronously. The advantage of this whole-context chain is that it orders asynchronous tasks: editions are rebuilt before the dashboard is created. If a task is known to be fully independent of other tasks it could just be launched immediately.
New Celery tasks:
keeper.tasks.editionrebuild.rebuild_edition(): copies a build on S3 onto the edition.
keeper.tasks.dashboardbuild.build_dashboard(): triggers LTD Dasher.
Edition.set_pending_rebuildto use the new
Adds logging with structlog.
Structlog is configured to generate key-value log strings in test/development and JSON-formatted strings in production.
@log_route decorator creates a new logger and binds metadata about a request, such as a unique request ID, method and path.
It also logs the response latency and status when the route returns.
The auth decorators bind the username once the user is known.
In this version we’ve dropped the
nginx-ssl-proxy pod that we’ve used thus far and adopted the standard Kubernetes Ingress resources for TLS termination instead.
This means that the Keeper service is now a NodePort-type service.
The advantage of using Ingress is that we can rely on Google to maintain that resource and ensure that the TLS-terminating proxy is updated with new security patches.
Migrate to setuptools-based packaging. LTD Keeper is now
pip install‘d into the Docker image at build time using the local sdist distribution (there are no plans to put LTD Keeper itself on PyPI).
Use setuptools_scm to automatically establish the application version based on the Git tag.
Automate the creation of the Docker image in Travis CI. The image is tagged with the branch or tag name. The build for the
masterbranch is labeled as
Build and testing are coordinated with a brand new Makefile.
Added the explicit idea of tracking modes to edition resources.
This determines whether or not an edition is updated with a new build.
The mode is set with the
mode field of the
The default tracking mode (
git_refs) is to update if a build resource has the right git ref (a tag or branch name).
lsst_doc tracking mode allows an edition to watch for builds with git refs formatted as
v<Major>.<Minor> and always publish the newest such tag.
This supports the revised LSST DM document release procedure: https://developer.lsst.io/v/DM-11952/docs/change-controlled-docs.html
Removed some technical debt and drift in the Kubernetes deployment templates.
Support non-DM JIRA ticket types (such as
tickets/LCR-N) when auto-slugifying.
Upload directory redirect objects to S3 that tell Fastly to redirect a browser from a directory path to the
Fastly API interactions.
Minimum viable API with Edition, Build, and Product routes.
Interaction with AWS S3 and Route53 with product provisioning and build uploads.