NEXT
3.0.3 (23 February 2021)
Documentation
The help output for
build
now correctly describes the default behaviour of the--download
and--no-download
options.The help output for
build
now warns about the need to escape wildcards or quote the whole pattern when using--download
so as to avoid expansion by the shell.
3.0.2 (16 February 2021)
Bug fixes
update
more gracefully handles Docker not being installed. Although the command still exits with error (as it currently serves only to pull the latest Docker image), an uncaught exception isn't thrown. More improvements to come later with #87.version
now gracefully handles Docker not being installed when--verbose
is given instead of throwing an uncaught exception.version
now includes Python information when--verbose
is given, which is helpful for debugging Python issues, e.g. which Python install is being used.The Docker (
--docker
) runner forbuild
,shell
, andview
no longer requests a TTY connected to the container when stdin is not itself a TTY (e.g. run from a non-interactive shell). This avoids a fatal error from Docker ("the input device is not a TTY").Distribution metadata was fixed so that the LICENSE file is no longer attempted to be installed under the Python installation prefix (e.g.
/usr/local
). It is instead included inside the "egg-info" directory alongside the code in the Python site libraries.
Development
- Revamp CI by switching from Travis CI to GitHub Actions, expanding the test matrix to macOS and Python 3.9, and adding an integration test for the "native" build runner.
3.0.1 (12 February 2021)
Hotfix for a missing transitive dependency on s3fs via fsspec, which caused
nextstrain build --aws-batch
invocations to fail when s3fs was not installed.
3.0.0 (11 February 2021)
The minimum Python version for installing the Nextstrain CLI itself is now 3.6.
Features
- build: Uploads and downloads for remote AWS Batch builds are now streamed without the use of temporary local files. This halves the local storage overhead needed and also speeds up the transfer of large builds since:
- Uploading can start immediately without first writing the whole archive locally
- Unmodified files do not need to be downloaded, just their metadata
build: The results of remote builds may now be selectively downloaded (or not downloaded at all). Two new
nextstrain build
options are available:--download
--no-download
The former may be given multiple times and specifies patterns to match against build dir files which were modified by the remote build. The latter skips downloading results entirely, which is useful if all you care about are the logs (such as when re-attaching to a build or when a build uploads results itself elsewhere). The default is still to download every modified file.
Currently this functionality is limited to AWS Batch (--aws-batch
) builds,
as it is the only remote environment supported.
Bug fixes
build: Python bytecode files (
__pycache__
and*.pyc
) are no longer uploaded or downloaded from remote builds on AWS Batch.build: Log messages about individual file uploads/downloads to AWS Batch are now printed before each operation, instead after, so you can see what files are taking a while instead of being in the dark until it completes.
remote download: A better error message is now produced when a prefix-less
s3://
URL is provided without the--recursively
option.
Documentation
Clarify how remote builds on AWS Batch acquire AWS credentials.
Fix broken links into AWS documentation for boto3.
Switch to our Nextstrain theme for Sphinx
Some documentation has been shuffled around to better fit within the larger docs.nextstrain.org effort. Redirects were put into place for any moved RTD URLs.
Development
Various improvements to the Read The Docs and Sphinx setup.
Upgrade locked Pipenv development environment.
Fix type checking failures under newer versions of mypy.
2.0.0.post1 (15 June 2020)
Documentation
- The AWS Batch documentation and 2.0.0 release notes (below) now include
information about the additional necessity of granting users the ability to
iam:PassRole
for the role used by Batch jobs.
2.0.0 (2 June 2020)
Features
- build: The AWS Batch runner now supports overriding the image hardcoded in
the Batch job definition. Use the
--image
command-line option, theNEXTSTRAIN_DOCKER_IMAGE
environment variable, or thedocker.image
config setting. This means that both--docker
and--aws-batch
builds will now use the same container image, increasing reproducibility and customizability.
This is a potentially-breaking change, as it requires your AWS IAM users
are allowed to perform the batch:RegisterJobDefinition
action and
iam:PassRole
for the your configured job role (typically
NextstrainJobsRole). The example NextstrainJobsAccessToBatch IAM
policy in the AWS Batch
docs is updated to reflect these new privileges.
build: The new
--cpus
and--memory
options allow limits to be specified for containerized (Docker, AWS Batch) builds. These both automatically inform Snakemake's resource scheduler and the AWS Batch instance size selection. If your builds use Snakemake-based workflows, using these new options is better than specifying--cores …
or--resources mem_mb=…
directly.version: Verbose output now includes the "native" versions of Augur and Auspice, if available.
Bug fixes
view: Auspice v2 dataset JSONs are now detected and included in the list of available datasets message printed to the console.
view: Auspice v1 datasets are now only listed if both the tree and meta JSON files exist. Previously, incomplete datasets with only the tree JSON were listed.
Documentation
- The README now documents known issues with Windows.
Development
Pipenv is now used to provide an easier and more consistent development environment.
pytest is now used to run mypy, flake8, and doctests.
1.16.7 (20 May 2020)
Bug fixes
- AWS Batch builds are now more resilient in the face of transient network or client errors when uploading the build directory and following build logs. Thanks Tony Tung!
1.16.6 (20 May 2020)
Bug fixes
- The previous release did not pass mypy's type checks due to a technicality; mypy has now been placated.
Documentation
check-setup: Success or failure is (hopefully) more clearly messaged now. This was muddied over time by adding support for the native and AWS Batch runners, and we've seen several support requests because of confusion here.
Installation instructions in the README now document all supported computing environments, or "runners". Pipx is also mentioned as a nice alternative to Pip.
Detaching from AWS Batch builds with
--detach
and Control-Z is now mentioned in the AWS Batch documentation.A direct reference to the AWS Batch User Guide on memory management details was added, because AWS docs can be hard to navigate.
The units documented to be used by the
--aws-batch-memory
option are now correct.Runner-selection options (
--docker
,--native
,--aws-batch
) are now in their own option group to reduce clutter among the help output. They are only visible with--help-all
now, like development options. With the use ofnextstrain check-setup --set-default
now emphasized, these options need not be as prominent.The top-level description in
nextstrain --help
output now says more than three words. :D
1.16.5 (22 April 2020)
Features
- build: AWS Batch jobs now require Ctrl-C to be pressed twice within 10s to cancel a job. This is an additional guard on top of 1.16.4's change so that if you accidentally press Ctrl-C once you can't accidentally press it again 30 minutes later and ruin your build.
1.16.4 (22 April 2020)
Features
build: AWS Batch jobs now require Ctrl-C to be pressed twice to cancel the job. This guards a potentially unwanted action from being used accidentally by requiring confirmation with a second Ctrl-C.
build: AWS Batch jobs now report the current job status when re-attaching. Previously the current job status was never reported, only the next status transition. For a PENDING or RUNNING job, it might be some time to the next transition.
1.16.3 (22 April 2020)
Features
- build: AWS Batch jobs now include more detail about why the container exited when available. This surfaces useful messages like "OutOfMemoryError: Container killed due to memory usage" in addition to the exit status.
Documentation
- check-setup: Describe how
--set-default
chooses an environment. Thanks Mary Larrousse!
Bug fixes
- Fix missing import in our gzip utilities which could cause a runtime error
(NameError) when running
nextstrain remote download
on S3 objects with aContent-Encoding
header set to a value other thangzip
ordeflate
. This circumstance is unlikely, but not impossible.
Development
- Use Flake8 for static runtime safety and correctness checks
1.16.2 (16 March 2020)
Bug fixes
deploy/remote upload: Some files, but not all, were being truncated during upload due to a bug in gzip compression handling. Now the whole file makes it to its destination. More details in #62.
build: The default arguments for
snakemake
are no longer used if a different program to run is specified with--exec
.
1.16.1 (25 February 2020)
Documentation
- Update README to include the latest usage information, which mentions the new
remote
command.
1.16.0 (25 February 2020)
Features
The
deploy
command is now an alias for theremote upload
command.The new
remote list
,remote download
, andremote delete
commands allow listing, downloading, and deleting remote datasets and narratives which were uploaded usingdeploy
/remote upload
. Currently only direct s3:// destinations are supported, but its anticipated that Nextstrain Groups will be supported as first-class destinations in the future.
Bug fixes
deploy/remote upload: Invalid credentials are now properly caught and messaged about.
deploy/remote upload: Files are now deployed/uploaded using streaming compression instead of buffering the whole file in memory first.
1.15.0 (18 February 2020)
Features
Environment variables for AWS credentials (
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
, andAWS_SESSION_TOKEN
) are now passed through to builds. This lets builds which use data on S3 to work transparently, for example.Environment variables for ID3C (
ID3C_URL
,ID3C_USERNAME
, andID3C_PASSWORD
) are now passed through to builds. This lets Seattle Flu Study builds which use data in an ID3C instance to work transparently.
1.14.0 (24 September 2019)
No changes since 1.14.0b1, described below.
1.14.0b1 (17 September 2019)
Features
- The
build
command now supports detaching from and re-attaching to builds run on AWS Batch (--aws-batch
).
This adds a feature we've wanted from the beginning of the CLI. By starting
the build with --detach
, a remote job is submitted and the command
necessary to --attach
to the job later is printed. This command includes
the job id and can be used as many times as desired, including while the
remote job is running or after it has completed. It will even work on other
computers or for other people, although you may need to modify the local
build path to a directory of your choosing. The directory may be empty, in
which case all build context will be restored there from the remote job.
AWS Batch builds may also be interactively detached by pressing Control-Z.
Normally this would suspend a Unix process (which could then be resumed with
fg
or bg
or SIGCONT), but in the same spirit, nextstrain build
will
detach from the remote job instead and exit the local process. This also
parallels nicely with our existing Control-C job cancellation support.
There are currently no facilities to track job state locally or list outstanding jobs, but these features may be added later if it seems they'd be useful. As it stands with this new feature, one pattern for launching multiple detached jobs and picking them up later is:
# Loop over several `nextstrain build` commands, appending the last
# line to a shell script.
nextstrain build --aws-batch --detach build-a/ | tail -n1 | tee -a pickup-jobs.sh
nextstrain build --aws-batch --detach build-b/ | tail -n1 | tee -a pickup-jobs.sh
…
# Then, sometime later:
bash pickup-jobs.sh
- The
--aws-batch
runner for thebuild
command no longer requires permission to perform the globally-scoped AWS IAM actions3:ListAllMyBuckets
. Instead, it uses theHEAD <bucket>
S3 API which requires eithers3:ListBucket
, which can be scoped to specific buckets in IAM grants, ors3:HeadBucket
, which is globally-scoped but does not reveal bucket names. More details on these IAM actions are in the S3 documentation.
1.13.0 (10 September 2019)
Features
- The
deploy
command no longer requires permission to perform the globally-scoped AWS IAM actions3:ListAllMyBuckets
. Instead, it uses theHEAD <bucket>
S3 API which requires eithers3:ListBucket
, which can be scoped to specific buckets in IAM grants, ors3:HeadBucket
, which is globally-scoped but does not reveal bucket names. More details on these IAM actions are in the S3 documentation.
1.12.0 (5 September 2019)
Features
- The
deploy
command now supports files other than JSON data files, such as Markdown narratives, by setting the correct content type on upload.
1.11.2 (3 September 2019)
Bug fixes
- This release fixes a regression in the
view
command which caused "connection reset" or "empty response" errors. The regression only affected versions 1.11.0 and 1.11.1 of the Nextstrain CLI when used with Auspice 1.38.0 via thenextstrain/base:build-20190828T223744Z
image. Thanks to Thomas Adams for the excellent bug report!
1.11.1 (30 August 2019)
Bug fixes
- The
check-setup
command no longer errors when Docker isn't installed.
1.11.0 (30 August 2019)
Features
The
view
command now supports--native
flag to run in the native ambient environment.The
check-setup
command now supports a--set-default
flag to save the first supported environment to the Nextstrain CLI's config file. This means that you don't have to specify--native
(or--aws-batch
) every time if you don't have/want Docker support.
Bug fixes
- The
--verbose
flag to theversion
command will no longer cause the Docker image to be downloaded when it isn't available locally.
Documentation
- Describe the somewhat annoying process of how to increase the disk space available to AWS Batch jobs in the AWS Web Console.
1.10.2 (23 August 2019)
Bug fixes
- The environment variables used by Augur are now passed through to --docker and --aws-batch builds.
1.10.1 (26 March 2019)
Features
- The
view
command now sports a--port
option to use an alternate port for the viewer.
1.10.0 (22 February 2019)
Features
- Add ability to specify vCPU and memory when running AWS Batch jobs via
--aws-batch-cpus
and--aws-batch-memory
or via specification in~/.nextstrain/config
or via environment variablesNEXTSTRAIN_AWS_BATCH_CPUS
andNEXTSTRAIN_AWS_BATCH_MEMORY
. This requires corresponding proper setup of compute environment in AWS Batch console.
1.9.1 (11 February 2019)
Features
- Restore the modification times of files when unzipping results from an AWS Batch run. This allows Snakemake's dependency resolution to properly determine file staleness, which in turn allows local builds to continue where AWS Batch builds leave off (e.g. running the bulk of the computation on AWS Batch and then iterating on subsequent trivial steps locally).
1.9.0 (8 February 2019)
Features
- Builds run on AWS Batch no longer delete the build dir zip file from S3 or the job log stream from CloudWatch, making it easier to debug and troubleshoot Batch builds. The Batch setup documentation is updated to note that the previously suggested retention policies are now the only thing preventing runaway data storage costs (and thus a must).
Bug fixes
- Declare missing dep on setuptools, used via
pkg_resources
by thecheck-setup
andupdate
commands. setuptools nearly always exists already on Python installs, especially when nextstrain-cli is installed using pip, but nearly always is not always.
Development
- Static type checking now passes again thanks to a work around for a mypy bug related to namespace packages.
1.8.1 (21 January 2019)
Features
- The
check-setup
command now tests if the local Docker image is new enough for this version of the CLI.
1.8.0 (18 January 2019)
Bug fixes
Docker images between
build-20190115T232255Z
andbuild-20190116T000613Z
(inclusive) broke thenextstrain view
command. It is fixed in this version of the CLI, 1.8.0, in tandem with new images, starting withbuild-20190119T045444Z
. If yourview
command is broken, runningnextstrain update
and following the instructions to upgrade to version 1.8.0 of the CLI should resolve the issue.AWS Batch builds now avoid uploading files matching
environment*
in the build directory, as such files are commonly used for storing sensitive environment values.
1.7.3 (28 December 2018)
Features
- The automatic check for newer versions of the CLI, which happens on the
update
andcheck-setup
commands, now produces a better, more-likely-to-work suggested invocation of pip to perform the upgrade.
1.7.2 (28 December 2018)
Features
- The
build
command now runssnakemake
with the--printshellcmds
option for improved log output.
Bug fixes
User-provided paths are now resolved strictly—they must exist—on both Python 3.5 and ≥3.6, not just 3.5. This discrepancy was unlikely to result in any noticeable problems because of other existence checks which were performed. Nevertheless, the change is good housekeeping and helps ensure robustness.
The
update
command no longer errors on Python 3.6.0 and 3.6.1 when the~/.nextstrain/
does not exist (for example, when upgrading from CLI versions before 1.7.0). #37
Development
Continuous integration testing now includes a much fuller range of Python versions in order to more quickly catch bugs like the one affecting
update
above.Continuous integration testing switched to running the zika-tutorial, a simplified build more suitable for our needs.
1.7.1 (5 December 2018)
Bug fixes
- The
shell
command no longer throws an unexpected exception about a missingwarn()
function when a non-existent build directory is given. The user-friendly error is printed instead, as expected.
1.7.0 (26 November 2018)
Features
Builds can now be run remotely on AWS Batch by passing the
--aws-batch
flag to thebuild
command. Seenextstrain build --help
for more information. Setup required to support this is documented indoc/aws-batch.md
.The
update
command now pulls down new images by theirbuild-*
tag instead of tracking the mutablelatest
tag. Our build tags are, most importantly, not updated after creation and thus are suitable references for reproducible runs. The output ofnextstrain version --verbose
now includes the specific build tag.The
check-setup
command now tests the amount of memory available to containers and warns if it less than 2GB. This is particularly important on Windows and macOS where Linux containers are run inside a VM on the host. The VM may have limited memory allocated to it, leading to out-of-memory errors in builds.
Documentation
Installation instructions in the README are now more detailed.
A big picture overview of where the CLI fits into the Nextstrain ecosystem is included in the README for situating newcomers.
The README now refers to Docker Desktop, the new name for Docker Community Edition.
1.6.1 (25 September 2018)
Features
- The
shell
command announces you're entering the build environment, prints information about mapped volumes, and describes how to leave the environment.
1.6.0 (18 September 2018)
Bug fixes
- On Windows, fix an issue where the
build
andshell
commands spawn thedocker run
process but also immediately return the user to the command-line. Forshell
, the situation was weirder still because the user ended up with two command prompts (cmd.exe and bash) but only one appeared to win when reading from the shared stdin.
1.5.0 (4 September 2018)
Features
The
build
command now supports a--native
flag to run the build outside of any container image, that is, within the ambient environment. That environment might be provided by conda or a cluster module system or custom installs or some other thing. Docker remains the default, although it may be explicitly specified with--docker
. Other runners are planned for the future. The idea is that the cli as a user-facing tool for Nextstrain is separate from a containerized Nextstrain environment (although the two work well together).The
build
,view
, andshell
commands now show an abbreviated set of common options when passed--help
. The full set of options is available using--help-all
. The idea is to make the initial output more approachable.
Development
The README now describes how to annotate the type of variable and use other typing features in a way that's compatible with Python 3.5
Package metadata for PyPi is slightly improved.
1.4.1 (11 August 2018)
Documentation
- Minor updates to README and command help strings
1.4.0 (9 August 2018)
Features
- A new
shell
command launches an interactive shell (bash) inside the build environment, which is useful for running ad-hoc commands and debugging.
1.3.0 (9 August 2018)
Features
The
update
command now prunes old Docker images after downloading new ones. This functionality relies on our new, labeled images. Older images will have to be manually pruned as a one-time step. Seedocker image prune
for more information. Note that locally built images which are tagged nextstrain/base:latest will be pruned whenupdate
is run. Protect such images by giving them an additional tag.The versions of the Docker image and individual Nextstrain components in the image are shown when the
version
command is run with the--verbose
flag.
1.2.0 (1 August 2018)
Features
A new
deploy
command supports uploading data files to S3, allowing the complete pathogen build lifecycle to happen using this package.The
check-setup
andupdate
commands now check if the CLI itself is out of date and could be updated.
Documentation
- Brief descriptions of the changes in each release are now kept in the
CHANGES.md
file. You're reading it!
Development
Describe basic development practices for this package in the README.
Commit to semantic versioning, which I'd been neglecting previously when bumping versions.
Static type checking is now supported for a small fraction of the source code and runs clean under mypy. This is included in Travis CI testing. The goal is to add more type annotations going forward.