Dear NGLess users,
I’m happy to announce the first beta of NGLess 1.6.0 (1.6.0-beta1).
NGLess has been completely
rewritten in Rust (with the help of our good friend Claude who wrote 21k lines of rust). Since the very first version, NGLess
was implemented in Haskell. As of 1.6, the Haskell codebase has been
retired and the Rust implementation is now the sole, supported
version of NGLess.
The goal of the rewrite was not to
change what NGLess does, it was to make NGLess easier to build,
install, and contribute to:
Simpler builds & distribution. No
more GHC/Stack/Nix toolchain; a single Rust binary that is much
simpler to compile and ship.
Same behavior. The new binary is validated to
produce identical output to the old Haskell binary for ngless
"1.5"+ scripts.
Installation
A prebuilt binary is available from github:
chmod +x ngless
./ngless --version
The external tools (samtools,
bwa, minimap2, prodigal, megahit) are not bundled, but are expected to be available in your environment.
We will not release the beta versions on conda, but the final 1.6.0 will be updated there.
Testing status
The current implementation passes all of
the predefined functional tests (all 99 of them), which diff its
output byte-for-byte against the output of the old Haskell binary. In
addition, some internal larger-scale testing reproduced our previous
results — and showed excellent performance, in particular
being faster when a small number of CPUs are used. That said,
NGLess 1.6 has not yet been thoroughly tested in the wild,
which is exactly why this is a beta: please put it through its paces
and report anything unexpected.
Known regressions & changes to be aware of
HTML run reports are not yet implemented. This is the biggest missing feature in the beta. If you rely on the
end-of-run HTML/JS report, this release does not produce one yet.
The console output is unchanged. We plan to restore this before the
final 1.6.0.
Some errors are detected differently (or later) than
before. Correct scripts should run identically, but some errors are now detected differently, the exact wording, timing
(e.g. caught later at runtime rather than early during
validation), or which of two competing errors is reported may differ
from the Haskell version. We do consider errors that are now caught later than before to be a regression, and we’d like to fix
these where possible; if you hit one, please report it. If
your script is valid, you should see no difference.
Floating point calculations are slightly differently. For example, the previous version would output 9.706432359438262 but the new version prints 9.706432359438264. This is not considered an issue to fix, though.
The debug output is different. This is also not considered an issue to fix.
Compatibility
Scripts declaring version 1.5 or newer run unchanged. You can also declare ngless
"1.6" (which maps to 1.5 semantics, so
output is intended to be identical, but we did bump the version as it's a complete rewrite, so it is hard to be 100% sure).
Scripts declaring older versions (0.5–1.4)
are no longer supported by this implementation. If you rely
on older scripts, please update them to version 1.5+ (or keep using
an older NGLess release).
Also in this
release
Fixed a bug when writing comments to
compressed output files.
Fixed load_sample_from_yaml (it previously took the wrong arguments).
Added the {always_3_fq_files} format flag to write().
Better “did you mean” suggestions for
mistyped arguments/flags (now matches on prefixes).
Fixed a bug where multiplication of
non-integer values used addition instead of multiplication.
Trying the beta
This is a beta release. Please test it
against your own pipelines and report anything that behaves
differently from previous versions. Bug reports, and especially any
output discrepancies, are very welcome on the GitHub issue tracker.
Best, Luis
--