-
Notifications
You must be signed in to change notification settings - Fork 57
/
Copy pathReleaseNotes.txt
executable file
·227 lines (145 loc) · 8.76 KB
/
ReleaseNotes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
=========== Changes for Bpipe Version 0.9.13 ============
- CHANGE: Supported version of Java is now JDK11 and version of
Groovy is 3.0.10
- FEATURE: Many improvements to agents including:
- a single Bpipe agent can now listen to
multiple queues
- support for JMS selectors so that different
commands can be handled with different settings
- FIX: bpipe.$LAUNCHER_PID.run.pid race condition
serialised as part of the command branch which
- FEATURE: support for a "bootstrap" config file
(bpipe.bootstrap.config) that is loaded first and
can perform meta-configuration (configuration
of downstream configuration files)
- FEATURE (experimental): quiet mode to cause stages to output
messages only if they actually execute something
- FEATURE: improved scalability to very large numbers of
input files and output directories
- FEATURE: allow AMQ connection config to come from file
- FEATURE: Improvements to AWS support, including:
- retry AWS commands for api rate limit exceeded, and insufficient
capacity errors with customisable settings
- ability to provide initScript
- improved "transfer" mode that relies only on SSH to transfer files
to the instance
- support for stopping rather than terminating instances, either to make
them reusable or to aid in troubleshooting
- improved support for running docker containers on AWS EC2 instances
- support for reading AWS credentials file and specifying profile
to provide credentials
- FEATURE: support for stage level configuration in bpipe.config
to configure commands
- FEATURE: support for specifying Anaconda environments to run
commands in, where Bpipe handles activating the environment so
that use of the environment is not hard coded in the pipeline
- FEATURE: Support setting of uid to current user in docker commands
- FIX: "bpipe stop" killing non-bpipe processes when using containers on Mac
- FEATURE: many improvements to "dev mode" where bpipe steps through
the pipeline interactively and shows highlighted commands
- FIX: various file handle leaks related to output directories
- FEATURE: Add new execOptions for Singularity container config
- FEATURE: Add bpipe pipeline id and command id to Name tag attached to EC2 instances
because: useful for dev / test where maintaining ongoing
- FEATURE: add support to read .netrc file to obtain API authentication credentials
- FEATURE: Allow configuration of custom options to Docker command
- FEATURE: support for hooks and durable replies in JMS agent : even if
Bpipe errors and is retried, the replyTo will be respected
- FEATURE: support to specify non default platform for container images
=========== Changes for Bpipe Version 0.9.12 ============
- FEATURE: add display of cores into stats table
- FEATURE: support for Groovoy 3.0 and JDK11+
- FEATURE: experimental 'dev' mode - use `bpipe dev`
to enter test mode that highlights
input and outputs in commands, hot updates with changes
and shows branch/command context
- FEAUTRE: support for optional outputs via .optional pseudo
extension
- FEATURE: named branches - use multiply syntax with
groovy Map literal as alternative to list, to create named
parallel branches
- FEATURE: dynamic splits. Use `forwardSplit(...)` to create
pipeline branches that are dynamically determined by result
of a previous stage. (see [docs](https://docs.bpipe.org/Guides/ParallelTasks/#splitting-dynamically-on-arbitrary-metadata))
- FEATURE: Command configuration can now be done based on stage name rather
than command name. This significantly relieves the need to add dedicated
config names to commands in order to adjust their configuration.
- FIX: Slurm executor polling not operating correctly
- FIX: Don't create stray empty .bpipe folders
- FIX: Autoarchive leaving behind empty dirs on some file systems
- DEPRECATED: the Multi command is no deprecated as there are no scenarios where
this is preferred over paralleising using branching structures
- FIX: wrong input may be used in from(multiple) transform() to() construct
- FEATURE: duplicate output files are now explicitly detected and warned about
- FEATURE: enhanced warnings and messages when transform / produce / filter
do not create expected files
=========== Changes for Bpipe Version 0.9.11 ============
- FEATURE: support for required attribute on notification
channels to ensure pipeline failure if channel config fails
- FEATURE: experimental support for virtual file extensions / file type
mappings , defined using: filetype <virtual ext> : [ 'ext.one', 'ext.two',...]
- FEATURE: agent will wait for commands to finish if it is asked to run
multiple in the same directory at the same time
- FEATURE: support for "queue" command : schedules pipeline to run after
another one finishes in a directory
- FIX: only match process owned by user when checking for running bpipe process
- FEATURE(experimental): from clause can now be added in pipeline definition to
cause inputs to prioritise from a given stage, with branch matching
- FEATURE: update aws sdk to support latest instance types
- FEATURE: EC2 commands now show name of stage via name tag, eg. in aws console
- FEAUTURE: accept aws creds from environment if not defined in bpipe.config
using standard AWS env variables for token / secret
- FEATURE: support direct transfer of files via scp for AWSEC2 executor.
If storage config setting is not defined and 'transfer' setting is true
then Bpipe will utilise scp to transfer input/output files to EC2 instances
- FEATURE: allow queue to be overridden within pipeline when sending
to JMS channel via explicit 'queue' property
- FEATURE: support replyTo as property for replying to JMS messages
(rather than JMSReplyTo)
- FEATURE: improved support for archiving: autoarchive and ability to run
archive command from outside bpipe dir
- FEATURE: implement support for 'required' and 'unique' as options on
gitlab messages : ensures pipeline reports error status if
gitlab calls fail
- FEATURE: support for experimental "when" feature to make stages conditional
- FEATURE: mask 'token' config property as sensitive info from log files
- FEATURE: support for setting R_LIBS_USER with R { r_libs="..." }
setting in bpipe.config
- FEATURE: add checks to result message returned in reply-to on
JMS messages
- FEATURE: allow specification of gitlab projects by id when
sending to gitlab
- FEATURE: show custom title in bpipe startup when configured
using title : '...' or title '...'
- FEATURE: support to set env variables for commands via bpipe.config,
by added env = <key> : <value>, ... to command config
- FIX: http send not retried on bpipe retry if initial attempt failed
- FIX: errors printed by trying to send JSON via SMTP
- FIX: adjust bpipe agent jvm options to reduce footprint on large servers
- FIX: fix transform of wildcard (eg: *.txt) to multiple outputs
resulting in regex error
- FIX: undefined pipeline variable somtimes shows bpipe internal stack trace
- FIX: possible concurrentmodification errors when user-specificed config
referenced concurrently in stages / reports
- FIX: fix multilevel branch names not reflected in file names
- FIX: error thrown when creating diagram using 'bpipe diagram ...'
- Support for reservation of gpus with torque (set gpus=N in command config)
- FIX: body of message relayed in JMS message header (causing parse fail in
some clients / protocols)
- FIX: fix regex in transform matching multi inputs only resolving single
- FIX: fix error when literal regex used as transform input specifier
- FIX: previous dirty state carrying over causing unexpected
cleanup of good files
- FIX: put retry around output dir watcher to handle race conditions
/ file system hiccups better
- FIX: Explicitly set pwd for singularity as this seems not always
to occur by default
- FIX:: fix downstream reference to output created by check failing
on retry
- FIX: inability to create gitlab issue with empty content
- FIX: json output double encoded in .bpipe/sent files
- FIX: some commands not running with set -e by default
- FIX:: fix warning printed for dirty files even though
they don't exist, already notified
- FIX: remove duplicates from printing of test outputs
- FIX: remove printout of shebang stripped statement