R81 pre-release / Parallel execution

Overview

The central new feature in Release 81 is its support for parallel backup execution - an ability to execute multiple backup steps at once.



The central new feature in Release 81 is its support for parallel backup execution - an ability to execute multiple backup steps at once. Parallel execution

Prior to R81, the file system scan and file ID retrieval were already done

in a parallel fashion, using several threads.



Release 81 extends this behavior to all phases of backup, allowing the program to execute all suitable operations in parallel with each other.



If you are familiar with the /mt switch of robocopy, it's a similar idea, but with automatic thread count management and several other important improvements.



Prior to R81, the file system scan and file ID retrieval were already donein a parallel fashion, using several threads.Release 81 extends this behavior to all phases of backup, allowing the program to execute all suitable operations in parallel with each other.If you are familiar with the /mt switch of robocopy, it's a similar idea, but with automatic thread count management and several other important improvements. Use cases

In practical terms, parallel execution results in a *very* noticeable speed increase in the following cases:



⦁ Backups with a lot of small- to medium-sized files

⦁ Backups that require creating a large number of folder

⦁ Backups that require deleting a large number of folders or files

⦁ Backups that "touch" a lot of folders



For a bit of anecdotal data see here -



In practical terms, parallel execution results in a *very* noticeable speed increase in the following cases:⦁ Backups with a lot of small- to medium-sized files⦁ Backups that require creating a large number of folder⦁ Backups that require deleting a large number of folders or files⦁ Backups that "touch" a lot of foldersFor a bit of anecdotal data see here - https://bvckup2.com/wip/03102019 Configuration

Parallel execution is enabled by default with the thread count set to the number of CPUs for local backups and to the double of that for network backups.



To set the thread count to some fixed custom value use the following override -



conf.exec.threads <thread-count>



See



Parallel execution is enabled by default with the thread count set to the number of CPUs for local backups and to the double of that for network backups.To set the thread count to some fixed custom value use the following override -conf.exec.threads See https://bvckup2.com/support/forum/topic/1140 for how to do this. Versions

Pre-release 1.80.99.1 and newer. Pre-release 1.80.99.1 and newer.

1 Pre-release 80.99.2

⦁ Fixed an issue with handling failures when _retrying_ a backup of a

_new_ file.



More specifically, if a new file appears at source, the backup is run and

it fails with a retryable error mid-copying. For example, a network

connection goes away. This will cause the program to retry the copying

after a pause. If this retry would also fail with an error, then bvckup2

would pop up the "something is wrong" message and shut down. ⦁ Fixed an issue with handling failures when _retrying_ a backup of a_new_ file.More specifically, if a new file appears at source, the backup is run andit fails with a retryable error mid-copying. For example, a networkconnection goes away. This will cause the program to retry the copyingafter a pause. If this retry would also fail with an error, then bvckup2would pop up the "something is wrong" message and shut down.

2 Pre-release 80.99.3

⦁ Fixed an issue with the retrying mechanism panicking when the last

attempt fails. There was an invalid self-consistency check (an assert),

which is as ironic as it gets. A bug in a bug checking code. ⦁ Fixed an issue with the retrying mechanism panicking when the lastattempt fails. There was an invalid self-consistency check (an assert),which is as ironic as it gets. A bug in a bug checking code.

3 Pre-release 80.99.4

⦁ Fixed a crashing issue with the archive trimming code that surfaced

every time an archive scan would identify and remove expired items. ⦁ Fixed a crashing issue with the archive trimming code that surfacedevery time an archive scan would identify and remove expired items.

4 Pre-release 80.99.5

⦁ Reworked retrying logic that is used to re-execute failed steps in a

presence of transient errors (e.g. network disconnects). This also

fixed an issue that caused self-inflicted crashes in cases when

copying of a _new_ file succeeded after several attempts.



⦁ Reworked logging module to report step info as it's being executed

IF this step happens to be the only one active and no other steps are

being scheduled until it completes.



This happens when bvckup2 is backing up a very large file, at which

point it will finish all other active steps, but won't queue any new ones.

This is done to allow the bulk copying to proceed in peace if you will.



Prior to 99.5 all step-specific info was recorded into a backup log only

after the step finished. However if there's exactly one step in progress

we can dump its progress into log as we go - which is exactly what 99.5

implements. ⦁ Reworked retrying logic that is used to re-execute failed steps in apresence of transient errors (e.g. network disconnects). This alsofixed an issue that caused self-inflicted crashes in cases whencopying of a _new_ file succeeded after several attempts.⦁ Reworked logging module to report step info as it's being executedIF this step happens to be the only one active and no other steps arebeing scheduled until it completes.This happens when bvckup2 is backing up a very large file, at whichpoint it will finish all other active steps, but won't queue any new ones.This is done to allow the bulk copying to proceed in peace if you will.Prior to 99.5 all step-specific info was recorded into a backup log onlyafter the step finished. However if there's exactly one step in progresswe can dump its progress into log as we go - which is exactly what 99.5implements.

5 Pre-release 80.99.6

⦁ Fixed an invalid self-consistency check in the logging module that

caused 99.5 to shut down with "Something happened" message.



This was caused by change #2 in 99.5. ⦁ Fixed an invalid self-consistency check in the logging module thatcaused 99.5 to shut down with "Something happened" message.This was caused by change #2 in 99.5.

6 Pre-release 80.99.7

⦁ Added support for working with live files. See here for details -

https://bvckup2.com/wip/17122019



This feature is configured with the following per-job override:



conf.scanning.src.requery_meta <file-name-pattern>



For example,



conf.scanning.src.requery_meta *.txt

conf.scanning.src.requery_meta 123*



will force the program to double-check size and timestamps for

files with .txt extension and files with names starting with 123.



See

an override.



⦁ Fixed a rare logging issue that caused the program to exit with the

"Something happened" message. ⦁ Added support for working with live files. See here for details -This feature is configured with the following per-job override:conf.scanning.src.requery_meta For example,conf.scanning.src.requery_meta *.txtconf.scanning.src.requery_meta 123*will force the program to double-check size and timestamps forfiles with .txt extension and files with names starting with 123.See https://bvckup2.com/support/forum/topic/1140 for how to addan override.⦁ Fixed a rare logging issue that caused the program to exit with the"Something happened" message.

7 Pre-release 80.99.8

⦁ Fixed another issue with handling a successful backup of a new file

after running into a retryable error. ⦁ Fixed another issue with handling a successful backup of a new fileafter running into a retryable error.

8 Pre-release 80.99.9

⦁ Fixed an issue with an incorrect logging level when failing to update

meta of a directory (the "Updating directory" operation)



⦁ + All changes from the mainstream 80.9 release

https://bvckup2.com/support/forum/topic/1206/6935 ⦁ Fixed an issue with an incorrect logging level when failing to updatemeta of a directory (the "Updating directory" operation)⦁ + All changes from the mainstream 80.9 release

9 Pre-release 80.99.11

⦁ Another pass over the retrying-on-errors logic. Apparently, a network

hiccup may not necessarily kill _all_ active operations. Sometimes it can

cause just some of them to error out, while the rest will keep on going.



⦁ + All changes from the mainstream 80.10 and .11 releases

https://bvckup2.com/support/forum/topic/1206/6941

https://bvckup2.com/support/forum/topic/1206/6962



There was no 80.99.10 release. ⦁ Another pass over the retrying-on-errors logic. Apparently, a networkhiccup may not necessarily kill _all_ active operations. Sometimes it cancause just some of them to error out, while the rest will keep on going.⦁ + All changes from the mainstream 80.10 and .11 releasesThere was no 80.99.10 release.

10 Pre-release 80.99.12

⦁ Added UI support for configuring multi-threaded backup execution.



New Processing section:

https://bvckup2.com/wip/r81-backup-settings-new.png



The same expanded:

https://bvckup2.com/wip/r81-exec-settings.png



Additional details:

https://bvckup2.com/wip/r81-exec-settings-details.png



⦁ Reworked all UI animations to use FPS-based sequencing. Previously,

animation used the "run for X ms using Y steps". X and Y were hand-

coded and because of that the frames-per-second value (FPS) was all

over the place. With this release there's now a single global FPS value

for all animations and another for all fade-ins and fade-outs.



⦁ Resolved (hopefully the last) issue with the retrying logic - there was a

race condition in the code that preemptively cancelled queued, but not

yet executed steps when the program ran into a retryable error. ⦁ Added UI support for configuring multi-threaded backup execution.New Processing section:The same expanded:Additional details:⦁ Reworked all UI animations to use FPS-based sequencing. Previously,animation used the "run for X ms using Y steps". X and Y were hand-coded and because of that the frames-per-second value (FPS) was allover the place. With this release there's now a single global FPS valuefor all animations and another for all fade-ins and fade-outs.⦁ Resolved (hopefully the last) issue with the retrying logic - there was arace condition in the code that preemptively cancelled queued, but notyet executed steps when the program ran into a retryable error.

11 Pre-release 80.99.13

⦁ Fixed an issue with program closing down when needing to trim the

$Archive. Trimming is also done in multi-threaded (parallel) fashion

now, but it was done _before_ the parallel execution was configured.

No config => no way to trim => whoops.



Kudos to Greg for the traces.



⦁ Fixed an issue with overly strict self-consistency check in the retrying

code. Had to do with retrying the backup of _new_ files. If this sort of

operation was queued and then cancelled due to other steps failing

(e.g. because the network connection went down), then in some cases

the program would panic by seeing a new-copy step in a retry state. ⦁ Fixed an issue with program closing down when needing to trim the$Archive. Trimming is also done in multi-threaded (parallel) fashionnow, but it was done _before_ the parallel execution was configured.No config => no way to trim => whoops.Kudos to Greg for the traces.⦁ Fixed an issue with overly strict self-consistency check in the retryingcode. Had to do with retrying the backup of _new_ files. If this sort ofoperation was queued and then cancelled due to other steps failing(e.g. because the network connection went down), then in some casesthe program would panic by seeing a new-copy step in a retry state.

12 Pre-release 80.99.14

⦁ Added an option of waking up computer to run scheduled backups.

https://bvckup2.com/wip/18042020



⦁ Added an option of password-protecting UI access in service mode.

https://bvckup2.com/wip/17042020



⦁ Internal changes to handle licensing-related cases due to upgrading

to a new major release (once this preview turns into R81).



⦁ Reworked Basic-to-Pro upgrade flow to be simpler - click on the link,

pay up, come back in the app, click on Refresh license. Done.



⦁ Slimmed down the uninstaller by 10K. Simply by _not_ using standard

library to convert integers into their hex string form.



⦁ Fixed uninstaller to remove all Task Scheduler auto-start entries (as

different versions created them in slightly different formats).



⦁ Fixed (hopefully the final) issue with retrying logic.



+ Any accumulated changes from recent R80 versions. ⦁ Added an option of waking up computer to run scheduled backups.⦁ Added an option of password-protecting UI access in service mode.⦁ Internal changes to handle licensing-related cases due to upgradingto a new major release (once this preview turns into R81).⦁ Reworked Basic-to-Pro upgrade flow to be simpler - click on the link,pay up, come back in the app, click on Refresh license. Done.⦁ Slimmed down the uninstaller by 10K. Simply by _not_ using standardlibrary to convert integers into their hex string form.⦁ Fixed uninstaller to remove all Task Scheduler auto-start entries (asdifferent versions created them in slightly different formats).⦁ Fixed (hopefully the final) issue with retrying logic.+ Any accumulated changes from recent R80 versions.

Topic is locked.