Repository navigation
_block:xyz_ and _blocker_ to better manage command queue dependencies #52
Description
One of the problems with trying to automate a number of tasks is dependancies. The idea I have to work around this is _block:<name> and _blocker_.
This will work in the following manner:
| Argument | Description |
|---|---|
| _blocker_ | All commands before this must be completed before any other items can be added into the queue |
| _block:_ | All commands in a block are run and queued sequentially |
Take the following runfile, as an example:
_block:file-creation_
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
_block:file-creation_
_blocker_
nmap _target_ -oN _target_/output/scans/_target_-nmap
In this example, because the items in the _block_ are always loaded sequentially they shouldn't ever run out of order. Additionally, because we have a blocker before nmap, this item won't run until the block is finished.
Alternatively, we could have a command list like the following:
_block:nmap_
mkdir _target_
mkdir _target_/output
mkdir _target_/output/scans
nmap _target_ -oN _target_/output/scans/_target_-nmap
_block:nmap_
nikto --host _target_
In this example, the block would run the same as before, but assuming the thread count is high enough then nikto would begin to run immediately, passing results back to the terminal (whilst nmap and file creation happened in the background).
Activity
[-]_block:xyz_ and _blocker_ to manage command queues[/-][+]_block:xyz_ and _blocker_ to better manage command queues dependancies[/+][-]_block:xyz_ and _blocker_ to better manage command queues dependancies[/-][+]_block:xyz_ and _blocker_ to better manage command queue dependencies[/+]codingo commentedon Aug 13, 2019
Blocks can be nested since everything in a block is sequential anyway, nesting will make no difference. That said, a warning should be shown if a block is not closed. This shouldn't need to be an exception, but the user should be made aware since everything after the block would then run sequentially.
codingo commentedon Aug 13, 2019
In case it's not clear from the above as well, it's important that the user can add multiple blocks to a single job. The idea of this is that the threader will optimize by taking jobs off the command list file, but by having blocks you can have your sequential tasks within the same file (instead of making shell scripts that run outside of the command queue).
iamOgunyinka commentedon Aug 14, 2019
So I have an example I'd like clarity on. Suppose there are 8 tasks scheduled to run:
and should be run on
5 threads/workers. The default behavior of your implementation is to pick 5 tasks first(not sure if the order of appearance matters?) and on completion, run the other 3. I think that's OK since the tasks seem to be independent of one another.Now, enter the
_block:<name>_block(let's call itTaskBlock) and the sequence-before segment_blocker_. Given the 8 tasks above, if a_blocker_is indicated beforecopy foo bar/like so,_blocker_segment (starting withcopy foo bar/) must wait until the tasks above the_blocker_is completed?_blocker_segment is completed, does the execution of the tasks follow the sequential order or they can be multi-threaded again before the next blocker is encountered?TaskBlock? Guaranteed sequential or multi-threaded?codingo commentedon Aug 15, 2019
Gotcha, so the intended functionality is that once a
_blocker_is hit, queuing of those tasks will be halted until the blocker is resolved. Once it's resolved commands may queue again and do not need to be completed sequentially. Items before a_blocker_do not need to run sequentially, only items within a_block.Items within a taskblock are guaranteed to be sequential.
codingo commentedon Aug 15, 2019
Currently being tested in #55
Merge pull request #60 from iamOgunyinka/flat
codingo commentedon Aug 19, 2019
Closed with #60, bounties to be awarded to #59 and #60