command: add pipe command (#182) - #587

I tested this in production for uploading a 500 GB file that was the output of tar and it seemed to work well! Just wanted to add my 2 cents here.

I did this by

1. Forking the branch and running go install github.com/wanderlog/s5cmd/[email protected] to install it
2. Running this command:

tar -cf - "$INPUT" | zstd -1 --stdout | \
  ~/go/bin/s5cmd --endpoint-url "$S3_ENDPOINT" --credentials-file "$S5CMD_CREDENTIALS_FILE" \
    pipe --part-size "$PART_SIZE_MB" "$S3_DESTINATION"

Verifying the command completed successfully

I think with refactoring cp command, we might able to use some of the code for stuff like common flags, content type guessing, override logic but not exactly sure if it worths, what do you think? @ahmethakanbesel @igungor

I think with refactoring cp command, we might able to use some of the code for stuff like common flags, content type guessing, override logic but not exactly sure if it worths, what do you think? @ahmethakanbesel @igungor

cp has the NewSharedFlags function, which defines 18 different common flags for cp and sync. pipe uses 13 of them. We can reduce the number of shared flags to use them all in cp, sync and pipe, but it can affect the user experience.

We can have a common ShouldOverride function. It may take a list of available flags or a parameter to determine called by cp or pipe.

@ahmethakanbesel Can you update the changelog please?

I updated the changelog.

@ahmethakanbesel There's a conflict on CHANGELOG.

🎖️

So cool, thank you all! I've been waiting for this!