Everyday commands

Text-processing tools at a glance: grep searches for matching strings, sed edits matched text, awk formats and manipulates fields.

vi

:$          # jump to the last line

Keyboard layout is wrong in vi

A key-mapping issue on the tiny default vimrc. Change set compatible to set nocompatible:

# Ubuntu 20.04 / 22.04
sudo sed -i 's/set compatible/set nocompatible/' /etc/vim/vimrc.tiny

# Ubuntu 18.04 and older
echo 'set nocompatible' >> ~/.vimrc

Listing and copying

ls -lh                 # human-readable sizes
ls -l --block-size=MB  # sizes in MB
ls -lrt DIR            # newest last

cp -R * /var/www/html/ # recurse into directories

find

find /var/log -iname '*.log' -type f          # files only, case-insensitive
find $HOME -type f -printf '%s %p\n' | sort -nr | head -10   # 10 largest files
find . -name "*.log" -printf '%s %p\n' | sort -nr | head -5

> msg.log               # truncate a file to zero bytes

curl

Option	Meaning
-i, --include	Include the response headers in the output
-s, --silent	No progress meter or error messages
-X, --request METHOD	Use a custom HTTP method (default GET)
-H, --header 'H: v'	Add a request header
-u, --user user:pass	HTTP authentication (prompts if password omitted)
-o FILE / -O	Save to FILE / to the remote filename
-C -	Resume a partial download
-d 'data'	Send a POST body

curl -o jquery.min.js https://cdn.jsdelivr.net/npm/jquery/dist/jquery.min.js
curl -O https://cdn.jsdelivr.net/npm/jquery/dist/jquery.min.js
curl -C - -O http://releases.ubuntu.com/21.04/ubuntu-21.04-desktop-amd64.iso
curl -d "user=foo&pass=12345" https://www.example.com/post.cgi

# Always append a newline to response bodies
echo '-w "\n"' >> ~/.curlrc

POST a JSON body

Build the body with a here-document, then send it:

BODY=$(cat << EOF
[
  {
    "controllerId": "${TARGET_ID}",
    "name": "QDevice",
    "securityToken": "${SECURITY_TOKEN}"
  }
]
EOF
)

curl -X POST "https://host/rest/v1/targets" \
  -u "${USERNAME}:${PASSWORD}" \
  -H 'Content-Type: application/json' \
  -d "$BODY" \
  -s | json_pp

Archives

# tar + gzip
tar -zcvf archive.tar.gz directory/     # create
tar -zxvf archive.tar.gz                # extract

# zip / 7z
sudo apt install zip p7zip-full
unzip file.zip -d target_dir
7z x package.7z

# gzip a single file
gzip file.txt

tree

sudo apt install tree
tree -C                # colourised
tree -f                # full paths
tree -L 2              # limit to two levels
tree -P '*.pl'         # only files matching a pattern
tree -F                # mark dirs with / and executables with *

dos2unix and rename

sudo apt install dos2unix rename
dos2unix script.sh     # convert CRLF line endings to LF

Shell snippets

# Default values
${VAR:-fallback}
var="${1:-2022}"       # $1, or 2022 if unset

# whoami vs $USER: whoami is the OS user, $USER is the login/ssh user
me=$(whoami)
me=$USER

# List every available command
compgen -c

# Show each command before running it (debug)
set -x

Example of extracting a version from a jar filename with awk:

current_ver=$(find build/libs -name '*.jar' ! -name '*plain.jar' | head -1 \
  | awk -F'[-]' '{if ($5=="SNAPSHOT.jar") print $4"-SNAPSHOT"; else print substr($4,1,length($4)-4)}')

find build/libs -name '*.jar' ! -name '*plain.jar' | xargs -I{} cp -f {} ../

Regular expressions

# Match: SET <anything> =
SET\s[\w\W]*\s=\s