In the fast-paced world of Linux systems like virtual machines and containers, shell scripting stands out as the go-to tool for making repetitive tasks easier. It's like having a shortcut for tasks you do often, so you don't have to type the same commands again and again. Instead, you create a script once, and then you can run it whenever you need to do that task. Whether you're managing servers, setting up software, or just making sure everything runs smoothly, getting the hang of shell scripting helps you work smarter.
Shell scripting is a type of programming that allows users to automate tasks on a Unix-based operating system.
Just like other program files have extensions like .exe or .py, shell scripts have their own too: .sh. If you're curious to see how shell scripting is used in real-world applications, you can check it out by clicking here: Click Me.
Shebang
That quirky "#!" you see at the start of some scripts, is like a secret code that tells your computer which interpreter to use when running the script. It's super handy because it means you can write scripts in different languages and still run them smoothly without any extra fuss.
These shebang examples indicate the path to different shell interpreters, such as Bash, Bourne shell (sh), Dash, and Korn shell (ksh), allowing the script to be executed using the specified interpreter.
And /bin/bash is the path where the interpreters is there.
#!/bin/bash
#!/bin/sh
#!/bin/dash
#!/bin/ksh
In many shell scripts, you might notice that the shebang line is either #!/bin/bash or #!/bin/sh, but the script ends up being executed by the Bash shell regardless. This happens because of internal linking in Linux. When you specify #!/bin/sh, it often gets linked to #!/bin/bash, which is executed by the Bash shell.
However, in recent years, some UNIX systems have started linking #!/bin/sh to #!/bin/dash instead of Bash. So, if you want your script to be executed by the Bash shell reliably, it's recommended to use #!/bin/bash as the shebang line. This ensures consistent behavior across different systems and versions.
In this blog, we'll continue with the bash executables or interpreter for the shell script as it's widely used.
Writing the shell script with Vim
Vim is a text editor program for Linux and other Unix-like operating systems.
It's a powerful tool used primarily by programmers, system administrators, and power users for editing text files directly in the terminal.
Vim stands for "Vi IMproved" and is an enhanced version of the older Vi text editor.
If you want to create a file in Linux you can do by various following ways :-
gedit <filename>.<extension>
touch <filename>.<extension>
echo "Hello Anurag!!" > a.txt
# output of the command which is in the left side of ">" will be assigned in a.txt
# if that file doesn't exit, a new file is created of that name and the content will save in it.
vim <filename>.<extension>
Creating file through vim
vim a.sh on executing this command the interface will look like this.
On clicking the insert key from the keyboard you will enter in the insert mode and now you can write the file in the term itself.
After writing in the file, press the escape key Esc, then you can do the following as you do in the other text editor like Notepad which are:-
To save the file on closing ->
:wqTo exit without saving ->
:q
This is the simple way you can create write and build the file in the terminal.
Cool, so now we created a shell file directly from the terminal.
If you want to execute it then give executable permission to it by chmod +x filename and give its path ./a.sh or sh a.sh and the following output you can see.
And if you want to edit something in the file or see the content you can again use the vim a.sh and you are good to go. If you only want to see the what is the content of the file you can also use cat a.sh.
Commands used to write shell script
As you know some of the quite common commands like cd, touch, man, mkdir, rmdir, rm, mv, cp, pwd, etc.
Here are some of the different commands that'll be quite helpful.
history
If you want to see the command you executed you can use it.
So no need to learn the command you'll learn on the go.
chmod
It'll be helpful when we have to give the permission to any file.
chmod +r-> + and then r/w/x means giving permission to the file.chmod -r-> - and then r/w/x means taking away permission from the file.chmod 471-> sets the file permissions to 471, which means the owner can read, write, and execute the file (4 + 2 + 1), the group can read (4), and others have executable permissions (1).4 -> first integer denotes the permission for the -> Root or Owner.
7 -> second integer denotes the permission for the -> Group.
1 -> third integer denotes the permission for the -> Others.
If you want to further break down how the 4 denoting the
read permission-> The 4 is denoted by three bits ->100
Note: if you want to Somment the command or line in the shell script you can use
#before it.
top
Purpose: Displays Linux processes in real-time.
Usage: Typically used to monitor system performance and resource usage.
Example:
topThis command opens an interactive display showing processes ordered by various criteria like CPU or memory usage.
grep
Purpose: Searches for patterns in input data.
Usage: Often used to filter output based on specified criteria.
Example:
ps aux | grep sshThis command lists all processes (
ps aux) and filters (grep ssh) for those containing "ssh".
sed
Purpose: Stream editor for filtering and transforming text.
Usage: Used to perform text substitutions, deletions, and more.
Example:
echo "Hello World" | sed 's/Hello/Hi/'This command replaces "Hello" with "Hi" in the output: "Hi World".
awk
Purpose: Text processing language for pattern scanning and processing.
Usage: Useful for data extraction and reporting.
Example:
ps aux | awk '{print $2, $11}'This command lists process IDs and their associated command names from
ps auxoutput.
free
Purpose: Displays system memory usage.
Usage: Shows total, used, and free memory.
Example:
free -mThis command displays memory statistics in megabytes (
-m).
df -h
Purpose: Displays disk space usage.
Usage: Shows filesystem usage in a human-readable format.
Example:
df -hThis command displays disk space usage across all mounted filesystems in a human-readable format (
-h).
Control commands
ifstatement: Used for conditional execution.if [ condition ]; then # commands elif [ condition ]; then # commands else # commands fiExample:
if [ $age -ge 18 ]; then echo "You are an adult." else echo "You are a minor." fiNote:-
-eq: Equal to-ne: Not equal to-lt: Less than-le: Less than or equal to-gt: Greater than-ge: Greater than or equal to
casestatement: Similar to switch-case in other languages, used for multi-way branching.case $variable in pattern1) # commands ;; pattern2) # commands ;; *) # default commands ;; esacExample:
case $1 in start) echo "Starting..." ;; stop) echo "Stopping..." ;; restart) echo "Restarting..." ;; *) echo "Usage: $0 {start|stop|restart}" ;; esacforloop: Iterates over a list of items.for var in list; do # commands doneExample:
for i in 1 2 3 4 5; do echo "Number: $i" donewhileloop: Repeats a block of commands as long as a condition is true.while [ condition ]; do # commands doneExample:
count=1 while [ $count -le 5 ]; do echo "Count: $count" count=$((count + 1)) doneuntilloop: Repeats a block of commands until a condition becomes true.until [ condition ]; do # commands doneExample:
count=1 until [ $count -gt 5 ]; do echo "Count: $count" count=$((count + 1)) doneselectloop: Provides a menu for selecting options.select var in list; do # commands doneExample:
select option in "Option 1" "Option 2" "Option 3"; do case $option in "Option 1") echo "You chose Option 1" ;; "Option 2") echo "You chose Option 2" ;; "Option 3") echo "You chose Option 3" ;; *) echo "Invalid option" ;; esac donebreakandcontinue: Control the flow within loops.break: Exit from the loop.continue: Skip the remaining part of the loop and start from the next iteration.
Example:
for i in {1..10}; do
if [ $i -eq 5 ]; then
break
fi
echo "Number: $i"
done
for i in {1..10}; do
if [ $i -eq 5 ]; then
continue
fi
echo "Number: $i"
done
These control commands and structures help create more complex and efficient Bash scripts by providing mechanisms to make decisions and repeat tasks.
Real-life example
If we want to make a shell script that will make a directory and inside the directory make 10 files inside it whose names must be "file 1", "file 2", ... respectively, and the content in each file should be "Hey I'm file 1", "hey I'm file 2", etc.
First of all, make a shell file by executing the following command.
vim ab.sh # shell file
Then press Insert key, and paste the below code in that. Then press Esc key and enter :wq! to save the file and exit.
#!/bin/bash
# Create a directory
mkdir my_directory
# Move into the directory
cd my_directory
# Loop to create 10 files
for i in {1..10}; do
# Create file with content
echo "Hey I'm file $i" > "file $i"
done
# List all the files
ls
# Move back to previous directory (optional)
cd ..
Then give the permission to execute the shell file ab.sh
chmod +x ab.sh
# Then execute the shell script
./ab.sh
Output:-
And that's it, whenever we have to make a folder that has 10 files inside it, having some content, then you can simply execute the shell.
Functions
Functions in shell scripting allow you to organize and reuse code segments, improving readability, maintainability, and modularity of your scripts. Here’s how you define and use functions in Bash (a common shell scripting language):
Defining Functions
In Bash, you define functions using the function keyword followed by the function name, parentheses (), and curly braces {} containing the function's code.
Or, more commonly without the function keyword:
function_name() {
# Function body
# Commands and logic go here
}
Example
Here's an example of a function that prints a greeting:
# Define a function named greet
greet() {
echo "Hello, welcome to Shell scripting!"
}
# Call the function
greet
Function Parameters
You can pass parameters to functions in Bash. Within the function, these parameters are accessed using positional parameters like $1, $2, etc.
# Define a function with parameters
greet_user() {
echo "Hello, $1! Welcome to Shell scripting!"
}
# Call the function with a parameter
greet_user "John"
Example with Return Value
In Bash, functions do not have a return statement like in other programming languages. Instead, the last command executed in the function determines the function's return status. You can capture output using command substitution.
# Define a function that calculates square of a number
square() {
echo $(($1 * $1))
}
# Call the function and capture the result
result=$(square 5)
echo "Square of 5 is: $result"
Scope
Variables defined inside a function are local to that function by default. To access global variables within a function, you need to declare them with the global keyword.
global_var="Global variable"
# Define a function accessing global variable
access_global() {
echo "Inside function: $global_var"
}
# Call the function
access_global
echo "Outside function: $global_var"
Why is Shell scripting needed?
Shell scripting plays a crucial role in DevOps by automating various tasks related to infrastructure management, code deployment, and configuration management. Here are some specific use cases:
Infrastructure Management:
Shell scripts help set up and manage the servers, databases, and other resources your applications need to run.
For example, they can automatically create new servers in the cloud, install software, and configure everything so your application works correctly.
Code Deployment:
They're handy for getting your code from your computer to where it needs to go, like a website or an app.
For instance, they can build your code, run tests to make sure it works, and then put it live on a server.
Configuration:
Shell scripts make sure all the settings for your applications are correct and consistent.
They can set up things like what database your app should use, how it should talk to other services, and where it should keep its logs.
Real-world example
Imagine you're a DevOps engineer entrusted with managing a massive fleet of 10,000 Linux-based virtual machines (VMs) in your company's infrastructure. Ensuring the health and optimal performance of these VMs is paramount for smooth operations.
One day, a developer reports issues with certain VMs - some are unresponsive, others are exhibiting high memory usage, and some are plagued by sluggish processes.
To efficiently handle this, you decide to automate the health monitoring process. You swiftly write a robust shell script designed to diagnose and resolve common VM issues. This script not only detects anomalies like unresponsiveness, excessive memory consumption, or sluggish processes but also takes remedial actions as necessary. After thoroughly testing it, you push the script to the company's Git repository, making it accessible to your team.
To ensure continuous monitoring and resilience, you configure the script to execute periodically on all VMs. This scheduled task runs at regular intervals, meticulously scanning each VM for any signs of trouble. Upon detection, the script promptly intervenes, executing predefined commands to diagnose and rectify the issues.
This following shell script can be used to monitor the health of the VM, and on assigning it to the crone job it will do the monitor regularly at specific time.
#!/bin/bash
##################################
# Author: Anurag Kumar
# Date: 27-06-2024
#
# Description: Bash script to monitor VM health and log results.
##################################
# Function to check VM health
check_vm_health() {
# Example commands to monitor VM health
echo "[$(date +'%Y-%m-%d %H:%M:%S')] Starting VM health check..."
# Check CPU usage
cpu_usage=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
echo "[$(date +'%Y-%m-%d %H:%M:%S')] CPU Usage: ${cpu_usage}%"
# Check memory usage
mem_usage=$(free | awk '/Mem/{printf("%.2f"), $3/$2*100}')
echo "[$(date +'%Y-%m-%d %H:%M:%S')] Memory Usage: ${mem_usage}%"
# Check disk usage
disk_usage=$(df -h | awk '$NF=="/"{printf "%s", $5}')
echo "[$(date +'%Y-%m-%d %H:%M:%S')] Disk Usage: ${disk_usage}"
# Add more checks as needed: network status, process status, etc.
echo "[$(date +'%Y-%m-%d %H:%M:%S')] VM health check completed."
}
# Log file path
log_file="/var/log/vm_health.log"
# Redirect all output to log file
exec >> $log_file 2>&1
# Main function to initiate VM health check
main() {
echo "[$(date +'%Y-%m-%d %H:%M:%S')] Initiating VM health monitoring..."
check_vm_health
echo "[$(date +'%Y-%m-%d %H:%M:%S')] VM health monitoring completed."
}
# Execute main function
main
Explanation:
Script Overview:
This script defines a function
check_vm_health()to monitor VM metrics such as CPU usage, memory usage, and disk usage.The script redirects all output to a log file (
/var/log/vm_health.log) to maintain a record of monitoring results.
Cron Job Setup:
- You can schedule this script to run regularly by, adding a cron job entry. This cron job runs the script
/path/to/your/script.shafter every specific time.
- You can schedule this script to run regularly by, adding a cron job entry. This cron job runs the script
Customization:
Customize the
check_vm_health()function to include additional health checks specific to your VM environment (e.g., network status, specific processes, etc.).Adjust the log file path (
log_file) as per your system's requirements.
Logging:
- The script uses
exec >> $log_file 2>&1to redirect both standard output and error output to the specified log file (/var/log/vm_health.log).
- The script uses
Thanks to this proactive approach, your team can swiftly identify and address VM problems before they escalate, ensuring uninterrupted service and optimal performance across the entire infrastructure.
This is one example, that tells how important the shell script is, for the DevOps engineer, to make our application a perfect running environment.
Script for Github
There are lots of scripts that you can write in order to interact with the application, with the help of the API provided by the Application like Github API docs, https://docs.github.com/en
Example
Like this following script to list all the collabratore of the organization.
#!/bin/bash
# GitHub API URL
API_URL="https://api.github.com"
# GitHub username and personal access token
USERNAME=$username
TOKEN=$token
# User and Repository information
REPO_OWNER=$1
REPO_NAME=$2
# Function to make a GET request to the GitHub API
function github_api_get {
local endpoint="$1"
local url="${API_URL}/${endpoint}"
# Send a GET request to the GitHub API with authentication
curl -s -u "${USERNAME}:${TOKEN}" "$url"
}
# Function to list users with read access to the repository
function list_users_with_read_access {
local endpoint="repos/${REPO_OWNER}/${REPO_NAME}/collaborators"
# Fetch the list of collaborators on the repository
collaborators="$(github_api_get "$endpoint" | jq -r '.[] | select(.permissions.pull == true) | .login')"
# Display the list of collaborators with read access
if [[ -z "$collaborators" ]]; then
echo "No users with read access found for ${REPO_OWNER}/${REPO_NAME}."
else
echo "Users with read access to ${REPO_OWNER}/${REPO_NAME}:"
echo "$collaborators"
fi
}
# Main script
echo "Listing users with read access to ${REPO_OWNER}/${REPO_NAME}..."
list_users_with_read_access
In order to execute this you have to first of all export the Github username and Access Token.
export username="your-username"
export token="provide-github-token"
TO make the github token of your to access the github application follow below steps :-
https://github.com/settings/apps > Personal access tokens > Tokens (classic)
Generate new token > Generate new token (classic) > Provide the token name > Select which of the Access you want to give to the token > Generate Token.
Take the token from here and export it.
To execute shell script, use the relative path of shell script file followed by the command line arguments. The first argument should be the organization name, and the second argument should be the repository name within the organization.
For example:
relative/path/to/script OrganizationName RepositoryName
In the shell script, you can access these arguments using $1 and $2. Here, $1 represents the organization name, and $2 represents the repository name.
The output you can see.
Thank you for reading this blog. It covers a wide range of topics, and I've made every effort to simplify each for easy understanding.
If you like this blog, please do like it.
Check out my Portfolio website for connecting with me or just to say Hi !!.