Unix - Environment variable

Unix - File System Basics

Linux file hierarchy System

file system is a logical collection of files on a partition or disk. A partition is a container for information and can span an entire hard drive if desired.
Your hard drive can have various partitions which usually contains only one file system, such as one file system housing the / file system or another containing the /home file system.
One file system per partition allows for the logical maintenance and management of differing file systems.
Everything in Unix is considered to be a file, including physical devices such as DVD-ROMs, USB devices, floppy drives, and so forth.

Directory Structure:

Unix uses a hierarchical file system structure, much like an upside-down tree, with root (/) at the base of the file system and all other directories spreading from there.
A UNIX filesystem is a collection of files and directories that has the following properties:

• It has a root directory (/) that contains other files and directories.
• Each file or directory is uniquely identified by its name, the directory in which it resides, and a unique identifier, typically called an inode.
• By convention, the root directory has an inode number of 2 and the lost+found directory has an inode number of 3. Inode numbers 0 and 1 are not used. File inode numbers can be seen by specifying the -i option to ls command.
• It is self contained. There are no dependencies between one filesystem and any other.

The directories have specific purposes and generally hold the same types of information for easily locating files. Following are the directories that exist on the major versions of Unix:

Directory	Description
/	This is the root directory which should contain only the directories needed at the top level of the file structure.
/bin	This is where the executable files are located. They are available to all user.
/dev	These are device drivers.
/etc	Supervisor directory commands, configuration files, disk configuration files, valid user lists, groups, ethernet, hosts, where to send critical messages.
/lib	Contains shared library files and sometimes other kernel-related files.
/boot	Contains files for booting the system.
/home	Contains the home directory for users and other accounts.
/mnt	Used to mount other temporary file systems, such as cdrom and floppy for the CD-ROM drive and floppy diskette drive, respectively
/proc	Contains all processes marked as a file by process number or other information that is dynamic to the system.
/tmp	Holds temporary files used between system boots
/usr	Used for miscellaneous purposes, or can be used by many users. Includes administrative commands, shared files, library files, and others
/var	Typically contains variable-length files such as log and print files and any other type of file that may contain a variable amount of data
/sbin	Contains binary (executable) files, usually for system administration. For example fdisk and ifconfig utlities.
/kernel	Contains kernel files

Navigating the File System:

Now that you understand the basics of the file system, you can begin navigating to the files you need. The following are commands you'll use to navigate the system:

Command	Description
cat filename	Displays a filename.
cd dirname	Moves you to the directory identified.
cp file1 file2	Copies one file/directory to specified location.
file filename	Identifies the file type (binary, text, etc).
find filename dir	Finds a file/directory.
head filename	Shows the beginning of a file.
less filename	Browses through a file from end or beginning.
ls dirname	Shows the contents of the directory specified.
mkdir dirname	Creates the specified directory.
more filename	Browses through a file from beginning to end.
mv file1 file2	Moves the location of or renames a file/directory.
pwd	Shows the current directory the user is in.
rm filename	Removes a file.
rmdir dirname	Removes a directory.
tail filename	Shows the end of a file.
touch filename	Creates a blank file or modifies an existing file.s attributes.
whereis filename	Shows the location of a file.
which filename	Shows the location of a file if it is in your PATH.

Unix - Shell Useful Resources

If you want to list down your website, book or any other resource on this page then please contact at webmaster@tutorialspoint.com

• Bell Labs - The Creation of the UNIX Operating System. Gives overview and history of UNIX operating system.
• BSD UNIX - FreeBSD is an advanced UNIX operating system for modern server, desktop, and embedded computer platforms.
• Linux Online - Linux is a free Unix-type operating system originally created by Linus Torvalds with the assistance of developers around the world.
• Unix @ Wikipedia - A brief description of Unix Operating system.
• The Unix Forums - A forum for Unix lovers. Share your ideas and thoughts with other Unix experts.

Useful Books on Unix

 

 

Regular expressions In BASH

Regulat expression in Linux Part-1

Regular expressions (Regexp)is one of the advanced concept we require to write efficient shell scripts and for effective system administration.
Basically regular expressions are divided in to 3 types for better understanding.

1)Basic Regular expressions
2)Interval Regular expressions (Use option -E for grep and -r for sed)
3)Extended Regular expressions (Use option -E for grep and -r for sed)


What is a Regular expression?
A regular expression is a concept of matching a pattern in a given string.
Which commands/programming languages support regular expressions?
vi, tr, rename, grep, sed, awk, perl, python etc.

Basic Regular Expressions

Basic regular expressions: This set includes very basic set of regular expressions which do not require any options to execute. This set of regular expressions are developed long time back.

^                  –Caret/Power symbol to match a starting at the beginning of line.

$                   –To match end of the line

*                  –0 or more occurrence of previous character.

.                   –To match any character

[]                  –Range of character

[^char]         –negate of occurrence of a character set

<word>        –Actual word finding

–                    -Escape character

Lets start with our Regexp with examples, so that we can understand it better.

^ Regular Expression

Example 1: Find all the files in a given directory
ls -l | grep ^-
As you are aware that the first character in ls -l output, - is for regular files and d for directories in a given folder. Let us see what ^- indicates. The ^ symbol is for matching line starting, ^- indicates what ever lines starts with -, just display them. Which indicates a regular file in Linux/Unix.
If we want to find all the directories in a folder use grep ^d option along ls -l as shown below
ls -l | grep ^d
How about character files and block files?
ls -l | grep ^c
ls -l | grep ^b
We can even find the lines which are commented using ^ operator with below example
grep ‘^#’ filename
How about finding lines in a file which starts with ‘abc’
grep ‘^abc’ filename
We can have number of examples with this ^ option.

$ Regular Expression

Example 2: Match all the files which ends with sh
ls -l | grep sh$
As $ indicates end of the line, the above command will list all the files whose names end with sh.
how about finding lines in a file which ends with dead
grep ‘dead$’ filename
How about finding empty lines in a file?
grep ‘^$’ filename

 * Regular Expression

Example 3: Match all files which have a word twt, twet, tweet etc in the file name.
ls -l | grep ‘twe*t’
How about searching for apple word which was spelled wrong in a given file where apple is misspelled as ale, aple, appple, apppple, apppppple etc. To find all patterns
grep ‘ap*le’ filename
Readers should observe that the above pattern will match even ale word as * indicates 0 or more of previous character occurrence.

. Regular Expression

Example 4: Filter a file which contains any single character between t and t in a file name.
ls -l | grep ‘t.t’
Here . will match any single character. It can match tat, t3t, t.t, t&t etc any single character between t and t letters.
How about finding all the file names which starts with a and end with x using regular expressions?
ls -l | grep ‘a.*x’
The above .* indicates any number of characters
Note: .* in this combination . indicates any character and it repeated(*) 0 or more number of times.
Suppose you have files as..
awx
awex
aweex
awasdfx
a35dfetrx
etc.. it will find all the files/folders which start with a and ends with x in our example.

[] Square braces/Brackets Regular Expression

Example 5: Find all the files which contains a number in the file name between a and x
ls -l | grep ‘a[0-9]x’
This will find all the files which is
a0xsdf
asda1xsdfas
..
..
asdfdsara9xsdf
etc.
So where ever it finds a number it will try to match that number.
Some of the range operator examples for  you.
[a-z] –Match’s any single char between a to z. [A-Z] –Match’s any single char between a to z. [0-9] –Match’s any single char between 0 to 9. [a-zA-Z0-9] – Match’s any single character either a to z or A to Z or 0 to 9 [!@#$%^] — Match’s any ! or @ or # or $ or % or ^ character. You just have to think what you want match and keep those character in the braces/Brackets.

[^char] Regular Expression

Example6: Match all the file names except a or b or c in its filenames
ls | grep  ’[^abc]‘
This will give output all the file names except files which contain a or b or c.

<word> Regular expression

Example7: Search for a word abc, for example I should not get abcxyz or readabc in my output.
grep ‘<abc>’ filename

Escape Regular Expression 

Example 8: Find files which contain [ in its name, as [ is a special charter we have to escape it
grep "[" filename
or
grep '[[]‘ filename
Note: If you observe [] is used to negate the meaning of [ regular expressions, so if you want to find any specail char keep them in [] so that it will not be treated as special char.
Note: No need to use -E to use these regular expressions with grep. We have egrep and fgrep which are equal to “grep -E”. I suggest you just concentrate on grep to complete your work, don’t go for other commands if grep is there to resolve your issues.

Regular Expression in Linux Part- 2

This is our second part on Regular Expressions in Linux.

Interval Regular expressions

These are used to mention no of character/character set reputation info. Note that interval regular expression and extended reg require -E option with  grep.
Note: In order to use this set of regular expressions you have to us -E with grep command and -r option with sed commands

{n} –n occurrence of previous character
{n,m} – n to m times occurrence of previous character
{m, } –m or more occurrence of previous character.
Example 1: Find all the file names which contain “t” and  t repeats for 3 times consecutively.
ls -l | grep -E ‘t{3}’
-E option is used to extend regexp understanding for grep.
Example 2: Find all the file names which contain l letter in filename with 1 occurrence to 3 occurrence consecutively.
ls -l | grep -E ‘l{1,3}’
Example 3: Find all the file names which contains k letter 5 and more in a file name.

ls -l | grep -E 'k{5,}'

This is bit tricky, let me explain this. Actually we given a range i.e 5 to infinity(Just given only comma after 5).

Extended regular expressions

These regular expressions extend the regular expression features.
Note:In order to use this set of regular expressions you have to us -E with grep command and -r option with sed commands
+ –one more occurrence of previous character
| — or option, we can specify either a character to present in the pattern.
? — 0 or one occurrence of previous character
() — grouping of character set.
Example 1: Find all the files which contains f letter, one more occurrences.
ls -l | grep -E ‘f+’
Example 2: Find all the files which may contain a or b in its file name
ls -l | grep -E ‘a|b’
Example 3: Find all the files which may contain t or 1 occurrence of t in filename
ls -l | grep -E ‘t?’
for example i have below files
test
best
see
do
my grep command will list test, best files as output.
Note: My grep output contain all these files though see and do files do not contain t, the is because we given ? which will search for 0 or 1 occurrence of previous character. See and do contains 0 t’s in its name, so it will find these files too.
Example 4: Find all the files which contains ab in the sequence
ls -l | grep -E ‘(ab)’
This will give all the files which contains ab in the file name consequently.
Please stay tuned to our next article on grep command and how to use it.

Grep command with Regular expressions -Part-3

n this post we will see how to use extended regular expressions to increase the power of grep command even better than Basic regular expression.

Extended regular expressions:

+               --Match one or more occurrences of previous character.

|             -- Match Either character

?             – Match 0 or 1 occurrence of previous character.

()            –match a group of characters

{number}      –Match number of occurrence of a character

{1, 3}       –Match a character which is 1 to 3 times repetition

{5, }       –Match a repeated character which is repeated 5 or more times.

Note1: In order to use this extended regular expressions we have to use –E option give grep the capability to understand Extended regular expressions.

Note2: egrep is nothing but grep –E, so try to avoid it if grep itself can do the work for you. Why to learn new command?

Examples:

Example1:Search for a words which contains one or more occurrence of ‘b’ between a and c.

grep –E ‘ab+c’ filename

Example2: Search for a word which contains zero or one occurrence of b between a and c

grep –E ‘ab?c’ filename

Example3: Search for a word which contains either a or b, a and b between d, e characters

grep –E ‘da|be’ filename 

Example4: Search for a word which contains either a or b, but not both a and b between d, e characters

grep –E ‘d(a|b)e’ filename 

Example5: Search for a word which contains only 2 ‘b’ between a and c character

grep –E ‘ab{2}c’ filename

Example6: Search for a word which contains 3 to 4 ‘b’ between a and c character

grep –E ‘ab{2,4}c’ filename

Example7: Search for a word which contains 3 or more ‘b’ between a and c character

grep –E ‘ab{3, }c’ filename 

Note: When we are using {} we have to give a range, but in this example we did not give range we just started the range but did not end the range which indicates infinity.


* ************************  enjoy ***************

Shell Manpage Help

All the Unix commands come with a number of optional and mandatory options. It is very common to forget complete syntax of these commands.
Because no one can possibly remember every Unix command and all its options, there has been online help available since Unix's earliest days.
Unix's version of help files are called man pages. If you know any command name but you do not know how to use it, then Man Pages are here to help you at every step.

Syntax:

Here is the simple command to get the detail of any Unix command while working with the system:

$man command

$man  ls

$man  cp  

Example:

Now you imagine any command for which you want to get help. Assuming you want to know about pwd then you simply need to use the following command:

$man pwd

The above command would open a help for you which would give you complete information about pwd command. Try it yourself at your command prompt to get more detail on
You can get complete detail on man command itself using the following command:

$man man

Man Page Sections:

Man pages are generally divided into sections, which generally vary by the man page author's preference. Here are some of the more common sections:

Section	Description
NAME	Name of the command
SYNOPSIS	General usage parameters of the command.
DESCRIPTION	Generally describes of the command and what it does
OPTIONS	Describes all the arguments or options to the command
SEE ALSO	Lists other commands that are directly related to the command in the man page or closely resembling its functionality.
BUGS	Explains any known issues or bugs that exist with the command or its output
EXAMPLES	Common usage examples that give the reader an idea of how the command can be used.
AUTHORS	The author of the man page/command.

So finally, I would say that man pages are a vital resource and the first avenue of research when you need information about commands or files in a Unix system.

ftp server in rhel6

What is FTP ?

• ftp server is used to transfer files between server and clients.
•  All major operating system supports ftp.
•  ftp is the most used protocol over internet to transfer files. Like most Internet operations, FTP works on a client/ server model. 
• FTP client programs can enable users to transfer files to and from a remote system running an FTP server program.
•  A user on the remote system has to log in to an account on a server and can then transfer files to and from that account's directories only.
• A special kind of user account, named ftp, allows any user to log in to it with the username “anonymous.”
• This account has its own set of directories and files that are considered public, available to anyone on the network who wants to download them.

Service Profile:-
Package-   vsftpd     (very secure file transfer protocol daemon)
daemon-  vsftpd
configuration file-
1-   /etc/vsftpd/vsftpd.conf
2-  /etc/vsftpd/ftpusers  (Contains users list to deny permanently)
3- /etc/vsftpd/user_list   (Contain users list to allow or deny)
port number-
20  (data transfer)
21  (control connection)
log file-
/var/log/xferlog
/var/log/vsftpd.log
DocumentRoot-  /var/ftp/

Note-
/var/ftp/pub is the ftp path for anonymous user to upload and download. 

Install Vsftpd FTP Server

#yum install vsftpd*  -y

#service vsftpd restart
#chkconfig vsftpd on
#netstat -tunlp | grep vsftpd
#mkdir  -p /var/ftp/pub/upload
#mkdir -p /var/ftp/pub/download
#cd /var/ftp/pub/download
#touch   a1 a2 a3 a4 a5
#ls
#chmod  -R  777 /var/ftp/pub
#service iptables stop
#chkconfig iptables off

Note- 
*Bydefault download operation is allow for all ftp and anonymous users in ftp server configuration file.
*You can use command line and gui tool for download any files from ftp server.

Ftp client tools:
1- ftp
2-lftp
3- wget
4- firefox
5- filezila


How to Accessing the FTP Server  from Linux Client machine - for GET operation test

On client machine-

#rpm -qa ftp*
#yum install ftp*  -y
#ftp 192.168.0.1

Open the configuration file, edit like this :

# vi /etc/vsftpd/vsftpd.conf

FTP Server Configuration – For  Anonymous Access 
 *Generally, the anonymous user has no permission to upload file in ftp server.
 *To provide  upload in ftp server follow the steps:
*Append Thease all lines :-

anonymous_enable=YES
anon_upload_enable=YES
anon_mkdir_write_enable=YES
anon_root=/var/ftp/pub    
ftpd_banner=Welcome to KR Network Cloud FTP server. 

FTP Server Configuration- For ftp users only

local_enable=NO                        (Restrict Access to Anonymous User Only)
write_enable=NO                          (Disable  FTP upload)

FTP Server Configuration-  ftp limit settings

local_max_rate=50
max_per_ip=5
max_clients=10
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_file=/var/log/vsftpd.log
xferlog_std_format=YES
idle_session_timeout=600
data_connection_timeout=120
listen=YES         (for ipv4)
#listen_ipv6=YES
pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES

FTP server Configuration- limits users their home directory only

chroot_local_user=YES      

:wq
save and exit

#service vsftp restart
#chkconfig vsftpd on

for troubleshoot Selinux Error before upload any files from client side..

#chcon -R -t public_content_rw_t  /var/ftp/pub
#ls -lZ /var/ftp/pub
#getsebool -a | grep ftp
#setsebool -P  allow_ftpd_anon_write=1
#setsebool -P allow_ftpd_full_access=1
#setsebool -P ftp_home_dir=1
#getsebool  -a | grep ftp

FTP Server Configuration- Other required Settings.

FTP Command

Command	Description
ascii	Set ascii transfer type
bell	Beep when command completed
binary	Set binary transfer type
bye	Terminate ftp session and exit
cd	Change remote working directory
cdup	Change remote working directory to parent directory
chmod	Change file permission of remote file
dir	List content of remote directory
delete	Delete remote file
get	Recive file from remote server
image	Set binary transfer type
lcd	Change local working directory
ls  -a	List content of remote directory
mdelete	Delete multiple file
mdir	Make directory on the remote machine
mget	Get multiple file
mkdir	Make directory on the remote machine
mls	List content of multiple directory
mput	Send multiple file
pwd	Print working directory on remote machine
quit	Terminate ftp session and exit
rmdir	Remove directory on the remote machine
rename	Rename file
rstatus	show status of remote machine
type	set file transfer type

diff & ac Command Example

Diff Command Examples

diff command compares two different files and reports the difference. The output is very cryptic and not straight forward to read.

Syntax: 

#diff [options] file1 file2

option  :

-w in the diff command will ignore the white space while performing the comparison. 

[root@desktop17 ~]# cat abc1
network cloud
kr network cloud
hello
[root@desktop17 ~]# cat abc2
network cloud
hi
kr network cloud
hello
[root@desktop17 ~]# diff abc1 abc2
1a2
> hi

[root@desktop17 ~]#  

Duplicate File Changes Patch

Diff and Patch command:- 

* diff output stored in a file is called a "patchfile"

 use  -u for "unified" diff, best in patchfiles

* patch duplicates changes in other files (user with care!)

  use -b to automatically backup changed files.

[root@desktop17 ~]# cat abc1
network cloud

hi 

kr network cloud
hello

[root@desktop17 ~]#cp abc1 abc2
[root@desktop17 ~]# cat abc2
network cloud

hi 

kr network cloud
hello

[root@desktop17 ~]# vim abc2

edit some thing new.

:wq 

[root@desktop17 ~]# diff -u abc1 abc2 
--- abc1    2013-06-14 15:28:46.845938983 +0530
+++ abc2    2013-06-14 15:28:33.439937131 +0530
@@ -1,4 +1,3 @@
 network cloud
-hi
 network cloud
 hello
[root@desktop17 ~]# diff -u abc1 abc2  > /tmp/abc.patch
[root@desktop17 ~]# 
[root@desktop17 ~]# 
[root@desktop17 ~]# patch  -b abc1 /tmp/abc.patch 
patching file abc1
[root@desktop17 ~]# cat abc1
network cloud

hi
kr network cloud
hello
 

 Means Confirm thathave been applied by running diff again.

 You should see no output, indicating that the files are now identical.

Display total connect time of users

Ac command will display the statistics about the user’s connect time.

Connect time for the current logged in user

With the option –d, it will break down the output for the individual days. In this example, I’ve been logged in to the system for more than 6 hours today. On Dec 1st, I was logged in for about 1 hour.

# ac –d

Dec  1  total        1.08
Dec  2  total        0.99
Dec  3  total        3.39
Dec  4  total        4.50
Today   total        6.10

Connect time for all the users

To display connect time for all the users use –p as shown below. Please note that this indicates the cumulative connect time for the individual users.

# ac -p 

        harry                               3.64
        madison                             0.06
        sanjay                              88.17
        nitesh                              105.92
        rakesh                              111.42
        total 309.21

Connect time for a specific user

To get a connect time report for a specific user, execute the following:

# ac -d nitesh

Jul  2  total       12.85
Aug 25  total        5.05
Sep  3  total        1.03
Sep  4  total        5.37
Dec 24  total        8.15
Dec 29  total        1.42
Today   total        2.95