Bash Script to Read 2 Files and Write the Uniqe Values to a New File

This department presents several shell script examples.

Hello Globe

Example ix. Hello World

              #!/bin/sh echo "Hello earth"            

Using Arguments

Example 10. Shell Script Arguments

              #!/bin/bash  # example of using arguments to a script repeat "My commencement proper noun is $i" echo "My surname is $2" repeat "Total number of arguments is $#"            

Save this file equally name.sh, gear up execute permission on that file past typing chmod a+ten proper name.sh then execute the file similar this: ./proper name.sh.

            $ chmod a+x proper name.sh $ ./name.sh Hans-Wolfgang Loidl My first name is Hans-Wolfgang My surname is Loidl Total number of arguments is 2          

Version 1: Line count case

The first example only counts the number of lines in an input file. It does so past iterating over all lines of a file using a while loop, performing a read operation in the loop header. While there is a line to procedure, the loop body will be executed in this case only increasing a counter by ((counter++)). Additionally the current line is written into a file, whose name is specified past the variable file, by echoing the value of the variable line and redirecting the standard output of the variable to $file. the electric current line to file. The latter is not needed for the line count, of course, only demonstrates how to cheque for success of an operation: the special variable $? volition contain the return lawmaking from the previous command (the redirected echo). By Unix convention, success is indicated by a render code of 0, all other values are fault code with application specific pregnant.

Another important issue to consider is that the integer variable, over which iteration is performed should ever count downwardly so that the analysis can find a bound. This might require some restructuring of the code, as in the following example, where an explicit counter z is introduced for this purpose. Subsequently the loop, the line count and the contents of the concluding line are printed, using echo. Of course, there is a Linux command that already implements line-count functionality: wc (for give-and-take-count) prints, when called with option -l, the number of lines in the file. We use this to check wether our line count is right, demonstrating numeric operations on the mode.

            #!/bin/bash # Simple line count instance, using bash # # Bash tutorial: http://linuxconfig.org/Bash_scripting_Tutorial#8-2-read-file-into-fustigate-array # My scripting link: http://www.macs.hw.ac.uk/~hwloidl/docs/index.html#scripting # # Usage: ./line_count.sh file # -----------------------------------------------------------------------------  # Link filedescriptor 10 with stdin exec 10<&0 # stdin replaced with a file supplied as a first argument exec < $ane # remember the name of the input file in=$1  # init file="current_line.txt" permit count=0  # this while loop iterates over all lines of the file while read LINE practice     # increase line counter      ((count++))     # write electric current line to a tmp file with proper name $file (not needed for counting)     repeat $LINE > $file     # this checks the return code of echo (not needed for writing; simply for demo)     if [ $? -ne 0 ]       then echo "Error in writing to file ${file}; check its permissions!"     fi done  repeat "Number of lines: $count" echo "The concluding line of the file is: `cat ${file}`"  # Note: Y'all can accomplish the same past simply using the tool wc like this echo "Expected number of lines: `wc -50 $in`"  # restore stdin from filedescriptor 10 # and shut filedescriptor 10 exec 0<&10 x<&-          

As documented at the start of the script, it is called like this (you must have a file text_file.txt in your current directory):

            $            ./line_count.sh text_file.txt          

	Sample text file
	You can get a sizable sample text file past typing: $ cp /dwelling/msc/public/LinuxIntro/WaD.txt text_file.txt

Several versions of line counting across a set of files

This section develops several shell scripts, each counting the total number of lines across a set of files. These examples elaborate specific beat out features. For counting the number of lines in ane file we use wc -l. Every bit a simple exercise you can supercede this control with a phone call to the line counting script in a higher place.

Version 1: Explicit For loop

Nosotros use a for-loop to iterate over all files provided as arguments to the script. We can admission all arguments through the variable $*. The sed command matches the line count, and replaces the entire line with just the line count, using the back reference to the kickoff substring (\1). In the for-loop, the shell variable n is a counter for the number of files, and s is the total line count and so far.

              #!/bin/fustigate # Counting the number of lines in a listing of files # for loop over arguments  if [ $# -lt 1 ] then   echo "Usage: $0 file ..."   go out 1 fi  echo "$0 counts the lines of code"  l=0 n=0 s=0 for f in $* practise 	l=`wc -l $f | sed 's/^\([0-9]*\).*$/\1/'` 	echo "$f: $l"         north=$[ $north + 1 ]         southward=$[ $s + $l ] done  repeat "$n files in total, with $s lines in total"            

Version 2: Using a Shell Part

In this version we define a function count_lines that counts the number of lines in the file provided as argument. Inside the function the value of the argument is retrieved by accessing the variable $1.

              #!/bin/fustigate # Counting the number of lines in a list of files # function version  count_lines () {   local f=$1     # this is the return value, i.e. non local   l=`wc -l $f | sed 's/^\([0-9]*\).*$/\ane/'` }  if [ $# -lt i ] then   repeat "Usage: $0 file ..."   get out i fi  repeat "$0 counts the lines of lawmaking"  l=0 northward=0 southward=0 while [ "$*" != ""  ] practise         count_lines $i         echo "$1: $l"         n=$[ $n + i ]         s=$[ $southward + $l ] 	shift done  echo "$n files in full, with $south lines in full"            

Version three: Using a return lawmaking in a role

This version tries to use the return value of the role to return the line count. Still, this fails on files with more than 255 lines. The return value is intended to just provide a return code, e.k. 0 for success, 1 for failure, but not for returning proper values.

              #!/bin/bash # Counting the number of lines in a list of files # function version using return code # WRONG version: the return code is limited to 0-255 #  so this script will run, simply print wrong values for #  files with more 255 lines  count_lines () {   local f=$one     local m   1000=`wc -l $f | sed 's/^\([0-9]*\).*$/\1/'`   return $m }  if [ $# -lt one ] so   repeat "Usage: $0 file ..."   exit 1 fi  echo "$0 counts the lines of lawmaking"  l=0 n=0 due south=0 while [ "$*" != ""  ] practice         count_lines $1 	fifty=$?         repeat "$i: $50"         n=$[ $due north + 1 ]         s=$[ $s + $fifty ] 	shift done  echo "$due north files in full, with $s lines in total"            

Version 4: Generating the file list in a shell function

              #!/bin/fustigate # Counting the number of lines in a list of files # function version  # office storing list of all files in variable files get_files () {   files="`ls *.[ch]`" }  # function counting the number of lines in a file count_lines () {   local f=$1  # 1st statement is filename   50=`wc -l $f | sed 'southward/^\([0-nine]*\).*$/\1/'` # number of lines }  # the script should be called without arguments if [ $# -ge 1 ] then   echo "Usage: $0 "   exit 1 fi  # split by newline IFS=$'\012'  echo "$0 counts the lines of code"  # don't forget to initialise! l=0 n=0 s=0 # call a function to become a listing of files get_files # iterate over this list for f in $files do         # call a function to count the lines         count_lines $f         echo "$f: $fifty" 	# increase counter         northward=$[ $n + 1 ] 	# increase sum of all lines         s=$[ $s + $50 ] done  repeat "$northward files in total, with $s lines in total"            

Version 5: Using an array to shop all line counts

The example below uses crush arrays to store all filenames (file) and its number of lines (line). The elements in an array are referred to using the usual [ ] notation, due east.g. file[1] refers to the first element in the array file. Notation, that bash only supports i-dimensional arrays with integers every bit indizes.

See the section on arrays in the Advanced Fustigate-Scripting Guide:.

              #!/bin/bash # Counting the number of lines in a listing of files # role version  # role storing list of all files in variable files get_files () {   files="`ls *.[ch]`" }  # function counting the number of lines in a file count_lines () {   f=$1  # 1st argument is filename   50=`wc -fifty $f | sed 's/^\([0-ix]*\).*$/\1/'` # number of lines }  # the script should exist chosen without arguments if [ $# -ge 1 ] and then   repeat "Usage: $0 "   exit ane fi  # split past newline IFS=$'\012'  echo "$0 counts the lines of code"  # don't forget to initialise! l=0 n=0 due south=0 # call a function to get a list of files get_files # iterate over this list for f in $files do         # call a function to count the lines         count_lines $f         echo "$f: $fifty"loc 	# shop filename in an assortment 	file[$north]=$f 	# store number of lines in an assortment 	lines[$n]=$l 	# increase counter         n=$[ $n + one ] 	# increment sum of all lines         southward=$[ $s + $fifty ] washed  echo "$n files in full, with $south lines in total" i=five echo "The $i-th file was ${file[$i]} with ${lines[$i]} lines"            

Version 6: Count only files we ain

              #!/bin/bash # Counting the number of lines in a list of files # for loop over arguments # count just those files I am owner of  if [ $# -lt i ] then   echo "Usage: $0 file ..."   exit i fi  repeat "$0 counts the lines of code"  fifty=0 north=0 due south=0 for f in $* do   if [ -O $f ] # checks whether file owner is running the script   then        l=`wc -l $f | sed 's/^\([0-9]*\).*$/\i/'`       echo "$f: $l"       due north=$[ $n + ane ]       s=$[ $s + $fifty ]   else       keep   fi done  echo "$n files in total, with $s lines in total"            

Version vii: Line count over several files

The final example supports options that tin can be passed from the control-line, e.g. by ./loc7.sh -d 1 loc7.sh. The getopts shell function is used to iterate over all options (given in the following string) and assigning the current option to variable name. Typically it is used in a while loop, to gear up shell variables that will be used later. We use a pipage of cat and awk to impress the header of this file, up to the first empty line, if the help selection is chosen. The main function of the script is a for loop over all non-option command-line arguments. In each iteration, $f contains the name of the file to process. If the date options are used to narrow the telescopic of files to process, we use the engagement and an if-statement, to compare whether the modification fourth dimension of the file is within the specified interval. Only in this case practise we count the number of lines as before. After the loop, nosotros impress the total number of lines and the number of files that have been processed.

Example 11. Version seven: Line count over several files

                #!/bin/fustigate ############################################################################ # # Usage: loc7.sh [options] file ... # # Count the number of lines in a given list of files. # Uses a for loop over all arguments. # # Options: #  -h     ... help message #  -d n ... consider only files modified within the last n days #  -w n ... consider only files modified within the concluding n weeks # # Limitations:  #  . only one option should exist given; a 2d one overrides # ############################################################################  help=0 verb=0 weeks=0 # defaults days=0 m=ane str="days" getopts "hvd:w:" proper noun while [ "$name" != "?" ] ; do   instance $name in    h) aid=one;;       v) verb=1;;       d) days=$OPTARG       m=$OPTARG       str="days";;    west) weeks=$OPTARG       k=$OPTARG       str="weeks";;   esac    getopts "hvd:w:" proper name done  if [ $aid -eq 1 ]  then no_of_lines=`cat $0 | awk 'Brainstorm { n = 0; } \                                  /^$/ { print n; \                                         leave; } \                                       { northward++; }'`       repeat "`head -$no_of_lines $0`"       leave  fi  shift $[ $OPTIND - 1 ]  if [ $# -lt 1 ] then   repeat "Usage: $0 file ..."   leave one fi  if [ $verb -eq i ]   then echo "$0 counts the lines of code"  fi  fifty=0 n=0 s=0 for f in $* do   x=`stat -c "%y" $f`   # modification date   d=`engagement --appointment="$x" +%y%1000%d`   # date of $k days/weeks ago   eastward=`engagement --date="$k $str ago" +%y%m%d`   # at present   z=`date +%y%m%d`   #echo "Stat: $10; At present: $z; File: $d; $m $str ago: $east"   # checks whether file is more than recent then req   if [ $d -ge $e -a $d -le $z ] # ToDo: fix yr wrap-arounds   and so        # be verbose if we plant a recent file       if [ $verb -eq 1 ]          then echo "$f: modified (mmdd) $d"       fi       # do the line count       l=`wc -l $f | sed 's/^\([0-9]*\).*$/\one/'`       echo "$f: $50"       # increase the counters       n=$[ $n + one ]       s=$[ $southward + $l ]   else       # not strictly necessary, because it's the stop of the loop       continue   fi washed  echo "$n files in full, with $s lines in full"              

	Exercise
	Extend Version 7 of the line count example above, to also compute the total number of bytes and the full number of words in the input file. $ ./loc8.sh text_file.txt loc8.sh ii files in total, with 1494438 bytes, 746 words, 27846 lines in total Hint: cheque the man page for wc.

jimenezsurney.blogspot.com

Source: http://www.macs.hw.ac.uk/~hwloidl/Courses/LinuxIntro/x864.html

Share this post