Wednesday 5 June 2013

Unix material


------------------------------------------Directories:-----------------------------------
File and directory paths in UNIX use the forward slash "/"
to separate directory names in a path.
examples:
/              "root" directory
/usr           directory usr (sub-directory of / "root" directory)
/usr/STRIM100  STRIM100 is a sub directory of /usr

------------------------Moving around the file system:----------------------------
pwd                            Show the "present working directory", or current directory.
cd                               Change current directory to your HOME directory.
cd /usr/STRIM          100  Change current directory to /usr/STRIM100.
cd INIT                        Change current directory to INIT which is a sub-directory of the current
                                     directory.
cd ..                             Change current directory to the parent directory of the current directory.
cd $STRMWORK      Change current directory to the directory defined by the environment
                                      variable 'STRMWORK'.
cd ~bob                      Change the current directory to the user bob's home directory (if you have permission).

----------------------------Listing directory contents:----------------------------------
ls        list a directory
ls -l     list a directory in long ( detailed ) format
   for example:
$ ls -l
drwxr-xr-x    4 cliff    user        1024 Jun 18 09:40 WAITRON_EARNINGS
-rw-r–r–    1 cliff    user      767392 Jun  6 14:28 scanlib.tar.gz
^ ^  ^  ^     ^   ^       ^           ^      ^    ^      ^
| |  |  |     |   |       |           |      |    |      | 
| |  |  |     | owner   group       size   date  time    name
| |  |  |     number of links to file or directory contents
| |  |  permissions for world
| |  permissions for members of group
| permissions for owner of file: r = read, w = write, x = execute -=no permission
type of file: – = normal file, d=directory, l = symbolic link, and others…
ls -a          List the current directory including hidden files. Hidden files start
                  with "."
ls -ld *     List all the file and directory names in the current directory using long format. Without the "d" option, ls would list the contents
                 of any sub-directory of the current. With the "d" option, ls just lists them like regular files.

--------------------------Changing file permissions and attributes------------------------
chmod 755 file      Changes the permissions of file to be rwx for the owner, and rx for
                                  the group and the world. (7 = rwx = 111 binary. 5 = r-x = 101 binary)
chgrp user file        Makes file belong to the group user.
chown cliff file         Makes cliff the owner of file.
chown -R cliff dir    Makes cliff the owner of dir and everything in its directory tree.
You must be the owner of the file/directory or be root before you can do any of these things.

-----------------------------------Moving, renaming, and copying files:--------------------------------
cp file1 file2                     copy a file
mv file1 newname         move or rename a file
mv file1 ~/AAA/                move file1 into sub-directory AAA in your home directory.
rm file1 [file2 ...]              remove or delete a file
rm -r dir1 [dir2...]             recursivly remove a directory and its contents BE CAREFUL!
mkdir dir1 [dir2...]           create directories
mkdir -p dirpath              create the directory dirpath, including all implied directories in the path.
rmdir dir1 [dir2...]            remove an empty directory

-----------------------------------------Viewing and editing files:----------------------------------------

cat filename
            Dump a file to the screen in ascii.
more filename        Progressively dump a file to the screen: ENTER = one line down SPACEBAR = page down  q=quit
less filename          Like more, but you can use Page-Up too. Not on all systems.
vi filename               Edit a file using the vi editor. All UNIX systems will have vi in some form.
emacs filename     Edit a file using the emacs editor. Not all systems will have emacs.
head filename          Show the first few lines of a file.
head -n  filename    Show the first n lines of a file.
tail filename              Show the last few lines of a file.
tail -n filename          Show the last n lines of a file.

kill -9 on pid 8323100

open unix and kill this process quickly
ps -ef | grep 8323100

kill -9 8323100

kill: 8323100: no such process
isqk@dev247.cs.ctc  [/home/net/isqk] [2]

WC -l #$DIR_REJECT#/RETAIL_COMPETITION_#ESPDATE#_#ESPTIME#_insert.rej

[username@mysite]$ ls -l
will produce a long listing format that includes the permissions, owner, group, size and modified date of each file:
isqk@dev247.cs.ctc  [/home/net/isqk] [0]
$cd ..

isqk@dev247.cs.ctc  [/home/net] [0]  ----Wrong
$cd /ibwk/
ksh: /ibwk/:  not found

isqk@dev247.cs.ctc [/home/net] [1]
$cd ibwk

isqk@dev247.cs.ctc [/home/net/ibwk] [0]
$ls -a
.                      .profile               BASE_POSITION_DIM.txt  gova.txt               suri.txt               y
..                     .vi_history            gov.txt                sur.txt                x

isqk@dev247.cs.ctc  [/home/net/ibwk] [0]
$vi .profile

PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.

export PATH

if [ -s "$MAIL" ]           # This is at Shell startup.  In normal
then echo "$MAILMSG"        # operation, the Shell checks
fi                          # periodically.

. /apps/IBM/InformationServer/Server/DSEngine/dsenv

export APT_CONFIG_FILE=/apps/qat/pvlr/dscfg/pvlr_px_4node.cfg



isqk@dev247.cs.ctc  [/home/net/ibwk] [1]
$cd ..

isqk@dev247.cs.ctc  [/home/net] [0]
$cd isqk

isqk@dev247.cs.ctc  [/home/net/isqk] [0]
$ls -a
.            ..           .profile     .vi_history

isqk@dev247.cs.ctc  [/home/net/isqk] [0]
$vi .profile




PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.

export PATH

if [ -s "$MAIL" ]           # This is at Shell startup.  In normal
then echo "$MAILMSG"        # operation, the Shell checks
fi                          # periodically.

-----Go to ibwk user id and copy the last two lines and Paste that into upper data below.------

i----insert mode

. /apps/IBM/InformationServer/Server/DSEngine/dsenv

export APT_CONFIG_FILE=/apps/qat/pvlr/dscfg/pvlr_px_4node.cfg



i----insert mode
Save and Quit
wq!
$orchadmin dump  -field        ------> View the Particular fields data in dataset.

isqk@dev247.cs.ctc  [/apps/dev/etlb/dsdata] [127]
$orchadmin  dump REPLENISHMENT_STORE_ORDER_AGGR.ds

How to copy the content of Dataset to a flat file.

Step 1) source the dsenv file
Step 2) export the APT_CONFIG_FILE
export APT_CONFIG_FILE=/opt/IBM/InformationServer/v81/Server/Configurations/default.apt
Step 3) Then Run the command

$orchadmin dump > test_ds.txt (orchadmin dump -delim '|' Frame_t.ds >> test_ds.txt)
($orchadmin dump -field "ORGANIZATION_UNIT_NUM" -field "ORGANIZATION_UNIT_ID" -field
"ORGANIZATION_UNIT_HISTORY_ID" REPLENISHMENT_STORE_ORDER_AGGR_Stg.ds)

------------------------------------------Orchadmin  commands---------------------------

Orchadmin is a command line utility provided by datastage to research on data sets.
The general callable format is : $orchadmin [options] [descriptor file]
1. Before using orchadmin, you should make sure that either the working directory or the $APT_ORCHHOME/etc  contains the file “config.apt” OR
The environment variable $APT_CONFIG_FILE  should be defined for your session.
Orchadmin  commands
 The various commands available with orchadmin are
1. CHECK: $orchadmin check
Validates the configuration file contents like  , accesibility of all nodes defined in the configuration file, scratch disk definitions and accesibility of all the nodes etc. Throws an error when config file is not found or not defined properly

2. COPY : $orchadmin copy
Makes a complete copy of the datasets of source with new destination descriptor file name. Please not that
a. You cannot use UNIX cp command as it justs copies the config file to a new name. The data is not copied.
b. The new datasets will be arranged in the form of the config file that is in use but not according to the old confing file that was in use with the source.

  3. DELETE : $orchadmin < delete | del | rm >  [-f | -x] descriptorfiles….
The unix rm utility cannot be used to delete the datasets. The orchadmin delete or rm command should be used to delete one or more persistent data sets.
-f options makes a force delete. If some nodes are not accesible then -f forces to delete the dataset partitions from accessible nodes and leave the other partitions in inaccesible nodes as orphans.
-x forces to use the current config file to be used while deleting than the one stored in data set.

4.  DESCRIBE: $orchadmin describe [options] descriptorfile.ds
This is the single most important command.
1. Without any option lists the no.of.partitions, no.of.segments, valid segments, and preserve partitioning flag details of the persistent dataset.
-c : Print the configuration file that is written in the dataset  if any
-p: Lists down the partition level information.
-f: Lists down the file level information in each partition
-e:  List down the segment level information .
-s: List down the  meta-data schema of the information.
-v:  Lists all segemnts , valid or otherwise
-l : Long listing. Equivalent to -f -p -s -v -e

5. DUMP: $orchadmin dump [options] descriptorfile.ds
The dump command is used to dump(extract) the records from the dataset.
Without any options the dump command lists down all the records starting from first record from first partition till  last record in last partition.
-delim ‘’ : Uses the given string as delimtor for fields instead of space.
-field : Lists only the given field instead of all fields.
-name : List all the values preceded by field name and a colon
-n numrecs : List only the given number of records per partition.
-p period(N) : Lists every  Nth record from each partition starting from first record.
-skip N: Skip the first N records from each partition.
-x : Use the current system configuration file rather than the one stored in dataset.

6. TRUNCATE: $orchadmin truncate [options] descriptorfile.ds
Without options deletes all the data(ie Segments) from the dataset.
-f: Uses force truncate. Truncate accessible segments and leave the inaccesible ones.
-x: Uses current system config file rather than the default one stored in the dataset.
-n N: Leaves the first N segments in each partition and truncates the remaining.

7. HELP: $orchadmin -help OR $orchadmin -help
Help manual about the usage of orchadmin or orchadmin commands.

1)orchadmin diskinfo
  -a     print information for all nodes
-np    print information for specified node pool
 -n      print information for specified nodes
-q   print summary of information only
No option specified , the default node pool is used


1)orchadmin dump –field “column name” datasetname.ds-----TO DISPLAY PERTICULAR FIELD
EX: $orchadmin dump -field "LANDING_ID" EDW_LANDING.IMPORT_PROJECT.ds
2)orchadmin dump datasetname.ds --------TO DISPLAY ALL ROWS AND COLUMNS
EX: $orchadmin dump EDW_LANDING.IMPORT_PROJECT.ds
3) Ds records datasetname.ds ------------TO DISPLAY NO.OF RECORDS IN A DATASET
EX: $dsrecords EDW_LANDING.IMPORT_PROJECT.ds
4) orchadmin diskinfo
  -a     print information for all nodes
-np    print information for specified node pool
 -n      print information for specified nodes
EX: $orchadmin diskinfo -n "node0"
 -q   print summary of information only
No option specified, the default node pool is used
EX: $orchadmin diskinfo


;J<@KAEB4NM53Q

DB_ORACLE_PASSWORD_STAGING=LDH@9:VA=9BN0F7I=;J

-------------------------Data----------------------------
-rw-r--r--    1 igns     dstage         4000 Dec 13 03:03 IA169B_EDW_STORE_ORDERS.txt.bkp
-rw-r--r--    1 igns     dstage     79476800 Dec 16 07:13 IA169B_EDW_STORE_ORDERS_20111214_154901.txt
-rw-r--r--    1 igns     dstage     79836240 Dec 16 07:13 IA169B_EDW_STORE_ORDERS_20111215_123748.txt
-rw-r--r--    1 igns     dstage     80609800 Dec 16 07:13 IA169B_EDW_STORE_ORDERS_20111216_010100.txt
-rw-r--r--    1 igns     dstage     79836240 Dec 16 07:23 IA169B_EDW_STORE_ORDERS.txt
------------------------------Data---------------------------------
isqk@dev247.cs.ctc  [/apps/dev/iadata] [0]
$cp IA169B_EDW_STORE_ORDERS_20111214_154901.txt IA169B_EDW_STORE_ORDERS.txt
cp: IA169B_EDW_STORE_ORDERS.txt: Permission denied

isqk@dev247.cs.ctc  [/apps/dev/iadata] [1]
$su igns
igns's Password:

isqk@dev247.cs.ctc  [/apps/dev/iadata] [0]

$cp IA169B_EDW_STORE_ORDERS_20111214_154901.txt IA169B_EDW_STORE_ORDERS.txt

Awk Introduction and Printing Operations

Awk is a programming language which allows easy manipulation of structured data and the generation of formatted reports. Awk stands for the names of its authors “Aho, Weinberger, and Kernighan”
The Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform associated actions.
Some of the key features of Awk are:
  • Awk views a text file as records and fields.
  • Like common programming language, Awk has variables, conditionals and loops
  • Awk has arithmetic and string operators.
  • Awk can generate formatted reports
Awk reads from a file or from its standard input, and outputs to its standard output. Awk does not get along with non-text files.
Syntax:
 
awk '/search pattern1/ {Actions}
     /search pattern2/ {Actions}' file
In the above awk syntax:
  • search pattern is a regular expression.
  • Actions – statement(s) to be performed.
  • several patterns and actions are possible in Awk.
  • file – Input file.
  • Single quotes around program is to avoid shell not to interpret any of its special characters.

Awk Working Methodology

  1. Awk reads the input files one line at a time.
  2. For each line, it matches with given pattern in the given order, if matches performs the corresponding action.
  3. If no pattern matches, no action will be performed.
  4. In the above syntax, either search pattern or action are optional, But not both.
  5. If the search pattern is not given, then Awk performs the given actions for each line of the input.
  6. If the action is not given, print all that lines that matches with the given patterns which is the default action.
  7. Empty braces with out any action does nothing. It wont perform default printing operation.
  8. Each statement in Actions should be delimited by semicolon.
Let us create employee.txt file which has the following content, which will be used in the
examples mentioned below.

$cat employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

Awk Example 1. Default behavior of Awk

By default Awk prints every line from the file.
$ awk '{print;}' employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000
In the above example pattern is not given. So the actions are applicable to all the lines.
Action print with out any argument prints the whole line by default. So it prints all the
lines of the file with out fail. Actions has to be enclosed with in the braces.

Awk Example 2. Print the lines which matches with the pattern.

$ awk '/Thomas/
> /Nisha/' employee.txt
100  Thomas  Manager    Sales       $5,000
400  Nisha   Manager    Marketing   $9,500
In the above example it prints all the line which matches with the ‘Thomas’ or ‘Nisha’. It has two patterns. Awk accepts any number of patterns, but each set (patterns and its corresponding actions) has to be separated by newline.

Awk Example 3. Print only specific field.

Awk has number of built in variables. For each record i.e line, it splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0 represents whole line. NF is a built in variable which represents total number of fields in a record.
$ awk '{print $2,$5;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000
 
$ awk '{print $2,$NF;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000
In the above example $2 and $5 represents Name and Salary respectively. We can get the Salary using  $NF also, where $NF represents last field. In the print statement ‘,’ is a concatenator.

Awk Example 4. Initialization and Final Action

Awk has two important patterns which are specified by the keyword called BEGIN and END.
Syntax: 
 
BEGIN { Actions}
{ACTION} # Action for everyline in a file
END { Actions }
 
# is for comments in Awk
Actions specified in the BEGIN section will be executed before starts reading the lines from the input.
END actions will be performed after completing the reading and processing the lines from the input.

$ awk 'BEGIN {print "Name\tDesignation\tDepartment\tSalary";}
> {print $2,"\t",$3,"\t",$4,"\t",$NF;}
> END{print "Report Generated\n--------------";
> }' employee.txt
Name      Designation           Department            Salary
Thomas    Manager                 Sales               $5,000
Jason        Developer              Technology            $5,500
Sanjay      Sysadmin               Technology            $7,000
Nisha       Manager                 Marketing              $9,500
Randy      DBA                      Technology            $6,000
Report Generated
--------------
In the above example, it prints headline and last file for the reports.

Awk Example 5. Find the employees who has employee id greater than 200

$ awk '$1 >200' employee.txt
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000
In the above example, first field ($1) is employee id. So if $1 is greater than 200, then just do the default print action to print the whole line.

Awk Example 6. Print the list of employees in Technology department

Now department name is available as a fourth field, so need to check if $4 matches with the string “Technology”, if yes print the line.
$ awk '$4 ~/Technology/' employee.txt
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
500  Randy   DBA        Technology  $6,000
Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be  performed.

Awk Example 7. Print number of employees in Technology department

The below example, checks if the department is Technology, if it is yes, in the Action, just increment the count variable, which was initialized with zero in the BEGIN section.
$ awk 'BEGIN { count=0;}
$4 ~ /Technology/ { count++; }
END { print "Number of employees in Technology Dept =",count;}' employee.txt
Number of employees in Tehcnology Dept = 3
Then at the end of the process, just print the value of count which gives you the number of employees in Technology department.
-------------------------------------------------awk COMMAND-------------------------------------------------
     awk command is used to manipulate the text.This command checks each line of a file, looking for patterns that match those given on the command line.

SYNTAX:
  The Syntax is
     awk '{pattern + action}' {filenames}   or  awk <pattern> '{print <stuff>}' <file>


OPTIONS:
     -W version Display version information and exit.
-F Print help message and exit.



EXAMPLE:
     Lets create a file file1.txt and let it have the following data: Data in file1.txt
14   15   16 
15   15   11 
5     56    6 
5     25    1 

To print the second column data in file1.txt

awk '{print $2}' file1.txt
This command will manipulate and print second column of text file (file1.txt). The output will look like

15
15
56
25


To multiply the column-1 and column-2 and redirect the output to file2.txt:

awk '{print $1,$2,$1*$2}' file1.txt > file2.txt
Command Explanation:
$1  : Prints 1st column
$2  : Prints 2ndcolumn
$1*$2  : Prints Result of $1 x $2
file1.txt  :  input file
>  :  redirection symbol
file2.txt  :  output file

The above command will redirect the output to file2.txt and it will look like,
14    15    210
15    15    225
5      56    280
5      25    125
Let's start with a file, words.data, that contains these lines:
nail hammer wood
pedal foot car
clown pie circus
Now we'll use the print function in awk to plug the words from each input line into a template, like this:
awk '{print "Hit the",$1,"with your",$2}' words.data
Hit the nail with your hammer
Hit the pedal with your foot
Hit the clown with your pie
Say some of the data in your input file is numeric, as in the grades.data file shown here:
Rogers 87 100 95
Lambchop 66 89 76
Barney 12 36 27
You can perform calculations like this:
awk '{print "Avg for",$1,"is",($2+$3+$4)/3}' grades.data
Avg for Rogers is 94
Avg for Lambchop is 77
Avg for Barney is 25
So far, we haven't specified any value for pattern in these examples, but if you want to exclude lines from being processed, you can enter something like this:
awk /^clown/'{print "See the",$1,"at the",$3}' words.data
See the clown at the circus
Here, we told awk to consider only the input lines that start with clown. Note also that there is no space between the pattern and the print specifier. If you put a space there, awk will think the input file is '{print and will not work. But all this is just the tip of the awk iceberg--entire books have been written on this command. If you are a programmer, try the man awk command.