File and directory
paths in UNIX use the forward slash "/"
to separate directory names in a path.
examples:
/
"root" directory
/usr directory usr
(sub-directory of / "root" directory)
/usr/STRIM100 STRIM100 is a sub directory of /usr
------------------------Moving
around the file system:----------------------------
pwd
Show the "present working directory", or current directory. cd
Change current directory to your HOME directory. cd /usr/STRIM
100 Change current directory to /usr/STRIM100. cd INIT
Change current directory to INIT which is a sub-directory
of the current
directory. cd ..
Change current directory to the parent directory of the current directory.
cd $STRMWORK Change current directory to the
directory defined by the environment
variable 'STRMWORK'. cd ~bob
Change the current directory to the user
bob's home directory (if you have permission).
ls list a directory ls -l list a directory in long ( detailed ) format
for
example:
$ ls -l
drwxr-xr-x 4 cliff
user 1024 Jun 18 09:40
WAITRON_EARNINGS
-rw-r–r– 1 cliff
user 767392 Jun 6 14:28 scanlib.tar.gz
^ ^ ^ ^ ^
^
^
^ ^
^ ^
| | | | |
|
|
| |
| |
| | | | | owner
group size date
time name
| | | | number of links to file or
directory contents
| | | permissions for world
| | permissions for members of group
| permissions for owner of file: r = read, w = write, x = execute -=no
permission
type of file: – = normal file, d=directory, l = symbolic link, and others…
ls -a
List the current directory including hidden files. Hidden files start
with "." ls -ld * List all the file and directory names
in the current directory using long format. Without the "d" option,
ls would list the contents
of any sub-directory of the current. With the "d" option, ls just
lists them like regular files.
--------------------------Changing file permissions and attributes------------------------
chmod 755
file Changes the permissions of file to be rwx
for the owner, and rx for
the group and the world. (7 = rwx = 111 binary. 5 = r-x = 101 binary)
chgrp user file Makes file belong to
the group user.
chown cliff file Makes cliff the owner of
file.
chown -R cliff dir Makes cliff the owner of dir and everything in
its directory tree.
You must be the owner
of the file/directory or be root before you can do any of these things.
-----------------------------------Moving, renaming, and copying files:--------------------------------
cp file1 file2
copy a file mv file1 newname move or
rename a file mv file1 ~/AAA/
move file1 into sub-directory AAA in your home directory. rm file1 [file2 ...]
remove or delete a file rm -r dir1 [dir2...]
recursivly remove a directory and its contents BE CAREFUL! mkdir dir1 [dir2...] create
directories mkdir -p dirpath
create the directory dirpath, including all implied directories in the
path. rmdir dir1 [dir2...]
remove an empty directory
-----------------------------------------Viewing
and editing files:----------------------------------------
cat filename
Dump a file to the screen in ascii. more filename Progressively dump a
file to the screen: ENTER = one line down SPACEBAR = page down q=quit less filename Like more, but you
can use Page-Up too. Not on all systems. vi filename
Edit a file using the vi editor. All UNIX systems will have vi in
some form. emacs filename Edit a file using the emacs editor.
Not all systems will have emacs. head filename Show
the first few lines of a file. head -n filename Show the first n lines of a file. tail filename Show
the last few lines of a file. tail -n filename
Show the last n lines of a file.
$orchadmin dump -field ------> View the Particular fields data
in dataset.
isqk@dev247.cs.ctc
[/apps/dev/etlb/dsdata] [127]
$orchadmin dump
REPLENISHMENT_STORE_ORDER_AGGR.ds
How to
copy the content of Dataset to a flat file.
Step 1) source the dsenv file
Step 2) export the APT_CONFIG_FILE
export APT_CONFIG_FILE=/opt/IBM/InformationServer/v81/Server/Configurations/default.apt
Step 3) Then Run the command
Orchadmin
is a command line utility provided by datastage to research on data sets.
The
general callable format is : $orchadmin
[options] [descriptor file]
1.
Before using orchadmin, you should make sure that either the working directory
or the $APT_ORCHHOME/etc contains the file “config.apt” OR
The
environment variable $APT_CONFIG_FILE should be defined for your session.
Orchadmin commands
The
various commands available with orchadmin are
1.
CHECK: $orchadmin check
Validates
the configuration file contents like , accesibility of all nodes defined
in the configuration file, scratch disk definitions and accesibility of all the
nodes etc. Throws an error when config file is not found or not defined
properly
2.
COPY : $orchadmin copy
Makes
a complete copy of the datasets of source with new destination descriptor file
name. Please not that
a.
You cannot use UNIX cp command as it justs copies the config file to a new
name. The data is not copied.
b.
The new datasets will be arranged in the form of the config file that is in use
but not according to the old confing file that was in use with the source.
The
unix rm utility cannot be used to delete the datasets. The orchadmin delete or
rm command should be used to delete one or more persistent data sets.
-f
options makes a force delete. If some nodes are not accesible then -f forces to
delete the dataset partitions from accessible nodes and leave the other partitions
in inaccesible nodes as orphans.
-x
forces to use the current config file to be used while deleting than the one
stored in data set.
Awk is a programming language which allows easy manipulation of structured
data and the generation of formatted reports. Awk stands for the names of its
authors “Aho, Weinberger, and Kernighan” The Awk is mostly used for pattern scanning and processing. It searches one
or more files to see if they contain lines that matches with the specified
patterns and then perform associated actions. Some of the key features of Awk are:
Awk views a text file as
records and fields.
Like common programming
language, Awk has variables, conditionals and loops
Awk has arithmetic and
string operators.
Awk can generate formatted
reports
Awk reads from a file or from its standard input, and outputs to its
standard output. Awk does not get along with non-text files.
Syntax:
awk '/search pattern1/ {Actions}
/search pattern2/ {Actions}' file
In the above awk syntax:
search pattern is a regular
expression.
Actions – statement(s) to be
performed.
several patterns and actions
are possible in Awk.
file – Input file.
Single quotes around program
is to avoid shell not to interpret any of its special characters.
Awk Working Methodology
Awk reads the input files
one line at a time.
For each line, it matches
with given pattern in the given order, if matches performs the
corresponding action.
If no pattern matches, no
action will be performed.
In the above syntax, either
search pattern or action are optional, But not both.
If the search pattern is not
given, then Awk performs the given actions for each line of the input.
If the action is not given,
print all that lines that matches with the given patterns which is the
default action.
Empty braces with out any
action does nothing. It wont perform default printing operation.
Each statement in Actions
should be delimited by semicolon.
Let us create employee.txt file which has the following content, which will
be used in the
examples mentioned below.
$cat employee.txt
100 Thomas Manager Sales $5,000
200 Jason Developer Technology $5,500
300 Sanjay Sysadmin Technology $7,000
400 Nisha Manager Marketing $9,500
500 Randy DBA Technology $6,000
Awk Example 1. Default behavior of Awk
By default Awk prints every line from the file.
$ awk '{print;}' employee.txt
100 Thomas Manager Sales $5,000
200 Jason Developer Technology $5,500
300 Sanjay Sysadmin Technology $7,000
400 Nisha Manager Marketing $9,500
500 Randy DBA Technology $6,000
In the above example pattern is not given. So the actions are applicable to
all the lines.
Action print with out any argument prints the whole line by default. So it
prints all the
lines of the file with out fail. Actions has to be enclosed with in the braces.
Awk Example 2. Print the lines which matches with the pattern.
$ awk '/Thomas/
> /Nisha/' employee.txt
100 Thomas Manager Sales $5,000
400 Nisha Manager Marketing $9,500
In the above example it prints all the line which matches with the ‘Thomas’
or ‘Nisha’. It has two patterns. Awk accepts any number of patterns, but each
set (patterns and its corresponding actions) has to be separated by newline.
Awk Example 3. Print only specific field.
Awk has number of built in variables. For each record i.e line, it splits
the record delimited by whitespace character by default and stores it in the $n
variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0
represents whole line. NF is a built in variable which represents total number
of fields in a record.
$ awk '{print $2,$5;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000
$ awk '{print $2,$NF;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000
In the above example $2 and $5 represents Name and Salary respectively. We
can get the Salary using $NF also, where $NF represents last field. In
the print statement ‘,’ is a concatenator.
Awk Example 4. Initialization and Final Action
Awk has two important patterns which are specified by the keyword called
BEGIN and END.
Syntax:
BEGIN { Actions}
{ACTION} # Action for everyline in a file
END { Actions }
# is for comments in Awk
Actions specified in the BEGIN section will be executed before starts reading
the lines from the input.
END actions will be performed after completing the reading and processing the
lines from the input.
In the above example, it prints headline and last file for the reports.
Awk Example 5. Find the employees who has employee id greater than 200
$ awk '$1 >200' employee.txt
300 Sanjay Sysadmin Technology $7,000
400 Nisha Manager Marketing $9,500
500 Randy DBA Technology $6,000
In the above example, first field ($1) is employee id. So if $1 is greater
than 200, then just do the default print action to print the whole line.
Awk Example 6. Print the list of employees in Technology department
Now department name is available as a fourth field, so need to check if $4
matches with the string “Technology”, if yes print the line.
$ awk '$4 ~/Technology/' employee.txt
200 Jason Developer Technology $5,500
300 Sanjay Sysadmin Technology $7,000
500 Randy DBA Technology $6,000
Operator ~ is for comparing with the regular expressions. If it matches the
default action i.e print whole line will be performed.
Awk Example 7. Print number of employees in Technology department
The below example, checks if the department is Technology, if it is yes, in
the Action, just increment the count variable, which was initialized with zero
in the BEGIN section.
$ awk 'BEGIN { count=0;}
$4 ~ /Technology/ { count++; }
END { print "Number of employees in Technology Dept =",count;}' employee.txt
Number of employees in Tehcnology Dept = 3
Then at the end of the process, just print the value of count which gives
you the number of employees in Technology department.
awk
'{print "Avg for",$1,"is",($2+$3+$4)/3}' grades.data Avg
for Rogers is 94
Avg for Lambchop is 77
Avg for Barney is 25
So far, we haven't specified any value for pattern
in these examples, but if you want to exclude lines from being processed, you
can enter something like this:
awk /^clown/'{print "See
the",$1,"at the",$3}' words.data
See the clown at the circus
Here, we told awk to consider only the
input lines that start with clown. Note also that there is no space
between the pattern and the print specifier. If you put a space there, awk
will think the input file is '{print and will not work. But all this is
just the tip of the awk iceberg--entire books have been written on this
command. If you are a programmer, try the man awk command.