Tagged: shell script

Newline / convert newline shell script

When pasting data to a text file, for use with python and other data editing tools, it may be better to use a code text editor, like BB Edit or Text Wrangler. These apps can save your .txt files with UNIX line returns which is necessary to make some scripts work properly.

Technically the line breaks are invisible, but using the ‘tr’ command, in a UNIX shell, you could display them as any character you like, eg:

In the terminal type:
cat /path/some/file.txt | tr ‘\n’ ‘U’

In the above example, if the input file uses UNIX style line returns, \n ,each line will have a ‘U’ at the end, or nothing if the file is using another type of return.

To change line returns from Mac to Unix in a data file, try the shell script below.



newFile=`basename -s ".txt" $theFile`.UnixLineReturns.txt

theDir=`dirname $theFile`

echo "theFile: $theFile"
echo "newFile: $newFile"
echo "theDir: $theDir"

cat "$theFile" | tr ‘\r’ ‘\n’ > "$theDir/$newFile"

Rsync backup

The scripts, below, allow you to sync directories or drives easily. Great for backups.

Save the scripts to text files with the .sh extension. Make them executable (chmod +x).


#!/bin/sh if [ $# -ne 2 ]; then echo 1>&2 "USAGE: syncDir.sh /source/dir /backup/dir/" exit 127 fi rsync --delete -av "$1" "$2"

Usage, in terminal window: path/to/syncDir.sh /source/dir/ /backup/dir/. It is recommended to set up mySync.sh, second script below, to execute syncDir.sh and supply the arguments for source directory and target backup directory.

Customize the paths to the syncDir.sh script, your source directory and the target backup directory.

~/scripts/syncDir.sh /Volumes/sourceDir/ /Volumes/backupDir

Execute the script in a terminal window (drag and drop and hit enter, or enter the path to the script and hit enter), or set up a cron job to execute this script on a schedule.

Important note!
Seems that sometimes we need to run rsync as superuser, to allow the “–delete” flag to do it’s work deleting or moving files that you have deleted or moved on your source directory. So you can run syncDir.sh as superuser by typing:

sudo ~/scripts/syncDir.sh /Volumes/sourceDir/ /Volumes/backupDir

I recommend testing these scripts thoroughly, with temporary directories and files, before using them on valuable data!

vi text editor introduction

vi, text editing program for the Unix/Mac terminal.

To open vi, open a file, or create a file type: vi file_name in a terminal window. Or more specifically vi path/to/filename.ext. For example: vi ~/Desktop/hello.sh will open, or create, a file on the Desktop called hello.sh.

To start adding text you need to be in insert mode! hit the i key to get into insert mode.

To delete text, in insert mode, press x for single characters. Press dw for entire words. Press dd to delete entire lines.

To save a file, first hit the ESC key, to enter “command mode”, then type: :w, then hit ENTER (colon+w ENTER)

To exit vi , first hit the ESC key, then type : :q, then hit ENTER (colon+q ENTER)

You can save AND exit by typing :wq, while in command mode.

To move to the beginning of a line, in command mode hit 0 (zero), the end hit $.

Vi cheat sheet

more info here
and here

Shell scripting notes

A shell script is a script written for the shell, or command line interface: Terminal. Typical operations performed by shell scripts include file manipulation, program execution, and printing text.

Examples of uses for shell scripts:

  • Editing large text or data files.
  • Automating tasks, such as backup operations.

You can write a shell script to a text file to perform various tasks on your Mac.


#!/bin/sh if [ -f ~/Desktop/hello.sh ]; then say Hello! fi

Save this code to a text file named hello.sh, on your desktop. It is best to use a program like Text Wrangler, or vi. Open the Terminal (in the Applications/Utilities/ folder) and type: chmod +x ~/Desktop/hello.sh, to make the file executable. To run the script type: ~/Desktop/hello.sh in a terminal window.

#!/bin/sh tells the shell to run the program in sh, a type of shell program.
if and fi are the beginning and end of an “if statement”. Allows execution of certain commands if certain arguments are true.
[ -f ~/Desktop/hello.sh ] is the argument. It means: if the file hello.sh exists on the desktop, do the commands that follow the “then” statement.
say Hello! is the command. This should cause your Mac to say “Hello!”.

You will need learn some basic Unix commands to get around in the command line interface. Read a Mac Terminal tutorial (pdf). The basics can be learned quickly. Below are a few commands that you need to know:

, list the contents of the current directory.
pwd, print the path to the current directory to screen.
cd, change directory. Ex: cd ~/ will change directories to your user home directory.
chmod, changes permissions. Vital, but best to do a tutorial on this one.
mkdir, create a new directory.

Using the Terminal in OSX and Linux is similar.

Utilities/commands useful for shell scripting.

awk A programming language that is designed for processing text-based data. awk video
grep Text search utility originally written for Unix. grep video grep video 2
cat Unix program used to concatenate and display files. cat video
sed is a Unix utility that (a) parses text files and (b) implements a programming language which can apply textual transformations to such files. Sed video

Example grep: this lists (ls) the items in the current directory, and recognizes directories in the list by using the -F option ( you will note the / after list items that are directories). The pipe (|) forwards the result to grep, which filters for directories because we used “/” as the argument.

ls -F | grep /

Example grep and awk: df lists the drives and partitions and disk space on the computer. -h makes it more (h)uman readable. Pipe (|) into grep is filtered for one of the disks, named disk0s2. Pipe (|) awk prints only the 5th column ($5). The result is just the disk space used ex: 39%.

df -h | grep disk0s2 | awk '{print $5}'

or just use awk in this format to get the same result

# more efficient df -h | awk '/disk0s2/ {print $5}' # prints it out in human readable formatting. df -h | awk '/disk0s2/ {print $1 " is " $5 " used. "}'

More info from shell scripting primer from Apple:


Advanced Bash-Scripting Guide

An example of a shell script in action, and a detailed explanation of its functionality.

Extracting raw keyframe data from After Effects keyframe data

You can easily “export” keyframe data from After Effects by copying the key frames in the timeline, then pasting them to a text file.

We want to extract raw key frame data, from After Effects formatted key fame data, via shell script, so that it can be used by Maya or other animation apps.

After Effects keyframe data looks like this:

Adobe After Effects 8.0 Keyframe Data Units Per Second 29.97 Source Width 900 Source Height 506 Source Pixel Aspect Ratio 1 Comp Pixel Aspect Ratio 1 Effects Sound Keys #1 Output 1 #22 Frame 1 0.000261479 2 0.00608461 3 0.0153011 4 0.0274689 5 0.0395869 6 0.0493024 7 0.0562797 8 0.0557284

We need something more like this:

0.000261479 0.00608461 0.0153011 0.0274689 0.0395869 0.0493024 0.0562797 0.0557284

Use this shell script to extract the keys: AfterEffectsKeyExtract.sh
Read this for a detailed explanation of the script.
Usage (in the terminal): path/to/AfterEffectsKeyExtract.sh path/to/keyFrameData.txt

Use this shell script to batch process several files:
Usage (in the terminal): path/to/BatchExtract.sh path/to/directory

Also see: http://oliverwolfson.com/importing-keyframe-data-to-maya/

Here is a primer on shell scripting. You can run a shell script on a Mac or in Linux, through the Terminal, or on a PC with a app like Cygwin.

UPDATE! The NEW! The python script, linked here, will do the same, import After Effects Sound Keys keyframes, from .txt files, to Maya, but it will also take care of the formatting, so there is no need to run the shell script on your After Effects keyframes before importing. Look for the NEW! script.

Rename multiple files with Mac shell script

Simple renaming and renumbering script for multiple files

num=0 for j in *.jpg do mv "$j" `printf "myFILE.%05d.jpg" $num` num=`echo "$num + 1" | bc` done

Usage: In any text editor, modify the script so that the extension in the second line (here .jpg) exactly matches the extension of the files you want to modify. Also, change the file name in line four (here myFILE) to whatever name you want to use. In the Terminal, cd to the directory with your files. Paste the script to the terminal, hit enter.

Note, the way this is set up, the script will rename every file of the specified .extension in the current directory.

Replace file name suffix:

Another script. This script changes the suffix of each specified file in a directory. In this example from “myName_000_01.JPG” to “myName_000.jpg”

for i in *.JPG do mv $i `basename -s _01.JPG $i`.jpg done

Edit the the first line of the script so that it works on the appropriate files. In this example we have several files that have the extension of .JPG in common. Each of these files will be renamed.

The “mv” unix command is used to rename the files. Using the “basename” utility and the “-s” flag we can designate what part original name we want to keep, rather the suffix we want to remove. In this example the suffix we want to remove is “_01.JPG”.

Specify the new suffix that you want to add (here it’s “.jpg”).

Next, open a terminal window. Navigate to the directory where your files reside. Enter “sh” in the window, hit enter. Paste the script, and hit enter. That’s it.

scripts written with the assistance of Jeshua Lacock of OpenOSX.com