MESSAGE
| DATE | 2015-03-27 |
| FROM | Ruben Safir
|
| SUBJECT | Subject: [NYLXS - HANGOUT] Bash Shell tricks
|
I looked this up today so I'll pass it along
Bash For Loop Spaces
by nixCraft on March 13, 2008 ·
16 comments
· LAST
UPDATED February 4, 2014
in BASH Shell , Linux
, UNIX
How do I use bash for loop with spaces in a file name on Unix or Linux
operating system? The following code fails with errors:
#!/bin/bash
files=$(ls *.txt)
dest="/nas/server/dest"
for f in $files
do
cp "$f" $dest
done
How do I fix this problem on bash shell?
Tutorial details
Difficulty Easy
(rss
)
Root privileges No
Requirements None
Estimated completion time 5m
For demonstration purpose create files as follows. cd to /tmp and use
mkdir command to create a directory file called test:
cd /tmp/
mkdir test
cd test
Create a set of files:
echo "foo" > "This is a test.txt"
echo "bar" > "another file name with lots of spaces .txt"
date > "current date and time.txt"
ls -l /etc/*.conf > "My configuration files.lst"
echo "Eat in silence; work in silence." > quote.txt
echo "Eat in silence; work in silence." > quote.txt
echo "Pride is blinding" > "A Long File Name . doc"
To list directory contents use ls command as follows:
|$ ls -l|
Sample outputs:
total 48
-rw-r--r-- 1 vivek wheel 18 Feb 4 16:01 A Long File Name . doc
-rw-r--r-- 1 vivek wheel 1092 Feb 4 15:54 My configuration files.lst
-rw-r--r-- 1 vivek wheel 4 Feb 4 15:54 This is a test.txt
-rw-r--r-- 1 vivek wheel 4 Feb 4 15:54 another file name with lots of spaces .txt
-rw-r--r-- 1 vivek wheel 29 Feb 4 15:54 current date and time.txt
-rw-r--r-- 1 vivek wheel 33 Feb 4 15:58 quote.txt
Understanding problem
Try to read file name with spaces in a for or while loop using following
syntax instead of ls command:
Syntax
for f in *
do
echo "$f"
done
Sample outputs:
A Long File Name . doc
My configuration files.lst
This is a test.txt
another file name with lots of spaces .txt
current date and time.txt
quote.txt
Let us try to copy files using a bash for loop
to $dest directory:
#!/bin/bash
dest="/nas/path/to/dest"
################################################################
## Do not use ls command to read file names in shell for loop ##
################################################################
for f in *.txt
do
# do something with $f now #
cp "$f" "$dest"
done
Wrap command line args $-at- (positional parameters) in double quotes
You can pass command line args too. The following is a bad example:
#!/bin/bash
for f in $-at-
do
echo "|$f|"
done
Run it as follows:
|./script *.txt|
Sample outputs:
|This|
|is|
|a|
|test.txt|
|another|
|file|
|name|
|with|
|lots|
|of|
|spaces|
|.txt|
|current|
|date|
|and|
|time.txt|
|quote.txt|
The following is correct way to deal with command line args in bash for
loop:
#!/bin/bash
for f in "$-at-"
do
echo "|$f|"
done
Run it as follows:
|./script *.txt|
Sample outputs:
|This is a test.txt|
|another file name with lots of spaces .txt|
|current date and time.txt|
|quote.txt|
while loop with spaces example
find . | while read -r file
do
echo "$file"
done
Or better:
find . -type f -print0 | xargs -I {} -0 echo "|{}|"
OR
find . -type f -print0 | xargs -I {} -0 cp "{}" /path/to/dest/
for loop with spaces example using IFS
*Warning*: Avoid using $IFS variable and is considered as a bad
practice, but presented here for historical reasons. See the discussion
below for more information.
O=$IFS
IFS=$(echo -en "\n\b")
for f in *
do
echo "$f"
done
IFS=$O
To process all files passed as command line args:
#!/bin/bash
O=$IFS
IFS=$(echo -en "\n\b")
for f in "$-at-"
do
echo "File: $f"
done
IFS=$O
Download PDF version
Found an error/typo on
this page?
Featured Articles:
* 30 Cool Open Source Software I Discovered in 2013
* 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X
* Top 30 Nmap Command Examples For Sys/Network Admins
* 25 PHP Security Best Practices For Sys Admins
* 20 Linux System Monitoring Tools Every SysAdmin Should Know
* 20 Linux Server Hardening Security Tips
* Linux: 20 Iptables Examples For New SysAdmins
* Top 20 OpenSSH Server Best Security Practices
* Top 20 Nginx WebServer Best Security Practices
* 20 Examples: Make Sure Unix / Linux Configuration Files Are Free
From Syntax Errors
* 15 Greatest Open Source Terminal Applications Of 2012
* My 10 UNIX Command Line Mistakes
* Top 10 Open Source Web-Based Project Management Software
* Top 5 Email Client For Linux, Mac OS X, and Windows Users
* The Novice Guide To Buying A Linux Laptop
{ 16 comments… read them below or add one
}
1
TheBonsai March 26, 2009 at 10:22 am
Hi.
The first two examples give the wrong expression that globbing or
the positional parameters aren’t “word-aware”.
The first example (globbing) works correctly *without* touching the
internal field separators.
The second example (positional parameters) works correctly when you
quote the positional parameter mass-expander using doublequotes:
|
for x in "$-at-"; do
...
done
|
The third example should use the read-switch for “raw read”, and it
should use the default REPLY variable: The raw-reading prevents Bash
from interpreting special characters like line continuation, using
REPLY makes it reading the whole line, instead of trying to
interpret words in it.
|
find . | while read -r; do
echo "$REPLY"
done
|
However, you will face the problem with the while-loop being run in
a subshell: You can’t communicate back to the main shell, e.g. by
setting variables. So try Process Substitution:
|
while read -r; do
echo "$REPLY"
done < <(find .)
|
References:
Mass-usage of positional parameters
The read builtin command
Introduction to expansions and substitutions
Regards,
Jan
Reply
2
Peter March 26, 2009 at 10:59 am
You should not manipulate IFS for such primitive jobs! Publishing
this as a tutorial is not good.
Reply
3
Philippe Petrinko November 21, 2009 at 12:51 pm
Hi Vivek,
This subject is interesting, but false.
If you try this
1) create a file with lots of spaces :
touch “file with spaces (lots) in its name”
2) check if a simple [ for ] loop will process it:
# for f in *; do echo -e “:$f:”; done
this will print OK:
:”file with spaces (lots) in its name:
Therefore proving that there is no need to modify IFS variable,
(The Bonsai and Peter are right)
Keep up the good work, thanks for your site and interesting topics.
— Philippe
Reply
4
george February 4, 2010 at 1:09 pm
The technique with ‘read’ isn’t necessary. It depends on the source
of the strings for ‘for’.
The problem:
$ touch “hello there”
$ touch hell_is_for_children
$ for i in `ls hell*`; do echo -e “:$i:” ; done
:hell_is_for_children:
:hello:
:there:
Two solutions:
Use read:
$ ls -1 hell* | while read FILENAME; do echo “:$FILENAME:” ; done
:hell_is_for_children:
:hello there:
Note that this will not work, because echo puts everything on the
same line, even though it comes out of ls -1 on multiple lines:
$ echo `ls -1 hell*` | while read FILENAME; do echo “:$FILENAME:” ; done
:hell_is_for_children hello there:
Although for some reason this does work (note the double quotes):
$ echo “`ls -1 hell*`” | while read FILENAME; do echo “:$FILENAME:”
; done
Second solution:
You can’t set IFS to newline for some reason:
$ IFS=$(echo -ne ‘\n’)
$ echo -n :$IFS: | xxd
0000000: 3a3a ::
Unless its followed by something
$ IFS=$(echo -ne ‘\n ‘)
$ echo -n :$IFS: | xxd
0000000: 3a20 3a : :
Zeroing it out doesn’t work:
$ IFS=
$ for i in `ls -1 hell*`; do echo “:$i:” ; done
:hell_is_for_children
hello there:
Note there’s no trailing/leading colons in the above result.
You can get the \n in there if you trail it with another character. How
about 0x0d?
$ IFS=$(echo -en “\n015?)
$ for i in `ls -1 hell*` ; do echo “:$i:” ; done
:hell_is_for_children:
:hello there:
So that should do it. You’ll find another google result that sets
IFS to $(echo -en “\n\b”) and I couldn’t figure out why \b for the
life of me, until I went through all the above pain myself :)
Reply
5
Philippe Petrinko February 4, 2010 at 3:00 pm
To George,
I don’t get your point.
Let’s create 3 files
|
touch "hello there"
touch "I feel I am getting too much spaces"
touch hell_is_for_children
|
There is no problem selecting files beginning with “hell” using this
simple line:
|
for f in hell*; do echo ":${f}:"; done|
|# which gives:
:hell_is_for_children:
:hello there:
|
My Unix Guru always told me: Keep It Short and Simple.
To Vivek: regarding “To process all files passed as command line args”:
(and please note the typo “ars” instead of “args”)
Create this script as “sample01.sh”
|
#!/bin/bash
for f
do
echo ":${f}:"
done
|
Then run sample01.sh script:
|
bash sample01.sh "this-one-has-no-spaces" "this one has many spaces"
"Some Spaces Too"
#which gives:
:this-one-has-no-spaces:
:this one has many spaces:
:Some Spaces Too:
|
you can avoid “for f in $-at-” and stick to “for f”
Thanks for this interesting topic,
— Philippe
Reply
6
george February 5, 2010 at 4:39 am
Philippe – it depends on your SOURCE of filenames. Yes, a simple
shell glob will work if you’re just trying to match a pattern:
|for x in hell*|
but my examples used backticks to demonstrate problems when
*commands* are the source of filenames. eg.:
|`cat filenames.txt`|
In these cases, you may have to change IFS or use read. I used |`ls
-1`| to make the examples easier to reproduce.
Reply
7
Philippe Petrinko February 5, 2010 at 10:03 am
To George:
“but my examples used backticks to demonstrate problems when
*commands* are the source of filenames. ”
As the question of this topic is : “How do I use bash for loop with
spaces in a file name?”
I am afraid I disagree. The point is using the output of a command
in an appropriate way.
And there are ways which do not require modifying IFS variable.
Let’s try this:
|
touch "Hello, World"
touch "myfile with spaces"
touch somefile
|
Then you can use [ls -1] this way:
|
ls -1|while read f; do echo ":${f}:"; done
:Hello, World:
:myfile with spaces:
:somefile:
|
Next, using a file content.
Let’s create the list of file names
|
ls -1>list.txt
|
Again, no problem (and BTW we can skip using [cat] which involves
useless overhead)
|
while read f; do echo ":${f}:"; done < list.txt
:Hello, World:
:myfile with spaces:
:somefile:
|
Last, but not least, [bash] has a builtin Variable which takes input
of [read] function.
(always a good thing to remind !)
That is, one could simplier write
|
while read; do echo ":${REPLY}:"; done < list.txt
|
or
|
ls -1|while read; do echo ":${REPLY}:"; done
|
Nevertheless, I admit there may be some times when you need to alter
IFS – but not in the cases you proposed.
— Philippe
Reply
8
george February 6, 2010 at 8:18 am
Well Philippe, we’re down to opinions, not facts.
1) I don’t think modifying IFS is that big of a deal – that’s what
its there for.
2) I don’t think this is very readable:
|while read; do echo ":${REPLY}:"; done < list.txt|
To the eye, it looks like you’re redirecting list.txt to ‘done’.
Definitely not clear that the receiver of STDIN is way back there in
the front. The overhead for cat’ing is nothing compared to the worth
of having the source of data on the left side where it belongs.
3) Using $REPLY violates many readability and maintainability
standards. Its name doesn’t make any sense in the context (who
exactly is replying?).
Reply
9
TheBonsai February 6, 2010 at 8:53 am
-at-george
at 2) it’s more correct to say you redirect to the ‘done’ than to
the ‘read’. Both is wrong, of course, because you redirect to the
compound command.
at 3) you need REPLY here or you have code that looses spaces
Reply
10
Philippe Petrinko February 6, 2010 at 3:24 pm
To George:
1) Is IFS modification a big deal ?
Yes, you just proved it.
Because of all defensive programming techniques I have learnt,
modifying IFS should kept to avoided unless necessary.
a) Modifying IFS it is useless in this case, it can be solved by
basic commands. Keep it short & simple.
b) As your code shows it, it is not for beginners, IFS processing as
many side-effects and is complex to handle.
So, it is more prone to programming errors. For instance:
– To forget to restore IFS value. Code may be long, and IFS
restoration may be out of view.
– Using commands which rely on Environment variables leads to code
that is much more context-dependant – and therefore less readable.
This is exactly what your experience shows, you said: “I couldn’t
figure out why for the life of me, until I went through all the
above pain myself”
2) Regarding code readability and being puzzled by right-hand
redirection:
Well, it’s a fact, and not an opinion, that input redirection with
“<" is just basic shell functionality.
Nevertheless, no problemo. Write:
|
cat list.txt | while read; do echo ":${REPLY}:"; done
or
cat list.txt | while read
do
echo ":${REPLY}:";
done
|
3) Using $REPLY …
I did not choose this name – Bash programmers did. Nevertheless, you
don’t like it, drop it:
|
cat list.txt | while read fname
do
echo "${fname}";
done
|
that’s it !
See, no IFS modification needed, readable, from left to right.
I just prefer the more efficient (using just shell internals
functions) …
|
while read fname
do
echo "${fname}";
done < list.txt
|
…because this is the use shell input redirection was designed for.
Cheers,
— Philippe
Reply
11
TheBonsai February 6, 2010 at 3:33 pm
-at-Philipe
Same applies to your read. If you don’t use the automatic REPLY,
read looses data (as it doesn’t read the line as line, but as set of
words).
Reply
12
Philippe Petrinko February 6, 2010 at 3:51 pm
To the Bonsai:
I am not sure of what you really mean, I assume you say this code
does not work.
The request is to print file names that may contains spaces – this
works.
I have checked this code before. Let’s check it again in my
gnome-terminal:
|
while read fname; do echo ":${fname}:"; done < list.txt
:Hello, World:
:list.txt:
:myfile with big spaces:
:myfile with spaces:
:somefile:
|
It works. I don’t know if you tried it, but it just works on my
Debian Lenny/Bash.
Or may I have misunderstood your comment?
Would you show me where this code does not fit to the purpose ?
Reply
13
TheBonsai February 6, 2010 at 5:13 pm
I mean this:
|
$ sed 's/^/>>>/;s/$/<<>> leading and trailing spaces <<<|
|$ while read fname; do echo ":${fname}:"; done < list.txt
:leading and trailing spaces:
|
Reply
14
TheBonsai February 6, 2010 at 5:14 pm
Sorry, seems I broke the HTML code parser there :/
Reply
15
Philippe Petrinko February 6, 2010 at 6:49 pm
-at-Bonsai
Yeah, right :-/
$REPLY was designed to take raw input (I mean the whole line without
word splitting).
And then yes, in this case, if we do not use $REPLY,
one of the simpliest way to have [read] include leading and trailing
spaces (or any FS caracter)
is to modify IFS variable.
Reply
16
chris February 4, 2014 at 8:47 am
nixcraft is now seen as untrustworthy by me
|
|
