How to get bash find exec to directly execute commands instead of using a temp file?

Question

I've written this BASH script:

find ./build/html -name '*.html' \( -exec echo ../emojize_pngorsvg.py \"{}\" \> \"{}.emojized\" \&\& rm \"{}\" \&\& mv \"{}.emojized\" \"{}\" >> ../emojize_commands.sh \; \)
cat ../emojize_commands.sh
chmod +x ../emojize_commands.sh
../emojize_commands.sh

It loops over all of the HTML files it finds in the ./build/html dir and it creates a single file called emojize_commands.sh which looks something like this:

../emojize_pngorsvg.py "./build/html/Home/Technical/Guides/Windows/Force-activate-a-Windows-evaluation/index.html" > "./build/html/Home/Technical/Guides/Windows/Force-activate-a-Windows-evaluation/index.html.emojized" && rm "./build/html/Home/Technical/Guides/Windows/Force-activate-a-Windows-evaluation/index.html" && mv "./build/html/Home/Technical/Guides/Windows/Force-activate-a-Windows-evaluation/index.html.emojized" "./build/html/Home/Technical/Guides/Windows/Force-activate-a-Windows-evaluation/index.html"
../emojize_pngorsvg.py "./build/html/Home/Technical/Guides/Windows/Various-Windows-Tips/index.html" > "./build/html/Home/Technical/Guides/Windows/Various-Windows-Tips/index.html.emojized" && rm "./build/html/Home/Technical/Guides/Windows/Various-Windows-Tips/index.html" && mv "./build/html/Home/Technical/Guides/Windows/Various-Windows-Tips/index.html.emojized" "./build/html/Home/Technical/Guides/Windows/Various-Windows-Tips/index.html"
../emojize_pngorsvg.py "./build/html/Home/Technical/Guides/Windows/Configure-RDP-to-connect-with-stored-credentials/index.html" > "./build/html/Home/Technical/Guides/Windows/Configure-RDP-to-connect-with-stored-credentials/index.html.emojized" && rm "./build/html/Home/Technical/Guides/Windows/Configure-RDP-to-connect-with-stored-credentials/index.html" && mv "./build/html/Home/Technical/Guides/Windows/Configure-RDP-to-connect-with-stored-credentials/index.html.emojized" "./build/html/Home/Technical/Guides/Windows/Configure-RDP-to-connect-with-stored-credentials/index.html"
# Much more lines here

Then it executes emojize_commands.sh - which works perfectly as expected!

I'm trying to remove the need for it to use a temporary file to do all of this. I'd rather it just execute the commands directly instead of needing to echo them to a file first.

I've tried removing echo and >> ../emojize_commands.sh (plus all of the other lines of code) so I am left with just this:

find ./build/html -name '*.html' \( -exec ../emojize_pngorsvg.py \"{}\" \> \"{}.emojized\" \&\& rm \"{}\" \&\& mv \"{}.emojized\" \"{}\" \; \)

But when I run it I get a bunch of "file not found" errors!

How can I get find to work how I want please?

Answer 1

You could have the -exec run each line via bash -c instead of echoing it to a file to be run later. That would also enable (require, in fact) adjusting your quoting, ultimately allowing a simpler and clearer command. For example:

find ./build/html -name '*.html' \
  -exec bash -c '../emojize_pngorsvg.py "{}" > "{}.emojized" && mv "{}.emojized" "{}"' \;

I took the liberty of some additional simplification, too. Specifically, it is unnecessary to remove the original file before mving the new one to the original name. Also, the \( and \) in the original command served no useful purpose, so I dropped them.

Also, I generally pipe find's output into xargs instead of using an -exec action. You could consider that, but it probably does not provide enough advantage in this case to justify the conversion.

Answer 2

Spawn a shell and run those commands there.

find ./build/html -name '*.html' -exec sh -c '
for f; do
  ../emojize_pngorsvg.py "$f" >"$f.emojized" &&
  mv "$f.emojized" "$f"
done' sh {} +

Answer 3

Since you already know where the repository is and which type of file you are looking for, the use of find is not optimal. you could simply use a for loop instead:

for file in ./build/html/*.html;do python ../emojize_pngorsvg.py "${file}" > "${file}.emojized"; mv "${file}.emojized" "${file}"; done;

Answer 4

My chosen solution

I have ended up using and accepting @John Bollinger's answer, which I ran through ShellCheck and modified to account for SC2156 (insecure variable injection). The command iterates over the files only once and is easy to understand:

find ./build/html -name '*.html' -exec sh -c '../emojize_pngorsvg.py "$1" > "$1.emojized" && mv "$1.emojized" "$1"' sh {} \;

---

Other solutions

I also like this which is basically @jhnc's solution:

find ./build/html -name '*.html' -exec bash -c 'for f; do ../emojize_pngorsvg.py ${f} > ${f}.emojized && mv ${f}.emojized ${f}; done' -- {} +

It differs in that it uses ${f} instead of "$f".

However I'm not sure what to think of the 2nd loop that has been introduced (now a for as well as a find) - similarly for @oguz ismail's answer.

---

An alternative option could've been a for instead of a find, but unfortunately this does not support dir tree recursion.

---

I decided against using @jhnc's other solution of shopt -s globstar + for f in build/html/**/*.html; ... based upon an unrelated answer which warns that using glob can be slow with large dir trees - although it would seem that some performance testing is needed here, as it appears to be a gray area.

---

I didn't look into the parallelization solutions suggested @F. Hauri - Give Up GitHub's, but it does appear to be possible to use with my own use-case of this being part of a large DOCKERFILE container creation script, and it needing to block the thread until execution completes.

---

@ikegami's solution presents another possible way to redirect output, but I didn't have time to try it this time unfortunately.

---

Community

Please feel free to correct anything I may have misunderstood or not considered! Thank you very much to all who posted.