Find all unique url's from Apache log files

I needed to build a list of all unique hits that had been made on a website in Apache.

Here's what I came up with using awk and sed.  This should match any HTTP 2xx or 3xx requests and strip of any GET request parameters.


awk '$9 ~/^(2|3)/ {print $7}' somelogs* | sed 's/\?.*$//' | sort | uniq

In-place file editing with sed

This took me a while to work out why this would not work. When using the -i option with sed you need to specify a backup extension to be used (at least on my 10.5 mac you do!)

e.g.

sed -i '' -e 's/something/something else/g' -e 's/again/again again/g' *.py