Extract bot agents from Nginx access log
mouse 171 · person cloud · link
HowTo
Nginx
Last update
2024-06-07
2024
06-07
« — » 
1
2
3
# head access.log
85.208.96.209 - - [2024-06-01T17:32:29+02:00]  200 "GET /... HTTP/1.1" 8381 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" "-"
...

1
cat access.log | sed -r 's/.+\[(.{19}).+compatible; *([^ ;]+).+/\1\t\2/ ; s/.+\[(.{19}).+ "-" "([^ ]+) .+/\1\t\2/' > agents.log

1
2
3
4
5
# head agents.log
2024-06-01T17:32:24 PetalBot
2024-06-01T17:32:29 SemrushBot/7~bl
2024-06-01T17:32:30 AhrefsBot/7.0
...