Is good way to show hourly hits count by searching apache/nginx logs in bash?

Issue

I want to setup a scheduled daily job in my app server, it will send me a report of hourly hits filtered by the status code.
For example, I want to receipt these reports everyday,

#hourly report of 200 status code

200_report.txt

2022-07-18T01:00:00 50
2022-07-18T02:00:00 100
2022-07-18T03:00:00 200
...

40x_reports and 50x_reports give me the similar content while filtered by 40x codes and 50x codes.

I dont want to use any complex tool, it would better I can achieve this by using one or a few bash command.

Any idea?

Thanks

Solution

I think rquery is the exact command you are looking for. You will be able to filter and group the content parsed by regex pattern.

[ rquery]$ ./rq -q "parse /\\\"(?P<origip>[^\n]*)\\\" (?P<host>\S+) (\S+) (?P<user>\S+) \[(?P<time>[^\n]+)\] \\\"(?P<request>[^\n]*)\\\" (?P<status>[0-9]+) (?P<size>\S+) \\\"(?P<referrer>[^\n]*)\\\" \\\"(?P<agent>[^\n]*)\\\"/|filter status=200 | select truncdate(time,3600), count(1) | group truncdate(time,3600) | sort truncdate(time,3600)" logs/access.log
29/Jun/2022:10:00:00 +1000      11
29/Jun/2022:11:00:00 +1000      210
29/Jun/2022:12:00:00 +1000      43
29/Jun/2022:14:00:00 +1000      41
29/Jun/2022:15:00:00 +1000      145
[ rquery]$ ./rq -q "parse /\\\"(?P<origip>[^\n]*)\\\" (?P<host>\S+) (\S+) (?P<user>\S+) \[(?P<time>[^\n]+)\] \\\"(?P<request>[^\n]*)\\\" (?P<status>[0-9]+) (?P<size>\S+) \\\"(?P<referrer>[^\n]*)\\\" \\\"(?P<agent>[^\n]*)\\\"/|filter status like '50*' | select truncdate(time,3600), count(1) | group truncdate(time,3600) | sort truncdate(time,3600)" logs/access.log
29/Jun/2022:11:00:00 +1000      3
29/Jun/2022:12:00:00 +1000      2

Answered By – WeDBA

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published