[SOLVED] Using zgrep to look at dpkg logs

[chimak111]

kubicle mentioned the use of zgrep here in the context of searching dpkg's logs.

I use this small yad script

#!/bin/bash
package_name="$(yad --center --width=300 --title "Search dpkg logs" --entry --entry-label="Enter the exact package_name:" 2>/dev/null)"
zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

which, thanks to input from jlittle and kubicle, is much better than what I posted originally!

#!/bin/bash
package_name="$(yad --center --width=300 --title "Search dpkg logs" --entry --entry-label="Enter the exact package_name:" 2>/dev/null)"
zgrep -e "status installed $package_name:" /var/log/dpkg.log* > /tmp/zgrep.txt
zgrep -e "status not-installed $package_name:" /var/log/dpkg.log* >> /tmp/zgrep.txt
cat /tmp/zgrep.txt | sort -k4
sleep 1s
rm /tmp/zgrep.txt

But
it requires the exact package name
and
the archived logs should still be in /var/log

To make the second condition more likely, I've bumped up the retention of logs (/var/log/apt/history.log and /var/log/dpkg.log) from the default rotate 12 to rotate 60.

[jlittle]

Thank you, I've always just used zcat whatever | grep, but that loses the file name, so I'll try to think of zgrep next time.

BTW, your script could use a sub-shell:

(  zgrep -e "status installed $package_name:"       /var/log/dpkg.log*;
zgrep -e "status not-installed $package_name:" /var/log/dpkg.log* ) |
sort -k4

but the two searches are so similar,

zgrep -Ee "status (not-)?installed $package_name:" /var/log/dpkg.log* |
sort -k4

(An egrep regex, -E, to avoid backslash-itis, they'd have to be doubled)

[chimak111]

zgrep -Ee "status (not-)?installed $package_name:" /var/log/dpkg.log* |
sort -k4

Thanks for that! Although just "-E" works for me. "-Ee" also works whereas just "-e" does nothing in the current context.

Now, I have

zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r

The sed bit allows sorting by date and time.

[jlittle]

I only left in the -e because you'd used it. I don't normally, with only one regex, I've seen scripts where the writers like -e with grep, maybe they're big sed users, where multiple -e phrases are common.

I'd wondered about the -k4. The fourth space-delimited field is always "installed" for me; maybe your date format has more spaces, mine is ISOish with one space.

[chimak111]

This is my "default" output where date is separated by a space from time but there's no space between the file/archive name and date:

$ zgrep -E "status (not-)?installed snapd:" /var/log/dpkg.log*
/var/log/dpkg.log:2020-05-13 21:53:19 status installed snapd:amd64 2.42.1+18.04
/var/log/dpkg.log:2020-05-13 21:53:36 status not-installed snapd:amd64 <none>
/var/log/dpkg.log.24.gz:2018-04-26 18:25:46 status installed snapd:amd64 2.32.5+18.04
/var/log/dpkg.log.24.gz:2018-05-15 22:48:04 status installed snapd:amd64 2.32.5+18.04
/var/log/dpkg.log.24.gz:2018-05-15 22:48:10 status not-installed snapd:amd64 <none>
/var/log/dpkg.log.5.gz:2019-12-11 16:33:51 status installed snapd:amd64 2.42.1+18.04
$

And this is the new output with a "sed" step (to introduce a space between the file/archive name and date) followed by "sort -k2,3 -r". Now the listing is sorted better, IMO.

$ yad-zgrep-dpkg-logs.sh
/var/log/dpkg.log: 2020-05-13 21:53:36 status not-installed snapd:amd64 <none>
/var/log/dpkg.log: 2020-05-13 21:53:19 status installed snapd:amd64 2.42.1+18.04
/var/log/dpkg.log.5.gz: 2019-12-11 16:33:51 status installed snapd:amd64 2.42.1+18.04
/var/log/dpkg.log.24.gz: 2018-05-15 22:48:10 status not-installed snapd:amd64 <none>
/var/log/dpkg.log.24.gz: 2018-05-15 22:48:04 status installed snapd:amd64 2.32.5+18.04
/var/log/dpkg.log.24.gz: 2018-04-26 18:25:46 status installed snapd:amd64 2.32.5+18.04
$

[kubicle]

You could pipe to column for a nicer output, as in:

zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

If you wish to do more advanced output formatting (and/or avoid long pipe chains), then "awk" is your tool (it will be able to all your sed/sort/column in one go, admittedly on a learning curve, though)

[chimak111]

You could pipe to column for a nicer output, as in:

zgrep -E "status (not-)?installed $package_name:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

If you wish to do more advanced output formatting (and/or avoid long pipe chains), then "awk" is your tool (it will be able to all your sed/sort/column in one go, admittedly on a learning curve, though)

Perfect! (Although your comment about using awk will gnaw away at me)

[Snowhog]

Your command string has $package_name (to be replaced by what ever package name you want to search against), but if I keep the $ (say, $snapd) no results are found. If I don't keep the $ (snapd), then the command string works.

[kubicle]

Your command string has $package_name (to be replaced by what ever package name you want to search against), but if I keep the $ (say, $snapd) no results are found. If I don't keep the $ (snapd), then the command string works.

The command is meant to be used in a script (see the first post) where "$package_name" is a variable.

[kubicle]

I don't really want to use yad, so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

#!/bin/bash
zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

Usage:
command <part_of_packagename>

Could be improved with some commandline options (for example, when you actually want to search for an exact match)

[chimak111]

I don't really want to use yad,

Since a terminal has to be open in any case I guess yad is unnecessary. But are there any other reasons for not wanting to use yad generally?

so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

#!/bin/bash
zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

Usage:
command <part_of_packagename>

Could be improved with some commandline options (for example, when you actually want to search for an exact match)

What does

[^ ]*

do? Does it stand for a string of any number of non-space characters non-greedily?

In the specific context of "status (not-)?installed <string>:",

.*${1}.*

seems to work just as well.

[kubicle]

Since a terminal has to be open in any case I guess yad is unnecessary. But are there any other reasons for not wanting to use yad generally?

The main reason is that since it's a cli tool, the switch to gui input is not really to my liking (and if I was to use a gui dialog, I'd prefer "kdialog")

What does

[^ ]*

do? Does it stand for a string of any number of non-space characters non-greedily?

Yes, it'll match any number of non-space chars

In the specific context of "status (not-)?installed <string>:",

.*${1}.*

seems to work just as well.

In that specific string it shouldn't make much of difference, it's just a habit of mine to avoid .* wildcards when possible in long line REGEXes, ít's somewhat prone to extensively wide matches. But since the search string is so exact in this script, it shouldn't make a difference in this case. Still, it's sort of future-proofing in case I wish to modify the script more.

[chimak111]

Re. kdialog, I too would prefer it, but it doesn't seem possible to format the windows as well as is possible with yad.

And yad has this cute color picker (accessed with yad --color) although I came across mention of a cli color picker which is not available for Ubuntu. I'll open a new thread on that.

[chimak111]

I added a "test" to ensure a search string is provided:

#!/bin/bash
echo "enter the_string"
read the_string
[ "$the_string" ] || { echo "You forgot the search string!" ; exit 1 ; }
zgrep -E "status (not-)?installed [^ ]*$the_string[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

[chimak111]

I don't really want to use yad, so I made a one-liner that accepts the package name as argument (and doesn't need an exact match, partial matches should be fine)

#!/bin/bash
zgrep -E "status (not-)?installed [^ ]*${1}[^ ]*:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

Usage:
command <part_of_packagename>

Could be improved with some commandline options (for example, when you actually want to search for an exact match)

I modified the code further to

• exit the script if the user presses Enter without providing a search string
• allow for the use of an exact match or part of the package name. For example, entering snapd will provide only snapd results whereas entering .*snapd.* will provide results for libsnapd-glib1 as well

#!/bin/bash

echo "enter the package name;"; echo "use .* as prefix/suffix if the exact package name is not known"
read the_string
[ "$the_string" ] || { echo "You forgot the search string!" ; exit 1 ; }
zgrep -E "status (not-)?installed $the_string:" /var/log/dpkg.log* | sed 's/:/: /' | sort -k2,3 -r | column -t

---

However, a notable failure of using zgrep on dkpg logs appears to relate to the very first (oldest) dpkg.log, the one recording the initial install.

For example, the code doesn't show status not-installed for packages such as casper, ubiquity-slideshow-kde, and many more that were automatically purged from the system during the installation process.

I've noted this on Kubuntu 18.04. I'll check in some *buntu 20.04 VMs as well.

---

Okay, it's the same in a VM of Kubuntu 20.04.It's as if dpkg is unaware of what is recorded in the corresponding /var/log/apt/history.log in the section beginning with

Start-Date: 2020-01-23  16:01:05
Requested-By: kubuntu (999)
Purge: ...