A FAST offline archive index/search utility.

[rainbowsally]

A FAST offline archive index/search utility.

In this experiment let's say we want to create an offline pool of deb files we can look through for filenames. Perhaps we want to know what supplies a specific <library>.so file for programming or development purposes.

What if it doesn't exist? What if we need to create a dev pkg for this one lib containing only a symlink to the one with the numeric extension?

Of course we could sudo and create the link manually. But how many times? And how could we remove things we add manually if we discover we made a mistake?

So this may be useful if you are interested in developing your own software and if nothing else it may have a few script tricks in it you can add to your bash (or other?) snippets file.

I have tried looking up filenames using dpkg and dpkg-deb utilities and it has taken 20 to 200 minutes to find all references to "kdev" in the DVD archives.

The Experiment: (not dangerous.)

Let's build the index and then try the same thing with grep. [Skip down to the results below the code section for a sec. -added after testing]

Start by making a symlink to the 'pool' you want to index in a folder you can test in. No SUDO NEEDED. NONE WANTED. We can do this on our Desktop and if you want to keep the index move it later.

Here's the utility which I have included in my ~/bin folder which is in my path, but you can put it in your working dir for testing.

[Possible bug. We assume all the deb files have unique names because we will put all the list files in a flat directory changing the list file's extension from *.deb to *.files. If EXACT duplicate names occur, the first will be overwritten by the second by dfl.create. This problem is extremely unlikely but it is possible.]

##############################################################
# to load the definitions in bash type 'source debfilelist.sub'
# type dfl.help for synopsis of functions available.
##############################################################
# debfilelist.sub (class dfl)
# needs symlink to archive 'pool' directory in current folder.
# the list directory is relocatable but this program needs to
# run where the user has read-write privileges.

DFL_ARCHIVE=main
DFL_LISTDIR=$DFL_ARCHIVE"-list"

##############################################################
## subroutines for main functions

# true (-L) if file or directory exists
exists()
{
 if [ -x $1 ]; then
  return 0; 
 else 
  return 1; 
 fi
}

dfl.msg() # msg
{
 kdialog --title "Deb File List Creator" --msgbox "$@"
}

##############################################################
## main functions

dfl.init()
{
 if ! exists pool; then
 dfl.msg "
 Can't find the symlink to 'pool' in the current folder. 
 
 We need a temporary link to the pool where '$DFL_ARCHIVE'
 is so we can generate the '$DFL_ARCHIVE-lists' in the current
 folder.
 
 Abort now. 
 "
 return
 fi
 mkdir -p $DFL_LISTDIR
 if ! exists $DFL_LISTDIR; then
 dfl.msg "
 Can't create '$DFL_LISTDIR'
 
 You may need read/write permissions for this folder, or you 
 can build the lists in one you can write in and move the 
 lists later.
 
 Abort now
 "
 return
 fi
 # ok if we get this far
 dfl.msg "
 Looks ok. 

 Type dfl.help at any time for the function quick ref
 "
}

# print out filename minus path and if it has a link, print that
# also, dereferenced (i.e., include the path of the referenced file
# Note: this will only work for files linked in the same directory
# such as shared libs. Used only by get_filelist.
_dfl_print_a_file() # fullpath [link]
{
 # print filename
 echo `basename $1`
 # if it has a link print that with the path prefixed
 if [ -n "$2" ] ;then
# echo `dirname $1`"/$2";  
 echo `dirname $2`;  
 fi
}


dfl.get_filelist() # debfile -- displays file list
{  
 local p=`dpkg-deb --contents $1 | sed 's|.*\./|/|; /\/$/d; s| -> |\n|'`
 for i in $p; do
  _print_a_file $i
 done
}


dfl.create()
{
 local listname
 for i in `find pool/$DFL_ARCHIVE/* -type f -name *.deb`; do
  listname=`basename $i .deb`".files"
  echo creating $listname
  get_filelist $i > $DFL_LISTDIR/$listname
 done
} 

# time to create list of 300,000 files on my system
# 21 minutes

dfl.help()
{
echo "
 commands:
  dfl.init		- creates list directory and checks for pool
  dfl.create		- creates a new set of file lists
  dfl.update		- creates only new lists
  dfl.clean	-	- removes lists that are no longer valid
 "
}

dfl.help

On my system:
Time to create the index: 21 minutes. Size 4.7 megs (15 megs on disk)
Time to gzip it, less than a second. Size 1.3 megs
Time to xz it, 5 seconds. Size 786K (!!!!)
Time to unpack either one, less than a second.

Time to find ALL references to 'kdev' in main-list: 0.6 seconds.

Here's the first listed.

main-list/app-install-data_0.12.04_all.files:kdevelop:kde4__kdevelop.desktop

And here's the last (1,198th in the listing).

main-list/util-linux_2.19.1-2ubuntu3_i386.files:blockdev.8.gz

If we don't need the names of the files in the packages we can create a list of only the packages to check out.

dfl.locate() # archive "str"
{
 local archive=$1
 local str=$2
 local list=""
 # remove from colon to end
 list=`grep $str $archive"-list | sed 's|:.*||'"
 echo "$list" | uniq
}

With this utility I found what I needed. I needed the kde3 support libs which was in a package found here.

main-list/kdevplatform3-libs_1.2.3-0ubuntu1_i386.files

Note that many of the dependencies may not be evident until run-time when you make your own packages.

One other missing dependency was libpcre.so.0 which I symlinked to libpcre.so.3 for now and will add to my kdevevlop3 package (though it should probably be in kdevplatform3-libs).

I don't know why the DVD has the kdevplatform3 package and not kdevelop3 itself, but I'm glad it had it.

Someone must feel the same way I do about kdevelop4?

I'm using a version of kdevelop3 from <closed>suse 10.0, converted to a deb package with a utility based on one I posted recently in these forums.

http://kubuntuforums.net/forums/inde...opic=3119796.0

The version of that utility I'm using now creates a control template and pre/post installation scripts. Now with this lookup utility it will be able to look up packages that contain dependencies that can be derived from using ldd on the executables to see what they need before installation.

Will update that post soon, if all goes well.