Announcement

Collapse
No announcement yet.

getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

    After upgrading to 10.04LTS i noticed that nepomukservices, strigi? and a new thing called virtuoso-t once again was more or less stopping my computer with a combined near 100% CPU load. I am not very surprised by this, as it has kind of been the norm on the last releases (all KDE 4 ones, right?). To get my desktop environment to communicate with the user again at a reasonable efficency, I have had to disable the whole ensemble of services and gizmos as far as is possible from the system settings panel. I could leave it at that.

    Only seeing that it is still there, in an LTS release by default (after near complete uselessness in n releases) indicate that someone somewhere must have found it useful - and working at least partly to the intentions, which are quite visionary.

    So I wondered, is there anyhow I can tune this beast to perform well on my computer?

    I have a toshiba laptop which already have reached the noble age of 3 years old. It has a dual-core processor although it was never one of the really strong ones. Also, my hard-disk is encrypted, and I do believe this may be part of the problem, as hard-disk scanning includes decryption and writing (the database) necessarily adds encryption to the processors to-do list.

    The symptoms I experience is frequent (at least once per hour, but I think more often) heavy hits to system performance because all the nepomukservices, virtuoso-t, (and strigi?) with associated encrypt / decrypt surge to close to 100%, which makes my user interface practically stop, until these processes have finished their job (I estimate 5-10 minutes for this each time, but have not done the statistics).

    My idea is that if it somehow was possible to limit the intensity of CPU craving by the sum of these associated processes and desprioritize them, and maybe do the scans less frequently, it may be possible to have a useable desktop search and a practically workable computer at the same time.

    Does anyone have any success stories I can learn from, along the lines of the ideas I have posted above, or completely different ways that the desktop search gizmos can actually get to work without paralyzing the computer? I am willing to give it one more shot if I can get some leads.

    #2
    Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

    Well, I can just tell you, that I also have a Toshiba laptop (see my sig line), and have all three - nepomuk, strigi, and akonadi running. My laptop suffers no performance degradation with all of these running. And, I have full compositing 3D desktop effects enabled.
    Windows no longer obstructs my view.
    Using Kubuntu Linux since March 23, 2007.
    "It is a capital mistake to theorize before one has data." - Sherlock Holmes

    Comment


      #3
      Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

      Earlier:

      > Topic: Is Nepomuk (search in Dolphin) completely useless?
      > Topic: nepomuk db
      > Topic: Unwanted applets in taskbar


      Summary: The Nepomuk is working here OK, but it is a tool that is under heavy development. You need to tune the Nepomuk settings so that it is not indexing everything.
      Before you edit, BACKUP !

      Why there are dead links ?
      1. Thread: Please explain how to access old kubuntu forum posts
      2. Thread: Lost Information

      Comment


        #4
        Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

        Originally posted by Snowhog
        Well, I can just tell you, that I also have a Toshiba laptop (see my sig line), and have all three - nepomuk, strigi, and akonadi running. My laptop suffers no performance degradation with all of these running. And, I have full compositing 3D desktop effects enabled.
        Telling me you succeeded without leaving clues? Is this supposed to be a challenge? I thought this was the one with the solutions :P
        (I am on my way to work and dont have time to look into doing anything real until later )

        @Rog131 I actually think I read most of those before posting but I will try again

        Comment


          #5
          Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

          @He-111: Pardon me for jumping in, but I noticed from one of your earlier posts in this thread, that you are using file encryption. Strigi is supposed to be "a program for fast indexing and searching your personal data". Virtuoso-nepomuk is the database server for the searcher. I suspect that the need to decrypt your files as part of the periodic index updating operation is at the root of your problem.

          As is usually the case, there is a trade-off between security and convenience. If you REALLY need to have the security of encrypted files on your laptop, I would suggest that you disable file-indexing. Do you really use file searching that often? Alternatively, you could leave the vast majority of your files unencrypted and reserve encryption for the places where it matters. In point of fact, it is a serious breach of security to have critical files on a laptop. A friend of mine used to say, "Never trust a computer you (or somebody else) can pick up!"

          Comment


            #6
            Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

            IF you wind up turning desktop search (strigi) off you may whant to consider using the " locate " comand
            it's WAY faster eneyway.

            for exampel just try a
            Code:
            locate video | less
            a yes it is wise to pipe it thrugh less/more as in this exampel it will spit out severall hundred mabe thousend lines of file's that have video in the name
            and you want to be abel to scrole thrugh it........and it spit it out in less than 1 second more like as soon as I hit enter.

            this came up just as quick

            vinny@desktop:~$ locate music.txt | less
            /home/vinny/music.txt
            /usr/share/games/egoboo/modules/advent.mod/gamedat/igmusic.txt
            /usr/share/games/egoboo/modules/healer.mod/gamedat/igmusic.txt
            /usr/share/games/egoboo/modules/palsand.mod/gamedat/igmusic.txt
            /usr/share/games/egoboo/modules/rogue.mod/gamedat/igmusic.txt
            (END)
            as you can see it's seartch can be fuzzey used like this but a few extra options and

            vinny@desktop:~$ locate -b '\music.txt'
            /home/vinny/music.txt
            vinny@desktop:~$
            if you wernt aware of this do a

            Code:
            man locate
            for a list of the options and usage

            it's allreadey a part of the system and will use no more resorses than your box siting idel





            VINNY
            i7 4core HT 8MB L3 2.9GHz
            16GB RAM
            Nvidia GTX 860M 4GB RAM 1152 cuda cores

            Comment


              #7
              Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

              I strongly sympathize with He-111! I have the same experience ever since I moved to Kubuntu (8.10). I've started a similar thread on Akonadi a year ago (but it seems to work for Contacts now, thanks to Snowhog).

              I'm not using any of these technologies and I don't have any need to do so. But it appears somewhat odd, when the Devs in all there wisdom time and again decide to include a seemingly useless (and resource-hungy) component in every new release (and enable them by default). Makes you wonder if there might be something to it after all...

              Like He-111, I also have hard disk encryption, and if that's the problem, that's fine and I will keep them disabled. (In that case it may be a smart move to disable file indexing by default when HD encryption is present...)

              @askrieger: unfortunately my 3 year old laptop is the only computer I actually own... I have a pretty powerful workstation but it is strictly speaking not mine, and I'm also using it as a server. IMHO my personal files are better off on my much better protected laptop.

              Chopstick

              Comment


                #8
                Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                Take a look at this thread. It gives a good impression of the improvements since Interpid Ibex (8.10).

                From my point of view, it must be pretty close to zero improvement because the problem is exactly the same. Even back then, the claim (by those closer to buntuisatwah) was that consideration must be given to nepomuk being in heavy development, and as not unusual, it was mentioned "it can be disabled, you know", etc.

                I still find it strange that this tool was considered apt for distribution in an LTS release. And I still wonder how much "heavy development"is needed before it stops eating my CPU completely, freezing my computer, etc - because at the end of the day that is simply about setting priorities right, right?

                That is where I get a problem. Good KDE style should leave some more control to the user. We are getting only the big red button with the "STOP! DON'T THINK ABOUT STARTING THIS PROCESS" letters on it (or maybe just STOP ) when what we need is something more like a throttle. A way to regulate the amount of gas your machine spends on this process. Should be doable with the resources available to a software under "heavy development" that apparently enjoys corporate sponsorship.

                PS> I limited the search context severely, set the max available memory to something surprisingly high (like 100xdefault) and it works a little bit better now. See I am not whining, I am offering constructive criticism and looking forward to the release when the nepomuk heavy development phase is over (Kubuntu Zealous Zebra gogogogogo! )

                Comment


                  #9
                  Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                  Thanks for pointing me to this thread, He-111! It was interesting, but only confirmed what I was thinking.
                  I also totally agree with what you say above, I don't see how Nepomuk is fit for an LTS release, and I don't understand why you cannot simply throttle it (or will it be so utterly useless if it only gets 10% CPU time...); that would really be the KDE way.
                  I also read the Summary on the Nepomuk website... at best it sounds like a research proposal and at worse it sounds like an advertisement ...

                  As for these Desktop search engines... I don't use them at all. The essential difference between the Internet and my computer is that the Internet is a chaotic mess that has to be navigated somehow, while my computer is managed by myself, and I usually know exactly where my files are.

                  The only use case for tagging and searching I can imagine are libraries for media/music and documents/papers. My music library is still sorted folder-wise (and rarely tagged), and for papers I recently started to use Mendeley (any opinions on that?).

                  Comment


                    #10
                    Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                    Originally posted by Chopstick

                    I'm not using any of these technologies and I don't have any need to do so. But it appears somewhat odd, when the Devs in all there wisdom time and again decide to include a seemingly useless (and resource-hungy) component in every new release (and enable them by default). Makes you wonder if there might be something to it after all...
                    +1

                    Even though I have tens of thousands of documents, images, and music files on my desktop system, they are organized in a hierarchical directory structure that lets me find what I need. This was the only sane approach to accumulating user data in the times prior to "desktop search" capabilities. Therefore, strigi, nepomuk, and akonadi are solutions to a problem I don't have, and don't ever expect to have. I originally suspected that it was simply a "solution" for people who can't get organized.

                    However, I recently came across a little dissertation that does make a plausible case for a productive application of this capability:

                    http://sidux.com/index.php?module=Ne...isplay&sid=585

                    I'm not a recipient of zillions of e-mail messages reporting technical problems, that I would wish to search in the future, so it's not applicable to my life. But at least I can see a legitimate purpose now.

                    Comment


                      #11
                      Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                      Originally posted by dibl

                      Even though I have tens of thousands of documents, images, and music files on my desktop system, they are organized in a hierarchical directory structure that lets me find what I need. This was the only sane approach to accumulating user data in the times prior to "desktop search" capabilities. Therefore, strigi, nepomuk, and akonadi are solutions to a problem I don't have, and don't ever expect to have. I originally suspected that it was simply a "solution" for people who can't get organized.

                      However, I recently came across a little dissertation that does make a plausible case for a productive application of this capability:

                      http://sidux.com/index.php?module=Ne...isplay&sid=585

                      I'm not a recipient of zillions of e-mail messages reporting technical problems, that I would wish to search in the future, so it's not applicable to my life. But at least I can see a legitimate purpose now.
                      As a student in EE, I use the internet excessively to access information, which often come in form of pdf. Most of them will be saved in appropriate folders according to the course they belong to. So at exam time, I am facing 5-6 folders each filled with several large pdfs.
                      There is only so much info you can put into a files' name, and so it gets impossible to just quickly look something up. Consider a file that is dealing with several subjects, or has magically moved to the wrong folder, or maybe I'd be able to find information in files from previous courses. Maybe I want to find a MATLAB or LaTex file that has a complicated and rarely used command in it which I'd like to use again and somehow know I did this before...
                      All these problems can be solved by Strigi/Nepomuk, it could show me all files dealing with a specific subject in a heartbeat. I absolutely loved it for the 30 min it usually worked before I had to spent another 2h+ rebuilding the index I hope KDE4.5 fixes that (of course by now most of the exams are already over...)

                      So, count me as a fan.

                      Comment


                        #12
                        Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                        Yes, there are cases where desktop search is useful, but does this justify the inclusion and default activation of desktop search whether one needs it or not?

                        Personally, I do not encrypt my files. Perhaps, I should, but I trust my firewall (foolish me) and the fact that as a Linux user, I'm not an easy target. On one occasion, my firewall was down for a few days. I took my wife's windoze box off the net and forced here to do her communications through my computer. In two or three days I logged an astounding number of unsuccessful attempts to penetrate my system. None of them were successful

                        I do organize my data in a hierarchy so that, even though I have some cognitive difficulties, I can find things fairly easily. When I do need to use Tools>"Find File" in Konq, I just wait it out, but this is a rare occurence because there are 18 first generation subdirectories under Documents and most of them have subdirectories as well. So, if a seventy year old geezer, with officially diagnosed memory problems does not need desktop search, does the average user?

                        Comment


                          #13
                          Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                          I agree with THHHB that media and document libraries are reasonable use cases for file indexing, in particular for music and scientific papers (if you read my previous post). But it is good that you mention it again THHHB - I will give file indexing for my library another try with KDE4.5.

                          For my scientific library I recently started to use Mendeley which allows me to tag and add meta data and bibliographic info to documents, which is already half way there, only the in-file search function is missing; so I have to rely on good tagging, which is some work (but comparatively little compared to the time it takes to read and understand the paper).
                          (Btw. if anyone can recommend an open-source alternative to Mendeley, I would be interested to hear.)

                          Chopstick

                          Comment


                            #14
                            Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                            Two comments re. Mendeley:
                            (1) There are other, similar programs available for Linux. Please see this Wikipedia article. You might find something else you like better (or not).

                            (2) Back in the previous century, I used to keep references to all the papers I cited in my publications in bibtex format in a single file (with the imaginative title "refs.tex"). Whenever, I wrote a paper, I would add additional items to the file. I could then search the file in EMACS, for the entry that I wanted. Of course this had the drawback that the system was entirely dependent on my memory (which I could usually rely on in those days) for at least one author or title keyword, e.g. "Didn't one of the guys from the Center for Advanced Trivia at Enormous State University write a paper on that last year?"

                            Comment


                              #15
                              Re: getting that desktop search thing to actually work (Nepomuk/virtuoso-t/strigi?)

                              Originally posted by THHHB
                              So, count me as a fan.
                              You may find this of interest: http://doc4.mandriva.org/bin/view/labs/Blog?language=en, even though it has nothing to do with Kubuntu.

                              Regards .....
                              Kubuntu User #9802
                              Linux Registered User #357086

                              Comment

                              Working...
                              X