Announcement

Collapse
No announcement yet.

Using Regex in Bash

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Using Regex in Bash

    I always hated working with regex but seems to be the best tool for the job in this case, or atleast it seems like it to me. Please correct me if I am wrong. I want to be able to search a directory(but not subdirectories) for avi, mp4 or mkv files. If I stick to basic regex and search only for a single file type, say avi, it works. The problem is searching for multiple file types.

    Code:
    find . -prune -type f -regex '.*.avi'
    Works fine.

    These lines do not, but I am not sure why:
    Code:
    find . -prune -type f -regex (avi|mp4)
    find . -prune -type f -regex (avi|mp4|mkv)
    find . -prune -type f -regex (.*.avi|.*.mp4|.*.mkv)
    Last edited by Xplorer4x4; Sep 10, 2012, 04:50 PM.
    OS: Kubuntu 12.10/Windows 8
    CPU: Intel Core i7 2600K
    Motherboard: Gigabyte GA-Z77X-UD5H
    Memory: 2x4GB Corsair Dominator
    Graphics Card: MSI R7770
    Monitor: Dell 2208WFP
    Mouse: Mionix NAOS 5000
    PSU: Corsair 520HX
    Case: Thermaltake Mozart TX
    Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
    Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

    #2
    Without fiddling with bash escapes, you could just use find's "or" capability, as in
    Code:
    find * -prune -type f -regex '.*log' -o -regex '.*py'
    (find . -prune returns no results for me ... surely '.' is a directory and gets pruned before anything happens?)

    I think what's happening with your syntax is that (avi|mp4) matches avip4 and avmp4, ie the | does not imply grouping of what's around it. No ... not that ... this seems to work:
    Code:
    find * -prune -type f -regex '.*log\|.*py'
    Looks like you need to escape the | character even though it's quoted.
    Last edited by SecretCode; Sep 10, 2012, 11:44 AM.
    I'd rather be locked out than locked in.

    Comment


      #3
      Sorry I added prune manually to working code in the post. I had prune working, but forgot how now. My goal in using prune was to search . and . only excluding subdirectories. Maybe I misunderstood the basis of the prune command.
      OS: Kubuntu 12.10/Windows 8
      CPU: Intel Core i7 2600K
      Motherboard: Gigabyte GA-Z77X-UD5H
      Memory: 2x4GB Corsair Dominator
      Graphics Card: MSI R7770
      Monitor: Dell 2208WFP
      Mouse: Mionix NAOS 5000
      PSU: Corsair 520HX
      Case: Thermaltake Mozart TX
      Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
      Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

      Comment


        #4
        -prune might work if it was after the filters. Anyway, see my edit to my above post.
        I'd rather be locked out than locked in.

        Comment


          #5
          Seems prune works fine in your edit:
          Code:
          find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv'
          I put an avi files in a subfolder of the working dir and named it exclude.avi so that if it showed up in the results, I would automaticly catch it.

          I am curious, would "or" be lighter then using one long regex?

          And finally, to piece this all together, I need to ftp it to my NAS, but I am a bit stumped on how to handle this via a service menu. I assume I am going to be best off including the find and ftp commands together in to a bash script.

          nas.desktop file:
          Code:
          konsole --hold --workdir %f -e sh ~/.kde/share/kde4/services/ServiceMenus/wd-live/ftptowdlive-new.sh
          nas-new.sh
          Code:
          find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp '192.168.1.160'
          Now I realize the bash script is probably going to need a HereDoc, but for the time being, I am just trying to establish the basic ftp connection after the search but I get this
          Code:
          find: missing argument to `-exec'
          I just want to see if I understand what this is saying, that it is expecting the results from find to be directed to the command after -exec?

          So the final product needs to look something like this:
          Code:
          find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp 192.168.1.160 <<End-Of-Session
          binary
          cd /some/nas/folder/
          put "{}"
          bye
          End-Of-Session
          OS: Kubuntu 12.10/Windows 8
          CPU: Intel Core i7 2600K
          Motherboard: Gigabyte GA-Z77X-UD5H
          Memory: 2x4GB Corsair Dominator
          Graphics Card: MSI R7770
          Monitor: Dell 2208WFP
          Mouse: Mionix NAOS 5000
          PSU: Corsair 520HX
          Case: Thermaltake Mozart TX
          Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
          Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

          Comment


            #6
            Originally posted by Xplorer4x4 View Post
            Seems prune works fine in your edit:
            Code:
            find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv'
            Yes, but I had to change the '.' to '*' for it to work.

            On reflection I would have used -maxdepth 1 for this
            Code:
            find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv'
            Originally posted by Xplorer4x4 View Post
            I am curious, would "or" be lighter then using one long regex?
            Lighter on my head to think about it, that's all!

            Originally posted by Xplorer4x4 View Post
            And finally, to piece this all together, I need to ftp it to my NAS, but I am a bit stumped on how to handle this via a service menu. I assume I am going to be best off including the find and ftp commands together in to a bash script.

            nas.desktop file:
            Code:
            konsole --hold --workdir %f -e sh ~/.kde/share/kde4/services/ServiceMenus/wd-live/ftptowdlive-new.sh
            nas-new.sh
            Code:
            find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp '192.168.1.160'
            Now I realize the bash script is probably going to need a HereDoc, but for the time being, I am just trying to establish the basic ftp connection after the search but I get this
            Code:
            find: missing argument to `-exec'
            I just want to see if I understand what this is saying, that it is expecting the results from find to be directed to the command after -exec?

            So the final product needs to look something like this:
            Code:
            find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp 192.168.1.160 <<End-Of-Session
            binary
            cd /some/nas/folder/
            put "{}"
            bye
            End-Of-Session
            When you use -exec you need to include the filename at some point using {} - and end it with \; (a ; escaped from the shell) - you don't have that in your first example. In the second, you don't have the \; - not even sure if a multi-line script can be passed to -exec.

            Anyway, plain ftp is so old-fashioned. Why not use something like rsync? Saves all this 'binary' and 'cd' and 'bye'.

            Code:
            find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv' -exec rsync {} 192.168.1.60:/some/nas/folder/ \;
            (not tested)

            or even
            Code:
            rsync *.{avi,mp4,mkv} 192.168.1.60:/some/nas/folder/
            (also not tested)
            I'd rather be locked out than locked in.

            Comment


              #7
              Ftp may be old fashioned but I have generally preferred it over sftp, scp, etc as the overhead has always hurt my thorough put substantially. Although it has been awhile since I tried sftp over the intranet or internet and not at all on my current isp so I will try to dig in to it tomorrow to see how big a difference it makes.

              Sent from my DROID2 Global
              OS: Kubuntu 12.10/Windows 8
              CPU: Intel Core i7 2600K
              Motherboard: Gigabyte GA-Z77X-UD5H
              Memory: 2x4GB Corsair Dominator
              Graphics Card: MSI R7770
              Monitor: Dell 2208WFP
              Mouse: Mionix NAOS 5000
              PSU: Corsair 520HX
              Case: Thermaltake Mozart TX
              Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
              Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

              Comment


                #8
                I tried both sets of code and they both fail. The first:

                Code:
                Exec=konsole --hold --workdir %f -e find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv' -exec rsync {} 192.168.1.60:22/tmp/media/usb/USB2/c45f5474-ce19-4d5b-a483-672d1ace407f/1.New/ \;
                Error:
                Error processing Exec field in
                The error comes in a pop up, not konsole.
                The second:
                Code:
                Exec=konsole --hold --workdir %f -e rsync *.{avi,mp4,mkv} 192.168.1.160:/tmp/media/usb/USB2/c45f5474-ce19-4d5b-a483-672d1ace407f/1.New/
                Error:
                rsync: connection unexpectedly closed (0 bytes received so far) [sender]
                rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
                In the case of the second example, I also noticed it prompts me for a password, but if I ssh in manually from konsole, it handles authentication via private/public keys just fine.

                Besides authentication, I manually tested an SFTP transfer via Filezilla. Using SFTP I got speeds of less then 1MB/s. In fact it peaked at about 1MB/s. This is working with a roughly 1GB video file running about 45 min in length. At this speed, it will take roughly 30 minuets to transfer. When switching to FTP though, I see speeds of about 8MB/s average and peaks of 10MB/s(NAS is limited by a 100MB port). At this rate, SFTP is just unbearable.
                OS: Kubuntu 12.10/Windows 8
                CPU: Intel Core i7 2600K
                Motherboard: Gigabyte GA-Z77X-UD5H
                Memory: 2x4GB Corsair Dominator
                Graphics Card: MSI R7770
                Monitor: Dell 2208WFP
                Mouse: Mionix NAOS 5000
                PSU: Corsair 520HX
                Case: Thermaltake Mozart TX
                Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
                Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

                Comment


                  #9
                  Why are you running them via .desktop files? I can't see the advantage. If you need to, it may be simpler to put the entire command in a shell script and call that from the .desktop file.
                  eta - and definitely debug them from the console, not in desktop files

                  I'm really surprised at those ftp/sftp speed differences ... may have to do some tests.
                  I'd rather be locked out than locked in.

                  Comment


                    #10
                    It keeps the dir much cleaner and organized to have it in a .desktop file then clog it up with a .desktop file and at least one bash script for each desktop file. In this case I would have a desktop file plus about 5 bash scripts for all 5 parent directories on the NAS.

                    As for ssh vs ftp speeds, let me know how your tests go. I realize sftp(and similar) will have some overhead because of the encryption, but this seems very extreme especially given we are talking about intranet speeds.
                    OS: Kubuntu 12.10/Windows 8
                    CPU: Intel Core i7 2600K
                    Motherboard: Gigabyte GA-Z77X-UD5H
                    Memory: 2x4GB Corsair Dominator
                    Graphics Card: MSI R7770
                    Monitor: Dell 2208WFP
                    Mouse: Mionix NAOS 5000
                    PSU: Corsair 520HX
                    Case: Thermaltake Mozart TX
                    Cooling: Thermalright TRUE Black Ultra-120 eXtreme CPU Heatsink Rev C
                    Hard Drives: 1x180 GB Intel 330 SSD - 1xWD 1 TB Caviar Black - 1xWD 2 TB Caviar Green - 2xWD 3 TB Caviar Green

                    Comment

                    Working...
                    X