Announcement

**Snowhog** · Nov 19, 2018, 09:27 AM

Originally posted by MoonRise View Post

OK, so I think I might know the issue and really have no clue how to fix.

This is a script and variables start off with a "$".

See my code:

Code:

awk -F $'\t' '$6 != ""' $myFl >> $myFlf

See the "$6"? It is being interpreted as a variable and therefore comes up empty. AWK in konsole knows how to interpret that (Column 6) because it isn't script based and not interpreted as variable. In the script it is and therefore is blank. So, what to do!?!?

I'm so fried now I can't think of how to force it to represent itself as is.

Send me a sample of the file you are using with this script. I'll play around with it (if you don't mind).

**MoonRise** · Nov 19, 2018, 12:13 PM

Sure. See attached.

Attached Files

EDI_867_20181118182952.zip (686 Bytes, 11 views)

**Snowhog** · Nov 19, 2018, 12:27 PM

Got it.

Clarification: When I open the extracted .txt file and expand the window so there isn't any line wrapping, I see extra white spaces between some columns on some of the lines. I'm assuming that the columns starting position should be the same for every line, but in the context of what you are doing with the line contents using awk, the extra spacing won't/doesn't matter, correct?

This is the extracted .txt file as I see it in a terminal using cat:

Pauls-iMac

ownloads paul$ cat EDI_867_20181118182952.TXT

Code:

ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922        20171014    00999111101    20171014    0926586923    20171010    1    44.7    Lst East SUPP CH SRV EDI  #999    9920 W. EntrtyPLEX DR    KAYSVILLE    UT    84037-959    EA    990003055
ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926627797    20171010    4    178.8    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926751452    20171012    1    44.7    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926516685    20171007    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926678606    20171011    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926515844    20171007    2    124.9    SUPLYY DEPTSTRE CHAN #9    9900 E SOMEWHR AVENUE    FRESNO    CA    93720    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922        20171014    00999111101    20171014    0926584818    20171010    1    44.7    SUPLYY DEPTSTRE CHAIN #9    9900 E SOMEWHR AVENUE, STE 9 ESTON    FL    33331-350    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    YZ9999998    20171014    00999113401    20171014    0926715863    20171012    3    256.5    SUPLYY DEPTSTRE CHAIN #9 LLC    900 SOMEWHR, STE 9 ESTON    MIAMI    FL    33126    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    ZZ9999999    20171014    00999111101    20171014    0926548404    20171009    1    44.7    SUPPLY DEPTSTRE CHAINS LLC    900 SOMEWHER AVE    MIAMI    FL    33155    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    XY9999997    20171014    00999113201    20171014    0926712287    20171012    2    110.1    Lst East SUPPL CH SRV EDI  #9    9920 West EntrtyPLEX RD    OCOEE    FL    34761    EA    990003055

Pauls-iMac

ownloads paul$

**MoonRise** · Nov 19, 2018, 01:04 PM

Originally posted by Snowhog View Post

Got it.

Clarification: When I open the extracted .txt file and expand the window so there isn't any line wrapping, I see extra white spaces between some columns on some of the lines. I'm assuming that the columns starting position should be the same for every line, but in the context of what you are doing with the line contents using awk, the extra spacing won't/doesn't matter, correct?

This is the extracted .txt file as I see it in a terminal using cat:

Pauls-iMac

ownloads paul$ cat EDI_867_20181118182952.TXT

Code:

ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922        20171014    00999111101    20171014    0926586923    20171010    1    44.7    Lst East SUPP CH SRV EDI  #999    9920 W. EntrtyPLEX DR    KAYSVILLE    UT    84037-959    EA    990003055
ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926627797    20171010    4    178.8    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926751452    20171012    1    44.7    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926516685    20171007    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926678606    20171011    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926515844    20171007    2    124.9    SUPLYY DEPTSTRE CHAN #9    9900 E SOMEWHR AVENUE    FRESNO    CA    93720    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922        20171014    00999111101    20171014    0926584818    20171010    1    44.7    SUPLYY DEPTSTRE CHAIN #9    9900 E SOMEWHR AVENUE, STE 9 ESTON    FL    33331-350    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    YZ9999998    20171014    00999113401    20171014    0926715863    20171012    3    256.5    SUPLYY DEPTSTRE CHAIN #9 LLC    900 SOMEWHR, STE 9 ESTON    MIAMI    FL    33126    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    ZZ9999999    20171014    00999111101    20171014    0926548404    20171009    1    44.7    SUPPLY DEPTSTRE CHAINS LLC    900 SOMEWHER AVE    MIAMI    FL    33155    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    XY9999997    20171014    00999113201    20171014    0926712287    20171012    2    110.1    Lst East SUPPL CH SRV EDI  #9    9920 West EntrtyPLEX RD    OCOEE    FL    34761    EA    990003055

Pauls-iMac

ownloads paul$

Correct, some line could have extra white spaces and are necessary, one reason the file is TAB delimited. However, as you see with Column 3 it is totally blank BUT that column has to exist and I don't care if it has something or not. When we get to Column 6 is where I do care if something is there. If a row at column 6 is blank I don't want to include it in the output. Hope that explains better.

You can run that AWK command from console, you will have to explicitly state the file name of course, and it will do just that. I was initially ecstatic about it until I tried to put in a bash script.

One of the problems is I am good at coding but I don't get to do it often so some things I have to go over again. You should see my Note folder of all things programming. Didn't help here though.

**MoonRise** · Nov 19, 2018, 01:13 PM

Here is the output when run from konsole:

Code:

ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926627797    20171010    4    178.8    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999997    00500011        XYZ  DC SALT LAKE CITY    0050001131 20171014 0922    XY9999997    20171014    00999111101    20171014    0926751452    20171012    1    44.7    SUPLYY DEPTSTRE #9    9900 WEST SOMEWHER AVENUE    NAMPA    ID    83687    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926516685    20171007    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926678606    20171011    1    62.45    Lst East SUPPL CH SRV EDI  #99    9920 West EntrtyPLEX DR.    VISALIA    CA    93291-623    EA    990003055
ZZ9999998    00500011        YZZ  DC SACRAMENTO    0050001131 20171014 0922    YZ9999998    20171014    00999112501    20171014    0926515844    20171007    2    124.9    SUPLYY DEPTSTRE CHAN #9    9900 E SOMEWHR AVENUE    FRESNO    CA    93720    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    YZ9999998    20171014    00999113401    20171014    0926715863    20171012    3    256.5    SUPLYY DEPTSTRE CHAIN #9 LLC    900 SOMEWHR, STE 9 ESTON    MIAMI    FL    33126    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    ZZ9999999    20171014    00999111101    20171014    0926548404    20171009    1    44.7    SUPPLY DEPTSTRE CHAINS LLC    900 SOMEWHER AVE    MIAMI    FL    33155    EA    990003055
ZZ9999999    00500011        ZZZ DC ORLANDO VISTA    0050001131 20171014 0922    XY9999997    20171014    00999113201    20171014    0926712287    20171012    2    110.1    Lst East SUPPL CH SRV EDI  #9    9920 West EntrtyPLEX RD    OCOEE    FL    34761    EA    990003055

**Snowhog** · Nov 19, 2018, 02:06 PM

Originally posted by MoonRise View Post

Correct, some line could have extra white spaces and are necessary, one reason the file is TAB delimited. However, as you see with Column 3 it is totally blank BUT that column has to exist and I don't care if it has something or not. When we get to Column 6 is where I do care if something is there. If a row at column 6 is blank I don't want to include it in the output. Hope that explains better.

It does. Thank you.

The example file you provided doesn't have any lines where column 6 contains 'no data'. So that I'm crystal clear (a necessary condition!), column 6 (in every line provided) contains the value 20171014. Is that correct?

**Snowhog** · Nov 19, 2018, 02:34 PM

From my brother-in-law:

change awk command line
from ... 'if($6 != ...
to ... 'if(\$6 != ...

escape the dollar sign ($) so that the command passed to awk by the shell passes the literal $6, instead of the shell's substitution of the sixth argument on the bash command line.

Alternatively, put the awk command(s) in a separate file for awk to execute, rather than passing the awk command(s) on the command line itself. Use an editor (not the bash "echo" command) to build this awk script file.

Assuming the file "myFl.awk" contains the line (created via an editor, (ie, vi):

{if($6 != "") {print}}

Then the bash command becomes:

awk -F $'\t' -f myFl.awk $myFl >> $myFlf

**MoonRise** · Nov 19, 2018, 03:47 PM

Well; the first part I've done that and get the below:

Code:

awk: cmd. line:1: {if(\$6 != "") {print}}
awk: cmd. line:1:     ^ backslash not last character on line
awk: cmd. line:1: {if(\$6 != "") {print}}
awk: cmd. line:1:     ^ syntax error

Doing the command file, no errors are thrown but I still get an empty file. So, on this one, I have the awk call with the -f but with the path to the .awk file ie /home/moonrise/Documents/deliv867/myFl.awk. Is that allowed? Seems I've seen that is correct but just need to ask.

**Snowhog** · Nov 19, 2018, 04:18 PM

Yeah, we figured that out as well.

Try this modified script:

Code:

#! /bin/sh
#


cd /home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI
for file in /home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI/*; do
myDATE=`date '+%Y%m%d'`
myTIME=`date '+%H%M%S'`
myFl="EDI_file_${myDATE}${myTIME}.TXT"
myFlf="EDI_file_f_${myDATE}${myTIME}.TXT"
mv ${file##*/} "/home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI/$myFl"
echo $myFl
sed -i '/BLINDED/d' $myFl
done
awk -F $'\t' '{$6 != ""}' $myFl >> $myFlf

Using your provided file, it works.

**MoonRise** · Nov 19, 2018, 05:29 PM

That is odd then. Still get an empty file here. Maybe there is something with konsole itself then. I guess I'll move it to another system I have and see what it does there. I've made no modifications to konsole other than color of text.

**Snowhog** · Nov 19, 2018, 05:36 PM

In order to 'test' the script, I made 'some' changes (your directories don't exist at here).

The "modified" script I tested, and that worked with the provided .txt file, is:

Code:

#! /bin/sh # for file in /User/paul/dropEDI/*; do myDATE=`date '+%Y%m%d'` myTIME=`date '+%H%M%S'` myFl="EDI_file_${myDATE}${myTIME}.TXT" myFlf="EDI_file_f_${myDATE}${myTIME}.TXT" mv ${file##*/} "/User/paul/dropEDI/$myFl" echo $myFl #sed -i '/BLINDED/d' $myFl done awk -F $'\t' '{$6 != ""}' $myFl >> $myFlf

The provided file didn't contain what the sed -i line was looking for, so I commented it out. But otherwise, the script above worked on the provided file and produced the second file with the filtered lines.

Added:
What is the name of the 'original' source file that the script runs against?

What are the names of the source files that this script is reading? I'm skeptical as to the myFl= and myFlf= variable creation lines. Are the source files named EDI_file_ and EDI_file_f_ or something else?

**MoonRise** · Nov 19, 2018, 05:39 PM

OK, will have to do same other system. Will let you know. I'm wondering if it's my konsole environment.

**MoonRise** · Nov 19, 2018, 06:01 PM

Must be me. Doesn't work on this system either. What is installed on both is gawk which should be fine for this.

Thanks anyway!! I really do appreciate the second eyes.

**MoonRise** · Nov 19, 2018, 06:34 PM

Works on the machine it will be running on which is a cygwin64 environment.

Environment issues here is the only thing I can think of.

**Snowhog** · Nov 19, 2018, 07:50 PM

We got it working.

Compare this slightly modified script that we tested on your provided file. That is, got it working on my iMac with it's bash.

Code:

#! /bin/sh
#


cd /home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI

for file in /home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI/*; do

  myDATE=`date '+%Y%m%d'`
  myTIME=`date '+%H%M%S'`
  myFl="EDI_file_${myDATE}${myTIME}.TXT"
  myFlf="EDI_file_f_${myDATE}${myTIME}.TXT"

  mv ${file##*/} "/home/moonrise/Documents/My_Work/fakeServer/delivEDI/dropEDI/$myFl"

  sed -e '/BLINDED/d' -i "" $myFl
done


awk -F $'\t' '$6 != ""' $myFl >> $myFlf

Announcement

Problem using AWK in shell script

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment