BASH Redirection Failures

[rec9140]

So I have something that needs some BASH redirection, but it seems to fail

The issue is:

tail -f ~/mysecretlog.log | grep -E "string1|string2"

Ok.. this works, fine..

The problem comes in when I want to save this data to a file, ie redirect stdout to a file. ie:


tail -f ~/mysecretlog.log | grep -E "string1|string2" >> AnotherSecret.log

waahaaahommmppp.. FAIL.

YES. I have tried explicitly telling it to send stdout to the file:
tail -f ~/mysecretlog.log | grep -E "string1|string2" &>> AnotherSecret.log

tail -f ~/mysecretlog.log | grep -E "string1|string2" 1> AnotherSecret.log

Nope. and nope.

No data is added to the file.

The source is a LIVE log data that is derived from another script... Sure it would be great or better or even "proper" to do this via that script... There is a long list of issues as to why thats not going to happen, right now at least.

There is something in this I am missing on redirecting the results that stdout gets fine, to a file.

shrug...

Any one got some hints Thanks!

[Snowhog]

You need to include greps 'Other Options' line buffering on output to your command.

tail -f ~/mysecretlog.log | grep -E --line-buffered "string1|string2" >> AnotherSecret.log

-f` Option in Tail Command in Linux


This option is mainly used by system administration to monitor the growth of the log files written by many Unix program as they are running. This option shows the last ten lines of a file and will update when new lines are added. As new lines are written to the log, the console will update with the new lines.

The prompt doesn’t return even after work is over so, we have to use the interrupt key to abort this command. In general, the applications writes error messages to log files. You can use the -f option to check for the error messages as and when they appear in the log file.

[rec9140]

You need to include greps 'Other Options' line buffering on output to your command.

tail -f ~/mysecretlog.log | grep -E --line-buffered "string1|string2" >> AnotherSecret.log

This is the key...

Another option on this, now with the source of the issue is:
stdbuf -o0

Which nukes this hidden buffer.. which if its really waiting for 4K worth of data, I find that hard to believe since doing this with out the redirection is "real time" to the data received, verified by another source...

Thanks...

[other snark removed after better judgement review... ]