Some Bash Questions

[Xplorer4x4]

Sorry for the generic thread title but I am trying to write a script to auto set up a server for a friend of mine, and a few things have me stumped so I figured I would lump them in here.

1. Salting a Password - I understand what salting is, but what I don;t get is how to use the password after salting it. For example:

perl -e 'print crypt("password", "salt"),"\n"'

becomes:

sa3tHJ3/KuYvI

Now maybe my syntax is off here, but let's use the password "password," salt it, and assign it ass password for the current logged in user.

rootpw=`perl -e 'print crypt("password", "salt"),"\n"'`
echo "$user:$rootpw" | chpasswd

Now if I try to log in with password or sa3tHJ3/KuYvI, it claims the password is wrong. What am I doing wrong here?


2. I want to test for the fastest mirror which is up to date or at most 6 hours behind, and automaticly apply this to my sources.list file. I know how to test for a mirror, and I know how to edit the sources list, but I don't know how to grab the result from netstat and use that in sed. Here is what I have. The empty line is to represent where I am stuck:

wget http://ftp.de.debian.org/debian/pool/main/n/netselect/netselect_0.3.ds1-14+squeeze1_amd64.deb && sudo dpkg -i netselect*.deb
sudo netselect -v -s10 -t20 `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors | grep -P -B8 "statusUP|statusSIX" | grep -o -P "(f|ht)tp.*\"" | tr '"\n' '  '`

sed -i 's|old/with/slashes|new/with/slashes|g'

[james147]

1) Linux hashes the passwords with a salt anyway, so it is a little redundant for that purpose.

You should not try to log in with "password" as that is not your password, your password is the salted hash of password so you should be able to log in with sa3tHJ3/KuYvI.

Also, what is $user set to?, what does echo "$user:$rootpw" evaluate to? is it set to the username you want to change (note that $user is undefined by default, $USER is defined to the currently logged in user).

2) Can you give some sample output of sudo netselect -v -s10 -t20 `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors as I am not currently on a computer that can run that command.

Also, grep | grep | tr | sed sounds like it would be simpler to do in awk.

[Xplorer4x4]

1. I got $user and $USER mixed up. Is there a way to write the salted pass to a txt file for retrial/deletion when the script is done? Inside a script it does not echo/print anything in the shell(then again maybe that was because the $USER variable was wrong).

2. http://pastebin.com/3gD7ePZu - In this case, the output I would want to grab is "3 ftp://91.121.124.139/ubuntu/" more specifically the url itself(I am sure that's obvious but just saying). It was late and I was thinking it had a different out put, so this may be tricker then I suspected.

[james147]

What is this script meant to do? If you just want to automatically setup a password why not just:

echo "test:a password" | chpasswd

chpasswd will hash the password for you and store it (with the salt) in /etc/shadow there is no need for manual salting when setting a Linux password.

Is there a way to write the salted pass to a txt file for retrial/deletion when the script is done?

some_command > a_file

will save the output of some_command to a_file (overwriting the file, use >> to append instead)

Inside a script it does not echo/print anything in the shell(then again maybe that was because the $USER variable was wrong).

It wont as it currently stands since you pass the output of the echo to another command, but what I meant was what does it print if you add the line echo "$user:$rootpw" with no pipes or anything so you can see what it is passing to chpasswd or even just comment out the pipe:

rootpw=`perl -e 'print crypt("password", "salt"),"\n"'`echo "$user:$rootpw" #| chpasswd

then see what it prints out.

2. http://pastebin.com/3gD7ePZu - In this case, the output I would want to grab is "3 ftp://91.121.124.139/ubuntu/" more specifically the url itself(I am sure that's obvious but just saying). It was late and I was thinking it had a different out put, so this may be tricker then I suspected.

Looks like you have read this. And if I am correct the sudo netselect -v -s10 -t20 `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors | grep -P -B8 "statusUP|statusSIX" | grep -o -P "(f|ht)tp.*\"" | tr '"\n' ' '` line should return something similar to:

60 ftp://mirrors.se.eu.kernel.org/ubuntu/70 http://ubuntu.alex-vichev.info/
77 http://ftp.citylink.co.nz/ubuntu/
279 http://ubuntu.mirrors.tds.net/pub/ubuntu/
294 http://mirror.umd.edu/ubuntu/
332 http://mirrors.rit.edu/ubuntu/
364 ftp://pf.archive.ubuntu.com/ubuntu/
378 http://mirror.csclub.uwaterloo.ca/ubuntu/
399 ftp://ubuntu.mirror.frontiernet.net/ubuntu/
455 http://ubuntu.mirror.root.lu/ubuntu/

Am I right in this?

If so there are a couple of things wrong with your snippet, firstly, backticks are gnerally discouraged, instead you should use $() (there are quite a few reason for this I wont go into). Also, if this script is meat to set up a server it is likley going to need to be run as root (or at least should be) making the use of sudo irrelevant. And lastly, you are not storing the output of that command anywhere, it is just being printed to the screen, you should store it in a variable. You should also exit the script if one of the commands fails to run, you can do this with command || exit 1

if [[ "$UID" -ne "0" ]] # Check to see if we are root if not print a message and exit
then
    echo "This script must be run as root"
    exit 1
fi

# The rest of the script without the use of sudo

# Use explicit naming
NETSELECTPKG=netselect.deb
wget http://ftp.de.debian.org/debian/pool/main/n/netselect/netselect_0.3.ds1-14+squeeze1_amd64.deb -O $NETSELECTPKG || exit 1 # exit if the command fails
dpkg -i $NETSELECTPKG || exit 1
# Tidy up after ourselves
rm -f $NETSELECTPKG

# Note that we must use $() instead of the backticks here as backticks to not nest at all
MIRRORS=$(netselect -v -s10 -t20 $(wget -q -O- https://launchpad.net/ubuntu/+archivemirrors | grep -P -B8 "statusUP|statusSIX" | grep -o -P "(f|ht)tp.*\"" | tr '"\n' '  '))

# This will sort the mirrors by the first column then remove it so you are left with a list of mirrors sorted by speed
MIRRORS=$(echo "$MIRROR" | sort | awk '{print $2}')

# This will return the first (probably fastest) mirror
MIRROR=$(echo "$MIRRORS" | head -n1)

# Do a bit of checking
if [[ -z "$MIRROR" ]]
then
    echo "No mirror found"
    exit 1
fi

# And finally fix the sources.list
awk -v mirror="$MIRROR" '{ {$2 = mirror}; print }' /etc/apt/sources.list> /tmp/sources.list
mv /tmp/sources.list /etc/apt/sources.list

NOTE: This code it almost completely untested, use at your own risk and make sure you check it before you run it

[Xplorer4x4]

1. I will come back to this(passwords), as there is more to it but I think I am close to figuring it out!

2. As for the mirrors, It says unknown command on line 21 which is:

MIRRORS=echo "$MIRROR" | sort | awk '{print $2}'

I tried to change $MIRROR to MIRRORS$, and it got a bit further, but still did not function properly. This one is way over my head so I am going to have to wait for a fix on your script.

3. Use explicit naming - I am curious what the point of this is? It looks neat from a programing stand point, but looks like extra work for no reason. It is something I will keep in mind but I just dont see the use in this case.

Thanks for all the help, guidance, and patience James!

[james147]

opps sorry that should be

MIRRORS=$(echo "$MIRROR" | sort | awk '{print $2}')

[Xplorer4x4]

No worries. You warned it was untested. That brings us to a new error: "No mirror found"

[james147]

No worries. You warned it was untested. That brings us to a new error: "No mirror found"

That is due to this check:

# Do a bit of checkingif [[ -z "$MIRROR" ]]
then
    echo "No mirror found"
    exit 1
fi

I would add in some debug code to see what is happening:

...

# Note that we must use $() instead of the backticks here as backticks to not nest at all
MIRRORS=$(netselect -v -s10 -t20 $(wget -q -O- https://launchpad.net/ubuntu/+archivemirrors | grep -P -B8 "statusUP|statusSIX" | grep -o -P "(f|ht)tp.*\"" | tr '"\n' '  '))
[B]echo "MIRRORS=$MIRRORS"[/B]


# This will sort the mirrors by the first column then remove it so you are left with a list of mirrors sorted by speed
MIRRORS=$(echo "$MIRROR" | sort | awk '{print $2}')
[B]echo "MIRRORS2=$MIRRORS"[/B]


# This will return the first (probably fastest) mirror
MIRROR=$(echo "$MIRRORS" | head -n1)
[B]echo "MIRROR=$MIRRORS
exit 1 # no point in continuing until we know the above are correct[/B]


...

Bits that have been added are in bold.

What does that print out now? Note it wont work fully due to the exit 1, that is just so we can see what's happening and fix it before running the later commands.
What does that

[Xplorer4x4]

MIRRORS=    3 http://mirror.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu/
   12 ftp://ftp-stud.hs-esslingen.de/ubuntu/
   13 http://ubuntu.unitedcolo.de/ubuntu/
   15 http://mirror.netcologne.de/ubuntu/
   16 ftp://ucho.ignum.cz/ubuntu/
   16 http://nl3.archive.ubuntu.com/ubuntu/
   17 http://mirror.i3d.net/pub/ubuntu/
   18 http://ftp.uni-mainz.de/ubuntu/
   19 http://ubuntu.fastbull.org/ubuntu/
   20 ftp://artfiles.org/ubuntu.com/
MIRRORS2=
./sources.sh: line 41: unexpected EOF while looking for matching `"'
./sources.sh: line 43: syntax error: unexpected end of file

That's what I got this time.

Line 41:

awk -v mirror="$MIRROR" '{ {$2 = mirror}; print }' /etc/apt/sources.list> /tmp/sources.list

There is no line 43 in this case, so the problem must be a missing symbol some where, correct?

[james147]

...

# This will return the first (probably fastest) mirror
MIRROR=$(echo "$MIRRORS" | head -n1)
[B]echo "MIRROR=$MIRRORS
exit 1 # no point in continuing until we know the above are correct[/B]


...

Opps, missed the " off the end of $MIRRORS.

[Xplorer4x4]

lol I must have been in a rush out the door earlier to miss something so obvious. New output:

MIRRORS=    3 http://mirror.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu/
    5 http://ftp.uni-mainz.de/ubuntu/
   10 http://ftp-stud.hs-esslingen.de/ubuntu/
   13 http://mirror.netcologne.de/ubuntu/
   13 http://ubuntu.unitedcolo.de/ubuntu/
   17 http://archive.ubuntu.mirror.dkm.cz/
   17 ftp://mirror.i3d.net/ubuntu/
   17 http://nl3.archive.ubuntu.com/ubuntu/
   18 http://ucho.ignum.cz/ubuntu/
   18 http://artfiles.org/ubuntu.com/
MIRRORS2=
MIRROR=

[james147]

So the problem is with the second assignment of MIRRORS:

# This will sort the mirrors by the first column then remove it so you are left with a list of mirrors sorted by speed
MIRRORS=$(echo "$MIRROR" | sort | awk '{print $2}')

Can you see the error?

Spoiler alert:

MIRROR is not defined yet, it should be MIRRORS,

# This will sort the mirrors by the first column then remove it so you are left with a list of mirrors sorted by speed
MIRRORS=$(echo "$MIRRORS" | sort | awk '{print $2}')

This is why you should test your code before publishing it

[jlittle]

James147, your persistence is admirable (no disrespect to the OP is implied at all).
Is that sort meant to be numeric, or am I spoiling?

Regards, John Little

[Xplorer4x4]

MIRROR is not defined yet, it should be MIRRORS,

lol didn't I mention that already(although not in those words)? I removed the exit command, tested the script, and it works great! Also, hinting earlier where you had used $() in this piece of code helped me realize what you meant when you mentioned $(). Initially I interrupted it as is rather then placing variables and such inside the parenthesis.

I hate to be a bother, but again, I do not quite see the point in the "explicit naming." Would you mind elaborating?

James147, your persistence is admirable (no disrespect to the OP is implied at all).

Not sure how that could be seen as disrespect, but I agree James has proven to be very patient!

[james147]

lol didn't I mention that already(although not in those words)?

O yes sorry, it a programming habit of only looking at the first error, fixing it then retesting the code that I might have ignored part of your response also, this is why you should not answer questions at 5AM because you cannot sleep

I hate to be a bother, but again, I do not quite see the point in the "explicit naming." Would you mind elaborating?

There was no real reason for using multiple lines except it makes it easier to read and debug. You can combine the pipes into one expression or split them out into more if you wanted. But at the time I figured that those steps would be handy to see if something went wrong. It makes no difference to the end result however. It is slightly less efficient splitting it up, but for this situation it is not going to make any noticeable difference (only if you are handling thousands of mirrors would you even start to be able to measure the overhead).

Overall when you do not need high performance it is better to go with the more readable solution to make maintenance easier and you should never optimize before you have a working solution