avr 29 2011

Hit by multipath bug.

Tag: UnixUggla @ 11 h 58 min

Short article to keep a note about this bug.
We (colleague and I) added a lun to a RHEL 5.4 and configured it with multipath.
Everything was fine until we rebooted. We were stuck at early boot up stage with :

Cannot make directory [/var/lib] : Read-only file system

Looking for some ideas regarding this bug we finally found this RH identified bug : https://bugzilla.redhat.com/show_bug.cgi?id=409741
Issue is exactly the one describe in comment #11 by Blaz Podrzaj.

To fix it :

  1. we move /var/lib/multipath/bindings to /etc/multipath/bindings.
  2. edit /etc/multipath.conf and add the directive bindings_file           /etc/multipath/bindings in the defaults section.

This is a bit different from the solution provided by Blaz Podrzaj comments but we prefer not to have symlinks.
To my mind, having various solutions to fix an issue is something that makes linux really « fun ».


avr 19 2011

Rescan tape drive

Tag: UnixUggla @ 14 h 08 min

Quick article to keep track of the command.

Following this message in dmesg :

st0: Error 10000 (sugg. bt 0x0, driver bt 0x0, host bt 0x1).

Looks like the command below fix the issue.

echo "1" > /sys/class/scsi_tape/st0/device/rescan


mar 18 2011

Obtenir une vision synthétique du stockage sous Linux avec une baie XP (HP).

Tag: ScriptsUggla @ 15 h 11 min

Un script perl qui combine les sorties de différents outils (xpinfo, multipath, mount) pour produire une vue complète du stockage disque.

Ce script fonctionne sur une machine Linux (RHEL5.5) avec une baie XP (HP).

Note : Ce script fonctionne mais peut probablement être grandement amélioré


#!/usr/bin/perl
# --------------------------------------------------------------------------------
# Script :   map.pl
# Revision : 1.1
# Author :   Rene Ribaud
# Description :
# --------------------------------------------------------------------------------
use strict;
use warnings;
#use diagnostics;   # Only use it for debugging purpose.

# --------------------------------------------------------------------------------
# Global variables
# --------------------------------------------------------------------------------
# Time
my ($sec,$min,$hour,$day,$month,$year,$wday,$yday,$isdst);
my $timestamp;

# Data
# Define a full structure
#my %mpaths =(
#        name =>{
#               size => undef,
#               uuid => undef,
#               dm => undef,
#               fs => undef,
#               disk =>         {
#                               diskname =>     {
#                                               port => undef,
#                                               lunid => undef,
#                                               sfnum => undef,
#                                               ldev => undef,
#                                               type => undef,
#                                               xp => undef,
#                                               },
#                               },
#               },
#        );

my %mpaths;

# --------------------------------------------------------------------------------
# Functions
# --------------------------------------------------------------------------------

# --------------------------------------------------------------------------------
# Main
# --------------------------------------------------------------------------------

# Get variable from command line

# Get timestamp
($sec,$min,$hour,$day,$month,$year,$wday,$yday,$isdst)=localtime();
$timestamp=sprintf("%4d%02d%02d-%02d:%02d:%02d",($year+1900),($month+1),$day,$hour,$min,$sec);

# Slurp tools data into arrays in memory
my @xpinfo=`xpinfo -i`;
my @multipath=`multipath -ll`;
my @mounts=`mount`;

for (my $line=0; $line<scalar(@multipath); $line++){
if ($multipath[$line] =~ /^mpath/){   # Check & parse line : mpath23 (360060e8015276a000001276a0000d0fc) dm-8 HP,OPEN-V
        (my $mpath, my $uuid, my $dm, my $type) = split ('\s+',$multipath[$line]);
        $uuid=~s/\(//g;
        $uuid=~s/\)//g;
        $mpaths{$mpath}->{"dm"}=$dm;
        $mpaths{$mpath}->{"uuid"}=$uuid;
        $mpaths{$mpath}->{"type"}=$type;
        # Crosscheck into mount to get the fs name
        # /dev/mapper/mpath14 on /oradata/crs/voting1 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)
        my $fs;
        my @mount = grep(/$mpath\s/,@mounts);
        if (scalar(@mount==0)){
                $fs="N/A";
                }
                else
                {
                (undef,undef,$fs) = split('\s+',$mount[0]);
                }
        $mpaths{$mpath}->{"fs"}=$fs;

        # Next line to parse : [size=36G][features=1 queue_if_no_path][hwhandler=0][rw]
        $line++;
        if ($multipath[$line] =~ /^\[size=/){
                (my $size) = $multipath[$line] =~ /\[size=([0-9]+\w)/;
                $mpaths{$mpath}->{"size"}=$size;

                # Next line to parse : \_ round-robin 0 [prio=0][active]
                $line++;
                if ($multipath[$line] =~ /^\\_/){
                        # Next line to parse :  \_ 0:0:4:13 sdad 65:208 [active][ready]
                        $line++;
                        if ($multipath[$line] =~ /^ \\_/){
                                my $no_more_device=0;
                                do{
                                        (undef,undef,undef,my $disk, my $sf, undef) = split ('\s+',$multipath[$line]);
                                        $mpaths{$mpath}->{"disk"}->{$disk}->{"sfnum"}=$sf;
                                        # Crosscheck into xpinfo
                                        my @device = grep(/$disk\s/,@xpinfo);

                                        # Handle the case where device was not unmapped correctly
                                        # Unmapped from XP and not remoove from multipath.
                                        if (scalar(@device) == 0){
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"port"} = "Not found";
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"lunid"}= "N/A";
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"ldev"}= "N/A";
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"type"}= "N/A";
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"xp"}= "N/A";
                                        }
                                        else{
                                                #/dev/sdu                     76  04  02  CL2H  d0:f1  OPEN-V           00075626
                                                (undef,undef,undef, my $lunid, my $port, my $ldev, my $type, my $xp) = split('\s+',$device[0]);
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"port"}=$port;
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"lunid"}=$lunid;
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"ldev"}=$ldev;
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"type"}=$type;
                                                $mpaths{$mpath}->{"disk"}->{$disk}->{"xp"}=$xp;
                                        }
                                        if ((defined($multipath[$line+1])) && ($multipath[$line+1] =~ /^ \\_/)){
                                                $line++;
                                                }
                                                else {
                                                $no_more_device=1;
                                                }
                                        } while ($no_more_device==0);
                                }
                        }
                }
        }
}

# Print mapping
printf("%-34s %-9s %-7s %-6s %-6s %-8s %-6s %-6s %s\n","UUID","xp","type","size","dm","mapth","lunid","ldev","fs");
foreach my $mpath (sort(keys(%mpaths))){
        my @first_disk=keys(%{$mpaths{$mpath}->{"disk"}});
        # Check to report disk not unmapped correctly
        printf("%-34s %-9s %-7s %-6s %-6s %-8s %-6s %-6s %s\n"  ,$mpaths{$mpath}->{"uuid"}
                                                                ,$mpaths{$mpath}->{"disk"}->{$first_disk[0]}->{"xp"}
                                                                ,$mpaths{$mpath}->{"disk"}->{$first_disk[0]}->{"type"}
                                                                ,$mpaths{$mpath}->{"size"}
                                                                ,$mpaths{$mpath}->{"dm"}
                                                                ,$mpath
                                                                ,$mpaths{$mpath}->{"disk"}->{$first_disk[0]}->{"lunid"}
                                                                ,$mpaths{$mpath}->{"disk"}->{$first_disk[0]}->{"ldev"}
                                                                ,$mpaths{$mpath}->{"fs"}
                                                                );
        my $notfoundcount=0;
        foreach my $disk (keys(%{$mpaths{$mpath}->{"disk"}})){
                printf("%-6s %-8s %-4s\n",$disk,$mpaths{$mpath}->{"disk"}->{$disk}->{"sfnum"},$mpaths{$mpath}->{"disk"}->{$disk}->{"port"});

                if ($mpaths{$mpath}->{"disk"}->{$disk}->{"port"} eq "Not found"){
                        $notfoundcount++;
                        }

                if ($notfoundcount == scalar(keys(%{$mpaths{$mpath}->{"disk"}}))){
                        printf("This device was found in multipath but not reported by xpinfo.\n");
                        printf("Multipath device was probably not flushed after unmapping the device.\n");
                        printf("Try to run multipath -f %s.\n",$mpath);
                        }

                }

print "\n";
}

Output result :

UUID                               xp        type    size   dm     mapth    lunid  ldev   fs
360060e8015276a000001276a0000d0f1  00075626  OPEN-V  201M   dm-13  mpath14  02     d0:f1  /oradata/crs/voting1
sdai   66:32    CL1F
sdc    8:32     CL2F
sday   67:32    CL1H
sds    65:32    CL2H

360060e8015276a000001276a0000d0f2  00075626  OPEN-V  201M   dm-14  mpath15  03     d0:f2  /oradata/crs/voting2
sdd    8:48     CL2F
sdaz   67:48    CL1H
sdaj   66:48    CL1F
sdt    65:48    CL2H

360060e8015276a000001276a0000d0f3  00075626  OPEN-V  201M   dm-15  mpath16  04     d0:f3  /oradata/crs/voting3
sde    8:64     CL2F
sdu    65:64    CL2H
sdba   67:64    CL1H
sdak   66:64    CL1F

360060e8015276a000001276a0000d0f4  00075626  OPEN-V  201M   dm-16  mpath17  05     d0:f4  /oradata/GFSTEUR1/pfile
sdal   66:80    CL1F
sdv    65:80    CL2H
sdbb   67:80    CL1H
sdf    8:80     CL2F

360060e8015276a000001276a0000d0f5  00075626  OPEN-V  36G    dm-17  mpath18  06     d0:f5  /oradata/GFSTEUR1/undo
sdg    8:96     CL2F
sdbc   67:96    CL1H
sdw    65:96    CL2H
sdam   66:96    CL1F

360060e8015276a000001276a0000d0f8  00075626  OPEN-V  99G    dm-18  mpath19  09     d0:f8  /oradata/RMAN/rman01
sdh    8:112    CL2F
sdan   66:112   CL1F
sdbd   67:112   CL1H
sdx    65:112   CL2H
...


mar 02 2011

ssh trucs et astuces

Tag: UnixUggla @ 18 h 02 min

Quelques trucs et astuces avec ssh.

  • Ignorer la clef de host :
    ssh -o stricthostkeychecking=no server
    
  • Mode batch, aucun « prompt » (password, clef) n’est demandé, en gros ca marche ou pas :
    ssh -o batchmode=yes server
    
  • Mode quiet , aucun message d’alerte ou diagnostique est affiché :
    ssh -q server
    
  • Utiliser une commande avec des quotes (ex : awk). Il faut protéger les quotes avec des double quotes ("'") :
    ssh server 'vgdisplay | grep "VG Name" | awk '"'"'{print $NF}'"'"''
    

mar 01 2011

Gestion des espaces dans les boucles « for » avec IFS

Tag: ScriptsUggla @ 14 h 17 min

Une astuce pour gérer les espaces dans les boucles « for » en shell (bash, ksh).
Rien ne vaut un petit exemple pour montrer le problème :

root@hx04970: /root/home/root/nene # ll
total 0
-rw-r--r--   1 root       sys              0 Feb 28 14:09 1
-rw-r--r--   1 root       sys              0 Feb 28 14:09 2
-rw-r--r--   1 root       sys              0 Feb 28 14:09 3
drwxr-xr-x   2 root       sys             96 Feb 28 12:04 tata tutu
drwxr-xr-x   2 root       sys             96 Feb 28 12:03 titi
drwxr-xr-x   2 root       sys             96 Feb 28 12:03 toto

Imaginons que je fasse une boucle sur l’ensemble des fichiers de l’arborescence ci dessus.

root@hx04970: /root/home/root/nene # for i in *;do echo $i;done
1
2
3
tata tutu
titi
toto

Ici tout se passe bien, pas de problèmes, le répertoire « tata tutu » est bien interprété.

Maintenant je complique un peu en voulant faire une boucle sur les répertoires seulement :

root@hx04970: /root/home/root/nene # for i in $(find . -type d);do echo $i;done
.
./toto
./titi
./tata
tutu

Et la c’est le drame :) .

L’espace sur le répertoire « tata tutu » est vu comme un séparateur et le shell croit que nous avons 2 répertoires tata et tutu.
Le bon administrateur Unix sait qu’il faut éviter les espaces c’est mal. (C’est comme : on ne croise jamais les effluves : c’est mal –> cf Ghostbuster).

Cool ! Mais si le répertoire est un share cifs avec des utilisateurs de windows, il y a de grande chance que l’on retrouve plein d’espaces dans les noms de fichier.

La meilleure méthode de résolution du problème ci dessus est d’utiliser la « variable » IFS.
Un petit man ksh me donne :

IFS Internal field separators, normally space, tab,
and newline that are used to separate command
words resulting from command or parameter
substitution, and for separating words with the
special command read. The first character of the
IFS parameter is used to separate arguments for
the « $* » substitution (see Quoting below).

IFS est la « variable » qui définie le séparateur, par défaut espace, tab et nouvelle ligne.

En redéfinissant le séparateur, sans l’espace et le tab on obtient :

root@hx04970: /root/home/root/nene # IFS=$'\n' && for i in $(find . -type d);do echo $i;done
.
./toto
./titi
./tata tutu

Le problème est résolu.

Pour aller plus loin : http://tldp.org/LDP/abs/html/internalvariables.html#IFSH