Veritas FS
Petit tutorial sur Veritas FS, gestion du file system de VERITAS.
Last Updated: Thu May 10 14:23:37 CDT 2001
A1000 Dual Port Setup and Configuration
with Veritas Volume Manager and Filesystem
———————————————————–
Overview:
The following tutorial documents the steps used to configure a single
A1000 array to be shared between two hosts. It then documents the
steps used to create simple Veritas volumes and filesystems on each
of the logical array drives, as well as the steps necessary to deport
and import the volumes between the two hosts.
This tutorial is based on an actual install of the configuration used
in the examples.
Hardware configs:
Sun StorEdge A1000 – SCSI Target ID 1
w/ 7 18GB disk drives
d0 d1 d2 HS d0 d1 d2
———————————————–
| 2,0 2,1 2,2 2,3 | 1,0 1,1 1,2 |
———————————————–
| |
| controller 1 | controller 1
| |
————– ————-
| E220R | | E220R |
————– ————-
node1 node2
| |
c0t0d0 c0t0d0 Internal root drive
c0t1d0 c0t1d0 Internal root mirror
– Both E220R's have a dual-channel Differential Ultra-SCSI card, with the
first port (controller 1) connected to the A1000 array.
– The A1000 array was configured as 3 logical drives, plus one hot spare.
The logical drives consist of 2 18GB disk drives mirrored (RAID 1).
– The A1000 logical drives are seen as:
c1t1d0, c1t1d1, c1t1d2
– Final configs are setup as:
Internal Drives:
Boot drive c0t0d0 encapsulated as rootdisk and mirrored to disk01 (c0t1d0)
A1000 Drives are all simple (concatenated) veritas volumes with vxfs filesystems:
/mh = disk02 = c1t1d0 = RAID 1 hardware mirror using array drives 1,0 and 2,0
/ms = disk03 = c1t1d1 = RAID 1 hardware mirror using array drives 1,1 and 2,1
/amt = disk04 = c1t1d2 = RAID 1 hardware mirror using array drives 1,2 and 2,2
Hardware RAID Hot Spare drive is 2,3
– Steps used to configure RAID:
1. Installed A1000 in Rack
2. Cabled A1000 to port 1 of dual-port differential SCSI card on node1
3. Set SCSI ID switch on back of A1000 to target 1
4. powered on A1000
5. powered on E220R
6. At boot prompt, verified the array was seen:
setenv auto-boot? false
reset-all
probe-scsi-all
7. Booted E220R
8. Verified OS and app patch requirements per Sun Infodoc 20029
"A1000/A3x00/A3500FC Software/Firmware Configuration Matrix"
(only patch required was the RAIDManager version 6.22 jumbo
patch 108834-09)
9. Installed the RAIDManager ver. 6.22 software from the included CD:
mount -r -F hsfs /dev/sr0 /cdrom
cd /cdrom/… (don't remember the exact path)
pkgadd -d . SUNWosar SUNWosafw SUNWosamn SUNWosau
This installs the following packages:
system SUNWosafw Open Storage Array Firmware
system SUNWosamn Open Storage Array Man Pages
system SUNWosar Open Storage Array (Root)
system SUNWosau Open Storage Array (Usr)
10. Verified that /etc/osa/mnf does not have a period (.) in the name.
There is a known problem with this. If it does, change to an
underscore (_). This was per the A1000 install notes at:
http://www.eng.auburn.edu/pub/mail-lists/ssastuff/Solaris8-A1000.html
11. Installed the RAIDManager jumbo patch 108834-09
12. Performed a reconfiguratoin reboot:
touch /reconfigure
init 6
13. Verified the A1000 could be seen, and what firmware revision level
it was:
/usr/lib/osa/bin/raidutil -c c1t1d0 -i
Firmware was currently at:
03.01.02.35
14. Upgraded firmware to current versions. The current versions
are included as part of the patch install, and are stored in
/usr/lib/osa/fw.
To install the latest boot code:
/usr/lib/osa/bin/fwutil 03010304.bwd c1t1d0
To install the latest firmware:
/usr/lib/osa/bin/fwutil 03010363.apd c1t1d0
15. Verified the A1000 was updated:
/usr/lib/osa/bin/raidutil -c c1t1d0 -i
Reports something similar to:
LUNs found on c1t1d0.
LUN 0 RAID 5 103311 MB
Vendor ID Symbios
ProductID StorEDGE A1000
Product Revision 0301
Boot Level 03.01.03.04
Boot Level Date 07/06/00
Firmware Level 03.01.03.63
Firmware Date 03/15/01
raidutil succeeded!
16. Now verify that the drives can be all seen:
/usr/lib/osa/bin/drivutil -i c1t1d0
This will report something similar to:
Drive Information for ig028_002
Location Capacity Status Vendor Product Firmware Serial
(MB) ID Version Number
[1,0] 17274 Optimal IBM DDYST1835SUN18G S96H 010811E164
[2,0] 17274 Optimal IBM DDYST1835SUN18G S96H 0108109219
[1,1] 17274 Optimal IBM DDYST1835SUN18G S96H 0108115692
[2,1] 17274 Optimal IBM DDYST1835SUN18G S96H 010811E211
[1,2] 17274 Optimal IBM DDYST1835SUN18G S96H 010810V958
[2,2] 17274 Optimal IBM DDYST1835SUN18G S96H 01081WH714
[2,3] 17274 Spare-Stdby IBM DDYST1835SUN18G S96H 010810V946
17. Delete the default lun 0 configuration:
/usr/lib/osa/bin/raidutil -c c1t1d0 -D 0
Reports something similar to:
LUNs found on c1t0d0.
LUN 0 RAID 5 103644 MB
Deleting LUN 0.
Press Control C to abort.
LUNs successfully deleted
raidutil succeeded!
18. Created the RAID 1 drive mirrors by mirroring the following
pairs of drives:
1,0 –> 2,0 mirrored as logical unit 0 (d0)
1,1 –> 2,1 mirrored as logical unit 1 (d1)
1,2 –> 2,2 mirrored as logical unit 2 (d2)
Commands used:
/usr/lib/osa/bin/raidutil -c c1t1d0 -l 1 -n 0 -s 0 -r fast -g 10,20
/usr/lib/osa/bin/raidutil -c c1t1d0 -l 1 -n 1 -s 0 -r fast -g 11,21
/usr/lib/osa/bin/raidutil -c c1t1d0 -l 1 -n 2 -s 0 -r fast -g 12,22
19. Created the hot spare drive using drive 2,3:
/usr/lib/osa/bin/raidutil -c c1t1d0 -h 23
20. Partitioned and labeled each new drive with a single slice 2
partition as the whole disk using the format command.
At this point the A1000 was ready to go on the first system. If this was
the only system, you would just build your filesystems and mount the drives
at this point, the same as any other drive.
Because we were dual-porting this between two nodes of a future cluster,
we now needed to configure the second node. We now need to change the
scsi-initatior-id of the second controller to 6 (from the default of 7).
This is so both scsi controllers can be connected to the array at the
same time.
To configure the second node:
1. Leave the array disconnected from the second system for now
2. Power on the second E220R
3. Update the nvramrc to set the controller to id 6 per
the Sun Infodoc 20704 "Setting the scsi-initatior-id on PCI
Systems with Sun Cluster Software". (This applies to any
dual ported systems, cluster or not.)
A. From the OBP prompt, get list of controllers:
ok setenv auto-boot? false
ok reset-all
ok probe-scsi-all
B. edit nvramrc using the path for the scsi controller(s)
that you are changing:
ok nvedit
0: probe-all install-console banner
1: cd /pci@1f,4000/scsi@2,1
2: 6 " scsi-initiator-id" integer-property
3: device-end
4: cd /pci@1f,4000/scsi@2
5: 6 " scsi-initiator-id" integer-property
6: device-end
7: banner (Control C)
C. Do a ctrl-c, and store the nvramrc:
ok nvstore
D. Set the system to use the nvramrc:
ok setenv use-nvramrc? true
E. Do a reset:
ok reset-all
4. Verify the nvramrc settings were saved and that the
scsi-initiator-id was changed to 6 on the card:
ok cd /pci@1f,4000/scsi@2,1
ok .properties
It should report something like:
"scsi-initiator-id 000000006"
5. Cable the second system (node2) via port 1 of the dual-port
differential SCSI card to the second SCSI port on the A1000.
6. Reset the system again and then probe the scsi bus
to verify it sees the array:
ok reset-all
ok probe-scsi-all
7. Reset the auto-boot parameter, and then reset the
system and allow it to boot:
ok setenv auto-boot? true
ok reset-all
8. Install the RAIDManager ver 6.22 software and jumbo patch
as you did on the first node.
9. Verify the RAIDManager software can see the configured array:
/usr/lib/osa/bin/raidutil -c c1t1d0 -i
/usr/lib/osa/bin/drivutil -i c1t1d0
*** DO NOT CONFIGURE THE RAID – IT IS ALREADY CONFIGURED ***
10. Verify the the OS utilities (format, prtvtoc, etc.) can see the drives.
At this point, the hardware is all configured. Next we need to
configure the Volume Manager and File System software.
VXVM/VXFS configs:
1. Install the VXVM and VXFS software:
mount -r -F hsfs /dev/sr0 /cdrom
cd /cdrom/… (don't remember the exact path)
pkgadd -d . VRTSvxvm VRTSvmdev VRTSvmdoc VRTSvmman VRTSvmsa VRTSvxfs VRTSfsdoc
This installs the following packages:
system VRTSfsdoc VERITAS File System Documentation Package
system VRTSvmdev VERITAS Volume Manager, Header and Library Files
system VRTSvmdoc VERITAS Volume Manager (user documentation)
system VRTSvmman VERITAS Volume Manager, Manual Pages
system VRTSvmsa VERITAS Volume Manager Storage Administrator
system VRTSvxfs VERITAS File System
system VRTSvxvm VERITAS Volume Manager, Binaries
NOTE: The manpages and docs are all optional. Also, the latest
packages can be obtained via the Veritas ftp site after
contacting Veritas.
2 – Install the Veritas licenses:
vxserial -c
Enter the license key for each product. At a minimum, you need the
base volume manager key, and the veritas filesystem key. If you will
be using RAID 5, also enter that key.
3 – Run vxinstall to complete the installation:
vxinstall
This will prompt for a quick or custom install. Select Quick Install
1. Quick Installation
– encapsulate the boot drive c0t0d0
– use default disk names
– initial the mirror drive c0t1d0
– initial all drives on controller 1 (the array)
NOTE: In order to properly encapsulate the boot drive, you need to have:
– an unused cylinder at the beginning or end of drive
– slices 3 and 4 must be unused
To do this, I usually label the boot drive to have the root
slice 0 start at cylinder 1, and then only use slices 1,5,6,7 for
swap and the other filesystems.
4 – Reboot the system when prompted
5 – Verify all drives were configured and are seen:
vxdisk list
6 – Verify the root drive was encapsulated. The /etc/vfstab file and
df should both show that the root filesystems now are using /dev/vx
devices.
7 – Delete the array drives from rootdg (quick install only creates rootdg
and places all drives in it.):
vxdg -g rootdg rmdisk disk02
vxdg -g rootdg rmdisk disk03
vxdg -g rootdg rmdisk disk04
8 – Now initialize the datadg disk group. You do this by naming the first
disk that will be in the group:
vxdg init datadg disk02=c1t1d0s2
9 – Now add the remaining drives to datadg:
vxdg -g datadg adddisk disk03=c1t1d1s2
vxdg -g datadg adddisk disk04=c1t1d2s2
10 – Mirror the root drive. First we mirror the root filesystem and
make the mirror drive bootable:
/etc/vx/bin/vxrootmir disk01
11 – Now mirror the remainder of the root volumes:
vxassist -g rootdg mirror swapvol disk01
vxassist -g rootdg mirror usr disk01
vxassist -g rootdg mirror opt disk01
vxassist -g rootdg mirror var disk01
NOTE: You cannot reboot until the volumes have completed the mirroring
process. If you do, you have to start them again. To verify the
mirrors are done, run:
vxprint -ht
And each volume shows as being ENABLED and ACTIVE, i.e.:
v usr – ENABLED ACTIVE 4283208 fsgen – ROUND
pl usr-01 usr ENABLED ACTIVE 4283208 CONCAT – RW
sd rootdisk-03 usr-01 rootdisk 31080352 4283208 0 c0t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4283208 CONCAT – RW
sd disk01-03 usr-02 disk01 2709400 4283208 0 c0t1d0 ENA
12 – Now build the datadg volumes. In this example, we are building 3 simple
volumes, one per disk, and we use the filesystem name as the volume name.
A. Get the maxsize of the drive(s):
vxassist -g datadg disk02
B. Now create the mh volume using the maxsize returned above (17228m)
on disk02:
vxassist -g datadg make mh 17228m layout=concat disk02
C. Now create the ms volume using the maxsize returned above (17228m)
on disk03:
vxassist -g datadg make ms 17228m layout=concat disk03
D. Now create the amt volume using the maxsize returned above (17228m)
on disk04:
vxassist -g datadg make amt 17228m layout=concat disk04
13 – Now build the VXFS filesystems on each volume:
mkfs -F vxfs -o largefiles /dev/vx/rdsk/datadg/mh
mkfs -F vxfs -o largefiles /dev/vx/rdsk/datadg/ms
mkfs -F vxfs -o largefiles /dev/vx/rdsk/datadg/amt
14 – At this point the volumes and filesystems are ready to go. Use vxprint
to verify all configs:
vxprint -ht | more
The output will look like:
Disk group: rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH USETYPE PREFPLEX RDPOL
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
dg rootdg default default 0 989258071.1025.node1
dm disk01 c0t1d0s2 sliced 4711 35363560 –
dm rootdisk c0t0d0s2 sliced 4711 35363560 –
v opt – ENABLED ACTIVE 20662120 fsgen – ROUND
pl opt-01 opt ENABLED ACTIVE 20662120 CONCAT – RW
sd rootdisk-04 opt-01 rootdisk 10418232 20662120 0 c0t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 20662120 CONCAT – RW
sd disk01-04 opt-02 disk01 6992608 20662120 0 c0t1d0 ENA
v rootvol – ENABLED ACTIVE 607848 root – ROUND
pl rootvol-01 rootvol ENABLED ACTIVE 607848 CONCAT – RW
sd rootdisk-02 rootvol-01 rootdisk 0 607848 0 c0t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 607848 CONCAT – RW
sd disk01-01 rootvol-02 disk01 0 607848 0 c0t1d0 ENA
v swapvol – ENABLED ACTIVE 2101552 swap – ROUND
pl swapvol-01 swapvol ENABLED ACTIVE 2101552 CONCAT – RW
sd rootdisk-01 swapvol-01 rootdisk 607848 2101552 0 c0t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 2101552 CONCAT – RW
sd disk01-02 swapvol-02 disk01 607848 2101552 0 c0t1d0 ENA
v usr – ENABLED ACTIVE 4283208 fsgen – ROUND
pl usr-01 usr ENABLED ACTIVE 4283208 CONCAT – RW
sd rootdisk-03 usr-01 rootdisk 31080352 4283208 0 c0t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4283208 CONCAT – RW
sd disk01-03 usr-02 disk01 2709400 4283208 0 c0t1d0 ENA
v var – ENABLED ACTIVE 7708832 fsgen – ROUND
pl var-01 var ENABLED ACTIVE 7708832 CONCAT – RW
sd rootdisk-05 var-01 rootdisk 2709400 7708832 0 c0t0d0 ENA
pl var-02 var ENABLED ACTIVE 7708832 CONCAT – RW
sd disk01-05 var-02 disk01 27654728 7708832 0 c0t1d0 ENA
Disk group: datadg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH USETYPE PREFPLEX RDPOL
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
dg datadg default default 59000 989266501.1167.node1
dm disk02 c1t1d0s2 sliced 4095 35282944 –
dm disk03 c1t1d1s2 sliced 4095 35282944 –
dm disk04 c1t1d2s2 sliced 4095 35282944 –
v amt – ENABLED ACTIVE 35282944 fsgen – SELECT
pl amt-01 amt ENABLED ACTIVE 35282944 CONCAT – RW
sd disk04-01 amt-01 disk04 0 35282944 0 c1t1d2 ENA
v mh – ENABLED ACTIVE 35282944 fsgen – SELECT
pl mh-01 mh ENABLED ACTIVE 35282944 CONCAT – RW
sd disk02-01 mh-01 disk02 0 35282944 0 c1t1d0 ENA
v ms – ENABLED ACTIVE 35282944 fsgen – SELECT
pl ms-01 ms ENABLED ACTIVE 35282944 CONCAT – RW
sd disk03-01 ms-01 disk03 0 35282944 0 c1t1d1 ENA
15 – If the volumes/filesystems are to be managed as part of a cluster
the should now be added to it. If not, then the filesystems can
be added to node 1 using standard methods:
mkdir /mh
mkdir /ms
mkdir /amt
edit /etc/vfstab and add the 3 new volumes/filesystems
mountall
At this point, all filesystems should be online and operational. To manually
failover the array between nodes:
1 – Unmount the filesystems on node 1
umount /mh
umount /ms
umount /amt
2 – Deport the datadg disk group from node 1 (you can also use
the vxdiskadm menu for this):
vxdg deport datadg
3 – Import the datadg disk group on node 2 (you can also use
the vxdiskadm menu for this):
vxdg import datadg
NOTE: If the disks show as offline on node 2 (vxdisk list),
then you need to put them online before importing, i.e:
vxdisk online c1t1d0s2
Also, if the first node has failed and you were unable
to deport the disks first, you would need to force the
import with the -f option.
4 – Enable the imported volumes:
vxrecover -s
This will attempt to enable all volumes. If you wish to
enable only certain volumes, use:
vxrecover -s [volume_name]
If you forget this step, you will get an error about the
volume not existing when you try to mount/access it.
5 – Mount the filesystems
mount /dev/vx/dsk/datadg/mh /mh
mount /dev/vx/dsk/datadg/ms /ms
mount /dev/vx/dsk/datadg/amt /amt
To fail back, you would use the same procedure.