Sun StorEdge 3510 FC Array Configuration on Solaris 8
Introduction
This document details some of the configuration required in setting up a Sun StorEdge 3510 FC Array with two LUN's. This document assumes a few things....
- You have system administration experience with Sun Solaris 8 running Veritas Volume Manager (hereafter referred to as VxVM) version 3.5.
- You have used storage subsystems before and are familiar with the terminology
- You have correctly connected your 3510 to your server, and the array has 12 x 72Gb disks installed.
- You have used Sun Hardware before (this procedure was carried out on a Sunfire 4800 domain) and are familiar with it - including OpenBoot, etc.
- Unless noted, the Korn Shell (ksh) is used.
I have written this document as a guide to people being given a system with a 3510 attached, but in an "unknown" state - i.e. you want to reconfigure the array to meet your needs, deleting any volumes/drives and associated data in the process.
This document will not hold your hand and is meant for experienced users only! It is a terse overview of commands used. You can read the documentation for your hardware, OpenBoot, sccli, VxVM, etc., if you are unsure of any of the techniques used - I see no point in duplicating already published documentation.
Preparation
First of all, we will bring the system down to runlevel 0 (i.e. OpenBoot) and ensure that the system can see the 3510 attached to it. In our case the 3510 has got one LUN mapped - we will be keeping this, but to show the full configuration we'll remap later when we recreate the logical volume.
# init 0
ok reset all
Resetting...
.
.
.
ok probe-scsi-all
/ssm@0,0/pci@1d,600000/SUNW,qlc@1
/ssm@0,0/pci@1d,600000/SUNW,qlc@1
*************************** Local Loop Devices ***************************
Device AL-PA a7 loopId 28 Hard-addr 28 Port WWN 216000c0ff801515
Lun 0 DISK SUN StorEdge 3510 327Q
.
.
.
ok boot -rs
Okay, the system can see the 3510. If the output of probe-scsi-all is bleak and you cannot see the 3510, check the FC connections and try again.
Once the system comes up in single user mode, we will assume that the currently installed OS has no support for the 3510, requires patching, and version of VxVM running on the machine doesn't yet have the Active Library Support installed.
On the system I am working on, I require the following packages to be installed. Where and how you obtain these depends upon your available media, and/or if you have a support agreement with Sun.
SUNWcfpl.pkg SUNWcfplx.pkg SUNWsan.pkg SUNWsccli.pkg SUNWscsa.pkg SUNWscsd.pkg SUNWscsu.pkg SUNWscui.pkg
For now, we've thrown all of these packages into /var/tmp/3510_sun/. Note... if the following scripts don't work, and the pkgadd or patchadd require interactive input, you can use a here document to automate the process. Or, just install each package and patch individually. The admin file method of using pkgadd is sure to work, however.
Install all the packages, accepting defaults, something like....
# cd /var/tmp/3510_sun
# ls *.pkg | while read pkg; do
> pkgadd -a `echo "${pkg}" | sed 's/pkg/adm/'` -d ${pkg}
> done
If a reboot is required, hold off until all of the packages are installed, but before you install any of the patches mentioned in the following step. Note: this patch list is undoubtedly a little (or a lot ;-) ) out of date, but it sufficed for the purposes of the server I was building. The patches I installed were (make note of the order, too):
111412-11 111095-13 111413-10 111096-06 111097-12 111846-06 111847-07
Unzip the patch to a temporary location, install it and then remove the temporary directory. Let's assume the patches are in var/tmp/3510_patch
# cd /var/tmp/3510_patch
# for patch in 111412-11.zip 111095-13.zip 111413-10.zip \
> 111096-06.zip 111097-12.zip 111846-06.zip 111847-07.zip; do
> unzip -d /tmp ${patch}
> cd /tmp/${patch%%.*}
> patchadd .
> cd /tmp
> \rm -rf ${patch}
> cd /var/tmp/3510_patch
> done
... answer questions as required ...
If you are requested to do a reboot, do so only after all patches are installed, but before going onto the next stage of the process.
We now need to add Veritas Active Library Support (version 1.3) for the 3510 so that VxVM can recognise the array. We have stored the package in /var/tmp/3510_vxvm
# cd /var/tmp/3510_vxvm
# zcat VRTS3510.13.tar.Z | tar xvf -
# vxfctl mode # should show mode: enabled
# pkgadd -d . VRTS3510 # answer all/yes to questions
# vxdctl enable
# vxddladm listsupport | grep 3510 # should list output similar to....
libvxsun3510.so A/A SUN StorEdge
# vxdctl enable
# vxdmpadm listenclosure all # You should now see that VxVM has
# vxdmpadm listctlr all # claimed the 3510(s) - woohoo!
Don't sit too easily though... that was just the first part, getting the system to recognise the 3510. Now, we must configure the 3510 for our needs!!
It's probably a good idea to do a reboot at this stage just to make sure everything is happy....
# reboot -- -rs
Configuration
Now we can look at configuring the 3510. We are going to (after deleting any existing data/configuration on the array) set up two global spare disks, assign the remaining 10 physical drives into two RAID 5 logical drives, then create one logical volume out of both logical drives. Then we'll map that to a LUN. Let's get to it.
First, we'll check for any existing configuration on the 3510. We'll use the sccli command to do this - for further information on the sccli command, see it's man page.
There are two ways to execute sccli commands, you can execute the commands from within your shell, or use sccli in interactive mode.
e.g.
# sccli /dev/es/ses3 show logical-volumes
-or-
# sccli /dev/es/ses3
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli> show logical-volumes
sccli supports manipulation of enclosures over the network too - just put the IP address of the 3510 inplace of the device file. Read the man page for further information.
If you don't know the device, just type sccli --list. sccli will then try to determine if any 3510's are attached to the system, and list them accordingly. Make a note of the correct device from the the resulting output.
Okay, let's proceed with the configuration. As I stated above, the first step is checking existing configuration, then we'll remove it if necessary. We'll have a look if there are any logical volumes and/or logical drives are configured. I'll be using sccli in command line mode from my shell.
# sccli /dev/es/ses3 show logical-volumes
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
As you can see, we have no logical volumes defined. Good. This'll save us deleting it. If a volume is defined, you'll see something like the following:
# sccli /dev/es/ses3 show logical-volumes
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
LV LV-ID Size Assigned LDs
-----------------------------------------
lv0 22E5B3F3 544.93GB Primary 2 ld0,ld1
We can then delete it....
# sccli /dev/es/ses3 delete logical-volume lv0
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: lv0:: deleted logical volume
Now, we'll do the same for any defined logical-drives...
# sccli /dev/es/ses3 show logical-drives
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
LD LD-ID Size Assigned Type Disks Spare Failed Status
-------------------------------------------------------------------------
ld0 72BE7D18 272.47GB Primary RAID5 5 2 0 Good
ld1 01558EE6 272.47GB Secondary RAID5 5 2 0 Good
Let's delete them....
# sccli /dev/es/ses3 delete logical-drive ld1
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: ld1: deleted logical drive
# sccli /dev/es/ses3 delete logical-drive ld0
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: ld0: deleted logical drive
With those removed, we can now setup the cache policy
# sccli /dev/es/ses3 set cache-policy write-back random
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
Now we'll want to configure some global spare disks. To do this, check the output of show disks and see what the ID's are before you start
# sccli /dev/es/ses3 show disks
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
Ch Id Size Speed LD Status IDs
----------------------------------------------------------------------
2 16 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 17 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 18 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 19 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 20 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 21 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 22 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 23 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 24 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 25 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 26 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 27 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
We see something important here.... it looks like this enclosure has been used as a supplementary secondary enclosure, as the ID's start at 16 instead of 0. This shouldn't cause a problem, but seeing as this is now a stand-alone primary enclosure, we'll reset the ID's. This is, unfortunately, a physical step. Walk into your server room, pull the plastic cover off of the left hand side of the array, and toggle the switch until a zero is displayed. Replace the cover and return to your desk.
First, reset the controller
# sccli reset controller
WARNING: This is a potentially dangerous operation.
The controller will go offline for several minutes.
Data loss may occur if the controller is currently in use.
Are you sure? yes
sccli: shutting down controller...
sccli: controller is shut down
sccli: resetting controller...
sccli: controller has been reset
Now, have a look at the output from show disks again to ensure ID's have been reset.
# sccli /dev/es/ses3 show disks
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
Ch Id Size Speed LD Status IDs
----------------------------------------------------------------------
2 0 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 1 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 2 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 3 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 4 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 5 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 6 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 7 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 8 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 9 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 10 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 11 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
Good, the device ID's are reset, now we'll add two global spare disks. It's pretty much up to you which disks you mark as global
# sccli /dev/es/ses3 configure global-spare 2.5
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
# sccli /dev/es/ses3 configure global-spare 2.11
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
# sccli /dev/es/ses3 show disks
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
Ch Id Size Speed LD Status IDs
----------------------------------------------------------------------
2 0 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 1 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 2 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 3 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 4 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 5 68.37GB 200MB NONE STAND-BY HITACHI DK32EJ72FSUN72G 2Q09
2 6 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 7 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 8 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 9 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 10 68.37GB 200MB NONE NEW HITACHI DK32EJ72FSUN72G 2Q09
2 11 68.37GB 200MB NONE STAND-BY HITACHI DK32EJ72FSUN72G 2Q09
As you can see from the above output, our two global spare disks are setup - they appear in state STAND-BY to indicate they're ready for use if required.
Now we are ready to use create our two logical drives. This will take some time whilst the drives are initialised....
# sccli /dev/es/ses3 create logical-drive raid5 2.0,2.1,2.2,2.3,2.4
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: created logical drive 4227D9B2
# sccli /dev/es/ses3 create logical-drive raid5 2.6,2.7,2.8,2.9,2.10
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: created logical drive 20C727D1
Use the following command to monitor the initialisation of the logical drives.
# sccli /dev/es/ses3 show logical-drives
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
LD LD-ID Size Assigned Type Disks Spare Failed Status
-------------------------------------------------------------------------
ld0 4227D9B2 0MB Primary RAID5 5 6 0 Initing
ld1 20C727D1 0MB Primary RAID5 5 6 0 Initing
Keep monitoring the initialisation process until both logical drives show status "Good"
# sccli /dev/es/ses3 show logical-drives
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
LD LD-ID Size Assigned Type Disks Spare Failed Status
-------------------------------------------------------------------------
ld0 4227D9B2 272.47GB Primary RAID5 5 6 0 Good
ld1 20C727D1 272.47GB Primary RAID5 5 6 0 Good
Now we can create our logical volume using both of these logical drives. This is very dependent on your requirements. You could also create two seperate logical volumes (each containing one logical drive), map these to two seperate LUN's - and then use 3PP software such as VxVM to mirror the two sides. Whilst we are using VxVM on the system (to mirror the root disks for one thing), we'll let the firmware of the 3510 take care of the disks using the inbuilt fault-tolerance of RAID5 together with the global spare disks added earlier.
# sccli /dev/es/ses3 create logical-volume ld0,ld1 primary
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: created logical volume 22E5B3F3
Let's check that the logical volume has been created successfully....
# sccli /dev/es/ses3 show logical-volumes
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
LV LV-ID Size Assigned LDs
-----------------------------------------
lv0 22E5B3F3 544.93GB Primary 2 ld0,ld1
Perfect - we now have our 544.93GB logical volume ready for mapping to a LUN!
# sccli /dev/es/ses3 show channels
Ch Type Media Speed Width PID / SID
--------------------------------------------
0 Host FC 2G Serial 40 / NA
1 Host FC 2G Serial NA / 42
2 Drive FC 2G Serial 11 / 15
3 Drive FC 2G Serial 14 / 15
4 Host FC 2G Serial 44 / NA
5 Host FC 2G Serial NA / 46
6 Host LAN N/A Serial NA / NA
# sccli /dev/es/ses3 map lv0 0.40.0
sccli: selected device /dev/es/ses3 [SUN StorEdge 3510 SN#07B8B9]
sccli: mapping lv0 to 0.40.0
We've found the PID (target) for Channel 0 and mapped the logical volume accordingly.
Now, we'll configure Solaris so that we see the /dev/es/ses3 enclosure and assign a "normal" device file to the enclosure.
# devfsadm
# cfgadm
Ap_Id Type Receptacle Occupant Condition
N0.IB8 PCI_I/O_Boa connected configured ok
N0.SB2 CPU_V2 connected configured ok
c0 scsi-bus connected configured unknown
c1 scsi-bus connected configured unknown
c2 scsi-bus connected unconfigured unknown
c3 scsi-bus connected unconfigured unknown
c4 fc-private connected configured unknown
c6 fc-private connected configured ok
# cfgadm -c configure c4
Okay, we've run devfsadm to rebuild the /dev and /devices directories. Then we ran cfgadm to view controller configuration - as you can see c4 has type fc-private (i.e. Fibre Channel) and is in state unknown, so we've used cfgadm -c configure c4 to configure that controller.
Now we just need to label the disk. Use the format command for this. If you don't see the "configured with capacity of blah GB" message, then reboot -- -rs. The system will then complain about it's unlabled disk as it boots, and you can enter format then.
# format
Searching for disks...done
c4t40d0: configured with capacity of 544.90GB
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/ssm@0,0/pci@18,700000/pci@1/scsi@2/sd@0,0
1. c1t0d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/ssm@0,0/pci@1c,700000/pci@1/scsi@2/sd@0,0
2. c4t40d0 <SUN-StorEdge3510-327P cyl 35286 alt 2 hd 255 sec 127>
/ssm@0,0/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff87b8b9,0
3. c6t40d1 <SUN-StorEdge3510-327P cyl 35425 alt 2 hd 127 sec 127>
/ssm@0,0/pci@18,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff87b9bd,1
Specify disk (enter its number): 3
selecting c4t40d0
[disk formatted]
Disk not labeled. Label it now? yes
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> type
AVAILABLE DRIVE TYPES:
0. Auto configure
1. Quantum ProDrive 80S
2. Quantum ProDrive 105S
3. CDC Wren IV 94171-344
4. SUN0104
5. SUN0207
6. SUN0327
7. SUN0340
8. SUN0424
9. SUN0535
10. SUN0669
11. SUN1.0G
12. SUN1.05
13. SUN1.3G
14. SUN2.1G
15. SUN2.9G
16. Zip 100
17. Zip 250
18. SUN36G
19. SUN-StorEdge3510-327P
20. SUN-StorEdge3510-327P
21. other
Specify disk type (enter its number)[20]: 0
c4t40d0: configured with capacity of 544.90GB
<SUN-StorEdge3510-327P cyl 35286 alt 2 hd 255 sec 127>
selecting c4t40d0
[disk formatted]
format> label
Ready to label disk, continue? yes
format> quit
Now we just need to make VxVM aware of the new addition....
# vxdctl enable
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced rootdisk rootdg online
c1t0d0s2 sliced rootmirr rootdg online
c4t40d0s2 sliced - - error
c6t40d1s2 sliced disk2 foodg online
# vxdiskadm
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
..snip..
Select an operation to perform: 1
..snip..
Select disk devices to add: [<pattern-list>,all,list,q,?] list
DEVICE DISK GROUP STATUS
c0t0d0 rootdisk rootdg online
c1t0d0 rootmirr rootdg online
c4t40d0 - - error
c6t40d1 disk2 foodg online
Select disk devices to add: [<pattern-list>,all,list,q,?] c4t40d0
Here is the disk selected. Output format: [Device_Name]
c4t40d0
Continue operation? [y,n,q,?] (default: y)
..snip..
Which disk group [<group>,none,list,q,?] (default: rootdg) foodg
Use a default disk name for the disk? [y,n,q,?] (default: y) n
Add disk as a spare disk for foodg? [y,n,q,?] (default: n)
Exclude disk from hot-relocation use? [y,n,q,?] (default: n) y
The selected disks will be added to the disk group foodg with
disk names that you will specify interactively. They will be
marked as nohotuses and excluded from hot-relocation use.
c4t40d0
Continue with operation? [y,n,q,?] (default: y)
The following disk device has a valid VTOC, but does not appear to have
been initialized for the Volume Manager. If there is data on the disk
that should NOT be destroyed you should encapsulate the existing disk
partitions as volumes instead of adding the disk as a new disk.
Output format: [Device_Name]
c4t40d0
Encapsulate this device? [y,n,q,?] (default: y)
The following disk has been selected for encapsulation.
Output format: [Device_Name]
c4t40d0
Continue with encapsulation? [y,n,q,?] (default: y)
Enter disk name for c4t40d0 [<name>,q,?] (default: foodg01) disk2mirr
The disk device c4t40d0 will be encapsulated and added to the disk group
foodg with the disk name disk2mirr.
Use a default private region length for this disk?
[y,n,q,?] (default: y)
The first stage of encapsulation has completed successfully. You
should now reboot your system at the earliest possible opportunity.
The encapsulation will require two or three reboots which will happen
automatically after the next reboot. To reboot execute the command:
shutdown -g0 -y -i6
This will update the /etc/vfstab file so that volume devices are
used to mount the file systems on this disk device. You will need
to update any other references such as backup scripts, databases,
or manually created swap devices.
Add or initialize other disks? [y,n,q,?] (default: n)
At this stage the new disk should be initialised and ready for use in VxVM and ready to use. If it shows a status of 'error' try rebooting the server.
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced rootdisk rootdg online
c1t0d0s2 sliced rootmirr rootdg online
c4t40d0s2 sliced disk2mirr foodg online
c6t40d1s2 sliced disk2 foodg online
That's it - the 3510 is configured, and is under the control of Veritas Volume Manager. You can now use it to store pictures of cats, recipes for hearty soups or your ever-expanding Oracle database of people with funny names.
Cheers
Kevin Waldron
kevin@zazzybob.com
Disclaimer! - This article is provided for guidance only, and does not replace the relevant official documentation and manuals. I will not be held liable for any hosed systems and/or data.
