Back to home
Poor man SAN This project really came to life because i wanted a better solution for a situation i needed a quick cluster of xen servers, but didn't have the resources to go and buy raid cards and SAS disks for all the nodes.

The idea was this: i have a nice hp dl380 g7 laying around, some Qlogic HBA 4g cards, SFP fiber cables and some 1 U servers with no room or fancy hot swap bays.
Knowing EMC SAN systems, i figured: why not build myself a SAN system, using this hardware and freebsd?

Sure, it's not a top of the line setup with a nice FC storage, but it will do the job just fine for what i need it to do.
So here is my solution to an entire setup based on freebsd and a SAN on fiber channel.
Preparations First and foremost, the qlogic cards need to go in the servers. Each just gets one, since i have no need for multipath and other fancy stuff.
Also, i picked up a HP storageworks 8/8 switch for 30 euro's, with 4 8G sfp units in it. It will work with a straight connection between the HBA's, but that will restrict you to just one host on the storage.

The switch has to be configured with zones. I simply set up all the ports in 1 zone, it doesn't matter in this case to isolate all nodes in a seperate zone (i.e. initiator zones).

Next, it's setting up the storage server so it serves up target luns on the SAN, then configuring the qlogic cards so the nodes can boot from a lun on the storage and that's it!
So here goes: the entire description on how you set it up
Storage setup The storage i have is a DL380-G7 machine with 1 SAS disk cage, 1 Xeon cpu, 36G of memory and 8 200G SAS disks.
I have it laying around from another project, but anything will do really, up to and including a nice big storage based on SSD disks!

First, install the OS. In this case: freebsd 11.1, freely downloadable at www.freebsd.org. Make sure you include the kernel source! It's needed for the kernel re-compile.
For ease of editting, i like to have nano editor on my systems, it is easier to use than vi.
Freebsd uses the spi kernel driver for qlogic cards, but won't function as a target. To make things work, 2 things need to be done:
When the server boots, the next step is to create storage luns.
First, make sure the HBA is working by using ctladm:
checking hba
#ctladm port -l

Port Online Frontend Name     pp vp
0    NO     camsim   camsim   0  0  naa.50000000e5683b01
1    YES    ioctl    ioctl    0  0
2    YES    tpc      tpc      0  0
3    YES    camtgt   isp0     0  0  naa.5001438000c43a9a
In the above list, port 3 is the HBA, reporting it is online (i.e. seeing light). If it is offline, probably there is something wrong with the fiber that is connected, or nothing is connected at all.
take note of the naa. id number of that port, it is needed later!

Note: If you get a NO and no id number, try the command: ctladm port -o on isp0

Now that the HBA seems to work, storage is needed!
For this, i employed ZFS. Not really a necessity, because the server has a RAID card in it, but it allows to make volumes without having to partition the harddisk volume and is more flexible.
For this, you need to know your disk names. Mine were da0 and da1 (2 raid 5 volumes).
creating storage
#gpart create -s GPT da0
#gpart add -t freebsd-zfs da0
#gpart label disk0 da0p1
#gpart create -s GPT da1
#gpart add -t freebsd-zfs da1
#gpart label disk0 da1p1
#zpool create targetspool /dev/da0p1 /dev/da1p1
#df
...
targetspool  used.. etc.
The final df command should show the targetspool is mounted as a file system.
Now we can proceed to create lun's for the strage by creating zfs volumes:
creating volumes
#zfs set sync=disable targetspool
#zfs create -V 5G targetspool/osboot1
#zfs create -V 500G -b 512 targetspool/vmstore1
This will create 1 boot lun's and 1 storage for VM's.
Yes, the first command will kill ZIL entirely, but in some cases it does improve performance.
The reason for the -b 512 is that if you decide to use VMware on your nodes, they seem to prefer this blocksize.

Last part: create the LUN's for the HBA
For this, you create a file called /etc/ctl.conf
creating ctl.conf
#nano /etc/ctl.conf
-- add the lines:
portal-group pg0 {
        discovery-auth-group no-authentication
}

lun osboot1 {
        backend block
        path /dev/zvol/datastore1/osboot1
        ctl-lun 0
        device-id OSBOOT1
        device-type 0
        serial 000000001
        option vendor "FreeBSD"
        option product "osboot1"
}

lun vmstore1 {
        backend block
        path /dev/zvol/datastore1/esxstore1
        ctl-lun 1
        device-id VMSTORE1
        device-type 0
        serial 100000001
        option vendor "FreeBSD"
        option product "vmstore"
}

target naa.5001438000c43a9a {
        auth-group no-authentication
        portal-group pg0
        port isp0
        lun 0 osboot1
        lun 1 vmstore1
}
Note: once you start installing on these luns, do NOT change the order in which they are set in ctl.conf! This is because when installing, all the luns are 'seen' by the installer. If you change the lun order, the bootvolume changes and so the reference to the harddisk used.
This will result in not bootable machines. So it's a good idea to keep a copy of this file somewhere...

The last thing to do is enable ctld in our config:
creating ctl.conf
#echo ctld_enable="YES" >> /etc/rc.conf
After a reboot, nfs should be up and running, and the targets should be visible on the SAN.

To make sure the luns are connected, use the command
checking luns
ctladm devlist -v
You should see your created luns attached with the correct configuration.
Node setup The nodes can vary, but all have one thing in common: no disks in it.
They all will boot from the storage server over FC.
Installing them is straight-forward:

Put a HBA in it and boot it up. Keep an eye on the boot sequence, as soon as you see the HBA (Qlogic in this case) bios starting, hit ctrl-Q
now, in the card config bios, select 'adapter settings'.
first, set the 'host adapter bios' to enabled.
Then, select 'selectable boot settings'. Set 'selectable boot' to enabled, and use the cursor keys to go to the first boot port name, hit enter
If everything is working correctly, you will see the target on the list.
Select the target and the luns that are visible to the card will be displayed.
Select the correct lun number.
Save it all, and reboot.

Now, if there is no OS installed, any OS can be installed as usual on that volume.
If there is, it will boot that os
Extention 1 - provisioning One other thing came to mind when working on this. In the server business there is something called 'provisioning'. Particularly in blade systems this is used.
This setup can actually do something like it with a little bit of preparation!
For this, each HBA is assigned a number (1... x).
For the setup, we create a temporary (high number) lun for each card. It really doesn't matter they all point to the same ZFS volume, it's just for configuring the HBA.
On each HBA, we do the setup as before, making sure the boot lun is the same number as the card is assigned. Once done, we can remove all the luns for them.

Now, whenever we want a parcitular OS booting on a particular server, we check the card number, point the lun number for that card to the correct zfs volume and it will boot.

It is a bit more work, because you will need luns ready where the OS is installed.
So for instance:
lun 0 - some windows install
lun 1 - some linux install

Because of the previous note about not changing lun numbers, these have to be available always, and cannot be used twice!
So booting 2 machines with the same OS from the same volume is impossible.
The reason the 'old' luns need to be available is because again, the install was made to a particular lun number. The booter of the OS will remember this, and will try and find the OS on that lun, regardless of what lun you started it from.

Example:
You configured card number 1 to boot from lun number 101. Now, you enter in ctl.conf that lun 101 uses the same path as lun 0 does.
When booting, the card will boot from lun 101, but then the bootloader there will immediately switch to lun 0, because that is where it was installed initially.
If lun 0 isn't available, it will crash and stop booting.
Extention 2 - redundancy It is possible to make the setup redundant. FC has the option of multipathing. This setup is easy extendable to it by simply sticking 2 HBA cards in all the servers, get another SAN switch and that pretty much is it.
It depends then on the OS how to support it.
Also, it is possible to setup the 2 HBA cards to boot from the exact same LUN. Only one will be selected by the server bios to boot from. If you set one HBA as the first, and the other one as the second one, the server can boot as long as one of the FC is active.
On the storage, the ctl.conf gets copied for the target, but with the correct target number (naa.xxxxxxx)
Extention 3 - more storage Adding more storage or more storage servers is easy: set it up using the server setup above, but make sure the target in ctl.conf is correct (particularly: the naa.xxxxxxx number).
Also, with another storage, you can select one for all the boot luns, the other one for data sharing.
Extention 4 - storage replication Far from being perfect, and there are better ideas for it, but having the setup (specially if it is fully redundant) it is possible to have storage replication in a simple way, provided the OS can do ZFS!
All you need: a LUN on both storage server of the same size. Then on the node/host server, you simply make a zfs mirror over these to luns and it will replicate all data on both luns.
again, this is in no way the same as a blocklevel sync on a dedicated 10Gbit link. But as a cheap temporary solution to non-critical data... why not?