What is better a raid configuration or single disks working in an edius sp system?

[redgum]

Hi Ron, there's a flaw in your math. It doesn't matter how many single drives you have their speed factor is still 1 (not 2) so a Raid will always be faster than a single drive. Fragmentation doesn't enter the equation because a file could fragment to 100 pieces on a single drive and to 3 pieces on a Raid (or vice versus). Fragmentation will effect both setups equally depending on how it happens and over which we have no control.
And every file is written to its source in an NLE including those directed to or from a scratch disc. If this didn't happen your edit would come to a grinding halt with the NLE unable to find its files.
Hard disks have and are the bottleneck in any system with their mechanical processing of files, hence the recommendation from manufacturers to install the fastest hard drive possible. That's why SSD will be so advantageous.
Losing files is not an issue, for instance it doesn't matter if you have one 500Gb drive or two 250Gb drives in Raid 0, if either drive crashes you lose all your information but the Raid offers (using your figures) 1.7 times the speed of a single drive. Also, the speed of a Raid is recommended by all NLE manufacturers particularly with larger files.
I was just interested in Brandons comments and reasoning.

[Ron Evans]

Hi Ron, there's a flaw in your math. It doesn't matter how many single drives you have their speed factor is still 1 (not 2) so a Raid will always be faster than a single drive.

Not so. A RAID will be faster for a single operation of read or write. However if the user only has a RAID of two hard drives it will depend on the RAID controller and hard drives used, how efficiently it can read and write at the same time. In the worst case it will be 1.7 divided by two ie .85 as apposed to 1 for a transfer between two single hard drives which only have to read or write not both at the same time. ie 15% slower. With three or more hard drives on a hardware controller it will always be a lot faster than single hard drives. For very small files it really doesn't matter anyway. For large streaming files then the RAID or single hard drive must read continuously and write continuously and then seek and interleave operation does become important in the overall speed of operation. With these larger files DV or HDV only need 3.5MBps of throughput which is at least 10 times less than the performance of almost any drive one would buy today. In all my tests hard drives are not the bottleneck but RAM and CPU certainly are the issue. If there is not enough RAM then the swap file location is also important and may end up being the bottleneck if it is on the boot drive.. My projects are theatre shows with three or four cameras that run to 2 hours or more and I have never had a problem using the approach I mentioned. Rendering is the issue and this is CPU and RAM based not hard drives. I accept the fact that if I was using uncompressed files the issue may be different but still maintain the 2 hard drive RAID is not the best solution even in this case for the reasons stated above. It makes administration of files easier.
I don't understand your comments of files written to source. Could you explain. Editing information is written to the project file, source files are not altered at all. Rendered files are extra to the source files and the reason that scratch/temp files are written. This again is the reason for me having project and temp files on discs other than the source files and final render to yet another disc and definitely not the boot drive. All discs run at full speed and only have to read or write never both at the same time. In the case of someone with just the 2 disc array it will have to manage all these operation at the same time. It will most definitely be slower for long programs. For short programs it may be driven purely by buffer size!!
Ron Evans

[pjsssss]

Ron, a quick request...these long posts would be soooo much easier to read if they had a few paragraphs instead of one big one.

[redgum]

Not so. A RAID will be faster for a single operation of read or write. However if the user only has a RAID of two hard drives it will depend on the RAID controller and hard drives used, how efficiently it can read and write at the same time. In the worst case it will be 1.7 divided by two ie .85 as apposed to 1 for a transfer between two single hard drives which only have to read or write not both at the same time. ie 15% slower. With three or more hard drives on a hardware controller it will always be a lot faster than single hard drives.

Ron, can we look past the CPU & RAM for a moment and concentrate on the RAID 0.
You made the comment that with 3 or 4 drives a Raid will be much faster than a single drive. Overall, that is very true. But the principal of read/write doesn't change between a two disc Raid and say a four disc Raid. The process MAY put a file in four different places.
With read/write you're assuming a very basic operation and that both processes consume the full attention of the hard drive which is never the case unless as you say the files are uncompressed or in fact HQ AVI. Normal HDV/DV files are not in question.
Your theory appears to be based on a software or on-board hardware Raid both of which are dependent on the CPU and RAM. I run 8 drives off a 3Ware 8PL Raid card which is blazenly fast but even with two drives under just about any condition it well outperforms two single drives. Of course, all reasonable Raid cards carry their own CPU and RAM which addresses the other issues you spoke of.

[Ron Evans]

There is a difference whether the RAID is two drives or more. RAID 0 requires CPU process to divide the files and manage the write to the discs in the array and assemble for a read. This is not free, it takes CPU, RAM and time. When this has to simultaneously both read and write to the same two drives it takes even more of these resources and time. The individual drives do not escape both a read and write operation with fragmentation of the file and multiple seeks on just two drives rather than a potentially contiguous file on a clean single drive. In the single drive case there is a single read or write operation and in the case of the RAID both drives have multiple seek operation for both read and write simultaneously. The load on individual drives and the processing is proportionally less as the number of drives increase.

Most people for whom video is a hobby ( like me) buy hard drives one at a time and are rarely the same. I for instance have drives in sizes from 160G to 1T. To gain the advantage of the RAID 0 drives have to be the same otherwise the capacity and speed will be dependent on the smallest and slowest drive, times the number of drives. To use more than 3 or 4 drives in a RAID 0 increases the risk of drive failure and total loss of all data. This is not something I want to do even as a hobby.

If one doesn't need the performance( for HDV and DV) why bother with the complexity. If the business requires the performance and ease of file management then a RAID 5 is the best choice with associated costs and the need for at least 4 identical drives. Hence my view that even as a hobby RAID 0 is not something that I want to use. It's only real value was to get a larger drive when drives were small which is what I did when I had drives too small to edit anything of use. I started with 200Meg drives !!! The moment drives got larger and faster I stopped using RAID.

Ron Evans

[swsw1550]

A non-redundant disk array, or RAID level 0, has the lowest cost of any RAID organization because it does not employ redundancy at all. This scheme offers the best performance since it never needs to update redundant information. Surprisingly, it does not have the best performance. Redundancy schemes that duplicate data, such as mirroring, can perform better on reads by selectively scheduling requests on the disk with the shortest expected seek and rotational delays. Without, redundancy, any single disk failure will result in data-loss. Non-redundant disk arrays are widely used in super-computing environments where performance and capacity, rather than reliability, are the primary concerns. A bit about raid arrays..........



Level 0 is a 'striped' disk array without fault tolerance. It provides data striping (spreading out blocks of each file across multiple disk drives) but no redundancy. This improves performance but does not deliver fault tolerance. If one drive fails then all data in the array is lost.


Level 1 does 'mirroring' and 'duplexing'. It provides disk mirroring. Level 1 provides twice the read transaction rate of single disks and the same write transaction rate as single disks.


Level 2 does 'error-correcting coding'. It isn't a typical implementation and rarely used. Level 2 stripes data at the bit level rather than the block level.


Level 3 is 'bit-interleaved parity'. It provides byte-level striping with a dedicated parity disk. Level 3, which cannot service simultaneous multiple requests, also is rarely used.


Level 4 is 'dedicated parity drive'. It is a commonly used implementation of RAID. Level 4 provides block-level striping (like Level 0) with a parity disk. If a data disk fails, the parity data is used to create a replacement disk. A disadvantage to Level 4 is that the parity disk can create write bottleneck

Level 5 is 'block interleaved distributed parity'. It provides data striping at the byte level and also stripe error correction information. This results in excellent performance and good fault tolerance. Level 5 is one of the most popular implementations of RAID.


Level 6 is 'independent data disks with double parity'. It provides block-level striping with parity data distributed across all disks.



Level 0+1 is 'a mirror of stripes'. It isn't one of the original RAID levels. Two RAID 0 stripes are created, and a RAID 1 mirror is created over them. Used for both replicating and sharing data among disks.



Level 10 is 'a stripe of mirrors'. It isn't one of the original RAID levels. Multiple RAID 1 mirrors are created, and a RAID 0 stripe is created over these.



Level 7 is a trademark of "Storage Computer Corporation" that adds caching to Levels 3 or 4.


RAID 50 should really be called "RAID 03" because it is implemented as a striped (RAID level 0) array whose segments are RAID 3 arrays. RAID 50 has the same fault tolerance as RAID 3 as well as the same fault tolerance overhead. High data transfer rates are achieved thanks to its RAID 3 array segments. High I/O rates for small requests are achieved thanks to its RAID 0 striping. Maybe a good solution for sites who would have otherwise gone with RAID 3 but need some additional performance boost. Very expensive to implement. All disk spindles must be synchronized, which limits the choice of drives. Byte striping results in poor utilization of formatted capacity.

RAID S:
RAID S is "EMC Corporation's" proprietary striped pairty RAID system used in its Symmetrix storage systems.

Anyway hope this helps and explains about the different Raids ect.

Steve

[swsw1550]

Also guys this is another option that I run sometimes, If you want the best of both worlds, RAID0 and RAID1 can be combined to create either RAID0+1 or RAID1+0. RAID0+1 is a mirror of stripes, so that when one drive fails in an array, it can be rebuilt from the other. This type of array cannot rebuild data from a failure on more than one sub array. RAID1+0 is a stripe of mirrors where a drive from each mirror could fail without causing data loss and is therefore more fault tolerant to multiple drive failures.. In each of these, you get the advantage of both fault tolerance and performance, but you need double the amount of drives for the same capacity as RAID0 (so this means at least 4 drives).

Steve

[robertse]

Steve,

Would this implementation be best served with a separate raid card or would a software based raid be able to acheive this? vista64, cpu I7, asus p6t mb

[STORMDAVE]

I use RAID 6. Couldn't be happier.

6x1.5TB drives=6TB of space. 2 drives are used for redundancy. If 2 drives die, then I still have the data on the remaining 4 drives. I did a test on this and it works flawlessly. I took out 2 random drives, and the data was still there. I put them back in and it rebuild the array in under 24 hours.

I use an Areca hardware raid card. I highly recommend using a hardware RAID card. It is secure, doesn't use the CPU and has many options.

I also got over 300MB/sec on read and write (natively) on this setup. I can also get 110MB/sec per editing machine via Gigabit lan.

[gdame]

Some different types of raid and a bit on how it works, I hope this helps you a bit,
RAID 0
RAID 0 is also called striping and treats two or more physical drives as one large drive. This style of RAID provides no redundancy, however it does provide speed advantages. If you installed two 80Gb drives configured as RAID 0, your PC would see this as one 160Gb drive going almost twice as fast as a single drive. The reason for this is because the data is read from both physical drives simultaneously.

RAID 1
RAID 1 is also called mirroring. In this configuration, the RAID controller keeps identical copies of the data on each drive. If one drive fails, you have a copy of the information on the other drive. When the failed drive is replaced, your RAID controller will copy the information back such that your redundancy is restored. There is no speed boost with RAID 1, the advantage is redundancy.

RAID 5
RAID 5 is the deluxe option. It is not supported by all controllers so it is important to confirm it is supported when reviewing controller options. RAID 5 uses a minimum of three drives; however the number can be as many as eight or more for larger implementations. RAID 5 looks at the data on all the drives and uses this detail to calculate a checksum. The checksum is stored as an overhead to the data such that if any single drive fails, the array can be rebuilt by recalculating the data based on the checksums. RAID 5 provides excellent performance by supplying the speed of many drives reading and writing simultaneously, combined with checksum redundancy. Your effective drive capacity is calculated by the number of drives in the array minus 1. e.g. 5 x 80GB drives in RAID 5 will give you 320 GB.

Mix’n’match your RAID
You can combine different RAID configurations in order to gain advantages from the various RAID types. For example, you could combine RAID 1 and 0 to get both speed and redundancy ie RAID 10. In this example, if you used 4 X 80Gb drives to get a logical 80Gb drive, it would provide twice the speed of a single drive layered with redundancy. If any single drive fails, simply replace it and continue working. There will be no data loss or need to restore as two of the drives will be holding a mirror image of the other two. Choosing which RAID types to mix can be complex as there are many technical considerations.

RAID 5 is the most common RAID implementation for servers, however if you are seeking additional performance and redundancy, you can use RAID 50 which combines two arrays of RAID 5 striped with RAID 0 to provide additional performance.

Cheers
Steve

Steve, You left out RAID 6, My favorite!

[Raja]

I use RAID 6. Couldn't be happier.

6x1.5TB drives=6TB of space. 2 drives are used for redundancy. If 2 drives die, then I still have the data on the remaining 4 drives. I did a test on this and it works flawlessly. I took out 2 random drives, and the data was still there. I put them back in and it rebuild the array in under 24 hours.

I use an Areca hardware raid card. I highly recommend using a hardware RAID card. It is secure, doesn't use the CPU and has many options.

I also got over 300MB/sec on read and write (natively) on this setup. I can also get 110MB/sec per editing machine via Gigabit lan.

StormDave, I also use Gigabit LAN for my current setup. I noticed in your signature that you have Gigabit LAN x 2 on your Dataserver?? Have you got both network ports plugged into a switch (some sort of load balancing) or have you only 1 gigabit port connected (to get 110MB/sec)?