| Sign in: |
| Members log in here with your user name and password to access the your admin page and other special features. |
|
|
|
|
|| SportsShooter.com: Member Message Board
WANTED: DEAD OR ALIVE. Archiving solution for the office.
Garrett Hubbard, Photographer
 |
Washington | D.C. | USA | Posted: 2:18 PM on 10.11.10 |
->> I am looking for some advice on an archiving solution for many Terabytes of data that has been living on each users external drive. I don't think we have the budget for a centralized server -- however you all may know something that I don't. The goal is to have a multi-user accessible & searchable list of the contents of those drives.
I am suggesting that we use buy a few external SATA controllers and duplicate the material on our existing external disks onto bare SATA drives and put them on the shelf as our archive. My boss suggested taking a screenshot of the directories/folder structure and placing those with each drive that goes on the shelf. Since drives are inexpensive, we may duplicate each drive.
I think I like that as an inexpensive archive solution. However, I don't know of a way to make this searchable by any database. Any ideas or alternative solutions? Thank you in advance for taking the time to give feedback and advice! |
|
Israel Shirk, Photographer, Assistant
|
Boise | ID | US | Posted: 3:00 PM on 10.11.10 |
->> To bring all the storage together you could just plug in the drives to a single computer and do software RAID, sharing to other computers as necessary... Not the fastest thing but archives generally don't need to be. Later on you can upgrade to a dedicated hardware RAID solution and just drop the same drives into it. (though it makes more sense to just go ahead with the hardware to me)
If you're going to pick up new drives too, you could install the new drives to the RAID and start it off clean with a RAID-5 (or RAID-0 - more reliable) array to maximize storage space. Label each drive when you add it so that if it fails you don't have to guess which one to replace. Dump the old drives' data onto the raid array, and after you have done the dump, add the drives to the array to increase redundancy. You can do all sorts of things to increase the number of drives you can attach - adding new SATA (or RAID/SATA) controllers and port expanders are both pretty simple to do.
If you use a hardware RAID array, you're limited to the drives you can attach to one card - I think it's 64 or 128, depending on the card. Hardware RAID's are usually faster and more reliable - bugs in the OS won't affect the array's consistency.
I'd be pretty hesitant to store drives where people have to walk up and pick them up; there's so much that can go wrong from static shock and dropping to drives getting mixed up, etc. At the same time, you have to monitor the RAID array periodically to check which drives have failed and replace them. |
|
Philip Johnson, Photographer
|
Garland | TX | USA | Posted: 3:27 PM on 10.11.10 |
->> I have to disagree with Israel on the Raid 0; it does not provide any archiving protection. Raid 0 is only striping the data between two drives. Raid 1 provides mirroring of your data. Raids 5 and 10 or 0+1 are your other options.
Checkout the information on OWC about raids, it helped make a little clearer picture for me. I'm not sure that you can use existing drives for the raid configuration. I think it will reformat the drives if you try that. |
|
Travis Haughton, Photographer
|
Crystal Lake | IL | USA | Posted: 4:50 PM on 10.11.10 |
| ->> Drobo |
|
Tom Story, Photographer
|
Tempe | AZ | USA | Posted: 5:04 PM on 10.11.10 |
| ->> I did something like this. But you'd think an outfit as big as yours would have something like a Merlin system with is multi users and searchable database, etc. |
|
Jim Colburn, Photo Editor, Photographer
|
McAllen | TX | USA | Posted: 5:06 PM on 10.11.10 |
| ->> Do you have an old Mac Tower hanging around? Throw in some big-ass hard drives, install 10.5 or later, connect it to your Mac network via Ethernet and you'll have a search-able (via Spotlight) file server. |
|
Israel Shirk, Photographer, Assistant
|
Boise | ID | US | Posted: 10:53 AM on 10.12.10 |
| ->> Sorry, meant RAID-0... |
|
Israel Shirk, Photographer, Assistant
|
Boise | ID | US | Posted: 10:54 AM on 10.12.10 |
| ->> 1... |
|
John Blankfort, Photographer
|
Monterey | CA | USA | Posted: 4:54 PM on 10.12.10 |
->> Garrett,
I know you are on a budget, but I would go with a RAID system. RAID 5 will give you the most bang for the buck and still protect you from a single drive failure. Of course, it will NOT protect you if someone steals the RAID box or you have a fire.
I would recommend a two step approach to protect you from both hardware failure and also provide off site storage.
1) Setup a RAID 5 box
2) Use external hard drives to backup the RAID box and take it off site.
For the RAID box… I would consider either a Drobo or a Qnap. They are much smaller than a desktop and you don’t need a keyboard or monitor. They also offer easy integration with a UPS, so when the power goes off you don’t risk data corruption. I have a Qnap. It’s got built in backup, FTP server, web server and programmable alarms that generate emails if a drive fails, get to hot or the machine reboots for no reason. You can also easily access Qnap remotely and they even have a iphone app.
I don’t know how much data you have, but three 2TB drives under RAID 5 gives you about 3.6TB of storage. Good 2TB drives are about $120 bucks and a 4 bay Qnap is about $900. So, for about $1260 you would have 3.6TB of storage. If you need more storage in the future… Just slap in another hard drive and the qnap will rebuild the RAID for you. You can get Qnaps from 2 bays to 8 bays. Buy one that gives you a little room to grow.
Most likely you can use your existing external drives for off site backup. |
|
Garrett Hubbard, Photographer
|
Washington | D.C. | USA | Posted: 5:15 PM on 10.12.10 |
->> Thanks to all who have replied here and via direct message. I really appreciate your help. I should have been more clear on the quantity of TB we're looking to store/archive.
when I say we have "many" TB of data, I'm estimating 20 TB--and that number is growing every day. |
|
Kevin M. Cox, Photographer, Assistant
 |
Galveston & Houston | TX | US | Posted: 6:53 PM on 10.12.10 |
->> It sounds like you're going to need to separate out the physical storage and searchable archive aspect of this project. Once you decide on physical storage you can then tackle how to search it.
For a more "small office" approach the ReadyNAS 3200 will get you around 22 TB of space using 2 TB drives:
http://www.readynas.com/?cat=73
Combine that with some software package like Portfolio to get the searchable aspect going.
You also haven't specified if the clients are Mac or PC which could make a difference in recommended software applications. |
|
Israel Shirk, Photographer, Assistant
|
Boise | ID | US | Posted: 2:13 AM on 10.13.10 |
->> Skip on drobo and the other proprietary systems... They're not real raid systems, so if anything goes south you're SOL.
Set up your own NAS using FreeNAS, a nice RAID controller, and a couple external towers to hold drives. Get two of these and fill with drives: (plug them in with eSATA to the main computer)...
FreeNAS links: http://www.freenas.org, installation video here: http://www.youtube.com/watch?v=5S8ixAR4Opo
For the raid controller, you want a 3ware or LSI that does SATA+SAS (serially attached storage). A lot of the others do everything in software, which is very very slow. You want one that'll support at least 4 external ports (so you can attach 4 external drive towers). Check compatibility with FreeBSD version 7.2 for the controller to make sure it'll work. I would use the RAID controller's BIOS setup to control all RAID functions rather than FreeNAS's - when you set up the RAID in FreeNAS, it's doing it in software (slow and not super-reliable), rather than in the nice controller you just paid for.
For the towers get a few of these: http://www.amazon.com/Sans-Digital-TowerRAID-TR8M-B-Multiplier/dp/B001LF40R... You plug them in to the eSATA ports on the RAID card, and they allow you to plug more drives in. There might be something more fitting but you get the idea - it needs to multiply the ports and provide power and ventilation.
The server itself just needs to be pretty barebones - a nice gigabit network card, good motherboard and processor and good case (plenty of ventilation in a clean, away-from-people location). Run FreeNAS from a compact flash card or usb stick and then after you've installed it, make a duplicate of the card so you have a backup.
The nice thing about this system is that if the server itself goes south, your data (on the external enclosures) just needs to be attached to a new (preferably similar) server with the same FreeNAS installation to recover. Same for the enclosures themselves. You can expand at will by adding more enclosures and drives (or increasing drive capacity). |
|
Israel Shirk, Photographer, Assistant
|
Boise | ID | US | Posted: 2:14 AM on 10.13.10 |
| ->> (the "get two of these..." should point to the amazon link) |
|
David Meyer, Photographer
|
Kent Miller, Photographer
|
Garrett Hubbard, Photographer
|
Washington | D.C. | USA | Posted: 5:48 PM on 10.14.10 |
->> Great stuff. thanks everyone.
Kent, for offline data, I'm going to pass along the solution Smiley Pool sent me. Check out the video demo's. Great solution IMO.
http://www.cdfinder.de/index.html
I'll try and resurrect this thread and let you know what we choose and why. |
|
Kent Miller, Photographer
|
New York | NY | USA | Posted: 11:20 PM on 10.16.10 |
| ->> Garrett, Thanks.. I am giving it a try right now. Let me know if you need any info on the steps I take. k |
|
Return to --> Message Board Main Index
|