company location

Get In Touch

Why not join us on Facebook? We only post about useful things like new releases, and unlike some others we promise we won't spam you with messages!

           MLtek on Facebook        MLtek on Twitter

We have recently signed up to CrowdReviews.com and we would really appreciate it if you could take a few minutes to leave us some feedback. All reviews are visible to everyone, and your review will be given particular weight if you can verify it with your LinkedIn profile.

 

 

Should I archive my file server to the cloud?

Product Home | Pricing | Download | Documentation | Release Notes | Customers

*Since writing this article Amazon have come a long way and they now offer thier extremely attractive storage tier called 'Glacier'. We fully support archiving to Amazon Glacier through an Amazon Storage Gateway*

We get asked at least once a week if ArchiverFS can move old files to the cloud. The answer is always a resounding yes! However, we advise people to think this through very carefully before they embark on such a project as costs can quickly spiral.

Given the frequency with which we get asked this question we decided to write this article to help people in the decision making process.

The decision to deploy a file server archiving solution brings multiple advantages, including the following:

  • If you migrate all your old files to second line storage you free up space on your expensive high performance first line storage thereby increasing it's lifespan.Disk Drive
  • Backups are faster and you need to back up less data. Old data rarely changes.
  • Restores are quicker. In an emergency you can restore just the frequently used files first and everyone gets back up and running sooner.
  • Compliance. Many countries now have legislation that places obligations on companies to only keep data for as long as it is needed.

There are other drivers but these seem to be the main reasons our customers decide to deploy a product like ArchiverFS.

What actually is the cloud?

I wonder just how many people reading this have sat in a presentation listening to someone tell us about the latest greatest thing through the medium of PowerPoint, when up pops 'the' slide. You know the one, it explains how the solution works and  right smack in the middle of it is a big and normally badly drawn cloud symbol. Ask any technical person what that symbol represents and they'll normally give you an answer like "it's the providers hosting platform\datacenter(s)", but ask a non technical person what it represents and you will often get an answer like "it's the part of the providers solution that holds the data".

So what actaully is it and where did the name the 'cloud' come from?

Back in the late 90's the internet was speeding up in a big way. Lots of businesses were getting broadband for the first time and providers saw an opportunity. Until then most solutions were on-premise meaning that software and\or hardware would be deployed to the customers site. The speeding up of the internet and the spread of business quality internet access enabled providers to change the business model and start hosting their solutions themselves. This change saw in a big rise in the 'subscription' model for software products, or as the software suppliers prefer to call it the 'Software As Service' model.

The challenge at the time was how on earth you could you explain this shift to non-technical senior staff, say the senior partner at an accountancy practice for example?

Salesmen needed something ethereal and 'fluffy' that they could point to and say "Your data is there!". What did they and their marketing managers reach for when they looked for a symbol to represent the new model?

Yup you guessed it.. a cloud.......

Cloud Graphic

So, if you are a technically minded person the 'cloud' is probably ... "A representation of the providers hosting platform which normally comprises of a server and storage infrastructure housed within the providers own (sometimes purpose built) datacenter, which is typically accessible via the internet and whilst benefiting from economies of scale and usually best in class design and tooling still uses technology not too dissimilar to most SME infrastructures."

If you aren't technically minded the cloud is probably... "A representation of the providers service that I don't need to worry about, they host it and they can deal with the issues. We just pay each month instead of a big investment up front and our staff get to use the latest greatest tools.".

Could using the cloud be the answer?

At first glance the idea of offloading all of your old and unused data to someone else's infrastructure seems like a great idea, and maybe it is. You don't have to spend time sourcing additional hardware, putting together project teams, updating infrastructure and supporting documentation, deploying physical kit and all of the other activities that go along with such an activity. In this example we'll use an 'average' file system. Obviously everyone's file system is different, but a lot of our customers file systems seem to fall in the range of 20TB to 50TB of total storage.

Now I know there is obviously selection bias in those numbers, after all this represents OUR customers. Our customers tend to represent the upper end of the spectrum of corporate file systems, the largest file system is 3.4 PetaBytes and the average tends to be around 20TB to 50TB of total storage. As that is the range we know we'll go with 30TB which is pretty much smack bang in the middle.

What do you need to start?

The first thing you'll need is somewhere to migrate your old files too! In this example you'll need around 20TB of storage to make a serious dent in the live file system. Pricing was obtained early in April 2017 and as the providers below are always launching new products you should probably check for yourself just in case they have recently released new VM sizes or storage options.

Each entry represents the best option we could find to obtain 20TB of storage with each provider.

Amazon AWS's offering get's you around 20TB of storage for $2,200 per month.

VM NamevCPURAMInstance StoragePrice Per Hour
d2.4xlarge16122GB12 x 2TB HDD$3.062 p/hour

How about Microsoft Azure? Better.. a similar amount of storage for $718.40 per month.

VM NamevCPURAMInstance StoragePrice Per Month
A347GB22x 1TBS30 HDD$718.40 p/m

And lastly, Google Cloud. About 20TB for $998.42 a month

VM NamevCPURAMInstance StoragePrice Per Month
n1-standard-4415GB22x 1TBS30 HDD$998.42 p/m

With the figures above in mind the best option on the table looks to currently be Microsoft Azure. There may well be other charges involved like bandwidth usage charges, VPN charges and others. Even if one took the Azure option you could be looking at nearly $1000 a month with the extra charges involved.

Customers tend to quickly realise that for under $10,000 as a one off cost they can deploy a glorified NAS device with >20TB of storage onsite. The conversation then almost invariably turns to a request for a recommendation on a suitable device that is tried and tested alongside our software.

So far only one customer has gone ahead with a cloud archiving project, and that was because a) their file server was about to run out of space and b) they didn't have any space in thier server rack for a new device so they would have needed to open up another rack at nearly $1300 a month. They were planning on decommissioning some kit in the near future, so when they have created some space they will be deploying a NAS device and migrating all their files back in house.

But if I really REALLY want to use the cloud, how would I do it?

The process of setting up cloud based file archiving is actually really simple!

Dont Panic!

  1. Sign up to the hosting provider of choice and create a new VM with the appropriate specification (don't forget the extra drives!).
  2. Create a site to site VPN from your datacenter to your cloud platform.
  3. Create additional VPN's from your users sites if required so they can access the cloud server.
  4. Set your cloud servers DNS to point to one of your Active Directory Domain Controllers.
  5. Join your cloud server to your AD.
  6. Configure the disks that are attached to your cloud server as one volume (at last a use for software RAID!).
  7. Create a new share on your cloud server.
  8. Download and install ArchiverFS on your Cloud server.
  9. Create a job to move all of the old content from your live file system to your cloud server.
  10. Watch the used space on your live file server plummet.

The process is really simple and can be carried out in a just a few hours (initial data sync aside). The problem isn't the technology, it's the pricing.

Driven by the number of enquiries we receive, we are actively working on a solution to this and are carrying out a fesability study on setting up a dedicated hosting service. Initially it would be based in the UK, but if it does go ahead and if it is popular enough we will expand it to other countries. We are hoping to hit much lower price points than any of the big players for storage provided specifically for archival purposes when used alongside our software.

The idea is to give customers their own segregated private VM with the potential for 100's of TB's of storage to be attached.

If you have any questions or you would like to express an interest in our potential hosting service please don't hesitate to contact us via [email protected], we would love to hear from you!

Mark Laverty
MLtek