Topic: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

As many of you may have noticed over the past months, our downloads sponsor SimpleCDN has had increasingly many problems with their infrastructure. Very slow downloads have become quite common and even corrupt downloads have been a major source of frustration from time to time.

Many times, we have contacted them, only to get replies that were all similar: they can't reproduce the problem, they've fixed the problem or — quite often — no reply at all.
Last week, they've promised us to move us to their new infrastructure, but have failed to do so. Since we don't seem to be able to get through to them, we have unfortunately also been unable to get new sponsored credit.

Since the release of the October 2010 DriverPacks, the number of downloads has grown to an amazing rate (20 GB/hour was not uncommon, with >3 TB or >1 million requests in the past week alone). This means that we're now burning rapidly through our remaining credit. This means that by the evening of October 5, GMT at the latest, downloads are likely to stop working (unless SimpleCDN allows us to go negative, which is undocumented).

So, we're now working rapidly on an alternative. We're talking to a possible alternative CDN partner, but chances of success are low there.
The more likely outcome is that we'll launch our own Bittorrent tracker. This implies that the auto-update feature of the DriverPacks BASE will stop working. It will also mean that you can no longer download the DriverPacks with just a browser: you'll have to use a Bittorrent client, such as uTorrent or Transmission.
This will free us from finding a CDN provider that sponsors all bandwidth, and from the alternative: paying for a CDN, which would be absolutely impossible to pay from just the donations that we're getting right now. We'd need several (many) hundreds of dollars of donations every month to be able to pay that, and that's not realistic.

We will keep you posted!

Thanks for your understanding.

UPDATE: We've got the situation completely under control again. We did find a new CDN partner after all. Details will follow soon.

Last edited by Wim Leers (2010-10-07 23:57:10)

Founder of DriverPacks.net — wimleers.com

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Unfortunately I think that this news will only exacerbate the problem and people will try downloading the DriverPacks because they may not be available in the near future.  Sort of like "panic buying" during impending natural disasters.  This will only cause our rapidly-dwindling credit to evaporate even faster.
The lack of site donations may or may not have been a contributing factor.
Even with using BitTorrent as our primary distribution method (which I like a lot), we'd still need a primary "seed" server to upload DriverPacks to.  That server would get hammered on release-days.

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Even with torrents the same issue arises... many will leach and few will seed and we still need to pay for a primary server. The torrent option is just a variation on the same two primary options we have always depended on, sponsors or donors. Since those two seem to be non-options, sadly that only leaves the other choice we have been trying to dodge for years... charging for our service(s) in some fashion. The reality of it is the fee can be ridiculously small. since we have a million downloads a week and ten packs that represents 100,000 users a week. The fee could be a dollar a year per person (heck, it could be a measly ten cents)... It's almost a joke that we should have to make donating a dollar mandatory but obviously the honor system won't ever work. It's truly a very sad day for DriverPacks.

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

FWIW I quoted the op over @ UBCD4Win's forums. Just trying to raise awareness, maybe it will help.

SteelTrepid wrote:

Crap, I'm sorry to hear this.  I wish there was something that I could do but I am normally in about the same boat as you guys.

I'll see if any of my few contacts are worth anything.

Thread is here.

http://d1syubgj0w3cyv.cloudfront.net/cdn/farfuture/qRi-6bSxUY_t4ZJ4TTzZb_mfY2Kr8TTYx9fCIGsvH1s/perpetual:forever/userbar/donator-2.png

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

OverFlow wrote:

... charging for our service(s) in some fashion. The reality of it is the fee can be ridiculously small. since we have a million downloads a week and ten packs that represents 100,000 users a week. The fee could be a dollar a year per person (heck, it could be a measly ten cents)... It's almost a joke that we should have to make donating a dollar mandatory but obviously the honor system won't ever work. It's truly a very sad day for DriverPacks.

I have to agree with the idea and if it's put to a vote the general populace would mostly agree, whether they would pay is a different story.
Pay to Use analysis says that only 1 in 5 will pay and if it's only $1 most would say it's not worth sending money or to use Paypal etc.
If the price was $5 per year then the 1 in 5 scenario works as an average. Small price to pay for the ease of use they gain.
It seems people are more willing to spend $5 than $1.

Most of the people using DP are techies or people who reload a lot and I'm sure they would realize that they have to pay the piper at some point or it all comes to a screeching halt.

Another option is Rapidshare as we have been using for the nightlies but they would probably complain after a couple of weeks.

Just my 2 cents worth or is that $1.

Stu

6

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

There's also eDonkey / Kad networks that does not require the server and is not shaped by ISPs like Bittorent

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

What additional costs would be accrued at CDN if this bandwidth whirlpool continues?

If they've provided decent capacity and performance (for the most part?), perhaps we should entertain staying with them.

Torrenting is a wonderful distributed technology, but the BASE would need changing in a very significant way. Not that Jeff isn't up to it, and as it's been said, the DP userbase IS a very technical bunch. You have all the talent you need just waiting to be called upon. My own thirty-five years have taught me many things, and I stand ready to do whatever I can to help y'all see this through.

My idea: stay with CDN, solicit dues/fees/payments/donations (whatever you call it) to cover the overages, and slipstream bittorrent protocol as a secondary download mechanism into the BASE product.

If bandwidth utilization is indeed this high, it is incumbent for technical professionals to deploy a better mousetrap. Bittorrent capitalizes on bandwidth, and more really IS merrier. Let's turn this data dragon on its head and start worrying about something else, shall we?

As a longtime user, I am IN on the donation part. You tell me how much ('cuz I know y'all are a fair bunch) and I'll send it. It really is time to monetize this (not so) little venture. It's field tested and battle proven. Now all we need is the fiscal support to take it to the next level.

NT5 has a LOT of life left in it, no matter what the progressives may say. You've built a great technology that deserves to continue growing and thriving. HEY! Maybe a few of you can even get a long overdue payment for all the valuable time and effort you've invested here these last few years.

OK, people. Let's DO this thing.

LET'S ROLL.


Mark Strelecki
DriverPack Believer and Supporter

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Thanks Mark,

To pay out of pocket for the CDN at our current usage levels... http://simplecdn.com/pricing

we had
Yesterdays Data Trans: 498.72 GB
Last Week  Data Trans: 3.36 TB
Last Week Total Requests: 1,059,509

and with our current setup with all the old packs available on the server about 16gig of storage.

according to the the price chart 500~1000 dollars more a month would be needed...
The donations we get now don't even cover Wim for his out of pocket that he pays for this site/forum.

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

I think you should distribute the DriverPacks via torrents in any case even if it's going to be just an alternative distribution method.

You don't have to set up your own bittorrent tracker as you can find trackers which don't allow any pirated content. You don't have to have a seeding server because popular torrents have a lot of seeders and this torrent is going to be popular. It doesn't require any maintenance and costs nothing. If you want to have a lot of seeders after the initial seeding, you can limit your upload speed forcing users to seed longer and prevent them from leaving early. But I think it isn't necessary.

Actually driverpacks have been available on several torrent trackers for years. If I take a look at just one of them, I can see that it uploads 337 gigabytes per day. This number is based on the current commutative upload speed(4 megabytes per second) and this speed will increase even more when people get home from work. There are so many seeders that a seeding server would be seeding close to nothing.


OverFlow wrote:

since we have a million downloads a week and ten packs that represents 100,000 users a week.

OverFlow wrote:

Last Week  Data Trans: 3.36 TB

By the way, these numbers don't add up.

That would mean that the average download size is 3.5MB (3.36*1024*1024/1,000,000) but driverpacks are bigger than that.

Let's suppose that users download only the XP driverpacks(753,65 MB).
That makes it about 4,679 users a week (3.36*1024*1024/753).

Last edited by BadPointer (2010-10-07 00:56:06)

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

We are still devising a plan, but it's possible that we may move to a split distribution where the torrents would be free, but the CDN would be available to paid users.  This would help subsidize our costs and give users a tiered service.  It's just an idea.

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

I can only repeat what mr_smartepants said.

@Badpointer: you're absolutely right, the numbers don't add up, at least not the way you would initially think. Those numbers originate from the SimpleCDN control panel. We didn't generate them.
The thing is that people use download accelerators. Download accelerators accelerate downloads by requesting multiple chunks of the same file simultaneously. E.g. the first 20% of a file is a chunk, the second 20% is the second chunk, etc., and all these chunks are downloaded simultaneously. Then SimpleCDN would register 5 requests.
See, SimpleCDN merely registers requests. But those requests may be only partial file downloads.

Founder of DriverPacks.net — wimleers.com

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

I like mr_smartepants' idea.

I'm aware of download accelerators. My point was that the "100,000 users a week" estimate is not correct. So I suggested calculating this number the other way.

Last edited by BadPointer (2010-10-08 00:40:03)

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Great news!

mr_smartepants wrote:

We are still devising a plan, but it's possible that we may move to a split distribution where the torrents would be free, but the CDN would be available to paid users.  This would help subsidize our costs and give users a tiered service.  It's just an idea.

Good idea!

Thank you!

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

mr_smartepants wrote:

We are still devising a plan, but it's possible that we may move to a split distribution where the torrents would be free, but the CDN would be available to paid users.  This would help subsidize our costs and give users a tiered service.  It's just an idea.

What do you propose as the paid CDN price range ???

My checkbook is ready.

Stu

Last edited by BigBrit (2010-10-10 07:47:00)

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

<begin opinion>
Our current file hosting situation cannot continue as-is.
In the past few days we had 590,000 web page hits, 255 GB of download traffic on Oct 7, 305 GB of download traffic on Oct 8, 154 GB of download traffic so far today.  Our peak was on release-day...over 500 GB of download traffic in one day (half a TERABYTE).  That amount of traffic is not sustainable without significant costs.
The new CDN server is getting hammered.  Everyone needs to think long and hard about which side of the fence they sit on.  All I can say is to be prepared for tiered service.  I think that it's the only way DriverPacks will survive into the future.  Our donation bucket is not overflowing and Wim's pockets are not bottomless!

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

If those bandwidth numbers are accurate (and I'm certain they are), then torrenting technology becomes not just an option, but an eventuality.

And MUCH sooner, rather than later, it would seem.

TERABYTES per week sounds like a job for Super Torrents.....<grin>

That amount of data transfer is remarkable, if not incredible. I had no idea of the volume. WOW!

Can we agree that collective, distributed bandwidth beats paying for thousands of duplicate streams from a for-pay content distributor?

DriverPacks could be their OWN cdn. Maybe even do it for OTHER projects. (Hey, I'm just throwing stuff out here.)

Torrents are another field tested, battle proven technology and would be a HUGE RELIEF for this project. They have the power to completely blunt the need for paying a cdn to perform mindless repetitions (downloads) at no small cost, apparently. (Bless Wim and the contributors for getting us all this far.)

Contributions could then be accepted in currency as well as peer and seeding bandwidth, and we users have a LOT of that, don't we?

I'll bet the contributors to this forum ALONE could create all the seeding bandwidth you'd need to do this job.

Well, that's just my opinion, and I could be wrong.....



MARK STRELECKI
Atlanta, GA.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

This is all good stuff... The main idea that keeps repeating is apparently tiered service.

Let's keep the ideas flowing

Free Torrents and paid CDN seems like a choice that would be agreeable to all.

Torrents would be a manual process requiring a third party app... that would certainly encourage people to pay a nuisance fee to get the downloads directly and automatically from DriverPacks BASE. Simple and effective, and still allows our app to be free to anyone.

Further the packs can be automatically verified with MD5 hash checks against the server for all packs found in the DriverPacks BASE folder. This would prevent any issues of corruption from the Torrented downloads. Seems easy to do and effective for both paid and non paid users.

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Possibly the paid service would have the username/password of the DriverPacks forums linked in the app to prevent pirating.  IE, the app would be a free download for all but to enable the 'Pro' features you'd have to use your username/password from the forums and that would be verified online (with IP comparison to prevent duplication from different IP domains.)

Read BEFORE you post.  HWID tool   DriverPacks Tutorial   DONATE!
http://driverpacks.net/userbar/admin-1.png
Not all heroes wear capes, some wear Kevlar!

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

I agree... however it would need to be a bit more adaptive than that... most people do not have a static IP. So the IP would have to be reverse DNS lookup to verify ISP and Region. I'm sure Wim could put a server side module in place to prevent cheating. It would be a simple matter to bust an account holder who was downloading from the USA and Russia within the same hour and cancel their login / account big_smile.

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

I don't think you have to worry too much about cheating as it might not be so widespread because torrents are faster and make it easier to choose driverpacks for download.

Besides, the speed of CDN isn't really impressive and DriverPacks BASE updates only 10 XP driverpacks. Users have to download other driverpacks from the website one by one which isn't very convenient. If you're going to require users to enter login/pass for every driverpack, it will be even less convenient. Ideally, DriverPacks BASE should be able update all driverpacks.

I agree that you should charge for the access to CDN.

Last edited by BadPointer (2010-10-11 08:06:24)

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

The speed of the CDN was always impressive in our experience? I am aware of no reports of slow downloads. I am aware of issues with incomplete downloads. If you were having slow downloads I wish that you had reported it. I have never been able to achieve the speed on any torrent (ever) that I get from the CDN.

In our (the teams) experience the CDN would max out any download bandwith. Those of us with the the high bandwidth connections have always gotten our full 30 Mbs download speeds. I don't know what you think is impressive but to me maxing out a 30 Megabit a second connection is pretty impressive. Over 4 megabytes a second is faster than a cheap IDE HDD. I'm sorry that I can't agree with you, but in my book that is impressive. Again, if you were having speed issues I wish that you had reported it. Perhaps our old provider had no server clusters in your country OR perhaps it was an issue that could have been addressed... if it were reported.

I think the idea would be that BASE would authenticate for the user. The username and password would only have to be entered once into the GUI. I would probably encrypt the login info on the local machine in such a way that it would always be available / cached on that machine. It would also be simple enough to have a cached login for the site downloads too, since that feature already exists for the main team members.

The 3rd party DriverPack will be available from the GUI in the future... However it is not likely that it will ever be a feature of DriverPacks BASE, which was not designed in such a way as to easily allow new main packs to be added. It will VERY likely be a feature of the new app we will develop for NT6 platforms.

DP BartPE Tutorial   DP_BASE Tutorial   HWID's Tool     Read BEFORE you post    UserBars!
http://driverpacks.net/userbar/admin-1.png
The DriverPacks, the DP_Base program, and Support Forum are FREE!.

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Regrading the speed. I don't download driverpacks that often so I don't know how the speed was changing over time. I remember downloading at my full speed of 6 Mbs about half a year ago. I tried downloading yesterday and the speed was about 3 Mbs. I'm going to measure it again when I get home.

Update:
The average download speed is 4 Mbs out of my max 6 Mbs.

Last edited by BadPointer (2010-10-12 02:09:04)

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Possibly the paid service would have the username/password of the DriverPacks forums linked in the app to prevent pirating.  IE, the app would be a free download for all but to enable the 'Pro' features you'd have to use your username/password from the forums and that would be verified online (with IP comparison to prevent duplication from different IP domains.

Spam link removed....

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

hello,im new member in this forum.

SPAM link removed by moderator

Re: SimpleCDN unresponsive: downloads may be compromised [UPDATE]

Torrenting is a wonderful distributed technology, but the BASE would need changing in a very significant way. Not that Jeff isn't up to it, and as it's been said, the DP userbase IS a very technical bunch. You have all the talent you need just waiting to be called upon. My own thirty-five years have taught me many things, and I stand ready to do whatever I can to help y'all see this through.

SPAM link removed by moderator