The Aftermath
Mar 21 2008, 06:37 PMFor those who don't know, we have been under DDoS, meaning distributed denial of service for the past 2 days. Finding a solution to block this attack on the server has been long and drawn out.
We believe at this point, the attack can be held off. Finally, all network settings are in place. The fact that you can read this message is enough to say that it's working now.
We really apologize for all of this. In the next post I will put a series of questions that you can read about this entire situation and what's gone on for the past 2 days. I also know that not everything is working properly, due all of these network changes. I'll fix them as I can.
Thank you all for sticking with us through these hard times.
-jcink
EDIT: If your forum is getting a SQL Error, this is because of the DDoS attacks and Jcink will fix it as soon as he can. Usually refreshing works but there are some forums where it wont. Please do NOT PM Jcink or I about this issue as I will just point you to this topic 
-Skullmonkey
Mar 21 2008, 06:38 PM
What happened?
At 12:45 AM, Easten time, Thursday, March 20 the server went under a DDoS attack. What is a DDoS attack? It means distributed denial of service attack. What this means in basic is that someone on the internet is using lots of other peoples computers to send bogus requests to the server. This bogs down the bandwidth, and floods all connections with useless garbage so nothing valid can get inside. Hence why the site and all services have been "down."
First Response
My first response to the attack was this: change the server IP address. Seemed simple enough and it worked last time. This time however, it did not work. Why? Someone was directly attacking the domain name, jcink.com, particularly the b1.jcink.com subdomains.
In an effort to find out what domain was being attacked, I scrambled, and only added IP records for two domains, my main one jcink.com and forum.jcink.com, as well as ONLY support.b1.jcink.com. Whenever I pointed b1.jcink.com, the attack hit and hit hard.
After working on this until 5am on Thursday, I left the support ffiorums online and posted a message stating that I'd try to get this solved at 11AM. I thought the attacker may have stopped by 11AM but it did not.
At that point, I called my ISP to see if they could do anything. Unfortunately they told me there was little they could do at their end and simply brought me through basic procedures of getting my connection online. No help there.
Now I began looking for DDoS solutions and eventually I remembered one from the past. It allowed me to install a hardware based firewall to guard the server. I spent from around 3PM to 6PM on Thursday installing and configuring this system, as the attack continued.
Around 6:30 PM when I had everything set up, the attack seemed to have stopped, or at least died down a lot. In any case we were up and running again and it appeared the firewall was doing great and the attack had died off.
9PM EST hits and I'm still not sure exactly what went on. I believe the attack started again and got worse, so we got taken down. It couldn't be held off; it was passing all connections through our regular router and hardware router. I determined this double pass was slowing everything down and preventing good blocking. The router HAD to be moved aside and put on a separate line, but I didn't have the hardware to do so.
I left the system plugged in with no router 1AM on friday and went to bed, and the site remained online all night. Around 2PM I bought the piece of hardware I needed and began chipping away at the hardware firewall to allow it to connect. I thought it would only take around 15 minutes to complete but it lasted much longer than that. I had a lot of complications getting it to work, and that's what I've spent all the time up until now on.
What's happening now?
We're still being attacked, but we're online. It rages on, but the health of our connection is excellent. At the moment, the firewall is blocking it all off, which is why you can even read this message.
Do you know who did it, who is doing it, and why?
No we do not know this information for sure. I've gotten several emails of theories mailed to.me, which I thank you all for sending, but no proof of any of it. I also do not know exactly which site(s) are being hit.
Will there be any more downtime?
There could be. There's still some configuration that the firewall might need, and there's always the chance the attack will become stronger. I'm confident at this point though we're pretty much alright for now. But no promises. Just remember, they are still attacking but we're blocking it. [ edit: march 26th: they have stopped ]
What is the state of the site and services?
I realize there are some bugs going on now because of this. The network changes have made a few issues internally and externally. Please post in support section if you're experiencing issues, and I'll try to fix everything up. I am already aware of some areas in need of fixing and I'll be working on those over the next day or so.
Is there anything you can do?
At the moment, no. You guys have all been great and I couldn't thank you enough for being patient with me during these hard times.
The only thing you need to do is; if you have your own domain name please update the IP address to 67.81.196.235 if you haven't already, OR (recommended) set a CNAME on b1.jcink.com so you'll never have to update it again.
Once again. thanks to everyone for your patience. I couldn't have gotten as far as I did without it, and we really, really apologize for all of this.
Mar 21 2008, 06:38 PM
No problem. You did a great job getting it all back.
Mar 21 2008, 06:39 PM
ur welcome and thank you so much
Mar 21 2008, 06:39 PM
Hopefully its over.
Mar 21 2008, 06:40 PM
As long as everything is safe again I'm happy.
Keep it up Jcink.
Mar 21 2008, 06:40 PM
QUOTE (chipper @ March 21, 2008 05:38 pm) No problem. You did a great job getting it all back.
just like he said. good work jcink
Mar 21 2008, 06:41 PM
I know it aint over its been on and off and will be on and off I cant get on Sagas so im depressed
Mar 21 2008, 06:42 PM
Aye, superb J
Any chance you can fix the errors on the hosted sites soon? Mine are all still down even though though domains are all updated
Mar 21 2008, 06:42 PM
Thank you Jcink, for your hard work the past 2 days
Mar 21 2008, 06:45 PM
I Hope this is over , this kill my league activity so hard
Mar 21 2008, 06:46 PM
wat league are you talking about
Mar 21 2008, 06:48 PM
Amazing work Jcink. Thanks for getting everything back up so quickly.
Mar 21 2008, 06:49 PM
I wonder who it was. And why.
Mar 21 2008, 06:50 PM
QUOTE (Dagger13 @ March 21, 2008 05:46 pm) wat league are you talking about
Either NDSL (which he's in), or USDL or something 4 letters that starts with U that he's running.
Mar 21 2008, 06:52 PM
wats NDSL stand for and USDL
Mar 21 2008, 06:53 PM
GBSL , My Basketball Player League
Global Ballerz Sim League
Mar 21 2008, 06:54 PM
Jcink,
Is this attack finally over? Or will the servers be down again?
Mar 21 2008, 06:55 PM
so wat do you do and let me get a link please
Mar 21 2008, 07:07 PM
nah i cant advertise here
Mar 21 2008, 07:07 PM
QUOTE (Rush)yay
Mar 21 2008, 07:08 PM
QUOTE (Arroyo30 @ March 21, 2008 06:07 pm) http://gbsl.b1.jcink.com/index.php?
you just fell for his trap, he made you break a rule.
Mar 21 2008, 07:09 PM
oh fk i dont know that
Mar 21 2008, 07:09 PM
Quick, edit it Arroyo!
Mar 21 2008, 07:09 PM
sorry
done lol , If you wanna the link aim me at arroyobsn
Mar 21 2008, 07:12 PM
First post updated.
Mar 21 2008, 07:13 PM
lol
Mar 21 2008, 07:16 PM
4 PM's about it so far >_>
Mar 21 2008, 07:19 PM
The second post has been updated and should answer a lot of questions for everyone. If there are any questions, however, that I missed, please post here.
Mar 21 2008, 07:26 PM
I'll repeat myself for about the third or fourth time, superb as ever Jcink, you never fail to amaze me with your commitment and desire to keep Jcink going to provide free board/site hosting for the masses and also to stop the bastards who keep trying to do this. I know I've mentioned this to you a few times, but when I'm not broke, a donation is winging its way to you. Promise
Oh, and I have errors on all my sites, but I can't see the free hosting forum anymore (when did it go?) so I'll have to post in here I guess. If you could get those fixed when you can that'd be great.
Thanks once again
Mar 21 2008, 07:27 PM
Will you ever know the exact identity of the persons doing this, or exactly what forum they were attacking?
Mar 21 2008, 07:31 PM
QUOTE (Incorporated @ March 21, 2008 06:27 pm) Will you ever know the exact identity of the persons doing this, or exactly what forum they were attacking?
I might -- if someone steps forward and tells me so, or if I can gain more data from the firewall.
Mar 21 2008, 07:35 PM
Yay, sites are back up, awesome, cheers Jcink
Mar 21 2008, 07:36 PM
If these attacks continue and you encounter more downtime, would you take down your JFB service(since it seems to be the source of your troubles)?
Mar 21 2008, 07:47 PM
No. I would continue to try to stop the attacks.
Mar 21 2008, 07:52 PM
Jcink is the best. Ever.
Mar 21 2008, 08:15 PM
Whoever you are: I am aware you've just stopped the attack hardware firewall cpu stopped, I suggest not trying again or you could wind up getting in "trouble"
Mar 21 2008, 08:16 PM
thank you very much for getting everything back inorder
Mar 21 2008, 08:24 PM
Bumping to the top.
Mar 21 2008, 08:36 PM
My members said my site is slow LOL
Mar 21 2008, 08:45 PM
Hmm, it could be the firewall but it's pretty fast for me everywhere else. I notice the amount of requests on the server is coming in at normal rate. but, If anyone else if having speed issues please let me know.
Mar 21 2008, 08:51 PM
I have found that some of the Skins imported are slow...the BGS Skin has been moving at snail speed. I also know the Aura skin is doing the same....otherwise everything else seems to be normal.
Mar 21 2008, 08:52 PM
Unfortunately that's because resourceempire.com is down so the hosted content on those skins is messed up. I will hopefully be able to fix the domain IP address when taylor W comes back.
Mar 21 2008, 08:53 PM
I figured that was most likely the problem...I have just been telling members to switch to skins that I have made and the images hosted with photobucket until everything gets back to normal. Keep up the good work buddy. Everyone is behind you.
Mar 21 2008, 08:58 PM
Forums look good on my half. Great job, Jcink.
Oh, one thought, it was kinda harrowing when it happened earlier, so is there any chance that like you could create a page when something like this happens and have it redirect us to it saying what's going on or something? I don't know, something. Just wondering.
Mar 21 2008, 09:06 PM
I think I am going to create a section off site called status.jcink.com on someone elses server so you guys can get information. I would have given information if I could but it was really hard considering I technically had no hosting to do so from...
Mar 21 2008, 09:09 PM
I think that would be a good idea....will help let everyone know the status of everything. If I had hosting for you to use you would be more the welcome to use it...but I don't have....which is the reason that I use this great service as it is. Maybe get like a Freewebs site or something....some sort of free hosting site....something that you can just use as a back up to posting information and such....if something like this was to happen again.
Mar 21 2008, 09:11 PM
QUOTE (Jcink @ March 21, 2008 09:06 pm) I think I am going to create a section off site called status.jcink.com on someone elses server so you guys can get information. I would have given information if I could but it was really hard considering I technically had no hosting to do so from...
Great Idea!
Mar 21 2008, 09:19 PM
Jcink how is the firewall blocking do you expect another downtime?
Mar 21 2008, 09:21 PM
It's doing fine. At the moment, from looking at the logs it seems the attack has died down or has stopped completely.
Regarding more downtime, there could be. There's still some configuration that the firewall might need, and there's always the chance the attack will become stronger. I'm confident at this point though we're pretty much alright for now.
Mar 21 2008, 09:24 PM
Sweet! a good idea!
Mar 21 2008, 09:40 PM
lol chipper thats one in a million , by the way this is the_bommar from blacktop kings)
Mar 22 2008, 12:59 AM
Thanks for all the hard work Jcink! I hope you get some rest soon!
And I too agree on having an off site to get info! Great Idea.
Mar 22 2008, 02:25 AM
Jcink, did the information I gave you help at least a little? Or was it completely garbage and couldn't be used to your server needs? I wasn't really sure what I could do to help, but I figured that was the very least I could try to do.
Mar 22 2008, 02:28 AM
i unfortunately didnt get your email ;/
Mar 22 2008, 04:33 AM
Posted both first and second post to my forums to say why we're down alot.
Mar 22 2008, 07:09 AM
awesome job guys
Mar 22 2008, 09:30 AM
ok, my question is this- if we had purchased a domain, would we have been able to keep our boards online during this attack or would they have still been down??
Mar 22 2008, 11:31 AM
QUOTE (Kelly @ March 22, 2008 08:30 am) ok, my question is this- if we had purchased a domain, would we have been able to keep our boards online during this attack or would they have still been down??
The forums would still be offline. The domain name would function properly, but the hosting service that points to the domain was down. If Jcink's servers are down, then your domain is basically pointing to no-where.
Mar 22 2008, 12:08 PM
QUOTE (Seifer44 @ March 22, 2008 10:31 am) QUOTE (Kelly @ March 22, 2008 08:30 am) ok, my question is this- if we had purchased a domain, would we have been able to keep our boards online during this attack or would they have still been down??
The forums would still be offline. The domain name would function properly, but the hosting service that points to the domain was down. If Jcink's servers are down, then your domain is basically pointing to no-where.
ok, thanks!
Mar 22 2008, 03:47 PM
Again not sure where to post this but I thought I'd alert you in here. FTP for me is not currently working so you may want to look into that
Mar 22 2008, 03:50 PM
Should work now. Port was not open.
Mar 22 2008, 03:51 PM
Yep, perfect
Mar 23 2008, 12:30 PM
good job jcink
Mar 23 2008, 06:13 PM
Ok not sure where to put this but my domain is still not working i can get into my forum by using the jcink link but i can not get onto it through my domain
Mar 23 2008, 06:33 PM
Alrighty. Make a new topic in JFB support section and post your domain. I'll take a look at that for you.
Mar 24 2008, 10:20 PM
This is never going to end.....
No offense Jcink but you know this is going to end up happening again in like 4 or 5 months. THose people won't give up.
It's there some be all end all protection you can use?
Mar 25 2008, 01:03 AM
I've said it once, I'll say it again: an end all protection solution would cost me $xx,xxx which we do not have, and even then it's no guarantee. Ask anyone online "how do you protect against DDoS for sure" and you'll get the same answer. Big companies and organizations don't spend money on this unless they have to and if they have it.
At the moment, we've got a good effort going. I've got two servers now, one serving as purely a hardware firewall and the main server, which has a software firewall that I wrote to stop the attacks that plagued us last year (and it's done its job well, let me tell you). In any case, the hardware firewall worked really well, and it fended them off. They gave up within 3 hours of me being online with the new hardware firewall and there hasn't been a peep from them in the logs since.
It's old, from 2000, but still very relevant to today, with any DDoS situation:
QUOTE "If they can shut down Yahoo, they can shut down anybody." -- http://www.news.com/2100-1023-236621.html
The fact of the matter is, to deal with it, you need more resources than your attacker can handle. This takes time, money, and effort. And experience. Every time something like this happens I learn more about it.
Let me set something straight as well. I know every time I post about this a lot of our older members say "Oh god, here we go again" and rightfully so, but what am I really supposed to do besides handle it and try to move on the best I can?
I can say one thing, my solutions do work every time, in the end. I want to go over something here. Last year, in 2007, we saw a lot of DoS attacks and that created quite a bit of downtime which you all remember, right around this time too. I gave the label to them: "DDoS." No, what these were, were just attacks from ONE or two people, who flooded the server and crashed the CPU. THAT is DoS, with one D.
Believe it or not, a lot of servers are vulnerable to just DoS out of the box... it's easy to do since all you need is one person (yourself). I'm not going to encourage you or show you how, but just read online and watch some video on how these people do this. I tried a lot of the server tools online to stop those and nothing worked, time and time again until something worked... which it did, and I'll get to that in a minute.
There's another part of the case which made the "DoS" of last year worse. Badly coded parts of IPB were crashing the server often and I realized nearly half of the times I cried "DDoS" and "Dos" when it was just a performance problem. Lack of RAM, and simply poorly done areas of IPB like the search engine, post deletion area, and today's top 10 posters to name a few. I soon learned how to track those down and all of those parts have been re-written. How do you think I found out about this?. Because I have far better logging and detection on things like that now.
In any case, after July, when the RAM was installed and the script for the DoS was finalized: have there been any attacks? There sure have. If I were to show you my software firewall logs, well there's been around 50 attempts and that's no lie. We've snoozed right past them; the server did it's job and simply blocked those IP addresses. The server itself hadn't been cold-rebooted in over 140 days, (and when it did in November before that, it was simply to install a hard drive) something that happened very often back in those days.
The first *real* DDoS attack happened here. Last month. On February 1st. The plan there was simply to change the IP, and I privately applied some updates to the router thinking that it'd fend an attack off in the future. But I was wrong because he we are this month. And this month, I deployed my "backup plan #1" as stated in that topic which I wouldn't reveal any information about, and still won't post all that many details regarding it beyond saying that there's a new hardware firewall in place.
And guess what? It worked. I got to see it in action as it happened, tweaked it, and made it work. It was a tough two days and nights for me to get through it but I worked it out and I did get service back -- and blocked the attack. I'm already thinking ahead too about cost-effective plans for the next attack, whatever it may be, and whenever it may be.
Mar 25 2008, 07:41 AM
QUOTE (Jcink @ March 25, 2008 12:03 am)
but what am I really supposed to do besides handle it and try to move on the best I can?
Amen to that! I think you have done a wonderful job!
There will always be DDoS attacks as long as there is internet - its just what "they" do. They find your weak point and attack.. Once they realize its not beating JCink down they will move on to someone else..
Mar 25 2008, 09:16 AM
QUOTE (TJ @ March 24, 2008 09:20 pm) This is never going to end.....
No offense Jcink but you know this is going to end up happening again in like 4 or 5 months. THose people won't give up.
It's there some be all end all protection you can use?
The solution stay off the net. The Internet core protocols as they exist today are inherently prone to attack. Lucky for us the attacks so far devised are prone to filtering. Jcink has done all he can short of a very costly Co-Location. He has one of the top three hardware firewalls, in fact short of a Cisco box i don't know if there is anything better. Nobody has any right to complain!
Mar 26 2008, 01:03 PM
I don't think anyone has a right to complain about the downtime. It was for a few days, not a few weeks like it could've been. It's been fixed, it's not still happening like it could be. Jcink has put forth countless amounts of time and cash towards this entire service, along with an unmeasurable amount of dedication. With that type of dedication, of course you're going to want to keep it online!
I have the utmost respect for Jcink for being tough enough to fight these battles and not give up, along with continuing his dedication towards keeping everything online and updated. That's another key feature, the updates are always worth it and rather frequently. If anyone should complain, it's Jcink for TJ saying to just give up. Giving up means failure, who wants that?
Mar 26 2008, 03:40 PM
Wow. That maybe why my site was down before..Lol I thought I mistype the url.