HWBOT Server hack attempt & move to Amazon Cloudservices

Yesterday at approx 23h CET we noticed a hacker trying to take control of both our production and test servers. Luckily we were able to fend them off, but after applying security patches the production server refused respond, and did not come back alive after a reboot.

16 hours later and four trips to the data center later we called it quits and gave up on repairing the production server, and started the move to the Amazon Cloud. Something we were planning to do later this year anyway. Members with a keen eye might already noticed we started to use the Amazon Content Delivery Network a few weeks ago to host the images, hosting our servers 'in the cloud' would have been the next logical step.

As this move was rushed, some functionality is still in progress of being restored: hundreds gigabytes of screenshots are being uploaded (at a slow 1mbit/sec), the forums need to be reinstalled, the news engine, and I'm sure bugs will popup from running on a virtual server instead of a dedicated ubuntu linux machine. Fingers crossed, and our apologies for the downtime!


53

Belgium richba5tard says:

The good news is the hackers did not touch our data, and no submissions or screenshots were lost. It seems they were trying to convert the HWBOT servers into dumb bots, to scan a vast array of ip's for weaknesses.

I'll continue to restore all features, but as I've had no sleep since yesterday morning I'll need to take a break soon.

United States xxbassplayerxx says:

Damned hackers!

*shakes stick*

I needed my daily HWBot fix!

United States I.M.O.G. says:

Thanks for the update. Good luck getting everything going again - what a pain! Cloud is a good move though, especially if its using a CDN to serve pages with how distributed your visitor base is across the world.

This is probably lower priority compared to having a site that runs... But you may also want to take a peak at the points algorithm... GTPP are not what they used to be. Overclockers.com used to have about 2485 GTPP (our GTPP had shrunk since earlier this month: http://webcache.googleusercontent.com/search?q=cache:OrkQzCkAhrkJ:hwbot.org/team/overclockers.com/+&cd=1&hl=en&ct=clnk&gl=us), but with the server change we lost about 300 GTPP. Madshrimps and xtremelabs lost many more GTPP than us even...

With some looking into, you should find one of two things:

1. The current totals for team points aren't accurate
2. The previous totals for team points weren't accurate, and they are correct currently

Belgium richba5tard says:

Thanks IMOG, I'll need to investigate once things calm down. The forum move has been completed too. Next is the images.

Germany Moose83 says:

My Team freeocen also lost points. Myself lost 200 Power points!

United States Killbuzzjrad says:

Well I'm going to give it some time since it's still buggy but my team and myself have lost a lot of points. :( I'm sure it will be fixed in the next few days.

United States Bobnova says:

Thanks for the hard work resurrecting HWBot, it's appreciated.

Belgium richba5tard says:

Thanks bobnova. : ) Wordpress has been reinstalled too, news is showing up properly now too. There were some issues with the Amazon RDS ('in the cloud' mysql instance), causing an exhausted connection pool, but this has been resolved *knocks on wood* The last major thing I need to do is upload a few hundred gigabytes of screenshots, it's going to take days using my 1mbit upload cap.

United States I.M.O.G. says:

Overnight me a hard drive, I'll get 'em upped for you RB. :D Nice work on stuff so far.

Also, I'd like to be the first to thank the hackers... In the past 5 minutes of browsing HWBot, I've never seen the site respond so smoothly! :D

EDIT: This score is an example of a GTPP which is no longer receiving GTPP:
http://hwbot.org/submission/2294253_i.m.o.g._aquamark_radeon_hd_4890_443866_marks

When trying to recalculate in order to get GTPP assigned to it, this is the error it throws back (FUNCTION madmark.teamGlobalPowerPoints does not exist):

[CODE]Failed to synchronize rankings: Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: FUNCTION madmark.teamGlobalPowerPoints does not exist Error Code: 1305 Call: replace delayed into rank_entry (rank_id, rank, result_id, participant_id, points) select ?, @rank:=@rank+1 rank, result_id, team_id, teamGlobalPowerPoints(?, @rank) points from ( select result_id, team_id from ( select * from ( select result_id, score, user_id, team_id, ranking, date from result join application app on (result.application_id = app.application_id) where app.application_id = ? and (app.multi_threaded='N' or effective_cores = ?) and best_hardware_submission = 'Y' and status_id < 10 and team_id != 0 order by score desc ) as sorted group by team_id order by score desc, date asc ) as by_team ) as sorted_by_team bind => [4 parameters bound] Query: DataModifyQuery(sql=" replace delayed into rank_entry (rank_id, rank, result_id, participant_id, points) select ?, @rank:=@rank+1 rank, result_id, team_id, teamGlobalPowerPoints(?, @rank) points from ( select result_id, team_id from ( select * from ( select result_id, score, user_id, team_id, ranking, date from result join application app on (result.application_id = app.application_id) where app.application_id = ? and (app.multi_threaded='N' or effective_cores = ?) and best_hardware_submission = 'Y' and status_id < 10 and team_id != 0 order by score desc ) as sorted group by team_id order by score desc, date asc ) as by_team ) as sorted_by_team ")[/CODE]

United States sin0822 says:

ye the site is much faster now. Good luck with your move.

United States hokiealumnus says:

Sucks about the hackers, but I second the thanks for the early move; it's moving so much faster now! Sorry you're having to deal with it on someone else's schedule instead of your own though. Hope you get some sleep. :)

Belgium richba5tard says:

Yeah i agree, moving to the amazon cloud makes it faster for everyone. The amazon database issue is not resolved yet though, still working on this!

United States l0ud_sil3nc3 says:

The site is probably the fastest now I have ever seen it, awesome work richba5tard and team.

United States hokiealumnus says:

Yea, the DB is very angry with the front page just now. Whatever though, take your time. We'll still be here benching; ready to submit whenever you can get it all sorted. :)

Belgium richba5tard says:

Glad to hear it's much faster in the US now, even though the new data center is in Ireland!

The database issues have been solved. Finally I can get some sleep!! :D

India sumonpathak says:

Awesome work rich... the team points are off...but i guess you guys will have it sorted...Good luck :)

Germany der8auer says:

Thanks for the nice and fast work :) realy appreciated!

United States Tsunamijuan says:

As much of a pain in the ass as this has been for you guys, I think the silver lining of improved availability and performance for those on the other side of the pond from the EU. Is vastly better. We appreciate the work you guys put in.

Denmark M.Beier says:

Its faster, good you guys got rid of them intruders with ease!

United States I.M.O.G. says:

GTPP are also slowly coming back to the teams, so maybe that bug is working itself out.

Australia Dinos22 says:

Thanks for the effort Rich, hopefully no more hacker issues from now on

Wish we could convert our ln2 computation power to kick that hackers arse :D

Norway knopflerbruce says:

Too bad the team league is down so we can't check the impact of the missing GTPP:p

Germany Moose83 says:

TPP and GTPP are back in my Team. Also the site is faster than ever! Very good work!!! But Login is not working for me.

Australia Jimba says:

Find the hacker..and hack him back..:P

good to see its all sorted now though.

also just idea but because you have to redo everything (for the move) wouldn't it also be a good idea to add features/functions that people want or where in the works?

Iran Amateurs says:

Did he/she change any frequency of server ? Heheheeee :D
I can not submit my aquamark file !!! Hwbot can get the file but won't submit !!!
Now that you are changing the site, what is your opinion about IPB for forum instead of VB. Is it possible?

Belgium richba5tard says:

Hi Poorya_lion, I actually like VB more than IPB. Matter of taste?

Everything seems to be ok now, except that resizing some images during submission take 100% CPU load for a very, very long time. I can not reproduce it, so I think it only occurs for some kind of images. If someone has trouble making a submission with images attached, please forward those problematic images to me!

United Kingdom borandi says:

Nice one for the info :)
If you're interested, mobile site is fubared as well. Secondary priority though.

Belgium richba5tard says:

Got some more bugs squashed, running HWBOT on a virtual server in "the cloud" is a bigger difference dan dedicated hosting than I hoped. Site responds swiftly though! When it works. :D

India sumonpathak says:

^^any chance of updating to vb 4.2 :D

Sweden Calathea says:

The site is super fast now. I love it.

Iran Amateurs says:

Thanks Frederik,
Actually I'm not insisting for IPB, It is OK to be in VB, just want to know your opinions.
I submit it right now, there is no problem with site and OH MY GOD , It is so fast , I love it.

Belgium leeghoofd says:

Who let Bill get anywhere near a PC ?

United States I.M.O.G. says:

I also prefer vb to ipb. On the backend and with moderation it is just way better.

sumonpathak said: ^^any chance of updating to vb 4.2 :D


Vb5 is nearing, probably doesn't make sense to go to 4.2 now.

Mobile site has been working fine here.

Norway knopflerbruce says:

I see some pics wont show any more, saying something like "the requested content could not be found. Please try again later". This is something that will be fixed, no? :p

Belgium richba5tard says:

Yes, no pics have been lost but I need to upload ±200 GB using a 1 mbit upload line. At full speed, that would take me 19 days. If I'm not banned by my ISP by then. :) I'm figuring out a better way to solve this issue.

Canada Vinster says:

It's probably a 30yr old fat guy still living in his mom's basement too... seriously needs to get a life... thanks for the continued hard work... and it is quite a bit faster for me as well.

Singapore rayhow says:

Thanks for the hard work and Good Luck!!!!!!

Belgium FunSkilZZ says:

stupid hackers !!!!

United States I.M.O.G. says:

richba5tard said: Yes, no pics have been lost but I need to upload ±200 GB using a 1 mbit upload line. At full speed, that would take me 19 days. If I'm not banned by my ISP by then. :) I'm figuring out a better way to solve this issue.
I'm sure someone would happily up the files for you? I could do 5Mbps up, and I have a trustworthy friend that does 60Mbps up.

United States Tsunamijuan says:

My trick for big files is to do it from a school. Like college or High school. Usually if you ask nicely they are happy to help out. Sometimes they ask for a small donation or something. But usually you can just do what you need from a library or make an arrangement with the tech staff.

Australia Dinos22 says:

richba5tard said: Hi Poorya_lion, I actually like VB more than IPB. Matter of taste?

Everything seems to be ok now, except that resizing some images during submission take 100% CPU load for a very, very long time. I can not reproduce it, so I think it only occurs for some kind of images. If someone has trouble making a submission with images attached, please forward those problematic images to me!


VB is the best, definitely dont change it

Belgium Massman says:

Thanks for the big big effort, Frederik! Awesome work!

India sumonpathak says:

I.M.O.G. said:
Vb5 is nearing, probably doesn't make sense to go to 4.2 now.


Gimme link to download :D

Belgium richba5tard says:

Image backup restore 90% complete!

United States sin0822 says:

it looks good man.

United States Mr.Scott says:

Server is slow as hell and I can't submit for a challange. I get the OOPS screen and some BS about your coding skills. It's been like that since around noon here.

Bulgaria I.nfraR.ed says:

Yes, I got this:

[CODE]org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. No space left on device[/CODE]

Belgium Massman says:

When did you get it?

Germany Moose83 says:

Same here, i get same error and cant submit and have no acces to mainpage.

Bulgaria I.nfraR.ed says:

Massman said: When did you get it?


It works now.
I was trying to upload a score and was failing on the screenshot, it seems.

Belgium richba5tard says:

"No space left on device" => Amazon disk was full. Only 8GB so I need to take care that the disks don't go full.

Belgium richba5tard says:

The error said "no space left on device": the server failed to store the attached image because it ran out of diskspace. The new amazon servers are faster but have less disk space available.

Bulgaria I.nfraR.ed says:

Yeah, figured that out. You're doing great job and the site is very fast now. Rev4 have never been as fast as it is at the moment.

Please log in or register to comment.