DhO email Blacklisted, suspect hack, need help ASAP

thumbnail
Daniel M Ingram, modified 9 Years ago at 5/4/14 4:55 PM
Created 9 Years ago at 4/11/14 3:59 AM

DhO email Blacklisted, suspect hack, need help ASAP

Posts: 3268 Join Date: 4/20/09 Recent Posts
I started getting messages a few days ago from people who couldn't get email replies when they requested new passwords, and then, after asking Chris Stavros of Omegabit about it, I received this:


"Hi Daniel,

I had this looked at. There are a bunch of email messages stuck in the outbound queue. I'm afraid it looks like your mail server (the outbound mail engine on your portal server), has been blacklisted and many messages are not getting routed to their destination because of reputation issues. See:

http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a72.29.184.157&run=toolpage

(see the excerpt from the mail logs below, as well)

There may be others not on this list - the raw mail logs also list additional listers, but once your one one of these master lists, you have to clean it up from the top. You can follow the various listers and learn more about how to petition to be removed - it is an arduous process and sometimes you just have to wait for the reputation to clear naturally.

It is most likely that there is a mail form exposed somewhere on the site - do you have the "invite" portlet or a mail form available anywhere? Some user has probably figured out how to exploit it for sending spam via the portal. It should be removed. If none exists, this could be some sort of hack - though, we've never observed one of this nature (it is usually a form or the invite portlet). It could hypothetically be some sort of exploit of the message board engine. But, I would look for portlets that generate email in the layouts, first.

The fastest way to resolve this (after the hole is plugged by removing the portlet - assuming there is one), would be to reroute email through a different mailhost that is not blacklisted - preferably, one with advanced spam filtering and reporting capabilities.

You might start by trying to route it through whatever handles your interactivebuddha.com mail (looks like Google?) - this would be a good idea for many reasons including source reputation (it will match your domain), and that they would probably catch the spam before you got blacklisted, saving you grief.

You can set the SMTP host to Google's, with your account credentials, and it <should> allow you to relay via their mail servers. You may need to enable this for your domain. See for more info:

https://support.google.com/a/answer/2956491?hl=en

The SMTP mail setting for the portal can be adjusted here:

Control Panel -> Server Administration -> Mail -> (set the Outgoing SMTP Server setting to match your relay-authorized Google account).

If you can help us to find the means that is being exploited for mail in the portal, I can offer one alternate option: a one-time remedy, which would be to reassign your outbound mail IP address to something that is not blacklisted from our network (essentially, a new public IP for mail routing). We would be assuming some risk for that action if it were determined to be a malicious attempt to hide intentional spamming by our upstream providers. So, it is important that we are diligent and beyond reproach in terms of making every effort to address the root cause before exercising that option; it is a last resort. Routing through your legitimate domain/mail provider is the "correct" approach, from a reputation perspective.

Concerning helping your users that are having issues: My recommendation is to manually reset their password to something for them, and to email them directly, at which point they should be able to login and reset it to whatever they prefer. You may have to do that for users that have locked themselves out or requested a new account until this mail reputation issue can be sorted out.

I hope this helps - let me know if you have questions or need help on how to proceed.

-Chris"

This is just one step more tricky than I likely can pull off: anyone want to help? I tried following the instructions in the support.google.com/a/answer etc link but can't find the admin console for google. I am only intermediately technical. Anyone offer to help with this?

Thanks very much,

Daniel
J J, modified 9 Years ago at 4/11/14 6:13 AM
Created 9 Years ago at 4/11/14 6:13 AM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 225 Join Date: 3/31/14 Recent Posts
I think that was my bad because I kept putting fake emails (blahblah@blahblah.com) when I registered. And I made a lot of accounts. Let me know if you need any help as I think I've done this before.

-James
thumbnail
Chris G, modified 9 Years ago at 4/11/14 6:13 AM
Created 9 Years ago at 4/11/14 6:13 AM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 118 Join Date: 8/22/09 Recent Posts
Hi Daniel,

I'm not really an expert, but you should be able to find the admin console for google at http://admin.google.com. If that doesn't work, try http://admin.google.com/interactivebuddha.com . This is assuming that you have a Google apps account for interactivebuddha.com (do you?)

I set up a google apps for business account to look at this, and the instructions for setting up the SMTP relay that Omegabit pointed you to (https://support.google.com/a/answer/2956491?hl=en) seem accurate.

You'll need to first set up the relay on Google's end, then set up DhO to route mail through Google's servers.
thumbnail
Dada Kind, modified 9 Years ago at 4/11/14 10:19 AM
Created 9 Years ago at 4/11/14 10:19 AM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 633 Join Date: 11/15/13 Recent Posts
Just sent you an email with potentially useful information about exploits.

Does Google handle your mail?
Ivo B, modified 9 Years ago at 4/11/14 12:02 PM
Created 9 Years ago at 4/11/14 12:02 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 42 Join Date: 2/11/13 Recent Posts
There are this things called RBLs.
http://www.rblmon.com/blog/what-are-rbls-and-how-do-they-work/

Looks like you're on it. Thats bad because people use those lists to fight spam. I dont aprove of RBLs since they are anarchistic. Owner decides if you'll be put off. There are other ways to fight spam. I can help with setting up clean email server until things are cleared up on new IP. I've seen your problems in other thread but i was put off by liferay platform. From what I see you/we basically need a forum with PMs and wiki. Not much else. That can be hosted very light! Including email server. I can help if forum is easily extractable from liferay, otherwise...

You could host everything for 20 bucks plus 5 bucks for backup per month with vps provider im using. One of the best there is. Thats what i use for personal server. Same setup for company i work for. Very professional VPS people.

I can help set up DNS too. No problem.

Well, ttyl
Ivo B, modified 9 Years ago at 4/11/14 12:05 PM
Created 9 Years ago at 4/11/14 12:05 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 42 Join Date: 2/11/13 Recent Posts
James Yen:
I think that was my bad because I kept putting fake emails (blahblah@blahblah.com) when I registered. And I made a lot of accounts. Let me know if you need any help as I think I've done this before.

-James


Sorry james, but i had to laugh here emoticon
You're funny
J J, modified 9 Years ago at 4/11/14 2:55 PM
Created 9 Years ago at 4/11/14 2:55 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 225 Join Date: 3/31/14 Recent Posts
Ivo B:
James Yen:
I think that was my bad because I kept putting fake emails (blahblah@blahblah.com) when I registered. And I made a lot of accounts. Let me know if you need any help as I think I've done this before.

-James


Sorry james, but i had to laugh here emoticon
You're funny


Thanks, I try

=p
thumbnail
Daniel M Ingram, modified 9 Years ago at 4/11/14 6:11 PM
Created 9 Years ago at 4/11/14 6:11 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 3268 Join Date: 4/20/09 Recent Posts
Dear Ivo,

Current best theory is that Liferay 5.2.2 and the version of JBoss we are using are both vulnerable to attack and exploitation.

I have tried for 2 years, over $2000 and now 7 server/Liferay/tech guys to upgrade to 6.2 and all so far have failed to do this.

The last guy to work on this, Manish, vanished for a while but has reappeared. Apparently he had a death in the family. Now he says he will help with this, but I am not holding my breath, as he started a year ago.

We will see how this goes.

I am not sure if we will be on Omegabit or on hosted on OSX on a MacMini I now have sitting here on my desk for that purpose. We will see.

If we end up on Omegabit, and we end up on 6.2, then in theory our problem is solved, but if not, then I may need your solution.

When you say you could extract us from this Liferay nightmare and put us on another platform: what would you put us on and what capabilities would we lose/gain?

Thanks,

Daniel
Ivo B, modified 9 Years ago at 4/11/14 11:22 PM
Created 9 Years ago at 4/11/14 11:21 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 42 Join Date: 2/11/13 Recent Posts
This site basically runs around bulettin board or forum, so this would be main priority.
Biggest job is to convert this forum into another one. We would probably need to write
a database conversion script.
There's also information in FAQ and wiki. Minor problem. Nothing much is left afterwards.
Or is there?

I would keep this functionality and try to implement it with proven light solutions.
Something like lighttpd+(fluxBB/myBB...).

A good email server setup is also needed. Postfix + dovecot + amavis/SA/clamav setup comes
to my mind.

Anyway... This (mainly db conversion) would need to be done in peace when current problems are over.
I've seen someone is offering professional assistance( liferay thread ). Maybe that would be best first step
and then we could all work together toward long term solution.

All the best
thumbnail
Daniel M Ingram, modified 9 Years ago at 4/13/14 7:37 PM
Created 9 Years ago at 4/13/14 7:37 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 3268 Join Date: 4/20/09 Recent Posts
Dear All,

As a precautionary measure to hopefully make it harder for whatever is going to happen (meaning possibly the DhO being hijacked to send spam), certain features, specifically new account creation and resetting forgotten passwords, have been turned off. Also, the mail and invite portlets have been disabled. I hope this may stop the problem. In the meantime, my apologies. In theory this will be resolved by the end of April. In practice, don't hold your breath, and pay attention to it instead... ;)

Practice well, and enjoy what access you may have to the DhO,

Daniel Ingram
Dharma Overground Underlord
thumbnail
Travis Gene McKinstry, modified 9 Years ago at 4/15/14 1:52 PM
Created 9 Years ago at 4/15/14 1:52 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 208 Join Date: 7/26/12 Recent Posts
Daniel M. Ingram:
Dear All,

As a precautionary measure to hopefully make it harder for whatever is going to happen (meaning possibly the DhO being hijacked to send spam), certain features, specifically new account creation and resetting forgotten passwords, have been turned off. Also, the mail and invite portlets have been disabled. I hope this may stop the problem. In the meantime, my apologies. In theory this will be resolved by the end of April. In practice, don't hold your breath, and pay attention to it instead... ;)

Practice well, and enjoy what access you may have to the DhO,

Daniel Ingram
Dharma Overground Underlord


Many of us understand you are doing your best.

Thanks for the website. For what it's worth, if the site becomes compromised, I enjoyed it while it lasted emoticon optimistically, I'm sure the problem will be fixed soon.
thumbnail
Daniel M Ingram, modified 9 Years ago at 4/16/14 11:29 PM
Created 9 Years ago at 4/16/14 11:29 PM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 3268 Join Date: 4/20/09 Recent Posts
Apparently shutting those things down did stop the spam access and the 50,000, yes, 50,000 bad emails in the queue have been removed.

We await a permanent solution, but in the meantime, stabilization has occurred.
thumbnail
b man, modified 8 Years ago at 6/21/15 7:50 AM
Created 8 Years ago at 6/21/15 7:50 AM

RE: DhO email Blacklisted, suspect hack, need help ASAP

Posts: 199 Join Date: 11/25/11 Recent Posts
Hi Daniel, 

Does this mean we can turn email notifcaiton back on?

If you havent already and want to further explore options to reroute the email,  let me know, I know a good tech guy who could possibly be persuaded to assist. 


Thanks, 

Bman

Breadcrumb