DhO email Blacklisted, suspect hack, need help ASAP [Daniel M. Ingram]


Daniel M. Ingram - 2014-04-11 08:59:44 - DhO email Blacklisted, suspect hack, need help ASAP

I started getting messages a few days ago from people who couldn't get email replies when they requested new passwords, and then, after asking Chris Stavros of Omegabit about it, I received this:


"Hi Daniel,

I had this looked at.  There are a bunch of email messages stuck in the outbound queue.  I'm afraid it looks like your mail server (the outbound mail engine on your portal server), has been blacklisted and many messages are not getting routed to their destination because of reputation issues.  See:

    http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a72.29.184.157&run=toolpage  

(see the excerpt from the mail logs below, as well)

There may be others not on this list - the raw mail logs also list additional listers, but once your one one of these master lists, you have to clean it up from the top.  You can follow the various listers and learn more about how to petition to be removed - it is an arduous process and sometimes you just have to wait for the reputation to clear naturally.

It is most likely that there is a mail form exposed somewhere on the site - do you have the "invite" portlet or a mail form available anywhere?  Some user has probably figured out how to exploit it for sending spam via the portal.  It should be removed.  If none exists, this could be some sort of hack - though, we've never observed one of this nature (it is usually a form or the invite portlet).  It could hypothetically be some sort of exploit of the message board engine.  But, I would look for portlets that generate email in the layouts, first.

The fastest way to resolve this (after the hole is plugged by removing the portlet - assuming there is one), would be to reroute email through a different mailhost that is not blacklisted - preferably, one with advanced spam filtering and reporting capabilities.

You might start by trying to route it through whatever handles your interactivebuddha.com mail (looks like Google?) - this would be a good idea for many reasons including source reputation (it will match your domain), and that they would probably catch the spam before you got blacklisted, saving you grief.

You can set the SMTP host to Google's, with your account credentials, and it <should> allow you to relay via their mail servers.  You may need to enable this for your domain.  See for more info:

    https://support.google.com/a/answer/2956491?hl=en

The SMTP mail setting for the portal can be adjusted here:

    Control Panel -> Server Administration -> Mail -> (set the Outgoing SMTP Server setting to match your relay-authorized Google account).

If you can help us to find the means that is being exploited for mail in the portal, I can offer one alternate option: a one-time remedy, which would be to reassign your outbound mail IP address to something that is not blacklisted from our network (essentially, a new public IP for mail routing).  We would be assuming some risk for that action if it were determined to be a malicious attempt to hide intentional spamming by our upstream providers.  So, it is important that we are diligent and beyond reproach in terms of making every effort to address the root cause before exercising that option; it is a last resort.  Routing through your legitimate domain/mail provider is the "correct" approach, from a reputation perspective.

Concerning helping your users that are having issues:  My recommendation is to manually reset their password to something for them, and to email them directly, at which point they should be able to login and reset it to whatever they prefer.  You may have to do that for users that have locked themselves out or requested a new account until this mail reputation issue can be sorted out.

I hope this helps - let me know if you have questions or need help on how to proceed.

-Chris"

This is just one step more tricky than I likely can pull off: anyone want to help? I tried following the instructions in the support.google.com/a/answer etc link but can't find the admin console for google. I am only intermediately technical. Anyone offer to help with this?

Thanks very much,

Daniel

-------------------

James Yen - 2014-04-11 11:13:31 - RE: DhO email Blacklisted, suspect hack, need help ASAP

I think that was my bad because I kept putting fake emails (blahblah@blahblah.com) when I registered. And I made a lot of accounts. Let me know if you need any help as I think I've done this before.

-James

-------------------

Chris G - 2014-04-11 11:13:57 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Hi Daniel,

I'm not really an expert, but you should be able to find the admin console for google at http://admin.google.com. If that doesn't work, try http://admin.google.com/interactivebuddha.com . This is assuming that you have a Google apps account for interactivebuddha.com (do you?)

I set up a google apps for business account to look at this, and the instructions for setting up the SMTP relay that Omegabit pointed you to (https://support.google.com/a/answer/2956491?hl=en) seem accurate.

You'll need to first set up the relay on Google's end, then set up DhO to route mail through Google's servers.

-------------------

Droll Dedekind - 2014-04-11 15:19:59 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Just sent you an email with potentially useful information about exploits.

Does Google handle your mail?

-------------------

Ivo B - 2014-04-11 17:02:50 - RE: DhO email Blacklisted, suspect hack, need help ASAP

There are this things called RBLs. 
http://www.rblmon.com/blog/what-are-rbls-and-how-do-they-work/

Looks like you're on it. Thats bad because people use those lists to fight spam. I dont aprove of RBLs since they are anarchistic. Owner decides if you'll be put off. There are other ways to fight spam. I can help with setting up clean email server until things are cleared up on new IP. I've seen your problems in other thread but i was put off by liferay platform. From what I see you/we basically need a forum with PMs and wiki. Not much else. That can be hosted very light! Including email server. I can help if forum is easily extractable from liferay, otherwise...

You could host everything for 20 bucks plus 5 bucks for backup per month with vps provider im using. One of the best there is. Thats what i use for personal server. Same setup for company i work for. Very professional VPS people.

I can help set up DNS too. No problem.

Well, ttyl

-------------------

Ivo B - 2014-04-11 17:05:17 - RE: DhO email Blacklisted, suspect hack, need help ASAP



Sorry james, but i had to laugh here
You're funny

-------------------

James Yen - 2014-04-11 19:55:58 - RE: DhO email Blacklisted, suspect hack, need help ASAP



Thanks, I try

=p

-------------------

Daniel M. Ingram - 2014-04-11 23:11:52 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Dear Ivo,

Current best theory is that Liferay 5.2.2 and the version of JBoss we are using are both vulnerable to attack and exploitation.

I have tried for 2 years, over $2000 and now 7 server/Liferay/tech guys to upgrade to 6.2 and all so far have failed to do this.

The last guy to work on this, Manish, vanished for a while but has reappeared. Apparently he had a death in the family. Now he says he will help with this, but I am not holding my breath, as he started a year ago.

We will see how this goes.

I am not sure if we will be on Omegabit or on hosted on OSX on a MacMini I now have sitting here on my desk for that purpose. We will see.

If we end up on Omegabit, and we end up on 6.2, then in theory our problem is solved, but if not, then I may need your solution.

When you say you could extract us from this Liferay nightmare and put us on another platform: what would you put us on and what capabilities would we lose/gain?

Thanks,

Daniel

-------------------

Ivo B - 2014-04-12 04:21:47 - RE: DhO email Blacklisted, suspect hack, need help ASAP

This site basically runs around bulettin board or forum, so this would be main priority.
Biggest job is to convert this forum into another one. We would probably need to write
a database conversion script.
There's also information in FAQ and wiki. Minor problem. Nothing much is left afterwards. 
Or is there?

I would keep this functionality and try to implement it with proven light solutions.
Something like lighttpd+(fluxBB/myBB...).

A good email server setup is also needed. Postfix + dovecot + amavis/SA/clamav setup comes
to my mind.

Anyway... This (mainly db conversion) would need to be done in peace when current problems are over. 
I've seen someone is offering professional assistance( liferay thread ). Maybe that would be best first step
and then we could all work together  toward long term solution.

All the best

-------------------

Daniel M. Ingram - 2014-04-14 00:37:31 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Dear All,

As a precautionary measure to hopefully make it harder for whatever is going to happen (meaning possibly the DhO being hijacked to send spam), certain features, specifically new account creation and resetting forgotten passwords, have been turned off. Also, the mail and invite portlets have been disabled. I hope this may stop the problem. In the meantime, my apologies. In theory this will be resolved by the end of April. In practice, don't hold your breath, and pay attention to it instead... ;)

Practice well, and enjoy what access you may have to the DhO,

Daniel Ingram
Dharma Overground Underlord

-------------------

Travis Gene McKinstry - 2014-04-15 18:52:36 - RE: DhO email Blacklisted, suspect hack, need help ASAP



Many of us understand you are doing your best.

Thanks for the website. For what it's worth, if the site becomes compromised, I enjoyed it while it lasted optimistically, I'm sure the problem will be fixed soon.

-------------------

Daniel M. Ingram - 2014-04-17 04:29:28 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Apparently shutting those things down did stop the spam access and the 50,000, yes, 50,000 bad emails in the queue have been removed.

We await a permanent solution, but in the meantime, stabilization has occurred.

-------------------

triple think - 2014-04-27 21:12:38 - RE: DhO email Blacklisted, suspect hack, need help ASAP

Sorry to hear about the digital influenza. Good to hear the ship has been righted.


all the blessed
triplethink / nathan :::=}>