Linux.com

Feature: Open Source

Using Mozilla 1.3's "Junk" email filter

By on April 14, 2003 (8:00:00 AM)

Share    Print    Comments   

- By <SLASH HREF="http://roblimo.com" ID="d6ce25c2cc3b127d06ee7072e4e96563" TITLE="" TYPE="LINK">Robin 'Roblimo' Miller</SLASH> -
My new Mandrake 9.1 installation includes Mozilla 1.3, which comes with a simple <SLASH HREF="http://www.paulgraham.com/better.html" ID="43d4426f3a32906ab1fce1bbca66899b" TITLE="" TYPE="LINK">Bayesian email filter</SLASH> that does a pretty fair job of getting rid of junk email before you waste eyeball time on it. But it is not a perfect utility, and unless you set it up correctly some of the email you want to read will inevitably end up in the "Junk" folder. This article explains some tricks I found to prevent this from happening, all of which are dirt-simple to implement once you know about them.

The "Junk" feature implementation is easy: You start by clicking on the "Tools" button in the Mozilla header while you're in the email (messenger) section of Mozilla. You set the "Junk" filters up to your taste, including assigning an appropriate folder in which to place the auto-determined spam. Then, whenever you get a piece of email you consider spam you click the "Junk" button in the Mozilla header, which puts a little wastepaper basket next to that email's subject line in your list of inbox messages. Once "Junked," you send that message to your trash bin by using the usual "Delete" button. Before long, messages that are similar to the ones you have marked "Junk" automagically get shunted to your "Junk" folder. Yay! No more penis or breast enlargement offers, no more incomprehensible Chinese or Korean characters that take forever to display on an old/slow computer. No more spam.

Except it isn't quite that easy.

I subscribe to a number of email lists, and I found that emails from those lists were being consigned to the "Junk" folder along with all the offers to date beautiful Russian women, make big money from home with my computer, get paid for my opinions (which I already do. come to think of it), and buy wholesale pipe fittings from factories in Hong Kong.

So I did the RTFM (Read The Fine Manual) thing and clicked the "Help" button up in my Mozilla header, chose "Help Contents," and selected "Controlling Junk Mail" from the table of contents. That got me nothing but this sad entry:

Controlling Junk Mail



text to come

Using Junk Mail Controls



text to come

[ Return to beginning of section ]
Wah!

Next step: Jumped on Google with the keywords "Mozilla 1.3 email junk filter" and scanned through the pages it brought up, starting with this FAQ, which turned out to be slightly outdated, and at the same time mentioned some "features to come" that don't seem to have been implemented in my copy of Mozilla.

I tried doing something else I figured out from reading Mozilla email filtering rules in general: Adding "good" email list addresses to my address book, since the "Junk" filter doesn't act against addresses you have chosen to save. This semi-worked, and I say "semi-worked" because to make it work 100% I'd need to save the email address of every person who posts to every email list I'm on that transmits the poster's email address as a "from" and uses the list address as a "reply to." Tedious!

Next phase: I used the original Mozilla email filters to create an "if message is 'to' [list address], put it in 'lists' folder" rule. This action eliminates the "list mail is probably junk" problem, but would give me false positives from many low-traffic lists I'm on until I spotted traffic from them in my "Junk" folder and made filters for all of them. Making an email filter in Mozilla is simple -- merely click-click -- but when you're a hard-core, commercial-level email user (as I am) who deals with heavy email traffic, this can take a substantial amount of time. Besides, I prefer (don't ask why; it's just an old habit) all my email in a single inbox, and to go through it in the order it is received.

Toggle magic!

This turned out to be the right trick. Although I didn't spot this feature in any of the (scanty) writings I found about Mozilla's "Junk" email sorting feature, it seems you can go through the "Junk" folder and unclick the "Junk" icon on all messages automatically consigned to that folder that shouldn't be there. Do this enough (like for two or three days), and the program will learn that Politech, SLUG, and other list emails aren't spam.

I now have a working spam filter that gets rid of at least 95% of all the junk people send me, but hasn't slapped any email I want to see into the "Junk" folder for several days. I still check because that's how I am, but it's been a while -- at least 1000 emails -- since I last saw a false positive. And it took no special program download or installation, and no fees of any kind, to make this happen, just some easy clickety-click action while reading my email with Mozilla.

This is another major win for the Mozilla team. All they need to do now is document this feature so we don't have to grope around, trying to learn how to use it correctly. And I'm sure they will handle this little task before long.

Share    Print    Comments   

Comments

on Using Mozilla 1.3's "Junk" email filter

Note: Comments are owned by the poster. We are not responsible for their content.

Mozilla junk mail filters

Posted by: Anonymous Coward on April 14, 2003 08:11 PM
The junk-mail filter interface gets even better in 1.4a. Try it, you'll like it.

#

Mozilla Spam Filter works!

Posted by: Anonymous Coward on April 14, 2003 08:16 PM
I've been "training" the Mozilla spam filter for about 5 days, and have less than 1 false positive for every 100 messages that I receive.. and since it is still in "training" mode, I expect these false positives to drop to 0 within a month.

Truly amazing that this software is free.

#

Mozilla Spam filters dont delete off POP server

Posted by: Anonymous Coward on April 14, 2003 09:23 PM
Everyone keeps raging on about various Spam Filters but I find that they do not stop you from downloading the Spam (which is what I want) by deleting known Spam off your POP or IMAP mail server. The fact that you have to actually download "all" the email first before it can filter it is rather self defeating. Sure you might not see it, but it using up bandwith + time just downloading it, just so a filter can put it in another directory on your hard drive.<nobr> <wbr></nobr>:-(

I feel that this feature could be built into many Email clients (with you having the option wether to us it or not) as at present I use kshowmail (at sourceforge.net) under Linux and eremove (at tucows.com) under Windows to do just that job (delete unwanted mail from my ISP's POP mail server). And there are quite a few other programs for both Linux and windows that can delete unwanted junk or spam off your POP server. They have been around for many years, so it is not as if I am proposing something new !!!

So why can not all those Email Clients (like Mozilla-Mail) incorporate this feature as well ?

Untill some one gets wise (the writers of Mozilla) I will continue to use Email-removers (off POP server) befor I ever get my mail. BTW many Email-removers have the option to open your Email Client after you have deleted the Spam/Junk, so why bother with Filtering that forces you to download all that Spam/Junk first ?

 

#

Re:Mozilla Spam filters dont delete off POP server

Posted by: Anonymous Coward on April 14, 2003 09:36 PM
"So why can not all those Email Clients (like Mozilla-Mail) incorporate this feature as well ?"

i am not familiar with the kshowmail and eremove that you mention, but i suggest that an MTA (or later filter on the MTA) that does it is even better than a specially modified client that does it (for example a spamassasin equipped procmail or sendmail that rejects rather than delievering, spam). simply put, because even a specially modified client still has to sort through the mail stored on the server. it may not be YOUR disk space or bandwidth, but its still disk space and bandwidth.

now the case can be made against the entire argument here about not even wasting disk space and bandwidth. that case is also simple, none of the filters are perfect and most people dont want their mail dumped or filtered AND rejected. they want it filtered and delivered to another mailbox (ie Junk as in this article) so that they can periodically review it. yeah its a pain, no it doesnt save disk space or bandwidth, but the reality is that no filter is perfect.

#

Re:Mozilla Spam filters dont delete off POP server

Posted by: Anonymous Coward on April 15, 2003 02:37 AM
See Marc Merlin's <A HREF="http://marc.merlins.org/linux/exim/sa.html" TITLE="merlins.org">SA-Exim</a merlins.org>.

You can set various thresholds. I personnally deliver all mail with a score below 5 as regular mail, tag messages between 5 and 8, and teerbrube with 8 an up. False positives? Above 8, they will simply bounce!

#

Re:Mozilla Spam filters dont delete off POP server

Posted by: Anonymous Coward on April 15, 2003 09:25 AM
I use to the old ICQ's pop mail notification of deleting email. Although i am a fan of deleting things on the server...having a junk mail control on the server could be a little dangerous, if it deletes a genuwine email, most likly you will never see it. Those things dont have recycle bins.

#

Surpise

Posted by: Anonymous Coward on April 15, 2003 04:36 PM
Ever wonder how the spam deleters that affect your pop3 mailbox directly work?

They DOWNLOAD the message, determine if it's spam, and if so send the "Delete" command to the pop server.

So you're not saving any time/bandwidth/etc using them.

The only exception is a filter that works soley on headers, in which case, the utilty would only have to d/l the headers. Problem isheaders alone are not enough to determine spammage anymore.

#

Re:Surpise

Posted by: Anonymous Coward on April 15, 2003 06:36 PM
I think you should try out Kshowmail or Eremove first !! Neither download the message just the From: by name & email address - Subj: - Size: - Date:

Both have options that allow you to individually VIEW an Email (ie. if you want to check before you delete) and Eremove has a really good option that lets you ONLY view XYZ lines (mine is set at 100)

So one is able to seperate the Spam/Junk/Unwanted very easy, and just not delete the ones you want.

Try them out before you make coments - might not be your cup of tea ? but I find them very usefull. Also they are both FREE - and there are other simila programs that do the same job also.

Why ALL Email Clients dont bother to include this facility AS AN OPTION for those that want it is beyond me !!!!!!!!!!

 

#

Surprise yourself

Posted by: Anonymous Coward on April 17, 2003 06:41 AM
Read the RFC that defines POP3 (RFC1939). You CANNOT download only specific header lines. CANNOT. There is no such command in the POP3 standard. These tools probably use POP3's TOP command to download the complete header, along with a specified number of lines of the body of the message (ie "TOP 2 1" to download the header and first line of message number 2).

Any tool that claims that it works without downloading the message should actually say that it downloads the header and tells the POP3 server what to do with it, and hides the process from you so that it APPEARS that no downloading has taken place.

This still limits you to filtering based on headers, which may or may not be good enough for your needs.

--
Anthony E. Greene mailto:agreene@pobox.com

#

bug (request for enhancement) 45609

Posted by: Anonymous Coward on April 15, 2003 09:57 PM
Remember that opensource projects rely on contributions. The fact is that there is a request for this feature in bugzilla ( bug 45609 http://bugzilla.mozilla.org/show_bug.cgi?id=45609 ). It's not that there has been a decision not to implement it, but rather that no one has donated the code yet.

#

Re:Mozilla Spam filters dont delete off POP server

Posted by: Anonymous Coward on April 18, 2003 03:44 AM

Those 'magical programs' would have to download the message anyway, otherwise they couldn't take a look at it themselves. So what happens, basically, is that you download all messages (spam+nospam) during checking, and once again all nospam messages with your regular email client.

What exactly is the benefit of such programs?

#

When RTFM doesn't work

Posted by: Anonymous Coward on April 14, 2003 09:50 PM
"So I did the RTFM (Read The Fine Manual) thing and clicked the "Help" button up in my Mozilla header, chose "Help Contents," and selected "Controlling Junk Mail" from the table of contents. That got me nothing but this sad entry:


        Controlling Junk Mail

text to come


        Using Junk Mail Controls

text to come"

I usually check for a help file, and come up with missing, outdated, or just plain inaccurate information more often than not. Which leads to my conclusion that the oft-given command "RTFM!" indicates that the person telling me to RTFM has probably not read TFM, nor tried to get the problem solved using the information in TFM.

Could we declare a feature freeze until the documentation catches up?

#

"Junk" Mail

Posted by: Tall Mario on April 14, 2003 11:06 PM
People are constantly trying to divise solutions that get rid of so called "junk" mail, but most people have never considered the alternative: why not actually read it? Most of these people are just trying to make a buck, selling their products or software. With the current downturn in the economy, the least we can do to help out is to take a look at what they're selling. Some of these people actually have really good quality stuff to offer.

#

Re:"Junk" Mail

Posted by: Anonymous Coward on April 14, 2003 11:24 PM
99% is junk. If the quality would be 99% I would read 5%. Time is money!

#

Re:"Junk" Mail

Posted by: Anonymous Coward on April 15, 2003 12:42 AM
HAHAHAHAHAHAHAHAH.....
good joke... hahahah...

spam with a useful content...hahahah

keep the good humour.

#

Re:"Junk" Mail

Posted by: Anonymous Coward on April 15, 2003 01:18 AM
"5) Porn. Endless porn come-ons. Tired of them."

I have to agree. I never thought I'd ever be tired of porn, but the spammers actually did it!

#

Re:"Junk" Mail

Posted by: Anonymous Coward on April 16, 2003 03:10 AM
LOL, your parents want you to read porn headed to them? They obviously want to avoid the bees and flowers discussion.

#

Re:It's JUNK That's Why We Don't Read IT

Posted by: Anonymous Coward on April 17, 2003 07:28 AM
Ahh, the poor junk mail spamers. Lets all support those poor, poor ass-wipes.

You can buddy, not me!

#

Why Bother?

Posted by: akseli on April 15, 2003 02:14 AM
Hey, until a few days ago i would recieve anywhere from 1 to 100 spam/junk e-mails a day. I use Outlook 2002 (XP) and used it's built in spam detection tool. But as just about everysingle other think that microsoft includes in its software the spam filter wasn't the most apt to do the job. It works very much like the Mozilla filter, you have to "train" it to detect spam and non-spam e-mails. Now I really don't have time to "train" a computer program to delete spam, I mean the whole idea is to waste less time. So what did I do, I used google, after trying a few junk mail removers for Outlook I found exactly what i wanted, an integrated, powerfull, and fast solution for managing spam. It's called Cloudmark Spamnet... never heard of it? Neither had I, but it works, and it works extremely well too. I didn't have to train it, I just configured it once (1 minute) and I was up and running. In the last week I have had 1 e-mail not detected as spam--- not bad, and NO e-mails that were NOT spam have been confused for spam, and I'm subscribed to quite a few newsletters too.

I'm not saying that Mozilla is bad, actually it's great, but I prefer Outlook using Cloudmark Spamnet.

(I never really was a great Netscape/Mozilla fan anyway)
for linux users mozilla or evolution are probably the two best alternatives when looking at e-mail.

#

Re:Why Bother?

Posted by: Anonymous Coward on April 15, 2003 03:57 AM
Hey, if you want to give M$ permission to look inside your computer and change things without your express permission or knowledge, go for it, man. Whatever floats your boat.

#

Re:Why Bother?

Posted by: Anonymous Coward on April 16, 2003 02:27 AM
I'd rather say "why bother about outlook?". Haven't you thought about that before writing?
I don't bother about outlook cause I can't use it and I don't want it anyway. Not useful for me.

#

Re:Why Bother?

Posted by: Anthony E. Greene on April 17, 2003 06:55 AM
The best (most efficient) mail system for Linux is fetchmail+procmail+mutt+sendmail_or_postfix.

The GUI mailers are okay if you get a few dozen or less messages a day, but once you get into the hundreds for incoming and the dozens for outgoing, speed and efficiency during use start counting for a lot more than ease during the learning phase.

Procmail is especially useful. You can make it do just about anything with a message. I have filters that add a Reply-To header for those mailing lists that default to replying to the sender instead of the list. I have filters that check my whitelist and blacklist, and filters that change the MIME-Type headers for some PGP messages to enable automatic decrypt/verify processing by my mailer. Each filter that delivers to my Inbox adds a header that tells me which filter delivered the message. That makes it easy to make any required adjustments. I have filters that delete duplicate messages and filters that automatically detach and store certain attachments. Procmail is by far the most powerful mail filtering I've ever seen.

Mutt is so configurable that I don't even want to get into it here. No other mailer I've seen even comes close, and I've used lots of them on Windows and Linux.

#

mozilla junk does not work

Posted by: Anonymous Coward on April 15, 2003 02:39 AM
Nor does NewsForge - I tried to create an account, and tells me that my password is incorrect! I haven't created a password, yet. Oops.

Anyhow, I use Moz from a few boxes; w32 & Linux. The good news is that I get the same behavior on all of them. The bad news is that the spam filter does not work.
I also use the prebuilt release binaries of Moz 1.3.

Also, Robin's instructions are unclear, in particular this excerpt:

The "Junk" feature implementation is easy: You start by clicking on the "Tools" button [in email]<nobr> <wbr></nobr>... You set the "Junk" filters up to your taste

There is no menu option for "Junk filters" under "Tools". Here are the options:

  • Search Messages
  • Search Addresses
  • Message Filters
  • Junk Mail Controls
  • Mark Selected Messages as Junk
  • Mark Selected Messages as Not Junk
  • Run Junk Mail Controls on Selected Messages
  • Run Filters on Selected Folder
  • Import
  • Password Manager
  • Download Manager
  • Web Development
  • Switch Profile

This is not to nitpick, as it is confusing, between "Run Junk Mail Controls on Selected Messages" and "Run Filters on Selected Folder", which are quite different things.

I can click on the Junk Icon in the toolbar, and the Junk Icons show up in the column to the right. So far, so good.

The problem is that I tried clicking on the "Delete" toolbar icon, as suggested by Robin, this is no different from hitting the Delete key - the email message goes into Trash, not into the Junk folder.

Running Junk Mail Controls on Selected Messages does not work, either. I can, however, drag messages marked as junk into the junk folder.

There is no automation going on here. No new messages show up in my Junk folder.

If you want to contact me, about the NewsForge bug, strip the spam guard from <tt>bingalls@fit-zones*SpamGuard*.com</tt>

#

Re:mozilla junk does not work

Posted by: Anonymous Coward on April 15, 2003 09:22 AM
Are you marking the email as Junk...hit the junk button

#

Re:mozilla junk does not work

Posted by: Anonymous Coward on April 15, 2003 10:43 AM
you're confused. As you're training it by marking spams that get through you must mark them as junk then delete them in the normal way. Over time Mozilla recognises them as junk and moves them automatically.

#

Mozilla Spam filter woe

Posted by: Anonymous Coward on April 15, 2003 03:12 AM
I've got a problem with mozilla spam filter as well. I've been successful to get it running in RH 8.1. However since I upgraded to RH 9.0 (clean install, not upgrade), Moz 1.3 spam filter has never worked for me.

I have enabled the spam control, set it to automatically trash any incoming junk, but after around 150 junk mails manually marked, the spam filter is not marking any new incoming (blantantly junk) mail. It feels as though the junk marking is not being run.

Does anyone have the same problem or can come up with a solution?
Is this one of the 'training' mode that I have to somehow get the spam filter out of?
If I delete all the junk emails, will the spam filter still remember its training?

#

Re:Mozilla Spam filter woe

Posted by: Anonymous Coward on April 16, 2003 02:05 AM
Yes, I continue to see the same thing...but I haven't upgraded from RH 8.0 yet, so I suspect that the RH upgrade has nothing to do with the problem. It worked great for a few days catching about 200 spams per day with no false positives; but then inexplicably it no longer does anything at all no matter how much additional training I give it. I had previously trained it on both good and bad email.

#

Email Filters

Posted by: Anonymous Coward on April 15, 2003 02:06 PM
I'm a Email Filter! Use WebMail to site & remove before downloading. Bit of a hassle but positive

#

Superfluous article

Posted by: Anonymous Coward on April 15, 2003 04:05 PM
I think the author's ranting about the lack
of help is unfair. It seems completely obvious that one should toggle the junk icon to
declare a message *not* junk.

#

Re:Superfluous article

Posted by: Anonymous Coward on April 15, 2003 11:21 PM
why is it unfair ? are you saying that it's ok to include "help" buttons in software but not include the help ? jeez.

#

Re:Superfluous article

Posted by: Anonymous Coward on April 16, 2003 02:12 AM
No, it's not completely obvious! Like how do you train it, or even know that you have to train it? When I first clicked on the junk button the little icon toggled but the message stayed in my inbox. I was still staring at all this spam after hitting the button...does this think work I thought? So I too tried to read the help and I too was disappointed.

So then I even thought I'd visit the Mozilla website hoping that there'd at least be some simple instructions there. Nope, not even to be found in the release notes. All this hype about this big new feature and not even a single sentence about how to use it!

#

Re:Superfluous article

Posted by: Anonymous Coward on April 16, 2003 11:21 PM
Never used Google, huh?

http://www.mozilla.org/mailnews/spam-howto.html
http://www.mozilla.org/mailnews/spam.html

#

How I THINK it works

Posted by: Anonymous Coward on April 17, 2003 12:54 PM
I believe the point of the article is the Junk features are not intuitive. At first, I didn't think it even worked. And the TOTAL lack of documentation was maddening, because unlike lots of things in Mozilla, this feature does NOT appear to make sense. At least initially.

But the article did NOT clearly say what to do. So, here, in a nutshell, is what worked for me.

1. Start using Mozilla for mail. Set up your account(s) accordingly. Most of you have already done this.

2. Choose the Junk Mail Controls option from the Tools menu and tick the box to turn the feature on. While there, tick the option to not mark as Junk mail received from people in your address book, and tick the option to automatically move Junk mail to a Junk Folder. If Mozilla doesn't create the folder for you after you press OK, create it yourself.

2 1/2. Now, bear with me on this one. Go back to Junk Mail Controls and un-tick the option to automatically move Junk mail. But leave the other options alone.

3. Start receiving junk mail. If you're like me, you won't have long to wait. Ugh. Select a piece of junk mail and press the Junk button in Mozilla's control bar. Notice a little icon for Junk has appeared in the caption listing for the offensive mail.

4. Notice that NOTHING ELSE HAS HAPPENED. It's still in your Inbox! Did you assume like me that the message would move to the Junk mail folder, similar to how pressing Delete sends mail to the Trash? Didn't happen! Well, that's NORMAL. Now, get ready for the lame part, probably fixed in version 1.4: MOVE (i.e. drag) THE MAIL NOW MARKED JUNK TO THE JUNK MAIL FOLDER YOURSELF.

5. Repeat.

6. A faster way is to just tick the Junk indicators in the message captions for all your Junk mail at once and then select/move it en-masse into the Junk folder. This way none of the Junk has a chance to do any HTTP or URL stuff that occurs when you view it.

7. The important thing is you've MARKED the offensive mail as Junk. It doesn't feel like you've done anything, but you HAVE. Over time, you'll start to notice new, incoming mail being marked as Junk automatically!

8. Now, you know the Junk Mail feature is WORKING!

9. Go back to Junk Mail Controls and re-Tick the feature to automatically move Junk mail to the Junk Folder.

10. Now, you'll notice Junk mail doesn't appear in your Inbox; it's all going to the Junk Folder on its own. This is what you were after in the first place.

11. You're not quite done! You never will be, because some Junk Mail is bound to slip through and sit in your Inbox, while something important may be labelled Junk by mistake and wind up in your Junk mail folder (you need to check it every so often). Depending on which it is, toggle on or off that Junk indicator in the caption list, then manually MOVE the mail to the appropriate folder. The more you do this, the smarter the feature will become.

CONCLUSION: Now that I see it working, I'm really happy it was supplied with Mozilla. It's terrific and is working real well for me. But, there was that initial trial & error before I figured it out. Hopefully, this will help you get cracking before version 1.4 and its better interface gets released.

#

Re:How I THINK it works

Posted by: Anonymous Coward on April 21, 2003 08:36 AM
>Now, get ready for the lame part, probably fixed in version 1.4:
>MOVE (i.e. drag) THE MAIL NOW MARKED JUNK TO THE JUNK MAIL FOLDER YOURSELF.

Stupid, there is no need to move the junk mail into the junk folder, just mark it as junk and delete it.

#

"Junk" folder

Posted by: Anonymous Coward on May 29, 2003 06:50 AM
When using Mozilla 1.3 on Windows 2000, junk mails were being marked, but not moved to the Junk folder.

I renamed the folder "Junk" (WITH " QUOTATION MARKS " IN THE FOLDER NAME) and it started moving them as expected.

#

This story has been archived. Comments can no longer be posted.


 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya