Search This Blog

Saturday 3 February 2018

Email Spam, Yahoo, DMARC and Mailing Lists: A Journey of Discovery

I belong to a small hobby organization, and assist with simple techy things that can help the group in various ways, like helping maintain the website.
A while back, the group was looking for a simpler way to be sure that they could communicate effectively and easily among the groups executive, using email.  This sounds easy, but in a group of 10, someone always seems to get left out of group emails sent from personal email accounts, or the wrong email gets included in a list, or emails get left in someone's personal list, that are no longer needed.
The group experimented with setting up a shared gmail account for this purpose, but gmail account security makes it very hard to share a single account, accessed from multiple locations.  Next up was a google discussion group, but that had it's own list of complexities.
So I suggested we just set up an Old School email list service.  You know, the ones were you define an email account, give a list of other emails to forward to, and just send anythng you want to share to it, for automatic forwarding to everyone.  The list gets maintained in one place, and we are done.
Well, they still exist, but most are commercial Marketing remailers, with tons of bells and whistles, and probably lots of other strings attached.  Some are free for entry level use, but I didn't spend too much time on them after looking into FreeLists.org .

FreeLists.org has been around for quite a while.  They are truly free, although they do not accept just anyone.  However our group met their criteria, so we set up our first list, and populated it with some emails and started testing.  Everything went fairly well, and we were happy, until one of our members sent out a test, and we discovered that his Reply To: email had been Redacted, and replaced with "dmarc-noreply@freelists.org" .  An interesting problem!  Since I already had one request for advice in the hands of the FreeLists Staff, I decided to nibble at this one for a while myself.

DMARC and Spam:
In my research, I learned there is an anti-spam toolset used by major email providers world-wide, known as DMARC.  One of the things DMARC does is allow email providers to identify the domain names they use to send email from (like gmail.com, cogeco.com, rogers.com, etc), and to embed that knowledge in several secure ways in emails and in Domain Name Servers (the "address book" of the internet), to allow receiving email services to securely determine if a given email has actually come from the email service it says it originated from (the "domain name" test).

Reject or Ignore:
Email sending services can also optionally tell a receiving email service what to do with email that fails this "domain name" test: either reject the email, or ignore it (allowing the receiving email service to decide what to do about it instead).  When DMARC was first rolled out in 2012 - 2013, this failure action was defaulted to ignore, by virtually everyone who implemented DMARC.

Yahoo got Spammed:
In early April, 2014, Yahoo.com was hit by major email spammers, who were spoofing the yahoo.com domain name, and creating havoc among yahoo email users and others.  Yahoo made a quick decision one weekend to change the DMARC failure action for their domain name from ignore to reject.  In doing so, all mail list and other remailing applications that included any yahoo email users, broke instantly  (their email spoofing problem was solved, though).  Not only did they fail, but the resulting error messages began bouncing back, and some of the bounces resulted in further rejects and bounces... See https://www.pcworld.com/article/2141120/yahoo-email-antispoofing-policy-breaks-mailing-lists.html for further information.

Fixing Mail Lists:

Mail List and other remailing service providers needed to deal with this, including our Mail List provider Freelists.org.  DMARC also provided some work-arounds.  See for example https://www.spamresource.com/2014/04/run-email-discussion-list-heres-how-to.html . At the time, FreeLists.org Facebook page reported:
   " Freelists - Freelists.org     April 19, 2014 ·
     A solution to DMARC-related issues has been developed and is being tested on a handful of lists that reported deliverability problems to us. Should feedback be positive, we'll continue rolling this out to the entire site in the coming days.
     Let us (staff@freelists.org) know if you'd like advance access to the fix, which replaces the subscriber's From: address with an "address redacted" sort of message when posting IF his mail provider publishes DMARC records."
Another common fix was to ask Yahoo email users to switch to another major email provider, like Gmail or Outlook...

So why did our Rogers.com Member get Redacted?
Examining the email header of messages from our Rogers User showed references to Yahoo!  Now that is very interesting!  No other user of our list had such references in their email headers.  But further research quickly uncovered the fact that Rogers has outsourced it's email to Yahoo.ca!!
I then discovered the toolset at the website dmarcian.com: https://dmarcian.com/dmarc-inspector/yahoo.ca
I ran this for the Rogers.Yahoo.ca email service, and determined that "p=reject", which is why our member's email is getting redacted.  His email provider prevents him from full use of a mail list.

In conclusion, our member gets to decide if a redacted reply address is okay when he uses our list, or if he will set up a different email account, to gain full use of the list.
We can also move forward to add a list that allows us to email amongst our total membership efficiently.

Saturday 20 January 2018

My NAS Experiences...

About a year ago, I embarked on the learning curve needed to implement a NAS (Network Accessable Storage) Server on my home Network.  A good opportunity to put one more of those old cast-off PCs to work, and find a home for those digital photos accumulating in various places.  

System 1: NAS4Free and PIII

After reading up on the several freeware type NAS implementations available, I decided an old PIII system with 500MB RAM would be good enough.  The box was small, did not use as much power as some of the other boxes I had sitting in the corner.  I stuck an old Maxtor 40GB PATA hard drive in it for the OS and NAS software, scrounged around until I found a PCI bus SATA card to use, and installed a 1TB SATA drive for Data.  INstalled Ubuntu Server OSA, which I had some experience with, then installed my first choice, NAS4Free.  Although I got it installed okay, I seemed to have a lot of trouble setting up appropriate security to be able to see it from other Windows and Linux boxes on my Network.  Maybe it was me; others have used it successfully, but I was having lots of frustration...

Version 2: Switch to OpenMediaVault (OMV), & Pictures!

So I started over, with OMV V2.1  OMV's install ISO does a very nice job of installing Debian and OMV on the Maxtor drive, and finding and setting up the Data drive on the 1TB SATA drive I had installed.  I was able to easily setup a number of data shares, and several classes of users, and see everything easily across my network.  The web admin panel works very well.  
I started to load pictures, from my Win10 PC, and had some more learning to do.  Fortunately others have been there before, and I can refer you to Awasu Consulting's very good articles on Open Media Vault, esp. the one entitled "Configuring Open Media Vault", and the section "Uploading Files to the NAS".
- https://awasu.com/weblog/omv-bpi/configuring-omv/

Other backgrounders I found helpful include:
- https://www.howtogeek.com/176471/how-to-share-files-between-windows-and-linux/
- https://www.howtogeek.com/howto/16196/how-to-disconnect-non-mapped-unc-path-drives-in-windows/

Soon I had uploaded many thousands of pics and videos from several vacations of the past several years, and made them available across our home network, and accessible from several different kinds of devices.  I have found Kodi to be an excellent Media Server, running on inexpensive Android based OTT boxes (about $30 each), feeding from the NAS box, and connected to TVs and a stereo in several rooms of the house.  Kodi is also easily controlled either locally on the TV, with the included IR remote control, or over WiFi, from our Android or iOS phones and tablets.  A very nice arrangement!

More Data: Movies and Music

Of course, there is lots more to do after these initial successes!  I added another 0.7TB SATA drive I had on hand, configured and shared it, and started to add Movies from our in-house collection, accumulated over a number of years.  Soon there were 200 Movies online and easily accessible, complete with supporting information courtesy of the Kodi installed movie information scraper apps.  

And all those music CDs!  Some had already been ripped, but we had lots more to do... Soon there were 1000 or so music CDs online, although since some had been ripped by different people, using different tools and formats, there was a lot of cleanup of existing digital files, so all the music was presented in a common way...  The Windows Media tools, and others, help a lot in setting this up, and Kodi is very happy using the repository so produced.

Setback 1: SATA Card Failure & System 2

And then the plug in PCI bus SATA card I was using failed....  I didn't have another, but I did have lots of other PCs in the corner.  This time I picked a slightly newer, more powerful Acer box, with all the right interfaces, 1GB of RAM, and an AMD Athlon 64b cpu, running at 2.7GHz.  A bit more power hungry, but we were happy with the results of having a NAS server online.  The hard drives installed easily, and the system booted up.  Back in business!  (I ordered another PCI SATA card, just to have around for future use).

Setback 2: Power Supply Failure, and System 3

A week ago, the power went out in our neighbourhood for a few hours.  A rare occurrence, but no big deal, except the NAS server did not restart.  Upon investigation, discovered that it's power supply, a Liteon branded 250W ATX12V supply, had failed.  Some research showed that this was a not uncommon problem in Acer PCs with this supply, once you add additional SATA drives to them!!  
A search through my collection of surplus PCs showed a few had ATX12V supplies, but not with the same motherboard connections (20pin vs 24pin power connectors).  There was also the question of just how much power the mb needed, and if the extra 4 pins were needed.  Unfortunately, Acer refuses to make any tech information on their motherboards public, so there is no way to answer these questions!  A new replacement supply was going to be about $40, but I did have a surplus PC, with the appropriate hard drive interfaces, and a better power supply reputation (an IBM / Lenovo PC).  It also had an even faster cpu, and 1.5GB of RAM.
So I moved the hard drive collection over, leaving the existing CD-ROM drive in place on the IDE0 cable, adding the Maxtor 40GB OS/NAS drive to the same cable, and plugging the 2 SATA Data drives to the 2 available SATA connectors, after removing the SATA hard drive that was originally in this machine.  I was expecting an easy reboot, and back in business.  Not so easy....

GRUB RESCUE>

Well, I had never run into this problem before.  It was accompanied with the error: "hd0 read error"
The BIOS sees the hard drives okay, although it is enumerating them in a different order than the previous PC did.  Did this matter?  No idea...
  The system boots ok from the Super Grub Rescue CD Recovery disk, and sees the hard drives okay.  I try several Grub recovery methods, but no luck.  But sometimes the CD doesn't boot... and I realize the BIOS doesn't see the CD drive...
  After more googling, I finally realize I should check the jumper settings on the PATA devices, since they are both on the same cable...  Yup, two Masters. (Shows how long it has been since I had to configure IDE drives!)
After fixing that problem, I have reliable CD booting, but still no success with Grub repairs.
  On to a more general purpose System Rescue Bootable CD, and it's also time to disconnect the SATA drives, just in case...
  Gparted is happy with the partitions it is finding on the 40GB drive, as described in the OMV information.  However, other tools are not happy with the ext4 file system, complaining of a invalid superblock.
  I switched to using Testdisk, after finding several good references online describing how to use it effectively (it is a complex tool).  By this time, I am in uncharted territory wrt my knowledge of Linux and it's file system details, so this is both fascinating and frustrating territory.  I am learning a lot. I am using my "Practical Guide to Ubuntu Linux" (4th Edition, by Mark Sobell) a lot, too.
  I was able to obtain the list of backup superblocks on the disk, tried fsck repairs with most of them, but no success.

Reinstall OMV, V3.0.94

I was wary of a re-install, since I did not want to have to rebuild the contents of the data disks, if they had to be re-initialized.  However, after more research, it looked like OMV could mount and share existing populated data disks, and that turned out to be the case.  I downloaded the latest stable release of OMV, rebuilt the 40GB OS disk with no problems, added the 2 data disks to the system, and began reconfiguring the system on my network.  Although my previous NAS configuration notes were not quite complete, I had no difficulty mounting and sharing the drives and their data folders, adding the users I had previously setup, configuring services and access rights, and getting everything back to normal.  And my configuration notes are in great shape now!!
So overall a success, although it took several days to work through the details.

Next time, I will go to a full rebuild of the OS drive much quicker...