Configure corporate mailing lists for OpenCulinary C.I.C.

[jayaddison]

Is your feature request related to a problem? Please describe.
Project development of RecipeRadar is managed by OpenCulinary C.I.C.. Project-related communication (announcement, development discussions, automated emails, and so on) are to be provided using public mailing lists, similar to the Debian project mailing lists.

Describe the solution you'd like
A set of public mailing lists for development of RecipeRadar, including web archives available for browsing. These should include:

• reciperadar-announce - announcements including major updates, features, and events.
• reciperadar-development - development-related discussion.
• reciperadar-feedback - feedback received through the application from users.

This set of mailing lists is likely to expand and change over time.

[jayaddison]

Suggested domain name for the web view of the mailing lists, and email addresses for the mailing lists:

• lists.reciperadar.com
• reciperadar-announce@lists.reciperadar.com
• reciperadar-development@lists.reciperadar.com
• reciperadar-feedback@lists.reciperadar.com

[jayaddison]

Infrastructure

As a best practice, it usually makes sense to provide separation between infrastructure components that don't have software-level dependencies (the mailing lists are for discussion about RecipeRadar -- but RecipeRadar could exist without the mailing lists, and the mailing lists could exist without RecipeRadar), and so we should design for that.

Currently both HTTP and HTTPS traffic for *.reciperadar.com domains arrive at a single IP address and webserver (haproxy), so it would likely be easiest to extend that webserver configuration to reroute traffic for lists.reciperadar.com. At the moment that is, I think, the only place where infrastructure-sharing for this feature would make sense. It'd be worthwhile to review the haproxy.cfg to consider any edge cases and problems.

Although the RecipeRadar service is containerized, I think the argument around avoiding infrastructure sharing between independent components extends to the configuration of the mailing lists, so I think we should not use containerization for these mailing lists.

I think GNU Mailman probably makes sense as the system to deploy.

Permissions

• Only the reciperadar-development list should allow posting by the public.
• To begin with, the reciperadar-feedback list should only accept posts from the RecipeRadar api application. It may be worth reconsidering that later.
• The reciperadar-feedback lis should only accept posts from addresses approved by the mailing list administrators.
• All of the mailing lists should allow confirmation-based subscriptions.
• All of the mailing lists should have complete, public published archives available on the web.

Edit: updated remove suggestion of using GNU Mailman; currently the Courier MTA is being considered instead.

[jayaddison]

I think GNU Mailman probably makes sense as the system to deploy.

Eh, maybe. It has a lot of dependencies, and it requires two persistence/query engines: one for persistent storage of email content, and one for search. For the former, we would likely use PostgreSQL since we have it in our stack already; for the latter, any backend supported by django-haystack should work.

However: I don't think we would want to use the Haystack OpenSearch backend, because we wouldn't want mailing list activity to affect the performance of the RecipeRadar online service. PostgreSQL would be a good option since it uses different storage I/O paths in our environment and it has mature built-in full-text-search support. However, it is not currently supported as a django-haystack search backend.

I recommend that we look into the public-inbox project instead as a way to host our mailing list infrastructure. It aligns well with our software licensing preferences, provides git cloning support for archiving and mirroring, and looks like it should require minimal infrastructure cost to support. It uses Xapian to provide mail archive search functionality. It does pull in a reasonably large number of transitive perl-based dependencies, but I think that's an acceptable tradeoff.

[jayaddison]

I've created https://github.com/openculinary/mailing-lists ... although am not completely certain that that's the correct approach; it seems that each individual mailing list is stored as a git repository, but I've created a single top-level git repository containing three git repositories that are in turn under source control.

I don't necessarily see a problem with that, but it's likely not the intended architecture and I'm not sure whether there are benefits to having meta-level source control. The downside is mostly that it's weird and potentially confusing, and perhaps slightly wasteful.

[jayaddison]

• configure openculinary.org outbound

  • smtp.openculinary.org
  • spf (allowed: mx)
  • dkim (message signing - requires key generation)
  • dmarc
  • port 587 (msa)
  • mx record
  • testing (send email from openculinary.org sender to reciperadar.com)

• mailing list git repositories

  • git.openculinary.org
  • read-only webserver (public-inbox-httpd)
  • forwarding from haproxy

• configure openculinary inbound

  • check and configure smtpd access policies
  • git repository write permissions
  • public-inbox stdin reader
  • testing (send local email to reciperadar-development@lists.reciperadar.com)
  • forward port 25 from router
  • testing (send external email to reciperadar-development@lists.reciperadar.com)

concerns:

• uptime for the mailserver
• (limited) separation of concerns in terms of infrastructure

references:

• https://wiki.gentoo.org/wiki/Complete_Virtual_Mail_Server
• https://wiki.debian.org/Exim

[jayaddison]

I recommend that we look into the public-inbox project instead as a way to host our mailing list infrastructure. It aligns well with our software licensing preferences, provides git cloning support for archiving and mirroring, and looks like it should require minimal infrastructure cost to support. It uses Xapian to provide mail archive search functionality. It does pull in a reasonably large number of transitive perl-based dependencies, but I think that's an acceptable tradeoff.

Further to this: I think that, roughly speaking, I'm suggesting using git repositories written by public-inbox as the primary source of truth for mailing list discussion content. That is: instead of populating mbox / Maildir files, I think we could read mail directly from stdin (from exim/postfix/whatever the MTA is) and then write the results to a git repository that is used to serve HTTP/HTTPS traffic, and is cloneable using standard git tools.

[jayaddison]

Hmm. I was thinking that postfix would be the mail transport agent to deploy here, but I feel somewhat uneasy about that after learning some of the corporate history of IBM in relation to Nazi Germany.

It looks like sendmail has become somewhat more corporate-sponsored recently. It could be worth a look next.

[jayaddison]

Hmm. I was thinking that postfix would be the mail transport agent to deploy here, but I feel somewhat uneasy about that after learning some of the corporate history of IBM in relation to Nazi Germany.

Note: it's not simply the sense of wrongdoing (that seems to ring true given a pattern of sales-heavy, numbers-over-people, revenue-oriented mindsets), but also the fact that the company failed to keep records and also pushed back on attempts to make amends. Perhaps they were secretly working with the allies as well, or something like that - and there was probably an element of that, but to my knowledge so far, that doesn't counterbalance complicity with a terrible regime.

[jayaddison]

It looks like courier-mta could be a good mail transfer agent option to begin with here. It can be configured to redirect incoming mail messages to a command-line process on standard-input, either using a legacy-aliases format (similar to sendmail), or a preferred dot-courier configuration format.

[jayaddison]

I recommend that we look into the public-inbox project instead as a way to host our mailing list infrastructure. It aligns well with our software licensing preferences, provides git cloning support for archiving and mirroring, and looks like it should require minimal infrastructure cost to support. It uses Xapian to provide mail archive search functionality. It does pull in a reasonably large number of transitive perl-based dependencies, but I think that's an acceptable tradeoff.

This continues to be the plan, however there are some additional details that I've been considering over the past few days:

• There'll be an ongoing possibility of unsolicited or otherwise unwelcome inbound messages, and although I don't think we should place strong authentication mechanisms in front of the mailing lists (because this could discourage diverse participation), I do think that there should be a simple confirmation process to ensure that senders intend and accept that their messages will be public. There is some precedent for this in the PostgreSQL mailing list guidelines, and also similarly for the previously-mentioned Debian project.
• Migration of messages from the existing RecipeRadar inboxes (my own account, and the contact/feedback inboxes) is an open question. There was not the same expectation of public viewership for those when they were created, and so I don't think it's reasonable to export and then make them public without the permission of the people who participated in them. However, I do think that we should archive some of the content (certainly content that presents itself as human-authored).

[jayaddison]

Currently both HTTP and HTTPS traffic for *.reciperadar.com domains arrive at a single IP address and webserver (haproxy), so it would likely be easiest to extend that webserver configuration to reroute traffic for lists.reciperadar.com. At the moment that is, I think, the only place where infrastructure-sharing for this feature would make sense. It'd be worthwhile to review the haproxy.cfg to consider any edge cases and problems.

In retrospect, there's one additional place where infrastructure-sharing makes sense here, in the short term: since we have fairly minimal hosting compute resources, I think we can place storage of the mailing list content and git repositories into the existing persistence I/O storage path.

[jayaddison]

🚧 This work is currently paused pending some other infrastructure migration plans (physically moving the server hardware).