My Understanding Of How UCE Actually Works
	     v1.2, mengwong@pobox.com 20010111
     http://www.mengwong.com/misc/postfix-uce-guide.txt

   http://www.postfix.org/uce.html exhaustively describes
   the syntax and semantics of several configuration
   options, but not their pragmatics.  How come
   configuration values can appear in multiple places?  What
   does it mean when they do?

Let's start by getting some preliminaries out of the way
before we move on to discussing the more complex UCE
restriction options.

First, a dress code requirement.  If the smtp client fails
either of the following variables, it's thrown out.

        smtpd_helo_required = yes/no
    strict_rfc821_envelopes = yes/no

The header_checks and body_checks variables define regexp
lookup table maps, which we'll look at later.

    header_checks = maptype:mapname (usually regexp.)
      body_checks = maptype:mapname (usually regexp.)

Maps are lookup tables.  Postfix supports several kinds of
maps (hash and regexp are the most common; see postconf -m),
and uses them for several functions, including alias maps
(which rewrite one address into another), transport maps
(which specify a transport for a given address), and access
maps, which tell postfix whether to permit or deny a given
smtp transaction, and can do other things besides --- things
weird and wonderful, such as recursively calling other
restriction lists, but I'm getting ahead of myself.

The meat of postfix's UCE configurations are found in
parameters that are all named smtpd_*_restrictions.

Access maps are one possible kind of smtpd restriction.
there are others.

Think of a restriction as a function that returns one of OK,
REJECT, or DUNNO.  If you're reminded of router access-lists,
good: you've recognized the pattern.  Some restrictions only
return OK or DUNNO, some return only REJECT or DUNNO, and some
are empowered to return a verdict either way.  If the
restriction doesn't have a strong feeling one way or the
other, it just returns a DUNNO, and postfix goes on to
evaluate the next restriction.  All restrictions may DUNNO
except check_relay_domains, which has to return either OK or
REJECT.  For that reason people generally set it as the last
restriction to be evaluated.

The restrictions are grouped into lists and evaluated in
order within each list.

If the restriction returns either OK or REJECT, the
list short-circuits and immediately concludes with that
value.

If a list concludes in REJECT, the smtp client will be denied.

The default result for all lists is OK.

Here's a symbolic representation:

  listA { r0; r1; r2;                     }
  listB {         r2; r3; r4;             }
  listC { r0;     r2; r3; r4; r5; r6;     }
  listD {         r2;         r5; r6; r7; }

Note that later lists have access to more and more
restrictions:

  listA may run                      r0, r1, and r2.
  listB may run all of listA's, plus r3 and r4.
  listC may run all of listB's, plus r5 and r6.

>>> Why "plus all of the previous list's restrictions"?

  We'll get to that later.  For now just keep in mind that
  each list has certain restrictions that belong to it; in
  addition to those, it can run other restrictions that
  belong to the preceding lists.

>>> So, what are the lists really called?

  listA:  smtpd_client_restrictions
  listB:  smtpd_helo_restrictions
  listC:  smtpd_sender_restrictions
  listD:  smtpd_recipient_restrictions

The order is important.  They are evaluated in that order,
and they "inherit" restrictions in that order.

If all the lists return OK, the client is 354 invited to
DATA the message over, and then after "." completion,
header_checks and body_checks run to give postfix one last
chance to reject the message instead of saying "250 OK:
queued as whatever."

>>> So, what are the functions available to each list?

All lists can run the following generic restrictions:

  permit (default when postfix runs off the end of a list)
  reject
  reject_unauth_pipelining

smtpd_client_restrictions can include all of the above, plus:

  check_client_access maptype:mapname
  permit_mynetworks
  reject_unknown_client
  reject_maps_rbl

smtpd_helo_restrictions can include all of the above, plus:

  check_helo_access maptype:mapname
  reject_invalid_hostname
  reject_unknown_hostname
  permit_naked_ip_address
  reject_non_fqdn_hostname

smtpd_sender_restrictions can include all of the above, plus:

  check_sender_access maptype:mapname
  reject_unknown_sender_domain
  reject_non_fqdn_sender

smtpd_recipient_restrictions can include all of the above, plus:

  check_recipient_access maptype:mapname
  permit_auth_destination
  permit_mx_backup
  reject_non_fqdn_recipient
  reject_unauth_destination
  reject_unknown_recipient_domain
  check_relay_domains (should be placed last, because it doesn't DUNNO)

smtpd_etrn_restrictions is a restriction list which isn't
really targeted at controlling UCE and therefore only gets a
passing mention here.  it can run the helo, client, and
generic sets of restrictions, but not the sender or
recipient restrictions.

Restriction classes are evaluated in the natural order of an
RFC821 transaction: client, helo, sender, recipient.

Within each restriction class, restrictions are evaluated in
the order they are listed.

If none of the restrictions has returned a REJECT, postfix
invites the client to send the data over.  After the data
has been received and the client has sent a ".", postfix
applies header_checks and body_checks.  This is your last
chance to reject the message.  Note: header_checks and
body_checks are only allowed to say REJECT or OK on the
right hand side; "550 foo bar" is not allowed.  (And you
probably wouldn't use OK anyway.)

Note: if you specify maptype:mapname without saying
check_*_access before it, the * is resolved according
to the smtpd_*_restrictions list that it appears in.

Note: while a restriction may result in a rejection early on
in the SMTP exchange, postfix waits until RCPT has been
received to send back a rejection code.  The reason:

20010111 comments by Dr. Liviu Daia <Liviu.Daia@imar.ro>:

  Some brain-dead Windows clients won't take no for an
  answer unless it comes at the RCPT TO stage in the SMTP
  negotiation.  Consequently, they'll blindly go ahead with
  DATA, making Postfix log piles of garbage to syslog.
  That's why Postfix by default postpones the SMTP checks to
  the RCPT TO stage.  You can disable this behaviour by
  setting $smtpd_delay_reject to "no", and the "moving"
  config options knobs will allow you to play with placing
  the checks to various other stages.  However, the default
  seems to work so well in practice that I suspect not many
  people willingly use that feature these days. :-) Wietse
  announced some time ago that the $smtpd_*_restrictions
  will eventually be unified in a single
  $smtpd_uce_restriction statement.

>>> What's the point of running a restriction in a later
    block?  Shouldn't all client-related restrictions belong
    inside the client_restrictions list, and so on?

Let's do an example.  Suppose you're big on spam-defense,
and so you've turned on reject_maps_rbl.  The gauntlet
you've erected looks like this:

client:    { reject_maps_rbl }
helo:      { }
sender:    { }
recipient: { permit_mynetworks, check_relay_domains }

You're happily rejecting any client who's listed in the RBL,
DUL, and RSS provided by MAPS.  Now, MAPS does mention that
some legitimate mail will be sacrificed in the interests of
keeping out more general spam.  For example, at the time of
this writing (200101), American Express's Internet Travel
Network, itn.net, is blacklisted by the RSS, but you don't
want to miss any important mail (smtp sender addresses of
pnr-notification@itn.net).  Taking matters into your own
hands, you explicitly OK that address in your sender-access
map, and you try:

client:    { reject_maps_rbl }
helo:      { }
sender:    { hash:sender-access }
recipient: { permit_mynetworks, check_relay_domains }

But wait!  That doesn't work, because reject_maps_rbl, which
runs before we even know what the sender address is, returns
REJECT, and that's enough for smtpd_client_restrictions to
deny the client.

What you really want is

client:    { }
helo:      { }
sender:    { hash:sender-access, reject_maps_rbl }
recipient: { permit_mynetworks, check_relay_domains }

so that the sender-access gets first crack at explicitly
approving the message before reject_maps_rbl denies it due
to RSS.

20010110 comment from Keith Matthews <keith_m@sweeney.demon.co.uk>:

  reject_maps_rbl checks are only of use in situations where
  you get mail directly from the open relay.  For many dial-up
  customers the fact that their ISP has gathered it
  automatically effectively disables the checks because they
  are carried out against the immediate client - i.e. the
  ISP's mail host !

  To be effective they need to be carried out against earlier
  relays, which is clearly much more work.

>>> Then what's the point of the first three lists?  If
>>> recipient_restrictions are a superset of all the others
>>> and if the order of evaluation is linear and fully
>>> specifiable, why not put everything in the
>>> recipient_restrictions list?

Maybe you want to express complex logic.  You might want to
set up a series of explicit permits followed by possible
denies:

    listA: { permit1; deny1; }
    listB: {    ...   deny2; }
    listC: {    ...   deny3; }

if permit1 returns OK, and you still want to test against
deny2 and deny3, then you need more than one list ---
otherwise, permit1 would short-circuit you past deny2 and
deny3.

    listA: { permit1; deny1; deny2; deny3; } # not what you want

>>> Gotcha --- but how complex can you get with only four
>>> restriction lists?

Ah, but you aren't restricted to only four lists.  The
smtpd_restriction_classes option lets you make up new
restriction lists.

As the smtpd man page says, smtpd_restriction_classes
declares the name of zero or more parameters that contain a
list of UCE restrictions. The names of these parameters can
then be used instead of the restriction lists that they
represent.

    smtpd_restriction_classes = mylist1,
			        mylist2
    mylist1 = check_sender_access foo:bar,
	      reject_non_fqdn_recipient,
	      reject_unknown_sender_domain,
	      check_recipient_access maptype:mapname
	      reject_non_fqdn_sender,
	      etc etc etc.
    mylist2 = permit_auth_destination,
	      permit_mx_backup,
	      reject_unauth_destination,
	      reject_unknown_recipient_domain

>>> Nice.  But how do these lists get called?

The smtpd_*_restriction lists get called naturally as the
SMTP transaction proceeds.  But restrictions lists that you
make up ... well, where's the entry point to those?

It turns out that access maps support a little-known
feature: on the right hand side of the access map you can
define OK, REJECT, and 500 and 400 error codes; this you
already knew.

But did you know that instead of saying OK or REJECT, you
can hop back up a level and define a list of UCE
restrictions?

Read uce.html carefully:
    Reject the request if the result is REJECT or "[45]XX
    text". Permit the request if the result is OK or RELAY
    or all-numerical. Otherwise, treat the result as another
    list of UCE restrictions.

That list is allowed to contain restrictions,
eg. reject_unknown_sender_domain, reject_non_fqdn_sender,
reject_unauth_destination, etc, etc --- but it is NOT
allowed to contain another access map.

That's when you bring out the smtpd_restriction_classes that
you made up.  You can set up my_restrictions to check
another access map:

    smtpd_restriction_classes = my_restrictions
    my_restrictions = reject_non_fqdn_sender,
		      check_sender_access regexp:other_sender_access

You could call my_restrictions directly from a standard
smtpd_*_restrictions list, or you could call from inside an
access map.

    smtpd_recipient_restrictions = check_recipient_access regexp:my_recipient_regexp

    my_recipient_regexp:
	/localuser1/      OK
	/localuser2/      REJECT
	/localuser3/      reject_unknown_sender_domain, reject_non_fqdn_sender
	/localuser4/      reject_unknown_sender_domain, my_restrictions

Mail to localuser4@localdomain gets tested against the
following restriction functions:

   1) check_recipient_access regexp:my_recipient_regexp
   2)    reject_unknown_sender_domain
   3)    my_restrictions:
   4)       reject_non_fqdn_sender
   5)       check_sender_access regexp:other_sender_access
   6)          ... and so on ...

(my_recipient_regexp could have been a hash.  I know.)

		     --- questions ---

>>> Shouldn't reject_unauth_pipelining be a yes/no parameter
>>> rather than a generic restriction?

20010111 comment by Dr. Liviu Daia <Liviu.Daia@imar.ro>:

  Actually, it shouldn't.  The reason for making it generic is
  that you might have some old Windows clients in your local
  network which you know in advance will try early pipelining,
  and you might want to allow them to send mail as well.

			   * * *

>>> Why does ITN.net, presumably a well-funded website with deep
>>> pockets backed by American Express itself still run as an
>>> open relay?

Good question.

			   * * *

>>> Is it worth hurting your users a lot in the short run
>>> just to teach third parties not to hurt other peoples' users
>>> a little in the long run?  (a little = receiving spam that
>>> you didn't want.  a lot = not receiving mail that you did.)

Perhaps it would be better to add a header line to the
message and pass it along for the LDA to shunt to a
low-priority mailbox, or for the MUA to flag as spam.

See http://www.kfki.hu/~kadlec/sw/postfix_patch.html for an
ingenious patch that allows users to decide whether they
want to use each particular MAPS (rbl, rss, dul) or not.

			   * * *

>>> These all look very similar:

  reject_unknown_client
  reject_unknown_hostname
  reject_unknown_sender_domain
  reject_unknown_recipient_domain

Yes, they do.

reject_unknown_client applies to the ip address and hostname of the connecting tcp/ip client.

reject_unknown_hostname applies to the hostname sent in the HELO string.

reject_unknown_sender_domain applies to the hostname part of MAIL FROM: <...@hostname>.

reject_unknown_recipient_domain applies to the hostname part of RCPT TO: <...@hostname>.

If you want to reject mail based on the "From: " and "To: "
headers in the message, use header_checks.

	   ----- appendix: man smtpd_check -----

If you look in src/man, you'll see that the man pages are
generated from the source code using a srctoman | nroff -man
pipeline.  Only some of the .c files, such as smtpd.c, are
converted into man pages; others, such as smtpd_check.c, are
not.  Here's how to view the man page on your own:

postfix-19991231-pl09% mantools/srctoman smtpd/smtpd_check.c | nroff -man | less