Wietse的一些性能优化建议！强烈建议看！！
第2 ��寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: 1.000.000 email posted in a single delivery process
View: Complete Thread （共有 46 条留言）
Original Format
新闻群组：mailing.postfix.users
日期：2001-09-26 14:14:54 PST
If you want to deliver 1000000 queue files in 3 hours, that is 10
milliseconds per file. That seems a bit optimistic to me, considering
the latencies of real disks (every file needs to be read from disk,
the cache is cold) and of real networks (having a local DNS server
should really make a difference).
Neither qmail nor Postfix currently has connection caching, which
can make a difference sending mail to sites that have slow servers.
0 - In the /etc/syslog.conf file, put a - in front of the maillog
filename, so that the stupid syslogd does not hammer the disk
after writing each logfile record.
1 - The default settings of two levels of directory hashing in the
queue should suffice. If you disagree, change this with:
postconf -e hash_queue_depth=3
postfix reload
2 - Configure an adequate number of delivery agents in the master.cf
file: 500+. On your typical Linux box, this will run the kernel
out of resources and Postfix will slam on the brakes until you
fix it.
3 - Test the setup with the smtp-source and smtp-sink programs.
For the test, specify a relayhost name that points to the
smtp-sink program, and send messages with thousands of recipients
into Postfix, and have Postfix perform one-recipient deliveries.
On the Postfix host:
postconf -e relayhost=smtp-sink.host:9999
smtp_destination_recipient_limit=1
On the mail sink host (can be same as postfix host, if you have
enough memory and process slots and space for open files):
./smtp-sink -c :9999 1000
On the smtp-source host (can be same as postfix host):
./smtp-source -c -m 1000 -s 10 -r 10000 -t foo@postfix.host postfix.host
Postfix breaks up the deliveries into transactions of 1 recipient
each, so one message with 10000 recipients will keep lots of
delivery agents busy.
4 - Before submitting the real mail:
postconf -e defer_transports=smtp relayhost=
smtp_destination_recipient_limit=$default_destination_recipient_limit
postfix reload
(and don't let anyone execute a "sendmail -q" command).
5 - SUBMIT THE MAIL VIA SMTP, not via /usr/sbin/sendmail. Otherwise
you are throwing away all the performance.
6 - On Doomsday,
postconf -e defer_transports=
postfix reload
sendmail -q
7 - From here on, do not touch the box. Each time you stop or reload
Postfix, or do a sendmail -q, it slows down dramatically.
Wietse
Alexander:
> (I've posted this message in QMAIL and EXIM mailing list.
> Many people say me that POSTFIX can do a good work)
>
> This is a case study and it's NOT for create a spamming system or other
> terrible mail bombing system.
> I HATE them !
>
> Title: the best mode to deliver 1.000.000 email messages (recipient are ALL
> different and email are ALL different) in few hours during a single delivery
> session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512 MB Ram,
> SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet connection (4
> Mbit/sec full bandwidth) and a fast DNS server near me.
>
> During the night a program running on the same server creates 1 million of
> email (getting user data and infos data from a SQL DB).
> This process need about 3 hours. Meanwhile email are created, they are
> "injected" in MTA queue (i.e. using qmail-inject or
> exim -odq -t) and they are NOT delivered: they stay in queue.
> The email are all different but short: about 800-1000 bytes each and the
> recipients are all different and are all remote.
> Yes, we can have a vary big number of different recipient domain but
> statistically we also have a large number of email directed on mass email
> system (Hotmail, Yahoo...)
>
> So 1.000.000 x 1.000 bytes = 1 GBytes data. Yes, like a technical
> information email system with personalized info for each recipient.
>
> Then every day at 8.00 AM the remote delivery needs to start (exim -q ,
> qmail-remote) and delivery is done once per day.
>
> Target: send 1 million of email in a very short time, shortest as possibile,
> it must end as soon as possibile. Bandwidth occupation is not a problem: we
> can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec = about 2000 seconds , 1
> hour and few minutes. 2 or 3 hours are ok anyway).
>
> And now the questions:
>
> 1) QMAIL vs EXIM vs POSTFIX: which is the best MTA to manage 1.000.000 email
> in queue ?
> (EXIM with option "split_spool_directory" and QMAIL patched to have a more
> "deep" in queue tree, POSTIFIX ??)
>
> 2) EXIM can deliver all messages during a single SMTP session to the same
> email system (useful for mass system like HOTMAIL,YAHOO, ....), QMAIL no.
> Is it faster to have either for example 300 (more or less) qmail-remote
> process running and sending 1 messages each or for example 20 (more or
> less) "exim -q" process running sending more messages in a single SMTP
> connection ? Remember: the target is to use less time than possible.
>
> Thanks for any suggestion about.
> POSTFIX ,I've never used it, can be an alternative ?
>
>
>
[quote]【[b]在 hzqbbc 发表的大作中提到：[/b]】
：第2 ��
：寄件者：Wietse Venema (wietse@porcupine.org)
：主旨：Re: 1.000.000 email posted in a single delivery process
：View: Complete Thread （共有 46 条留言）
[/quote]
另外一个人的性能调整建议，不过是产品化的建议。不错！虽然有些我不同意。
寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:08 PST
On 27 September 2001, Liviu Daia <Liviu.Daia@imar.ro> wrote:
[...]
> For a today's PC, something like 40-50 messages / second would still
> be an optimistic estimate;
That is, over a LAN. Expect a much lower rate over Intermet,
because of dead or slow destinations.
> that's about 6 times slower that what you want.
Regards,
Liviu Daia
--
Dr. Liviu Daia e-mail: Liviu.Daia@imar.ro
Institute of Mathematics web page: http://www.imar.ro/~daia
of the Romanian Academy PGP key: http://www.imar.ro/~daia/daia.asc
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
第 9 条留言
寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:20 PST
On 26 September 2001, Alexander <ml@ambrosa.it> wrote:
[...]
> Title: the best mode to deliver 1.000.000 email messages (recipient
> are ALL different and email are ALL different) in few hours during a
> single delivery session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512
> MB Ram, SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet
> connection (4 Mbit/sec full bandwidth) and a fast DNS server near me.
Wietse already explained how to setup Postfix for best performance.
Here's an additional checklist, mainly for the rest of the system:
- if possible, use OpenBSD or FreeBSD with softupdates rather than
Linux;
- if you insist on using Linux, use a 2.2.x kernel, not a 2.4.x one, and
turn OFF the swap; buy more RAM if you feel you're running out of it;
- use RAID 0+1, not 5, and (if you have that choice) mirror the stripes
instead of striping the mirrors;
- use ext2; Xfs and Jfs are not yet ready for prime time, and everything
else is slower than ext2 for mail handling;
- install a caching DNS on the SAME machine as the mail server, a nearby
one is not good enough; use djbdns for that instead of bind; turn off
logging for lookups, and give it plenty of memory to keep the cache;
- if you trust your UPS, turn off sync writes to the queue
("chattr -R -S /var/spool/postfix")
- if possible, send the logs to another machine instead of writing them
to a local file, or at least put the logs on different disk spindles
than the queue;
- apply Michael Tokarev's patch for CDB maps, and use CDB maps instead
of hash.
(I'm sure I forgot a few things too.)
[...]
> Target: send 1 million of email in a very short time, shortest as
> possibile, it must end as soon as possibile. Bandwidth occupation is
> not a problem: we can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec
> = about 2000 seconds , 1 hour and few minutes. 2 or 3 hours are ok
> anyway).
[...]
As Wietse said, that's overly optimistic. With careful tuning,
Postfix would perform better than Exim and Qmail, but IMHO, no MTA would
be able to send 1 million messages in an hour, even over a LAN. No
offense intended, but people who set themselves such a target have never
managed a mail systems in the real world. For a today's PC, something
like 40-50 messages / second would still be an optimistic estimate;
that's about 6 times slower that what you want.
Regards,
Liviu Daia
[quote]【[b]在 hzqbbc 发表的大作中提到：[/b]】
：第2 ��
：寄件者：Wietse Venema (wietse@porcupine.org)
：主旨：Re: 1.000.000 email posted in a single delivery process
：View: Complete Thread （共有 46 条留言）
[/quote]
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 06:45:59 PST
Matthias Andree:
> Well, in a LAN, it might actually be possible to get half a million
> mails through per hour without extensive tuning - however, that's
> without VERP and it assumes your destination machines are *FAST*.
To give you a data point: running smtp-source against smtp-sink
on a 750MHz pentium FreeBSD 4.3 machine without going over a LAN,
with 100 parallel connections:
tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 tail:9999
1.21 real 0.06 user 0.50 sys
That's 1 million messages in 20 minutes.
Same email load, two hosts connected by a 100BaseT ethernet switch,
target is P550 FreeBSD 4.1, with 100 parallel connections:
tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 spike:9999
1.05 real 0.06 user 0.23 sys
The times are pretty much comparable.
Do not expect anything near this performance with a queue on rotating
disk drives, though.
Wietse
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
:victory: 先收下了，研究研究
页:
[1]