Tags

Release 1.1.0

* Incompatible change: sqlite3 and per_user behaviour:
	Now both redis and sqlite3 follows the common principles for per-user
	statistics:
	1) If per-user statistics is enabled check per-user tokens ONLY
	2) If per-user statistics is not enabled then check common tokens ONLY
	If you need old behaviour, then you'd need to use separate classifier
	for per-user statistics.
* Implement redis statistics backend and cache
* Implement autolearning for statistics
* Reworked statistics architecture from scratch
* Add hyperscan (https://github.com/01org/hyperscan) engine for regular
	expressions:
	- add lazy loader for hyperscan databases
	- rework regexp cache to have joint pcre/hyperscan scanning
	- implement hyperscan pre-filter support
	- add compilation guards for bad expressions
	- implement `rspamadm control recompile` command
	- implement hyperscan cache monitoring
	- slides: <https://highsecure.ru/rspamd-hyperscan.pdf>
* Implement flexible task logging
* Rework fuzzy worker:
	- it is now possible to run multiple fuzzy workers;
	- implement lazy writing as sqlite3 is bad at concurrent writing;
	- add retries for simple sql commands in fuzzy backend;
	- use fine-grained transactions for fuzzy;
	- implement new multi-pubkeys mode;
	- allow encrypted only storages;
	- rework statistics for fuzzy;
	- add `rspamadm control fuzzystat` command for extended statistics;
	- implement human readable output for the previous command;
	- add condition script for learning fuzzy storage;
* Various fixes to SPF:
	- fix `redirect` records;
	- fix domains when parsing mx/ptr/a records in includes/redirects;
	- fix issues with multiple addresses in SPF records;
	- ignore SPF results in case of DNS failure;
	- adjust TTL of records when resolving subelements of SPF records;
	- always select `v=spf1` line if it is available
	- do not cache records with DNS failure in subrequests;
	- ignore records with temporary fails during subrequests resolving;
	- fix `RDNS_RC_NOREC` support;
* Add clang plugin for static analysis:
	- implement static checks for `rspamd_printf` format strings;
* Add 'allow_raw_input' option for non-mime messages
* Recognize types using libmagic
* Fix parsing of IPv6 received headers
* Add new interface of communication between workers in rspamd
* Add support for named socketpairs
* Don't write URLs by default as it is too verbose
* Set status for HTTP replies
* Try load `rspamd.conf.override`
* Implement words decaying for text parts to limit many checks
* Improve support of SA rules and plugins:
	- add check_for_shifted_date and check_for_missing_to_header eval rules;
	- add 'check_relays_unparseable' support;
	- add `check_for_mime('mime_attachement')` function;
	- use new re_cache interface for all SA rules;
	- add support for `Mail::SpamAssassin::Plugin::MIMEHeader`;
	- add support of 'special' SA headers to `exists` function;
	- fix issue when SA metas contain other metas;
	- fix freemail rules;
* Many fixes to the URL parser
* Match any newline character in regexps
* Fix resolving of upstreams and detection of poor IPv6 configurations
* Parse upstreams selection algorithm from the configuration line
* Add `reresolve` command to the control interface
* Generate fuzzy hashes from task metadata (URLs and headers)
* Add method to check if IP is local and `local_addrs` option
* Implement forced timeout for delayed filters
* Disable fast path of pcre-jit as it seems to be broken
* Bayes fixes:
	- new normalizer function;
	- really use weights of tokens from the OSB algorithm;
	- restore multiple classifiers support;
* Rules changes:
	- add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's;
	- improve empty image rule;
	- rework `FORGED_RECIPIENTS` rule;
	- reduce weight of `SUSPICIOUS_RECIPS`;
	- fix `*_NORESOLVE_MX` symbols in hfilter;
	- add `SUBJ_ALL_CAPS` rule with support of UTF8
	- add spamhaus SBL to uribl
	- fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules
	- remove `R_TO_SEEMS_AUTO` as it generates a lot of FP;
	- add new Message-ID regexp for Thunderbird (by @moisseev);
* Plugins changes:
	- allow ratelimit plugin to set symbol instead of pre-result
	- support IP DNS black lists for URIBL (e.g spamhaus SBL);
	- drop deprecated SURBL bits (by @fatalbanana)
	- rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana)
	- add `SURBL_BLOCKED` (by @fatalbanana)
	- add `CR_SURBL`
	- SURBL: allow fallthrough to default symbol (by @fatalbanana)
	- Settings: fix IP match (by @fatalbanana)
	- SURBL: add missing symbols to metric (by @fatalbanana)
	- allow processing images urls for SURBL
	- unconditionally disable SPF for authenticated users and local networks
* Rework ratelimit plugin
	- switch to `rates` instead of old and stupid strings to setup;
	- check if a bucket is zero and disable the corresponding limits'
	- turn off all buckets by default;
	- check either `rcpt` or `user` buckets, not all together'
	- document new `rates` and `symbol` options;
	- inform user about what buckets are used in the configuration;
* Add neural network **experimental** plugin
* Add a sample script to learn neural network from rspamd logs
* Add documentation strings support to rspamd:
	- add strings for the main configuration options;
	- document workers options;
	- add internal plugin options;
	- create `rspamadm confighelp` routine;
	- implement human readable output for the previous command;
	- add subtree search support;
	- add keyword search support;
* Documentation improvements, tutorials section, statistics description
* Many other minor and major bugfixes not noted here

Release 1.0.11

* Fix spf redirects
* Fix domains when parsing mx/ptr/a records in includes/redirects
* Fix unfolded base64 encoding
* Fix GError use-after-free
* Do not rewrite the original url when using redirector
* Fix parsing of fragment in urls
* Fix processing of HTML tags
* Improve empty image rule
* Avoid long double type
* Fix tokens weights in OSB algorithm
* Improve debugging for bayes

Release 1.0.10

* Fix settings application (#416)
* Fix another issue with fixed strings
* Fix hash function invocation
* Use the proper string for make_dns_request in lua_http
* Fix scan time output
* Update webui:
	- fix labels for greylisting
	- fix dimension of scan time

Release 1.0.9

* Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password
* Support passwords longer than 64 symbols

Release 1.0.8

* Add function to traverse AST atoms
* Allow dependencies on rspamd symbols for SA metas
* Fix memory corruption when timeout is removed in fuzzy check
* Fix encrypted fuzzy add processing
* Avoid use-after-free in controller session destructor
* Use session pool instead of task pool in fuzzy check
* Fix assembly in i386 mode (#413, #412)

Release 1.0.7

* Plugged memory leaks in internet address object & html parser
* Fixed static build
* Fixed multiple sigchld processing
* Fixed deletion of signal events after event processing loop
* Fixed build on ARM (#404 - reported by @Gottox)
* Fixed setting the default mask for SPF.
* Fixed sanitisation of HTTP query values
* Fixed parsing of the last header in encrypted HTTP messages
* Additions and fixes for test suite & benchmarks
* Added openssl aes-256-gcm support to libcryptobox & HTTP server
* Implemented support for starting multiple HTTP servers
* Implemented batch accept in HTTP server
* Added module to get data from HTTP headers (#285 - reported by @msimerson)
* Added `rspamadm control` command
* Added ability to sort counters output.
* Added ability to specify custom headers for rspamc client
* Fix architecture detection
* Converted history storage to the UCL format
* Allow flexible number of rows in history
* Fix action badges in WebUI
* Add universal cryptobox hash API
* Migrated to the optimized blake2b implementation adopted from Andrew Moon
* Allow explicit loading of specific modules
* Always load settings module
* Allow to add symbols from settings
* Fix double free in the controller fuzzy learn command
* Avoid endless loop when cannot open sqlite db
* Updated libucl

Release 1.0.6

* Fix build on i386
* Update CentOS7 service file patch (by @fatalbanana)
* Fix path to rspamadm in Debian init script (by @fatalbanana)
* Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD
* Fix portability issues
* Use cryptobox chacha for libottery
* Better support of 32 bit builds
* Fix header name tokens setup
* Fix levenstein distance method for words
* Add workaround for old libevent (#400)
* Fix microseconds in termination timer
* Fix some more issues with fixed strings
* Explicitly test CPU instructions even after CPUID call
* Do not check out of boundary memory
* Do not output broken emails
* Fix unknown symbols registration
* Handle SIGILL using longjmp
* Block signals when exiting event loop
* Fix incorrect allocation size
* Slightly optimize alignment
* Restore rspamd -t for compatibility
* Add more sanity checks for emails

Release 1.0.5

* Add rspamd control interface:
    - support `stat` command to get runtime stats of rspamd workers
    - support `reload` command to reload runtime elements (e.g. sqlite3 databases)
* Rework curve25519 library for modular design:
    - add Sandy2x implementation by Tung Chou
    - fix CPU detection for variables loading assembly
    - add testing for curve25519 ECDH
* New fixed strings library
* Add `R_SUSPICIOUS_IMAGES` rule
* Enable mmap in sqlite3
* Use new strings in the HTTP code
* Improve google perftools invocation
* Improve performance profiling in http test
* Reorganize includes to reduce namespace pollution
* Allow specific sections printing in configdump command
* Rework workers signals handlers to be chained if needed
* Update socketpair utility function
* Add control_path option for rspamd control protocol
* Fix ownership when listening on UNIX sockets
* Rework signals processing in main
* Remove extra tools from rspamd (they live in rspamadm now)
* Remove global rspamd_main
* Add global timeout for the overall task processing (8 seconds by default)
* Sanitize NULL values for fuzzy backend
* Store NM between encrypt/decrypt
* Add textpart:get_words_count method
* Fix generic DNS request in lua
* Tune hfilter weights
* Add support of IPv6 in hfilter
* Fix parsing of HTTP headers with IP addresses
* Sync with the recent libucl
* Various minor bugfixes

Release 1.0.4

* Add configdump routine to rspamadm
* Implement retransmits for fuzzy_check plugin
* Fix events processing for learning anf checking fuzzy hashes
* Avoid dependency on unneeded and uncompatible glib include
* Add `historyreset` command to the controller
* Fix loading of tokenizer config from dump (#389)
* Add sorting hints for the history
* Allow custom lua scripts for users/languages extraction (#388)
* Do not add FORGED_RECIPIENTS when 'To' is missing (#387)
* Do not add R_UNDISC_RCPT when 'To' is missing (#387)
* Add encryption to fuzzy check plugin
* Add encryption for fuzzy storage
* Add new epoch for encrypted fuzzy request
* Add encryption for `rspamd.com` storage
* Remove gmime processing for LDA mode as it is deadly broken
* Add routine to find end of headers position in mime messages
* Fix LDA headers folding
* Init libraries in rspamc client as well to avoid locale issues
* Avoid collision with locally installed includes
* Allocate and free memory with the same allocator in rspamadm (#385)
* Preserve expired fuzzy hashes counter
* Improvements in webui:
    - Add favicon.ico
    - Rework history table
    - Fix sorting for the history
    - Migrate to bootstrap 3 and jquery 2
    - Fix css bugs
    - Add glyphicons
    - Add reset history
    - Improve history buttons
    - Redraw graph to avoid display issues
    - Webui is now MIT licensed to match licensing policy of rspamd

Release 1.0.3

* Fix piechart clean slice (#380)
* Fix controller crashes when GString is reallocated (#381)
* Correctly set locale before start
* Set C locale for numeric values
* Add rspamadm routine:
    - add `pw` command to manage passwords
    - add `help` command for displaying help
    - add `configtest` command to check configuration files
    - add `keypair` command for generating encryption keys
    - add `fuzzy_merge` routine to merge fuzzy sqlite databases
    - add a simple manual page for rspamadm
* Allow metric registration for composite expressions
* Add strict mode for configtest
* Add logger counters
* Save and show learned messages count (#383)
* Add `no_stat` flag
* Add `task:set_flag` and `task:get_flags` (#382)
* Enable foreign keys in sqlite3
* Remove orphaned shingles from fuzzy storage
* Optimize synchronization steps for fuzzy storage
* Allow delayed conditions registration
* Add lua API for conditions registering

Release 1.0.2

* Fix critical bug in webui that prevents password from being sent
* Rework webui view:
    - Switch to d3.js for graphs
    - Improve piechart look
    - Rework colors for piechart
    - Fix layout for symbols
    - Fix refresh button
* Add descriptions for whitelist maps
* Fix build on arm (#379)
* Fix issue with the last element in the radix trie
* Add more tests for radix trie algorithm
* Allow to extract URLs from query strings of other URLs (#361)
* Initialize rrd fields before writing to file
* Fix double free if no password has been specified

Release 1.0.1

* Add writing to rrd from the controller
* Fixed lots of bugs in rrd code
* Adopt new DNS API in hfilter plugin (by @AlexeySa)
* Allow only one controller process to manage rrd file
* Set event base for fuzzy calls
* Improve fuzzy IO errors logging
* Add rra extraction function to rrd library
* Add graph handler to the controller
* Cache correct passwords to avoid too high CPU usage when working with webui
* Controller sockets are owned by router do not export them to task
* Optimize logging by skipping hash table search if it's empty
* Fix loading issue whith broken statfiles
* Print assertions from glib to rspamd logger
* Load legacy `lua/rspamd.local.lua`
* Update webui with some fixes to learning and scanning

Release 1.0.0.

* Rework symbols processing:
	- Improve sorting logic for symbols
	- Organize processing into multiple stages
	- Added asynchronous watchers for symbols
	- Added ability to organize dependencies between symbols
* Fixed URL redirector:
	- Use optimized POE loop
	- Organize dependencies
	- Fix startup
* New sqlite3 backend:
	- Allow to have per-languages and per-user statistics
	- Allow sqlite3 to be used as statistics backend
* Store tokenizer configuration within statfiles
* Improve bayes statistics:
	- Use headers and images metainformation in bayes
	- Suggest using of pre-processed tokens for statistics
	- Fix tokens normalization for OSB algorithm
* Rewrite url parsing:
	- Fix numerous issues with url extraction and normalization
	- Fix mailto urls
* Fix settings plugin to allow custom actions scores
* Improve rbl plugin
* Allow capturing patterns in rspamd lua regexp library
* Add GTUBE support
* Fix spamc legacy support
* Add DKIM support to RBL module
* Fix issues with multiple DKIM signatures
* Fix issue if rspamd cannot create statfiles (#331)
* Rework parts and task structure:
	- Now text_parts, parts and received are arrays
	- Pre-allocate arrays with some reasonable defaults
	- Use arrays instead of lists in plugins and checks
	- Remove unused fields from task structure
	- Rework mime_foreach callback function
	- Remove deprecated scan_milliseconds field
* Add ip_score plugin support (not enabled by default):
	- Can check for asn/country and network using DNS lookups
	- Can store and load reputation from redis server
* Improve PARTS_DIFFER rule to count merely different words
* New HTML parser:
	- Parses HTML parts using a set of state machines
	- Extracts useful data and exports it to lua functions:
		+ Styles
		+ Images
		+ URLs
		+ Colors
		+ Structure elements
	- Added HTML rules for some checks
* New version of LUA DNS API
* Table versions of many functions in LUA API
* Improve rspamc client:
	- Print execution time
	- Allow executing of external commands and passing output to them
	- Allow mime output mode when rspamc alters message according to rspamd
		checks and send it to an external command or stdout
* Allow scanning of local files using HTTP requests
* Rework configuration system:
	- Rules are now moved from the $CONFDIR to $RULESDIR to avoid ambiguity
	- All modules configurations are now split in $CONFDIR/modules.d/* to
		simplify upgrades
	- Move hfilter to plugins
	- Allow plugins and rules to define default scores to simplify metrics
		setup
	- Include overrides for all modules to honor local/automatic parameters
	- Tune scores for many modules
* Rework and enable DMARC plugin
* Add whitelist plugin for SPF/DKIM/DMARC based whitelisting
* Add some common domains to whitelists shipped with rspamd
* Rework logging:
	- Now each log entry supports module name and a `tag`. Tag is used to
		identify unique objects (such as tasks) when checking log files
	- It is possible to turn on debugging for the specific modules
	- Systemd logging is fixed
* Improve spamassassin plugin.
	- Now headers are matched more like SA
	- Improve support of Message-ID
	- Add support of ToCc header type
	- Fix :addr and :name in headers regexps
* Resurrect rrd support code
* Save controller stats between restarts
* Fixed tonns of bugs
* Added tonns of minor improvements and features
* Added more unit tests
* Create functional tests framework
* Added documentation for missing modules
* Added rpm/deb repositories and scripts
* Updated WebUI and libucl externals

Release 0.9.10:

* Do not dereference null pointer on learning.
* Fix some extreme cases in BAYES.
* Add a workaround to avoid bad HTML messages breaking.
* Build with -O2 flags by default.
* Add constraints to limit DNS requests count per task.
* Add workaround for SURBL DNS flood.
* Set error if rspamd cannot learn anything.