Repository navigation
feat: optionally look up disposable domains without building a Set of them - #326
Conversation
2c6cef4 to
9faaf2c
Compare
|
this is a tradeoff that we should not decide on. if you would want to have a smaller footprint and be fine with slower lookup time, you should have an easy way to configure it. so it would be ok for me if you add the additional bin format class and have an option to turn it on instead of using the set. in addition, i would want to have a spec that makes sure every entry is found in both implementations, such as |
|
Makes sense, thanks! The sorted list is now opt-in, and the default is unchanged (the Set): ValidEmail2.disposable_domains_lookup = :sorted_list
|
ValidEmail2.disposable_emails loads the ~180,000 disposable domains into a Set: one String object per domain plus its hash entry, about 35 MB of memory once the first email is checked. The file is sorted (see pull_mailchecker_emails.rb), so lookups now go through ValidEmail2.disposable_domains, a SortedDomainList keeping the file as a single String (2.6 MB) searched by bisection. disposable_emails still returns the same Set, built on demand. Once it is loaded, disposable_domains returns it, so code adding domains to it or removing them keeps working.
Lookups use the Set of disposable_emails by default again. Setting ValidEmail2.disposable_domains_lookup = :sorted_list switches them to the SortedDomainList, trading lookup speed for memory.
bb450d8 to
b854149
Compare
phoet
left a comment
There was a problem hiding this comment.
Just small changes, works otherwise.
As suggested in review.
|
@phoet changes done! |
|
@Spone could you run main in production with your lookup on and report back before I craft a new version? |
|
Yes of course! I'll do that and report back |
|
Hi @phoet I've been running the |
|
👍 |
|
@phoet can you push |
|
a new version should have been released, but nothing showed up on rubygems. have to investigate what went wrong. https://github.com/micke/valid_email2/actions/runs/43881741055/job/133349106539 |
|
wrestled it to do something |
Problem
ValidEmail2.disposable_emailsloads the ~180,000 disposable domains into aSet. Each domain becomes its ownStringplus a hash entry, so the first email checked grows the process by about 35 MB.Change
The domains file is sorted and has no duplicates:
pull_mailchecker_emails.rbwrites it with.uniq.sort. So it can be searched without building aSet, as an opt-in trade of lookup speed for memory:ValidEmail2::SortedDomainListkeeps the file as a single binaryString(about 2.6 MB) and answersinclude?by bisection over its lines. Each step reads one line around the middle offset, so there's no index to build.ValidEmail2.disposable_domainsis whatAddress#disposable_domain?and#disposable_mx_server?now use::set(the default), theSetofdisposable_emails, as before;:sorted_list, theSortedDomainList.Any other value raises an
ArgumentError.disposable_emailsstill returns the same mutableSet, built on demand. With:sorted_list, domains added to it or removed from it are ignored by the validations; the README says so.Measured
Against
config/disposable_email_domains.txt(178,700 domains). Memory is the process's RSS growth after loading and one lookup, each case in a fresh process::set:sorted_listA sorted list lookup is slower than a
Set's but stays well under the existingperform_under(0.0001)benchmark (100 µs).Specs
spec/sorted_domain_list_spec.rb:spec/valid_email2_spec.rb::setis the default, and an unknown lookup is rejected;:sorted_list, a disposable domain is detected without loading theSet;:sorted_list, every entry of theSetis found in the sorted list.spec/benchmark_spec.rb: a lookup benchmark with:sorted_list.bundle exec rake: 148 examples, 0 failures. RuboCop reports the same 2 offenses as onmain(Metrics/ClassLengthonAddress,Metrics/BlockLengthon the main spec).