I intend to send emails when a spider finnishes scraping. I have a MyStatsCollector class (placed in the root of the project) with a code like this:
class MyStatsCollector(StatsCollector):
def _persist_stats(self, stats, spider):
...
mailer = MailSender()
mailer.send()
The code is functional, it sends out the email when spider is done.
However, the code is also return an error:
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.10/site-packages/twisted/python/log.py", line 96, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/python/log.py", line 80, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- <exception caught here> ---
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/asyncioreactor.py", line 138, in _readOrWrite
why = method()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/tcp.py", line 248, in doRead
return self._dataReceived(data)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 329, in dataReceived
self._flushReceiveBIO()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 300, in _flushReceiveBIO
self._flushSendBIO()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 253, in _flushSendBIO
bytes = self._tlsConnection.bio_read(2 ** 15)
builtins.AttributeError: 'NoneType' object has no attribute 'bio_read'
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/asyncioreactor.py", line 138, in _readOrWrite
why = method()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/tcp.py", line 248, in doRead
return self._dataReceived(data)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 329, in dataReceived
self._flushReceiveBIO()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 300, in _flushReceiveBIO
self._flushSendBIO()
File "/opt/homebrew/lib/python3.10/site-packages/twisted/protocols/tls.py", line 253, in _flushSendBIO
bytes = self._tlsConnection.bio_read(2 ** 15)
AttributeError: 'NoneType' object has no attribute 'bio_read'
After some googling, I found this issue on Github where it is advised to implement sending emails with the async/await mechanism.
How can one do that? What parts do need to be modified?