Delphi EurekaLog and OmniThreadLibrary incompatible?

Question

I have a simple program that creates an OmniThread workerpool in the 'initialization' of a unit and destroys the same pool in the 'finalization' of that unit. This works fine as long as we do not use EurekaLog. If we include EurekaLog an access violation is raised during finalization of the application (after closing the app). This happens only once every 3 to 10 times the program ends; so it seems some sort of timing issue.

It seems that it all works fine, if we create the workerpool in the "normal" application flow (not in the initialization and finalization of a seperate unit).

The code of this unit is as follows:

unit Unit1;

interface

uses
  OtlParallel;

var
  _Worker: IOmniBackgroundWorker;

implementation

initialization
  _Worker := Parallel.BackgroundWorker.NumTasks(10)
    .Execute(
      procedure(const AWorkItem: IOmniWorkItem)
      begin
        //
      end
    );

finalization
  _Worker.Terminate(INFINITE);
  _Worker := nil;

end.

The main application is also simple:

uses
  {$IFDEF EurekaLog}
  EMemLeaks,
  EResLeaks,
  EDebugExports,
  EDebugJCL,
  EFixSafeCallException,
  EMapWin32,
  EAppVCL,
  EDialogWinAPIMSClassic,
  EDialogWinAPIEurekaLogDetailed,
  EDialogWinAPIStepsToReproduce,
  ExceptionLog7,
  {$ENDIF EurekaLog}
  Vcl.Forms,
  Unit1 in 'Unit1.pas';

{$R *.res}

begin
  Application.Initialize;
  Application.MainFormOnTaskbar := True;
  Application.Run;
end.

The callstack on the access violation is:

:5653e4c4 
OtlTaskControl.TOmniTask.Execute
OtlThreadPool.TOTPWorkerThread.ExecuteWorkItem($393A160)
OtlThreadPool.TOTPWorkerThread.Execute
System.Classes.ThreadProc($3976800)
EThreadsManager.NakedBeginThreadWrapper(???)
:76ee6359 KERNEL32.BaseThreadInitThunk + 0x19
:77628944 ntdll.RtlGetAppContainerNamedObjectPath + 0xe4
:77628914 ntdll.RtlGetAppContainerNamedObjectPath + 0xb4

I use the latest master checkout of OTL and EurekaLog version 7.9.1.4 update 1 hot-fix 4.

Is the way we create and destroy the workerpool correct? And, if so, is anybody familiar with issues in OTL/EurekaLog if they are used together?

Answer 1

It appears to be a flaw / not implemented in OTL:

destructor TOmniTaskControl.Destroy;
begin
  // TODO 1 -oPrimoz Gabrijelcic : ! if we are being scheduled, the thread pool must be notified that we are dying !

It looks like this code can have a task sheduled in the thread pool, which is still running.

Both TOmniTaskControl and TOmniTask share the same TOmniSharedTaskInfo instance. If TOmniTaskControl is deleted from finalization of OtlTaskControl, then any TOmniTask in still running thread pool will contain reference to already deleted TOmniSharedTaskInfo.

It is "not a problem" in app without EurekaLog, since memory of deleted object will be unchanged. Thus, access to already deleted object will be successful. However, adding EurekaLog means erasing disposed memory with a debug pattern. Thus, accessing already deleted object may fail.

Specifically, the code crashes when TOmniTask tries to access already deleted TOmniSharedTaskInfo:

procedure TOmniTask.InternalExecute(calledFromTerminate: boolean);
// ...
begin
  otCleanupLock.EnterWriteLock;
// ...
      finally
        otCleanupLock.EnterWriteLock;
        if assigned(otSharedInfo_ref.ChainTo) and
           (otSharedInfo_ref.ChainIgnoreErrors or (otExecutor_ref.ExitCode = EXIT_OK))
        then
          chainTo := otSharedInfo_ref.ChainTo; // - fails here inside System.@IntfCopy
        otSharedInfo_ref.ChainTo := nil;
// ...

The inaccessible :5653e4c4 location is just an address, which System.@IntfCopy tries to read when threating trashed fields from otSharedInfo_ref.

You can confirm that it is a "use after free" issue by disabling "Catch memory problems" option in EurekaLog (which overwrites object/interface VMT on disposal) and setting "When memory is released" option to "Do nothing". If you do that - the mentioned crash accessing already deleted otSharedInfo_ref / TOmniSharedTaskInfo inside TOmniTask.InternalExecute will no longer occur (indeed - because now the deleted otSharedInfo_ref will be unchanged, thus will be accessible).

Additionally, a new memory leak will now be found instead:

OtlSync.pas TOmniCriticalSection.Create
OtlSync.pas CreateOmniCriticalSection
OtlSync.pas TOmniCS.Initialize
OtlTaskControl.pas TOmniCS.Acquire
OtlTaskControl.pas TOmniTask.InternalExecute
OtlTaskControl.pas TOmniTask.Execute
OtlThreadPool.pas TOTPWorkerThread.ExecuteWorkItem
OtlThreadPool.pas TOTPWorkerThread.Execute
System.Classes.pas ThreadProc

This leak occurs at otSharedInfo_ref.MonitorLock.Acquire inside TOmniTask.InternalExecute. This happens near previosly mentioned chainTo := otSharedInfo_ref.ChainTo.

It is just another confirmation of "use after free" bug. That is because normally ostiMonitorLock: TOmniCriticalSection field of TOmniSharedTaskInfo would be finalized when the otSharedInfo_ref: TOmniSharedTaskInfo field is deleted as part of TOmniTaskControl's destruction.

However, if there is "use after free" bug - then the TOmniTaskControl is being deleted while there is TOmniTask alive still holding reference to the same TOmniSharedTaskInfo. This means that attempt to otSharedInfo_ref.MonitorLock.Acquire will re-initialize ostiMonitorLock: TOmniCriticalSection with a brand new critical section. And since the TOmniTaskControl and its TOmniSharedTaskInfo is already gone - there will be no code to cleanup this newly created critical section. Indeed, you just created a new critical section as a field inside already deleted object (TOmniSharedTaskInfo)!