What is the fastest\best way to get file names from a folder using PowerShell?

Question

The directory has 20k folders in it. In these folders there are subfolders and some files. I don't need to look into the subfolders. I need to get all the files with .EIA extension from the folders.

I know I could use Get-Item, Get-ChildItem for this but these cmdlet are too slow in the getting the data. Also, this script has to run every hour therefore, it cannot be taking superlong.

I was trying to use [System.IO.File]::GetFiles($path) but this gives an error

 Method invocation failed because [System.IO.File] does not contain a method named 'GetFile'

I have also tried

$pathEia = "\\Sidney2\MfgLib\AidLibTest\*\*.EIA"
 [System.IO.File]::GetFiles($pathEia)

This also throws an error:

 Exception calling "GetFiles" with "1" argument(s): "The filename, directory name, or volume label
     | syntax is incorrect. : '\\Sidney2\MfgLib\AidLibTest\*\*.EIA'"

I am using PowerShell Core 7.2 .Net Framework 4.8 Any help is appreciated. Thanks in advance.

Answer 1

Very similar to mklement0's helpful answer but using the instance methods from DirectoryInfo.

EnumerationOptions is available starting from .NET Core 2.1. This class has the property IgnoreInaccessible set to $true by default, in prior versions an exception would cause the enumeration to Stop:

...skip files or directories when access is denied (for example, UnauthorizedAccessException or SecurityException).

This answer requires PowerShell Core 7+.

# Skip the following Attributes:
#   2.    Hidden
#   4.    System
#   1024. ReparsePoint
#   512.  SparseFile

$enum = [IO.EnumerationOptions]@{
    RecurseSubdirectories = $false # Set to `$true` if you need to do a recursive search
    AttributesToSkip      = 2, 4, 1024, 512
}

$start  = [IO.DirectoryInfo]::new('\\Sidney2\MfgLib\AidLibTest')
$result = foreach($dir in $start.EnumerateDirectories()) {
    $dir.GetFiles('*.EIA', $using:enum)
}
$result | Format-Table

---

If you need to do a recursive search on the subfolders (if RecurseSubdirectories = $true), you can consider using multi-threading with ForEach-Object -Parallel.

$start  = [IO.DirectoryInfo]::new('\\Sidney2\MfgLib\AidLibTest')
$result = $start.EnumerateDirectories() | ForEach-Object -Parallel {
    $_.GetFiles('*.EIA', $using:enum)
}
$result | Format-Table

It's important to note that, using a parallel loop may or may not have an edge over an efficient linear loop (such as foreach), as mklement0 notes in his comment:

Parallelism works best for different disks/shares/computers.

Answer 2

Try the following:

$path = '\\Sidney2\MfgLib\AidLibTest'
$allFilePathsOfInterest =
  foreach ($dir in [System.IO.Directory]::GetDirectories($path)) {
    [System.IO.Directory]::GetFiles($dir, '*.EIA')
  }

Given that the input directory path is a full path, $allFilesOfInterest is an array of full file paths too.

If you want the file names only, use the instance methods of the [System.IO.DirectoryInfo] type instead of the static methods of the [System.IO.Directory] type, which allows you to access the .Name property of the [System.IO.FileInfo] instances being returned:

$path = '\\Sidney2\MfgLib\AidLibTest'
$allFileNamesOfInterest =
  foreach ($dir in [System.IO.DirectoryInfo]::new($path).GetDirectories()) {
    $dir.GetFiles('*.EIA').Name
  }

• Note the two-step approach - get subdirectories first, then examine their files - because I'm not aware of a standard .NET API that would allow you to process wildcards across levels of the hierarchy (e.g., \\Sidney2\MfgLib\AidLibTest\*\*.EIA').

• If you need more control over the enumeration of the files and directories, the GetDirectories and GetFiles methods offer overloads that accept a System.IO.EnumerationOptions instance, but, unfortunately, in PowerShell (Core) 7+ / .NET (Core) only:

  • Windows PowerShell / .NET Framework only offers overloads with a System.IO.SearchOption instance, but the only thing that controls is whether the enumeration is recursive.