Unlocking file using Restart Manager from Rust

Question

I'm trying to convert this function from C++ to Rust using the windows crate:

#include <iostream>
#include "windows.h"

#include <RestartManager.h>
#pragma comment(lib ,"Rstrtmgr.lib")

BOOL ReleaseFileLock(LPCTSTR pFilePath)
{
    BOOL bResult = FALSE;

    DWORD dwSession;
    WCHAR szSessionKey[CCH_RM_SESSION_KEY + 1] = { 0 };
    DWORD dwError = RmStartSession(&dwSession, 0, szSessionKey);
    if (dwError == ERROR_SUCCESS)
    {
        dwError = RmRegisterResources(dwSession, 1, &pFilePath,
            0, NULL, 0, NULL);
        if (dwError == ERROR_SUCCESS)
        {
            UINT nProcInfoNeeded = 0;
            UINT nProcInfo = 0;
            RM_PROCESS_INFO rgpi[1];
            DWORD dwReason;

            dwError = RmGetList(dwSession, &nProcInfoNeeded,
                &nProcInfo, rgpi, &dwReason);
            if (dwError == ERROR_SUCCESS ||
                dwError == ERROR_MORE_DATA)
            {
                if (nProcInfoNeeded > 0)
                {
                    //If current process does not have enough privileges to close one of
                    //the "offending" processes, you'll get ERROR_FAIL_NOACTION_REBOOT
                    dwError = RmShutdown(dwSession, RmForceShutdown, NULL);
                    if (dwError == ERROR_SUCCESS)
                    {
                        bResult = TRUE;
                    }
                }
                else
                    bResult = TRUE;
            }
        }
    }

    RmEndSession(dwSession);

    SetLastError(dwError);
    return bResult;
}

I successfully imported the necessary functions for that, but as a starting point I tried simply to start the session and close it and I got error STATUS_ACCESS_VIOLATION:

use windows_sys::core::PWSTR;
use windows::Win32::System::RestartManager::{
    RmStartSession,
    RmEndSession,
    RmRegisterResources,
    RmGetList,
    RmShutdown,
    RmForceShutdown,
};

fn release_file_lock(path: &str) -> Result<(), Box<dyn std::error::Error>> {    

    let psessionhandle = std::ptr::null_mut();
    let dwsessionflags: u32 = 0;
    let strsessionkey = std::ptr::null_mut();
    unsafe {
        let result = RmStartSession(psessionhandle, dwsessionflags, windows::core::PWSTR(strsessionkey));
        println!("{:?} {:?}", result, psessionhandle);
        RmEndSession(*psessionhandle); // < --- STATUS_ACCESS_VIOLATION
    };
    Ok(())
}

I would like to know what the flow of converting the code should be.

Currently, I'm looking in docs.rs/windows and follow every function and I'm trying to fit the parameters, but it looks like I missed something important in that process.

Answer 1

I would argue there is no single right way to convert code from one language to another. If you are unfamiliar with the language you can go layer by layer, like you did in the provided rust snippet. This way you may have an easier experience learning the concepts of the target language during your conversion process.

In your case I would recommend to focus on one api call at a time and check if the results match with the results of your C++ source code. Let me share some of my opinions on the conversion topic:

1. Keep the functions as identical as reasonable in the first iteration. For example your C++ code has the function signature BOOL ReleaseFileLock(LPCTSTR pFilePath) but Rust has release_file_lock(path: &str) -> Result<...>. In unicode mode a LPCTSTR refers to wide characters that represent UTF16 encoding in windows. Rust stores the strings in UTF8 but the api you later call expects UTF16. Keeping the signatures identical at first, ensures that your expectations of the functional behavior can also be identical.

2. Do not care about the specifics of error handling during conversion. You already started with Result<(), Box<dyn std::error::Error>> which implies some sort of basic error forwarding. I would generally keep it simple and skip the error handling part for the moment. If you encounter api functions returning Result<...> just do unwrap. Of course this may has the side effects of handle leaks due to unwrap but it is just faster for prototyping and converting code in an unfamiliar terrain. Later in the finalization process of the conversion, introduce error handling and step through the termination to ensure correctness of resource management.

3. Be extremely careful with types, type conversions and memory concepts! You should be aware of the differences of C++ and Rust and know what the api expects from the arguments you pass. Understand the intent and content of variables in the C++ code, then research how to achieve the same using Rust.

Lets walk through the conversion process and explore some mistakes you made during conversion. The following snippet shows a stripped down version of the C++ example for the baseline. I made some minor type modifications and moved RmEndSession into the successful branch.

#include <Windows.h>
#include <RestartManager.h>
#pragma comment(lib ,"Rstrtmgr.lib")

bool ReleaseFileLock(LPCWSTR pFilePath)
{
    DWORD dwSession = 0;
    WCHAR szSessionKey[CCH_RM_SESSION_KEY + 1] = { 0 };
    DWORD dwError = RmStartSession(&dwSession, 0, szSessionKey);
    if (dwError == ERROR_SUCCESS)
    {
        RmEndSession(dwSession);
        return true;
    }
    return false;
}

int main() {
    ReleaseFileLock(L"C:\\somepath");
}

Here would be the conversion to Rust:

use windows::{Win32::{System::RestartManager::{
    RmStartSession,
    RmEndSession, CCH_RM_SESSION_KEY,
}, Foundation::{ERROR_SUCCESS, WIN32_ERROR}}, core::{PWSTR, PCWSTR, w}};

unsafe fn release_file_lock(file_path: PCWSTR) -> bool {
    let mut session: u32 = 0;
    let mut session_key_buffer = [0_u16; (CCH_RM_SESSION_KEY as usize) + 1];
    let session_key = PWSTR(session_key_buffer.as_mut_ptr());
    let result = RmStartSession(&mut session, 0, session_key);
    if WIN32_ERROR(result) == ERROR_SUCCESS {
        RmEndSession(session);
        return true;
    }
    return false;
}

fn main() {
    unsafe {
        release_file_lock(w!("C:\\somepath"));
    }
}

As you can see the first iteration of the Rust solution is almost identical to the C++ solution. There are type differences like DWORD is u32 and WCHAR is u16 in Rust. There are additional type conversions required like WIN32_ERROR(result) and PWSTR(session_key_buffer.as_mut_ptr()) to convert the wide char slice to the type expected by the api call. But the underlying ideas and variables memory layouts are equivalent.

During your conversion attempt a lot of the C++ code concepts were certainly unintentionally left out or modified in a way that changed the intentions of variables. In the Rust version you:

1. Did not check if the api call was successful before doing something with the handle. To avoid garbage data during your debugging, you should always check the return value for success.

2. Used pointers let psessionhandle = std::ptr::null_mut() where originally values DWORD dwSession were declared. The confusion of values and pointers also caused the access violation due to a dereference of a null pointer. I many cases a mutable reference passed to a function in C++ like the &dwSession translates in Rust to &mut dwSession or for complex types to .as_mut_ptr(), so you just have to keep the original idea of the C++ code in mind.

3. Did not allocate a WCHAR szSessionKey[] as a buffer for the api but instead passed in a std::ptr::null_mut(). The api just wants a pointer to the memory where it can store this session key. In C++ this pointer came from the stack allocated WCHAR szSessionKey[] and you did not have to reference it like so &szSessionKey during the function call because in C++ the variable to a stack allocated memory block is already its pointer. In Rust you had to declare the stack memory and then specifically pass its pointer with .as_mut_ptr() to the PWSTR(*mut u16) to achieve the equivalent concept.

With this advice you should be able to convert the entire code. Keep in mind that after the first iteration you go back and introduce proper error handling and if desired you can also update the function arguments to more convenient types, like a string slice instead of windows specific types.

Answer 2

Before raising an access violation, the code prints out:

160 0x0

The first number, 160, is the return value of the RmStartSession() call. This is a system error code identified by the named constant ERROR_BAD_ARGUMENTS that translates to the error text "One or more arguments are not correct."

The first step in preventing the access violation is thus to observe the return value of RmStartSession() and bail if it is anything but ERROR_SUCCESS like so:

let result = RmStartSession(
    psessionhandle,
    dwsessionflags,
    windows::core::PWSTR(strsessionkey),
);
if result != ERROR_SUCCESS.0 {
    return Err("Failed to start session".into());
}

With that addressed, we can move ahead and figure out which argument(s) are wrong. The necessary information is available from Microsoft's documentation detailing the function call contract. The relevant parameters for RmStartSession are:

pSessionHandle:
A pointer to the handle of a Restart Manager session.

strSessionKey:
A null-terminated string that contains the session key to the new session. The string must be allocated before calling the RmStartSession function.

Either pointer must be valid and refer to caller-supplied memory. The former is a pointer to a DWORD (u32 in Rust) while the latter is a pointer to the start of an array of WCHARs (u16s in Rust) of length CCH_RM_SESSION_KEY + 1. Applying those changes leads to the following code:

fn release_file_lock(path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut session_handle = 0_u32;
    let mut session_key = [0_u16; CCH_RM_SESSION_KEY as usize + 1];
    unsafe {
        let result = RmStartSession(&mut session_handle, 0, PWSTR(session_key.as_mut_ptr()));
        if result != ERROR_SUCCESS.0 {
            return Err("Failed to start session".into());
        }
        println!("{:?} {:?}", result, session_handle);
        RmEndSession(session_handle);
    };
    Ok(())
}

Answer 3

Thanks to your assistance, I was able to delve deeper and successfully created the whole function and it works.

use windows::{Win32::{System::RestartManager::{
    RmStartSession,
    RmRegisterResources,
    RmEndSession, 
    RmGetList,
    RmShutdown,
    RmForceShutdown,
    CCH_RM_SESSION_KEY,
    RM_PROCESS_INFO
}, Foundation::{ERROR_SUCCESS, WIN32_ERROR, ERROR_MORE_DATA}}, core::{PWSTR, PCWSTR, w}};

unsafe fn release_file_lock(file_path: PCWSTR) -> bool {
    let mut session: u32 = 0;
    let mut session_key_buffer = [0_u16; (CCH_RM_SESSION_KEY as usize) + 1];
    let session_key = PWSTR(session_key_buffer.as_mut_ptr());
    let result = RmStartSession(&mut session, 0, session_key);
    if WIN32_ERROR(result) == ERROR_SUCCESS {
        let result = RmRegisterResources(
            session,
             Some(&[file_path]), 
             None, 
             None
        );
        if WIN32_ERROR(result) == ERROR_SUCCESS {
            let mut pnprocinfoneeded: u32 = 0;
            let mut rgaffectedapps: [RM_PROCESS_INFO; 1] = [RM_PROCESS_INFO{..Default::default()}];
            let mut lpdwrebootreasons: u32 = 0;
            let mut pnprocinfo: u32 = 0;
            let result = RmGetList(session, &mut pnprocinfoneeded, &mut pnprocinfo, Some(rgaffectedapps.as_mut_ptr()), &mut lpdwrebootreasons);
            if WIN32_ERROR(result) == ERROR_SUCCESS || WIN32_ERROR(result) == ERROR_MORE_DATA {
                if pnprocinfoneeded > 0 {
                    // If current process does not have enough privileges to close one of
                    // the "offending" processes, you'll get ERROR_FAIL_NOACTION_REBOOT
                    let result = RmShutdown(session, RmForceShutdown.0 as u32, None);
                    if WIN32_ERROR(result) == ERROR_SUCCESS {
                        // success
                        RmEndSession(session);
                        return true;
                    }
                } else {
                    // success
                    RmEndSession(session);
                    return true;
                }
            }
        }
        RmEndSession(session);
        return false;
    }
    return false;
}