Exporting Google Cloud User Accounts to AppEngine's Datastore Emulator

Question

I am working with an older AppEngine GCP application and I am trying to setup a local environment with an emulated representation of the user profiles stored in datastore. For this, I am trying to export the gmail user-profiles from my GCP workspace and import these into my datastore emulator.

Is the local ndb datastore and the datastore emulator completely disconnected from eachother? Because I cant seem to interact with the data imported to the emulator. Is it possible to import it to my local datastore.db directly?

So far, this is what I am doing: I went into GCP and exported the datastore contents as a folder with gzip files. I unpacked them and stored them on my computer.

I run my dev_appserver.py with "--support_datastore_emulator true" so that I get a datastore emulator port, then I use this in postman:

POST request - localhost:[my emulator port]/v1/projects/[my project]:import

body - {"input_url":"path/to/exported_data.overall_export_metadata"}

The request is successful, but I cant seem to find any update to my local ndb database. I can add new and query for ndb elements in the admin interactive console on http://localhost:8000/console, but none of the imported elements can be queried for.

I run the app with:

dev_appserver.py . --storage_path .tmp/ --support_datastore_emulator true --enable_console --dev_appserver_log_level=debug

And the emulator with

datastore start --data-dir=.tmp/ --project=[MY-PROJECT-ID]

I get this warning when starting the emulator:

WARNING: Reusing existing data in [.tmp/].

Which is kinda what I want, no? Since I want my local datastore and the emulator data to both be accessible through ndb calls.

I have added the following environment variables:

DATASTORE_DATASET=[MY-PROJECT-ID]
DATASTORE_EMULATOR_HOST=localhost:8081
DATASTORE_EMULATOR_HOST_PATH=http://localhost:8081/datastore
DATASTORE_HOST=http://localhost:8081;DATASTORE_PROJECT_ID=[MY-PROJECT-ID]
GOOGLE_APPLICATION_CREDENTIALS=C:\path\to\service_account_key.json

Answer 1

Not sure of the order in which you executed the steps but try this

1. Start datastore emulator by running

   gcloud beta emulators datastore start --data-dir=[DATA_DIR]

2. The above command should now give you a port number. You should note that the port number can change each time you start the emulator.

3. Import the data you exported from the cloud using the port number you got when you started the emulator (bullet 1).

4. Before you can run dev_appserver.py, you need to set the environment variables using the values from when you started the emulator (bullet 1). You can either set the values automatically or manually (see documentation)

5. Now that the environment variables have been set, you can now run dev_appserver.py using

   dev_appserver.py --support_datastore_emulator=true app.yaml