Downloading data from a production Google App Engine datastore to your Local Development Server

Preparation - Patch Local Development Server [1]

1. Make sure Remote API is enabled for your project
2. Make sure your Local Development Server (LDS) is running
3. Login to your LDS Admin Console in your browser (by visiting http://localhost:/_ah/login)
• e.g. http://localhost:9081/_ah/login (my application is deployed on port 9081)
4. Open your browser's Developer Tools (F12 in Chrome on Windows) and get the value for the dev_appserver_login cookie
5. Find lib/oauth2client/oauth2client/client.py in the LDS install directory (for me this resides at C:\Program Files (x86)\Google\google_appengine\lib\oauth2client\oauth2client\client.py)
6. Create a backup of this file (e.g. called client.py.bak)
7. Open the original file for editing and find the "new_request" function ("def new_request")
8. Before the line self.apply(headers) (for me this appears at Line 555), add the following:
• headers['Cookie'] = 'dev_appserver_login="your-cookie-value-here"';
9. Save the client.py file; and restart your LDS

Download data from App Engine

• Note: download_data doesn't allow specifying a query with which to download data. To limit data, filter by kind.
• Note: You can remove the --kind flag; but, that will download ALL DATA!
• Note: You can play with the --rps_limit / --batch_size flags; but, these settings worked well for me

1. Open a command-line tool (e.g. MS-DOS in Windows) to your application's directory
• e.g. C:\GitHub\my_project
2. Run the following command
• appcfg.py download_data --application=s~[APP_ID] --url=http://[APP_ID].appspot.com/_ah/remote_api --rps_limit=100 --batch_size=100 --kind=[KIND] --filename=[FILENAME].csv
• e.g. appcfg.py download_data --application=s~my-project --url=http://my-project.appspot.com/_ah/remote_api --rps_limit=100 --batch_size=100 --kind=User --filename=user.csv

Upload data to LDS

• Note: You can play with the --num_threads / --rps_limit / --batch_size flags; but, these settings worked well for me

1. Make sure your Local Development Server (LDS) is running
2. Open a command-line tool (e.g. MS-DOS in Windows) to your application's directory
• e.g. C:\GitHub\my_project
3. Run the following command
• appcfg.py upload_data --application=dev~[APP_ID] --url=http://localhost:[APP_PORT]/_ah/remote_api --filename=[FILENAME].csv --num_threads=1 --rps_limit=100 --batch_size=100
• e.g. appcfg.py upload_data --application=dev~my-project --url=http://localhost:9081/_ah/remote_api --filename=user.csv --num_threads=1 --rps_limit=100 --batch_size=100

Hope this helps and thanks for reading! Make sure to share this if you liked it.

Sources

1. https://code.google.com/p/googleappengine/issues/detail?id=12445#c23