Running Import
tsj siteinstance kessler_chaindrive_download
This is the script /home/typethink/.virtualenvs/pine-celery/bin/kessler_chaindrive_download.
This script makes a directory for downloading the batches and runs the downloader and imports the batches.
The downloader finds the last batch that chaindrive believes we've downloaded (batch_min) and the last chaindrive batch (batch_max) and downloads all of them.
There's a log in the directory.
The import script runs both imports (it's the pattern importer described below).
Note this script hardcodes Kessler's schema to limit it's run to Kessler's.
Scan for Product IDs
chain_prod $(cat kessler_missing_list )
This presupposes that kessler_missing_list is a list of whitespace separated product IDs.
chain_prod is in ~/bin on 23, you do not have to activate the virtualenv first.
This command uses zgrep to find files that have the interesting product IDs, then runs the management command for finding product definitions and product availability on those files.
How to run the current manual import:
tsj siteinstance cd ~/tmp/siteinstance import_chaindrive_pattern 2018-0?-??
For better or worse, these are the steps for running all the saved chaindrive data (since Jan 25th) through our importer.
This is quite long running, probably should run in tmux (disconnect fears).
To run for dev instead of live, change line 3 to be "tsj kess[tab]chain" instead of "tsj kess[tab]gm".
Management Commands
pointofsale_readit
This takes any number of gzipped product definition response files (made by our runner).
Imports the product definitions to Items.
Loads up images from the FTP too. Items without images will be marked incomplete.
Example:
~tmp/kesslers-chaindrive/2018-01-26$ manage.py pointofsale_readit -v2 chaindrive_get_product_definition*.data.gz
pointofsale_read_inventory
This takes any number of gzipped product availability response files (made by our runner).
Adds ItemLocations for the items for existing Locations (pos / location id is used for the lookup).
Items without quantity at any setup location will be marked inactive.
Example:
~tmp/kesslers-chaindrive/2018-01-26$ manage.py pointofsale_read_inventory -v2 chaindrive_get_product_availability_by_branch_*.data.gz
pointofsale_show
This takes either product definition or availability (--inventory) response files.
Displays entries in the availability or definitions file as parsed (shown as key sorted key value pairs).
Optionally takes a product id or product code, which only shows the given id or code.
Examples:
~tmp/kesslers-chaindrive/2018-01-26$ manage.py pointofsale_show --inventory --productcode DSZRD12294 chaindrive_get_product_availability_by_branch_*.data.gz
Show me the product data for ProductID in these two months of API data
~tmp/kesslers-chaindrive$ manage.py pointofsale_show --productid 297375 2019-0[67]-??/*def*.data.gz
Show me the inventory data for ProductID in these two months of API data
~/tmp/kesslers-chaindrive$ manage.py pointofsale_show --productid 297375 --inventory 2019-0[67]-??/*ava*.data.gz
Search the last 100 logs for this ProductID
~/log/kesslersdiamonds_gm$ grep 297375 $(ls -tr *$TSJ_SITE-20190[67]??.log|tail -100)
Shell:
chaindrive_import
This runs pointofsale_readit and pointofsale_read_inventory in the current directory against the appropriately named response files.
This command is in pine-celery's bin (on tsj23).
It also writes a log in log/$TSJ_SITE/
Example:
~tmp/kesslers-chaindrive/2018-01-26$ import_chaindrive
Python:
python -m tsj_pointofsale.chaindrive.runner
This downloads responses for the coded range of batches.
The downloaders are threaded, which means 4 simultaneous downloads.
The response time can vary from <0.1s for an empty response to >45s for a full 100 items in a product definition with more template lookups.
The long response time is what prompted saving responses.
Example (Notice this is on tsj22, do not commit runner to the repo, as it has a password):
~tmp/kesslers-chaindrive/2018-02-01$ python -m tsj_pointofsale.chaindrive.runner | tee download.log