Skip to content

enabling and applying reading geom from shp #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
remicousin wants to merge 1 commit into
base: master
Choose a base branch
from
Open

enabling and applying reading geom from shp #623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 4 additions & 26 deletions pepsico/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,20 @@

# Installation and Run Instructions

## Creating a conda environment with this project's dependencies
## Get Shapefiles

* see enacts' README
* Get the GAUL 2015 levels 0 and 1 from FAO at `https://data.apps.fao.org/catalog/organization/administrative-boundaries-fao?__cf_chl_f_tk=OgUUh7I4P2USfARHuVjb3Rfoqf7aqn2I9Rk18Y390T8-1782743011-1.0.1.1-K6UEdWxnYz2bczx3DjVit2KF0TaXe9b3nV.wnhusfMQ`

## Running the application in a development environment

* Activate the environment

`conda activate enactsmaproom`

* Create a development configuration file by copying `config-dev-sample.yaml` to `config-dev.yaml` and editing it as needed. Note that `config-dev.yaml` is in the `.gitignore` file so you won't accidentally commit changes that are specific to your development environment.
* Create a development configuration file by copying `config-dev-sample.yaml` to `config-dev.yaml` and editing it as needed, in particular to overwrite the default configuration values set in `config-defaults.yaml` such as data files paths and names. Note that `config-dev.yaml` is in the `.gitignore` file so you won't accidentally commit changes that are specific to your development environment.

* Start the development server using your development config file, e.g.:

`CONFIG=config-dev.yaml python app.py`
`CONFIG=config-dev.yaml pixi run python app.py`

* Navigate your browser to the URL that is displayed when the server starts, e.g. `http://127.0.0.1:8050/python_maproom/`

* When done using the server stop it with CTRL-C.

# Development Overview

see enacts' README

## Adding or removing dependencies

see enacts' README

## Building the documentation

see enacts' README

# Docker Build Instructions

TBD

# Support

* `[email protected]`
132 changes: 99 additions & 33 deletions pepsico/config-defaults.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,75 +14,141 @@ datasets:
shapes_adm_US-CA:
- name: CA-US-MX
color: black
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (46, 162, 259)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (46, 162, 259)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (46, 162, 259)
tolerance: 0.0025
is_checked: True
- name: Countries
color: black
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (9, 11, 20, 28, 30, 32, 39, 46, 48, 63, 71, 72, 98, 100, 103, 108, 111, 123, 162, 168, 174, 176, 180, 200, 208, 210, 251, 258, 259)
# sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (9, 11, 20, 28, 30, 32, 39, 46, 48, 63, 71, 72, 98, 100, 103, 108, 111, 123, 162, 168, 174, 176, 180, 200, 208, 210, 251, 258, 259)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (9, 11, 20, 28, 30, 32, 39, 46, 48, 63, 71, 72, 98, 100, 103, 108, 111, 123, 162, 168, 174, 176, 180, 200, 208, 210, 251, 258, 259)
tolerance: 0.0025
is_checked: True
- name: States
color: grey
sql: select (adm0_code, adm1_code) as key, adm1_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2014_1 where adm0_code in (46, 162, 259)
# sql: select (adm0_code, adm1_code) as key, adm1_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2014_1 where adm0_code in (46, 162, 259)
path: /data/remic/mydatafiles/pepsico/g2015_2014_1/g2015_2014_1.shp
geom: geometry
label: ADM1_NAME
selector: ADM0_CODE
selected: (46, 162, 259)
tolerance: 0.0025
is_checked: True

shapes_adm_SAMER:
- name: National
color: black
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
tolerance: 0.0025
is_checked: True
- name: States
color: grey
sql: select (adm0_code, adm1_code) as key, adm1_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2014_1 where adm0_code in (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
#sql: select (adm0_code, adm1_code) as key, adm1_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2014_1 where adm0_code in (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
path: /data/remic/mydatafiles/pepsico/g2015_2014_1/g2015_2014_1.shp
geom: geometry
label: ADM1_NAME
selector: ADM0_CODE
selected: (12, 14, 24, 32, 33, 37, 51, 57, 61, 71, 73, 81, 86, 99, 100, 107, 111, 158, 176, 180, 191, 194, 195, 209, 211, 228, 233, 246, 260, 263)
tolerance: 0.0025
is_checked: True

shapes_adm_SASIA:
- name: India-Pakistan
color: black
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (115, 188)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (115, 188)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (115, 188)
tolerance: 0.0025
is_checked: True
- name: National
color: grey
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (1, 2, 15, 23, 31, 52, 117, 132, 138, 154, 171, 175, 187, 231, 239, 250)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (1, 2, 15, 23, 31, 52, 117, 132, 138, 154, 171, 175, 187, 231, 239, 250)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (1, 2, 15, 23, 31, 52, 117, 132, 138, 154, 171, 175, 187, 231, 239, 250)
tolerance: 0.0025
is_checked: True
- name: States
color: grey
sql: select (adm0_code, adm1_code) as key, adm1_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2014_1 where adm0_code in (115, 188)
#sql: select (adm0_code, adm1_code) as key, adm1_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2014_1 where adm0_code in (115, 188)
path: /data/remic/mydatafiles/pepsico/g2015_2014_1/g2015_2014_1.shp
geom: geometry
label: ADM1_NAME
selector: ADM0_CODE
selected: (115, 188)
tolerance: 0.0025
is_checked: True

shapes_adm_Thailand:
- name: Thailand-Vietnam
color: black
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (240, 264)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (240, 264)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (240, 264)
tolerance: 0.0025
is_checked: True
- name: National
color: grey
sql: select adm0_code as key, adm0_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2012_0 where adm0_code in (15, 23, 31, 40, 44, 115, 116, 139, 149, 153, 171, 175, 183, 230)
#sql: select adm0_code as key, adm0_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2012_0 where adm0_code in (15, 23, 31, 40, 44, 115, 116, 139, 149, 153, 171, 175, 183, 230)
path: /data/remic/mydatafiles/pepsico/g2015_2014_0/g2015_2014_0.shp
geom: geometry
label: ADM0_NAME
selector: ADM0_CODE
selected: (15, 23, 31, 40, 44, 115, 116, 139, 149, 153, 171, 175, 183, 230)
tolerance: 0.0025
is_checked: True
- name: States
color: grey
sql: select (adm0_code, adm1_code) as key, adm1_name as label,
ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
from g2015_2014_1 where adm0_code in (240, 264)
#sql: select (adm0_code, adm1_code) as key, adm1_name as label,
# ST_AsBinary(ST_SimplifyPreserveTopology(coarse_geom, 0.0025)) as the_geom
# from g2015_2014_1 where adm0_code in (240, 264)
path: /data/remic/mydatafiles/pepsico/g2015_2014_1/g2015_2014_1.shp
geom: geometry
label: ADM1_NAME
selector: ADM0_CODE
selected: (240, 264)
tolerance: 0.0025
is_checked: True

maprooms:
Expand Down
1 change: 1 addition & 0 deletions pepsico/config-dev-sample.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ mode: debug # debug, devel or prod
dev_server_interface: 127.0.0.1
dev_server_port: 8050
dev_processes: 4
# to query db of geoms
db:
host: localhost # at IRI, set this to dlcomputemon1.iri.columbia.edu
password: null # at IRI, get the password from another developer
Expand Down
21 changes: 20 additions & 1 deletion pepsico/maproom_utilities.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from shapely.geometry.multipolygon import MultiPolygon
from shapely.geometry import Polygon
import sqlalchemy
import geopandas as geopd


# Mapping Utilities
Expand Down Expand Up @@ -39,8 +40,18 @@ def get_geom(level, conf, shapes_adm_name):
"""
if "bbox" in conf["datasets"] :
return synthesize_geom(conf["datasets"]["bbox"], level=level)
else:
elif 'sql' in conf["datasets"][shapes_adm_name][level]:
return sql2geom(conf["datasets"][shapes_adm_name][level]["sql"], conf["db"])
elif 'path' in conf["datasets"][shapes_adm_name][level]:
return shp2geom(
conf["datasets"][shapes_adm_name][level]["path"],
conf["datasets"][shapes_adm_name][level]["geom"],
conf["datasets"][shapes_adm_name][level]["label"],
conf["datasets"][shapes_adm_name][level]["selector"],
conf["datasets"][shapes_adm_name][level]["selected"],
#[int(s) for s in conf["datasets"][shapes_adm_name][level]["selected"].split()],
conf["datasets"][shapes_adm_name][level]["tolerance"],
)


def sql2GeoJSON(shapes_sql, db_config):
Expand Down Expand Up @@ -75,6 +86,14 @@ def sql2GeoJSON(shapes_sql, db_config):
return geom2GeoJSON(sql2geom(shapes_sql, db_config))


def shp2geom(path, geom, label, selector, selected, tolerance):
df = geopd.read_file(
path, columns=[geom, label, selector], where=f'{selector} in {selected}'
).rename(columns={geom: 'the_geom', label: 'label'})
#df['the_geom'] = df.the_geom.simplify(tolerance=tolerance)
return df


def geom2GeoJSON(df):
""" Form a GeoJSON dict from a geometric object

Expand Down
Loading
Loading