diff --git a/README.md b/README.md index 86b02ef..91e9688 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,14 @@ ## [DannyDannyDanny/Methodology](https://github.com/DannyDannyDanny/methodology/) +Conventions: +* [conventional comments](https://conventionalcomments.org/) +* [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) +* [conventional branch](https://conventional-branch.github.io/) + This guide focuses on designing, developing ~~and delivering~~ Data-Processing Sytems, written primarily in python, deploying to Linux servers, optionally communicating with external interfaces (APIs or DBs). The repo is actively being updated; _bookmark_ :bookmark: it, leave a _Star_ :star: or _Watch_ :eye: the repo. * [repo roadmap](#roadmap) + * ๐Ÿ”ด๐ŸŸก๐ŸŸข [example process roadmap](example_progress_roadmap.md) * [gitmoji](#gitmoji-reference) * [planning](project_planning_contracting.md) * ideation / design (coming soon) @@ -12,8 +18,9 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing S * Deployment (coming soon) * Launch (coming soon) * Maintenance (coming soon) +* [cloud engineering](cloud-engineering.md) * [data management](data_storage.md) -* [tools](tools.md) (:construction: under construction :construction:) +* [tools](tools.md) :construction: * [philosophy](philosophy.md) * [scripts](scripts/) * [emoji_shortlist](scripts/emoji_shortlist.py) diff --git a/cloud-engineering.md b/cloud-engineering.md new file mode 100644 index 0000000..b9556ea --- /dev/null +++ b/cloud-engineering.md @@ -0,0 +1,3 @@ +# Cloud Engineering + +* [general guidance](https://www.lockedinspace.com/posts/001.html) diff --git a/example_progress_roadmap.md b/example_progress_roadmap.md new file mode 100644 index 0000000..006a578 --- /dev/null +++ b/example_progress_roadmap.md @@ -0,0 +1,44 @@ +## Datasets + +Which datasets are in the cloud? + +๐Ÿ”ด - dataset in cloud +๐ŸŸก - dataset migration in progress +๐ŸŸข - dataset entirely migrated + +* Dataset1 +* ๐ŸŸข Dataset2 +* ๐ŸŸข Dataset 2 +* ๐ŸŸก Dataset 3 +* ๐Ÿ”ด Dataset 4 +* ๐Ÿ”ด Dataset 5 +* ๐Ÿ”ด Dataset 6 +* ๐Ÿ”ด Dataset 7 + +## Projects + +What projects are fully in the cloud? + +๐Ÿ”ด - project not in cloud +๐ŸŸก - project migration in progress +๐ŸŸข - project entirely in cloud + +* ๐ŸŸข Project 1 +* ๐ŸŸก Project 2 +* ๐ŸŸก Project 3 +* ๐Ÿ”ด Project 4 +* ๐Ÿ”ด Project 5 + +## Users + +Who is in the cloud? + +๐Ÿ”ด - no projects in cloud +๐ŸŸก - some projects in cloud +๐ŸŸข - all projects in cloud + +* ๐ŸŸข User 1 +* ๐ŸŸข User 2 +* ๐ŸŸก User 3 +* ๐ŸŸก User 4 +* ๐Ÿ”ด User 5 diff --git a/index.md b/index.md index 0e1749e..7e8bee9 100644 --- a/index.md +++ b/index.md @@ -1,2 +1,2 @@ -# Index For VIMWIKI +# Index For methodology/VIMWIKI * [readme.md](README.md) diff --git a/methodology-development.md b/methodology-development.md index 46c8ffb..b74b731 100644 --- a/methodology-development.md +++ b/methodology-development.md @@ -1,6 +1,14 @@ ### Development * Development Cycle [Agile](https://www.atlassian.com/agile/scrum/sprints) +#### Backlog Refinement +* Fibonacci Story Points: 1 2 3 5 8 13 20 40 100 +* Each Story should have an epic +* Example Description: + * **Background**: Who? Why? What? How? + * **Expected Outcome**: measurable change from before / after completion. + * **Dependencies**: descriptions / approval from ABC / acceptance criteria / uaser acceotance testing + ## Development Infrastructure * **[Setup Dev Machine](setup-dev-machine.md)** * **Setup Environments** @@ -17,6 +25,7 @@ * Setup [Travis-CI build testing](https://docs.travis-ci.com/user/customizing-the-build/#building-specific-branches) * Add an [`.EditorConfig` file](https://editorconfig.org/) * Add [git commit template](https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733#file-git-commit-template-md) + * Branching strategy: small teams might want to adopt [trunk based development](https://trunkbaseddevelopment.com/) * **Documentation** * Your repo should have a one-liner for isntallation andd running: (docker-compose.yml + `doco up`) @@ -77,6 +86,9 @@ git log --follow -- filename # search code in all history git log -S "some_old_varname" --oneline --all + +# files changed on your branch compared to origin/main i.e. **files changed** in a Github PR +git diff --name-only origin/main ``` ### Submitting multiple PRs diff --git a/python_snippets.md b/python_snippets.md index 592e014..14230fe 100644 --- a/python_snippets.md +++ b/python_snippets.md @@ -1,3 +1,46 @@ +## database file-caching + +``` +from pathlib import Path +from snowflake.connector import connect +import pandas as pd +import hashlib + +def get_df(query, env, use_cache=True): + print(f'{query[:100] = }') + if env == "prod": + role = "COSUTMER" + warehouse = "CLOWN_COSTUMES" + elif env == "sand": + role = "CUSTOMER" + warehouse = "CUSTOMER_INFORMATION" + else: + raise ValueError(f"env must be prod or sand - recieved: {env}") + + hash_query_4_digits = hashlib.shake_128(query.encode()).hexdigest(4) + cache_pkl_path = Path(f"/tmp/cache-{env}-{hash_query_4_digits}.pkl") + print(cache_pkl_path.exists(), cache_pkl_path) + if cache_pkl_path.exists() and use_cache: + print('loading cached data') + df = pd.read_pickle(cache_pkl_path) + return df + + print("fetching & caching data") + with connect( + authenticator="externalbrowser", + user="user@company.com", + account=f"company-{env}", + role=role, + warehouse=warehouse, + ) as con: + cur = con.cursor() + cur.execute(query) + df = cur.fetch_pandas_all() + df.to_pickle(cache_pkl_path) + return df +``` + + ## Python Kernel Management ``` # install new kernel "my_new_env" diff --git a/setup-dev-machine.md b/setup-dev-machine.md index 1e30913..096f6fb 100644 --- a/setup-dev-machine.md +++ b/setup-dev-machine.md @@ -1,17 +1,4 @@ # Dev Machine -๐Ÿšง I'm currently in the process of [switching to Linux](https://github.com/DannyDannyDanny/methodology/issues/5). - -๐Ÿงค Get comfortable with these tools on a system you are familiar with like MacOS or Windows (via Windows-Subsystem-Linux). - - -| Tool |Linux-Compatible | MacOS-compatible | Description | -|--- |--- | --- | --- | -| [`zsh`](https://www.zsh.org/) + [omz](https://github.com/ohmyzsh/ohmyzsh) | โœ… | โœ… | Preferred Shell + Configurator | -| [`lf`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI file browser | -| [`vim`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI text editor | -| ~[`mutt`](http://www.mutt.org/)~ | โœ… | โœ… | ~CLI email client~ (just use a browser for now) | - -๐Ÿ”– Further reading: -* [Unix as IDE](https://blog.sanctum.geek.nz/series/unix-as-ide/) -* [LARBS progs.csv](https://github.com/LukeSmithxyz/LARBS/blob/master/progs.csv) +This page is made redundant by: [DannyDannyDanny/dotfiles](https://github.com/DannyDannyDanny/dotfiles) +> TODO: delete this page