chore: minimize outdated page 🔥

Added conventions for comments, commits, and branches.

README.md

@@ -1,8 +1,14 @@
 ## [DannyDannyDanny/Methodology](https://github.com/DannyDannyDanny/methodology/)
 
+Conventions:
+* [conventional comments](https://conventionalcomments.org/)
+* [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/)
+* [conventional branch](https://conventional-branch.github.io/)
+
 This guide focuses on designing, developing ~~and delivering~~ Data-Processing Sytems, written primarily in python, deploying to Linux servers, optionally communicating with external interfaces (APIs or DBs). The repo is actively being updated; _bookmark_ :bookmark: it, leave a _Star_ :star: or _Watch_ :eye: the repo.
 
 * [repo roadmap](#roadmap)
+  * 🔴🟡🟢 [example process roadmap](example_progress_roadmap.md)
 * [gitmoji](#gitmoji-reference)
 * [planning](project_planning_contracting.md)
 * ideation / design (coming soon)
@@ -12,8 +18,9 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing S
 * Deployment (coming soon)
 * Launch (coming soon)
 * Maintenance (coming soon)
+* [cloud engineering](cloud-engineering.md)
 * [data management](data_storage.md)
-* [tools](tools.md) (:construction: under construction :construction:)
+* [tools](tools.md) :construction:
 * [philosophy](philosophy.md)
 * [scripts](scripts/)
   * [emoji_shortlist](scripts/emoji_shortlist.py)

cloud-engineering.md

@@ -0,0 +1,3 @@
+# Cloud Engineering
+
+* [general guidance](https://www.lockedinspace.com/posts/001.html)

example_progress_roadmap.md

@@ -0,0 +1,44 @@
+## Datasets
+
+Which datasets are in the cloud?
+
+🔴 - dataset in cloud
+🟡 - dataset migration in progress
+🟢 - dataset entirely migrated
+
+*  Dataset1
+* 🟢 Dataset2
+* 🟢 Dataset 2
+* 🟡 Dataset 3
+* 🔴 Dataset 4
+* 🔴 Dataset 5
+* 🔴 Dataset 6
+* 🔴 Dataset 7
+
+## Projects
+
+What projects are fully in the cloud?
+
+🔴 - project not in cloud
+🟡 - project migration in progress
+🟢 - project entirely in cloud
+
+* 🟢 Project 1
+* 🟡 Project 2
+* 🟡 Project 3
+* 🔴 Project 4
+* 🔴 Project 5
+
+## Users
+
+Who is in the cloud?
+
+🔴 - no projects in cloud
+🟡 - some projects in cloud
+🟢 - all projects in cloud
+
+* 🟢 User 1
+* 🟢 User 2
+* 🟡 User 3
+* 🟡 User 4
+* 🔴 User 5

index.md

@@ -1,2 +1,2 @@
-# Index For VIMWIKI
+# Index For methodology/VIMWIKI
 * [readme.md](README.md)

methodology-development.md

@@ -1,6 +1,14 @@
 ### Development
 * Development Cycle [Agile](https://www.atlassian.com/agile/scrum/sprints)
 
+#### Backlog Refinement
+* Fibonacci Story Points: 1 2 3 5 8 13 20 40 100
+* Each Story should have an epic
+* Example Description:
+  * **Background**: Who? Why? What? How?
+  * **Expected Outcome**: measurable change from before / after completion.
+  * **Dependencies**: descriptions / approval from ABC / acceptance criteria / uaser acceotance testing
+
 ## Development Infrastructure
 * **[Setup Dev Machine](setup-dev-machine.md)**
 * **Setup Environments**
@@ -17,6 +25,7 @@
   * Setup [Travis-CI build testing](https://docs.travis-ci.com/user/customizing-the-build/#building-specific-branches)
   * Add an [`.EditorConfig` file](https://editorconfig.org/)
   * Add [git commit template](https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733#file-git-commit-template-md)
+  * Branching strategy: small teams might want to adopt [trunk based development](https://trunkbaseddevelopment.com/)
 * **Documentation**
   * Your repo should have a one-liner for isntallation andd running: (docker-compose.yml + `doco up`)
 
@@ -77,6 +86,9 @@ git log --follow -- filename
 
 # search code in all history
 git log -S "some_old_varname" --oneline --all
+
+# files changed on your branch compared to origin/main i.e. **files changed** in a Github PR
+git diff --name-only origin/main
 ```
 
 ### Submitting multiple PRs

python_snippets.md

@@ -1,3 +1,46 @@
+## database file-caching
+
+```
+from pathlib import Path
+from snowflake.connector import connect
+import pandas as pd
+import hashlib
+
+def get_df(query, env, use_cache=True):
+    print(f'{query[:100] = }')
+    if env == "prod":
+        role = "COSUTMER"
+        warehouse = "CLOWN_COSTUMES"
+    elif env == "sand":
+        role = "CUSTOMER"
+        warehouse = "CUSTOMER_INFORMATION"
+    else:
+        raise ValueError(f"env must be prod or sand - recieved: {env}")
+
+    hash_query_4_digits = hashlib.shake_128(query.encode()).hexdigest(4)
+    cache_pkl_path = Path(f"/tmp/cache-{env}-{hash_query_4_digits}.pkl")
+    print(cache_pkl_path.exists(), cache_pkl_path)
+    if cache_pkl_path.exists() and use_cache:
+        print('loading cached data')
+        df = pd.read_pickle(cache_pkl_path)
+        return df
+
+    print("fetching & caching data")
+    with connect(
+        authenticator="externalbrowser",
+        user="user@company.com",
+        account=f"company-{env}",
+        role=role,
+        warehouse=warehouse,
+    ) as con:
+        cur = con.cursor()
+        cur.execute(query)
+        df = cur.fetch_pandas_all()
+        df.to_pickle(cache_pkl_path)
+        return df
+```
+
+
 ## Python Kernel Management
 ```
 # install new kernel "my_new_env"

setup-dev-machine.md

@@ -1,17 +1,4 @@
 # Dev Machine
 
-🚧 I'm currently in the process of [switching to Linux](https://github.com/DannyDannyDanny/methodology/issues/5).
+This page is made redundant by: [DannyDannyDanny/dotfiles](https://github.com/DannyDannyDanny/dotfiles)
-
+> TODO: delete this page
-🧤 Get comfortable with these tools on a system you are familiar with like MacOS or Windows (via Windows-Subsystem-Linux).
-
-
-| Tool |Linux-Compatible | MacOS-compatible | Description |
-|--- |--- | --- | --- |
-| [`zsh`](https://www.zsh.org/) + [omz](https://github.com/ohmyzsh/ohmyzsh) | ✅ |  ✅ | Preferred Shell + Configurator |
-| [`lf`](https://github.com/gokcehan/lf) | ✅ |  ✅ | CLI file browser |
-| [`vim`](https://github.com/gokcehan/lf) | ✅ |  ✅ | CLI text editor  |
-| ~[`mutt`](http://www.mutt.org/)~ | ✅ |  ✅ | ~CLI email client~ (just use a browser for now) |
-
-🔖 Further reading:
-* [Unix as IDE](https://blog.sanctum.geek.nz/series/unix-as-ide/)
-* [LARBS progs.csv](https://github.com/LukeSmithxyz/LARBS/blob/master/progs.csv)