From 5b3af97fc30b891d0d9f1dfcd28da0cb38a05121 Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Mon, 3 Oct 2022 10:31:02 +0200 Subject: [PATCH 01/11] :lipstick: make example roadmap --- example_progress_roadmap.md | 44 +++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 example_progress_roadmap.md diff --git a/example_progress_roadmap.md b/example_progress_roadmap.md new file mode 100644 index 0000000..006a578 --- /dev/null +++ b/example_progress_roadmap.md @@ -0,0 +1,44 @@ +## Datasets + +Which datasets are in the cloud? + +๐Ÿ”ด - dataset in cloud +๐ŸŸก - dataset migration in progress +๐ŸŸข - dataset entirely migrated + +* Dataset1 +* ๐ŸŸข Dataset2 +* ๐ŸŸข Dataset 2 +* ๐ŸŸก Dataset 3 +* ๐Ÿ”ด Dataset 4 +* ๐Ÿ”ด Dataset 5 +* ๐Ÿ”ด Dataset 6 +* ๐Ÿ”ด Dataset 7 + +## Projects + +What projects are fully in the cloud? + +๐Ÿ”ด - project not in cloud +๐ŸŸก - project migration in progress +๐ŸŸข - project entirely in cloud + +* ๐ŸŸข Project 1 +* ๐ŸŸก Project 2 +* ๐ŸŸก Project 3 +* ๐Ÿ”ด Project 4 +* ๐Ÿ”ด Project 5 + +## Users + +Who is in the cloud? + +๐Ÿ”ด - no projects in cloud +๐ŸŸก - some projects in cloud +๐ŸŸข - all projects in cloud + +* ๐ŸŸข User 1 +* ๐ŸŸข User 2 +* ๐ŸŸก User 3 +* ๐ŸŸก User 4 +* ๐Ÿ”ด User 5 From 10c457f31df79177db9d51fd397d3caec62ab6e1 Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Mon, 3 Oct 2022 10:33:20 +0200 Subject: [PATCH 02/11] :memo: add exercise roadmap --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ad3bf3e..96f4598 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,7 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing Sytems, written primarily in python, deploying to Linux servers, optionally communicating with external interfaces (APIs or DBs). The repo is actively being updated; _bookmark_ :bookmark: it, leave a _Star_ :star: or _Watch_ :eye: the repo. * [repo roadmap](#roadmap) + * ๐Ÿ”ด๐ŸŸก๐ŸŸข [example process roadmap](example_progress_roadmap.md) * [gitmoji](#gitmoji-reference) * [planning](project_planning_contracting.md) * ideation / design (coming soon) @@ -13,7 +14,7 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing S * Launch (coming soon) * Maintenance (coming soon) * [data management](data_storage.md) -* [tools](tools.md) (:construction: under construction :construction:) +* [tools](tools.md) :construction: * [philosophy](philosophy.md) * [scripts](scripts/) * [emoji_shortlist](scripts/emoji_shortlist.py) From 48e48735d5d02f5349943bd20301613977c89943 Mon Sep 17 00:00:00 2001 From: DannyDannyDanny Date: Tue, 25 Oct 2022 10:05:18 +0200 Subject: [PATCH 03/11] vimwiki: dth @ DK023900WSL --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index 0e1749e..7e8bee9 100644 --- a/index.md +++ b/index.md @@ -1,2 +1,2 @@ -# Index For VIMWIKI +# Index For methodology/VIMWIKI * [readme.md](README.md) From 00efbe079246592abd1a575f487b09f86bd40eae Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Mon, 19 Dec 2022 10:28:26 +0100 Subject: [PATCH 04/11] :memo: doc trunk based dev as branching strategy --- methodology-development.md | 1 + 1 file changed, 1 insertion(+) diff --git a/methodology-development.md b/methodology-development.md index 46c8ffb..f3fd26e 100644 --- a/methodology-development.md +++ b/methodology-development.md @@ -17,6 +17,7 @@ * Setup [Travis-CI build testing](https://docs.travis-ci.com/user/customizing-the-build/#building-specific-branches) * Add an [`.EditorConfig` file](https://editorconfig.org/) * Add [git commit template](https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733#file-git-commit-template-md) + * Branching strategy: small teams might want to adopt [trunk based development](https://trunkbaseddevelopment.com/) * **Documentation** * Your repo should have a one-liner for isntallation andd running: (docker-compose.yml + `doco up`) From f2dc9c1e8a8b9eabc91f68f3c29edcdb54d3a027 Mon Sep 17 00:00:00 2001 From: DannyDannyDanny Date: Wed, 28 Dec 2022 12:21:26 +0100 Subject: [PATCH 05/11] vimwiki: dth @ DK023900WSL --- README.md | 1 + cloud-engineering.md | 3 +++ 2 files changed, 4 insertions(+) create mode 100644 cloud-engineering.md diff --git a/README.md b/README.md index 3d5629c..80ad24a 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing S * Deployment (coming soon) * Launch (coming soon) * Maintenance (coming soon) +* [cloud engineering](cloud-engineering.md) * [data management](data_storage.md) * [tools](tools.md) :construction: * [philosophy](philosophy.md) diff --git a/cloud-engineering.md b/cloud-engineering.md new file mode 100644 index 0000000..b9556ea --- /dev/null +++ b/cloud-engineering.md @@ -0,0 +1,3 @@ +# Cloud Engineering + +* [general guidance](https://www.lockedinspace.com/posts/001.html) From c0a4e6eee37551e3271030b8f3398e33a8156f5c Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Thu, 13 Jul 2023 16:10:43 +0200 Subject: [PATCH 06/11] :note: add **files changed**-like git command --- methodology-development.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/methodology-development.md b/methodology-development.md index f3fd26e..e987b15 100644 --- a/methodology-development.md +++ b/methodology-development.md @@ -78,6 +78,9 @@ git log --follow -- filename # search code in all history git log -S "some_old_varname" --oneline --all + +# files changed on your branch compared to origin/main i.e. **files changed** in a Github PR +git diff --name-only origin/main ``` ### Submitting multiple PRs From 76e69d08999e8920c22f004214f4f4817de0628f Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Wed, 16 Aug 2023 11:20:30 +0200 Subject: [PATCH 07/11] :sparkles: add snowflake caching ;snowflake: --- python_snippets.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/python_snippets.md b/python_snippets.md index 592e014..317d161 100644 --- a/python_snippets.md +++ b/python_snippets.md @@ -1,3 +1,46 @@ +## database file-caching + +``` +from pathlib import Path +from snowflake.connector import connect +import pandas as pd +import hashlib + +def get_df(query, env, use_cache=True): + print(f'{query[:100] = }') + if env == "prod": + role = "COSUTMER" + warehouse = "CLOWN_COSTUMES" + elif env == "sand": + role = "CUSTOMER" + warehouse = "CUSTOMER_INFORMATION" + else: + raise ValueError(f"env must be prod or sand - recieved: {env}") + + hash_query_4_digits = hashlib.shake_128(query.encode()).hexdigest(4) + cache_pkl_path = Path(f"/tmp/cache-{env}-{hash_query_4_digits}.pkl") + print(cache_pkl_path.exists(), cache_pkl_path) + if cache_pkl_path.exists() and use_cache: + print('loading cached data') + df = pd.read_pickle(cache_pkl_path) + return df + + print("fetching & caching data") + with connect( + authenticator="externalbrowser", + user="dnth@lundbeck.com", + account=f"lundbeck-hlu{env}", + role=role, + warehouse=warehouse, + ) as con: + cur = con.cursor() + cur.execute(query) + df = cur.fetch_pandas_all() + df.to_pickle(cache_pkl_path) + return df +``` + + ## Python Kernel Management ``` # install new kernel "my_new_env" From cfedc6d9afb65efe52bf104e4b7cf4d730c032aa Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Tue, 22 Aug 2023 10:08:26 +0200 Subject: [PATCH 08/11] :art: clean up --- python_snippets.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/python_snippets.md b/python_snippets.md index 317d161..14230fe 100644 --- a/python_snippets.md +++ b/python_snippets.md @@ -28,8 +28,8 @@ def get_df(query, env, use_cache=True): print("fetching & caching data") with connect( authenticator="externalbrowser", - user="dnth@lundbeck.com", - account=f"lundbeck-hlu{env}", + user="user@company.com", + account=f"company-{env}", role=role, warehouse=warehouse, ) as con: From c3ef5fd602303220992ddea7dbd517f673101675 Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Tue, 3 Oct 2023 14:00:28 +0200 Subject: [PATCH 09/11] doc: backlog refinement :memo: --- methodology-development.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/methodology-development.md b/methodology-development.md index e987b15..b74b731 100644 --- a/methodology-development.md +++ b/methodology-development.md @@ -1,6 +1,14 @@ ### Development * Development Cycle [Agile](https://www.atlassian.com/agile/scrum/sprints) +#### Backlog Refinement +* Fibonacci Story Points: 1 2 3 5 8 13 20 40 100 +* Each Story should have an epic +* Example Description: + * **Background**: Who? Why? What? How? + * **Expected Outcome**: measurable change from before / after completion. + * **Dependencies**: descriptions / approval from ABC / acceptance criteria / uaser acceotance testing + ## Development Infrastructure * **[Setup Dev Machine](setup-dev-machine.md)** * **Setup Environments** From 0aaa7ee40bce18a03e3adb467581e413ad5a4106 Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Thu, 16 Oct 2025 10:20:01 +0200 Subject: [PATCH 10/11] doc: conventional commit/branch/comment Added conventions for comments, commits, and branches. --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 80ad24a..91e9688 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,10 @@ ## [DannyDannyDanny/Methodology](https://github.com/DannyDannyDanny/methodology/) +Conventions: +* [conventional comments](https://conventionalcomments.org/) +* [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) +* [conventional branch](https://conventional-branch.github.io/) + This guide focuses on designing, developing ~~and delivering~~ Data-Processing Sytems, written primarily in python, deploying to Linux servers, optionally communicating with external interfaces (APIs or DBs). The repo is actively being updated; _bookmark_ :bookmark: it, leave a _Star_ :star: or _Watch_ :eye: the repo. * [repo roadmap](#roadmap) From 0643fe969d754809c70ac4f0b2ccbb8725987540 Mon Sep 17 00:00:00 2001 From: Daniel Thoren Date: Tue, 18 Nov 2025 16:05:49 +0100 Subject: [PATCH 11/11] chore: minimize outdated page :fire: --- setup-dev-machine.md | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/setup-dev-machine.md b/setup-dev-machine.md index 1e30913..096f6fb 100644 --- a/setup-dev-machine.md +++ b/setup-dev-machine.md @@ -1,17 +1,4 @@ # Dev Machine -๐Ÿšง I'm currently in the process of [switching to Linux](https://github.com/DannyDannyDanny/methodology/issues/5). - -๐Ÿงค Get comfortable with these tools on a system you are familiar with like MacOS or Windows (via Windows-Subsystem-Linux). - - -| Tool |Linux-Compatible | MacOS-compatible | Description | -|--- |--- | --- | --- | -| [`zsh`](https://www.zsh.org/) + [omz](https://github.com/ohmyzsh/ohmyzsh) | โœ… | โœ… | Preferred Shell + Configurator | -| [`lf`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI file browser | -| [`vim`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI text editor | -| ~[`mutt`](http://www.mutt.org/)~ | โœ… | โœ… | ~CLI email client~ (just use a browser for now) | - -๐Ÿ”– Further reading: -* [Unix as IDE](https://blog.sanctum.geek.nz/series/unix-as-ide/) -* [LARBS progs.csv](https://github.com/LukeSmithxyz/LARBS/blob/master/progs.csv) +This page is made redundant by: [DannyDannyDanny/dotfiles](https://github.com/DannyDannyDanny/dotfiles) +> TODO: delete this page