diff --git a/README.md b/README.md index 91e9688..86b02ef 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,8 @@ ## [DannyDannyDanny/Methodology](https://github.com/DannyDannyDanny/methodology/) -Conventions: -* [conventional comments](https://conventionalcomments.org/) -* [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) -* [conventional branch](https://conventional-branch.github.io/) - This guide focuses on designing, developing ~~and delivering~~ Data-Processing Sytems, written primarily in python, deploying to Linux servers, optionally communicating with external interfaces (APIs or DBs). The repo is actively being updated; _bookmark_ :bookmark: it, leave a _Star_ :star: or _Watch_ :eye: the repo. * [repo roadmap](#roadmap) - * ๐Ÿ”ด๐ŸŸก๐ŸŸข [example process roadmap](example_progress_roadmap.md) * [gitmoji](#gitmoji-reference) * [planning](project_planning_contracting.md) * ideation / design (coming soon) @@ -18,9 +12,8 @@ This guide focuses on designing, developing ~~and delivering~~ Data-Processing S * Deployment (coming soon) * Launch (coming soon) * Maintenance (coming soon) -* [cloud engineering](cloud-engineering.md) * [data management](data_storage.md) -* [tools](tools.md) :construction: +* [tools](tools.md) (:construction: under construction :construction:) * [philosophy](philosophy.md) * [scripts](scripts/) * [emoji_shortlist](scripts/emoji_shortlist.py) diff --git a/cloud-engineering.md b/cloud-engineering.md deleted file mode 100644 index b9556ea..0000000 --- a/cloud-engineering.md +++ /dev/null @@ -1,3 +0,0 @@ -# Cloud Engineering - -* [general guidance](https://www.lockedinspace.com/posts/001.html) diff --git a/example_progress_roadmap.md b/example_progress_roadmap.md deleted file mode 100644 index 006a578..0000000 --- a/example_progress_roadmap.md +++ /dev/null @@ -1,44 +0,0 @@ -## Datasets - -Which datasets are in the cloud? - -๐Ÿ”ด - dataset in cloud -๐ŸŸก - dataset migration in progress -๐ŸŸข - dataset entirely migrated - -* Dataset1 -* ๐ŸŸข Dataset2 -* ๐ŸŸข Dataset 2 -* ๐ŸŸก Dataset 3 -* ๐Ÿ”ด Dataset 4 -* ๐Ÿ”ด Dataset 5 -* ๐Ÿ”ด Dataset 6 -* ๐Ÿ”ด Dataset 7 - -## Projects - -What projects are fully in the cloud? - -๐Ÿ”ด - project not in cloud -๐ŸŸก - project migration in progress -๐ŸŸข - project entirely in cloud - -* ๐ŸŸข Project 1 -* ๐ŸŸก Project 2 -* ๐ŸŸก Project 3 -* ๐Ÿ”ด Project 4 -* ๐Ÿ”ด Project 5 - -## Users - -Who is in the cloud? - -๐Ÿ”ด - no projects in cloud -๐ŸŸก - some projects in cloud -๐ŸŸข - all projects in cloud - -* ๐ŸŸข User 1 -* ๐ŸŸข User 2 -* ๐ŸŸก User 3 -* ๐ŸŸก User 4 -* ๐Ÿ”ด User 5 diff --git a/index.md b/index.md index 7e8bee9..0e1749e 100644 --- a/index.md +++ b/index.md @@ -1,2 +1,2 @@ -# Index For methodology/VIMWIKI +# Index For VIMWIKI * [readme.md](README.md) diff --git a/methodology-development.md b/methodology-development.md index b74b731..46c8ffb 100644 --- a/methodology-development.md +++ b/methodology-development.md @@ -1,14 +1,6 @@ ### Development * Development Cycle [Agile](https://www.atlassian.com/agile/scrum/sprints) -#### Backlog Refinement -* Fibonacci Story Points: 1 2 3 5 8 13 20 40 100 -* Each Story should have an epic -* Example Description: - * **Background**: Who? Why? What? How? - * **Expected Outcome**: measurable change from before / after completion. - * **Dependencies**: descriptions / approval from ABC / acceptance criteria / uaser acceotance testing - ## Development Infrastructure * **[Setup Dev Machine](setup-dev-machine.md)** * **Setup Environments** @@ -25,7 +17,6 @@ * Setup [Travis-CI build testing](https://docs.travis-ci.com/user/customizing-the-build/#building-specific-branches) * Add an [`.EditorConfig` file](https://editorconfig.org/) * Add [git commit template](https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733#file-git-commit-template-md) - * Branching strategy: small teams might want to adopt [trunk based development](https://trunkbaseddevelopment.com/) * **Documentation** * Your repo should have a one-liner for isntallation andd running: (docker-compose.yml + `doco up`) @@ -86,9 +77,6 @@ git log --follow -- filename # search code in all history git log -S "some_old_varname" --oneline --all - -# files changed on your branch compared to origin/main i.e. **files changed** in a Github PR -git diff --name-only origin/main ``` ### Submitting multiple PRs diff --git a/python_snippets.md b/python_snippets.md index 14230fe..592e014 100644 --- a/python_snippets.md +++ b/python_snippets.md @@ -1,46 +1,3 @@ -## database file-caching - -``` -from pathlib import Path -from snowflake.connector import connect -import pandas as pd -import hashlib - -def get_df(query, env, use_cache=True): - print(f'{query[:100] = }') - if env == "prod": - role = "COSUTMER" - warehouse = "CLOWN_COSTUMES" - elif env == "sand": - role = "CUSTOMER" - warehouse = "CUSTOMER_INFORMATION" - else: - raise ValueError(f"env must be prod or sand - recieved: {env}") - - hash_query_4_digits = hashlib.shake_128(query.encode()).hexdigest(4) - cache_pkl_path = Path(f"/tmp/cache-{env}-{hash_query_4_digits}.pkl") - print(cache_pkl_path.exists(), cache_pkl_path) - if cache_pkl_path.exists() and use_cache: - print('loading cached data') - df = pd.read_pickle(cache_pkl_path) - return df - - print("fetching & caching data") - with connect( - authenticator="externalbrowser", - user="user@company.com", - account=f"company-{env}", - role=role, - warehouse=warehouse, - ) as con: - cur = con.cursor() - cur.execute(query) - df = cur.fetch_pandas_all() - df.to_pickle(cache_pkl_path) - return df -``` - - ## Python Kernel Management ``` # install new kernel "my_new_env" diff --git a/setup-dev-machine.md b/setup-dev-machine.md index 096f6fb..1e30913 100644 --- a/setup-dev-machine.md +++ b/setup-dev-machine.md @@ -1,4 +1,17 @@ # Dev Machine -This page is made redundant by: [DannyDannyDanny/dotfiles](https://github.com/DannyDannyDanny/dotfiles) -> TODO: delete this page +๐Ÿšง I'm currently in the process of [switching to Linux](https://github.com/DannyDannyDanny/methodology/issues/5). + +๐Ÿงค Get comfortable with these tools on a system you are familiar with like MacOS or Windows (via Windows-Subsystem-Linux). + + +| Tool |Linux-Compatible | MacOS-compatible | Description | +|--- |--- | --- | --- | +| [`zsh`](https://www.zsh.org/) + [omz](https://github.com/ohmyzsh/ohmyzsh) | โœ… | โœ… | Preferred Shell + Configurator | +| [`lf`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI file browser | +| [`vim`](https://github.com/gokcehan/lf) | โœ… | โœ… | CLI text editor | +| ~[`mutt`](http://www.mutt.org/)~ | โœ… | โœ… | ~CLI email client~ (just use a browser for now) | + +๐Ÿ”– Further reading: +* [Unix as IDE](https://blog.sanctum.geek.nz/series/unix-as-ide/) +* [LARBS progs.csv](https://github.com/LukeSmithxyz/LARBS/blob/master/progs.csv)