DannyDannyDanny/Methodology

Repo Roadmap

• 2021Q1: Gitmoji -> (general) Methodology ✅
• 2021Q2: Version control methodology
• 2021Q3: Decide on sections
  • Python section
  • Organize other methodology into hierarchy
  • Split hierarchy into paragraphs / sections
• 2021Q4
  • Refine github flows
  • Add sub-pages
  • Add Custom utils.
• 2022Q1 Consultancy Method

This methodology repo is focused on Python Data Processing based projects with som external dependencies (APIs or DBs). The repo is actively being updated; Leave a start bookmark it or Watch the repo to get notificatied of updates.

Version control

Commit messages and gitmoji

Use the format :gitmoji: <commit message> in your commit messages. Only proper nouns, CamelCase and class names should have uppercases letters. Avoid using multiple gitmojis.

emoji	code	description
✨	:sparkles:	add new feature / function
🎨	:art:	refactor code / improve readibility / structure and lint flake8
🥅	:goal_net:	try/raise/handle errors or warnings
⚡	:zap:	improve performance / coverage / profiling
🐛	:bug:	fix this bug
🚚	:truck:	moving / renaming
♻️	:recycle:	work related to CI/CD, webhooks, automated tests
🚧 / 💥	:construction:/:boom:	work in progress / breaking changes
📝	:memo:	doc / logging / verbosity / code comments
🔥 / ⚰️	:fire:/:coffin:	remove (dead) code / file
👽	:alien:	changes related to APIs and Databases
📊	:bar_chart:	clean / transform / validate data
💄	:lipstick:	improve UI / UX, CLI or visualizations
📌	:pushpin:	dependency and environment versioning
🙈	:see_no_evil:	changes to .gitignore and exclusively local files
🔍	:mag:	changes in connection with PR-review change-requests

Missing a gitmoji? Get inspiration. Submit an issue (or Pull Request).

Oneliner install and run

When using git repositories, connect with ssh.

The README for your project should contain a one-liner code that dowloads, installs and runs a mini-test.

git clone https://github.com/user/repo.git cool-repo && cd cool-repo
pipenv install
pipenv run python .
#cd .. && rm -rf cool-user-repo-online

Git Flows

In progress! 🚧 Use git hooks!

# Git Log A DOG acronym
git log --all --decorate --oneline --graph

# when you have uncommitted changes
git add --patch <pattern>

# check what you've staged
git diff --cached

# NEW DEVELOPMENT FLOW
# clone repo with local_name enables several clones
git clone <repo> <local_name>

# optional: checkout (i.e switch to) a branch to continue developing from there
git checkout <feature/featurename>

# optional if you switched branch on the last step
# switched branch: branch out from 
git checkout <feature/newfeature>
git branch <feature/newfeature>
git push --set-upstream origin <feature/newfeature>

# delete branch locally (-d: normal, -D: force)
git branch -D <branch_name_1> <branch_name_2> <branch_name_1>
# delete remote
git push origin -d <branch_name>

# Comparing / Merging two branches
git merge --no-commit --no-ff <feature/otherfeaturebranch>

# get file from another branch
git checkout <otherbranch> -- path/to/dir

# delete branches not present on remote
git remote prune origin

Submitting multiple PRs

• Interactive rebase
• Rebase & Update PR

Context:

1. submit PR for feature1
2. from feature1 branch make new branch feature2
3. commit and push feature2 changes
4. submit PR
5. Both PRs approved

# there should be one or more flows that account for cases where:
# -> feature1 gets change requests
# -> feature1 is rejected and made obsolete

Python

Packages we like:

• Python Standard Library:
  • argparse to build CLI programs
• pipx or pyenv. environment management.
• Fabric for deployment over SSH

Pathlib

For navigating, reading and filesystem and file read/write.

from pathlib import Path
Path('.').iterdir()
Path('.').absolute()

Documentation

• Use Google's Markdown Style Guide for written documentation.
• Conceptual graphical documentation as inline embedded images.
• Save Matplotlib / Seaborn plots as .svg.
• Sketches and handwritten illustrations scanned / photographed as .png/.jpg.
• Scanned / photographed illustrations should be cropped, contrast and alligned.

Other methdology

• Closed Source Software: If you cannot check what it does / how it works, do not use it for the sake of security.
• Things you should do now
• Follow naming conventions
• Follow software engineering laws
• Planning
  • non-linear roadmaps
• DevOps
  • Do CodeReviews
  • Use scrum methodology
  • Test and Production environments must be the same and very well defined: FreeBSD image or Ubunut Docker container
• UX/UI
  • Understand, Ideate and Test, Implement UX.
  • Divide year into 26 fortnights labelled A-Z
  • Avoid dark patterns
• Database / DataWarehouse
  • Consider SQL (but don't use select *)
  • Be aware of feature casualties of large databases
  • Data Science - Hierarchy of Needs
• Security
  • Audit security
  • Understand SSH tunnels
  • Anticipatory Failure Determination
  • Secret Management for APIs
• Programming Personalizations
  • git bash solarized 🌞
  • Music for programming
  • OpenStack: Linting and Pre-commit Hooks
• Resolve Joel's 12 Questions
• Visualization
  • problem: it's difficult to include visualization in this workflow but essential to fast prototyping.
  • Perhaps text-to-diagram
  • AsciiFlow
  • Spectral, a diverging colormap from matplotlib.
  • Tikz Editor
• There's some gold nuggets this reddit post
• Philosophy / Mindset
  • the-importance-of-humility-in-software-development
  • doing it right vs doing it on time
• Details about pair programming
• Is extreme programming a thing?
• Collaboration (tools: csvbox, pyodide)
• Virtualize Everything! Awesome-Compose.
• open source cousin to GPL3

Sort Sections

Repo Badges

Coming soon! 🚧

Project Portfolio

• Consumption Predictor