10 KiB
10 KiB
Document Roadmap
- Sections by Project Phase (setup, develop, deploy, deliver)
- Each section has heading, sections-link and at least one bullet point
- Each section should covers fixed topics (version control, systems management, docuementation)
- Information that doesn't fit into phases goes to Topic Hierarchy
- Add Custom utils.
Sections
- Setup (Version Control, Environments, Documentation)
- Develop (Version Control, Server Management, Documentation)
- Deploy
- Deliver?
Develop
-
Setup Environments
- You should be able to run the code and develop without a dedicated server. Get virtualized with docker-compose.
- Deploy using Docker + Docker-Compose. Docker keeps all your application events separate from OS events.
- Add to
.bashrcon remote server (remove or replaceENV.): alias doco="docker-compose -f docker-compose.ENV.yml"; cd /opt/project/
- Add to
-
Setup Version Control
- Connect to your repos using ssh.
- Setup git credentials
git config --local user.email (initialis)@(client comain)
- Install and use pre-commit hooks on your dev machine
- Setup Travis-CI build testing
- Add an
.EditorConfigfile
-
Documentation
- Your repo should have a one-liner for isntallation andd running: (docker-compose.yml +
doco up)
- Your repo should have a one-liner for isntallation andd running: (docker-compose.yml +
Python Developement
- Python Standard Library:
- External Modules:
- Writing good python code:
- Avoid code smells
- pipx or pyenv for environment management. If the machine has python3.x installed, use venv from the standard library.
- Loguru for logging
Develop with Version Control
# Git Log A DOG acronym
git log --all --decorate --oneline --graph
# when you have uncommitted changes
git add --patch <pattern>
# check what you've staged
git diff --cached
# NEW DEVELOPMENT FLOW
# clone repo with local_name enables several clones
git clone <repo> <local_name>
# optional: checkout (i.e switch to) a branch to continue developing from there
git checkout <feature/featurename>
# optional if you switched branch on the last step
# switched branch: branch out from
git checkout <feature/newfeature>
git branch <feature/newfeature>
git push --set-upstream origin <feature/newfeature>
# delete branch locally (-d: normal, -D: force)
git branch -D <branch_name_1> <branch_name_2> <branch_name_1>
# delete remote
git push origin -d <branch_name>
# Comparing / Merging two branches
git merge --no-commit --no-ff <feature/otherfeaturebranch>
# get file from another branch
git checkout <otherbranch> -- path/to/dir
# delete branches not present on remote
git remote prune origin
# see all commits that affected somefile
git log --follow -- filename
# search code in all history
git log -S "some_old_varname" --oneline --all
Submitting multiple PRs
Context:
- submit PR for feature1
- from feature1 branch make new branch feature2
- commit and push feature2 changes
- submit PR
- Both PRs approved
# there should be one or more flows that account for cases where:
# -> feature1 gets change requests
# -> feature1 is rejected and made obsolete
Project Infrastructure
| Project Infrastructure | Example |
|---|---|
| Issue / Project tracking | Jira |
| Code repository | Github / RedMine |
| Knowledge Base / Document Center | Confluence / RedMine |
| Docker registry1 | Docker Hub / Harbor / AWS Contanier Registry |
Topic Hierarchy
- Documentation
- Use Google's Markdown Style Guide for written documentation.
- Conceptual graphical documentation as inline embedded images.
- Save Matplotlib / Seaborn plots as
.svg. - Sketches and handwritten illustrations scanned / photographed as
.png/.jpg. - Scanned / photographed illustrations should be cropped, contrast and alligned.
- DevOps
- Do CodeReviews
- Use scrum methodology
- Test and Production environments must be the same and very well defined: FreeBSD image or Ubunut Docker container
- Logs:
- Application logs
docker-compose logs - Linux-based OS logs:
/var/log/messages - ssh logs
/var/log/audit/audit.log
- Application logs
- UX/UI
- Visualization
- problem: it's difficult to include visualization in this workflow but essential to fast prototyping.
- Perhaps text-to-diagram
- AsciiFlow
- Spectral, a diverging colormap from matplotlib.
- Tikz Editor
- Matplotlib
- API
- JSON API is the easiet API - follow the rules
- Requesting - use Python-requests
- Database / DataWarehouse
- sqlAlchemy usage
- Pandas cheatsheet
- PostgreSQL (datetime functs)
- Consider SQL (but don't use
select *) - Be aware of feature casualties of large databases
- Data Science - Hierarchy of Needs
- Security
- Audit security
- Understand SSH tunnels
- Anticipatory Failure Determination
- Secret Management for APIs
- Programming Personalizations
- Code-tags (PEP-0350)
- vscode tip: turn off openDiffOnClick
- git bash solarized 🌞
- Music for programming
- OpenStack: Linting and Pre-commit Hooks
- Deploying
- Virtualize Everything! Awesome-Compose.
- AI / ML / Modelling
- open source cousin to GPL3
- Machine Learning Master is a great resource!
- Philosophy / Mindset / Principles
- the-importance-of-humility-in-software-development
- doing it right vs doing it on time
- Closed Source Software: If you cannot check what it does / how it works, do not use it for the sake of security.
- Software Design X Dieter Rams
- Collaboration / Working in a team
- SPACE (satification, performance, activity, communication, efficiency)
- Google's Four DevOp KPIs
- Details about pair programming
- Is extreme programming a thing?
- Collaboration (tools: csvbox, pyodide)
- Resolve Joel's 12 Questions
- keeping the repo nice
- keep repo nice with makefile
- Misc
- Things you should do now
- Follow naming conventions
- Follow software engineering laws
- There's some gold nuggets this reddit post
- Mac Software
-
Read about the difference between docker repos and registries ↩︎