the pipeline

layout: true

<div class="my-footer"><span>© Stefan G. Weichinger : oops! linux : @stefangw : 2019 </span></div>

---
class: center, middle, inverse

# ContinuWHAT?

## CI/CD with
<img src="https://about.gitlab.com/images/press/logo/png/gitlab-logo-1-color-white-rgb.png" width="25%" style="position: middle; top: 100px; left: 450;" />

.footnote[Stefan G. Weichinger, oops linux, 2019]

---

# about me

Stefan G. Weichinger <<a href="mailto:office@oops.co.at">office@oops.co.at</a>>

# `@stefangw`

- expert
- linux servers since the early 90s
- giving talks, writing for magazines, books ...
- always looking for well-paid work

.bg[ ![Buster JS](https://www.oops.co.at/css/img/oops-logo.gif) ]

https://www.oops.co.at

---
class: center, middle
# ContinuWHAT?

## CI/CD with
<img src="https://about.gitlab.com/images/press/logo/png/gitlab-logo-1-color-black-rgb.png" width="25%" style="position: middle; top: 100px; left: 450;" />

### an overview by Stefan G. Weichinger

DevTreff #5, Amstetten .created-on[2019-09-04]

---
name: cicd
class: center, middle
# What is CI/CD?

---
.head[
  ## CI :: goals, practices
]

Moderne Software-Entwicklung ist verteilt, dynamisch, komplex.

Wie bringt man viele Entwickler zusammen, gewährleistet lauffähigen und fehlerfreien/armen Code?

## "integrate at least daily"

* eine gemeinsame Codebase, Integration passiert quasi immer sofort
* build automatisieren
* build mit self-tests ausstatten
* build kurz und schnell halten
* ausführbare Version bereit stellen
---
.head[
  ## CI :: definition
]

"(in software engineering) continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day." .red[*]

.footnote[.red[*] wikipedia, img from medium.com
]

---
.head[
  ## CI ::  pros/cons
]

## pros

* Code Qualität geht rauf
* very dynamic
* Transparenz, Nachvollziehbarkeit, Feedback
* Workflow beschleunigt
* Kreativität wird ermöglich, Risiko gemindert

## cons
* Lernkurve, neue Prozesse
* Kosten (je nach Umsetzung)
* Anwendung auf existierende Projekte schwieriger
* Aufbau der automatisierten Tests etc = Arbeit
* very dynamic ;-)

---
.head[
  ## CD :: definition :: Continuous Delivery
]

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time</p>— Carl Caum (@ccaum) <a href="https://twitter.com/ccaum/status/372620989257232384?ref_src=twsrc%5Etfw">August 28, 2013</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

---
.head[
  ## CD :: definition :: Continuous Deployment
]

"Continuous Deployment :: eine weitergehende Form des Continuous Delivery, bei der auch die Auslieferung der Software auf die Produktiv-Infrastruktur durchgeführt wird" .red[*]

.footnote[.red[*] wikipedia
]

---
.head[
  ## CD :: definition
]

### Continuous Delivery vs. Deployment

<img src="https://puppet.com/sites/default/files/2016-09/puppet_continuous_diagram.gif" width="90%" style="position: middle;" /> .red[*]

.footnote[.red[*] puppet.com
]

---
.head[
  ## CI/CD :: in GitLab
]
<img src="https://about.gitlab.com/images/blogimages/cicd_pipeline_infograph.png" width="90%" style="position: middle;" />

---
.head[
  ## GitLab CI/CD :: overview
]
<img src="https://docs.gitlab.com/ee/ci/introduction/img/gitlab_workflow_example_11_9.png" width="90%" style="position: middle;" />
---
.head[
  ## CI/CD :: Pipeline
]

???
A pipeline consists of

- stages
- jobs
---
.left-column[
  ## About GitLab
  ### What is GitLab?
]
.right-column[
GitLab is an open source end-to-end software development platform with

- built-in version control
- issue tracking
- code review
- CI/CD
- ... and much more.

There's a Community Edition (CE) and a Enterprise Edition (EE).

Use it on premise or hosted (gitlab.com).

]

---

.left-column[
  ### GitLab CI/CD :: the Runner
]

.right-column[
#### What?
A Runner is an instance doing build jobs.

#### Where?
A Runner can be a virtual machine, a VPS, a bare-metal machine,
a docker container or even a cluster of containers.

#### How?
A runner might be set up to be specific for certain projects, groups or tags.
And it is running on a separate machine, not on the GitLab Server.

]

???

runners are separate workhorses
http API between gitlab server and runners
Shared runners vs. private or project runners

---
.head[
  ## Setup a Project :: Create the project in gitlab
]

standard git-procedure ...

```shell
$ git clone https://gitlab.com/stefangweichinger/hello-world
$ cd hello-world
# create great code here
$ git add. ; git commit -m "my great work"; git push
```

Reload the project page and check content.
---
.head[
  ## CI/CD :: enable the magic
]
The CI is configured by creating a `.gitlab-ci.yml` in the root path of your project:

### Add a build job to the CI
```yaml
variables:
  IMAGE_NAME: gitlab:5005/sgw/hello-world

before_script:
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

build_latest:
  only:
    - master
  tags:
    - docker-builder
  script:
    - docker build -t $IMAGE_NAME:latest .
    - docker push $IMAGE_NAME:latest
```
`add`, `commit` and `push` this file to the repository.

Watch the build live on your GitLab and check the Registry for the built image.
---
.head[
  # Run your Docker Image
]
Authenticate to the registry:

```bash
$ docker login gitlab:5005
```

Then run your image:

```bash
$ docker run -p 3000:3000 gitlab:5005/sgw/hello-world:latest
```

Access the application in your web browser:

http://gitlab:3000

???

crux here : ports, IPs, DNS, names,

---
class: center, middle, inverse
 # that was the easy part
---
.head[
  ## .gitlab-ci.yml :: more details
]

```yaml
image: docker:18

services:
- docker:dind

before_script:
  - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY

stages:
  - test
  - build

test:
  image: python:3-alpine
  stage: test
  before_script:
    - ''
  script:
    - pip install pytest
    - pip install -r requirements.txt
    - pytest --version
    - pytest helloworld.py
  allow_failure: true

build:
  stage: build
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - > docker build --pull --build-arg VCS_REF=$CI_COMMIT_SHA
      --build-arg VCS_URL=$CI_PROJECT_URL --cache-from $CI_REGISTRY_IMAGE:latest
      --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
```
---

.left-column[
### ... for the runaways
]

.right-column[
some other things:

- you will need routing and SSL/TLS ... something like [traefik](#traefik)
- GitLab docker registry: push/pull generated images
- CD: deploy to staging and production : use [environments](#environments)
- speed up things with [caching](#caching)
]

---
.left-column[
  ### best practises
  #### use variables = avoid hardcoding
]

.right-column[
GitLab offers variables for nearly(?) everything

```bash
export CI_JOB_TOKEN="abcde-1234ABCD5678ef"
export CI_REGISTRY="registry.example.com"
```

```yaml
# .gitlab-ci.yml
before_script:
  - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
```
]
---
.left-column[
  ### best practises
  #### use variables = avoid hardcoding
]

.right-column[
for example for tagging the docker images:

```bash
export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a"
export CI_COMMIT_SHORT_SHA="1ecfd275"
export CI_COMMIT_REF_NAME="master"
export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce"
export CI_REGISTRY="registry.example.com"
export CI_REGISTRY_USER="gitlab-ci-token"
export CI_REGISTRY_PASSWORD="longalfanumstring"
export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce"
```

```yaml
# .gitlab-ci.yml
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    # Then we tag it "latest"
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    # And we push it.
    - docker push $CI_REGISTRY_IMAGE:latest
```
]
---
name: caching
.left-column[
  ### best practises
  #### use caches
]

.right-column[
You can define which paths you want to cache for the next stage(s):

```yaml
example-job:
  stage: example
  script:
    - npm install
  cache:
    paths:
      - node_modules/
```

For bigger environments you can run a separate cache server and share cache between multiple runners.
]

---
.left-column[
  ### best practises
  #### use small docker images ... and cache them
]

.right-column[
instead of rebuilding everytime from scratch

```code
docker build --tag $CONTAINER_IMAGE:$CI_BUILD_REF .
```

you could benefit from the already available docker image layers via

```bash
docker build\
  --cache-from $CONTAINER_IMAGE:latest\
  --tag $CONTAINER_IMAGE:$CI_BUILD_REF\
  --tag $CONTAINER_IMAGE:latest\
  .
```
]

---
name: yaml_templates
.left-column[
  ### best practises
  #### yaml templates
]

.right-column[
`.gitlab-ci.yml` grows fast and gets complex rather soon.

To avoid repeating blocks you may use yaml templates, for example:

.pull-left[

```yaml
.job_template: &job_definition
  image: ruby:2.1
  services:
    - postgres
    - redis

test1:
  <<: *job_definition
  script:
    - test1 project

test2:
  <<: *job_definition
  script:
    - test2 project
```
]
.pull-right[

```yaml
.job_template:
  image: ruby:2.1
  services:
    - postgres
    - redis

test1:
  image: ruby:2.1
  services:
    - postgres
    - redis
  script:
    - test1 project

test2:
  image: ruby:2.1
  services:
    - postgres
    - redis
  script:
    - test2 project
```
]

]
---
name: environments
.left-column[
  ### best practises
  #### environments
]

.right-column[
*Environments* are like tags for your CI jobs, describing where code gets deployed.

Deployments are created when jobs deploy versions of code to environments, so every environment can have one or more deployments.

GitLab provides a full history of your deployments for each environment.

GitLab keeps track of your deployments, so you always know what is currently being deployed on your servers.
]

---
name: traefik
.left-column[
  ### best practises :: reverse proxy :: traefik
]

.right-column[
<img src="https://docs.traefik.io/img/internal.png" width="75%" style="position: fixed; top: 160px; left: 150;" />
]
---
name: demo
.head[
  ## a small demo
]

demo repository at gitlab.com : [hello-world](https://gitlab.com/stefangweichinger/hello-world)

???

small python script
CI builds docker image, pushes to project repo

2 environments

---

# Thanks!

For more information please contact:

Stefan G. Weichinger <<a href="mailto:office@oops.co.at">office@oops.co.at</a>>

# `@stefangw`

- expert
- linux servers since the early 90s
- giving talks, writing for magazines, books ...
- always looking for well-paid work

.bg[ ![Buster JS](https://www.oops.co.at/css/img/oops-logo.gif) ]

---

---
.left-column[
  ## appendix
  ### credits
]

.right-column[
### presentation

done with [remark](https://github.com/gnab/remark)

watch @GitLab :: https://stefangweichinger.gitlab.io/gitlab-talk/

slides: https://gitlab.com/stefangweichinger/gitlab-talk

done by Stefan G. Weichinger <<a href="mailto:office@oops.co.at">office@oops.co.at</a>>

### links

* consult your local search engine ... hundreds of articles and howtos ...
]
---
name: last-page
class: center, middle, inverse

## That's all, folks!