========
Releases
========

Schedule
========

New versions of NSCLI were typically released each month around the 15th, but due to support challenges we currently 
release on an as-needed basis. 

Versioning
==========

NSCLI uses an implementation of the `CalVer <http://calver.org/>`_ version number standard. Our specific implementation
is described by ``scripts/versioning.py``.

Publishing
==========

NSCLI releases are published on our `github <https://github.rackspace.com/NSI/nscli/releases>`_. They are also mirrored 
on Artifactory, which is where most users get them.

Artifactory Releases
--------------------

:Python: https://artifacts.rackspace.net/artifactory/nscli-pypi-local/nscli/
:Homebrew: https://artifacts.rackspace.net/artifactory/nsi-bottles-local/
:Debian: https://artifacts.rackspace.net/artifactory/nscli-debian-local/pool/

Process
=======

This section contains notes about our current release process, and is a working document that is subject to change at 
any time.

Making Releases
---------------

Prerequisites/Setup
^^^^^^^^^^^^^^^^^^^

- Have `PDM <https://pdm.fming.dev/>`_ installed. 

    - As of Jul 27th, 2023 - we have to stay on v2.7.4.
    - constraints.txt creation (scripts/generate_constraints.py) breaks due to missing section in pdm.lock.
    - Potential solution is to use `pdm lock --static-urls`. REF : https://github.com/pdm-project/pdm/pull/2101/files 

- GPG Key
    
    - Ensure you have a GPG key. Create one if needed.
        - gpg --list-keys (Check for gpg keys)
            - If ``gpg`` command is missing:
                - brew install gnupg
        - Create a GPG key
            - `Create GPG key <https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key>`_
    - Ensure you have your gpg configured on github.
        - `GIT GPG Config 1 <https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account>`_
    - Ensure git is configured with your gpg key. 
        - `GIT GPG Config 2 <https://docs.github.com/en/github/authenticating-to-github/telling-git-about-your-signing-key>`_

- YOU MUST HAVE NSCLI TAPPED IN HOMEBREW

    - `SupportTools Homebrew Repo <https://github.rackspace.com/SupportTools/homebrew-rs>`_

- Set your system python to 3.9.X

    - `Install pyenv <https://github.com/pyenv/pyenv#installation>`_ 
    - ``pyenv install 3.9.X``
    - ``pyenv global 3.9.X``

- Fork NSCLI and set a git remote of NSI/nscli as NSI::

    git remote add NSI git@github.rackspace.com:NSI/nscli.git

- Ensure you have gh-pages as a local branch tracking NSI/gh-pages ::

    git fetch --all
    git branch gh-pages NSI/gh-pages
            
    ! Validate the status of your gh-pages branch with ::

    git branch -vv | grep gh-pages
    
    ! It should look like this:
        - gh-pages                                58b4c5576 **[NSI/gh-pages]** rebuild docs

- Ensure you have master as a local branch tracking NSI/master ::

    git fetch --all
    git branch master NSI/master

    ! Validate the status of your master branch with

        git branch -vv | grep master
    
    ! It should look like this:
        - master                                ccdfb27f9 **[NSI/master]** Merge pull request #2904 from NSI/develop


Few days before release
^^^^^^^^^^^^^^^^^^^^^^^

**OPTIONAL** - (needs more testing) Update the lockfile with latest compatible packages ::

    deactivate
    make distclean
    git fetch --all
    git checkout develop
    git pull NSI develop 
    make pdm_lock

Two days before release
^^^^^^^^^^^^^^^^^^^^^^^

E-mail nscli@rackspace.com and let them know there will be a release soon, so they can finish any code they're in the 
middle of.

One day before release
^^^^^^^^^^^^^^^^^^^^^^

Perform a ``brew update && brew upgrade`` to avoid waiting for package upgrades on release day.

Release day
^^^^^^^^^^^^^^^^^^

1. Try to merge or reject any outstanding PRs on the development branch. If any PRs need to stay open past release, make 
   sure the authors know about it.

2. Pull down the development branch and build a release env for testing.::

    deactivate
    make distclean
    git fetch --all
    git checkout develop
    git pull NSI develop 
    git log
    git status
    make release_env
    source .venv/bin/activate
    nscli -v

3. **- SKIP -** *(pending tests review & validation)* Run the tests. Make sure they pass. ::

    pytest nscli/tests

4. Check the basic functionality of the develop branch:

    - Run ``nscli info`` on a device
    - SSH to a network bastion
    - SSH to a switch
    - SSH to a customer bastion
    - SSH to a firewall
    - SSH to an F5 or other device GUI

    Lab Devices to test against:

    - ASA 349763
    - F5 689680
    - WAF 863021
    
::

    nscli info 349763
    nscli netbast fra1
    nscli netbast ord1
    nscli switch C7-10-1E.ORD1
    nscli cbast dfw1
    nscli cbast syd2
    nscli ssh 349763
    nscli ssh 689680
    nscli ssh 689680 -g
    nscli ssh 863021
    nscli ssh 863021 -g

5. Clean up the release env and rebuild with a build env::

    deactivate
    make distclean
    make build_env
    source .venv/bin/activate

6. If you need to fix anything, merge it into the development branch and repeat steps 1-4 until everything works. 
   **You don't need to make a PR yet. Any changes can be merged alongside later commits in this process.** 

7. Make sure the nscli version is up-to-date in ``nscli/__init__``

8. Make sure the documentation is up-to-date in the development branch, especially ``Commands``

    - To find what has changed since the last release, you can do::
      
        git log -p $(git tag | tail -n 1)..
    
    - see ``man git-log`` for lots of filtering options.
    - Run ``make webpage`` and take a look at the generated page; make any adjustments needed to formatting in the 
      source docs::
        
        make webpage
    
    - Review ``docs/src/newsfragments``. If there are any news file still in here, troubleshoot why. Maybe they have
      the wrong extension?
      **Troubleshooting**::
    
        make -q docs/src/Changelog.rst  
        echo $?
        # 0 means up to date
        touch docs/src/newsfragments/<pickone>
        make -q docs/src/Changelog.rst
        echo $?
        # 1 means not up to date
        make webpage
        
        tl;dr:
            touch docs/src/newsfragments/<pickone>
            make webpage
    
    - Review HTML formatting and modify rst files as needed.
    
        ::
        
            open docs/gh-pages/index.html
    
    - **Note:** running ``make webpage`` is likely to stage a commit for the updates to the Changelog. Make sure this is 
      committed before proceeding.
    
        ::
        
            git status
            git add <filenames> (if you made additional changes)
            git commit -m "Rebuilds changelog vXX.X.XX"
        
9. Merge any documentation changes into the development branch. Make a PR to NSI/develop. ::

    git push
    ! create a PR from <your_fork>/develop -> NSI/develop
    
    
10. Deactivate from any applied virtual env ::

    deactivate

11. Create a pull request to merge the development branch into ``master``. Have someone from the NSI team review, 
    approve, and merge.

    - Title: ``NSCLI vYY.M.D``
    - kill ``pr-merge`` jenkins job
    - wait for ``branch`` job to finish, then "build now" on cancelled ``pr-merge``
    - merge into ``master``
    
12. Pull down ``master``::

    git checkout master && git pull --rebase git@github.rackspace.com:NSI/nscli.git master 

13. Export the environment variables that the Makefile needs in order to proceed.::

    export API_TOKEN=<ID_TOKEN>
        # Identity token for Artifactory
        # Found in artifactory > username > Identity Tokens
    export OAUTHTOKEN=<TOKEN>
        # For Rackspace GitHub
        # Found in github > settings > developer settings > personal access tokens

14. Clean up existing brew/nscli installation.::

    brew uninstall nscli && brew cleanup

15. Clean up potentially old nscli docker images and login to docker.

    - **Note:** Login credentials will be your SSO
    - **WARNING:** This will DELETE ALL local images. Upload / save them before running. 
    - Potentially update the command to only delete "NSCLI" images with "filter" 

    ::

    docker system prune -af && docker login docker.artifacts.rackspace.net && docker login nsi-docker-local.artifacts.rackspace.net

16. Update brew::

    brew update && brew upgrade

17. Clean up and build a new release environment to define a list of prod packages.::

    make distclean && make release_env

18. Create a list of prod packages in pdm_list.::

    make pdm_list
    
19. Create a new 'build' environment used for the remaining build process.::

    make build_env

20. Make the python sdists::

    make dist

21. Make the final copy of webpages.::

    make webpage

22. Upload the python sdist. This "deploys" a new nscli version.::

    ! check the changelog.html, verify all updates are present

    make upload

23. Build the debian packages. ::

    make -j 3 debian

24. Validate there are 3 debian packages built in ``./dist/debian``, then deploy the debian packages 
    - **Note:** If there are not 3 builds, investigate errors during step 23 and ensure you delete those docker images
    before rebuilding in step 23. ::

    make debian_upload

25. Homebrew

    1. Person1(arm64) - Build the nscli brew formula. :: 

        make brew_formula

    2. Have Person2(x86_64) catch up with setting their build_env, then hand over the generated Formula/nscli.rb to Person2(x86_64) and place it in Formula/nscli.rb.:: 

        ! Person2: Make sure you are on NSI master / develop (with latest changes pulled) 
        ! Person2: Create following directories at root (same folder as Makefile), if missing:  dist, Formula

        brew uninstall nscli && brew cleanup
        make distclean
        make build_env
        > copy Person1's(arm64 Mac) formula file to Formula/nscli.rb
        > MAKE SURE PERSON2: creates a "dist" FOLDER in root nscli folder

    3. Person1 & Person2 - Make a new bottle of nscli::
    
        make bottle
    
    4. Once the bottle is built, copy the sha256 from the ``bottle do`` from the formula and have Person1 manually insert it into their formula, nscli.rb, that will be uploaded.

        Sample Complete DSL ``bottle do``::
        
            bottle do
                root_url "https://artifacts.rackspace.net/artifactory/nsi-bottles-local"
                sha256 big_sur: "a3c9a00f97426019a60b37030402fc0f8968cd2500a565664b3257f59ad08cd8"
                sha256 cellar: :any, arm64_monterey: "f8627df9323b4db4c75d288610940e2dedc5424428ea94cf9391d1bcadc8e0e1"
            end

    5. Making the bottle will install nscli. Open a new terminal window and make sure the installed nscli works.::

        nscli -v
        nscli ssh 349763

    6. Upload the bottle to artifactory, and the new Formula to github. 
    
        - Person1(arm64)::
        
            make bottle_upload formula_upload
    
        - Person2(x86_64)::
        
            make bottle_upload
 
    7. Steps for Person2(x86_64) are complete. Remaining steps are done by Person1.

    8. Upload `/Formula/nscli.rb` file to mirror, so it can be used for "non Tap" installation, if needed.

         - Create a version specific sub-folder under content root:
            /home/netsec.devops/content/nscli/Formula/23.7.27/

        - Upload 'nscli.rb' file: 
            e.g: 
            scp /Formula/nscli.rb netsec.devops@files.lp.rackspace.com:/home/netsec.devops/content/nscli/Formula/23.7.27/


26. Upload the new web docs with ``make webpage_upload`` and review them at https://pages.github.rackspace.com/nsi/nscli/ ::

    make webpage_upload

27. Ensure someone merges the changes for the uploaded formula in the ``SupportTools/homebrew-rs`` repo. 
28. Tag ``master`` with the new version. The version number format is 'v' + the number, like ``v21.1.14``. Example: 
    ``git tag -s vYY.M.D master``. Push the signed tag to GitHub: ``git push NSI --tags`` ::

    git tag -s vYY.M.D master -m "NSCLI vYY.M.D" && git push NSI --tags

29. Create a new release in GitHub. Copy/paste the release notes from ``docs/src/Changelog.rst`` into the release notes. 
    Upload the new tarball and wheel as attachments.

    - Click ``Releases``
    - Target: ``master``
    - Tag: ``vYY.M.D``
    - Title: ``NSCLI vYY.M.D``
    - Copy/paste from ``docs/src/Changelog.rst``
    - Drag and drop ``.tar.gz`` and ``.whl`` files from ``./dist/``
    - Click ``Publish Release``

30. Make the release email with ::

        make email

    This will create the release e-mail and send it to you to preview. To
    send the real release e-mail, run ``scripts/send_release_email.py``. This currently doesn't work, so just forward
    the test e-mail, making sure to update the subject and fix the formatting.

    - **NOTE**: You will need to have a sshuttle session opened to the SMTP server.
        - First add a route to smtp.ord1.corp.rackspace.com
            - ``sudo route add 10.12.47.12/32 -iface en0``
        - Create the proxy
            - ``sshuttle -r cbast.dfw1.corp.rackspace.net smtp.ord1.corp.rackspace.com``
        - Remove the route once you've received the email
            - ``sudo route delete 10.12.47.12/32``
    - Troubleshooting:
        - make sure sso is up to date in keychain ``com.rackspace.nscli.sso``
        - If email fails, just copy/paste from ``docs/email/ReleaseEmail.html``

    To: ::

        GTS NSIA Engineers <GTS-Netsec-AE@rackspace.com>; 
    
    CC: ::

        GTS Engineering Leadership <GTS-Eng-LT@rackspace.com>; 
        GTSNetSec <GTSNetSec@rackspace.com>; 
        NSI Stakeholders <nsi_stakeholders@rackspace.com>; 
        GTS Network Defense <GTS-Network-Defense@rackspace.com>;
        NSCLI Dev <nscli@rackspace.com>; 
        GTS First Line Response - NetSec <GTSFLR-NETSEC@rackspace.co.uk>;
        Provisioning - NetSec <provisioning-netsec@rackspace.com>
        GSS Cloudflare EES <GSS-Cloudflare-EES@rackspace.com>;


    Subject: ``New NSCLI available: vYY.M.D``

    In the body, at the bottom add: ::

        BCC: GTS Engineering Leadership; GTSNetSec;  NSI Stakeholders; GTS Network Defense; NSCLI Dev; GTS First Line Response - NetSec; Provisioning - NetSec; GSS Cloudflare EES; 

31. Check out ``develop``. Bump the version number in ``nscli.__init__``. Push these changes to GitHub.::

        git checkout develop
        git log

        !! update ``__init__.py``

        git add nscli/__init__.py
        git commit -m "version bump"
        git push

        !! Create and merge a PR to ``NSI/develop``

