All code for this document is located at here.
To submit a package to Neuroconductor, the author/maintainer of the package provides the GitHub URL for the package. Once the package is submitted several initial checks are conducted. These checks ensure that the package has been created correctly. After initial checks are complete, the package must be verified by email. This verification is designed to prevent spam and allow the developer to stop a package if they would like to revise the package before re-submitting.
Once the verification is complete, the package is processed. Overall, the package is copied/cloned to a remote server. Standardized Travis CI and Appveyor configuration files, specific to Neuroconductor, are added. These are to ensure that the checks performed on these services are consistent for each package. Some parameters of the package DESCRIPTION file are changed. These parameters ensure that when a package is downloaded from Neuroconductor, the correct versions of the dependent packages are used.
Next, the package is pushed to the central Neuroconductor GitHub (https://github.com/neuroconductor) and submitted to Travis CI and AppVeyor to be built and checked on multiple systems. Parameters are set to ensure that Travis CI and AppVeyor use the correct versions of Neuroconductor packages for checking and external dependencies are installed. The author of the package receives an automatic email indicating whether the package was built successfully and is integrated with Neuroconductor together with a description file containing pertinent information about the process.
We use the terminology “Stable” and “Current” to differentiate a different status of development for a Neuroconductor package. On the initial submission, after all checks are passed, the package is incorporated into Neuroconductor and deemed the Stable version. The Current version of the package is the result of nightly pulls and mirror the latest package version from the developer’s GitHub repository. This provides Neuroconductor users with a way to use the latest versions of a package and at the same time it provides the Neuroconductor platform with a safe way of checking new versions of a package against the existing set of Current Neuroconductor packages. If a Current version of a package passes all the required Neuroconductor tests, we contact the developer of the package and suggest an official re-submission to Neuroconductor. If the newly re-submitted version of the package passes the checks against the Stable Neuroconductor packages, this version is incorporated to the Stable version of Neuroconductor.
neuroc.deps packageWe have created the neuroc.deps package that perform most of the backend operations on a Neuroconductor package. It can be installed as follows:
devtools::install_github("muschellij2/neuroc.deps")The most relevant function is use_neuroc_template, which is used to make many of the changes to the package. For a specific project, you should specify the user = "neuroconductor".
DESCRIPTION fileIn order to test packages against the relevant Neuroconductor packages, we change the DESCRIPTION file. We do this in the following ways:
Remotes field. Packages are installed using the install_github function, which reads this Remotes field to install dependencies if necessary. The Remotes field modifies and overrides the locations of dependencies to be installed. If a dependency for a package is present, then a newer version of the package will not be installed unless indicated by the user or indicated a newer version is necessary in the package (by the package (>= VERSION)) syntax) in the dependencies.bioViews field to a package in case there are Bioconductor package in the dependencies, to ensure install_github looks in that repository, as per the issue hadley/devtools#1254.covr package is added to the Suggests field if not already present in the dependencies (Depends, Imports, or Suggests). This is so that code coverage can be performed.If a package depends on the ANTsR workflow, a slightly modified set of continuous integration steps are performed as that build is highly technical.
For checking R packages, we use Continuous Integration services Travis CI, which builds on Linux and OS X operating systems, and Appveyor, which builds on Windows using MinGW.
The purpose is to ensure that the package can be built, installed, and checked on the respective systems with the appropriate dependencies.
For Travis CI, we delete the developer’s .travis.yml configuration script and replace it with the one located at https://github.com/muschellij2/neuroc.deps/blob/master/inst/neuroconductor_travis.yml.
before_install:
- fname=travis_helpers.sh
- wget -O ${fname} http://bit.ly/travis_helpers
- cat ${fname}; source ${fname}; rm ${fname}
- PROJECT_NAME=neuroconductor
- remove_neuroc_packageswhich remove any packages located on Neuroconductor from the Travis machine. As caching is done, these may be present from previous builds. The travis_helpers.sh file is a set of helper bash functions that backend the ghtravis package. Most of these are changes to DESCRIPTION file, but on Travis and not the GitHub.
The command:
- install_remotes_no_deplooks at the Remotes field in the DESCRIPTION file and runs install_github(..., upgrade_dependencies = FALSE). This ensures that the Neuroconductor packages will be those with the specific commit IDs at the time of running. No old Neuroconductor packages will be present as they were removed using remove_neuroc_packages.
The environmental variable of PACKAGE_NAME is created from the DESCRIPTION file. This may be different from the repository name from the user, but will be the same repository name on Neuroconductor, as all repos are neuroconductor/PACKAGE_NAME.
- export PACKAGE_NAME=`package_name`We add the following fields to the YAML: To ensure Bioconductor packages can be installed if necessary:
bioc_required: yes
use_bioc: yesSo that we ensure that no warnings are present in the installation (similar to CRAN):
warnings_are_errors: trueThat we have a similar threshold for packages similar to CRAN:
r_check_args: --as-cranAfter running R CMD check, the 00install.out and 00check.log are printed for diagnostic purposes.
When packages are being deployed, R CMD INSTALL --build is run so that they have the standardized naming conventions. The deployment information for Neuroconductor, including GitHub encrypted keys, are added. After building, the binary distribution is uploaded to the GitHub repository when tagged (from Neuroconductor’s backend not the developer).
After deployment, we use Coveralls.io and the covr package to run code coverage. We use type = "all" so that we provide coverage of tests, vignettes, and examples:
after_deploy:
- Rscript -e 'covr::coveralls(type = "all")'We plan to add Neuroconductor badges to the README.md file.
Currently, we only formally support packages that work in *nix type of operatings systems. We will check the package for Windows as a courtesy to Windows users, but do not provide a detailed level of support.
We use the neuroconductor_appveyor.yml, which changes the PATH variable to try to replicate a Windows machine using Rtools only and not installing MinGW.
Different from the YAML from devtools::use_appveyor(), we remove the following part:
- path: '\*_*.tar.gz'
name: Bitsas could overwrite Linux builds depeneding on the naming convention on Deployment.
If you need access to a secure key, such as a GitHub Personal Acccess Token (PAT), you do not to set them in your YAML files. Specifically with GitHub, if you push a secure key to a repository, GitHub will automatically deactivate that token (this may only apply to public repositories). In order to set an environment variable, such as GITHUB_PAT for GitHub authentication, you have to change the settings on the repository on the respective CI website.
In Travis CI you have to go to: https://travis-ci.org/USERNAME/REPO/settings, then the section labeled “Environment Variables”. Put GITHUB_PAT as the name and paste your unencrypted GitHub PAT in the Value field. When you build on Travis CI, you should see:
Setting environment variables from repository settings
$ export GITHUB_PAT=[secure]in the build logs. Now you can use the environment variable GITHUB_PAT in your code.
In Appveyor you have to go to: https://ci.appveyor.com/project/USERNAME/REPO/settings, then the section labeled “Environment” and click “Add Variable”. Put GITHUB_PAT as the name and paste your unencrypted GitHub PAT in the Value field. I believe you should click the lock to encrypt it.
devtools::session_info()## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.2 (2020-06-22)
## os macOS Catalina 10.15.7
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz America/New_York
## date 2021-02-18
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib source
## assertthat 0.2.1 2019-03-21 [2] CRAN (R 4.0.0)
## cachem 1.0.4 2021-02-13 [1] CRAN (R 4.0.2)
## callr 3.5.1 2020-10-13 [1] CRAN (R 4.0.2)
## cli 2.3.0 2021-01-31 [1] CRAN (R 4.0.2)
## colorout * 1.2-2 2020-06-01 [2] Github (jalvesaq/colorout@726d681)
## crayon 1.4.1 2021-02-08 [1] CRAN (R 4.0.2)
## desc 1.2.0 2020-06-01 [2] Github (muschellij2/desc@b0c374f)
## devtools * 2.3.2 2020-09-18 [1] CRAN (R 4.0.2)
## digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.2)
## ellipsis 0.3.1 2020-05-15 [2] CRAN (R 4.0.0)
## evaluate 0.14 2019-05-28 [2] CRAN (R 4.0.0)
## fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.0.2)
## fs 1.5.0 2020-07-31 [2] CRAN (R 4.0.2)
## glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
## htmltools 0.5.1.1 2021-01-22 [1] CRAN (R 4.0.2)
## knitr 1.31 2021-01-27 [1] CRAN (R 4.0.2)
## lifecycle 1.0.0 2021-02-15 [1] CRAN (R 4.0.2)
## magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.2)
## memoise 2.0.0 2021-01-26 [1] CRAN (R 4.0.2)
## pkgbuild 1.2.0 2020-12-15 [1] CRAN (R 4.0.2)
## pkgload 1.1.0 2020-05-29 [2] CRAN (R 4.0.0)
## prettyunits 1.1.1 2020-01-24 [2] CRAN (R 4.0.0)
## processx 3.4.5 2020-11-30 [1] CRAN (R 4.0.2)
## ps 1.5.0 2020-12-05 [1] CRAN (R 4.0.2)
## purrr 0.3.4 2020-04-17 [2] CRAN (R 4.0.0)
## R6 2.5.0 2020-10-28 [1] CRAN (R 4.0.2)
## remotes 2.2.0 2020-07-21 [2] CRAN (R 4.0.2)
## rlang 0.4.10 2020-12-30 [1] CRAN (R 4.0.2)
## rmarkdown 2.6 2020-12-14 [1] CRAN (R 4.0.2)
## rprojroot 2.0.2 2020-11-15 [1] CRAN (R 4.0.2)
## rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.0.2)
## sessioninfo 1.1.1 2018-11-05 [2] CRAN (R 4.0.0)
## stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
## stringr 1.4.0 2019-02-10 [2] CRAN (R 4.0.0)
## testthat 3.0.2 2021-02-14 [1] CRAN (R 4.0.2)
## usethis * 2.0.1 2021-02-10 [1] CRAN (R 4.0.2)
## withr 2.4.1 2021-01-26 [1] CRAN (R 4.0.2)
## xfun 0.21 2021-02-10 [1] CRAN (R 4.0.2)
## yaml 2.2.1 2020-02-01 [2] CRAN (R 4.0.0)
##
## [1] /Users/johnmuschelli/Library/R/4.0/library
## [2] /Library/Frameworks/R.framework/Versions/4.0/Resources/library