Skip to content

AWS + Google Cloud #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
Mar 21, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
7e0c22f
Created AWS folder
hadiparsianNIH Feb 24, 2025
add4a25
added first submodule to AWS directory
hadiparsianNIH Feb 25, 2025
f8c73de
Finished first submodule. need to add AWS Batch Setup
hadiparsianNIH Feb 26, 2025
d3b6993
finished first submodule
hadiparsianNIH Feb 26, 2025
8834828
Github Action: Lint Notebooks
Feb 26, 2025
5041d26
changed nextflow container for first submodule
hadiparsianNIH Feb 26, 2025
b9b0565
Changed format of submodule2
hadiparsianNIH Feb 26, 2025
f730d93
Github Action: Lint Notebooks
Feb 26, 2025
e83b6fc
trying to install packages in submodule2
hadiparsianNIH Feb 27, 2025
3473616
Github Action: Lint Notebooks
Feb 27, 2025
625711a
This should work for submodule2
hadiparsianNIH Feb 27, 2025
2462c24
This has worked for submodule2
hadiparsianNIH Feb 28, 2025
5942c52
R- 4.2.2 for submodule2
hadiparsianNIH Feb 28, 2025
a6fbcde
hope this is last commit for submodule 2 technichal
hadiparsianNIH Feb 28, 2025
12ec7ef
done with submodule 2!
hadiparsianNIH Feb 28, 2025
33c1f83
Github Action: Lint Notebooks
Feb 28, 2025
30ec71e
Updated installing packages in GCP using bash script
hadiparsianNIH Feb 28, 2025
7eb1eed
Github Action: Lint Notebooks
Feb 28, 2025
749d55f
added submodule3 to aws directory
hadiparsianNIH Mar 3, 2025
47c6e63
added bash installation for submodule 3
hadiparsianNIH Mar 4, 2025
b05d0e6
Done with submodule3 in AWS
hadiparsianNIH Mar 5, 2025
7472d8b
updated package installations for submodule 1,2,3
hadiparsianNIH Mar 5, 2025
0fc918a
converted submodule4 to AWS
hadiparsianNIH Mar 6, 2025
8f8e75d
Github Action: Lint Notebooks
Mar 6, 2025
be1e34f
Done with the module part. need to work on README
hadiparsianNIH Mar 6, 2025
09b1cca
Github Action: Lint Notebooks
Mar 6, 2025
2faf4ef
Done with the module part. need to work on README
hadiparsianNIH Mar 6, 2025
0a58972
Github Action: Lint Notebooks
Mar 6, 2025
519be8c
added "JASPAR2020" installation
hadiparsianNIH Mar 6, 2025
7cb19af
Github Action: Lint Notebooks
Mar 6, 2025
f69d033
Done with this module!!!
hadiparsianNIH Mar 7, 2025
57c983b
removed yml, package version, and session info from docs
hadiparsianNIH Mar 7, 2025
b850b08
moved "quiz files" out of "docs" and then removed "docs" directory.
hadiparsianNIH Mar 7, 2025
7c22d22
Github Action: Lint Notebooks
Mar 7, 2025
3704871
changed aws machine type to ml.m5.2xlarge
hadiparsianNIH Mar 11, 2025
1fe7b59
name of executor should be "awsbatch" no matter what. Corrected it.
hadiparsianNIH Mar 12, 2025
85e5201
Updated the link to the cloud formation template
hadiparsianNIH Mar 13, 2025
9f417e1
Github Action: Lint Notebooks
Mar 13, 2025
a7223eb
updated cloud formation templete link
hadiparsianNIH Mar 13, 2025
4762810
Github Action: Lint Notebooks
Mar 13, 2025
643b154
Increased "Volume size" in README to 100 GB
hadiparsianNIH Mar 14, 2025
45f69e8
Github Action: Lint Notebooks
Mar 14, 2025
1b81e21
modified submodule1 and 2 to add more clarification to nextflow.confi…
hadiparsianNIH Mar 17, 2025
6466261
Github Action: Lint Notebooks
Mar 17, 2025
4d77c17
fixed typos, miss spelling integration
hadiparsianNIH Mar 17, 2025
f3a6b22
Fixed typos
jghanaim04 Mar 17, 2025
7bd2570
Fixed typos
jghanaim04 Mar 17, 2025
da70f50
created directory "04-Bonus-New-Data" in AWS and GoogleCloud
hadiparsianNIH Mar 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
813 changes: 813 additions & 0 deletions AWS/01-RNA-Seq/RNA-seq.ipynb

Large diffs are not rendered by default.

31 changes: 31 additions & 0 deletions AWS/01-RNA-Seq/install_rna_seq_packages.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash

# Create a conda environment
conda create -n r-rna-seq r-base=4.3.3 -y

# Consider addressing your conda initialization instead.
source ~/.bashrc

# Activate the Conda Environment
conda activate r-rna-seq

# Install packages
conda install -c conda-forge -c bioconda bioconductor-deseq2 -y

conda install r-hexbin -y

conda install conda-forge::r-tidyverse -y

conda install bioconda::bioconductor-bsgenome -y

conda install bioconda::bioconductor-vsn -y

conda install bioconda::bioconductor-genomation -y

conda install bioconda::bioconductor-genomeinfodb -y

R -e 'install.packages(c("NMF","IRkernel"), repos="http://cran.rstudio.com/")'


# Install the kernel specification for Jupyter
R -e 'IRkernel::installspec(name = "R-RNA-Seq", displayname = "R-RNA-Seq")'
21 changes: 21 additions & 0 deletions AWS/01-RNA-Seq/rnaseq-aws.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
plugins {
id 'nf-amazon'
}

profiles {
aws {
process {
executor = 'awsbatch'
queue = 'nextflow-batch-job-queue' // Name of your Job queue
container = 'quay.io/nextflow/rnaseq-nf:v1.1'

}
workDir = 's3://your_bucket_name/rna-tmp/' // path of your working directory
params.outdir = 's3://your_bucket_name/rna-outputs/' // path of your output directory

fusion.enabled = true
wave.enabled = true
aws.region = 'us-east-1' // YOUR AWS REGION

}
}
1,000 changes: 1,000 additions & 0 deletions AWS/02-RRBS/RRBS-downstream.ipynb

Large diffs are not rendered by default.

27 changes: 27 additions & 0 deletions AWS/02-RRBS/install_rrbs_packages.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash

# Create a conda environment
conda create -n r-rrbs r-base=4.3.3 -y

# Consider addressing your conda initialization instead.
source ~/.bashrc

# Activate the Conda Environment
conda activate r-rrbs

# Install packages
conda install -c conda-forge r-data.table=1.16.4 -y
conda install bioconda::bioconductor-methylkit -y

conda install bioconda::bioconductor-genomicranges -y

conda install bioconda::bioconductor-genomation -y

conda install conda-forge::r-ggforce -y

conda install conda-forge::r-tidyverse -y

R -e 'install.packages(c("IRkernel","ggplot2"), repos="http://cran.rstudio.com/")'

# Install the kernel specification for Jupyter
R -e 'IRkernel::installspec(name = "R-RRBS", displayname = "R-RRBS")'
21 changes: 21 additions & 0 deletions AWS/02-RRBS/rrbs-aws.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
plugins {
id 'nf-amazon'
}

profiles {
aws {
process {
executor = 'awsbatch'
queue = 'nextflow-batch-job-queue' // Name of your Job queue
container = 'quay.io/nextflow/rnaseq-nf:v1.1'

}
workDir = 's3://nextflow-bucket-test/meth-tmp/' // Path of your working directory
params.outdir = 's3://nextflow-bucket-test/meth-outputs/' // Path of your output directory

fusion.enabled = true
wave.enabled = true
aws.region = 'us-east-1' // YOUR AWS REGION

}
}
1,840 changes: 1,840 additions & 0 deletions AWS/03-Integration/Integration.ipynb

Large diffs are not rendered by default.

62 changes: 62 additions & 0 deletions AWS/03-Integration/install_integration_packages.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#!/bin/bash

# Create a conda environment
conda create -n r-integration r-base=4.3.3 -y

# Consider addressing your conda initialization instead
source ~/.bashrc

# Activate the Conda Environment
conda activate r-integration

# Install packages

conda install conda-forge::r-pvclust -y

conda install conda-forge::r-ggnewscale -y

conda install conda-forge::r-ggridges -y

conda install conda-forge::r-europepmc -y

conda install conda-forge::r-ggseqlogo -y

conda install bioconda::bioconductor-methreg -y

conda install bioconda::bioconductor-bsgenome -y

conda install bioconda::bioconductor-clusterprofiler -y

conda install bioconda::bioconductor-pathview -y

conda install bioconda::bioconductor-enrichplot -y

conda install bioconda::bioconductor-repitools -y

conda install bioconda::bioconductor-rnaagecalc -y

conda install bioconda::bioconductor-motifmatchr -y

conda install bioconda::bioconductor-tfbstools -y

conda install conda-forge::r-doparallel -y

conda install bioconda::bioconductor-bsgenome.hsapiens.ucsc.hg38 -y

conda install -c conda-forge r-data.table=1.16.4 -y
conda install bioconda::bioconductor-methylkit -y

conda install bioconda::bioconductor-genomation -y

conda install bioconda::bioconductor-deseq2 -y

conda install conda-forge::r-tidyverse -y

conda install bioconda::bioconductor-motifstack -y



R -e 'install.packages(c("IRkernel"), repos="http://cran.rstudio.com/")'

# Install the kernel specification for Jupyter
R -e 'IRkernel::installspec(name = "R-Integration", displayname = "R-Integration")'
Loading
Loading