Merge pull request #792 from I2PC/release-3.23.07

Release 3.23.07

Author: EstrellaFG

.github/workflows/build.yml

@@ -0,0 +1,57 @@
+# Workflow name
+name: Build Xmipp
+
+# Specify when the Action should be triggered: when a pull request is opened against the 'devel' or 'master' branch
+on:
+  pull_request:
+    branches: [devel, master]
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  # This workflow contains a single job called "build"
+  build:
+    # The type of runner that the job will run on
+    runs-on: ubuntu-22.04
+
+    # Steps represent a sequence of tasks that will be executed as part of the job
+    # Disabling shallow clone is recommended for improving relevancy of reporting
+    steps:
+      # Installing apt dependencies
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libfftw3-dev libopenmpi-dev libhdf5-dev libtiff5-dev python3-numpy python3-dev libsqlite3-dev default-jdk git cmake libopencv-dev openmpi-bin
+      
+      # Setting up Python
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.9.15' # Version range or exact version of a Python version to use, using SemVer's version range syntax
+          
+      # Installing Python dependencies
+      - name: Install python dependencies
+        run: |
+           python -m pip install --upgrade pip
+           pip install scons numpy
+       
+       # Installing CUDA                
+      - uses: Jimver/cuda-toolkit@v0.2.10
+        id: cuda-toolkit
+        with:
+          cuda: '11.7.0'
+          linux-local-args: '[ "--toolkit" ]'
+      
+      # Using CCache to speed C/C++ compilation
+      - name: Install ccache
+        uses: hendrikmuhs/ccache-action@v1.2
+      
+      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+      - uses: actions/checkout@v3
+        with:
+            fetch-depth: 0
+
+      # Compiling Xmipp in the Pull Request branch
+      - name: Compile Xmipp
+        env:
+          CIBuild: 1
+          BUILD_TESTS: True
+        run: ./xmipp noAsk

.github/workflows/main.yml

@@ -0,0 +1,23 @@
+# Workflow name
+name: SonarCloud scan
+
+# Specify when the Action should be triggered: when a pull request is opened against the 'devel' or 'master' branch
+on:
+  pull_request:
+    branches: [devel, master]
+jobs:
+  build:
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Prepare Sonar scanner
+        run: |
+          wget -nv https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.6.0.2311-linux.zip
+          unzip -q sonar-scanner-cli-4.6.0.2311-linux.zip
+          echo "${PWD}/sonar-scanner-4.6.0.2311-linux/bin/" >> $GITHUB_PATH
+        working-directory: ${{ env.BUILD_DIR }}
+      - name: SonarCloud Scan
+        run: |
+          sonar-scanner -Dsonar.host.url=https://sonarcloud.io -Dsonar.organization=i2pc -Dsonar.login=$SONAR_TOKEN -Dsonar.projectKey=Xmipp
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

.github/workflows/scanBuild.yml

@@ -34,6 +34,7 @@
 .pydevproject
 .classpath
 .idea
+.vscode
 
 # Scons auxiliary files
 .sconsign.dblite
@@ -65,3 +66,8 @@ xmippEnv.json
 .devel_version
 /src/xmipp/Debug/
 /src/xmipp/applications/programs/version
+
+# Output files from compilation
+compileLOG.txt
+reportInstallation.tar.gz
+xmipp_version.txt

.gitignore

@@ -1,33 +1,64 @@
+   ## Release 3.23.07 - voting
+   ### Xmipp Programs 
+   - New programs
+      - image_peak_high_contrast(for detecting high contrast regions in tomographic reconstruction)
+      - misaligment_detection (to detect misalignment in tomographic reconstructions from high-contrast regions)
+      - deep_global_assignment
+      - deep_center
+   - Programs updated
+      - xmipp_angular_distance: new features
+      - tomo_extract_particles: new features
+      - subtract_projection: parallelization with mpi
+      - tomo_extract_subtomograms: allow downsampling of features
+      - angular_resolution_alignment:  Detect misalignment with resolution
+   - Programs fixed
+      - align_volume_and_particles: Fixed error  
+      - angular_project_library: Removed deterministic behaviour (mpi)
+      - volumen_subtraction: fixed bug
+   - Deprecated programs (For more details [visit](https://github.com/I2PC/xmipp/wiki/Deprecating-programs-and-protocols)) 
+      - classify_kmeans_2D
+      - rotational_spectra
+      - particle_boxsize
+   
+   ### Installation and user guide
+   - New clearer, more intuitive and informative installer. It also creates a file to facilitate user support.
+   - Refactored the deep learning toolkit, more robust and new environment with updated tensorflow.
+   - Updated requirement.
+     
+   ### More Xmipp 
+   - Added half precission suport to numpy
+   - Added the ability to read and write CIF files
+   - Modular design of winner filter
+   - Decoupling XmippTomo from XmippSPA
+   - Fix Zernike equation
+   - Removed all occurrences of non base-class default destructors
+   - Improved MultidimArray performance
+   - Added support for half precision floating point numbers in MRCs
+   - Assign tiff to gain files
+   - adding half maps labels  
+
+
+   
    ## Release 3.23.03 - Kratos
    ### Xmipp Programs 
+   
+  - New programs: tomo_confidence_map, tomo_extract_particlestacks, tomo_extract_subtomograms, tomo_tiltseries_dose_filter, psd_estimatator
   - Deprecated programs (for more details visit the [wiki](https://github.com/I2PC/xmipp/wiki/Deprecating-programs)):
     angular_distribution_show, apropos
     ctf_correct_idr, ctf_create_ctfdat , ctf_show , idr_xray_tomo , image_common_lines , metadata_convert_to_spider , metadata_selfile_create , mlf_refine_3d, ml_refine_3d, ml_tomo , mrc_create_metadata , pdb_construct_dictionary, pdb_restore_with_dictionary , reconstruct_admn , reconstruct_art_pseudo , resolution_ibw , resolution_ssnr , score_micrograph , reconstruct_fourier_starpu , tomo_align_tilt_series, tomo_align_dual_tilt_series, tomo_align_refinement, tomo_align_refinement, tomo_extract_subvolume, tomo_project_main, tomo_remove_fluctuations , tomo_align_tilt_series,transform_range_adjust , validation_tilt_pairs , volume_pca , volume_validate_pca , work_test , 6f4d983 , evaulate_coordinates , extract_subset , image_separate_objects , volume_enhance_contrast , volume_reslice , xray_import , xray_project , xray_psf_create , xray_reconstruct_art , gpu_correlation, gpu_utils, classify_significant, deepAlign.
-  - New programs: tomo_confidence_map, tomo_extract_particlestacks, 
-, tomo_extract_subtomograms, tomo_tiltseries_dose_filter, psd_estimatator
+
   - volume_from_pdb: fixing input pdb file being overwritten when '-centerPDB' flag was set
   - xmipp_phantom_movie:  adding support for fixed step shift & gain and dark image generation
   - CTF simulation allows astigmatism
   - xmipp_metadata_utility: Now join operations with an empty set will return a new empty set (previously no output file was generated).
   - xmipp_matrix_dimred: Program help improved. Exception is now thrown when the number of output dimensions is larger than the input dimensions
   - xmipp_angular_distance: Added itemId column to the output
 
-
-   ### Protocols scipion-em-xmipp
-  - New protocol status: beta, new, production and updated. Will appear in the left pannel of Scipion 
-  - Protocol subtract_projection: user experience improvements, no final mask by default, apply ciruclar mask in adjustment image to avoid edge artifacts, validate same sampling rate with tolerance in third decimal
-  - Protocol convert_pdb: Allowed to save centered PDB used for conversion. 
-  - Protocol align_volume_and_particles: add alingment validation
-  - Protocol FlexAlign: updating protocol to reflect changes in the executable, fixed test, removing unused protocol (Movie average)
-  - Protocol align_volume_and_particles:Align volume and particles adapted to tomography and works in the absence of tomo plugin.
-  - Protocol volume_consensus: validate same sampling rate with tolerance in third decimal
-  - Protocols deprecated (for more details visit the [wiki](https://github.com/I2PC/xmipp/wiki/Deprecating-programs)): protocol_deep _align, reconstruct_heterogeneous, protocol_metaprotocol_create_output, protocol_metaprotocol_discrete_heterogeneity_scheduler
-  
    ### Installation and user guide
    - Refactor and simplified Readme page.
    - Updating CUDA version compatibility
    - Updating gcc version availables
-   - Fixd Matlab installation
+   - Fixed Matlab installation
    - Added missing array include to fix compilation error with g++12
    - Alert and not block compilation if gcc - CUDA are not compatible
    - Avoid compilation warnings
@@ -59,23 +90,6 @@
   -  xmipp_micrograph_automatic_picking: Fixing memory leak
   -  subtract_projection: Fixed several bugs (improved results), added circular mask to avoid edge artifacts, added option to boost particles instead of subtract
 
-
-  ### Protocols scipion-em-xmipp
-  - Protocol_cl2d_align: The input can now be a set of averages or a set of 2D classes 
-  - Protocol_local_ctf: Default value are now changed for maxDefocusChange
-  - Protocol_apply_zernike3d: Now accepts either a Volume or SetOfVolumes and applies the coefficients in a loop in the deform step
-  - Protocol_postProcessing_deepPostProcessing: Managed GPU memory to avoid errors
-  - Protocol_resolution_deepres: Mandatory mask
-  - Protocol center particles and Gl2d (all options): Fix streaming
-  - Protocol_create_3d_mask: Allows volume Null=True
-  - Protocol_reconstruct_fourier: Set pixel size
-  - GL2D static: Bug fixing
-  - Protocol_trigger_data: Bug fixing
-  - Protocol_crop_resize: Set sampling rate of mrc files when cropping resizing volumes or particles
-  - subtract_projection: New protocol for boosting particles. Add protocol to wizard XmippParticleMaskRadiusWizard as now the protocol uses it
-
-  - **New tests:** deep_hand, pick_noise, screen_deep_learning, resolution_B_factor
-  - Fixed TestHighres test
 
   ### Installation and user guide
   - Various bug fixing
@@ -102,26 +116,6 @@
 - **volume_consensus**: Bug fixed
 - **ctf.h and angular_continuous_assign_2**: Changes for local defocus estimation #578
 
-### Protocols scipion-em-xmipp
-- **rotate_volume**: New protocol
-- **subtract_projection**: New implementation based on adjustment by regression instead of POCS and improved performance
-- **local_ctf**: Add new sameDefocus option + formatting
-- **compare_reprojections & protocol_align_volume**: Fast Fourier by default
-- **crop_resize**: Allows input pointers
-- **resolution_deepres**: Resize output to original size
-- **denoise_particles**: Added setOfAverages as input option
-- **process**: Change output from stk (spider) to mrcs (mrc)
-- **trigger_data**: Bug fixed
-- **screen_deeplearning**:  Added descriptive help" 
-- **center_particles**: Added summary info
-- **align_volume_and_particles**: Summary error fixed
-- **cl2d**: Summary errors solved 
-
-- **New tests:** test_protocol_reconstruct_fourier, test_protocols_local_defocus, test_protocols_local_defocus, TestXmippAlignVolumeAndParticles,  TestXmippRotateVolume
-- **Improved tests:** test_protocols_deepVolPostprocessing, test_protocols_xmipp_3d, Test ProjSubtracion
-- **Excluded tests:** test_protocols_zernike3d, test_protocols_metaprotocol_heterogeneity
-
-
 ### Installation and user guide
 - Version info printed at the end of the installation
 - Removed empty folder with cleanBin command

CHANGELOG.md

@@ -8,7 +8,7 @@
 Xmipp is a suite of image processing programs, primarily aimed at single-particle 3D electron microscopy, designed and managed by the [Biocomputing Unit](http://biocomputingunit.es/) located in Madrid, Spain.
 
 The Xmipp project is divided into four repositories. 
-This is the main repository, which contains the majority of the source code for the programs, additional scripts, and tests. Three remaining repositories, [XmippCore](https://github.com/I2PC/xmippCore/), [XmippViz](https://github.com/I2PC/xmippViz/), and [Scipion-em-xmipp](https://github.com/I2PC/scipion-em-xmipp), are automatically downloaded during the compilation/installation process.
+This is the main repository, which contains the majority of the source code for the programs, additional scripts, and tests. Three remaining repositories, [XmippCore](https://github.com/I2PC/xmippCore/), [XmippViz](https://github.com/I2PC/xmippViz/), and [Scipion-em-xmipp](https://github.com/I2PC/scipion-em-xmipp), are automatically downloaded during the installation process.
 
 
 ## Getting started
@@ -26,8 +26,10 @@ At least 2 processors are required to run Xmipp. In some virtual machine tools o
 
 ### Software dependencies
 #### Compiler
-Xmipp requires C++17 compatible compiler. We recommend either GCC or CLANG, in the newest version possible. We have good experience with GCC-8 and bad experience with GCC-7, in any case a version > 6 is required. If use GCC-11 and experience issues, [please visit this.](https://github.com/I2PC/xmipp/issues/583). For more details about the compilation proces and installation of gcc, please visit [compiler](https://github.com/I2PC/xmipp/wiki/Compiler)
+Xmipp requires C++17 compatible compiler. We recommend GCC, we have good experience with GCC-8.3 in any case a version > 8 is required. If use GCC-10.3 and CUDA=11 and experience issues, [please change the compiler version](https://github.com/NVIDIA/nccl/issues/494). If use GCC-11 and experience issues, [please visit this.](https://github.com/I2PC/xmipp/issues/583). For more details about the compilation proces and installation of gcc, please visit [compiler](https://github.com/I2PC/xmipp/wiki/Compiler)
 
+#### Cmake
+Xmipp requires Cmake 3.16 or above. To update it please [visit](https://github.com/I2PC/xmipp/wiki/Cmake-update-and-install)
 #### Cuda
 Xmipp supports Cuda 8 through 11.7. CUDA is optional but highly recommended. We recommend you to use the newest version available for your operating system, though Cuda 10.2 has the widest support among other Scipion plugins. Pay attemption to the [compiler - CUDA](https://gist.github.com/ax3l/9489132) compatibility.
 To install CUDA for your operating system, follow the [official install guide](https://developer.nvidia.com/cuda-toolkit-archive).
@@ -43,7 +45,7 @@ We sometimes see issues regarding the HDF5 dependency. Please visit the [HDF5 Tr
 
 `sudo apt install -y libfftw3-dev libopenmpi-dev libhdf5-dev python3-numpy python3-dev libtiff5-dev libsqlite3-dev default-jdk git cmake`
 
-`pip install scons numpy`
+`pip install scons` in the enviroment that xmipp will be compiled (scipion3 if you will run Xmipp within Scipion)
 
 Also a compiler will be required (`sudo apt install gcc-8 g++-8`)
 
@@ -76,7 +78,7 @@ Xmipp is installed in the build directory located in the same directory where th
 `source build/xmipp.bashrc`
 
 
-# That's all
+## That's all
 
 If you miss some feature or find a bug, please [create an issue](https://github.com/I2PC/xmipp/issues/new) and we will have a look at it, you can also contact us by xmipp@cnb.csic.es. For more details, troubleshootings and information visit the [wikip page.](https://github.com/I2PC/xmipp/wiki)