LargestSimulations

The aim of this page is to give a feeling of how large light scattering simulations can be done using ADDA. Hopefully, it will also present a variety of different applications, so that each example will tell an interesting short story by itself. Complexity of the light scattering simulation with ADDA depends on the many (connected) factors: scatterer size parameter, required discretization or accuracy, number of voxels in a computational grid, convergence speed of the iterative solver, and need for repeated calculations (e.g. orientation averaging). Therefore, this page consists of several categories (nominations) with several examples in each of them.

• Overview
• Largest size parameter
  • X1/N1. Cube with kD=2000 and refractive index 1.02
  • X2. Sphere with size parameter 320 and refractive index 1.05
  • X3. Aggregate of 1 million spheres
• Largest number of voxels
  • N2. Truncated slab
  • N3. Red blood cell with very fine discretization
• Largest processor time
  • T1. Sphere with size parameter 130 and refractive index 1.2
  • T2. Aggregate of 10⁵ spheres, averaged over orientations

Overview

Users are encouraged to submit new examples based on their own work/experience. This can be done either by a comment to this wiki page or contacting the maintainer of this page directly.

Before proceeding to the description of particular examples, we present a summary table. Note that the processor time is an ambiguous measure, since there is large difference in performance per working frequency between processors of different generations over the last 20 years (up to 10 times due to execution of larger number of instructions per cycle). Thus, its usefulness is similar to processor hours (related by a factor or roughly 2500).

Parameter	X1/N1	X2	X3	N2	N3	T1	T2
Shape	cube	sphere	aggregate	slab	red blood cell	sphere	aggregate
Volume-equivalent size parameter	1240	320	176	16	36	130	82
Refractive index (average)	1.02	1.05	1.5+10−4i	1.31+0.1i	1.03	1.2	1.5+10−4i
Dipoles (voxels) per wavelength	10	10	10	257	93	12	10
Largest 1D grid size	3168	1024	958	8192	1408	512	444
Number of voxels in a grid, 106	31795	1073	879	1073	1035	134	87
Number of occupied voxels, 106	31795	562	92	1073	619	70	9.2
Relative residual norm (-eps)	4×10−2	10−4	10−3	10−5	10−5	10−5	10−2
Iterative solver used	Bi-CG	Bi-CGStab	QMR	QMR	QMR	Bi-CGStab	QMR
Number of iterations (average)	300	334	9200	23	10	29200	1100
Number of runs of iterative solver	1	1	2	2	1	1	116
Number of processor cores	3168	512	1008	4096	560	64	240
Processor time, THz×hours	16	8.8	41	0.2	1.1	90	35
Total memory used, GiB	27930	698	647	1093	754	70	63
Year	2025	2008	2016	2013	2009	2006	2016
ADDA version	1.5.0-alpha3	0.79a2	1.3b4	1.2	0.79	0.75b	1.3b4
Supercomputer	Austral	MareNostrum	Sisu	Sisu	LISA	LISA	Sisu

The values in bold are the largest (the hardest to reach) ones for a particular parameter among all examples

Largest size parameter

X1/N1. Cube with kD=2000 and refractive index 1.02

Currently, this case wins by a large margin in both size and number of voxels, but is only a test simulation on a modern cluster. The major limitation is the refractive index, which had to be low (1.02) to ensure reasonable convergence of the iterative solver. The option -init_field wkb helped to extend the limit to some extent, but significantly larger values of refractive index still remain unreachable. Specifically, ADDA was executed with a command line:

adda_mpi -shape box -size 2000 -m 1.02 0 -grid 3168 -ntheta 9000 -eps 4 -init_field wkb -maxiter 300 -pol fcd -int fcd -iter bicg

To reach the record size we used the maximum available memory for a single job, occupying 44 192-core nodes, but could effectively employ only 3168 of the available cores (based on grid size). For the same reason, we used the least-memory-consuming iterative solver (Bi-CG), which is known to be not optimal with respect to wall time. Moreover, we limited the number of iterations to 300 and only reached the relative residual norm of 4×10⁻², which is hardly sufficient. Reaching the standard norm of 10⁻⁴ is possible, but will require at least 3 times larger processor time. These results were presented at ELS XXI conference, see slide 11 of the presentation.

X2. Sphere with size parameter 320 and refractive index 1.05

This was a benchmark simulation, performed to test the performance and extreme capabilities of ADDA on MareNostrum. It is published in Yurkin M.A. and Hoekstra A.G. The discrete-dipole-approximation code ADDA: capabilities and known limitations, J. Quant. Spectrosc. Radiat. Transfer 112, 2234–2247 (2011). ADDA was executed with a command line:

adda_mpi -m 1.05 0 -grid 1024 -size 640 -ntheta 1800 -iter bicgstab -eps 4

X3. Aggregate of 1 million spheres

The aggregates consisted of 1 million spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 301. ADDA was executed with a command line:

adda_mpi -m 1.5 0.0001 -shape read n1000000_dpl10.geom -dpl 10.0003 -ntheta 1800 -eps 3

This is the maximum size we could reach, but it was only for a single aggregate in fixed orientation. Thus, it was not included in the paper in contrast to the results for 10⁵ spheres, where we could perform orientation averaging.

Largest number of voxels

N2. Truncated slab

This was a part of a scalability study of ADDA on Sisu, performed by Antti Penttila. The overall parallel efficiency for the largest number of cores (4096) was estimated to be 63%, but it was 77% and 83% for 2048 and 512 cores respectively. ADDA was executed with a command line:

adda_mpi -size 100 -m 1.31 0.1 -grid 4096 32 8192 -shape box 0.0078125 2 

The size was chosen relatively small (corresponding to very large number of dipoles/voxels per wavelength) to keep the simulations fast. However, at least 10 times larger sizes seem feasible with such hardware. In particular, such simulations can be used to study optical properties of effectively infinite inhomogeneous (particulate) slabs.

N3. Red blood cell with very fine discretization

This simulation was a part of comparison of two methods for simulation of light scattering, published in Gilev K.V., Eremina E., Yurkin M.A., and Maltsev V.P. Comparison of the discrete dipole approximation and the discrete source method for simulation of light scattering by red blood cells, Opt. Express 18, 5681–5690 (2010).

Since both considered methods involve certain numerical errors, there was need for some reference, so that accuracy of the methods can be independently evaluated. Described simulation for very fine discretization was used as such a reference. ADDA was executed with a command line:

adda_mpi -lambda 0.4936 -m 1.03 0.0 -shape axisymmetric fung_100points.txt -size 7.5 -grid 1408 -scat_grid_inp scat_params_eryth.dat -yz

Moreover, convergence of DDA results with refining discretization was studied to estimate the accuracy of the reference itself.

Largest processor time

T1. Sphere with size parameter 130 and refractive index 1.2

This simulation was performed in frame of a benchmark study: Yurkin M.A., Maltsev V.P., and Hoekstra A.G. The discrete dipole approximation for simulation of light scattering by particles much larger than the wavelength, J. Quant. Spectrosc. Radiat. Transfer 106, 546–557 (2007). ADDA was executed 25 times in a row with a command line:

adda_mpi -m 1.2 0 -size 260 -grid 512 -ntheta 1440 -iter bicgstab -chpoint 17h [-chp_load]

where the last option was used for every incantation except the first one. Use of checkpoint system was necessary, since at that time no single job could use more than 1200 processor-hours on LISA.

T2. Aggregate of 10⁵ spheres, averaged over orientations

The aggregates consisisted of 100 000 spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 140. The orientation of the aggregate was averaged, by performing 58 independent ADDA simulations and using 512 scattering planes for each orientation. ADDA was executed with a command line:

adda_mpi -m 1.5 0.0001 -shape read 1h_10dpl.geom -dpl 10.0 -ntheta 1800 -eps 2 -orient avg avg-all.dat

The results were compared with other methods in Penttilä A., Markkanen J., Väisänen T., Räbinä J., Yurkin M.A., and Muinonen K. How much is enough? The convergence of finite sample scattering properties to those of infinite media, J. Quant. Spectrosc. Radiat. Transfer 262, 107524 (2021). One of the graphs is shown on another wiki page. Even larger aggregate has been simulated, but only in a fixed orientation.