This article provide details on running the simulation from the Windows command prompt.
Using the design environment (CAD/GUI)
-
FDTD
fdtd-solutions [options]
-
MODE
mode-solutions [options]
-
CHARGE, HEAT, DGTD and FEEM
device [options]
-
INTERCONNECT
interconnect [options]
Options
filename
- optional, the filename Opens the specified simulation or project file.
-v
- optional, Outputs the product version number.
scriptFile.lsf
- optional, the script filename Opens the specified script file.
-safe-mode
- optional, Turn on the safe mode.
-trust-script
- optional, Turn off the safe mode.
-run <scriptfile>
..\fdtd-solutions.exe -run <scriptfile> <simulationfile>
- optional, Run the specified script file.
- if a simulation file is required, this is added after the script file.
-nw
-hide
- optional, (-nw for FDTD only), -hide for other solvers.
- This hides the CAD window from appearing on the Desktop.
|
Notes: -nw and -hide command options
|
-use-solve
-run script.lsf -use-solve
- optional, used to run simulation in non-interactive engine mode.
- added after the -run and script file options.
- this will run the simulation based on the resource configuration saved in the CAD.
|
Notes: -use-solve command options
|
-logall
-
optional, Generates a log file for each simulation or sweep.
-exit
- optional, Exit application after running the script file.
-o
- Change the location that log files are saved to.
- All log files will be saved to the relative or absolute directory passed to -o.
- If the directory ends with .log this will be treated as a file name.
Useful when running INTERCONNECT with the -logall option.
Examples
Running a script while 'hiding' the CAD window and saving the log file on a different location.
"C:\Program Files\Lumerical\[[verpath]]\bin\interconnect.exe" -hide -run scriptfile.lsf -logall -o "C:\temp\logfiles\"
Running a script with a simulation file while 'hiding' the CAD window and disabling safe mode.
"C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-solutions" -nw -trust-script -run scriptfile.lsf simulationfile.fsp
Opening a simulation file.
"C:\Program Files\Lumerical\[[verpath]]\bin\mode-solutions.exe" simulationfile.lms
"C:\Program Files\Lumerical\[[verpath]]\bin\device.exe" simulationfile.ldev
Run simulations without using MPI
- FDTD
fdtd-engine [options]
- FDE
fde-engine [options]
- EME
eme-engine [options]
- varFDTD
varfdtd-engine [options]
- CHARGE
device-engine [options]
- HEAT
thermal-engine [options]
- DGTD
dgtd-engine [options]
- FEEM
feem-engine [options]
- MQW
mqw-engine [options]
Options
filename
- required, The name of the simulation or project file to run.
-t
- optional, Controls the number of threads used. If not used or left blank, it will use all available threads.
-fullinfo
- optional, It will print more detailed time benchmarking information to the log file based on walltime and cpu time measurements.
-log-stdout
- optional, Redirects the log file data to the standard output, rather than saving it to file.
- This option will be ignored when the simulation runs in graphical mode.
-mesh-only
- optional, Mesh the geometry without running the simulation.
-inmaterialfile <file>
- optional, Load simulation mesh data from a <file>.
-outmaterialfile <file>
- optional, Save simulation mesh data to <file> for use on another project.
-logall
- optional, Create a log file for each simulation or sweep.
- Logfiles are named filename_p0.log, filename_p1.log, filename_p2.log
- By default, only filename_p0.log is created.
-mr
Note: Requires to be executed using Git bash on Windows.
- optional, Print a simple memory usage report for a given simulation file to the standard output. Output can be piped or saved as a text file.*
-o
- optional, Change the location that log files are saved to.
- All log files will be saved to the relative or absolute directory passed to -o.
- If the directory ends with .log the last section will be treated as a file name.
-resume
- optional, available for FDTD simulations only.
- Resumes the simulation from the last check point.
- If no check point is found it will start the simulation job from the beginning.
- Enable the simulation checkpoint feature in the "Advanced Options" of the FDTD Solver object.
Examples
Run a simulation using 12 threads (cores).
"C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-msmpi.exe" -t 12 "C:\temp\example.fsp"
Running with 4 threads (cores) and saving the log files into a specific path.
"C:\Program Files\Lumerical\[[verpath]]\bin\varfdtd-engine.exe" -t 4 "C:\temp\example.lms" -o "C:\temp\logfiles\"
Running on the local computer with the -resume flag when check point is enabled in FDTD.
"C:\Program Files\Lumerical\[[verpath]]\bin\varfdtd-engine.exe" -t 4 -resume "\path\simulationfile.fsp"
Running simulations via the MPI
Using MPI to run the simulation job with the solver is done for the following used cases:
- Run several simulations at the same time on different machine or nodes. (Concurrent computing)
- Using several machine to run 1 single simulation to take advantage of their memory (RAM) as required by the simulation. (Distributed computing)
- Launch and run a simulation from a local machine to a different remote machine or node.
MPI is a complex application with many configuration options and versions. On Windows, Microsoft MPI, MPICH2 and Intel MPI are the supported MPI frameworks.
General MPI Syntax
mpiexec [mpi_options] solver [solver_options]
MPI Options
-n <#>
- FDTD, varFDTD and EME specify the <#> number of mpi processes
-hosts <hostlist>
- FDTD, varFDTD and EME or send job across multiple computers.
-hosts <hostfile>
- Overrides the '-n' option.
Where - hostlist : comma separated list of hosts or IP with corresponding number of processes
- hostfile : text file with 1 hostname/IP per line, with corresponding number of processes separated by a comma ':'
-nice -n19
- all solvers, specifies the process priority for load balancing.
-affinity
- only on Microsoft MPI, all solvers
- Microsoft MPI feature to lock processes to specific cores.
For additional information on MPI options, consult the MPI product documentation for further details:
"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -help
OR
"C:\Program Files (x86)\IntelSWToolsMPI\mpi\2018.4.274\intel64\bin\mpiexec.exe" -help
Supported MPI variants
It is necessary to use the version of the solver that is matched to the version of MPI being used to run the solver. See the list below for details.
Microsoft MPI
The following executable are included with the installation and suitable to use when running on the local machine. Not used when running jobs on remote machines.
- fdtd-engine-msmpi
- fde-engine-msmpi
- eme-engine-msmpi
- varfdtd-engine-msmpi
- device-engine-msmpi
- thermal-engine-msmpi
- dgtd-engine-msmpi
- feem-engine-msmpi
- mqw-engine-msmpi
Intel MPI
This is included with the installation and is used to run locally or remote machines that has more than 64 cores or multiple CPU/NUMA nodes.
- fdtd-engine-impi
- fde-engine-impi
- eme-engine-impi
- varfdtd-engine-impi
- device-engine-impi
- thermal-engine-impi
- dgtd-engine-impi
- feem-engine-impi
- mqw-engine-impi
MPICH2
The MPICH2 engine executable is provided as an optional installation that can be used to run locally or remote machines that has less than 64 cores or only 1 CPU/NUMA node.
- fdtd-engine
- fde-engine
- eme-engine
- varfdtd-engine
- device-engine
- thermal-engine
- dgtd-engine
- feem-engine
- mqw-engine
Examples
Microsoft MPI, with 4 processes and the -resume flag when checkpoint is enabled in FDTD.
"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 "C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-msmpi.exe" -t 1 -resume "\path\simulationfile.fsp"
Microsoft MPI, with 4 processes
"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 "C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-msmpi.exe" "C:\temp\example.fsp"
Intel MPI, using 4 processes
"C:\Program Files (x86)\IntelSWToolsMPI\mpi\2018.4.274\intel64\bin\mpiexec.exe" -n 4 "C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-impi.exe" "C:\temp\example.fsp"
Microsoft MPI, and setting job priority
"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 -priority 1 "C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-msmpi.exe" "C:\temp\example.fsp"
Intel MPI, distributed between two computers using 4 processes on each node.
"C:\Program Files (x86)\IntelSWToolsMPI\mpi\2018.4.274\intel64\bin\mpiexec.exe" -hosts 2 node1 4 node2 4 "C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine-impi.exe" -t 1 "C:\temp\example.fsp"
Pipe standard output to a text file
- The standard output does not appear in the Command Prompt window. In order to see the report you can simply pipe the output to a text file using the piping command ">".
- For example, to output the engine version number or memory usage report to a file, use the following syntax.
"C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine.exe" -v > "C:\temp\engine_version.txt"
"C:\Program Files\Lumerical\[[verpath]]\bin\fdtd-engine.exe" -mr "C:\temp\example.fsp" > "C:\temp\mem_usage_example.txt"
CPi - MPI test program
This test application allows users to ensure that MPI is properly configured, without the additional complication of running any Lumerical solver.
For example, this avoids any potential problems with product licensing, since both MPI and CPI are not licensed features.
MPICH2 Nemesis
Run CPi using 4 processes on the local computer.
"C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 4 "C:\Program Files\Lumerical\FDTD\mpitest\cpi-msmpi.exe"
The output of the CPI test should look something like this:
Process 2 on localhost
Process 1 on localhost
Process 3 on localhost
Process 0 on localhost
pi is approximately 3.1416009869231249, Error is 0.0000083333333318
wall clock time = 0.000049
Simple Multi-purpose Daemon (SMPD)
The SMPD service is part of the MPI (Message Passing Interface) Library process management system utility used for running simulations. SMPD is an application that runs in the background of each computer. Your local machine will send a request to the SMPD on the remote computer, asking it to start the simulation on that remote computer.
Microsoft MPI
The Microsoft MPI version of SMPD does not start automatically when the computer starts. This is why Microsoft MPI is not recommended for remote job launching.
Should you want to used Microsoft MPI for remote job launching, it is necessary to manually log into each computer used to run the simulation. Open a command prompt and execute:
C:\Program Files\Microsoft MPI\Bin\smpd.exe -debug
MPICH2
The MPICH2 version of SMPD is configured to start automatically when the computer starts.
Intel MPI
This version of SMPD also starts automatically when the computer starts.