This article shows how to run parameter or optimization sweeps with the Lumerical-AWS integration using the script prompt within FDTD.
These commands are executed "in-product" rather than running them from the Python shell or a Python script (.py) file.
These commands can be executed directly from the script prompt within the product's design environment or part of a script (.lsf) file.
Requisites
- AWS account. If you do not have one, create and activate and AWS account.
- Lumerical-AWS integration configuration
- AWS S3 bucket with the simulation files you want to run. See, Using S3 buckets for details.
- Your license server and compute nodes are up and running. See, Managing instances for details.
|
Notes: When using systems commands in the script prompt, backslash double quotes (\") will be used whenever quotes are required within the Python command. Running system commands from the script prompt will not open the command prompt or terminal window and display the standard output. To view the standard output, this must be stored as a text file. Add the Python installation directory to your PATH environment and change the working directory of that of the Lumerical-AWS module. Configuration of the Lumerical-AWS integration should be completed prior to executing the processes in this page. |
Using S3
To run simulation jobs using the Lumerical-AWS Python integration module, simulation files has to be copied over to S3.
Copying to S3 from the local machine can be done in 2 ways.
- Using the AWS management console and add an Object to your bucket.
Files and Folders are called Objects in S3. - Using the script prompt from within the product's CAD/GUI as shown in this guide.
Example:
- Simulation file: "sweep_AR_coating_example.fsp"
- S3 bucket: "s3://lum-aws-demo-bucket/"
(Copy an existing simulation ile to S3)
> cp('c:\location\filename.fsp, s3://bucketname/filename.fsp');
(Save the current working project file to S3)
> save('s3://lum-aws-demo-bucket/simulation_1001.fsp');
Launch nodes
> system('python -c "import lumerical.aws; lumerical.aws.start_compute_instances(name=\"NAME\", num_instances=\"NODE_COUNT\", vnc_password=\"VNC_PASSWORD\")" > "FILENAME.txt" ');
Where:
- NAME, VPC created during configuration.
- NODE_COUNT, is the number of compute nodes to launch.
- VNC_PASSWORD, optional and provides VNC password for individual access to the nodes. If left blank, it will take the Workgroup_ID as the VNC password.
- FILENAME.txt, change this to the location and filename you want to store your compute instance status information.
- Each compute node is configured to checkout license from your license server.
- Each node will have the resource manager configured to launch jobs, referencing the IPs of all other nodes in the Workgroup created by the start_compute_instance command.
- All the nodes created will have a unique and shared Workgroup_ID tag.
Example:
- VPC Name: "aws-demo"
- NODE _COUNT: 4
- VNC_PASSWORD: not defined
- Output Text file: "C:\node-status.txt"
> system('python -c "import lumerical.aws; lumerical.aws.start_compute_instances(name=\"aws-demo\", num_instances=4)" > "C:\node-status.txt" ');
After creating and launching the nodes, the following information will be stored in the output text file.
- Time Stamp, the time stamp when the nodes were launched.
- Workgroup_ID, the shared tag for all the nodes for the current 'Launch'.
- Number of nodes created.
- VNC_password, which is the 'WorkgroupID' for this set of nodes.
Run sweeps
Ensure that the simulation or project file is stored in your S3 bucket. Run your sweeps from S3.
> system('python -c "import lumerical.aws; lumerical.aws.run_parameter_sweep(name=\"NAME\", workgroup_id=\"WORKGROUP_ID\", s3_uri=\"S3_PATH_FILE\", sweep_name=\"SWEEP_NAME\")" ');
Where:
- NAME, VPC name created in the previous step.
- WORKGROUP_ID, is the workgroup_ID Tag created when launching your compute nodes.
- S3_PATH_FILE, is the full S3 bucket URI, including the folder and the simulation (.fsp) filename.
- SWEEP_NAME, is the name of the parameter sweep in your simulation.
Example:
> system('python -c "import lumerical.aws; lumerical.aws.run_parameter_sweep(name=\"aws-vpc01\", workgroup_id=\"aws-vpc01-00021\", s3_uri=\"s3:/lum-aws/paralleltest.fsp\", sweep_name=\"test_sweep\")" ');
Retrieve results
Load the project file directly from S3 and view the results on your local computer.
> load('s3://bucketname/filename.fsp');
Managing jobs and nodes
The nodes should have been 'Terminated' after the job completes. But if the nodes are not Terminated somehow even after you obtained and saved your results to your local machine, This can be terminated manually. We have 2 ways of removing or terminating the nodes to avoid recurring charges while these are not in use.
Killing simulation jobs
This will terminate all nodes with the corresponding 'Workgroup ID'. This action does not save any changes or results to your simulation file. This is similar to the 'Force Quit' button on the CAD/GUI resource manager. Nodes running under a different 'Workgroup ID' will not be affected.
> system('python -c "import lumerical.aws; lumerical.aws.kill_job(name=\"NAME\", workgroup=\"WORKGROUP_ID\")" ');
Terminating nodes
The command, "terminate_all_instances" will terminate ALL of your compute instances irregardless of Workgroup_ID, in the specified VPC. This will not save any changes or results to your simulation file.
> system('python -c "import lumerical.aws; lumerical.aws.terminate_all_instances(name=\"NAME\")" ');
See also:
Accessing individual nodes with VNC