Full Scale Performance: Single Process, Sharrow Off, Explicit Chunking #20

Activity

1. aletzdy commented on Jun 6, 2024

   

   aletzdy

   on Jun 6, 2024

   Last edited by aletzdy

   Author

   More actions

   Run with full sample, sharrow on, and single process (1 TB memory, Intel Xeon Gold 6342 @ 2.8GHz machine).

   Runtime is 1428.2 mins (compared to 1,433.1 mins for a similar run with NO chunking)
   Chunking is set to explicit in settings_mp.yaml, and explicit chunking is enabled for a number of models whose memory usage was higher than 300Gb in a previous sharrow run with same specs but with NO chunking. These models are:

   compute_accessbility
   mandatory_tour_scheduling
   non-mandatory_tour_scheduling
   trip_destination

   For these models, the argument explicit_chunk was set to 0.5 in each respective model's yaml file, indicating the need for 2 (=1/0.5) chunks. Further chunks may be added in other tests and as required.

   timing_log.csv
   activitysim.log
   memory_profile.csv

   

   Below is the run without chunking:

   <style> </style>

   event	full_rss_gb_chunking	full_rss_gb_No_chunking	Difference	% Diff accounting for skims
   preload_injectables	2.167452	2.527322	-0.35987	NA
   initialize_proto_population	173.9227	174.1077	-0.18509	NA
   compute_disaggregate_accessibility	206.3356	201.5561	4.779454	17.4%
   initialize_landuse	180.1346	175.3758	4.75886	375.3%
   initialize_households	198.2141	193.3762	4.837839	25.1%
   compute_accessibility	276.3907	367.4861	-91.0954	-47.1%
   av_ownership	184.3127	180.3675	3.945181	63.0%
   auto_ownership_simulate	202.3356	197.699	4.636545	19.7%
   work_from_home	193.0878	192.2186	0.869179	4.8%
   external_worker_identification	194.0765	192.6287	1.447735	7.8%
   external_workplace_location	194.758	192.3864	2.371576	13.0%
   school_location	292.9196	289.0295	3.890065	3.4%
   workplace_location	402.6434	397.1514	5.49206	2.5%
   transit_pass_subsidy	194.9363	192.3512	2.585076	14.2%
   transit_pass_ownership	194.0668	193.3555	0.711307	3.7%
   vehicle_type_choice	270.9691	265.4805	5.488607	6.0%
   adjust_auto_operating_cost	183.9534	178.5603	5.393015	121.1%
   transponder_ownership	188.1608	182.8532	5.30756	60.7%
   free_parking	194.9765	193.1514	1.825108	9.6%
   telecommute_frequency	195.4649	192.9583	2.506637	13.3%
   cdap_simulate	195.6484	193.3964	2.252005	11.7%
   mandatory_tour_frequency	195.0767	194.2196	0.857059	4.3%
   mandatory_tour_scheduling	330.5758	453.69	-123.114	-44.0%
   school_escorting	214.1988	208.6679	5.530911	16.0%
   joint_tour_frequency_composition	205.0339	201.3234	3.710493	13.6%
   external_joint_tour_identification	201.6065	201.7927	-0.18614	-0.7%
   joint_tour_participation	199.2767	194.9526	4.324151	20.7%
   joint_tour_destination	207.107	202.9316	4.175458	14.5%
   external_joint_tour_destination	196.1608	191.8689	4.291928	24.2%
   joint_tour_scheduling	212.2434	212.0547	0.188678	0.5%
   non_mandatory_tour_frequency	238.0545	233.7386	4.315931	7.2%
   external_non_mandatory_identification	208.9876	207.5797	1.407865	4.2%
   non_mandatory_tour_destination	383.1723	379.6374	3.534909	1.7%
   external_non_mandatory_destination	197.0708	192.72	4.350751	23.4%
   non_mandatory_tour_scheduling	227.4625	275.858	-48.3955	-47.6%
   vehicle_allocation	218.2018	217.5814	0.620433	1.4%
   tour_mode_choice_simulate	217.6302	212.2314	5.398741	14.2%
   atwork_subtour_frequency	196.0487	190.7215	5.327159	32.1%
   atwork_subtour_destination	295.5799	290.2229	5.357048	4.6%
   atwork_subtour_scheduling	211.7981	206.3961	5.401965	16.7%
   atwork_subtour_mode_choice	198.8897	193.8696	5.020123	25.4%
   stop_frequency	222.6066	217.8433	4.7633	10.9%
   trip_purpose	199.3885	195.7768	3.611701	16.7%
   trip_destination	426.6972	536.408	-109.711	-30.3%
   trip_purpose_and_destination	206.8114	201.6604	5.150953	18.7%
   trip_scheduling	208.1987	204.3938	3.804885	12.6%
   trip_mode_choice	224.9043	219.6366	5.267706	11.6%
   parking_location	292.0136	303.9549	-11.9413	-9.2%
   write_data_dictionary	210.6467	207.9639	2.682831	7.9%
   track_skim_usage	204.6225	202.1674	2.455048	8.7%
   write_trip_matrices	298.8238	296.5611	2.262761	1.8%
   write_tables	257.3621	255.2239	2.138206	2.6%
   finalizing	230.174	228.311	1.862975	3.4%