Design a distributed data processing system using Apache Spark

Design a distributed data processing system that efficiently transforms large-scale datasets (e.g., 4TB of data) using Apache Spark with constrained compute resources, addressing challenges like partitioning, memory management, and job optimization.