Worker pod resource limit in spring batch and spring cloud deployer kubernetes

Issue

I am trying to run a Spring Batch application in kubernetes cluster. I am able to enforce resource limits to the main application pod by placing the following snippet in the deployment yaml:

resources:
  limits:
    cpu: 500m
    ephemeral-storage: 500Mi
    memory: 250Mi

These settings are getting applied and can be seen in the pod yaml (kubectl edit pod batch).

However, these limits are not propagated to the worker pods. I tried adding the following properties in configmap of batch to set the cpu and memory limits:

SPRING.CLOUD.DEPLOYER.KUBERNETES.CPU: 500m
SPRING.CLOUD.DEPLOYER.KUBERNETES.MEMORY: 250Mi

However, the worker pods are not getting these limits. I tried providing the following env variables too, but still the limits were not applied to the worker pod:

SPRING_CLOUD_DEPLOYER_KUBERNETES_CPU: 500m
SPRING_CLOUD_DEPLOYER_KUBERNETES_MEMORY: 250Mi

The versions involved are:

  • Spring Boot: 2.1.9.RELEASE
  • Spring Cloud: 2020.0.1
  • Spring Cloud Deployer: 2.5.0
  • Spring Cloud Task: 2.1.1.RELEASE
  • Kubernetes: 1.21

How can I set these limits?

EDIT: Adding code for DeployerPartitionerHandler:

public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer) {

    Resource resource = this.resourceLoader.getResource(resourceSpec);

    DeployerPartitionHandler partitionHandler = new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
            "worker");

    commandLineArgs.add("--spring.profiles.active=worker");
    commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
    commandLineArgs.add("--spring.batch.initializer.enabled=false");
    commandLineArgs.add("--spring.cloud.task.closecontext_enabled=true");
    commandLineArgs.add("--logging.level.root=DEBUG");

    partitionHandler.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
    partitionHandler.setEnvironmentVariablesProvider(environmentVariablesProvider());
    partitionHandler.setApplicationName(appName + "worker");
    partitionHandler.setMaxWorkers(maxWorkers);

    return partitionHandler;
}

@Bean
public EnvironmentVariablesProvider environmentVariablesProvider() {
    return new SimpleEnvironmentVariablesProvider(this.environment);
}

Solution

Since the DeployerPartitionHandler is created using the new operator in the partitionHandler method, it is not aware of the values from the properties file. The DeployerPartitionHandler provides a setter for deploymentProperties. You should use this parameter to specify deployment properties for worker tasks.

EDIT: Based on comment by Glenn Renfro

The deployment property should be spring.cloud.deployer.kubernetes.limits.cpu and not spring.cloud.deployer.kubernetes.cpu.

Answered By – Mahmoud Ben Hassine

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published