Conda environment not working with nextflow

[MarcElosua]

Hi everyone, I am very new to nextflow so this could be a silly mmistake... I'm running into an error when using nextflow with a conda environment that I have no idea how to work around or debug...

I am trying to run this on an HPC cluster with a slurm workflow manager

I have this process in my test.nf file

// Declare syntax version
nextflow.enable.dsl=2

process processSampleSheet { 
    executor 'local' // No need to launch a job for this step
    
    storeDir params.main_dir + "/outputs"
    
    // output gemid csv file to channel
    output:
    path "sample-sheet-gemid.csv"
    
    script:
    """
	#!/usr/bin/env python3
	
	
	import secrets
	import string
	import pandas as pd

	# Function to generate random 8-character alphanumeric string
	def generate_random_string():
		alphabet = string.ascii_letters + string.digits
		return ''.join(secrets.choice(alphabet) for _ in range(8))

	print("${params.sampleSheetPath}")
	# Read and subset sample sheet
	meta = pd.read_csv("${params.sampleSheetPath}")

	# Add a unique gem_id for each sample
	meta['gem_id'] = meta.apply(lambda x: generate_random_string(), axis=1)

	# Write dataframe to csv
	meta.to_csv("sample-sheet-gemid.csv", header = True, index = False)
    """
}

and the nextflow.config file looks like:

// GLOBAL PARAMETERS
params.main_dir = '/temp_work/'
params.sampleSheetPath = params.main_dir + '/sample_sheet_test.csv'  // Your sample sheet file
params.fastqPath = params.main_dir + '/fastq_sub'  // Your sample sheet file
params.cellrangerPath = '/programs/x86_64-linux/cellranger/7.1.0/' // need to be in interactive node -> srun --pty /bin/bash
params.expect_cells = 10000
params.reference = '/reference_databases/ReferenceGenome/10xGenomics/refdata-gex-GRCh38-2020-A'
params.slurm_template = params.main_dir + '/scripts/slurm.template'

// Remove work and cacher and .nextflow files at the end of the process
cleanup = true

// SLURM parameters
// https://www.nextflow.io/docs/latest/config.html?highlight=queuesize
executor {
    name = 'slurm'
    queueSize = 250
    pollInterval = '15 sec'
}

// Path to the conda environment
conda.enabled = true
process.conda = '/home/melosua/miniconda3/envs/cellranger'

And I get this error

N E X T F L O W  ~  version 23.04.1
Launching `test.nf` [spontaneous_booth] DSL2 - revision: 597ea8dcd4
executor >  local (1)
executor >  local (1)
[92/7bcc8b] process > processSampleSheet     [100%] 1 of 1, failed: 1 ✘
[-        ] process > processFastq           -
[-        ] process > processFastQC          -
[-        ] process > processMultiQC         -
[-        ] process > processCellrangerCount -
ERROR ~ Error executing process > 'processSampleSheet'

Caused by:
  Process `processSampleSheet` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env python3
  
  
  import secrets
  import string
  import pandas as pd
  
  # Function to generate random 8-character alphanumeric string
  def generate_random_string():
  	alphabet = string.ascii_letters + string.digits
  	return ''.join(secrets.choice(alphabet) for _ in range(8))
  
  print("/temp_work/melosua/project/sample_sheet_test.csv")
  # Read and subset sample sheet
  meta = pd.read_csv("/temp_work/melosua/project/sample_sheet_test.csv")
  
  # Add a unique gem_id for each sample
  meta['gem_id'] = meta.apply(lambda x: generate_random_string(), axis=1)
  
  # Write dataframe to csv
  meta.to_csv("sample-sheet-gemid.csv", header = True, index = False)

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.run: line 126: /usr/bin/activate: No such file or directory

Work dir:
  /temp_work/melosua/project/work/92/7bcc8bf468d6575875f5ecb261d246

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

The .command.log file only has this error

cat /temp_work/melosua/project/work/92/7bcc8bf468d6575875f5ecb261d246/.command.log 
.command.run: line 126: /usr/bin/activate: No such file or directory

Line 126 in .command.run is:

source $(conda info --json | awk '/conda_prefix/ { gsub(/"|,/, "", $2); print $2 }')/bin/activate /home/melosua/miniconda3/envs/cellranger

And when I try to run that line locally it successfully loads the conda environment. As far as I can tell I am not able to get nextflow to call conda using process.conda but have no idea how to go about fixing it.... any help would be greatly appreciated!

[MarcElosua]

Figured this out, the order of $PATH was conflicting and there was another conda instance elsewhere that was being detected instead.

[mribeirodantas]

Thanks for sharing how you solved it, @MarcElosua 😉.

[HagenC]

And how did you solve it? I have the exact same problem and NO clue how to solve it!

[MarcElosua]

@HagenC try checking your $PATH and make sure there are no other conda instances anywhere. To me it was reading $PATH from the opposite direction it usually does so modifying that fixed the issue.

[HagenC]

@MarcElosua Thanks for your response! It turns out the issue was that I initially ran Nextflow from a conda environment, as the local server only had Java version 8 installed. The solution was to install Java version 22 within a conda environment and then install Nextflow locally.