rant: run --all and stack features are unusable in CI environments

[dennislapchenko]

I am aware this is a long standing issue of terragrunt and CI enviroments (like Atlantis). Issue of not being able to apply with plan file units containing inter-dependencies. Without run --all at least units can be slowly applied one by one (which is also a pain, but at least it works, one commit per unit). Testing out stacks just now I was hopeful that it has solved the CI issues. Sadly it is still exactly the same story as with run --all.

Key problem:
In CI environments like Atlantis, teams need deterministic plan files with correct outputs for each inter-dependency. Having mock outputs getting applied once and then re-running the apply seems like a clunky hack. Running applies without plan files is also dangerous.

What I am trying to understand, and perhaps this helps some other developers' sanity, are two things:

1. Are run --all and stack commands/infrastructures really intended to be always run manually? It is hard to believe so I am open to learn some key aspects that I might be missing. Perhaps gruntworks' internal pipeline platform has this solved and is a paid offering?
2. If first assumption is correct, perhaps there are some community solutions which enable flows like those with Atlantis to work reliably with stack feature?

[yhakbar]

Yes, the Gruntwork Pipelines product does have native support for Terragrunt Stacks:
https://www.gruntwork.io/platform/pipelines

Unfortunately, the way that OpenTofu and Terraform currently work, there is no way to signal to OpenTofu that some variables are unknown, and might need to be recalculated, even when a plan is saved. I've documented a workaround here in the past.

We also champion this issue to support this behavior natively in OpenTofu better. The OpenTofu engineer working on behalf of Gruntwork, @diofeher, has done some good work exploring this problem space, and might have an RFC to address it in OpenTofu in the near future.

[brontolinux]

Hello there.

I have a case that could be related to this discussion. If it's not the case (e.g. if an issue should be opened instead), just let me know and I'll amend.

We are observing a strange behaviour with terragrunt 0.89.4 and runner pools, where everything works fine when using run --all on the command line, but fails to find units to run on when run in Atlantis.

E.g. say that I have a stack in the directory testdir, with three units, each one in a subdirectory called prod, nonprod, uat, each one with its own terragrunt.hcl.

If I run terragrunt run --all plan in testdir on a terminal, terragrunt happily finds the units in the subdirectories and adds them to the queue:

Using runner pool for stack .
 The runner-pool runner at . will be processed in the following order for command plan:
- Unit ./nonprod
- Unit ./prod
- Unit ./uat

The same will fail in Atlantis, where the runner pool won't find any units in stack .

Using runner pool for stack .
 could not find any subfolders with Terragrunt configuration files
 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

unless I add these to the workflow:

        - env:
            # Preserve the usual behaviour when running commands like `terragrunt run --all plan`, when
            # we want it do descend in all units below the current one.
            # See https://terragrunt.gruntwork.io/docs/reference/cli/commands/run#queue-include-dir
            name: TG_QUEUE_INCLUDE_DIR
            value: "**"
        - env:
            # ditto
            name: TG_STRICT_CONTROL
            value: "double-star"

I can't really understand what it is that makes those variables needed in Atlantis, while everything works just fine. Unsure if it's another case for usability of terragrunt in CI/CD or something else, so trying with this discussion first.

Cheers,
--bronto

[yhakbar]

I'm seeing the following error, which is odd:

could not find any subfolders with Terragrunt configuration files

Are you sure you're running Terragrunt where you expect? Try running pwd && terragrunt find --json right before where you would run terragrunt run --all plan. Also, make sure you're not in a strange symlinked directory or anything.

[brontolinux]

Thanks for following up on this so far 🙏

I'm quite sure I am in the right place, yes. The error message says:

running 'sh -c' 'terragrunt run --all plan -- -input=false $(printf '%s' $COMMENT_ARGS | sed 's/,/ /g' | tr -d '\\') -no-color -out ../../../../atlantis.tfplan' in '/home/atlantis/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__': exit status 1: running "terragrunt run --all plan -- -input=false $(printf '%s' $COMMENT_ARGS | sed 's/,/ /g' | tr -d '\\\\') -no-color -out ../../../../atlantis.tfplan" in "/home/atlantis/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__": 

and as you can see in this output I just collected (after ensuring that all the relevant variables are set from Fargate in the Atlantis environment), I am in that directory. terragrunt find can't find any unit, but Linux' find finds the three terragrunt.hcl files easily:

atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ terragrunt run --all plan
14:03:30.657 DEBUG  Terragrunt Version: 0.89.4
14:03:30.657 DEBUG  TF_PLUGIN_CACHE_DIR already set to ../../../../../../../../../../.terraform.d/plugin-cache, skipping auto provider cache dir
14:03:30.664 DEBUG  Provider service initialized with cache dir: ../../../../../../../../../../.terraform.d/plugin-cache, user cache dir: ../../../../../../../../../../.terraform.d/plugins
14:03:30.665 INFO   Terragrunt Cache server is listening on 127.0.0.1:36699
14:03:30.665 INFO   Start Terragrunt Cache server
14:03:30.665 DEBUG  Searching for stack files in .
14:03:30.666 DEBUG  Starting provider cache service with cache dir: "../../../../../../../../../../.terraform.d/plugin-cache"
14:03:30.666 DEBUG  Provider cache service temp dir: /tmp/terragrunt/providers
14:03:30.666 DEBUG  Provider cache service is ready to process requests
14:03:30.683 INFO   Using runner pool for stack .
14:03:30.687 DEBUG  Provider cache service shutting down...
14:03:30.687 DEBUG  Provider cache service shutdown complete
14:03:30.687 INFO   Shutting down Terragrunt Cache server...
14:03:30.687 INFO   Terragrunt Cache server stopped
14:03:30.687 ERROR  could not find any subfolders with Terragrunt configuration files
14:03:30.687 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ pwd && terragrunt find --json
/home/atlantis/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__
14:04:47.585 DEBUG  Terragrunt Version: 0.89.4
14:04:47.585 DEBUG  TF_PLUGIN_CACHE_DIR already set to ../../../../../../../../../../.terraform.d/plugin-cache, skipping auto provider cache dir
[]
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ find . -type f -name terragrunt.hcl -print
./nonprod/terragrunt.hcl
./uat/terragrunt.hcl
./prod/terragrunt.hcl
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ 

These are the relevant part of the environment:

atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ env | awk '/^T[FG]_/ && !/password|token/' | sort
TF_PLUGIN_CACHE_DIR=/home/atlantis/.terraform.d/plugin-cache
TG_BACKEND_BOOTSTRAP=true
TG_LOG_LEVEL=DEBUG
TG_PROVIDER_CACHE=true
TG_PROVIDER_CACHE_DIR=/home/atlantis/.terraform.d/plugin-cache
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ 

As for symlinks, I ran a find /home/atlantis -type l -print, and found symlinks only inside some old .terraform directory from the time when we didn't have the provider cache. Running find /home/atlantis -path '*/.terraform' -prune -o -type l -print after that confirmed that there were no other symlinks around: the output was empty.

But wait, now comes the fun. If I set TG_QUEUE_INCLUDE_DIR and TG_STRICT_CONTROL, terragrunt find will still return an empty list:

atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ export TG_QUEUE_INCLUDE_DIR="**"
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ export TG_STRICT_CONTROL="double-star"
atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ terragrunt find --json
14:30:24.696 DEBUG  Enabled strict control(s): double-star
14:30:24.702 DEBUG  Terragrunt Version: 0.89.4
14:30:24.702 DEBUG  TF_PLUGIN_CACHE_DIR already set to ../../../../../../../../../../.terraform.d/plugin-cache, skipping auto provider cache dir
[]

But terragrunt run --all plan though...:

atlantis@ip-10-245-22-77:~/.atlantis/repos/rikstv/sre/rikstv.terraform.infra.atlantistesting/21/apps_sre___selftest__/apps/sre/__selftest__$ terragrunt run --all plan
14:30:58.205 DEBUG  Enabled strict control(s): double-star
14:30:58.205 DEBUG  Included directories set. Excluding by default.
14:30:58.206 DEBUG  Terragrunt Version: 0.89.4
14:30:58.206 DEBUG  TF_PLUGIN_CACHE_DIR already set to ../../../../../../../../../../.terraform.d/plugin-cache, skipping auto provider cache dir
14:30:58.214 DEBUG  Provider service initialized with cache dir: ../../../../../../../../../../.terraform.d/plugin-cache, user cache dir: ../../../../../../../../../../.terraform.d/plugins
14:30:58.215 INFO   Terragrunt Cache server is listening on 127.0.0.1:34711
14:30:58.215 DEBUG  Searching for stack files in .
14:30:58.215 INFO   Start Terragrunt Cache server
14:30:58.215 DEBUG  Starting provider cache service with cache dir: "../../../../../../../../../../.terraform.d/plugin-cache"
14:30:58.215 DEBUG  Provider cache service temp dir: /tmp/terragrunt/providers
14:30:58.215 DEBUG  Provider cache service is ready to process requests
14:30:58.228 INFO   Using runner pool for stack .
. . . 
14:30:58.469 DEBUG  Unit ./nonprod is included by glob **
14:30:58.469 DEBUG  Unit ./prod is included by glob **
14:30:58.469 DEBUG  Unit ./uat is included by glob **
14:30:58.469 DEBUG  Stack at ../__selftest__:
  => Unit ./nonprod (excluded: false, assume applied: false, dependencies: [])
  => Unit ./prod (excluded: false, assume applied: false, dependencies: [])
  => Unit ./uat (excluded: false, assume applied: false, dependencies: [])
14:30:58.470 INFO   The runner-pool runner at . will be processed in the following order for command plan:
- Unit ./nonprod
- Unit ./prod
- Unit ./uat

14:30:58.470 DEBUG  Runner Pool Controller: starting with 3 tasks, concurrency 2147483647
14:30:58.470 DEBUG  Runner Pool Controller: found 3 readyEntries tasks
14:30:58.470 DEBUG  Runner Pool Controller: running ./nonprod
14:30:58.470 DEBUG  Runner Pool Controller: running ./prod
14:30:58.470 DEBUG  Runner Pool Controller: running ./uat
14:30:58.470 DEBUG  Runner Pool Controller: found 0 readyEntries tasks
14:30:58.470 DEBUG  [nonprod] Running ./nonprod
14:30:58.470 DEBUG  [prod] Running ./prod
14:30:58.470 DEBUG  [uat] Running ./uat

So it seems like there is some inconsistency, unless terragrunt find is supposed to ignore those environment variables.

Anything else I can test, or that you'd like me to try?

[brontolinux]

Hello everyone

@yhakbar: did you have time to review my findings above?

Meanwhile, I did some more debugging, removing all environment settings from root.hcl and moving them to Fargate's environment, and confirmed that they are all set for Atlantis. In more detail, I added this pre-hook in Atlantis:

      - run: echo -n "Atlantis environment before multienv (secrets removed):\n$(env | egrep -vi '(secret|token|password)' | sort)\n"

and atlantis showed all TG_*, TF_* variables it was supposed to show, so the environment is sane.

[yhakbar]

terragrunt find will ignore hidden directories by default, which is probably what you're running into there. Try adding --hidden to your find call. Also the queue related flags are ignored by find. We intentionally don't support them there.

The filepath that you're showing is really weird. It's possible that's what is confusing Terragrunt. I'd recommend retesting on the latest version of Terragrunt, and testing on a filepath that doesn't have all those . and __. If units get picked up after doing that, you can try reintroducing special characters until the behavior is reproducible.

[brontolinux]

You are onto something here @yhakbar. In fact, Atlantis' default data dir is ~/.atlantis, where Atlantis will store its database, checked out repos, Terraform plans. I did a few experiments inside the Atlantis container, first by running unset TG_QUEUE_INCLUDE_DIR TG_STRICT_CONTROL, and then copying data in a subdirectory of the home directory. When the subdirectory was hidden (say, .foo), no units were discovered; when the subdirectory wasn't hidden (say, foo), everything worked just fine.

While I agree that it's fine and appropriate that terragrunt skips searching for units in a stack when they are in a hidden subdirectory under the working directory, I can't really make sense of why it should ignore everything when the working directory itself is inside an hidden directory. Was there a good reason to do so, or it's just a bug? In case, I will gladly open a separate issue for that.