Chaos

With the camel-chaos JAR on the classpath, chaos is switched on with profile dev (the default of camel run), or with camel.chaos.enabled=true (for a plain Main, a test or a deterministic simulation test). It is never switched on with profile prod, also when camel.chaos.enabled=true. Without a profile, the usual case for a plain Main, Spring Boot and Quarkus, chaos stays off unless camel.chaos.enabled=true.

With camel-main, when camel.chaos.* options are set and chaos would be switched on (profile dev or camel.chaos.enabled=true), but the camel-chaos JAR is not on the classpath, camel-main logs a WARN that the options are ignored.

When chaos is switched on, it installs its hooks when the CamelContext initializes, before the routes are created, and logs a WARN that it is on. The hooks do nothing until a rule is added. When chaos is off, nothing of it is installed (only the lifecycle strategy that made this decision stays, and does nothing), and camel.chaos.rules is ignored.

camel.chaos.rules and camel.chaos.scenario are marked insecure:dev: with camel-main or Camel Spring Boot and profile prod (which sets camel.security.policy=fail) an application with rules or a scenario configured refuses to start. Without a profile they are allowed with a WARN (as with camel run --runtime=spring-boot --chaos=<file>). Where this check does not run, chaos stays off with profile prod anyway. Rules set from Java (chaosConfig().withRules(…​)) are not checked either, as for every insecure:dev option.

The options are read from the properties component, and are camel-main options (ChaosConfigurationProperties).

For example, in application.properties:

Rules can also be added, changed and removed from Java while the application runs:

addRules adds rules separated by ;, as in camel.chaos.rules, all of them or none when one is not valid, so they are in place at the same time. checkRule says what a rule matches nothing of in the context now, see below.

With the Camel CLI, against a running integration (myapp is its name or pid):

chaos-add prints the id of the new rule (r1, r2, …​) with the rule in plain words, to check that it does what was meant (Added chaos rule r1: fail half of the sends to http:stock* with HTTP status 503, for 5 minutes), and chaos-change the version of the rule that replaces it, also with the rule in plain words. --rule can be given more than once (or a rule can hold several separated by ;): the rules are added together, all or none, which matters when one must be in place before the other fires, as each chaos-add takes a moment. The ids are per integration, so chaos-add, chaos-change and chaos-remove act on one integration: the one whose name or pid is given, and without one the only integration that runs, or else the only one of them that has chaos switched on. When that is not clear (no integration runs, or several do and not exactly one has chaos switched on), the command fails with the list of the running integrations, each with whether chaos is on (exit code 2), and the name or pid of one is needed; without a name these commands never act on all of them. chaos-clear without one acts on every running integration that has chaos switched on. The rules that chaos-clear removes stay in the report; with --reset the report starts again, for a new experiment: no rules, route counts or verdict, as for an application that just started (with the same seed, and the ids of new rules going on from the last one). An id that does not exist is "not found" (exit code 3). camel get chaos lists the current rules with their id, version, state, matches and faults, the observed rate (RATE), when the rule last fired (LAST) and the effect of its faults (EFFECT), and --report shows the report, with the recent faults (also as JSON with --json); see The live state of the rules. Their NOTE column says what a lifecycle rule does to its route (route poller down, back in 12s) and why a rule did nothing when it was skipped. Without a rule to show, camel get chaos says so on stderr (no rules, chaos off, or no integration with camel-chaos).

A rule that matches nothing in the integration now is added with a warning, which chaos-add and chaos-change print, and which is logged: an endpoint= or component= pattern that matches no endpoint of the context, an at=poll for endpoints that no route consumes with a scheduled poll, a route= or step= pattern that matches no route or step id, and an expectation of a route that does not exist. Such a rule fires 0 times, and is often a typo; the warning names some of the endpoints, routes or steps there are. It is not an error, as the routes may still change (a toD creates its endpoints when it sends, and a route can be added later). The rules of camel.chaos.rules are checked when the context has started.

To see what a rule would hit before it is added, chaos-add --dry-run checks it as chaos-add would, and adds nothing. It prints the rule in plain words, then each send, poll, route or step it matches now, with its route (and the step that sends), the sends known only when they send that may match (a toD), and the warnings:

A rule that is not valid is refused with the same reason as for chaos-add. A condition (when=) is checked on each message as it comes, so the check lists what the rule would hit if the messages met it, and the send of the dead letter channel (at=dlq) is not listed, as it is the error handler’s.

The commands go through the chaos action of the CLI connector, which calls ChaosManager: a rule goes through the same parser as camel.chaos.rules, and one that is not valid is refused with the reason. They work only when chaos is switched on in the integration (see Switching it on): they cannot switch it on, and with profile prod they are refused.

The chaos developer console (/q/dev/chaos with camel run --console) shows the same rules and report, as text or JSON. It only reads: a rule cannot be added, changed or removed over HTTP.

A scenario is the rules of an experiment in a file of the project, so it can be run again, shared and kept with the code, instead of rules typed once at the command line. By convention it is in a chaos folder, with the extension .chaos:

To start an application with a scenario:

--chaos=<file> adds camel-chaos as --chaos does, and sets camel.chaos.scenario to the file (it must end with .chaos; --chaos Shop.java still runs Shop.java). A plain Main sets camel.chaos.scenario itself. camel run * and camel export leave .chaos files out: they are not routes, and they do not go into an exported project.

To load a scenario into an application that runs:

A scenario is an experiment of its own: loading one removes the rules there are and starts a new report (as chaos-clear --reset does), then adds the rules of the scenario, all of them or none. A line that is not valid is refused with its line number (Chaos scenario outage.chaos, line 4: Invalid chaos rule […​]), and then the rules there are stay. The ids of the rules go on from the last one (r4 above, after three rules). In the report, each rule of a scenario says which one it came from ((scenario outage.chaos) in the text, scenario in the JSON).

The chaos screen of the Camel TUI saves the rules there are as a scenario (key s), with the seed: rules tried by hand become a file of the project, to run again. On its Errors tab, ! makes a rule of a real error, which fails the same way again (the same exception and message at the same step, or the same HTTP status at the same endpoint, ten times): fix the bug, then prove the fix against the same failure, and keep the rule in a scenario. And t on a rule asks its AI panel to make a test of it: a JUnit test with ChaosAssertions for a Maven project, or a scenario with an expectation and the camel run --chaos=<file> --max-seconds=<seconds> command, whose exit code is the verdict, for a folder that runs with camel run. The test has the seed, and the rule with draw=<its id>, so it fails the same messages as the rule did (see Time and the seed).

A rule is one line: the target, the fault and its parameters as name=value, then at, in, when, rate, from and until, and optionally an expectation. A value with spaces or a ; is quoted with " or ' (\", \' and \\ inside quotes are a quote and a backslash).

The patterns of the targets are matched as text, with for any characters; they are not regular expressions. The match is case-sensitive and a can stand anywhere in the pattern, unlike the patterns of interceptSendToEndpoint, which ignore case and can be regular expressions: a rule, also one from a tool, cannot make matching slow. An endpoint= pattern is matched against the endpoint URI, with or without its query parameters, and the // after the scheme does not count (kafka:orders* matches kafka://orders?brokers=b1). A component= pattern is matched against the scheme of the URI only, so a pattern with a : (component=http:localhost) is rejected, with the endpoint= pattern to use instead. Some components give their endpoints the URI of their address: undertow:http://inventory/stock has the endpoint URI http://inventory/stock. When the scheme of the endpoint URI is not a name of the component in the context, chaos puts the default name of the component in front, and matches undertow:http://inventory/stock (with component=undertow, not component=http). A camel-netty-http endpoint has the endpoint URI netty-http://http://inventory/stock, and a camel-netty-http component added under another name is matched with netty-http in front too. A route= pattern is matched against the route id, and a step= pattern against the id of the step (to1, or the id set with id("enrich")).

stall and lifecycle act on a route, not on its messages: route=poller stall duration=20s suspends route poller and resumes it 20 s later; route=orders lifecycle action=stop duration=1m stops and starts it again; route=orders lifecycle action=remove duration=10s removes it and adds the same route definition back. Each rule acts when it is added (a rule of camel.chaos.rules when the context has started), or from= later (route=poller stall duration=20s from=+1m), so at, rate and until are rejected; an expectation (expect=, within=) can be added.

A flapping route goes down more than once: route=orders stall duration=10s repeat=3 every=1m suspends route orders for 10 s three times, each time a minute after the last one went down. every must be longer than the duration, so the route is back before it goes down again, and the rule must be done within camel.chaos.maxDuration ((repeat - 1) * every + duration); repeat is at most 100. Each round acts as the first: a route that someone else stopped meanwhile is left as it is, and the rule ends there (skipped). Removing the rule between two rounds ends it, with the route as it is (back). The route= target is the id of one route, not a pattern, and the route must exist when the rule is added: a rule for a route that does not exist is rejected (with camel.chaos.rules the context does not start), and so is a duration longer than camel.chaos.maxDuration. While a lifecycle rule has a route removed, a rule for that route is still accepted (to change the rule, for example).

The action and the comeback run on a scheduled executor of the ExecutorServiceManager of the context (one thread, whose name ends with Chaos, created when the first such rule acts), not on the thread that adds the rule, so adding the rule returns at once, even while the route waits for its exchanges in flight to finish. The actions and comebacks of all rules run one after another on that thread, so a comeback can wait for the action of another rule, such as a stop that waits for its exchanges in flight; a rule that replaces one whose route is down acts after that route came back. The route is suspended, stopped, resumed and started through the route controller of the context (CamelContext.getRouteController()), and removed with CamelContext.removeRoute. A route that cannot be suspended (its consumer does not support it) is stopped and started instead, as Camel does for any suspend.

Chaos only brings back what it took down:

With the supervising route controller (camel.main.routeControllerSuperviseEnabled=true), a stop or suspend by chaos is a manual operation for the controller: it does not restart the route while it is down, and the comeback starts the route through the controller, which supervises it again (and retries a start that fails, with its back-off). A route that action=remove adds back is started through the route controller too, when adding its definition did not start it (the supervising controller takes a route that is added again under supervision, but does not start it), so it is supervised again as well. The supervising controller starts the routes in the background, so a rule of camel.chaos.rules can find its route not started yet, and then does nothing; add such a rule once the route runs.

action=remove needs the route definition of the route, so it works for the routes of the route model (Java, XML and YAML DSL). The definition that is added back is the one that was removed, as it was prepared, so the route comes back with the route configurations it had. A route created from a route template or a kamelet is rejected for action=remove (removing it closes the local beans of the template); stop and stall work for it.

The report shows what each lifecycle rule did, with the times after the rule was added, for example:

With repeat= the line tells the round, and between two rounds when the next one starts (round 1 of 3: suspended route orders at +2ms, resumed at +10s5ms (down 10s3ms); round 2 in 49s).

In JSON each action counts in fired.lifecycle, and the rule has a lifecycle object: action, route, duration, state (pending, acting, down, comingBack, back, skipped, left, failed or cancelled; pending also between two rounds), round (the times it took the route down), repeat and every (with repeat=), actsIn (the milliseconds until it acts, again), downAt and backAt (milliseconds after the rule was added, of the last round), backIn while the route is down, endedEarly when the rule was removed while the route was down, and the reason when it did not go as planned. camel get chaos --json and the chaos dev console give the same object.

ChaosManager.report() gives the report, as plain text (toText()) for a person and as JSON (toJson()) for a tool, and it is logged when the context stops (camel.chaos.reportOnShutdown), if any rule was added. It states the seed, each rule with its id and version, the rule in plain words (description, see below), its matches, the faults that fired at each hook and the ones that were skipped (with the reasons, such as a stream body, counted in skippedReasons), its state (active, waiting for its start with from=, ended, removed or replaced), and per route. The rules are in the order of their ids, and the versions of an id from the latest. Per route:

The routes are counted while there are rules (also ended ones). Camel does not call the route policies while the context stops, so the exchanges that finish while the context drains them are not counted.

Without an expectation the verdict is PASS, as chaos makes exchanges fail on purpose, and the report does not say that a message was lost, except for the messages that a rule with at=dlq kept from reaching the dead letter channel, which then handled them: those count as lost (in the report of the rule as lostAtDlq), and make the verdict FAIL, with the reason. An expectation, expect=orders:100 within=2m, says that route orders completes 100 more exchanges after the rule is added and before the time is up (by default until the rule ends: its from plus its until time, else camel.chaos.maxDuration, which is also the longest within after the start of the rule). An exchange of the route counts once, the first time it completes in the route. An exchange that the error handler handled (for example one that went to the dead letter channel) completes, and so counts as completed, not as lost. The route id is not checked when the rule is added, as the route may come later (a WARN says so when a rule is added or changed, or the context starts, and no route has that id). An expectation is met when the count is reached in time, and unmet when the time is up without it: then the verdict is FAIL, with the reason, and the missing exchanges count as lost (100 expected, 97 completed: 3 lost). Before that it is pending, and so is the report (isFinal() is false): a tool must treat a PASS with a pending expectation as not decided yet. An expectation is cancelled when its rule is removed or changed first, or when the context stops first (also when the context is started again): the messages may still arrive, so it is neither met nor unmet, and it does not keep the report from being final.

A run with chaos ends with the verdict of the report as its exit code, so a scenario can gate a build:

This is for runs in the foreground, with the default runtime or --runtime=main; a run in the background has no exit code to give.

The delay fault (and an error with after=) waits through TimeSource of camel-util, and the start and the end of a rule, its bursts, the time of an expectation and the times of a lifecycle rule are read through it too; the action and the comeback of a lifecycle rule are tasks of a scheduled executor of the ExecutorServiceManager. So in a deterministic simulation test (camel-test-dst) they run in virtual time: a 10 minute stall takes milliseconds, and with camel.chaos.seed set to the seed of the run, the same seed gives the same report again, times included. Chaos does not depend on camel-test-dst.

ChaosLifecycleDstTest in the tests of camel-chaos is such a test, and an example to copy: a route consumes a queue (a fake with a scheduled poll consumer) and calls a downstream system asynchronously (DstCamel.asyncStep, as a world needs for a route that is suspended or stopped while an exchange is in a downstream call). A stall or lifecycle rule, added while an exchange is in flight, takes the route down for up to 10 minutes while a step= rule fails the downstream call at a rate, and the test checks from the report that no message is lost (an expectation), that the redeliveries add up to the step faults, and that the route was down for exactly the duration.

The seed fixes the sequence of random draws of each rule (rate=<probability>, the delays of a range such as duration=100ms..2s, the random header values), not which exchange gets which draw: with several threads (concurrent consumers, parallel processing) the threads decide that, so two runs with the same seed can give different reports. With one thread and the same exchanges in the same order, the same seed gives the same faults.

Each rule draws by a key, with the seed: its id when it is added (r2), and its place in a scenario (the second rule draws as r2) or in camel.chaos.rules. So a rule added while the application runs, written alone in a scenario or a test, draws other choices there (it is the first rule, r1). draw=<id> names the key: route=shipping error rate=0.2 draw=r2 with the same seed fails the same messages as rule r2 did. The test that the Camel TUI makes of a rule (key t) writes it.

A finding of a chaos session becomes a regression test: the same rules, the same seed, and assertions on the report. ChaosAssertions throws an AssertionError with the report when it fails, so it works with any test framework. Add camel-chaos as a test dependency, and switch chaos on in the test with camel.chaos.enabled=true:

The rules can also come from a scenario (camel.chaos.scenario=classpath:chaos/outage.chaos), or be added in the test with ChaosManager.get(context).addRule(…​). With camel-test-main-junit6, set the same properties in the test configuration. In a deterministic simulation test (see Time and the seed), the times of the rules run in virtual time, and the report is read with ChaosManager.report() when the simulation is done.

The rules are also described as data, for the tools that help to write them (a form in a terminal UI, an AI that turns a request into rules, the docs): META-INF/org/apache/camel/chaos/chaos-model.json in the camel-chaos JAR, also in the Camel catalog (CamelCatalog.chaosModelJSon()). It has the targets, the hooks of each target and the faults each one offers (and whether it is reached only with at=), the parameters of each fault with their types and limits, the options, the kinds and ops of an expectation, the limits, and recipes: known rules in plain words, grouped by what goes wrong, each a rule with placeholders (${endpoint}, ${rate}) and what to pick them from (the endpoints the routes send to, the routes, the steps). To guide the one who picks a recipe, each says what it does, what it can uncover, when not to use it, and what to watch while it runs, such as:

The model is made from the tables of the parser, so it says what the parser accepts; a tool still adds a rule through ChaosManager or the CLI connector, which check it as for any other rule.