Operations Notes

In this section, we provide some notes that could be of interest for operations.

Important

As we have many different variations of the products, especially in operations, all examples here are provided in the barest way in form of docker commands. Please adapt that to whatever suites your environment.

Triggering loads

In all of those example, we assume that

You deployed our application already
Our application shares a network app_network
RabbitMQ is running as service rabbitmq and is reachable under that name in the app_network

All examples run a new rabbitmq container, will send a message and then exit. As such, the exact version of RabbitMQ that you use there is not really relevant - it is only used as a client to send a message. But feel free to change the latest label to whatever you have running as a server.

You can also exec into a running rabbitmq container and just execute the relevant rabbitmqadmin command.

Trigger actions through Graviton Messenger

Messenger is mainly a messaging middleware - many actions it does are mainly coordinated things that could triggered directly but through different means.

But we implemented a simple way to trigger the most common actions through queue messages, Messenger will then do whatever is needed in those cases.

Info

Messenger is subscribed to the routing key messenger.load on the messenger exchange.

Commands

Trigger a full “BSA load”:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=messenger routing_key=messenger.load payload=BSA-FULL

Only download the new files from BSA:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=messenger routing_key=messenger.load payload=BSA

Trigger a eVoja Data Load:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=messenger routing_key=messenger.load payload=EVOJA

Trigger a Swissquant Data Load:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=messenger routing_key=messenger.load payload=SWISSQUANT

Trigger actions through Graviton Loader

You can also send a message directly to Graviton Loader to just trigger load elements that are relevant to eVoja.

Graviton Loader takes JSON as message payload. You specify what to load through the parameters.load element, and you can control if dependencies should be loaded by the parameters.loadDeps boolean element.

Info

Graviton Loader is subscribed to the routing key ml3k.load on the ml3k exchange.

Commands

You can trigger the entire bulk load with the ALL element:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=ml3k routing_key=ml3k.load \
'payload={"loadId":1,"parameters":{"load":"ALL","loadDeps":true }}'

You can trigger a single element (and only that, without dependencies) with its name (like Customer):

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=ml3k routing_key=ml3k.load \
'payload={"loadId":1,"parameters":{"load":"Customer","loadDeps":false }}'

You can trigger multiple elements comma seperated without dependencies (like Customer and Account) like this:

docker run --rm \ 
--network app_network \
nexus.pub.swisscom.evoja.scbs.ch/evoja/rabbitmq:latest \
rabbitmqadmin --host=rabbitmq --username=guest --password=guest --vhost=/ publish exchange=ml3k routing_key=ml3k.load \
'payload={"loadId":1,"parameters":{"load":"Customer,Account","loadDeps":false }}'

RabbitMQ notes

Reactions to memory problems

There are (but should not) scenarios, where RabbitMQ allocated unusual amounts of memory.

This is mainly caused by the fact that temporary queues have been opened, received messages, but those messages were never collected (for whatever reason). This was historically caused by Graviton Loader, but was optimized in version 3.1.0 which was included since release EVO20R1 BF2.

In earlier releases, there could be situations where RabbitMQ reaches its memory high watermark (which defaults to 40%) and somes messages could be dropped or not be accepted for delivery.

As a workaround, you can either remove any container memory limit to a degree that 40% of that limit is sufficient, or you can set a new memory high watermark by setting the environment variable RABBITMQ_VM_MEMORY_HIGH_WATERMARK to a higher value (like 0.8 or 0.9) and redeploy.

Generally, if you have a Graviton Loader version older than 3.1.0, you are advised to update to fix that problem.

Analysis

RabbitMQ offers some handy commands that gives you insights into what is going on.

Firstly, it’s easiest to just expose (or otherwise access) port 15672 of the RabbitMQ container - there, a nice Web interface awaits you (default credentials guest/guest) where you can see everything. Just do that.

RabbitMQ UI

If for some reason you can not do that, exec into the running RabbitMQ container. There, with the rabbitmqctl command, you can query everything that you may want to know.

Please check the official documentation for the tool for whatever you need to know.

Some examples here:

List all queues

bash-5.0# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name	messages
clan	0
ssp	0
ml3k-controller	0
statekeeper	0
amq.gen-wmcsW_6-Vu2jfqamtsF4Qg	0
notification-email	0
statekeeper-queue	0
ml3k-Stager	0
ml3k-TargetLoader	0
statekeeper-ml3k	0
ml3k-daemon	0

List all exchanges

bash-5.0# rabbitmqctl list_exchanges
Listing exchanges for vhost / ...
name	type
amq.direct	direct
amq.fanout	fanout
	direct
amq.rabbitmq.trace	topic
amq.match	headers
amq.headers	headers
statekeeper	topic
amq.topic	topic
ml3k	topic

List all consumers (connections)

bash-5.0# rabbitmqctl list_consumers
Listing consumers in vhost / ...
queue_name	channel_pid	consumer_tag	ack_required	prefetch_count	active	arguments
clan	<rabbit@acde2b545329.1591648064.1087.0>	amq.ctag-Y7gGaXAIYXGTXiHKWV6qSw	true	0	true	[]
ssp	<rabbit@acde2b545329.1591648064.1092.0>	amq.ctag-9yODnfvJgvWb0XdFcS-rqQ	true	0	true	[]
statekeeper	<rabbit@acde2b545329.1591648064.1031.0>	amq.ctag-RQUWPKSpUhpGpJ-O9bkBCg	true	0	true	[]
amq.gen-wmcsW_6-Vu2jfqamtsF4Qg	<rabbit@acde2b545329.1591648064.30675.1>	ml3k	false	0	true	[]
notification-email	<rabbit@acde2b545329.1591648064.1063.0>	amq.ctag-o6tjMxqlibGWqBr6SrAzCw	true	0	true	[]
statekeeper-queue	<rabbit@acde2b545329.1591648064.1043.0>	amq.ctag-k6VdrVUBn2e6Nt6FRgQSoA	false	0	true	[]
statekeeper-ml3k	<rabbit@acde2b545329.1591648064.1047.0>	amq.ctag-7VlxbdDByAT6rnafbHeyTw	false	0	true	[]
ml3k-daemon	<rabbit@acde2b545329.1591648064.30675.1>	controller	false	0	true	[]

Each command also allows you to pass a list of elements (in case you need more information).

So to list all queues with specific fields:

bash-5.0# rabbitmqctl list_queues name,pid,exclusive,messages_ready_ram
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name	pid	exclusive	messages_ready_ram
clan	<rabbit@acde2b545329.1591648064.1101.0>	false	0
ssp	<rabbit@acde2b545329.1591648064.1096.0>	false	0
ml3k-controller	<rabbit@acde2b545329.1591648064.2027.0>	false	0
statekeeper	<rabbit@acde2b545329.1591648064.1056.0>	false	0
amq.gen-wmcsW_6-Vu2jfqamtsF4Qg	<rabbit@acde2b545329.1591648064.30683.1>	true	0
notification-email	<rabbit@acde2b545329.1591648064.1079.0>	false	0
statekeeper-queue	<rabbit@acde2b545329.1591648064.1067.0>	false	0
ml3k-Stager	<rabbit@acde2b545329.1591648064.30661.1>	false	0
ml3k-TargetLoader	<rabbit@acde2b545329.1591648064.30647.1>	false	0
statekeeper-ml3k	<rabbit@acde2b545329.1591648064.1086.0>	false	0
ml3k-daemon	<rabbit@acde2b545329.1591648064.30678.1>	false	0

You will find all possible values for the fields in the official documentation as referenced above.

Verbose log

If you need debug log output for the RabbitMQ container, you can do that during runtime as described in the official documentation.

exec into the container and execute:

rabbitmqctl set_log_level debug