apache-flink logging Logging configuration

Example

Local mode

In local mode, for example when running your application from an IDE, you can configure log4j as usual, i.e. by making a log4j.properties available in the classpath. An easy way in maven is to create log4j.properties in the src/main/resources folder. Here is an example:

log4j.rootLogger=INFO, console

# patterns:
#  d = date
#  c = class
#  F = file
#  p = priority (INFO, WARN, etc)
#  x = NDC (nested diagnostic context) associated with the thread that generated the logging event
#  m = message


# Log all infos in the console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss.SSS} %5p [%-10c] %m%n

# Log all infos in flink-app.log
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=flink-app.log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss.SSS} %5p [%-10c] %m%n

# suppress info messages from flink
log4j.logger.org.apache.flink=WARN

Standalone mode

In standalone mode, the actual configuration used is not the one in your jar file. This is because Flink has it own configuration files, which take precedence over your own.

Default files: Flink ships with the following default properties files:

• log4j-cli.properties: Used by the Flink command line client (e.g. flink run) (not code executed on the cluster)
• log4j-yarn-session.properties: Used by the Flink command line client when starting a YARN session (yarn-session.sh)
• log4j.properties: JobManager/Taskmanager logs (both standalone and YARN)

Note that ${log.file} default to flink/log. It can be overridden in flink-conf.yaml, by setting env.log.dir,

env.log.dir defines the directory where the Flink logs are saved. It has to be an absolute path.

Log location: the logs are local, i.e. they are produced in the machine(s) running the JobManager(s) / Taskmanager(s).

Yarn: when running Flink on Yarn, you have to rely on the logging capabilities of Hadoop YARN. The most useful feature for that is the YARN log aggregation. To enable it, set the yarn.log-aggregation-enable property to true in the yarn-site.xml file. Once that is enabled, you can retrieve all log files of a (failed) YARN session using:

yarn logs -applicationId <application ID>

Unfortunately, logs are available only after a session stopped running, for example after a failure.