↓
  1. Posts/

Python stdin stderr logging

···

Logging in Python is quite straight forward, except for by the BasicConfig writes to stderr and not to stdout

1
2
3
4
5
6
7
8

  import logging

  log_format = '%(name)s - %(levelname)s - %(message)s'
  logging.basicConfig(format=log_format)
  log = logging.getLogger(__name__)
  log.warning('I print to stderr by default')
  log.info('I print to stderr by default')
  log.error('I print to stderr by default')

This can cause info logs to be written to stderr and in AWS Glue it ends up in the error logs.

Setup logs to stdout 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

  import logging
  import sys

  root = logging.getLogger()
  root.setLevel(logging.DEBUG)

  handler = logging.StreamHandler(sys.stdout)
  handler.setLevel(logging.DEBUG)
  log_format = '%(name)s - %(levelname)s - %(message)s'
  formatter = logging.Formatter(log_format)
  handler.setFormatter(formatter)
  root.addHandler(handler)

  log =logging.getLogger(__name__)
  log.info("Writes to stderr by default')

Using print()

Clearly using logging is a but complicated for some usecases.

print() in python writes to stdout by default. But we can make it write to stderr.

1
2
3
4

  #!/usr/bin/env python
  import sys
  print('this prints to stdout')
  print("this prints to stderr", file=sys.stderr)

AWS Glue Python Shell logs

You can use print statements in your Glue Python Shell job for logging. Glue captures stdout and stderr by default.

There is no need to setup a logger in Glue Python Shell job.

Redirecting stderr and stdout

Most shells will show you stdout and stderr logs in the console. You can redirect error logs and stdout logs into different files.

Use 1> to redirect stdout and 2> to redirect stderr.

>> can be used to append operation

1
2

  myscript 2> /tmp/error.log 1> /tmp/out.log # overwrite
  myscript 2>> /tmp/error.log 1>> /tmp/out.log # append