Miraheze Offline Help

Tech:Graylog

Graylog is a centralized log management solution used to collect and analyze logs across WikiTide servers. The web interface is available at https://logging.wikitide.net/. Access is restricted to Technology team members, who can authenticate using their LDAP credentials.

Architecture

Graylog currently runs on graylog161.fsslc.wtnet. It includes the following services:

  • graylog-server – the main log collection and processing service
  • opensearch – handles indexing and searching of log messages
  • mongod – stores Graylog’s configuration data
                     +----------------------------------+                                        +------------------------------------------+
                     | test151.fsslc.wtnet             |                                        | graylog161.fsslc.wtnet                  |
                     | +------------+                   |                                        |                                          |
                     | |            |                   |                                        | +---------------+      +---------------+ |
+----------------+   | | MediaWiki  |-\                 |                                        | |               |      |               | |
|                |   | |            |  ---\             |                                    ------|graylog-server -------- opensearch    | |
| WikiTide User  |   | +------------+      --\          |             12210/tcp   ----------/    | |               |\     |               | |
|                |   |                   +------------+ |              ----------/               | +-------|-------+ \    +---------------+ |
+----------\-----+   | +-------------+   |            | |   ----------/                          |         |          |                     |
            ------\  | |             |   | syslog-ng  -----/          TLS encrypted              |          \         \                     |
                   ----|   NGINX     -----            | |                                        |  +-------|-------+  \  +---------------+ |
                     | |             |   +------------+ |                                        |  |               |   \ |               | |
                     | +-------------+    /             |                                        |  |     NGINX     |    ||    mongod     | |
                     |                   /              |                                        |  |               |     |               | |
                     | +-------------+  /               |                                        |  +------|--------+     +---------------+ |
                     | | /dev/log    | /                |                                        +---------|--------------------------------+
                     | | (kernel logs|/                 |                                                  |                                 
                     | | , etc.)     |                  |                                                  |                                 
                     | +-------------+                  |                                                  |                                 
                     |                                  |                                                  |                                 
                     +----------------------------------+                                        +---------|---------+                       
                                                                                                 |                   |                       
                                                                                                 |  Tech Team member |                       
                                                                                                 |                   |                       
                                                                                                 +-------------------+

In the above architecture, syslog-ng on test151 receives logs locally and forwards them to graylog-server. To enable this, set base::syslog::syslog_daemon to syslog_ng in Puppet. The base::syslog class will:

  • Install syslog-ng
  • Configure it to listen on 127.0.0.1:10514 for local logs
  • Use the system source for kernel and system logs

Streams

Streams in Graylog define how log messages are routed and who can access them. All messages go to the All messages stream by default. Custom streams restrict access based on roles. For example, MediaWiki Specialists only see MediaWiki and NGINX streams.

Querying the Data

Graylog uses a Lucene-like syntax for queries. To view available fields, go to Graylog Search and click the Fields sidebar tab.

Examples:

  • View NGINX logs for your IP:
  • nginx_remote_addr:"1.2.3.4"
  • View all SSH logs:
  • application_name:"sshd"
  • View all MediaWiki errors and warnings:
  • application_name:"mediawiki" AND (mediawiki_level:"ERROR" OR mediawiki_level:"WARNING")
  • View logs for a specific MediaWiki request:
  • mediawiki_reqId:"642df1294318d7551fab367e"

Access

The Graylog interface is not directly accessible without a SOCKS5 proxy, similar to Proxmox. Port 8089 is used locally for proxying. You may tunnel through any of:

  • graylog161.fsslc.wtnet (if you have direct access)
  • test151.fsslc.wtnet – MediaWiki test host
  • A MediaWiki or MediaWiki task host like mw151.fsslc.wtnet
  • A bastion host like bast161.wikitide.net

SmartProxy Setup

Install SmartProxy:

Then configure:

  • Go to Proxy Server > Add server
  • Name: WikiTide Proxy
  • Address: 127.0.0.1
  • Port: 8089
  • Protocol: SOCKS5
  • Save

Next:

  • Go to Proxy Rules > Add rule
  • Rule type: Search Domain and SubDomain
  • Domain: logging.wikitide.net
  • Apply Proxy: WikiTide Proxy
  • Save and click Save again at the bottom (make sure you click it in both places)

See this video for a quick walkthrough.

OpenSSH

If you’re using OpenSSH, you can create a dynamic SOCKS5 proxy with:

ssh -D 8089 <server>.<dcname>.wtnet

Replace with the server hostname (e.g., test151) and with the datacenter identifier (e.g., fsslc).

If using a bastion setup as described on Tech:SSH#OpenSSH, you can simply run:

ssh -D 8089 wikitidebast

This avoids making two SSH hops.

PuTTY

To configure PuTTY:

  • Select a server to connect to
  • Navigate to Connection > SSH > Tunnels
  • Enter 8089 in the Source port field
  • Choose the Dynamic and Auto radio buttons
  • Save the session

If you plan to leave PuTTY open while idle, the session may time out. To avoid this, see: How to prevent PuTTY timeout when idle

Administration

Graylog configuration is a mix of Puppet and web interface setup. MongoDB on graylog161.fsslc.wtnet stores persistent configuration.

Relevant Puppet classes:

Categories

Go to Source →

Prev
Tech:Grafana
Next
Tech:Graylog161