Understanding Systemd
Systemd is the init system and service manager for most modern Linux distributions. It uses "units" to manage system resources:
- Services — Execute and manage programs/daemons
- Sockets — Listen on ports and start services on-demand
- Targets — Group units together (similar to runlevels)
- Timers — Schedule tasks (like cron, but with systemd)
Basic systemctl Commands
# Start a service
sudo systemctl start nginx
# Stop a service
sudo systemctl stop nginx
# Restart a service
sudo systemctl restart nginx
# Reload configuration (without restart)
sudo systemctl reload nginx
# Check service status
systemctl status nginx
# Enable service to start on boot
sudo systemctl enable nginx
# Disable service from starting on boot
sudo systemctl disable nginx
# Check if service is enabled
systemctl is-enabled nginx
# Check if service is active (running)
systemctl is-active nginx
Targets and Runlevels
# List available runlevel targets
ls -al /lib/systemd/system/runlevel*
# Get current default target (runlevel)
systemctl get-default
# Set default target
sudo systemctl set-default multi-user.target # No GUI
sudo systemctl set-default graphical.target # With GUI
# Switch target immediately
sudo systemctl isolate multi-user.target
| Target | Description | Old Runlevel |
|---|---|---|
| poweroff.target | System shutdown | 0 |
| rescue.target | Single-user mode | 1 |
| multi-user.target | Multi-user, no GUI | 3 |
| graphical.target | Multi-user with GUI | 5 |
| reboot.target | System reboot | 6 |
Creating a Custom Service
Step 1: Create the Service File
# Service files go in /lib/systemd/system/ (or /etc/systemd/system/)
sudo nano /lib/systemd/system/myapp.service
Step 2: Write the Service Configuration
[Unit]
Description=My Custom Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target
[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Step 3: Reload and Test
# Reload systemd to recognize new service
sudo systemctl daemon-reload
# Start the service
sudo systemctl start myapp
# Check status and logs
sudo systemctl status myapp
sudo journalctl -xeu myapp
Step 4: Enable on Boot
# Enable the service (creates symlink in multi-user.target.wants/)
sudo systemctl enable myapp
# Verify it's enabled
systemctl is-enabled myapp
Service Types Explained
# Most common types:
Type=simple # Default. Process started by ExecStart is the main process
Type=forking # Service forks and parent exits (traditional daemons)
Type=oneshot # Process exits after doing its work (scripts)
Type=notify # Like simple, but service sends notification when ready
Type=idle # Similar to simple, but waits until all jobs finish
Example: Forking Service
[Service]
Type=forking
PIDFile=/run/myapp/myapp.pid
ExecStart=/opt/myapp/bin/myapp --daemon
ExecStop=/bin/kill -TERM $MAINPID
Example: Oneshot Service (Script)
[Unit]
Description=Initialize Application Data
[Service]
Type=oneshot
ExecStart=/opt/myapp/init-data.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Environment Variables
[Service]
# Single variable
Environment="NODE_ENV=production"
# Multiple variables
Environment="NODE_ENV=production" "PORT=3000" "DB_HOST=localhost"
# From a file
EnvironmentFile=/etc/myapp/environment
EnvironmentFile=-/etc/myapp/local.env # '-' means optional
Example environment file (/etc/myapp/environment):
NODE_ENV=production
PORT=3000
DATABASE_URL=postgres://localhost/myapp
SECRET_KEY=your-secret-key
Restart Policies
[Service]
# Restart options
Restart=no # Don't restart (default)
Restart=on-success # Restart only if exited cleanly
Restart=on-failure # Restart on non-zero exit, signal, timeout
Restart=on-abnormal # Restart on signal, timeout
Restart=on-abort # Restart on signal only
Restart=on-watchdog # Restart on watchdog timeout
Restart=always # Always restart
# Delay before restart
RestartSec=5
# Maximum restart attempts (10 restarts within 60 seconds)
StartLimitBurst=10
StartLimitIntervalSec=60
Service Dependencies
[Unit]
# Start after these units
After=network.target postgresql.service
# Require these units (fail if they fail)
Requires=postgresql.service
# Want these units (don't fail if they fail)
Wants=redis.service
# Don't start if these are starting
Before=nginx.service
# Conflict with these units
Conflicts=other-webserver.service
Viewing Logs with journalctl
# View logs for a specific service
journalctl -u myapp
# Follow logs in real-time (like tail -f)
journalctl -fu myapp
# View logs since last boot
journalctl -u myapp -b
# View logs from specific time
journalctl -u myapp --since "2025-01-10 10:00:00"
journalctl -u myapp --since "1 hour ago"
# Show only errors
journalctl -u myapp -p err
# Extended output with explanations
journalctl -xeu myapp
# Show kernel messages
journalctl -k
# Disk usage of journal
journalctl --disk-usage
# Clean old logs
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=500M
Complete Example: Node.js Application
Here's a production-ready service file for a Node.js application:
# /lib/systemd/system/nodeapp.service
[Unit]
Description=Node.js Application
Documentation=https://github.com/yourcompany/nodeapp
After=network.target
[Service]
Type=simple
User=node
Group=node
WorkingDirectory=/var/www/nodeapp
Environment=NODE_ENV=production
Environment=PORT=3000
ExecStart=/usr/bin/node /var/www/nodeapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=nodeapp
# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/www/nodeapp/uploads
[Install]
WantedBy=multi-user.target
# Deploy the service
sudo systemctl daemon-reload
sudo systemctl enable --now nodeapp
sudo systemctl status nodeapp
Troubleshooting
# Analyze boot time
systemd-analyze
# Check which services take longest to start
systemd-analyze blame
# Verify service file syntax
systemd-analyze verify /lib/systemd/system/myapp.service
# Show service dependencies
systemctl list-dependencies myapp
# List all failed units
systemctl --failed
# Reset failed state
sudo systemctl reset-failed myapp
| Issue | Solution |
|---|---|
| Service won't start | Check journalctl -xeu servicename |
| Permission denied | Verify User/Group in service file |
| Changes not applied | Run systemctl daemon-reload |
| Service starts but dies | Check RestartSec and exit codes |
| Not starting on boot | Ensure systemctl enable was run |
Resources
Need help with Linux server configuration? Contact us for consulting.