# [SOUTH AFRICA] SciELO Nginx Log Export to FTP --- Operations Manual

## 1. Purpose

This manual documents the daily process used to export the SciELO Nginx
access log to an FTP server while keeping a local monthly archive.

The workflow:

1.  Nginx writes requests to `scielo-access.log`.
2.  `logrotate` rotates the active log daily.
3.  The export script reads the newly rotated, uncompressed log without
    modifying it.
4.  The script creates a gzip copy using the required filename
    convention.
5.  The gzip file is validated.
6.  The file is uploaded to the FTP server.
7.  Only after the FTP server confirms a successful transfer is the
    generated copy stored in the local monthly archive.

The original Nginx/logrotate files remain under logrotate control.

------------------------------------------------------------------------

## 2. File Naming Convention

The exported filename is:

``` text
YYYY-MM-DD_scielo.za.log.gz
```

Example:

``` text
2026-10-09_scielo.za.log.gz
```

The local archive is organized by year and month:

``` text
/var/log/nginx/scielo-org-za/YYYY-MM/
```

Example:

``` text
/var/log/nginx/scielo-org-za/2026-10/2026-10-09_scielo.za.log.gz
```

Using `YYYY-MM` instead of only the month number prevents files from
different years from being mixed.

------------------------------------------------------------------------

## 3. Existing Nginx Log Rotation

The server currently uses the following logrotate configuration:

``` text
/var/log/nginx/*.log {
    create 0640 nginx root
    daily
    rotate 10
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
    endscript
}
```

### Important behavior

Because `delaycompress` is enabled, the most recently rotated file
remains uncompressed for one rotation cycle.

For example:

``` text
scielo-access.log
scielo-access.log-20261009
scielo-access.log-20261008.gz
scielo-access.log-20261007.gz
```

The export process must **not** modify or remove
`scielo-access.log-20261009`. It creates an independent compressed copy.

------------------------------------------------------------------------

## 4. FTP Credentials

Store FTP credentials separately from the export script.

Create:

``` bash
vi /root/.scielo-ftp.conf
```

Contents:

``` bash
FTP_USER='your_username'
FTP_PASS='your_password'
```

Protect the file:

``` bash
chmod 600 /root/.scielo-ftp.conf
chown root:root /root/.scielo-ftp.conf
```

Verify:

``` bash
ls -l /root/.scielo-ftp.conf
```

Expected permissions:

``` text
-rw-------. 1 root root ... /root/.scielo-ftp.conf
```

Do not commit this file to Git or copy its contents into operational
documentation.

> **Security note:** traditional FTP does not encrypt credentials or
> transferred data. If the service later supports SFTP or FTPS,
> migration to an encrypted transport is recommended.

------------------------------------------------------------------------

## 5. Daily Export Script

Create:

``` bash
vi /usr/local/sbin/export-scielo-za.sh
```

Use the following script:

``` bash
#!/bin/bash

set -u
umask 027

LOGDIR="/var/log/nginx"
ARCHIVE="$LOGDIR/scielo-org-za"

HOST="ftp.ratchet.scielo.org"
CREDENTIALS="/root/.scielo-ftp.conf"

# Load FTP credentials
if [ ! -r "$CREDENTIALS" ]; then
    echo "ERROR: credentials file not found: $CREDENTIALS"
    exit 1
fi

source "$CREDENTIALS"

if [ -z "${FTP_USER:-}" ] || [ -z "${FTP_PASS:-}" ]; then
    echo "ERROR: FTP_USER or FTP_PASS is not defined."
    exit 1
fi

# The date suffix generated by logrotate
DATA=$(date +%Y%m%d)

SOURCE="$LOGDIR/scielo-access.log-$DATA"

if [ ! -s "$SOURCE" ]; then
    echo "ERROR: rotated log was not found or is empty:"
    echo "$SOURCE"
    exit 1
fi

YEAR="${DATA:0:4}"
MONTH="${DATA:4:2}"
DAY="${DATA:6:2}"

FORMATTED_DATE="${YEAR}-${MONTH}-${DAY}"

FILENAME="${FORMATTED_DATE}_scielo.za.log.gz"
DESTDIR="$ARCHIVE/${YEAR}-${MONTH}"
DEST="$DESTDIR/$FILENAME"
TMP="$DESTDIR/.${FILENAME}.tmp"

mkdir -p "$DESTDIR"

echo "=============================================="
echo "SciELO ZA daily export"
echo "=============================================="
echo "Source      : $SOURCE"
echo "Export file : $FILENAME"
echo "Destination : $DEST"
echo

# Prevent duplicate processing
if [ -f "$DEST" ]; then
    echo "File has already been processed:"
    echo "$DEST"
    exit 0
fi

# Create an independent compressed copy.
# The logrotate-managed source file is not changed.
echo "Compressing..."

gzip -c "$SOURCE" > "$TMP"

if [ $? -ne 0 ]; then
    echo "ERROR: compression failed."
    rm -f "$TMP"
    exit 1
fi

# Validate the gzip stream
if ! gzip -t "$TMP"; then
    echo "ERROR: invalid gzip file."
    rm -f "$TMP"
    exit 1
fi

echo "Compression successful:"
ls -lh "$TMP"

# FTP transfer
echo
echo "Uploading to FTP..."

FTP_LOG=$(mktemp)

ftp -n -v "$HOST" >"$FTP_LOG" 2>&1 <<EOF
quote USER $FTP_USER
quote PASS $FTP_PASS
binary
put $TMP $FILENAME
quit
EOF

cat "$FTP_LOG"

# Detect common FTP failures
if grep -Eqi \
    '530 |550 |425 |426 |not connected|failed|failure|error|refused|timed out|no such|not found' \
    "$FTP_LOG"; then

    echo "ERROR: FTP reported a failure."
    rm -f "$FTP_LOG"
    rm -f "$TMP"
    exit 1
fi

# Require FTP server completion confirmation
if ! grep -Eqi '^226 ' "$FTP_LOG"; then
    echo "ERROR: FTP server did not confirm successful transfer."
    rm -f "$FTP_LOG"
    rm -f "$TMP"
    exit 1
fi

rm -f "$FTP_LOG"

echo
echo "FTP transfer completed."

# Archive only after FTP success
mv -f -- "$TMP" "$DEST"

if [ $? -ne 0 ]; then
    echo "ERROR: FTP transfer succeeded, but local archival failed:"
    echo "$DEST"
    exit 1
fi

echo
echo "=============================================="
echo "PROCESS COMPLETED"
echo "=============================================="
echo "Archived file:"
echo "$DEST"

exit 0
```

Set permissions:

``` bash
chmod 750 /usr/local/sbin/export-scielo-za.sh
chown root:root /usr/local/sbin/export-scielo-za.sh
```

------------------------------------------------------------------------

## 6. Why the Script Uses `gzip -c`

The command:

``` bash
gzip -c "$SOURCE" > "$TMP"
```

reads the rotated log and writes a compressed copy.

It does **not** replace or delete:

``` text
/var/log/nginx/scielo-access.log-YYYYMMDD
```

This separation is intentional. The original file remains under
`logrotate` lifecycle management.

------------------------------------------------------------------------

## 7. FTP Transfer Method

The FTP implementation uses the same connection method already validated
in the environment:

``` bash
ftp -n -v "$HOST" <<EOF
quote USER $FTP_USER
quote PASS $FTP_PASS
binary
put "$LOCAL_FILE" "$REMOTE_FILE"
quit
EOF
```

`binary` is required for gzip files.

The process does not force passive mode because the existing FTP
workflow has already been validated without that change.

### Successful transfer

The FTP server should return a `226` response, for example:

``` text
226 Transfer complete.
```

The script requires a `226` response before considering the upload
successful.

### Common authentication failure

``` text
331 Please specify the password.
530 Login incorrect.
```

This indicates an authentication problem, not an upload or passive-mode
problem.

Check the configured username/password and the credentials file.

------------------------------------------------------------------------

## 8. Manual Test

Run:

``` bash
/usr/local/sbin/export-scielo-za.sh
```

For a rotation dated October 9, 2026, the expected local file is:

``` text
/var/log/nginx/scielo-org-za/2026-10/2026-10-09_scielo.za.log.gz
```

Verify it:

``` bash
ls -lh /var/log/nginx/scielo-org-za/2026-10/
```

Test gzip integrity:

``` bash
gzip -t /var/log/nginx/scielo-org-za/2026-10/2026-10-09_scielo.za.log.gz
echo $?
```

Expected:

``` text
0
```

Inspect the beginning of the archived log:

``` bash
gzip -dc /var/log/nginx/scielo-org-za/2026-10/2026-10-09_scielo.za.log.gz | head
```

------------------------------------------------------------------------

## 9. Daily Scheduling

The observed log rotation normally occurs around 03:00--04:00. Run the
export after the rotation has completed.

Create:

``` bash
vi /etc/cron.d/scielo-za-export
```

Contents:

``` cron
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin

0 5 * * * root /usr/local/sbin/export-scielo-za.sh >> /var/log/export-scielo-za.log 2>&1
```

Set permissions:

``` bash
chmod 644 /etc/cron.d/scielo-za-export
chown root:root /etc/cron.d/scielo-za-export
```

Verify:

``` bash
cat /etc/cron.d/scielo-za-export
```

Monitor execution:

``` bash
tail -100 /var/log/export-scielo-za.log
```

------------------------------------------------------------------------

## 10. Historical Log Migration

Previously compressed logrotate files may look like:

``` text
scielo-access.log-20260930.gz
scielo-access.log-20261001.gz
scielo-access.log-20261002.gz
```

These files are already compressed and should **not** be recompressed.

They can be uploaded with a new remote filename and then moved/renamed
locally.

Example mapping:

``` text
scielo-access.log-20260930.gz
    -> 2026-09-30_scielo.za.log.gz

scielo-access.log-20261001.gz
    -> 2026-10-01_scielo.za.log.gz
```

Local destinations:

``` text
/var/log/nginx/scielo-org-za/2026-09/2026-09-30_scielo.za.log.gz
/var/log/nginx/scielo-org-za/2026-10/2026-10-01_scielo.za.log.gz
```

### Historical migration script

``` bash
#!/bin/bash

source /root/.scielo-ftp.conf

HOST="ftp.ratchet.scielo.org"
ARCHIVE="/var/log/nginx/scielo-org-za"

for f in /var/log/nginx/scielo-access.log-????????.gz; do

    [ -f "$f" ] || continue

    base=$(basename "$f")

    data="${base#scielo-access.log-}"
    data="${data%.gz}"

    year="${data:0:4}"
    month="${data:4:2}"
    day="${data:6:2}"

    FILE_REMOTE="${year}-${month}-${day}_scielo.za.log.gz"
    DESTDIR="${ARCHIVE}/${year}-${month}"
    DEST="${DESTDIR}/${FILE_REMOTE}"

    echo
    echo "=========================================="
    echo "Source      : $f"
    echo "FTP filename: $FILE_REMOTE"
    echo "Destination : $DEST"
    echo "=========================================="

    if ! gzip -t "$f"; then
        echo "ERROR: invalid gzip file: $f"
        continue
    fi

    mkdir -p "$DESTDIR"

    FTP_LOG=$(mktemp)

    ftp -n -v "$HOST" >"$FTP_LOG" 2>&1 <<EOF
quote USER $FTP_USER
quote PASS $FTP_PASS
binary
put $f $FILE_REMOTE
quit
EOF

    cat "$FTP_LOG"

    if grep -Eqi \
        '530 |550 |425 |426 |not connected|failed|failure|error|refused|timed out|no such|not found' \
        "$FTP_LOG"; then

        echo "ERROR: FTP transfer failed: $FILE_REMOTE"
        rm -f "$FTP_LOG"
        continue
    fi

    if ! grep -Eqi '^226 ' "$FTP_LOG"; then
        echo "ERROR: FTP server did not confirm transfer: $FILE_REMOTE"
        rm -f "$FTP_LOG"
        continue
    fi

    rm -f "$FTP_LOG"

    mv -f -- "$f" "$DEST"

    if [ $? -ne 0 ]; then
        echo "ERROR: FTP succeeded, but local move failed:"
        echo "$f -> $DEST"
        continue
    fi

    echo "OK: $FILE_REMOTE"
done
```

The order is intentionally:

``` text
Validate gzip
      |
      v
Upload to FTP
      |
      v
Receive FTP 226
      |
      v
Move/Rename locally
```

If FTP fails, the original historical file remains in place for a later
retry.

------------------------------------------------------------------------

## 11. Troubleshooting

### Rotated source file does not exist

Example:

``` text
ERROR: rotated log was not found or is empty:
/var/log/nginx/scielo-access.log-20261009
```

Check:

``` bash
ls -lh /var/log/nginx/scielo-access.log*
```

Also check logrotate status and execution time.

------------------------------------------------------------------------

### FTP login fails

Typical response:

``` text
331 Please specify the password.
530 Login incorrect.
```

Verify that credentials are loaded:

``` bash
source /root/.scielo-ftp.conf

printf 'FTP_USER=[%s]\n' "$FTP_USER"
printf 'FTP_PASS length=%d\n' "${#FTP_PASS}"
```

Do not print the actual password.

Check for Windows CRLF characters:

``` bash
cat -A /root/.scielo-ftp.conf
```

If lines contain `^M`, normalize them:

``` bash
sed -i 's/\r$//' /root/.scielo-ftp.conf
```

------------------------------------------------------------------------

### FTP transfer starts but fails

Relevant FTP response codes include:

``` text
425 Cannot open data connection
426 Connection closed; transfer aborted
550 Requested action not taken
```

Check firewall rules, FTP data-channel behavior, available disk space
and server-side permissions.

------------------------------------------------------------------------

### Gzip validation fails

Test manually:

``` bash
gzip -t FILE.gz
echo $?
```

A successful validation returns:

``` text
0
```

Do not upload or archive a file that fails gzip validation.

------------------------------------------------------------------------

### File was already processed

The daily script checks:

``` text
/var/log/nginx/scielo-org-za/YYYY-MM/YYYY-MM-DD_scielo.za.log.gz
```

If the file already exists, the script exits successfully without
overwriting or retransmitting it.

------------------------------------------------------------------------

## 12. Operational Verification

Useful daily checks:

``` bash
tail -100 /var/log/export-scielo-za.log
```

``` bash
find /var/log/nginx/scielo-org-za -type f -name '*_scielo.za.log.gz' -printf '%TY-%Tm-%Td %TH:%TM %s %p\n' | sort
```

Check gzip integrity for a specific file:

``` bash
gzip -t /var/log/nginx/scielo-org-za/2026-10/2026-10-09_scielo.za.log.gz
```

Check the most recent archive:

``` bash
find /var/log/nginx/scielo-org-za -type f -name '*_scielo.za.log.gz' -printf '%T@ %p\n' \
    | sort -nr \
    | head
```

------------------------------------------------------------------------

## 13. Recovery Considerations

The current daily script uses:

``` bash
DATA=$(date +%Y%m%d)
```

This means it expects the logrotate output for the current date to exist
when the script runs.

If FTP is unavailable or the scheduled job fails, the next day's run
does not automatically retry the previous day's source file.

For a more resilient production implementation, a future version should:

-   discover all eligible `scielo-access.log-YYYYMMDD` and `.gz` files;
-   determine whether the corresponding `YYYY-MM-DD_scielo.za.log.gz`
    has already been archived;
-   process only missing exports;
-   safely retry failed FTP transfers;
-   keep a clear success/failure audit trail.

This removes the dependency on a single daily execution window.

------------------------------------------------------------------------

## 14. Security and Preservation Notes

The export archive and the original Nginx logs serve different
operational purposes.

The export process should not alter the original logrotate-managed
source. Keeping the source untouched provides a reference copy if an
export must be regenerated or investigated.

Recommended controls include:

-   keep FTP credentials readable only by root;
-   do not store credentials in source control;
-   validate gzip files before transfer;
-   require an FTP success response before considering the transfer
    complete;
-   retain execution logs;
-   monitor failed scheduled jobs;
-   periodically verify that local and remote expected files are
    present;
-   migrate from unencrypted FTP to SFTP or FTPS when supported.

------------------------------------------------------------------------

## 15. Key Paths

  --------------------------------------------------------------------------------------
  Purpose                             Path
  ----------------------------------- --------------------------------------------------
  Active Nginx access log             `/var/log/nginx/scielo-access.log`

  Rotated Nginx logs                  `/var/log/nginx/scielo-access.log-YYYYMMDD[.gz]`

  Export archive                      `/var/log/nginx/scielo-org-za/YYYY-MM/`

  Daily export script                 `/usr/local/sbin/export-scielo-za.sh`

  FTP credentials                     `/root/.scielo-ftp.conf`

  Cron configuration                  `/etc/cron.d/scielo-za-export`

  Export execution log                `/var/log/export-scielo-za.log`

  FTP host                            `ftp.ratchet.scielo.org`
  --------------------------------------------------------------------------------------

------------------------------------------------------------------------

## 16. Expected Final State

After successful daily processing:

``` text
/var/log/nginx/
├── scielo-access.log
├── scielo-access.log-20261009
├── scielo-access.log-20261008.gz
│
└── scielo-org-za/
    └── 2026-10/
        ├── 2026-10-01_scielo.za.log.gz
        ├── 2026-10-02_scielo.za.log.gz
        ├── ...
        └── 2026-10-09_scielo.za.log.gz
```

The FTP server should contain the corresponding exported filenames:

``` text
2026-10-01_scielo.za.log.gz
2026-10-02_scielo.za.log.gz
...
2026-10-09_scielo.za.log.gz
```

------------------------------------------------------------------------

**Document purpose:** operational runbook for the SciELO Nginx
access-log FTP export workflow.