Server Crashes - cannot open json


#1

Hi there!
Our Ubuntu server crashes about once a week with this error below. We are on build 8000.
Any ideas?

[05/20 06:25:02] [CRITICAL] [default] Cannot open json file for reading: “/.local/share/High Fidelity - dev/domain-server/entities/models.json.gz.replace”
[05/20 06:25:02] [WARNING] [hifi.domain_server] Replacement file could not be read, it either doesn’t exist or is invalid.
[05/20 06:25:02] [DEBUG] [hifi.domain_server] BACKUP RULES:
[05/20 06:25:02] [DEBUG] [hifi.domain_server] Name: “Half Hourly Rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] format: “half_hourly_rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] interval: 1800
[05/20 06:25:02] [DEBUG] [hifi.domain_server] count: 5
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-half_hourly_rolling-2018-05-20_05-41-55.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-half_hourly_rolling-2018-05-20_05-11-24.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-half_hourly_rolling-2018-05-20_06-11-55.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-half_hourly_rolling-2018-05-20_04-10-24.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-half_hourly_rolling-2018-05-20_04-40-54.zip”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] lastBackup: 13.116 mins ago
[05/20 06:25:02] [DEBUG] [hifi.domain_server] Name: “Daily Rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] format: “daily_rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] interval: 86400
[05/20 06:25:02] [DEBUG] [hifi.domain_server] count: 7
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-16_18-16-11.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-15_18-16-10.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-13_22-15-39.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-19_18-16-47.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-17_18-16-12.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-14_18-15-44.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-18_18-16-42.zip”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] lastBackup: 12.137 hrs ago
[05/20 06:25:02] [DEBUG] [hifi.domain_server] Name: “Weekly Rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] format: “weekly_rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] interval: 604800
[05/20 06:25:02] [DEBUG] [hifi.domain_server] count: 4
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-weekly_rolling-2018-05-16_22-00-13.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-weekly_rolling-2018-05-09_21-59-47.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-weekly_rolling-2018-05-02_21-59-17.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-weekly_rolling-2018-04-25_21-59-03.zip”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] lastBackup: 80.413 hrs ago
[05/20 06:25:02] [DEBUG] [hifi.domain_server] Name: “Thirty Day Rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] format: “thirty_day_rolling”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] interval: 2592000
[05/20 06:25:02] [DEBUG] [hifi.domain_server] count: 12
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-thirty_day_rolling-2018-05-13_21-57-38.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-thirty_day_rolling-2018-04-13_21-57-21.zip”
[05/20 06:25:02] [DEBUG] [default] Checking “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-thirty_day_rolling-2018-03-14_21-57-15.zip”
[05/20 06:25:02] [DEBUG] [hifi.domain_server] lastBackup: 152.456 hrs ago
[05/20 06:25:02] [INFO] [default] Found 4 ice-server IP addresses for “dev-ice.highfidelity.com
[05/20 06:25:02] [INFO] [default] Set candidate ice-server socket to 54.183.149.152:7337
[05/20 06:25:02] [DEBUG] [default] Updating ice-server address in High Fidelity Metaverse API to “54.183.149.152”
[05/20 06:25:02] [DEBUG] [hifi.networking] QHostInfo lookup result for “stun.highfidelity.io” with lookup ID 1 is “xx.xx.xx.xxx”
[05/20 06:25:02] [DEBUG] [hifi.networking] Sending intial stun request to stun.highfidelity.io
[05/20 06:25:02] [CRITICAL] [default] Could not open backup archive: “/.local/share/High Fidelity - dev/domain-server/backups/autobackup-daily_rolling-2018-05-13_22-15-39.zip”
[05/20 06:25:02] [CRITICAL] [default] ERROR: -1000
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [default] Could not open backup archive: "/.local/share/High Fidelity - dev/domain-server/backups/autobackup-thirty_day_rolling-2018-05-13_21-57-3$
[05/20 06:25:02] [CRITICAL] [default] ERROR: -1000
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [default] Could not open backup archive: “/.local/share/High Fidelity - dev/domain-server/backups/backup-Bak-2018-04-17_15-54-41.zip”
[05/20 06:25:02] [CRITICAL] [default] ERROR: -1000
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Failed to find “mappings.json” while loading backup
[05/20 06:25:02] [CRITICAL] [hifi.asset-backup] Error: 0
[05/20 06:25:02] [DEBUG] [hifi.asset-backup] Found 24 backup assets to delete from disk.
[05/20 06:25:02] [WARNING] [hifi.asset-backup] Some backups did not load properly, aborting delete operation for safety.
[05/20 06:25:02] [DEBUG] [hifi.networking] New public socket received from STUN server is xx.xxx.xxx.xxx:xxxxx
[05/20 06:25:03] [DEBUG] [default] ice-server address updated with metaverse.
[05/20 06:25:03] [DEBUG] [default] Received a request for assignment type 7 from 127.0.0.1:45986
[05/20 06:25:03] [DEBUG] [default] Deploying assignment - UUID: {xxxxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxxxx}, Type: 0 - to 127.0.0.1:45986
[05/20 06:25:03] [DEBUG] [default] Deploying assignment - UUID: {xxxxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxxxx}, Type: 1 - to 127.0.0.1:50650


#2

Hi @Spinteractive,
Unfortunately we are not gonna be able to figure out much from this log alone.
Do you happen to have any core dumps from those crashes?

- Clement