Unable to connect to remote servers


#1

Sadly, I’m still (mostly) unable to be more than a ghost inworld. The behaviour is highly erratic, so it’s quite difficult to describe and practically impossible to replicate reliably for the purposes of a bug report. I will try categorise the different results, but I’m in a world of confusion!

Initial check

I’m pretty sure my ports are forwarded through my router correctly - but if anyone could verify, that would be great. Note that the TCP forwarding does seem to have an effect, especially when deactivated then reactivated (see 2nd situation below for more details).

First Situation

  1. ‘Ghost’ log on to a remote host. The result is I can see no voxels and no avis. Stats read Servers: 0 Avatars: 0. Interface log indicates no response from the STUN server:

    [2014-07-27T07:43:06] Sending intial stun request to 54.215.251.124
    [2014-07-27T07:43:07] QJsonObject({“status”: “success”})
    [2014-07-27T07:43:07] QJsonObject({“status”: “success”})
    [2014-07-27T07:43:07] Sending intial stun request to 54.215.251.124
    [2014-07-27T07:43:08] Sending intial stun request to 54.215.251.124
    [2014-07-27T07:43:08] Failed to lookup public address via STUN server at stun.highfidelity.io:3478. Using DS for STUN.

    When it’s like this, there are also constant log entries saying the nodelist has been cleared:

    [2014-07-27T08:43:22] Clearing the NodeList. Deleting all nodes in list.

Second Situation

  1. Sometimes Interlace connects with up to 6 Servers and voxels start to appear. This situation can often to kick in immediately after I make a change to the router settings (deactivate then reactivate port forwarding). If present, other avis appear. However, the AC connections will constantly drop and reconnect. I don’t know if it’s relevant, but the IP addresses listed don’t correspond with what I see on whatismyip.com.

    [2014-07-27T07:58:09] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
    [2014-07-27T07:58:09] Packet of type 5 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
    [2014-07-27T07:58:10] OAuth authorization code passed successfully to domain-server.
    [2014-07-27T07:58:10] NodeList UUID changed from “00000000-0000-0000-0000-000000000000” to “ae1c003f-ab5d-43c8-8bf1-f4278b76223a”
    [2014-07-27T07:58:10] Application title set to: Dave @ sandbox.highfidelity.io (build 917) - ?0.00000000
    [2014-07-27T07:58:11] NodeList UUID changed from “ae1c003f-ab5d-43c8-8bf1-f4278b76223a” to “3a135a9b-e85f-4450-a1ff-abc1205439e5”
    [2014-07-27T07:58:11] Added “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807
    [2014-07-27T07:58:11] Application title set to: Dave @ sandbox.highfidelity.io (build 917) - ?0.00000000
    [2014-07-27T07:58:11] Added “Particle Server” § {052168b7-9d81-4663-889b-f5d25074ef38} 54.241.217.33:43946 / 10.166.213.22:43946
    [2014-07-27T07:58:11] Added “Voxel Server” (V) {c2ff519b-ec39-4434-9249-ff37afc37cc7} 54.241.217.33:52471 / 10.166.213.22:52471
    [2014-07-27T07:58:11] Added “Model Server” (o) {3d236da9-212b-478b-ae93-374c8d2d6a30} 54.241.217.33:53630 / 10.166.213.22:53630
    [2014-07-27T07:58:11] VoxelSystem… voxel server {c2ff519b-ec39-4434-9249-ff37afc37cc7} added…
    [2014-07-27T07:58:11] Added “Avatar Mixer” (W) {c5f4f964-0448-4522-a491-a142a3034143} 54.241.217.33:46451 / 10.166.213.22:46451
    [2014-07-27T07:58:12] Activating public socket for node “Voxel Server” (V) {c2ff519b-ec39-4434-9249-ff37afc37cc7} 54.241.217.33:52471 / 10.166.213.22:52471
    [2014-07-27T07:58:12] Activating public socket for node “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807
    [2014-07-27T07:58:12] Activating public socket for node “Avatar Mixer” (W) {c5f4f964-0448-4522-a491-a142a3034143} 54.241.217.33:46451 / 10.166.213.22:46451
    [2014-07-27T07:58:12] Activating public socket for node “Particle Server” § {052168b7-9d81-4663-889b-f5d25074ef38} 54.241.217.33:43946 / 10.166.213.22:43946
    [2014-07-27T07:58:12] Activating public socket for node “Model Server” (o) {3d236da9-212b-478b-ae93-374c8d2d6a30} 54.241.217.33:53630 / 10.166.213.22:53630
    [2014-07-27T07:58:14] stats from new Model server… [0.000000, 0.000000, 0.000000, 1.000000]
    [2014-07-27T07:58:15] stats from new Particle server… [0.000000, 0.000000, 0.000000, 1.000000]
    [2014-07-27T07:58:16] stats from new Voxel server… [0.000000, 0.000000, 0.000000, 1.000000]
    [2014-07-27T07:58:18] Killed “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807 <---- Why?
    [2014-07-27T07:58:18] Packet of type 8 received from unknown node with UUID QUuid("{bc373ce1-5cb2-4498-a0e3-dd34f1949089}")

Note: There will often be many, many log messages describing packets from unknown nodes between reconnects:

[2014-07-27T07:53:38] Packet of type 8 received from unknown node with UUID QUuid("{bc373ce1-5cb2-4498-a0e3-dd34f1949089}")

or huge numbers of Packet hash mismatches:

[2014-07-27T08:47:41] Packet hash mismatch on 11 - Sender QUuid("{0faa0ff4-2c81-4bc5-8d4e-481fc452a785}")
[2014-07-27T08:47:41] Packet hash mismatch on 5 - Sender QUuid("{0faa0ff4-2c81-4bc5-8d4e-481fc452a785}")
[2014-07-27T08:47:41] Packet hash mismatch on 8 - Sender QUuid("{fccffd65-56ac-43f8-a91d-258a7c4dc947}")

Similar log, but with Thoys avi present - note the connect / drop / reconnect cycle. My IP address (from whitismyip.com) at the time was 1.10.220.52:

[2014-07-27T08:28:09] Packet of type 26 received from unknown node with UUID QUuid("{3d236da9-212b-478b-ae93-374c8d2d6a30}")
[2014-07-27T08:28:09] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:09] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:09] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:09] Packet of type 5 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:11] OAuth authorization code passed successfully to domain-server.
[2014-07-27T08:28:11] NodeList UUID changed from “00000000-0000-0000-0000-000000000000” to “0db9b872-bd13-46fa-800d-23ab8417644c”
[2014-07-27T08:28:11] Application title set to: Dave @ sandbox.highfidelity.io (build 917) - ?0.00000000
[2014-07-27T08:28:12] NodeList UUID changed from “0db9b872-bd13-46fa-800d-23ab8417644c” to “d4c9aa04-1569-4cf0-a528-943a65e94109”
[2014-07-27T08:28:12] Added “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807
[2014-07-27T08:28:12] Application title set to: Dave @ sandbox.highfidelity.io (build 917) - ?0.00000000
[2014-07-27T08:28:12] Added “Particle Server” § {052168b7-9d81-4663-889b-f5d25074ef38} 54.241.217.33:43946 / 10.166.213.22:43946
[2014-07-27T08:28:12] Added “Voxel Server” (V) {c2ff519b-ec39-4434-9249-ff37afc37cc7} 54.241.217.33:52471 / 10.166.213.22:52471
[2014-07-27T08:28:12] VoxelSystem… voxel server {c2ff519b-ec39-4434-9249-ff37afc37cc7} added…
[2014-07-27T08:28:12] Added “Model Server” (o) {3d236da9-212b-478b-ae93-374c8d2d6a30} 54.241.217.33:53630 / 10.166.213.22:53630
[2014-07-27T08:28:12] Added “Avatar Mixer” (W) {c5f4f964-0448-4522-a491-a142a3034143} 54.241.217.33:46451 / 10.166.213.22:46451
[2014-07-27T08:28:12] Activating public socket for node “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807
[2014-07-27T08:28:12] Activating public socket for node “Voxel Server” (V) {c2ff519b-ec39-4434-9249-ff37afc37cc7} 54.241.217.33:52471 / 10.166.213.22:52471
[2014-07-27T08:28:12] Activating public socket for node “Particle Server” § {052168b7-9d81-4663-889b-f5d25074ef38} 54.241.217.33:43946 / 10.166.213.22:43946
[2014-07-27T08:28:12] Activating public socket for node “Avatar Mixer” (W) {c5f4f964-0448-4522-a491-a142a3034143} 54.241.217.33:46451 / 10.166.213.22:46451
[2014-07-27T08:28:12] Activating public socket for node “Model Server” (o) {3d236da9-212b-478b-ae93-374c8d2d6a30} 54.241.217.33:53630 / 10.166.213.22:53630
[2014-07-27T08:28:15] stats from new Voxel server… [0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:15] stats from new Model server… [0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:15] stats from new Particle server… [0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:17] Clearing the NodeList. Deleting all nodes in list.
[2014-07-27T08:28:17] Killed “Particle Server” § {052168b7-9d81-4663-889b-f5d25074ef38} 54.241.217.33:43946 / 10.166.213.22:43946
[2014-07-27T08:28:17] Killed “Voxel Server” (V) {c2ff519b-ec39-4434-9249-ff37afc37cc7} 54.241.217.33:52471 / 10.166.213.22:52471
[2014-07-27T08:28:17] particle server going away… v[0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:17] Killed “Audio Mixer” (M) {bc373ce1-5cb2-4498-a0e3-dd34f1949089} 54.241.217.33:59807 / 10.166.213.22:59807
[2014-07-27T08:28:17] voxel server going away… v[0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:17] Killed “Avatar Mixer” (W) {c5f4f964-0448-4522-a491-a142a3034143} 54.241.217.33:46451 / 10.166.213.22:46451
[2014-07-27T08:28:17] Killed “Model Server” (o) {3d236da9-212b-478b-ae93-374c8d2d6a30} 54.241.217.33:53630 / 10.166.213.22:53630
[2014-07-27T08:28:17] NodeList UUID changed from “d4c9aa04-1569-4cf0-a528-943a65e94109” to “00000000-0000-0000-0000-000000000000”
[2014-07-27T08:28:17] VoxelSystem… voxel server {c2ff519b-ec39-4434-9249-ff37afc37cc7} removed…
[2014-07-27T08:28:17] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:17] model server going away… v[0.000000, 0.000000, 0.000000, 1.000000]
[2014-07-27T08:28:17] Packet of type 8 received from unknown node with UUID QUuid("{bc373ce1-5cb2-4498-a0e3-dd34f1949089}")
[2014-07-27T08:28:17] Application title set to: Dave @ sandbox.highfidelity.io (build 917) - ?0.00000000
[2014-07-27T08:28:17] Packet of type 8 received from unknown node with UUID QUuid("{bc373ce1-5cb2-4498-a0e3-dd34f1949089}")
[2014-07-27T08:28:17] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")
[2014-07-27T08:28:17] Packet of type 8 received from unknown node with UUID QUuid("{bc373ce1-5cb2-4498-a0e3-dd34f1949089}")
[2014-07-27T08:28:18] Packet of type 11 received from unknown node with UUID QUuid("{c5f4f964-0448-4522-a491-a142a3034143}")

Third Situation (localhost)

  1. If I run dev builds of DS and AC locally, I can connect with no problems at all with both the build dev build and the 917 build of Interface. Voxels appear, scripts work fine, etc, etc.

Remote server logs

Thoys was kind enough to send me some server logs from when I attempted to log in to his server. However, the results are inconclusive, as weirdly, I struggled to find log entries with corresponding timestamps, even in the AC and DS logs. This was also one of the less common occasions where the STUN server did return a new public socket to Interface, but the system would ‘Kill’ my agent two seconds after connection. My IP address, as supplied by the STUN server seems to vary. There are two different, but similar IP addresses in the log entries below, neither of which correspond with what I see on whatismyip.com.

First mention in AC log:

[DEBUG] [2014-07-26 13:34:05 +0200] [15238:15231] [avatar-mixer] Added “Agent” (I) {82289ca9-de12-46e2-b304-bf312e9f6c71} 1.10.220.3:56711 / 192.168.1.4:56711

but then, 2 seconds later in AC log:

[DEBUG] [2014-07-26 13:34:07 +0200] [15238:15231] [avatar-mixer] Killed “Agent” (I) {82289ca9-de12-46e2-b304-bf312e9f6c71} 1.10.220.3:56711 / 192.168.1.4:56711

First mention in DS log:

[DEBUG] [2014-07-26 13:34:24 +0200] [15540:14746] Added “Agent” (I) {3b70474f-d4f7-4b5e-b5f9-60103e318834} 1.10.220.3:56711 / 192.168.1.4:56711

First mention in Interface:

[2014-07-26T16:34:48] New public socket received from STUN server is 1.10.221.51:21745

STUN servers

I’ve been reading up a little on STUN servers and the like, it seems they’re not expected to work for all NAT types. Quoting http://en.wikipedia.org/wiki/STUN :

STUN does work with three types of NAT: full cone NAT, restricted cone NAT, and port restricted cone NAT. In the cases of restricted cone or port restricted cone NATs, the client must send out a packet to the endpoint before the NAT will allow packets from the endpoint through to the client.
STUN does not work with symmetric NAT (also known as bi-directional NAT) which is often found in the networks of large companies. TURN offers better results with symmetric NAT.

I am based in Thailand, so although my router setup is pretty standard, I have seen odd wide area network configurations mentioned in forums out here in the past - incompetence being the price of nepotism here!
A last resort (and a gain for those who are definitely behind symmetric NAT) might be to consider a TURN based solution instead / as well as the STUN server?

STUN server connection timing

To aid my own understanding, I also put together a timing diagram for the STUN server connection process:


Please let me know if I’ve not got this right!

System

Windows 7 Ulimate 64-bit SP1 Xeon X5650 12-core 2.67GHz nVidia GeForce GTX560M graphics 12GB RAM Interface tested using build 917 (and dev build for localhost test)

Thanks in advance for any help!

  • Dave

#2

@davedub,

I’ve worked on most of the hole-punching/NAT stuff to try and get nodes connected to each other in High Fidelity. Let’s see what we can figure out.

For starters, let’s see if you can consistently talk out of your network on the STUN port.

http://portquiz.net:3478/

Do you get the portquiz page when you go to that link?


#3

One thing I’ll add to look into on my list - when a connection to the STUN server fails, we should fall back to another STUN server on a different port and lastly attempt to use the domain-server as your STUN server.


#4

Hi b,

Thanks very much for your help with this.

portquiz.net consistently gives the following result:

Outgoing port tester
This server listens on all TCP ports, allowing you to test any outbound TCP port.
You have reached this page on port 3478.
Your network allows you to use this port. (Assuming that your network is not doing advanced traffic filtering.)
Network service: unknown
Your outgoing IP: 1.10.220.36

I also checked my IP on whatismyip.com - it is consistent: 1.10.220.36

With regards what Interface does on ‘no response’ from the STUN server, it seems to default to using the DS for STUN without trying any others, as each initial STUN request is to the same IP:

[2014-07-27T07:43:06] Sending intial stun request to 54.215.251.124

[2014-07-27T07:43:07] Sending intial stun request to 54.215.251.124
[2014-07-27T07:43:08] Sending intial stun request to 54.215.251.124
[2014-07-27T07:43:08] Failed to lookup public address via STUN server at stun.highfidelity.io:3478. Using DS for STUN.

Let me know if you need any other details. If there’s an apparent delay on my answers, please bear in mind I’m on UTC+7, so I’m usually working whilst you guys are tucked up in bed and visa versa :wink:

~ Dave


#5

Which version of Qt are you using?

I know we have a bug with 5.3.1 and the STUN packets. If you have 5.3.1 let me know, I’ll get a fix in today.


#6

I have version Qt5.2.0 installed in C:\Qt from when I compiled the code. Does the downloaded version of Interface use this location too, or do I need to look somewhere else as well?


#7

@b

I don’t know if there were any relevant changes to the STUN server, but although I am still unable to connect properly, the STUN server behaviour is now consistent. The STUN server now responds every time:

[2014-07-30T09:20:19] DS at sandbox.highfidelity.io is at 54.241.217.33
[2014-07-30T09:20:19] Sending intial stun request to 54.215.251.124

[2014-07-30T09:20:19] New public socket received from STUN server is 1.10.221.35:29265

This I verified as my IP correct IP address, as reported by portquiz.net. I also verified it using a local stack and a build from old code (about 10 days) - the STUN server comes back with my IP every time.

However, I am still unable to connect. The log shows the NodeList clearing every 6 seconds:

[2014-07-30T09:20:49] Clearing the NodeList. Deleting all nodes in list.
[2014-07-30T09:20:55] Clearing the NodeList. Deleting all nodes in list.

[2014-07-30T09:21:14] Clearing the NodeList. Deleting all nodes in list.
[2014-07-30T09:21:20] Clearing the NodeList. Deleting all nodes in list.

This is all completely consistent each time now. I have posted a full log here

Red Herring (I *think*)

I have also found a quirk of my network that has been affecting things - when I disable or re-enable the UDP port forwarding, I my ISP decides I should get a new IP address, every time! This seems to have the effect of connecting me briefly to the ACs, but the connections will not last long, and packets sent from the nodes are flagged as unknown when they start to arrive in Interface - below I can see the UUID for the audio mixer's packets is not being recognised. I also see that the destination IPs (i.e. local socket) for each added server / mixer does not match mine. [2014-07-30T09:46:16] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") [2014-07-30T09:46:16] Packet of type 11 received from unknown node with UUID QUuid("{330244f0-8cf9-43b8-b5e4-12d1cf2c85a3}") [2014-07-30T09:46:16] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") [2014-07-30T09:46:21] Clearing the NodeList. Deleting all nodes in list. [2014-07-30T09:46:28] Clearing the NodeList. Deleting all nodes in list. [2014-07-30T09:46:29] Displaying QWebView for OAuth authorization at "https://data.highfidelity.io/oauth/authorize?client_id=6e430d2c6dd845a2edf2a2cf9f9debff2517684c001aad5e87db252a7ad58a30&response_t ype=code&state=f01e2e6b-80c5-4902-836a-2e0d6a4ca750&redirect_uri=https://sandbox.highfidelity.io:40101/oauth" [2014-07-30T09:46:32] OAuth authorization code passed successfully to domain-server. [2014-07-30T09:46:32] NodeList UUID changed from "00000000-0000-0000-0000-000000000000" to "f01e2e6b-80c5-4902-836a-2e0d6a4ca750" [2014-07-30T09:46:32] Application title set to: Dave @ sandbox.highfidelity.io (build 926) - ?0.00000000 [2014-07-30T09:46:33] NodeList UUID changed from "f01e2e6b-80c5-4902-836a-2e0d6a4ca750" to "722aa746-bad5-450a-9e82-f9edfaada746" [2014-07-30T09:46:33] Application title set to: Dave @ sandbox.highfidelity.io (build 926) - ?0.00000000 [2014-07-30T09:46:33] Added "Avatar Mixer" (W) {330244f0-8cf9-43b8-b5e4-12d1cf2c85a3} 54.241.217.33:52852 / 10.166.213.22:52852 [2014-07-30T09:46:33] Added "Particle Server" (P) {57e86af1-7d75-45bb-9ce7-111a5034e673} 54.241.217.33:36904 / 10.166.213.22:36904 [2014-07-30T09:46:33] Added "Model Server" (o) {2ef0c596-cdac-4b61-9c17-cf867a76f727} 54.241.217.33:46174 / 10.166.213.22:46174 [2014-07-30T09:46:33] Added "Audio Mixer" (M) {bbbd648e-60b3-49da-8391-d5ec65cbe191} 54.241.217.33:50713 / 10.166.213.22:50713 [2014-07-30T09:46:33] Added "Voxel Server" (V) {dfccc3cf-a18b-4913-ba3f-1a82790b6436} 54.241.217.33:44197 / 10.166.213.22:44197 [2014-07-30T09:46:33] VoxelSystem... voxel server {dfccc3cf-a18b-4913-ba3f-1a82790b6436} added... [2014-07-30T09:46:34] Activating public socket for node "Audio Mixer" (M) {bbbd648e-60b3-49da-8391-d5ec65cbe191} 54.241.217.33:50713 / 10.166.213.22:50713 [2014-07-30T09:46:34] Activating public socket for node "Avatar Mixer" (W) {330244f0-8cf9-43b8-b5e4-12d1cf2c85a3} 54.241.217.33:52852 / 10.166.213.22:52852 [2014-07-30T09:46:34] Activating public socket for node "Particle Server" (P) {57e86af1-7d75-45bb-9ce7-111a5034e673} 54.241.217.33:36904 / 10.166.213.22:36904 [2014-07-30T09:46:34] Activating public socket for node "Model Server" (o) {2ef0c596-cdac-4b61-9c17-cf867a76f727} 54.241.217.33:46174 / 10.166.213.22:46174 [2014-07-30T09:46:34] Activating public socket for node "Voxel Server" (V) {dfccc3cf-a18b-4913-ba3f-1a82790b6436} 54.241.217.33:44197 / 10.166.213.22:44197 [2014-07-30T09:46:36] stats from new Particle server... [0.000000, 0.000000, 0.000000, 1.000000] [2014-07-30T09:46:36] stats from new Model server... [0.000000, 0.000000, 0.000000, 1.000000] [2014-07-30T09:46:36] stats from new Voxel server... [0.000000, 0.000000, 0.000000, 1.000000] [2014-07-30T09:46:43] Killed "Audio Mixer" (M) {bbbd648e-60b3-49da-8391-d5ec65cbe191} 54.241.217.33:50713 / 10.166.213.22:50713 [2014-07-30T09:46:43] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") [2014-07-30T09:46:43] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") [2014-07-30T09:46:43] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") [2014-07-30T09:46:43] Packet of type 8 received from unknown node with UUID QUuid("{bbbd648e-60b3-49da-8391-d5ec65cbe191}") But this behaviour (temp connection on IP address change) is probably a red herring - I guess the situation to focus on is the initial connection failing.

Nodelist::reset

The initial connection is now always the same, with the constant 'Clearing the NodeList. Deleting all nodes in list.' messages appearing afterwards, as per the log I linked to above. I set a breakpoiont in the code (old code, not updated for over a week), and found the debug message comes from LimitedNodeList::eraseAllNodes()

Callstack:

  • LimitedNodeList::reset()
  • NodeList::reset()

Before NodeList::reset(), there’s a Qt template function that I don’t know how to trace (template<class Obj, typename Ret> struct FunctionPointer…Ret (Obj::*)). Having not used template function before, I’m a bit stuck as to how to trace the source of the reset call back any further…

I hope this helps, I’ve been an alpha for 12 days now. It would be great to see the world, meet some people and get scripting!

  • Dave

#8

Apologies if I’m being dense, but why are you showing us the IP filter screen of your router? When I look at your attached screenshot, it looks like you are trying to set up a block on access between your router and an amazon web services IP address (based on reverse IP lookup). It isn’t sandbox, but maybe one of the other critical hifi services?

I’m not familiar with your router model, but if you are trying to set up port forwarding (shouldn’t be required to run the viewer though), you should be doing something in the NAT section of your router’s setup.

… I took a quick look at your router’s manual. I’d recommend removing both of those filter rules and see what happens. When you want to try running a server, I think you’ll do it via “Advanced Setup” then “NAT” and then “Virtual Server”.


#9

Hi mthome,

Thank you very much for taking the time to reply.

There were a couple of pages on the router config that looked like good candidates for opening up ports when I looked. I did find the NAT Virtual Server pages, but the settings needed specifying one of FTP, SSH, Telnet, etc. None seemed to fit the bill - I thought I needed some sort of UDP based setting that just wasn’t on the list - see open dropdown in image below.

Then I found the Filter page that did allow me to set a source and destination for IP / port for UDP and had a ‘Forward’ setting, so I went ahead and set the source IP as the STUN server that HiFi uses and tried to Forward to my local IP - very likely in error!

So, I’ve removed the filter settings. I’d tried to connect without any router settings in the past and tried again just now - Servers: 0, Avatars: 0

[2014-07-30T22:39:45] Clearing the NodeList. Deleting all nodes in list.
[2014-07-30T22:39:51] Clearing the NodeList. Deleting all nodes in list.
[2014-07-30T22:39:57] Clearing the NodeList. Deleting all nodes in list.
etc…

I then tried to connect with the virtual server settings as below - same result as usual. Servers: 0, Avatars: 0

[2014-07-30T22:41:09] Clearing the NodeList. Deleting all nodes in list.
[2014-07-30T22:41:10] Clearing the NodeList. Deleting all nodes in list.
etc…

It’s so frustrating!!!

Anyways, thanks again for help,

  • Dave

#10

It is possible that your ISP is blocking something. Perhaps someone from HighFidelity could list the outgoing ports used and you could try using ping and traceroute.


#11

@davedub,

@mthome is correct in that in general you shouldn’t need to open up any ports on your side for the Interface client to connect to a domain.

Taking a look at your logs in conjunction with Thoys’ domain so I can see if anything can be gleaned from there.


#12

It does seem like something would be going on at the level of your ISP that is causing a change in your perceived IP/port combination at the level of the domain-server or various assignment-clients.


#13

It would be interesting to have you test on a different connection if possible, or through a VPN. Another interesting test would be to run a scripted assignment-client against somebody’s domain to see if it has any better luck connecting.


#14

@mthome - Yep, it looks like my ISP is blocking something, or at least doing something weird with my IP address.

@b Thank you for your continued help with this, even though it looks like the fault is with my ISP, not HiFi - I very much appreciate it. To business:

Port forwarding - I have removed all settings from my router now, no forwarded ports, no filters.

I’ve bought myself a VPN and have run comparison tests. I also went to a local Internet Cafe that uses a different ISP and ran tests, both with and without VPN.

~ Build 940
~ The connection is unstable in all situations to varying degrees.
~ Chat doesn’t work in any of the situations (Chat window reads ‘You must be logged in to chat with others’).
~ All tests at sandbox.highfidelity.io, apart from one log that starts on localhost (no stack running).
~ I see from other’s screenshots that there are usually 6 connected ACs. I never get more than 5.

Results:

  1. No VPN (ISP: ToT - at home):
    No Voxels, Servers: 0
    Never connects to ACs.
    Actual IP address: 1.10.221.81
    STUN returned IP addresss: 1.10.221.81
    ACs connect to addresses: none ever returned
    Logs here.

  2. Using VPN (ISP: ToT - at home):
    Voxels visible, Servers: variable - from 0 to 5
    Connects to ACs, but kills audio mixer shortly afterwards.
    Actual IP address: 23.254.107.6
    STUN returned IP addresss: 23.254.107.6
    ACs connect to addresses: 10.166.213.22
    Logs here.

  3. No VPN (ISP: 3BB - at Internet Cafe):
    Voxels visible, Servers: 4 to 5 variable
    Lots of ‘Clearing the NodeList. Deleting all nodes in list.’ messages in first log. Keeps starting then killing Audio mixer AC in second.
    Actual IP address: ??? and 180.183.79.110
    STUN returned IP addresss: 180.183.79.110 and 180.183.79.110
    ACs connect to addresses: 10.166.213.22 and 10.166.213.22
    Logs here and here.

  4. Using VPN (ISP: - 3BB at Internet Cafe):
    Voxels visible, Servers: 4 to 5 variable
    Connection keeps dropping off and re-establishing (All ACs connecting then dropping, not just audio mixer).
    Actual IP address (VPN): 23.254.107.184 and 23.254.107.173
    STUN returned IP addresss: 23.254.107.184 and 23.254.107.173
    ACs connect to address: 10.166.213.22 and 10.166.213.22
    Logs here and here.

In 3 out of 4 situations, what’s very clear from the logs is that the ACs always think my address is 10.166.213.22, regardless of ISP, VPN or what the STUN server tells them. I hope that is due to a bug in HiFi, otherwise it’s slightly creepy!
Tomorrow night I have to go to Malaysia on business, so I’ll have the opportunity to test from a different country. Lucky timing I guess.

Running a scripted assignment client sounds interesting, how to proceed?

  • Dave

#15

double post removed - earlier post now showing


#17

Hey @davedub - where are you pulling that “ACs connect to address” from?


#18

I seem to remember reading in the code that the second IP address on these lines was the client IP:

[2014-07-31T12:18:00] Activating public socket for node “Particle Server” § {d899d984-7dd3-4fd8-90e7-74fe8258c7f8} 54.241.217.33:59584 / 10.166.213.22:59584

But as usual, could be very wrong!


#19

What you’re looking at there is actually the address for the particle-server, so it makes sense if that is not changing across your connections.

On connections where you are getting servers and losing them, that’s likely a packet rate issue - the connection is too slow for you to keep an active connection to all servers at once.

It would be helpful to get a speedtest from those connections if possible.

Something you can do to check that would be to see if turning off Voxels helps stabilize your connection. You can do that via the menu system. I believe it is under the Developer render options.

Not that you want to experience High Fidelity without voxels, but at least that would help us see if less bandwidth helps you keep the servers.


#20

Have you tried plugging your pc direct in your modem instead of your router ?

To rule out that your router is somehow bugging you …


#21

@MarcelEdward - thanks for the suggestion, but my modem / router is all in one. I have a cabled into it rather than use WiFi.

@b

I looked at the code again - I see the second IP is for the particle server’s local socket, not my client’s local socket - doh!

The packet rate issue sounds like a likely candidate, although my issues are far worse on the faster (i.e. no vpn) connection.
I briefly tried disabling voxels late last night, it seemed to make no difference.
If it is a slow connection issue (please forgive any naivety here) do you think it is worth me compiling up versions with increased AC timeouts (currently at 6 second I believe), just to see if it makes any difference?

I’ll be in Malaysia later tomorrow, the Internet connection’s usually better from there. Will run more tests…

  • Dave