Done… no change. I’m really curious why the systime doesn’t track the gpstime. Is systime influenced by the directly connected GPS receiver, or is this a reflection of the asterisk server? Does the master influence asterisk/app_rpt which in turn influences the slave RTCM’s? I’m trying to understand the flow of signaling, this will signal where to look for the problem. (i.e. is it an RTCM or Asterisk/app_rpt code issue?) Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 10:03 PM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** I recommend restarting asterisk after the changes. On Tue, Jun 12, 2018 at 9:45 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, Is Systime coming from the RTCM or app_rpt? Any thoughts on what could be causing this anomaly (Systime not syncing with GPS time)? You reference servers, by that do you mean instances of app_rpt? I’ll change to the LAB source for master timing and see if anything changes. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 9:23 PM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work. I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime. I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server. On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, “Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 3:35 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge. I am also using a Raspberry Pi 3B (not plus) The master should be the first one to get a time packet to the server, or they should all be tied exactly. Chuck On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, A couple facts: The network has very low latency (sub ms) across all nodes on the VLAN. The site labeled LAB has a lab quality GPSDO, by NOVUS Power. The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage. Question: Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output. Here is an example of the output from my implementation: GPSTime (Lab): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.220000 DrainTime: 06/11/18 20:16:57.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71 GPSTime (Valparaiso): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5 GPSTime (Crown_Point): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6 GPSTime (LaPorte): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6 GPSTime (Lab): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.220000 DrainTime: 06/11/18 20:17:19.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38 GPSTime (Valparaiso): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.260000 DrainTime: 06/11/18 20:17:19.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7 GPSTime (Crown_Point): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5 GPSTime (LaPorte): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1 Results AFTER resetting the MASTER: GPSTime (Lab): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.220000 DrainTime: 06/11/18 20:31:23.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86 GPSTime (Valparaiso): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8 GPSTime (LaPorte): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7 GPSTime (Crown_Point): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5 Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Monday, June 4, 2018 4:51 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1 RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 I would not be concerned about the times. They are not important. A better thing to look at is in the Asterisk CLI issue the command voter debug level 3 Then expect output like the following: GPSTime (South): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (East): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (South): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 GPSTime (East): 06/04/18 04:07:19.000000000 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 SysTime: 06/04/18 04:07:19.000000 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 DrainTime: 06/04/18 04:07:18.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (North): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them. Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server). If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one. I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock. If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM. If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs). Chuck On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Hayden, When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so: 06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018 I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Thursday, May 31, 2018 5:56 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Is the time in UTC? How far off is the system time? Have you done a full calibration of each board? What baud rate do you run on your GPS's? I tend to use 9600 baud and have had no issues with the GPS's I have used. There is various GPS debug tools available in the RTCM. They are documented here - https://wiki.allstarlink.org/wiki/RTCM_Client#GPS Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency. I can't comment on that as I built all my boards and they have external frequency references. Regards Hayden VK7HH From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network. Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments. Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously? Regards Hayden VK7HH