Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Chuck, I setup the NTP client and enabled the service, as well as moved the MASTER to LAB. This is the output from VOTER DEBUG LEVEL 3: GPSTime (Lab): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.200000 DrainTime: 06/15/18 15:02:39.160 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90 GPSTime (Valparaiso): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.260000 DrainTime: 06/15/18 15:02:39.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2 GPSTime (LaPorte): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2 GPSTime (Crown_Point): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 In the Asterisk console, I restarted Asterisk by asserting the “reboot now” command and this is the output: GPSTime (Valparaiso): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 299.6 GPSTime (LaPorte): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 328.1 GPSTime (Crown_Point): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.980 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.3 GPSTime (Lab): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.220000 DrainTime: 06/15/18 15:07:04.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.11 Waiting an hour and testing again yields: SysTime: 06/15/18 16:23:30.200000 DrainTime: 06/15/18 16:23:30.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.64 GPSTime (Valparaiso): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 298.1 GPSTime (Crown_Point): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 220.9 GPSTime (LaPorte): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 326.6 It would appear the NTP activity is slowly (expected) netting the SysTime, however, the LAB (master) is still off. If the SysTime comes from Asterisk, then what gives? An unanswered question: What is DrainTime describing, and what drives the displayed value? I notice on yours, everything matches. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: Chuck Henderson <rpt2@chuck.midlandsnetworking.com> Sent: Wednesday, June 13, 2018 2:34 AM To: kevin.babich@gmail.com Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** That is complicated and I don't remember. All the needed software is installed already with the allstarlink raspberry pi dial install that I used. I think I used 07-08-2017RAT_RC1.img but I can't remember for sure. I also can't remember exactly what I did to set up the time sync but I looked for clues something like man systemd-timesyncd timedatectl I had used ntpd in the past with x86 ACID but didn't need it for the DIAL version I am using now because this other method of getting the time was builtin and all I had to do was a google search and I landed on the exact files to edit to make it use my own dedicated ntp server which I have had running for many years. If it ever dies I will have to re-google how to rebuild it using current hardware. I quit trying to remember things that I can just ask "ok google". On Wed, Jun 13, 2018 at 1:22 AM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, How does one go about syncing the server in app_rpt? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 11:50 PM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Yes, many GPS will sometimes be off by one second in the RTCM. If I have to power cycle any of my voters/RTCMs I usually then do a reboot via the RTCM menu so that the GPS is already up and accurate when the RTCM boots and this seems to eliminate the one second off problem, most of the time. Both Jim and I tried to fix this and neither of us could. When all set up to troubleshoot it the problem never happens. The software has a couple of code segments to try to make it work even when exactly one second off but I don't think that works all the time either. Also in looking at the code I see that SysTime does not matter at all. It is just a reference to look at in the debug output. Mine exactly matches because I also sync my Linux time to a GPS reference separately from the app_rpt voter code. But I tried disabling that and setting the date and time both wrong and the voter still works because it keeps it's own time that is referenced to the master and not dependent on the system time. So the master time does need to be the first time that the voter channel driver receives each second. Other RTCMs time must arrive later than the master arrives but before the buffer length later. The RTCM-GPS time can be any amount wrong as long as all RTCM-GPSs are exactly the same as each other. Start with much bigger buffers than you need and gradually reduce them. On Tue, Jun 12, 2018 at 10:03 PM Tim Sawyer <tisawyer@gmail.com <mailto:tisawyer@gmail.com> > wrote: This may be a bit off topic but I've had problems with the BG7TBL units. One won't get a RTCM GPS lock with initial power on after it's been off for a while... like a day or more. The GPS Lock LED on the GPS comes on but doesn't on the RTCM. Subsequent immediate power cycle cures that. Another I've seen shifts it's time by one second after it's been running for some period. On the other hand I have one that works flawlessly. So I'm a little concerned about these. I don't know if there some incompatibility with the RTCM or if the BG7TLB units have inconsistent performance. Any thoughts on that possibly being repeated to the issue Kevin is having? On Tue, Jun 12, 2018 at 7:24 PM Chuck Henderson <rpt2@chuck.midlandsnetworking.com <mailto:rpt2@chuck.midlandsnetworking.com> > wrote: The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work. I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime. I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server. On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, “Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 3:35 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge. I am also using a Raspberry Pi 3B (not plus) The master should be the first one to get a time packet to the server, or they should all be tied exactly. Chuck On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, A couple facts: The network has very low latency (sub ms) across all nodes on the VLAN. The site labeled LAB has a lab quality GPSDO, by NOVUS Power. The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage. Question: Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output. Here is an example of the output from my implementation: GPSTime (Lab): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.220000 DrainTime: 06/11/18 20:16:57.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71 GPSTime (Valparaiso): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5 GPSTime (Crown_Point): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6 GPSTime (LaPorte): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6 GPSTime (Lab): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.220000 DrainTime: 06/11/18 20:17:19.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38 GPSTime (Valparaiso): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.260000 DrainTime: 06/11/18 20:17:19.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7 GPSTime (Crown_Point): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5 GPSTime (LaPorte): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1 Results AFTER resetting the MASTER: GPSTime (Lab): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.220000 DrainTime: 06/11/18 20:31:23.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86 GPSTime (Valparaiso): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8 GPSTime (LaPorte): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7 GPSTime (Crown_Point): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5 Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Monday, June 4, 2018 4:51 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1 RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 I would not be concerned about the times. They are not important. A better thing to look at is in the Asterisk CLI issue the command voter debug level 3 Then expect output like the following: GPSTime (South): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (East): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (South): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 GPSTime (East): 06/04/18 04:07:19.000000000 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 SysTime: 06/04/18 04:07:19.000000 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 DrainTime: 06/04/18 04:07:18.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (North): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them. Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server). If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one. I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock. If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM. If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs). Chuck On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Hayden, When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so: 06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018 I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Thursday, May 31, 2018 5:56 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Is the time in UTC? How far off is the system time? Have you done a full calibration of each board? What baud rate do you run on your GPS's? I tend to use 9600 baud and have had no issues with the GPS's I have used. There is various GPS debug tools available in the RTCM. They are documented here - https://wiki.allstarlink.org/wiki/RTCM_Client#GPS Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency. I can't comment on that as I built all my boards and they have external frequency references. Regards Hayden VK7HH From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network. Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments. Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously? Regards Hayden VK7HH
Kevin, Here is what I see in your data. If your network latency is very low as you say, then it looks like your GPS's don't all have the same time. LaPorte and Crown_Point must be exactly the same as each other and Valparaiso is probably within a millisecond, but lab is way off by about 60 milliseconds (could be somewhere between 41 and 79ms), this is a problem and couldn't be network latency or you would see that in a ping test. And all are about 200 to 280 milliseconds different than the NTP time reference that you are using which is not a problem but just effects the debug display. Drain time should be the time that audio is being removed from the buffer and being sent to the transmitter which is sometime in the past as configured by your buffers in the voter.conf. All change in 20ms jumps so a change of 20ms could just be a change of a fraction of a microsecond that incremented from one 20ms buffer slot to the next one. I would really suspect pps polarity in the RTCM config of either "lab" is wrong, or pps polarity in all 3 of the other sites is wrong to account for that kind of discrepancy. Wrong PPS polarity should not keep it from working. Change the pps polarity and wait a minute or two for it to resync and it should start working again. If it doesn't then restart that RTCM after 99 saving the new pps setting. Most PPS pulses I have looked at are between 1ms and 100 ms long. if your pps pulse is 50ms long then it would cause the time to be off by 50ms if you have it set wrong. If the pps pulse is 500ms long then it probably would not work at all if it is set to the wrong polarity because the time would be half a second off and I am sure you don't have that much buffer space. Do you know your pps pulse width? Is it the same in all of your GPS's? When I say pps polarity I am only talking about RTCM menu item 10 being 0 or 1. I have never used the GPS's that you are using so I don't know anything about how they behave. My experience is with the Garmin and with 3 other brands that act exactly the same as the Garmins. They all have configurable PPS pulse width. On Fri, Jun 15, 2018 at 4:41 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
I setup the NTP client and enabled the service, as well as moved the MASTER to LAB. This is the output from VOTER DEBUG LEVEL 3:
GPSTime (Lab): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.200000
DrainTime: 06/15/18 15:02:39.160
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90
GPSTime (Valparaiso): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.260000
DrainTime: 06/15/18 15:02:39.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2
GPSTime (LaPorte): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.280000
DrainTime: 06/15/18 15:02:39.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2
GPSTime (Crown_Point): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.280000
DrainTime: 06/15/18 15:02:39.240
In the Asterisk console, I restarted Asterisk by asserting the “reboot now” command and this is the output:
GPSTime (Valparaiso): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.960
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 299.6
GPSTime (LaPorte): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.960
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 328.1
GPSTime (Crown_Point): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.980
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.3
GPSTime (Lab): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.220000
DrainTime: 06/15/18 15:07:04.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.11
Waiting an hour and testing again yields:
SysTime: 06/15/18 16:23:30.200000
DrainTime: 06/15/18 16:23:30.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.64
GPSTime (Valparaiso): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 298.1
GPSTime (Crown_Point): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 220.9
GPSTime (LaPorte): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 326.6
It would appear the NTP activity is slowly (expected) netting the SysTime, however, the LAB (master) is still off. If the SysTime comes from Asterisk, then what gives?
An unanswered question: What is DrainTime describing, and what drives the displayed value? I notice on yours, everything matches.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* Chuck Henderson <rpt2@chuck.midlandsnetworking.com> *Sent:* Wednesday, June 13, 2018 2:34 AM *To:* kevin.babich@gmail.com *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
That is complicated and I don't remember. All the needed software is installed already with the allstarlink raspberry pi dial install that I used. I
think I used 07-08-2017RAT_RC1.img but I can't remember for sure. I also can't remember exactly what I did to set up the time sync but I looked for clues something like
man systemd-timesyncd
timedatectl
I had used ntpd in the past with x86 ACID but didn't need it for the DIAL version I am using now because this other method of getting the time was builtin and all I had to do was a google search and I landed on the exact files to edit to make it use my own dedicated ntp server which I have had running for many years. If it ever dies I will have to re-google how to rebuild it using current hardware. I quit trying to remember things that I can just ask "ok google".
On Wed, Jun 13, 2018 at 1:22 AM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
How does one go about syncing the server in app_rpt?
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Tuesday, June 12, 2018 11:50 PM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Yes, many GPS will sometimes be off by one second in the RTCM. If I have to power cycle any of my voters/RTCMs I usually then do a reboot via the RTCM menu so that the GPS is already up and accurate when the RTCM boots and this seems to eliminate the one second off problem, most of the time. Both Jim and I tried to fix this and neither of us could. When all set up to troubleshoot it the problem never happens. The software has a couple of code segments to try to make it work even when exactly one second off but I don't think that works all the time either.
Also in looking at the code I see that SysTime does not matter at all. It is just a reference to look at in the debug output. Mine exactly matches because I also sync my Linux time to a GPS reference separately from the app_rpt voter code. But I tried disabling that and setting the date and time both wrong and the voter still works because it keeps it's own time that is referenced to the master and not dependent on the system time.
So the master time does need to be the first time that the voter channel driver receives each second. Other RTCMs time must arrive later than the master arrives but before the buffer length later.
The RTCM-GPS time can be any amount wrong as long as all RTCM-GPSs are exactly the same as each other.
Start with much bigger buffers than you need and gradually reduce them.
On Tue, Jun 12, 2018 at 10:03 PM Tim Sawyer <tisawyer@gmail.com> wrote:
This may be a bit off topic but I've had problems with the BG7TBL units. One won't get a RTCM GPS lock with initial power on after it's been off for a while... like a day or more. The GPS Lock LED on the GPS comes on but doesn't on the RTCM. Subsequent immediate power cycle cures that. Another I've seen shifts it's time by one second after it's been running for some period.
On the other hand I have one that works flawlessly. So I'm a little concerned about these. I don't know if there some incompatibility with the RTCM or if the BG7TLB units have inconsistent performance.
Any thoughts on that possibly being repeated to the issue Kevin is having?
On Tue, Jun 12, 2018 at 7:24 PM Chuck Henderson < rpt2@chuck.midlandsnetworking.com> wrote:
The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work.
I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime.
I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server.
On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
“Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Tuesday, June 12, 2018 3:35 AM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Kevin,
Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge.
I am also using a Raspberry Pi 3B (not plus)
The master should be the first one to get a time packet to the server, or they should all be tied exactly.
Chuck
On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
A couple facts:
The network has very low latency (sub ms) across all nodes on the VLAN.
The site labeled LAB has a lab quality GPSDO, by NOVUS Power.
The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage.
Question:
Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output.
Here is an example of the output from my implementation:
GPSTime (Lab): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.220000
DrainTime: 06/11/18 20:16:57.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71
GPSTime (Valparaiso): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5
GPSTime (Crown_Point): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6
GPSTime (LaPorte): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6
GPSTime (Lab): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.220000
DrainTime: 06/11/18 20:17:19.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38
GPSTime (Valparaiso): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.260000
DrainTime: 06/11/18 20:17:19.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7
GPSTime (Crown_Point): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.280000
DrainTime: 06/11/18 20:17:19.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5
GPSTime (LaPorte): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.280000
DrainTime: 06/11/18 20:17:19.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1
Results AFTER resetting the MASTER:
GPSTime (Lab): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.220000
DrainTime: 06/11/18 20:31:23.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86
GPSTime (Valparaiso): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8
GPSTime (LaPorte): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7
GPSTime (Crown_Point): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5
Thoughts?
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Monday, June 4, 2018 4:51 AM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Kevin,
I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1
RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0
RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0
I would not be concerned about the times. They are not important.
A better thing to look at is in the Asterisk CLI issue the command voter debug level 3
Then expect output like the following:
GPSTime (South): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
sending KEEPALIVE (GPS) packet to client East digest 99479120
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
GPSTime (East): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
sending KEEPALIVE (GPS) packet to client East digest 99479120
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
GPSTime (South): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.980
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:19.000000000
SysTime: 06/04/18 04:07:19.000000
DrainTime: 06/04/18 04:07:18.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:19.000000000
SysTime: 06/04/18 04:07:19.000000
DrainTime: 06/04/18 04:07:18.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
sending KEEPALIVE (GPS) packet to client East digest 99479120
GPSTime (East): 06/04/18 04:07:19.000000000
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
SysTime: 06/04/18 04:07:19.000000
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
DrainTime: 06/04/18 04:07:18.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (North): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.980
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them.
Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server).
If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one.
I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock.
If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM.
If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs).
Chuck
On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com> wrote:
Hayden,
When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so:
06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed
GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D
GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018
I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Hayden Honeywood *Sent:* Thursday, May 31, 2018 5:56 PM
*To:* app_rpt-users@lists.allstarlink.org *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Is the time in UTC? How far off is the system time?
Have you done a full calibration of each board?
What baud rate do you run on your GPS's?
I tend to use 9600 baud and have had no issues with the GPS's I have used.
There is various GPS debug tools available in the RTCM. They are documented here -
https://wiki.allstarlink.org/wiki/RTCM_Client#GPS
Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency.
I can't comment on that as I built all my boards and they have external frequency references.
Regards
Hayden VK7HH
From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network.
Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments.
Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously?
Regards
Hayden VK7HH
Chuck, A quick comment on PPS polarity… I set RTCM (LAB) command 10 to the opposite of the original setting. Which in this case was changing non-inverted to inverted. It didn’t work… LAB / NOVUS GPSDO <System Time Not Set> Using Primary Voter Host This remained for an hour, until I switched it back. ALL OTHER / BG7TBL GPSDO I set RTCM (everything else) the same, the RTCM did indeed resync, however, this is the output from the RTCM console: 06/16/2018 16:47:19.060 Host Connection Lost (Pri) (192.168.40.210) 06/16/2018 16:47:19.060 Using Primary Voter Host (192.168.40.210) 06/16/2018 16:47:23.060 Time now syncronized to GPS 06/16/2018 16:47:23.500 Host Connection established (Pri) (192.168.40.210) 06/16/2018 16:50:00.580 Inbound (Eth Rx) packet out of bounds by: -800 ***This line continues without end*** Interestingly enough, the SYSTIME read the same (.180 - .280), however GPSTIME was off by ~200ms. This tells me that I have selected the correct PPS polarity. This condition wasn’t able to be tested on the lab quality NOVUS GPSDO, as the RTCM never synced back up. I believe this is telling me the PPS polarity is correct here as well, as inverting the input of the RTCM placed the packets too far out of time to sync. Is it worth having a share screen session to have a look for yourself? It is entirely possible that am incorrectly testing this setup. Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> On Behalf Of Chuck Henderson Sent: Saturday, June 16, 2018 12:48 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Here is what I see in your data. If your network latency is very low as you say, then it looks like your GPS's don't all have the same time. LaPorte and Crown_Point must be exactly the same as each other and Valparaiso is probably within a millisecond, but lab is way off by about 60 milliseconds (could be somewhere between 41 and 79ms), this is a problem and couldn't be network latency or you would see that in a ping test. And all are about 200 to 280 milliseconds different than the NTP time reference that you are using which is not a problem but just effects the debug display. Drain time should be the time that audio is being removed from the buffer and being sent to the transmitter which is sometime in the past as configured by your buffers in the voter.conf. All change in 20ms jumps so a change of 20ms could just be a change of a fraction of a microsecond that incremented from one 20ms buffer slot to the next one. I would really suspect pps polarity in the RTCM config of either "lab" is wrong, or pps polarity in all 3 of the other sites is wrong to account for that kind of discrepancy. Wrong PPS polarity should not keep it from working. Change the pps polarity and wait a minute or two for it to resync and it should start working again. If it doesn't then restart that RTCM after 99 saving the new pps setting. Most PPS pulses I have looked at are between 1ms and 100 ms long. if your pps pulse is 50ms long then it would cause the time to be off by 50ms if you have it set wrong. If the pps pulse is 500ms long then it probably would not work at all if it is set to the wrong polarity because the time would be half a second off and I am sure you don't have that much buffer space. Do you know your pps pulse width? Is it the same in all of your GPS's? When I say pps polarity I am only talking about RTCM menu item 10 being 0 or 1. I have never used the GPS's that you are using so I don't know anything about how they behave. My experience is with the Garmin and with 3 other brands that act exactly the same as the Garmins. They all have configurable PPS pulse width. On Fri, Jun 15, 2018 at 4:41 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, I setup the NTP client and enabled the service, as well as moved the MASTER to LAB. This is the output from VOTER DEBUG LEVEL 3: GPSTime (Lab): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.200000 DrainTime: 06/15/18 15:02:39.160 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90 GPSTime (Valparaiso): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.260000 DrainTime: 06/15/18 15:02:39.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2 GPSTime (LaPorte): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2 GPSTime (Crown_Point): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 In the Asterisk console, I restarted Asterisk by asserting the “reboot now” command and this is the output: GPSTime (Valparaiso): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 299.6 GPSTime (LaPorte): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 328.1 GPSTime (Crown_Point): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.980 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.3 GPSTime (Lab): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.220000 DrainTime: 06/15/18 15:07:04.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.11 Waiting an hour and testing again yields: SysTime: 06/15/18 16:23:30.200000 DrainTime: 06/15/18 16:23:30.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.64 GPSTime (Valparaiso): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 298.1 GPSTime (Crown_Point): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 220.9 GPSTime (LaPorte): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 326.6 It would appear the NTP activity is slowly (expected) netting the SysTime, however, the LAB (master) is still off. If the SysTime comes from Asterisk, then what gives? An unanswered question: What is DrainTime describing, and what drives the displayed value? I notice on yours, everything matches. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: Chuck Henderson <rpt2@chuck.midlandsnetworking.com <mailto:rpt2@chuck.midlandsnetworking.com> > Sent: Wednesday, June 13, 2018 2:34 AM To: kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** That is complicated and I don't remember. All the needed software is installed already with the allstarlink raspberry pi dial install that I used. I think I used 07-08-2017RAT_RC1.img but I can't remember for sure. I also can't remember exactly what I did to set up the time sync but I looked for clues something like man systemd-timesyncd timedatectl I had used ntpd in the past with x86 ACID but didn't need it for the DIAL version I am using now because this other method of getting the time was builtin and all I had to do was a google search and I landed on the exact files to edit to make it use my own dedicated ntp server which I have had running for many years. If it ever dies I will have to re-google how to rebuild it using current hardware. I quit trying to remember things that I can just ask "ok google". On Wed, Jun 13, 2018 at 1:22 AM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, How does one go about syncing the server in app_rpt? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 11:50 PM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Yes, many GPS will sometimes be off by one second in the RTCM. If I have to power cycle any of my voters/RTCMs I usually then do a reboot via the RTCM menu so that the GPS is already up and accurate when the RTCM boots and this seems to eliminate the one second off problem, most of the time. Both Jim and I tried to fix this and neither of us could. When all set up to troubleshoot it the problem never happens. The software has a couple of code segments to try to make it work even when exactly one second off but I don't think that works all the time either. Also in looking at the code I see that SysTime does not matter at all. It is just a reference to look at in the debug output. Mine exactly matches because I also sync my Linux time to a GPS reference separately from the app_rpt voter code. But I tried disabling that and setting the date and time both wrong and the voter still works because it keeps it's own time that is referenced to the master and not dependent on the system time. So the master time does need to be the first time that the voter channel driver receives each second. Other RTCMs time must arrive later than the master arrives but before the buffer length later. The RTCM-GPS time can be any amount wrong as long as all RTCM-GPSs are exactly the same as each other. Start with much bigger buffers than you need and gradually reduce them. On Tue, Jun 12, 2018 at 10:03 PM Tim Sawyer <tisawyer@gmail.com <mailto:tisawyer@gmail.com> > wrote: This may be a bit off topic but I've had problems with the BG7TBL units. One won't get a RTCM GPS lock with initial power on after it's been off for a while... like a day or more. The GPS Lock LED on the GPS comes on but doesn't on the RTCM. Subsequent immediate power cycle cures that. Another I've seen shifts it's time by one second after it's been running for some period. On the other hand I have one that works flawlessly. So I'm a little concerned about these. I don't know if there some incompatibility with the RTCM or if the BG7TLB units have inconsistent performance. Any thoughts on that possibly being repeated to the issue Kevin is having? On Tue, Jun 12, 2018 at 7:24 PM Chuck Henderson <rpt2@chuck.midlandsnetworking.com <mailto:rpt2@chuck.midlandsnetworking.com> > wrote: The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work. I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime. I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server. On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, “Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 3:35 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge. I am also using a Raspberry Pi 3B (not plus) The master should be the first one to get a time packet to the server, or they should all be tied exactly. Chuck On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, A couple facts: The network has very low latency (sub ms) across all nodes on the VLAN. The site labeled LAB has a lab quality GPSDO, by NOVUS Power. The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage. Question: Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output. Here is an example of the output from my implementation: GPSTime (Lab): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.220000 DrainTime: 06/11/18 20:16:57.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71 GPSTime (Valparaiso): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5 GPSTime (Crown_Point): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6 GPSTime (LaPorte): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6 GPSTime (Lab): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.220000 DrainTime: 06/11/18 20:17:19.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38 GPSTime (Valparaiso): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.260000 DrainTime: 06/11/18 20:17:19.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7 GPSTime (Crown_Point): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5 GPSTime (LaPorte): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1 Results AFTER resetting the MASTER: GPSTime (Lab): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.220000 DrainTime: 06/11/18 20:31:23.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86 GPSTime (Valparaiso): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8 GPSTime (LaPorte): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7 GPSTime (Crown_Point): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5 Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Monday, June 4, 2018 4:51 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1 RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 I would not be concerned about the times. They are not important. A better thing to look at is in the Asterisk CLI issue the command voter debug level 3 Then expect output like the following: GPSTime (South): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (East): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (South): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 GPSTime (East): 06/04/18 04:07:19.000000000 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 SysTime: 06/04/18 04:07:19.000000 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 DrainTime: 06/04/18 04:07:18.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (North): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them. Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server). If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one. I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock. If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM. If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs). Chuck On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Hayden, When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so: 06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018 I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Thursday, May 31, 2018 5:56 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Is the time in UTC? How far off is the system time? Have you done a full calibration of each board? What baud rate do you run on your GPS's? I tend to use 9600 baud and have had no issues with the GPS's I have used. There is various GPS debug tools available in the RTCM. They are documented here - https://wiki.allstarlink.org/wiki/RTCM_Client#GPS Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency. I can't comment on that as I built all my boards and they have external frequency references. Regards Hayden VK7HH From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network. Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments. Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously? Regards Hayden VK7HH
Just a little words to all that participate in that thread. Thanks a lot! I am learning a lot. Keep it rolling, very interresting stuff Le sam. 16 juin 2018 à 13:28, Kevin Babich <kevin.babich@gmail.com> a écrit :
Chuck,
A quick comment on PPS polarity… I set RTCM (LAB) command 10 to the opposite of the original setting. Which in this case was changing non-inverted to inverted. It didn’t work…
LAB / NOVUS GPSDO
<System Time Not Set> Using Primary Voter Host
This remained for an hour, until I switched it back.
ALL OTHER / BG7TBL GPSDO
I set RTCM (everything else) the same, the RTCM did indeed resync, however, this is the output from the RTCM console:
06/16/2018 16:47:19.060 Host Connection Lost (Pri) (192.168.40.210)
06/16/2018 16:47:19.060 Using Primary Voter Host (192.168.40.210)
06/16/2018 16:47:23.060 Time now syncronized to GPS
06/16/2018 16:47:23.500 Host Connection established (Pri) (192.168.40.210)
06/16/2018 16:50:00.580 Inbound (Eth Rx) packet out of bounds by: -800 ***This line continues without end***
Interestingly enough, the SYSTIME read the same (.180 - .280), however GPSTIME was off by ~200ms. This tells me that I have selected the correct PPS polarity. This condition wasn’t able to be tested on the lab quality NOVUS GPSDO, as the RTCM never synced back up. I believe this is telling me the PPS polarity is correct here as well, as inverting the input of the RTCM placed the packets too far out of time to sync.
Is it worth having a share screen session to have a look for yourself? It is entirely possible that am incorrectly testing this setup.
Thoughts?
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Saturday, June 16, 2018 12:48 AM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Kevin,
Here is what I see in your data. If your network latency is very low as you say, then it looks like your GPS's don't all have the same time. LaPorte and Crown_Point must be exactly the same as each other and Valparaiso is probably within a millisecond, but lab is way off by about 60 milliseconds (could be somewhere between 41 and 79ms), this is a problem and couldn't be network latency or you would see that in a ping test. And all are about 200 to 280 milliseconds different than the NTP time reference that you are using which is not a problem but just effects the debug display. Drain time should be the time that audio is being removed from the buffer and being sent to the transmitter which is sometime in the past as configured by your buffers in the voter.conf. All change in 20ms jumps so a change of 20ms could just be a change of a fraction of a microsecond that incremented from one 20ms buffer slot to the next one.
I would really suspect pps polarity in the RTCM config of either "lab" is wrong, or pps polarity in all 3 of the other sites is wrong to account for that kind of discrepancy.
Wrong PPS polarity should not keep it from working. Change the pps polarity and wait a minute or two for it to resync and it should start working again. If it doesn't then restart that RTCM after 99 saving the new pps setting.
Most PPS pulses I have looked at are between 1ms and 100 ms long. if your pps pulse is 50ms long then it would cause the time to be off by 50ms if you have it set wrong. If the pps pulse is 500ms long then it probably would not work at all if it is set to the wrong polarity because the time would be half a second off and I am sure you don't have that much buffer space. Do you know your pps pulse width? Is it the same in all of your GPS's?
When I say pps polarity I am only talking about RTCM menu item 10 being 0 or 1.
I have never used the GPS's that you are using so I don't know anything about how they behave. My experience is with the Garmin and with 3 other brands that act exactly the same as the Garmins. They all have configurable PPS pulse width.
On Fri, Jun 15, 2018 at 4:41 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
I setup the NTP client and enabled the service, as well as moved the MASTER to LAB. This is the output from VOTER DEBUG LEVEL 3:
GPSTime (Lab): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.200000
DrainTime: 06/15/18 15:02:39.160
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90
GPSTime (Valparaiso): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.260000
DrainTime: 06/15/18 15:02:39.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2
GPSTime (LaPorte): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.280000
DrainTime: 06/15/18 15:02:39.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2
GPSTime (Crown_Point): 06/15/18 15:02:39.000000000
SysTime: 06/15/18 15:02:39.280000
DrainTime: 06/15/18 15:02:39.240
In the Asterisk console, I restarted Asterisk by asserting the “reboot now” command and this is the output:
GPSTime (Valparaiso): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.960
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 299.6
GPSTime (LaPorte): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.960
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 328.1
GPSTime (Crown_Point): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.020000
DrainTime: 06/15/18 15:07:03.980
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.3
GPSTime (Lab): 06/15/18 15:07:04.000000000
SysTime: 06/15/18 15:07:04.220000
DrainTime: 06/15/18 15:07:04.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.11
Waiting an hour and testing again yields:
SysTime: 06/15/18 16:23:30.200000
DrainTime: 06/15/18 16:23:30.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.64
GPSTime (Valparaiso): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 298.1
GPSTime (Crown_Point): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 220.9
GPSTime (LaPorte): 06/15/18 16:23:30.000000000
SysTime: 06/15/18 16:23:30.280000
DrainTime: 06/15/18 16:23:30.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 326.6
It would appear the NTP activity is slowly (expected) netting the SysTime, however, the LAB (master) is still off. If the SysTime comes from Asterisk, then what gives?
An unanswered question: What is DrainTime describing, and what drives the displayed value? I notice on yours, everything matches.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* Chuck Henderson <rpt2@chuck.midlandsnetworking.com> *Sent:* Wednesday, June 13, 2018 2:34 AM *To:* kevin.babich@gmail.com *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
That is complicated and I don't remember. All the needed software is installed already with the allstarlink raspberry pi dial install that I used. I
think I used 07-08-2017RAT_RC1.img but I can't remember for sure. I also can't remember exactly what I did to set up the time sync but I looked for clues something like
man systemd-timesyncd
timedatectl
I had used ntpd in the past with x86 ACID but didn't need it for the DIAL version I am using now because this other method of getting the time was builtin and all I had to do was a google search and I landed on the exact files to edit to make it use my own dedicated ntp server which I have had running for many years. If it ever dies I will have to re-google how to rebuild it using current hardware. I quit trying to remember things that I can just ask "ok google".
On Wed, Jun 13, 2018 at 1:22 AM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
How does one go about syncing the server in app_rpt?
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Tuesday, June 12, 2018 11:50 PM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Yes, many GPS will sometimes be off by one second in the RTCM. If I have to power cycle any of my voters/RTCMs I usually then do a reboot via the RTCM menu so that the GPS is already up and accurate when the RTCM boots and this seems to eliminate the one second off problem, most of the time. Both Jim and I tried to fix this and neither of us could. When all set up to troubleshoot it the problem never happens. The software has a couple of code segments to try to make it work even when exactly one second off but I don't think that works all the time either.
Also in looking at the code I see that SysTime does not matter at all. It is just a reference to look at in the debug output. Mine exactly matches because I also sync my Linux time to a GPS reference separately from the app_rpt voter code. But I tried disabling that and setting the date and time both wrong and the voter still works because it keeps it's own time that is referenced to the master and not dependent on the system time.
So the master time does need to be the first time that the voter channel driver receives each second. Other RTCMs time must arrive later than the master arrives but before the buffer length later.
The RTCM-GPS time can be any amount wrong as long as all RTCM-GPSs are exactly the same as each other.
Start with much bigger buffers than you need and gradually reduce them.
On Tue, Jun 12, 2018 at 10:03 PM Tim Sawyer <tisawyer@gmail.com> wrote:
This may be a bit off topic but I've had problems with the BG7TBL units. One won't get a RTCM GPS lock with initial power on after it's been off for a while... like a day or more. The GPS Lock LED on the GPS comes on but doesn't on the RTCM. Subsequent immediate power cycle cures that. Another I've seen shifts it's time by one second after it's been running for some period.
On the other hand I have one that works flawlessly. So I'm a little concerned about these. I don't know if there some incompatibility with the RTCM or if the BG7TLB units have inconsistent performance.
Any thoughts on that possibly being repeated to the issue Kevin is having?
On Tue, Jun 12, 2018 at 7:24 PM Chuck Henderson < rpt2@chuck.midlandsnetworking.com> wrote:
The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work.
I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime.
I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server.
On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
“Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Tuesday, June 12, 2018 3:35 AM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Kevin,
Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge.
I am also using a Raspberry Pi 3B (not plus)
The master should be the first one to get a time packet to the server, or they should all be tied exactly.
Chuck
On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com> wrote:
Chuck,
A couple facts:
The network has very low latency (sub ms) across all nodes on the VLAN.
The site labeled LAB has a lab quality GPSDO, by NOVUS Power.
The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage.
Question:
Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output.
Here is an example of the output from my implementation:
GPSTime (Lab): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.220000
DrainTime: 06/11/18 20:16:57.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71
GPSTime (Valparaiso): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5
GPSTime (Crown_Point): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6
GPSTime (LaPorte): 06/11/18 20:16:57.000000000
SysTime: 06/11/18 20:16:57.280000
DrainTime: 06/11/18 20:16:57.220
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6
GPSTime (Lab): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.220000
DrainTime: 06/11/18 20:17:19.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38
GPSTime (Valparaiso): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.260000
DrainTime: 06/11/18 20:17:19.220
Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7
GPSTime (Crown_Point): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.280000
DrainTime: 06/11/18 20:17:19.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5
GPSTime (LaPorte): 06/11/18 20:17:19.000000000
SysTime: 06/11/18 20:17:19.280000
DrainTime: 06/11/18 20:17:19.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1
Results AFTER resetting the MASTER:
GPSTime (Lab): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.220000
DrainTime: 06/11/18 20:31:23.180
Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86
GPSTime (Valparaiso): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8
GPSTime (LaPorte): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7
GPSTime (Crown_Point): 06/11/18 20:31:23.000000000
SysTime: 06/11/18 20:31:23.280000
DrainTime: 06/11/18 20:31:23.240
Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5
Thoughts?
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Chuck Henderson *Sent:* Monday, June 4, 2018 4:51 AM *To:* Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Kevin,
I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1
RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0
RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0
I would not be concerned about the times. They are not important.
A better thing to look at is in the Asterisk CLI issue the command voter debug level 3
Then expect output like the following:
GPSTime (South): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:16.000000000
SysTime: 06/04/18 04:07:16.000000
DrainTime: 06/04/18 04:07:15.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
sending KEEPALIVE (GPS) packet to client East digest 99479120
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
GPSTime (East): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:17.000000000
SysTime: 06/04/18 04:07:17.000000
DrainTime: 06/04/18 04:07:16.980
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
sending KEEPALIVE (GPS) packet to client East digest 99479120
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
GPSTime (South): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.980
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:18.000000000
SysTime: 06/04/18 04:07:18.000000
DrainTime: 06/04/18 04:07:17.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:19.000000000
SysTime: 06/04/18 04:07:19.000000
DrainTime: 06/04/18 04:07:18.960
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
GPSTime (North): 06/04/18 04:07:19.000000000
SysTime: 06/04/18 04:07:19.000000
DrainTime: 06/04/18 04:07:18.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
sending KEEPALIVE (GPS) packet to client East digest 99479120
GPSTime (East): 06/04/18 04:07:19.000000000
sending KEEPALIVE (GPS) packet to client North digest dd455bb0
SysTime: 06/04/18 04:07:19.000000
sending KEEPALIVE (GPS) packet to client South digest 4a396b22
DrainTime: 06/04/18 04:07:18.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (North): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.960
Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4
GPSTime (East): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.980
Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2
GPSTime (South): 06/04/18 04:07:20.000000000
SysTime: 06/04/18 04:07:20.000000
DrainTime: 06/04/18 04:07:19.980
Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7
Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them.
Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server).
If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one.
I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock.
If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM.
If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs).
Chuck
On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com> wrote:
Hayden,
When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so:
06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed
GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D
GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018
I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity.
Kindest Regards,
Kevin Babich | N9IAA
Valparaiso, IN 46383
*From:* App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> *On Behalf Of *Hayden Honeywood *Sent:* Thursday, May 31, 2018 5:56 PM
*To:* app_rpt-users@lists.allstarlink.org *Subject:* Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
Is the time in UTC? How far off is the system time?
Have you done a full calibration of each board?
What baud rate do you run on your GPS's?
I tend to use 9600 baud and have had no issues with the GPS's I have used.
There is various GPS debug tools available in the RTCM. They are documented here -
https://wiki.allstarlink.org/wiki/RTCM_Client#GPS
Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency.
I can't comment on that as I built all my boards and they have external frequency references.
Regards
Hayden VK7HH
From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***
What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network.
Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments.
Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously?
Regards
Hayden VK7HH
To whom it may concern: I changed the NOVUS GPSDO to match the other sites, this was the output. At one point the SysTime matched the GPSTime, don’t know how or why it changed. It was previously postulated that the LAB time was earlier as it was closest to the Asterisk server, apparently not the case as one can observe a distant site with earlier time. GPSTime (Valparaiso): 06/20/18 17:10:49.000000000 SysTime: 06/20/18 17:10:49.260000 DrainTime: 06/20/18 17:10:49.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 300.5 GPSTime (Lab): 06/20/18 17:10:49.000000000 SysTime: 06/20/18 17:10:49.260000 DrainTime: 06/20/18 17:10:49.240 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.7 GPSTime (LaPorte): 06/20/18 17:10:49.000000000 SysTime: 06/20/18 17:10:49.280000 DrainTime: 06/20/18 17:10:49.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.7 GPSTime (Crown_Point): 06/20/18 17:10:49.000000000 SysTime: 06/20/18 17:10:49.280000 DrainTime: 06/20/18 17:10:49.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.7 Then after checking back, after restarting Asterisk, I found this: GPSTime (LaPorte): 06/20/18 16:57:02.000000000 SysTime: 06/20/18 16:57:02.020000 DrainTime: 06/20/18 16:57:01.960 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.4 GPSTime (Valparaiso): 06/20/18 16:57:02.000000000 SysTime: 06/20/18 16:57:02.260000 DrainTime: 06/20/18 16:57:02.220 Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 305.8 GPSTime (Lab): 06/20/18 16:57:02.000000000 SysTime: 06/20/18 16:57:02.280000 DrainTime: 06/20/18 16:57:02.240 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 232.8 GPSTime (Crown_Point): 06/20/18 16:57:02.000000000 SysTime: 06/20/18 16:57:02.280000 DrainTime: 06/20/18 16:57:02.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 221.0 Then booted Asterisk: GPSTime (LaPorte): 06/20/18 17:05:41.000000000 SysTime: 06/20/18 17:05:41.000000 DrainTime: 06/20/18 17:05:40.960 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 330.6 GPSTime (Lab): 06/20/18 17:05:41.000000000 SysTime: 06/20/18 17:05:41.260000 DrainTime: 06/20/18 17:05:41.220 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 232.2 GPSTime (Valparaiso): 06/20/18 17:05:41.000000000 SysTime: 06/20/18 17:05:41.260000 DrainTime: 06/20/18 17:05:41.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 296.1 GPSTime (Crown_Point): 06/20/18 17:05:41.000000000 SysTime: 06/20/18 17:05:41.280000 DrainTime: 06/20/18 17:05:41.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 221.7 And booted Asterisk once more: GPSTime (Lab): 06/20/18 17:16:26.000000000 SysTime: 06/20/18 17:16:26.000000 DrainTime: 06/20/18 17:16:25.980 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.9 GPSTime (Valparaiso): 06/20/18 17:16:26.000000000 SysTime: 06/20/18 17:16:26.260000 DrainTime: 06/20/18 17:16:26.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 296.7 GPSTime (Crown_Point): 06/20/18 17:16:26.000000000 SysTime: 06/20/18 17:16:26.260000 DrainTime: 06/20/18 17:16:26.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.3 GPSTime (LaPorte): 06/20/18 17:16:26.000000000 SysTime: 06/20/18 17:16:26.280000 DrainTime: 06/20/18 17:16:26.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 331.0 ****This is what I saw before the GPSDO change:*** GPSTime (Lab): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.200000 DrainTime: 06/15/18 15:02:39.160 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90 GPSTime (Valparaiso): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.260000 DrainTime: 06/15/18 15:02:39.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2 GPSTime (LaPorte): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2 GPSTime (Crown_Point): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 What is clear to me is that there appears to be no known rhyme or reason this is happening, nor a method of troubleshooting the same. This appears to affect the RTCM to properly generate waveforms synchronously which is necessary for Simulcast. Perhaps I’m expecting too much and need to lower the bar… it’s just ham radio… right? A quick comment on PPS polarity… I set RTCM (LAB) command 10 to the opposite of the original setting. Which in this case was changing non-inverted to inverted. It didn’t work… LAB / NOVUS GPSDO <System Time Not Set> Using Primary Voter Host This remained for an hour, until I switched it back. ALL OTHER / BG7TBL GPSDO I set RTCM (everything else) the same, the RTCM did indeed resync, however, this is the output from the RTCM console: 06/16/2018 16:47:19.060 Host Connection Lost (Pri) (192.168.40.210) 06/16/2018 16:47:19.060 Using Primary Voter Host (192.168.40.210) 06/16/2018 16:47:23.060 Time now syncronized to GPS 06/16/2018 16:47:23.500 Host Connection established (Pri) (192.168.40.210) 06/16/2018 16:50:00.580 Inbound (Eth Rx) packet out of bounds by: -800 ***This line continues without end*** Interestingly enough, the SYSTIME read the same (.180 - .280), however GPSTIME was off by ~200ms. This tells me that I have selected the correct PPS polarity. This condition wasn’t able to be tested on the lab quality NOVUS GPSDO, as the RTCM never synced back up. I believe this is telling me the PPS polarity is correct here as well, as inverting the input of the RTCM placed the packets too far out of time to sync. Is it worth having a share screen session to have a look for yourself? It is entirely possible that am incorrectly testing this setup. Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org> On Behalf Of Chuck Henderson Sent: Saturday, June 16, 2018 12:48 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Here is what I see in your data. If your network latency is very low as you say, then it looks like your GPS's don't all have the same time. LaPorte and Crown_Point must be exactly the same as each other and Valparaiso is probably within a millisecond, but lab is way off by about 60 milliseconds (could be somewhere between 41 and 79ms), this is a problem and couldn't be network latency or you would see that in a ping test. And all are about 200 to 280 milliseconds different than the NTP time reference that you are using which is not a problem but just effects the debug display. Drain time should be the time that audio is being removed from the buffer and being sent to the transmitter which is sometime in the past as configured by your buffers in the voter.conf. All change in 20ms jumps so a change of 20ms could just be a change of a fraction of a microsecond that incremented from one 20ms buffer slot to the next one. I would really suspect pps polarity in the RTCM config of either "lab" is wrong, or pps polarity in all 3 of the other sites is wrong to account for that kind of discrepancy. Wrong PPS polarity should not keep it from working. Change the pps polarity and wait a minute or two for it to resync and it should start working again. If it doesn't then restart that RTCM after 99 saving the new pps setting. Most PPS pulses I have looked at are between 1ms and 100 ms long. if your pps pulse is 50ms long then it would cause the time to be off by 50ms if you have it set wrong. If the pps pulse is 500ms long then it probably would not work at all if it is set to the wrong polarity because the time would be half a second off and I am sure you don't have that much buffer space. Do you know your pps pulse width? Is it the same in all of your GPS's? When I say pps polarity I am only talking about RTCM menu item 10 being 0 or 1. I have never used the GPS's that you are using so I don't know anything about how they behave. My experience is with the Garmin and with 3 other brands that act exactly the same as the Garmins. They all have configurable PPS pulse width. On Fri, Jun 15, 2018 at 4:41 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, I setup the NTP client and enabled the service, as well as moved the MASTER to LAB. This is the output from VOTER DEBUG LEVEL 3: GPSTime (Lab): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.200000 DrainTime: 06/15/18 15:02:39.160 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.90 GPSTime (Valparaiso): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.260000 DrainTime: 06/15/18 15:02:39.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 301.2 GPSTime (LaPorte): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 329.2 GPSTime (Crown_Point): 06/15/18 15:02:39.000000000 SysTime: 06/15/18 15:02:39.280000 DrainTime: 06/15/18 15:02:39.240 In the Asterisk console, I restarted Asterisk by asserting the “reboot now” command and this is the output: GPSTime (Valparaiso): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 299.6 GPSTime (LaPorte): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.960 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 328.1 GPSTime (Crown_Point): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.020000 DrainTime: 06/15/18 15:07:03.980 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 219.3 GPSTime (Lab): 06/15/18 15:07:04.000000000 SysTime: 06/15/18 15:07:04.220000 DrainTime: 06/15/18 15:07:04.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.11 Waiting an hour and testing again yields: SysTime: 06/15/18 16:23:30.200000 DrainTime: 06/15/18 16:23:30.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 229.64 GPSTime (Valparaiso): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 298.1 GPSTime (Crown_Point): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 220.9 GPSTime (LaPorte): 06/15/18 16:23:30.000000000 SysTime: 06/15/18 16:23:30.280000 DrainTime: 06/15/18 16:23:30.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 326.6 It would appear the NTP activity is slowly (expected) netting the SysTime, however, the LAB (master) is still off. If the SysTime comes from Asterisk, then what gives? An unanswered question: What is DrainTime describing, and what drives the displayed value? I notice on yours, everything matches. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: Chuck Henderson <rpt2@chuck.midlandsnetworking.com <mailto:rpt2@chuck.midlandsnetworking.com> > Sent: Wednesday, June 13, 2018 2:34 AM To: kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** That is complicated and I don't remember. All the needed software is installed already with the allstarlink raspberry pi dial install that I used. I think I used 07-08-2017RAT_RC1.img but I can't remember for sure. I also can't remember exactly what I did to set up the time sync but I looked for clues something like man systemd-timesyncd timedatectl I had used ntpd in the past with x86 ACID but didn't need it for the DIAL version I am using now because this other method of getting the time was builtin and all I had to do was a google search and I landed on the exact files to edit to make it use my own dedicated ntp server which I have had running for many years. If it ever dies I will have to re-google how to rebuild it using current hardware. I quit trying to remember things that I can just ask "ok google". On Wed, Jun 13, 2018 at 1:22 AM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, How does one go about syncing the server in app_rpt? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 11:50 PM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Yes, many GPS will sometimes be off by one second in the RTCM. If I have to power cycle any of my voters/RTCMs I usually then do a reboot via the RTCM menu so that the GPS is already up and accurate when the RTCM boots and this seems to eliminate the one second off problem, most of the time. Both Jim and I tried to fix this and neither of us could. When all set up to troubleshoot it the problem never happens. The software has a couple of code segments to try to make it work even when exactly one second off but I don't think that works all the time either. Also in looking at the code I see that SysTime does not matter at all. It is just a reference to look at in the debug output. Mine exactly matches because I also sync my Linux time to a GPS reference separately from the app_rpt voter code. But I tried disabling that and setting the date and time both wrong and the voter still works because it keeps it's own time that is referenced to the master and not dependent on the system time. So the master time does need to be the first time that the voter channel driver receives each second. Other RTCMs time must arrive later than the master arrives but before the buffer length later. The RTCM-GPS time can be any amount wrong as long as all RTCM-GPSs are exactly the same as each other. Start with much bigger buffers than you need and gradually reduce them. On Tue, Jun 12, 2018 at 10:03 PM Tim Sawyer <tisawyer@gmail.com <mailto:tisawyer@gmail.com> > wrote: This may be a bit off topic but I've had problems with the BG7TBL units. One won't get a RTCM GPS lock with initial power on after it's been off for a while... like a day or more. The GPS Lock LED on the GPS comes on but doesn't on the RTCM. Subsequent immediate power cycle cures that. Another I've seen shifts it's time by one second after it's been running for some period. On the other hand I have one that works flawlessly. So I'm a little concerned about these. I don't know if there some incompatibility with the RTCM or if the BG7TLB units have inconsistent performance. Any thoughts on that possibly being repeated to the issue Kevin is having? On Tue, Jun 12, 2018 at 7:24 PM Chuck Henderson <rpt2@chuck.midlandsnetworking.com <mailto:rpt2@chuck.midlandsnetworking.com> > wrote: The server needs to have the earliest time of all GPS time receivers. Master RTCM should be on the same LAN switch as the server and all other GPS time receivers must be at least some fraction of nano seconds delayed from the master time. Your LAB time arrives at the server before the master time arrives and that is not allowed for in the software. I am surprised that it even works at all. It should not work. I also am surprised that the SysTime does not exactly match with the Master Time “Valparaiso” as I checked 3 servers of mine here and the SysTime always exactly matches the master GPSTime. Master GPSTime is the reference for SysTime. I am reviewing the source to see what a non master time arriving before the master time might mess up. I know it has never worked for me when I have tried using an even slightly delayed master time like putting a couple of store and forward ethernet devices between the master time and the server. On Tue, Jun 12, 2018 at 4:31 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, “Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online. The PPS Polarity is set to non-inverted, the only selection which works. We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other. Perhaps the BG7TBL oscillator has an issue. I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Tuesday, June 12, 2018 3:35 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, Which one is the master? From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate. Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge. I am also using a Raspberry Pi 3B (not plus) The master should be the first one to get a time packet to the server, or they should all be tied exactly. Chuck On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Chuck, A couple facts: The network has very low latency (sub ms) across all nodes on the VLAN. The site labeled LAB has a lab quality GPSDO, by NOVUS Power. The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage. Question: Is the reported system time exactly as reported by the timebase? If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output. Here is an example of the output from my implementation: GPSTime (Lab): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.220000 DrainTime: 06/11/18 20:16:57.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71 GPSTime (Valparaiso): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5 GPSTime (Crown_Point): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6 GPSTime (LaPorte): 06/11/18 20:16:57.000000000 SysTime: 06/11/18 20:16:57.280000 DrainTime: 06/11/18 20:16:57.220 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6 GPSTime (Lab): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.220000 DrainTime: 06/11/18 20:17:19.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38 GPSTime (Valparaiso): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.260000 DrainTime: 06/11/18 20:17:19.220 Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7 GPSTime (Crown_Point): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5 GPSTime (LaPorte): 06/11/18 20:17:19.000000000 SysTime: 06/11/18 20:17:19.280000 DrainTime: 06/11/18 20:17:19.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1 Results AFTER resetting the MASTER: GPSTime (Lab): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.220000 DrainTime: 06/11/18 20:31:23.180 Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86 GPSTime (Valparaiso): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8 GPSTime (LaPorte): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7 GPSTime (Crown_Point): 06/11/18 20:31:23.000000000 SysTime: 06/11/18 20:31:23.280000 DrainTime: 06/11/18 20:31:23.240 Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5 Thoughts? Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Chuck Henderson Sent: Monday, June 4, 2018 4:51 AM To: Users of Asterisk app_rpt <app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> > Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Kevin, I just checked 3 of my RTCM's and got the following results. RTCM1: Current Time: Mon Jun 04, 2018 08:55:43.640 Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec Last Rx Pkt index: 1080, inbounds: 1 RTCM2: Current Time: Mon Jun 04, 2018 08:58:20.300 Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 RTCM3: Current Time: Mon Jun 04, 2018 08:58:48.060 Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec Last Rx Pkt index: 0, inbounds: 0 I would not be concerned about the times. They are not important. A better thing to look at is in the Asterisk CLI issue the command voter debug level 3 Then expect output like the following: GPSTime (South): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:16.000000000 SysTime: 06/04/18 04:07:16.000000 DrainTime: 06/04/18 04:07:15.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (East): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:17.000000000 SysTime: 06/04/18 04:07:17.000000 DrainTime: 06/04/18 04:07:16.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 GPSTime (South): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:18.000000000 SysTime: 06/04/18 04:07:18.000000 DrainTime: 06/04/18 04:07:17.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 GPSTime (North): 06/04/18 04:07:19.000000000 SysTime: 06/04/18 04:07:19.000000 DrainTime: 06/04/18 04:07:18.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 sending KEEPALIVE (GPS) packet to client East digest 99479120 GPSTime (East): 06/04/18 04:07:19.000000000 sending KEEPALIVE (GPS) packet to client North digest dd455bb0 SysTime: 06/04/18 04:07:19.000000 sending KEEPALIVE (GPS) packet to client South digest 4a396b22 DrainTime: 06/04/18 04:07:18.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (North): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.960 Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4 GPSTime (East): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2 GPSTime (South): 06/04/18 04:07:20.000000000 SysTime: 06/04/18 04:07:20.000000 DrainTime: 06/04/18 04:07:19.980 Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7 Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them. Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second. (this is with no signals present and with a very low network latency between RTCMs and the server). If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one. I also recommend not being logged into the RTCMs via telnet except to make setting changes. Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock. If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem. But do not evaluate if the problem is solved while you have an active telnet session to the RTCM. If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs). Chuck On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich@gmail.com <mailto:kevin.babich@gmail.com> > wrote: Hayden, When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so: 06/01/2018 00:22:12.240 Warning: GPS Data time period elapsed GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun 1 00:22:13 2018 I suspect this has to do with the receiver losing lock. However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel. The GPS antenna is designed with high RF immunity and is at a site with little activity. Kindest Regards, Kevin Babich | N9IAA Valparaiso, IN 46383 From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Thursday, May 31, 2018 5:56 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** Is the time in UTC? How far off is the system time? Have you done a full calibration of each board? What baud rate do you run on your GPS's? I tend to use 9600 baud and have had no issues with the GPS's I have used. There is various GPS debug tools available in the RTCM. They are documented here - https://wiki.allstarlink.org/wiki/RTCM_Client#GPS Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency. I can't comment on that as I built all my boards and they have external frequency references. Regards Hayden VK7HH From: App_rpt-users <app_rpt-users-bounces@lists.allstarlink.org <mailto:app_rpt-users-bounces@lists.allstarlink.org> > On Behalf Of Hayden Honeywood Sent: Wednesday, May 30, 2018 5:34 PM To: app_rpt-users@lists.allstarlink.org <mailto:app_rpt-users@lists.allstarlink.org> Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated*** What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480. These values were based upon the math Jim provided, which is a function of maximum latencies on the network. We rarely have greater than 1ms to any host on the network. Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments. Some users have had success with a dedicated RTCM as the master timing source (with no radio connected). Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously? Regards Hayden VK7HH
participants (3)
-
Chuck Henderson -
Kevin Babich -
Pierre Martel