[App_rpt-users] RTCM [RX Packet Out of Bounds] Updated

Tue Jun 12 21:30:43 UTC 2018

Chuck,

“Valparaiso” is the master, while “Lab” is closer to the server, it isn’t regularly online.  The PPS Polarity is set to non-inverted, the only selection which works.  We have more of the NOVUS GPSDO’s, I will try it to see if it remains in-sync with the other.  Perhaps the BG7TBL oscillator has an issue.  I know that one of his models had an issue with floating point math and didn’t report time accurately, I believe it was corrected in mine, but, maybe not.

Kindest Regards,

Kevin Babich | N9IAA

Valparaiso, IN 46383

From: App_rpt-users <app_rpt-users-bounces at lists.allstarlink.org> On Behalf Of Chuck Henderson
Sent: Tuesday, June 12, 2018 3:35 AM
To: Users of Asterisk app_rpt <app_rpt-users at lists.allstarlink.org>
Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***

Kevin,

Which one is the master?  From the looks of it I would say that "lab" should be the master as it appears to be the first one to arrive at the server so must have the lowest network delay or they are inaccurate.  Inaccuracy could be caused by having the PPS polarity wrong in the RTCM and are setting time based on the trailing edge of the pulse rather than the leading edge.

I am also using a Raspberry Pi 3B (not plus)

The master should be the first one to get a time packet to the server, or they should all be tied exactly.

Chuck

On Mon, Jun 11, 2018 at 8:46 PM Kevin Babich <kevin.babich at gmail.com <mailto:kevin.babich at gmail.com> > wrote:

Chuck,

A couple facts:

The network has very low latency (sub ms) across all nodes on the VLAN.

The site labeled LAB has a lab quality GPSDO, by NOVUS Power.

The other sites utilize the GPSDO, by BG7TBL, as available on eBay, all are the 2016 vintage.

Question:

Is the reported system time exactly as reported by the timebase?  If so, it would appear I have an issue with the accuracy of the GPSDO NMEA output.

Here is an example of the output from my implementation:

GPSTime (Lab):   06/11/18 20:16:57.000000000

SysTime:   06/11/18 20:16:57.220000

DrainTime: 06/11/18 20:16:57.180

Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.71

GPSTime (Valparaiso):   06/11/18 20:16:57.000000000

SysTime:   06/11/18 20:16:57.280000

DrainTime: 06/11/18 20:16:57.220

Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.06W, Elev: 299.5

GPSTime (Crown_Point):   06/11/18 20:16:57.000000000

SysTime:   06/11/18 20:16:57.280000

DrainTime: 06/11/18 20:16:57.220

Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.6

GPSTime (LaPorte):   06/11/18 20:16:57.000000000

SysTime:   06/11/18 20:16:57.280000

DrainTime: 06/11/18 20:16:57.220

Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.6

GPSTime (Lab):   06/11/18 20:17:19.000000000

SysTime:   06/11/18 20:17:19.220000

DrainTime: 06/11/18 20:17:19.180

Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 233.38

GPSTime (Valparaiso):   06/11/18 20:17:19.000000000

SysTime:   06/11/18 20:17:19.260000

DrainTime: 06/11/18 20:17:19.220

Got GPS (Valparaiso): Lat: 4131.82N, Lon: 08702.05W, Elev: 293.7

GPSTime (Crown_Point):   06/11/18 20:17:19.000000000

SysTime:   06/11/18 20:17:19.280000

DrainTime: 06/11/18 20:17:19.240

Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 217.5

GPSTime (LaPorte):   06/11/18 20:17:19.000000000

SysTime:   06/11/18 20:17:19.280000

DrainTime: 06/11/18 20:17:19.240

Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 324.1

Results AFTER resetting the MASTER:

GPSTime (Lab):   06/11/18 20:31:23.000000000

SysTime:   06/11/18 20:31:23.220000

DrainTime: 06/11/18 20:31:23.180

Got GPS (Lab): Lat: 4126.20N, Lon: 08654.13W, Elev: 231.86

GPSTime (Valparaiso):   06/11/18 20:31:23.000000000

SysTime:   06/11/18 20:31:23.280000

DrainTime: 06/11/18 20:31:23.240

Got GPS (Valparaiso): Lat: 4131.81N, Lon: 08702.05W, Elev: 301.8

GPSTime (LaPorte):   06/11/18 20:31:23.000000000

SysTime:   06/11/18 20:31:23.280000

DrainTime: 06/11/18 20:31:23.240

Got GPS (LaPorte): Lat: 4139.38N, Lon: 08645.96W, Elev: 327.7

GPSTime (Crown_Point):   06/11/18 20:31:23.000000000

SysTime:   06/11/18 20:31:23.280000

DrainTime: 06/11/18 20:31:23.240

Got GPS (Crown_Point): Lat: 4121.15N, Lon: 08724.18W, Elev: 218.5

Thoughts?

Kindest Regards,

Kevin Babich | N9IAA

Valparaiso, IN 46383

From: App_rpt-users <app_rpt-users-bounces at lists.allstarlink.org <mailto:app_rpt-users-bounces at lists.allstarlink.org> > On Behalf Of Chuck Henderson
Sent: Monday, June 4, 2018 4:51 AM
To: Users of Asterisk app_rpt <app_rpt-users at lists.allstarlink.org <mailto:app_rpt-users at lists.allstarlink.org> >
Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***

Kevin,

I just checked 3 of my RTCM's and got the following results.
RTCM1:
Current Time: Mon  Jun 04, 2018  08:55:43.640
Last Rx Pkt System time: 06/04/2018 02:55:56.040, diff: 21587600 msec
Last Rx Pkt Timestamp time: 06/04/2018 02:55:56.040, diff: 0 msec
Last Rx Pkt index: 1080, inbounds: 1

RTCM2:
Current Time: Mon  Jun 04, 2018  08:58:20.300
Last Rx Pkt System time: <System Time Not Set>, diff: -905657056 msec
Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec
Last Rx Pkt index: 0, inbounds: 0

RTCM3:
Current Time: Mon  Jun 04, 2018  08:58:48.060
Last Rx Pkt System time: <System Time Not Set>, diff: -905629296 msec
Last Rx Pkt Timestamp time: <System Time Not Set>, diff: 0 msec
Last Rx Pkt index: 0, inbounds: 0

I would not be concerned about the times.  They are not important.

A better thing to look at is in the Asterisk CLI issue the command
voter debug level 3

Then expect output like the following:

GPSTime (South):   06/04/18 04:07:16.000000000

SysTime:   06/04/18 04:07:16.000000

DrainTime: 06/04/18 04:07:15.960

Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7

GPSTime (North):   06/04/18 04:07:16.000000000

SysTime:   06/04/18 04:07:16.000000

DrainTime: 06/04/18 04:07:15.960

Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4

GPSTime (East):   06/04/18 04:07:16.000000000

SysTime:   06/04/18 04:07:16.000000

DrainTime: 06/04/18 04:07:15.980

Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2

sending KEEPALIVE (GPS) packet to client East digest 99479120

sending KEEPALIVE (GPS) packet to client North digest dd455bb0

sending KEEPALIVE (GPS) packet to client South digest 4a396b22

GPSTime (East):   06/04/18 04:07:17.000000000

SysTime:   06/04/18 04:07:17.000000

DrainTime: 06/04/18 04:07:16.980

Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2

GPSTime (South):   06/04/18 04:07:17.000000000

SysTime:   06/04/18 04:07:17.000000

DrainTime: 06/04/18 04:07:16.980

Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7

GPSTime (North):   06/04/18 04:07:17.000000000

SysTime:   06/04/18 04:07:17.000000

DrainTime: 06/04/18 04:07:16.980

Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4

sending KEEPALIVE (GPS) packet to client East digest 99479120

sending KEEPALIVE (GPS) packet to client North digest dd455bb0

sending KEEPALIVE (GPS) packet to client South digest 4a396b22

GPSTime (South):   06/04/18 04:07:18.000000000

SysTime:   06/04/18 04:07:18.000000

DrainTime: 06/04/18 04:07:17.960

Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7

GPSTime (North):   06/04/18 04:07:18.000000000

SysTime:   06/04/18 04:07:18.000000

DrainTime: 06/04/18 04:07:17.980

Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4

GPSTime (East):   06/04/18 04:07:18.000000000

SysTime:   06/04/18 04:07:18.000000

DrainTime: 06/04/18 04:07:17.980

Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2

GPSTime (South):   06/04/18 04:07:19.000000000

SysTime:   06/04/18 04:07:19.000000

DrainTime: 06/04/18 04:07:18.960

Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7

GPSTime (North):   06/04/18 04:07:19.000000000

SysTime:   06/04/18 04:07:19.000000

DrainTime: 06/04/18 04:07:18.960

Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4

sending KEEPALIVE (GPS) packet to client East digest 99479120

GPSTime (East):   06/04/18 04:07:19.000000000

sending KEEPALIVE (GPS) packet to client North digest dd455bb0

SysTime:   06/04/18 04:07:19.000000

sending KEEPALIVE (GPS) packet to client South digest 4a396b22

DrainTime: 06/04/18 04:07:18.980

Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2

GPSTime (North):   06/04/18 04:07:20.000000000

SysTime:   06/04/18 04:07:20.000000

DrainTime: 06/04/18 04:07:19.960

Got GPS (North): Lat: 4030.79N, Lon: 08857.99W, Elev: 245.4

GPSTime (East):   06/04/18 04:07:20.000000000

SysTime:   06/04/18 04:07:20.000000

DrainTime: 06/04/18 04:07:19.980

Got GPS (East): Lat: 4030.02N, Lon: 08855.92W, Elev: 250.2

GPSTime (South):   06/04/18 04:07:20.000000000

SysTime:   06/04/18 04:07:20.000000

DrainTime: 06/04/18 04:07:19.980

Got GPS (South): Lat: 4020.80N, Lon: 08856.35W, Elev: 225.7

Ignore the lines I marked in green and only be concerned with the lines that have "Time" in them.

Notice that each RTCM has 3 timestamp lines that are mostly together and they should be within about 40 ms of each other and should also be within about 40ms of the other RTCMs set of lines for that second.  (this is with no signals present and with a very low network latency between RTCMs and the server).

If you find that one RTCM is many ms out of line with the others, I recommend rebooting that one and/or the master one.

I also recommend not being logged into the RTCMs via telnet except to make setting changes.  Having an active telnet session to the RTCM can have a negative impact on it's ability to maintain time sync lock. Having active output on that telnet session will have a negative impact on it's ability to maintain time sync lock.  

If the above does not solve your problem then try increasing your tx buffer setting to 1400 and see if that eliminates your problem.  But do not evaluate if the problem is solved while you have an active telnet session to the RTCM.

If you still have a problem then send the output of your "voter debug level 3" (with no signals present and no active telnet sessions to RTCMs).

Chuck

On Thu, May 31, 2018 at 7:30 PM, Kevin Babich <kevin.babich at gmail.com <mailto:kevin.babich at gmail.com> > wrote:

Hayden,

When running DEBUG = 32, I see the following, but, only on occasion, say every minute or so:

06/01/2018 00:22:12.240  Warning: GPS Data time period elapsed

GPS-DEBUG: $GPRMC,002213.00,A,4131.82046,N,08702.05712,W,0.027,,010618,,,D*6D

GPS-DEBUG: mon: 5, gps_time: 1527812533, ctime: Fri Jun  1 00:22:13 2018

I suspect this has to do with the receiver losing lock.  However, it doesn’t make sense as the receiver reports a strong signal and doesn’t show signs of loss of signal on the front alarm panel.  The GPS antenna is designed with high RF immunity and is at a site with little activity.

Kindest Regards,

Kevin Babich | N9IAA

Valparaiso, IN 46383

From: App_rpt-users <app_rpt-users-bounces at lists.allstarlink.org <mailto:app_rpt-users-bounces at lists.allstarlink.org> > On Behalf Of Hayden Honeywood
Sent: Thursday, May 31, 2018 5:56 PM

To: app_rpt-users at lists.allstarlink.org <mailto:app_rpt-users at lists.allstarlink.org> 
Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***

Is the time in UTC? How far off is the system time?

Have you done a full calibration of each board?

What baud rate do you run on your GPS's?

I tend to use 9600 baud and have had no issues with the GPS's I have used.

There is various GPS debug tools available in the RTCM. They are documented here - 

https://wiki.allstarlink.org/wiki/RTCM_Client#GPS

Can you possibly try injecting a 9.6MHz signal in place of the crystal on board? Perhaps you have a variant of James KI0KN's problem where his crystals were off frequency.

I can't comment on that as I built all my boards and they have external frequency references.

Regards

Hayden VK7HH

From: App_rpt-users <app_rpt-users-bounces at lists.allstarlink.org <mailto:app_rpt-users-bounces at lists.allstarlink.org> > On Behalf Of Hayden Honeywood
Sent: Wednesday, May 30, 2018 5:34 PM
To: app_rpt-users at lists.allstarlink.org <mailto:app_rpt-users at lists.allstarlink.org> 
Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds] ***Updated***

What is your values in voter.conf for RX buffer (960) and in the RTCM client for TX buffer (480). Are all RTCM TX buffers set to the same value? Yes, 480.  These values were based upon the math Jim provided, which is a function of maximum latencies on the network.  We rarely have greater than 1ms to any host on the network.

Are all the GPS's in a location where they cannot be swamped with RF? Yes, and they utilize PCTEL antennas with filtering for use in RF dense environments.

Some users have had success with a dedicated RTCM as the master timing source (with no radio connected).   Has it been suspected that an RTCM doesn’t have the horsepower to handle both functions contemporaneously?

Regards

Hayden VK7HH

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.allstarlink.org/pipermail/app_rpt-users/attachments/20180530/24a4675d/attachment-0001.html>

------------------------------

Message: 3
Date: Wed, 30 May 2018 19:51:43 -0500
From: "Kevin Babich" <kevin.babich at gmail.com <mailto:kevin.babich at gmail.com> >
To: "'Users of Asterisk app_rpt'"
        <app_rpt-users at lists.allstarlink.org <mailto:app_rpt-users at lists.allstarlink.org> >
Subject: Re: [App_rpt-users] RTCM [RX Packet Out of Bounds]
        ***Updated***
Message-ID: <010001d3f879$8c2b3710$a481a530$@gmail.com <http://gmail.com> >
Content-Type: text/plain; charset="utf-8"

Hayden,

Do you know what exactly “RX Packet Out of Bounds” refers to?  I can infer its meaning, but, one can’t be certain.  Any thoughts why the system time is always so far off?  It has always been, since I began experimenting with the RTCM in 2012.  This is consistent across many RTCM’s.

_______________________________________________
App_rpt-users mailing list
App_rpt-users at lists.allstarlink.org <mailto:App_rpt-users at lists.allstarlink.org> 
http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users

To unsubscribe from this list please visit http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button"
You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem. 

_______________________________________________
App_rpt-users mailing list
App_rpt-users at lists.allstarlink.org <mailto:App_rpt-users at lists.allstarlink.org> 
http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users

To unsubscribe from this list please visit http://lists.allstarlink.org/cgi-bin/mailman/listinfo/app_rpt-users and scroll down to the bottom of the page. Enter your email address and press the "Unsubscribe or edit options button"
You do not need a password to unsubscribe, you can do it via email confirmation. If you have trouble unsubscribing, please send a message to the list detailing the problem. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.keekles.org/pipermail/app_rpt-users/attachments/20180612/600f164f/attachment.html>