Skip to content

Sending push notifications with 40k installations fails  #4242

Closed
Closed
Sending push notifications with 40k installations fails #4242

Description

@adammlevy
adammlevy
opened on Oct 8, 2017

Issue Description

I have a background job on my parse instance which fetches an XML feed every 1 minute and sends a push notification to a specific channel every time it detects new data from the feed. I have noticed that sometimes the push is sent and sometimes it is not sent. On all cases I see that the success block is called but when I look in the parse-dashboard I see the push in the sending state.

// push to channels which usually fails.  
Parse.Push.send({
		channels: [channel],
			data: { ...

Since I have no problem receiving push notifications on my dev instance (which only has 6 installations), on my prod instance I suspected this had to do with the fact that I am querying 40k installations where each channels record may contain 20 channels.
To prove the number of installations was the issue, I passed a query in the push request instead of using channels which would force the push to only be sent to my device

// Query pushes to only my device
var queryInstall = new Parse.Query(Parse.Installation);
queryInstall.equalTo('deviceType', 'ios');
queryInstall.equalTo('installationId', [MY_INSTALLATION_ID]);

Parse.Push.send({
	where: queryInstall,
	data: { ...

This query made every push a success to my device but if I remove the query and use channels instead no pushes are received. Note that although there are 40k installations, not all of them subscribe to notifications.

Steps to reproduce

Send a push to a channel observed by multiple Installation objects.

Expected Results

The push gets sent to all users.

Actual Outcome

The push notification does not get sent.

Environment Setup

  • Server

    • parse-server version : 2.6.3
    • Operating System: N/A
    • Hardware: N/A
    • Localhost or remote server? : Heroku (1 Web Dyno: 2x, 1 Worker Dyno: 2x)

  • Database

    • MongoDB version: 3.2.10
    • Storage engine: - Hardware:
    • Localhost or remote server?: mLab, Shared cluster 1GB

Logs/Trace

 verbose: REQUEST for [POST] /parse/push: {
   "where": {
     "channels": "CHPIncident-WestLA"
   },
  "data": {
     "alert": {
       "title": "CHP Incident West LA",
       "body": "West LA: traffic Collision-unknown injury  at I405 N   National Blvd"
     },
     "chpIncident": {
       "centerId": "LAHB",
       "dispatchId": "LACC",
       "logId": "171008LA01569",
       "logTime": "Oct  8 2017 10:58AM",
       "incidentId": "171008LA01569Oct  8 2017 10:58AM",
       "logType": "1183-Trfc Collision-Unkn Inj",
       "location": "I405 N / National Blvd",
       "drivingText": " traffic Collision-unknown injury at i 4 o 5 north   National boulevard ",
       "drivingTextSoCal": " traffic Collision-unknown injury at the 4 o 5 north   National boulevard ",
       "locationDesc": "NB 405 JSO NATIONAL",
       "area": "West LA",
       "broadcastMessage": "West LA: traffic Collision-unknown injury  at I405 N   National Blvd",
       "lat": 34.02685,
       "lon": -118.429908,
       "logDetails": [
         {
           "time": "Oct  8 2017 11:00AM",
           "details": "[2] POSS DUPE LG/1556 -- RP STATED DESC OF VEHS ON LG/1556 WAS DIFFERENT FRM HER TC"
         },
         {
           "time": "Oct  8 2017 10:59AM",
           "details": "[1] BLK NISS ALT, BLK Q5, UNK DESC 3RD VEH"
         }
       ],
       "logUnits": []
     },
     "expiration_time": "2017-10-08T22:02:15.039Z",
     "sound": "chp_incident.caf",
     "PushType": "CHPIncident"
   }

2017-10-08T18:02:17.681160+00:00 app[web.1]: } method=POST, url=/parse/push, host=carr.herokuapp.com, connection=close, user-agent=node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 6.11.4), accept=*/*, content-type=text/plain, x-request-id=3cd4fcd9-ef31-4039-8507-22e00ff14637, x-forwarded-for=54.205.54.184, x-forwarded-proto=https, x-forwarded-port=443, via=1.1 vegur, connect-time=1, x-request-start=1507485737673, total-route-time=0, content-length=1307, channels=CHPIncident-WestLA, title=CHP Incident West LA, body=West LA: traffic Collision-unknown injury  at I405 N   National Blvd, centerId=LAHB, dispatchId=LACC, logId=171008LA01569, logTime=Oct  8 2017 10:58AM, incidentId=171008LA01569Oct  8 2017 10:58AM, logType=1183-Trfc Collision-Unkn Inj, location=I405 N / National Blvd, drivingText= traffic Collision-unknown injury at i 4 o 5 north   National boulevard , drivingTextSoCal= traffic Collision-unknown injury at the 4 o 5 north   National boulevard , locationDesc=NB 405 JSO NATIONAL, area=West LA, broadcastMessage=West LA: traffic Collision-unknown injury  at I405 N   National Blvd, lat=34.02685, lon=-118.429908, logDetails=[time=Oct  8 2017 11:00AM, details=[2] POSS DUPE LG/1556 -- RP STATED DESC OF VEHS ON LG/1556 WAS DIFFERENT FRM HER TC, time=Oct  8 2017 10:59AM, details=[1] BLK NISS ALT, BLK Q5, UNK DESC 3RD VEH], logUnits=[], expiration_time=2017-10-08T22:02:15.039Z, sound=chp_incident.caf, PushType=CHPIncident


RESPONSE from [POST] /parse/push: {
   "headers": {
     "X-Parse-Push-Status-Id": "wtqQ4H2N5p"
   },
   "response": {
     "result": true
   }
 } X-Parse-Push-Status-Id=wtqQ4H2N5p, result=true
 verbose: _PushStatus wtqQ4H2N5p: sending push to 267 installations

Activity

  1. flovilmart commented on Oct 8, 2017

    @flovilmart
    flovilmart
    on Oct 8, 2017
    Contributor

    Can’t you provide the logs when running with VERBOSE=1? Do you have proper indexes in place?

  2. adammlevy commented on Oct 8, 2017

    @adammlevy
    adammlevy
    on Oct 8, 2017
    ContributorAuthor

    @flovilmart included log with push request and response. The only index I have on the Installation record is {"channels" : 1}

  3. flovilmart commented on Oct 8, 2017

    @flovilmart
    flovilmart
    on Oct 8, 2017
    Contributor

    There should be more logs than that, notably from the push adapter

  4. adammlevy commented on Oct 8, 2017

    @adammlevy
    adammlevy
    on Oct 8, 2017
    ContributorAuthor

    Sorry forgot to include that. For all the successful pushes I am seeing node-pre-gyp verb parse-server-push-adapter APNS APNS transmitted to [DEVICE_TOKEN] and for pushes that failed node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device [DEVICE_TOKEN]

  5. flovilmart commented on Oct 8, 2017

    @flovilmart
    flovilmart
    on Oct 8, 2017
    Contributor

    There should be the detailed error message as well, wether it’s a bad payload or a bad device token.

  6. adammlevy commented on Oct 8, 2017

    @adammlevy
    adammlevy
    on Oct 8, 2017
    ContributorAuthor

    After simplifying the payload by removing the chpIncident param (which should not be there in the first place) I am receiving more notifications on my device. I didn't have any problems sending that large of a payload when the push query only targeted my device.

  7. adammlevy commented on Oct 8, 2017

    @adammlevy
    adammlevy
    on Oct 8, 2017
    ContributorAuthor

    Ok, I just noticed this error in the log which I have never seen before:

    APNS error transmitting to device %s with error %s 60c0a4a22e0c3a4f5a472ccc90df8d1da6bc115da64e60f7276f35b200348e18 { VError: apn write failed: Not receiving Ping response after 150000 ms
2017-10-08T20:57:11.958304+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.stream.on.err (/app/node_modules/apn/lib/client.js:105:21)
2017-10-08T20:57:11.958339+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:96:13)
2017-10-08T20:57:11.958499+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.emit (events.js:188:7)
2017-10-08T20:57:11.958602+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at _connection._streamIds.forEach (/app/node_modules/apn/lib/protocol/endpoint.js:155:16)
2017-10-08T20:57:11.958720+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at Array.forEach (native)
2017-10-08T20:57:11.958890+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at EventEmitter.close [as _close] (/app/node_modules/apn/lib/protocol/endpoint.js:139:33)
2017-10-08T20:57:11.959063+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:101:20)
2017-10-08T20:57:11.959251+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at TLSSocket.emit (events.js:188:7)
2017-10-08T20:57:11.959476+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at _handle.close (net.js:497:12)
2017-10-08T20:57:11.959597+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS     at TCP.done [as _onclose] (_tls_wrap.js:332:7)
2017-10-08T20:57:11.959784+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   jse_shortmsg: 'apn write failed: Not receiving Ping response after 150000 ms',
2017-10-08T20:57:11.959975+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   jse_info: {},
2017-10-08T20:57:11.960136+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   message: 'apn write failed: Not receiving Ping response after 150000 ms',
2017-10-08T20:57:11.960327+00:00 app[web.2]: node-pre-gyp ERR! parse-server-push-adapter APNS   stack: 'VError: apn write failed: Not receiving Ping response after 150000 ms\n    at Stream.stream.on.err (/app/node_modules/apn/lib/client.js:105:21)\n    at emitOne (events.js:96:13)\n    at Stream.emit (events.js:188:7)\n    at _connection._streamIds.forEach (/app/node_modules/apn/lib/protocol/endpoint.js:155:16)\n    at Array.forEach (native)\n    at EventEmitter.close [as _close] (/app/node_modules/apn/lib/protocol/endpoint.js:139:33)\n    at emitOne (events.js:101:20)\n    at TLSSocket.emit (events.js:188:7)\n    at _handle.close (net.js:497:12)\n    at TCP.done [as _onclose] (_tls_wrap.js:332:7)' }

  8. flovilmart commented on Oct 8, 2017

    @flovilmart
    flovilmart
    on Oct 8, 2017
    Contributor

    Just to rewind a bit, sending a simple payload to a channels target works correctly?

  9. adammlevy commented on Oct 9, 2017

    @adammlevy
    adammlevy
    on Oct 9, 2017
    ContributorAuthor

    Yes. Now that I have simplified the payload I can see more pushes coming in and the push adapter always logs whether it was a successful push or a failed push. While I was sending the large payload I would never see the push adapter log so I wonder if it was just silently failing?

    Regarding the issue above, I think I found the solution in issue #90

  10. flovilmart commented on Oct 9, 2017

    @flovilmart
    flovilmart
    on Oct 9, 2017
    Contributor

    Perhaps APNS was not happy with those large payloads, or the node apn module don’t support those properly.

  11. flovilmart
    closed this as completedon Oct 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions