Our api.nottinghamcollege.ac.uk app, which provides various applications and digital signage data and layouts, began experiencing increased gateway timeouts and errors from around 3:30 PM on Tuesday 13th October.
This was identified as being related to a specific third-party integration and data source that was (and still is) experiencing issues. This led to increased errors and requests hanging for an extended period of time from our API app, which impacted performance when this service was being called. We experienced a degraded application state when these requests we being made, which lasted until around Wednesday 14th October 10:30 PM, when the root cause of the issue was found.
Because of the issue with the upstream data source, all services using this have been turned off until further notice. Some additional measures have been implemented to limit the impact of this going forward, but due to ongoing issues with the third-party this service remains off.
The service is Nottingham Travelwise, which provides live camera feeds of traffic cameras around Nottingham City. We use this to show current traffic levels around our campuses for help during travel. Due to ongoing issues with this service, camera images being requested are not returning live images and are experiencing long timeouts for such requests. Our API app was waiting an extended amount of time for these requests, leading to the issues described.
We are reviewing any potential areas with other integrations/third-parties which may have risks of extended query/wait times to prevent future issues of degrading our overall application/service.
At this time due to the issues with Nottingham Travelwise, all digital signage layouts using live camera feeds have been disabled until further notice.