tag:status.tuotempo.com,2005:/historytuotempo srl Status - Incident History2024-03-27T13:21:35+01:00tuotempo srltag:status.tuotempo.com,2005:Incident/168342042023-04-11T14:00:00+02:002023-04-12T11:49:18+02:00Degraded Performance due to a problem on Loab Balancer system (14:08 - 14:15)<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>14:00</var> CEST</small><br><strong>Resolved</strong> - This afternoon, we had an intermittent hardware issue on our loabalancer service. Our HA system correctly detected the issue and switched automatically to our backup loadbalancer the incoming traffic. We apologize for the problems.</p>tag:status.tuotempo.com,2005:Incident/74157092021-07-05T10:00:00+02:002021-07-05T16:08:06+02:00Degraded Performance due to a problem on cache system - ( 10:03 - 10:33 CET ; 11:44 - 12:40 CET )<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>Resolved</strong> - This morning, we had an issue on the caching system that have been identified and solved. We apologize for the problem.</p>tag:status.tuotempo.com,2005:Incident/69654052021-05-10T04:30:00+02:002021-05-10T16:54:00+02:00Degraded Performance due to DB node problems - ( 16:01 - 16:14 CET )<p><small>May <var data-var='date'>10</var>, <var data-var='time'>04:30</var> CEST</small><br><strong>Resolved</strong> - A DB node responded to incoming calls with degraded performance caused by network problems. The node has been terminated and replaced with a new one. The users relying on the cluster where DB node was present experienced delays.<br />We apologize for the incident</p>tag:status.tuotempo.com,2005:Incident/68422432021-04-26T08:30:00+02:002021-04-26T09:47:07+02:00Degraded performance on Application ( 08:16 - 08:26 CET )<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>08:30</var> CEST</small><br><strong>Resolved</strong> - Some customers experienced slow access or timeout on API service and backoffice system, caused by a database issue.<br />We found the source of issue and fixed it.<br />We apologize for the incident</p>tag:status.tuotempo.com,2005:Incident/67941132021-04-18T15:30:00+02:002021-04-20T13:10:59+02:00Connectivity problems on Main API endpoint<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>15:30</var> CEST</small><br><strong>Resolved</strong> - At 16/04/2021 11:56-11:58 CET and 18/04/2021 15:23-15:25 CET <br />AWS DNS service had problems, that caused us connectivity problems on our API service. <br />Some customers experienced timeouts responses on our platform in this time.<br />All issues was resolved and services going well</p>tag:status.tuotempo.com,2005:Incident/64984352021-03-09T10:00:00+01:002021-03-10T17:40:48+01:00API Errors - (09:46 - 09:48)<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - Some customers experienced issues with the API service where they received Forbidden responses.<br />We discovered the source of the problem inside our platform and fixed it.</p>tag:status.tuotempo.com,2005:Incident/63867292021-02-23T04:00:00+01:002021-02-24T12:40:37+01:00Degraded performance on API Service ( 3:35 - 03:55 CET )<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>04:00</var> CET</small><br><strong>Resolved</strong> - Some customers experienced slow access or timeout on API service and backoffice system.<br />We discovered and released a fix that resolve this issue caused by slowness of our database layer.</p>tag:status.tuotempo.com,2005:Incident/63416572021-02-20T08:30:00+01:002021-02-20T10:03:04+01:00Degraded performance on Application ( 08:30 - 09:40 CET )<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>08:30</var> CET</small><br><strong>Resolved</strong> - Some customers experienced slow access or timeout on API service and backoffice system, caused by scaling issues.<br />We found the source of issue and fixed it.<br />We apologize for the incident</p>tag:status.tuotempo.com,2005:Incident/36561252020-03-07T04:30:08+01:002020-03-07T04:30:08+01:00Degraded performance on Application ( 03:30 - 04:00 CET )<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>04:30</var> CET</small><br><strong>Resolved</strong> - AWS gateway outbound network is stable now and issue completely fixed</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>04:21</var> CET</small><br><strong>Monitoring</strong> - We fixed network problem with AWS gateway and monitoring situation</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>04:20</var> CET</small><br><strong>Identified</strong> - Some customers experienced slow access or timeout on API service and backoffice system.<br />We identified the issue caused to a AWS networks problem by outbound Gateway IP</p>tag:status.tuotempo.com,2005:Incident/36529002020-03-05T20:00:00+01:002020-03-06T12:03:23+01:00Degraded performance on API Service ( 18:20 - 19:40 CET )<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>20:00</var> CET</small><br><strong>Resolved</strong> - Some customers experienced slow access or timeout on API service and backoffice system.<br />We discovered and released a fix that resolve this issue caused by a wrong behavior of login api.</p>tag:status.tuotempo.com,2005:Incident/35667462020-02-14T12:30:00+01:002020-02-14T15:28:33+01:00Degraded performance on application ( 11:50 - 12:20 CET )<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:30</var> CET</small><br><strong>Resolved</strong> - Some customers experienced slow access or timeout on API service and backoffice system.<br />We discovered and resolved the issue caused by hardware problems of a node in one of our clusters.</p>tag:status.tuotempo.com,2005:Incident/29013012019-08-22T13:30:08+02:002019-08-22T17:22:18+02:00Degraded performance on application ( 12:30 - 13:40 CET )<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>13:30</var> CEST</small><br><strong>Resolved</strong> - The access to the tuotempo backoffice was slow in the above period of time due to a caching problem. The cause has been identified and a patch was applied to reduce system overload.</p>tag:status.tuotempo.com,2005:Incident/28376812019-08-08T22:45:56+02:002019-08-09T11:35:42+02:00API Errors (21:00 - 22:45 CET)<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>22:45</var> CEST</small><br><strong>Resolved</strong> - Some customers experienced issues with the API service.
<br />We discovered the source of the problem and fixed it.</p>tag:status.tuotempo.com,2005:Incident/27606102019-07-24T21:55:46+02:002019-07-25T10:59:55+02:00Strong delay due to Autoscaling Problems ( 20:15 - 21:55 CET )<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>21:55</var> CEST</small><br><strong>Resolved</strong> - We discovered a bug that in certain circumstances caused a not correctly Autoscaling of our API stack.</p>tag:status.tuotempo.com,2005:Incident/25331722019-06-11T14:50:21+02:002019-06-12T14:52:25+02:00Degraded Performance due to DB node problems - ( 14:50 - 15:18 CET )<p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>14:50</var> CEST</small><br><strong>Resolved</strong> - A DB node responded to incoming calls with degraded performance caused by filesystem problems. The node has been terminated and replaced with a new one. The users relying on the cluster where DB node was present experienced delays.</p>tag:status.tuotempo.com,2005:Incident/19073222018-09-14T13:36:56+02:002018-09-14T13:36:56+02:00Backoffice Login is not working<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:36</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:16</var> CEST</small><br><strong>Update</strong> - we have now fixed the issue</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>10:15</var> CEST</small><br><strong>Identified</strong> - THe access to the tuotempo backoffice has been broken in the last hours due to a caching problem. Now it has been resolved by a complete recaching of the admin credentials. The access could be reenabled though through the password reset (forgot password)</p>tag:status.tuotempo.com,2005:Incident/16373472018-03-12T10:37:00+01:002018-03-12T10:38:31+01:00Strong delays in platform response<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>10:37</var> CET</small><br><strong>Resolved</strong> - The Caching platform has been restored, and we are currently evaluating to build an increased High Availability by using an automatic failover system with a clusterized solution</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>10:05</var> CET</small><br><strong>Monitoring</strong> - The caching platform has been restored, we are monitoring the recovering of all services</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>10:04</var> CET</small><br><strong>Identified</strong> - The current problem is due to a caching platform OUTAGE. We are currently restoring that</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:49</var> CET</small><br><strong>Investigating</strong> - We are currently experiencing big delays in server response, we are currently investigating.</p>tag:status.tuotempo.com,2005:Incident/15581372018-01-08T13:23:21+01:002018-01-08T13:23:21+01:00Connectivity problems on Main API endpoint<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>13:23</var> CET</small><br><strong>Resolved</strong> - Caching database has been fully restored. Response times are Progressively returning to normal levels.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>12:58</var> CET</small><br><strong>Monitoring</strong> - We are seeing response times already normalising, the service is restoring progressively.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>12:44</var> CET</small><br><strong>Identified</strong> - The problem has been identified in a deterioration of cached responses database. we are currently rebuilding it. it is expected to have completed the recovering in about 20 minutes from now</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>12:25</var> CET</small><br><strong>Investigating</strong> - We are expriencing high delays on the main API endpoint. We are currently investigating but simulttaneously increasing both database and computational resources</p>tag:status.tuotempo.com,2005:Incident/10027842016-11-29T09:00:45+01:002016-11-29T09:00:45+01:00Major outage on database layer 7:00 to 7:22 GMT<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>09:00</var> CET</small><br><strong>Resolved</strong> - The service has gone down from 7:00 to 7:22 am (Greenwich Mean Time) due to a deadlock on one of the db clusters. The db service has been restarted and service is now back to normal. We are currently investigating on the detailed issue to prevent it and to develop possible auto-recovery measures.</p>tag:status.tuotempo.com,2005:Incident/5270222016-06-10T08:14:59+02:002016-06-10T08:14:59+02:00Network seems to be unreachable from certain world areas<p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>08:14</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>08:04</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>07:48</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.tuotempo.com,2005:Incident/3988252016-01-18T12:54:02+01:002016-01-18T12:54:02+01:00Database layer connectivity deteriorated<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>12:54</var> CET</small><br><strong>Resolved</strong> - Today we experimented a difficulty in the communications between Database nodes and loadbalancer.
<br />the problem has been resolved in 10 minutes</p>tag:status.tuotempo.com,2005:Incident/2700542015-06-24T16:44:30+02:002015-06-24T16:44:30+02:00Network seems to be unreachable from certain world areas<p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:44</var> CEST</small><br><strong>Resolved</strong> - Incident Has been completely restored. No data loss no inconsistency to report.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:36</var> CEST</small><br><strong>Monitoring</strong> - Cluster Restored, monitoring the situation</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:33</var> CEST</small><br><strong>Update</strong> - Still working on database cluster's nodes recovery</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:20</var> CEST</small><br><strong>Update</strong> - We have currently identified the problem on certain databse cluster nodes which are hosting some of our clients' accounts. still acting to restore.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:15</var> CEST</small><br><strong>Update</strong> - Some of the Accounts are unreachable at the moment.
<br />Still recovering</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>16:11</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.tuotempo.com,2005:Incident/2695922015-06-23T18:24:23+02:002015-06-23T18:24:23+02:00Degraded Performance due to DB node - 21 min (Jun 23 - 14:28 - 14:49) CET<p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>18:24</var> CEST</small><br><strong>Resolved</strong> - A new DB node added to manage high workload responded to incoming calls with degraded performance. The node has been terminated and replaced with a new one. The users relying on the slow DB node experienced delays.</p>tag:status.tuotempo.com,2005:Incident/2048882015-04-22T14:20:26+02:002015-04-22T14:20:26+02:00Service performance degradation: 14:03 - 14:10<p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>14:20</var> CEST</small><br><strong>Resolved</strong> - One of the frontend loadbalancers hanged. Autorecovery systems delayed in detecting it.
<br />We are working on refining the detection system.</p>tag:status.tuotempo.com,2005:Incident/1717052015-02-12T12:10:28+01:002015-02-12T12:10:28+01:00Degraded Performance due to Network problem - 26 min (Feb 11 - 18:00 - 18:26) CET<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:10</var> CET</small><br><strong>Resolved</strong> - Most of our clients experienced longer waiting time due to a new network problem. The problem has been resolved redirecting traffic to a different network. Since it was the second time in just 2 days, actions has been taken to automatically redirect in case of similar issues for reducing impact.</p>