tag:densityinc.statuspage.io,2005:/historyDensity Status - Incident History2024-03-28T14:40:33-04:00Densitytag:densityinc.statuspage.io,2005:Incident/198950862024-02-05T04:00:20-05:002024-02-09T11:03:40-05:00Maps Hosting Site Outage<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>04:00</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>12:43</var> EST</small><br><strong>Identified</strong> - We're having an outage with our hosting site. The team is aware and is working to resolve it.</p>tag:densityinc.statuspage.io,2005:Incident/194901652023-12-20T00:00:44-05:002023-12-20T10:46:39-05:00Webhooks Outage<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>00:00</var> EST</small><br><strong>Resolved</strong> - From December 12-20, webhooks were unavailable. This issue has now been resolved, and you do not need to do anything to resume receiving data.<br /><br />If you use webhooks to feed a data lake or other record of count data, you will need to backfill data from December 12-20, 2023. We recommend using the API query space events endpoint. If you need help downloading a CSV from the time frame, please let us know, and we can provide instructions.</p>tag:densityinc.statuspage.io,2005:Incident/194150982023-12-12T17:17:49-05:002023-12-12T17:17:49-05:00Entry Live Errors<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:17</var> EST</small><br><strong>Resolved</strong> - When we cut over the live Entry data to our improved count database, the system ran into unexpected errors. This caused a backup in Entry count, which resulted in incorrect live count for about 20% of spaces. This issue was resolved at 4:30 p.m. EST.<br /><br />The only impact to historical data and APIs was a slight delay in arrival, resolved at 3:45 p.m. EST. There is no incorrect data populating in either location.<br /><br />The legacy Dashboard had a bug with timestamps. If you use the legacy platform for building-level data, this is now available. <br /><br /> v2 APIs had a bug with representing timestamps on the response. If you ran into timestamp parsing errors on your end, you may have to re-pull data.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>11:34</var> EST</small><br><strong>Update</strong> - The Entry count backlog is currently clearing from the earlier outage. Until this is resolved, live data in Displays will be incorrect, and you may experience jumps in numbers. <br /><br />All v2 and v3 APIs are operating normally. If you use APIs to feed a data lake or other record of count data, and received API errors between 4-4:30 a.m. or 7-9 a.m., you will need to backfill your data. Not all customers received error messages, please check your API instance.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>09:58</var> EST</small><br><strong>Identified</strong> - Density performed scheduled maintenance this morning at 3 a.m. EST for v2 API changes and fixes to Entry Live (displays).<br /><br />-Between 4-4:30 a.m. EST there was a full outage across the platform due to the v2 cutover. This was resolved at 4:30 a.m. EST.<br /><br />-Between 7-9 a.m. EST a limited number of customers may have experienced v2 API errors. This was resolved at 9 a.m. EST.<br /><br />-At 5:30 a.m. EST, we began to experience issues with Entry data, related to the fixes for Entry Live. This is ongoing; updates to follow.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>09:28</var> EST</small><br><strong>Investigating</strong> - We are currently investigating a large number of errors in Entry data.</p>tag:densityinc.statuspage.io,2005:Incident/193553772023-12-06T13:18:34-05:002023-12-06T13:18:52-05:00Entry Counts Outage<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>13:18</var> EST</small><br><strong>Resolved</strong> - We faced an hour-long outage for our Entry count data into Atlas and APIs due to an AWS issue. This has been resolved. <br /><br />If you use our v3 API to feed a data lake or other record of count data, you will need to backfill data from 12:09-1:04 EST. We recommend refreshing the data for that time frame using the same endpoints.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>12:00</var> EST</small><br><strong>Investigating</strong> - Our Entry counts API is down, which is impacting Atlas and APIs. We will update as we uncover the cause.</p>tag:densityinc.statuspage.io,2005:Incident/191554182023-11-20T10:18:56-05:002023-11-20T10:21:13-05:00Webhooks Backup<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>10:18</var> EST</small><br><strong>Resolved</strong> - From November 14-20, webhooks were unavailable. This issue was resolved on November 20 at 5 a.m. EST. You do not need to do anything to resume receiving data. We have also put monitoring in place to prevent this from happening again.<br /><br />If you use webhooks to feed a data lake or other record of count data, you will need to backfill data from November 14-20. We recommend using the API query space events endpoint. If you need help downloading a CSV from the time frame, please let us know, and we can provide instructions.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>00:00</var> EST</small><br><strong>Identified</strong> - We have identified a backlog in our webhooks, which is in the process of being cleared.</p>tag:densityinc.statuspage.io,2005:Incident/187688472023-10-14T11:01:12-04:002023-10-14T11:01:12-04:00Live Entry Count Down<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>11:01</var> EDT</small><br><strong>Resolved</strong> - All systems have been restored and are operating normally.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>17:40</var> EDT</small><br><strong>Identified</strong> - Backend systems will be running overnight to restore normal functionality. All Entry data is available, but may experience delays or intermittent issues during this time.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:51</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing downtime with our Entry live availability features, including Displays. There will also be a delay in Entry historical data until the issue is resolved. The cause has been identified, and we are working to resolve the issue.</p>tag:densityinc.statuspage.io,2005:Incident/187659012023-10-12T12:54:49-04:002023-10-12T12:54:49-04:00Entry Count Pipeline is Down<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>12:54</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved. If you use the v2 API to feed a data lake or other record of count data, you will need to backfill the data during this outage. Please let us know if you need any assistance in this.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>10:23</var> EDT</small><br><strong>Monitoring</strong> - The Entry count system is down, impacting Dashboard, Displays and v2 count APIs. A fix has been implemented, and we are monitoring.</p>tag:densityinc.statuspage.io,2005:Incident/79933412021-09-14T14:18:01-04:002021-09-14T14:18:01-04:00Degraded Analytics API<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>14:18</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>14:08</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:37</var> EDT</small><br><strong>Identified</strong> - Our database provider is currently experiencing an outage, and our analytics API has been affected. No data has been lost, however, space analytics are currently unavailable.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>12:30</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:densityinc.statuspage.io,2005:Incident/80380332021-09-09T13:00:00-04:002021-09-20T13:08:38-04:00Data Integrity Issue<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>13:00</var> EDT</small><br><strong>Resolved</strong> - During a period from 9/5 to 9/9, a select number of API analytics endpoints did not return results for count data. Events (entrances / exits) were still available. No data was lost during this period. Live data (webhooks/websockets) was unaffected.</p>tag:densityinc.statuspage.io,2005:Incident/75562632021-07-20T10:20:44-04:002021-07-20T10:20:44-04:00Counts API Unavailable<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>10:20</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>10:08</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>10:08</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:55</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:26</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:24</var> EDT</small><br><strong>Investigating</strong> - We are seeing degraded performance retrieving data from the Counts API.</p>tag:densityinc.statuspage.io,2005:Incident/67964382021-04-20T16:19:24-04:002021-04-20T16:19:24-04:00Application Authentication Unavailable<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>16:19</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>16:05</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:05</var> EDT</small><br><strong>Update</strong> - We have enabled basic auth for Portfolio. At this time, we are awaiting the Auth0 incident to resolve to re-enable SSO. Please reach out to support@density.io if you need help.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>13:03</var> EDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue. Dashboard and Display have basic auth enabled. Please contact support@density.io if you would like to temporarily disable SSO.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:08</var> EDT</small><br><strong>Identified</strong> - We have re-enabled basic authentication (email/password) for customers of Dashboard. Portfolio and Display will be available via basic auth soon. <br /><br />SSO is currently disabled. We will post an update as soon as we have SSO re-enabled.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:55</var> EDT</small><br><strong>Update</strong> - Auth0 incident available here https://status.auth0.com/incidents/zvjzyc7912g5</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:53</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing an issue with a 3rd party provider (Auth0) that provides authentication to the Density Dashboard. We will sharing more updates shortly.</p>tag:densityinc.statuspage.io,2005:Incident/64144422021-02-27T09:30:00-05:002021-02-27T10:08:05-05:00AWS RDS Minor Version Upgrade<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>09:30</var> EST</small><br><strong>Resolved</strong> - During an automated minor version upgrade of our database, a set of configuration parameters became invalid. The parameters were updated to be consistent with a new localization spec for Postgres.</p>tag:densityinc.statuspage.io,2005:Incident/42115072020-05-26T20:00:00-04:002020-05-27T17:47:32-04:00Dashboard Reports Unavailable<p><small>May <var data-var='date'>26</var>, <var data-var='time'>20:00</var> EDT</small><br><strong>Resolved</strong> - Reports on the Dashboard were unavailable from May 27 12:00am UTC until May 27 12:30am UTC. The team identified the issue, and pushed a fix to production. No data was lost during this period.</p>tag:densityinc.statuspage.io,2005:Incident/36931872020-03-16T07:00:00-04:002020-03-16T09:40:59-04:00API downtime<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>07:00</var> EDT</small><br><strong>Resolved</strong> - The Density API became unavailable at 11am UTC. The API is back to a stable state as of 1:40pm UTC. There was no loss of data.</p>tag:densityinc.statuspage.io,2005:Incident/35815342020-02-18T13:39:03-05:002020-02-18T13:39:03-05:00API Downtime<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:39</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:37</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:34</var> EST</small><br><strong>Monitoring</strong> - We have identified the issue. We are currently monitoring the fix. Systems should be available shortly.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:05</var> EST</small><br><strong>Investigating</strong> - We are currently investigating an issue related to a database migration. The API and Dashboard are currently affected. We are still able to receive events from Density sensors and our websocket + webhook servers are operational.</p>