Restart completed. Again, note that some upstream connections may take a while to be re-established due to rate limits. (12:05 UTC — Apr 3)Planned bouncer upgrade is scheduled for Thursday, April 3rd. On Thursday at 12:00 UTC we will be rolling out an upstream update for soju, the software that powers the chat.sr.ht bouncer. The bouncer will be restarted, causing everyone to be disconnected. Most clients will reconnect automatically. Note that some upstream connections may take a while to b...| Issues on sr.ht status
Build queues are full and wait times are expected. Due to a minor issue with our build servers, build queues were unprocessed for a couple of hours this morning. The system is catching up with pending builds now, but wait times may be higher than normal for the next hour or so. (08:00 UTC — May 27)| Issues on sr.ht status
User registrations are open. We are now accepting new users once again. (12:00 UTC — Feb 18) User registrations are currently disabled. We are not accepting new user accounts until we are able to deploy more effective mitigations with respect to an ongoing spam attack. (20:00 UTC — Feb 17)| Issues on sr.ht status
Planned maintenance is complete. Full service is restored. (14:03 UTC — Jan 23) Planned maintenance is underway. (14:00 UTC — Jan 23) We are planning a brief interruption in service for pages.sr.ht. At 14:00 UTC we intend to place pages.sr.ht into a read-only mode to complete the last steps for its migration following the incident earlier this month. We anticipate no more than 30 minutes of downtime. (14:00 UTC — Jan 23)| Issues on sr.ht status
Planned maintenance is complete. Normal service is restored. Thank you for your patience. (16:00 UTC — Mar 22) Maintenance is underway The maintenance window is now open. (14:00 UTC — Mar 22) A planned outage is scheduled for March 22th. At 14:00 UTC, we plan to perform system upgrades across our fleet. This will cause intermittent outages which are expected to last 15-20 minutes each, over the course of 3 to 4 hours in total.| Issues on sr.ht status
Service restored. We have migrated the database to a temporary host and brought our services back online. We had to use one of our build servers to host it for the time being, so our build system is operating at half its normal capacity – you may notice longer-than-usual build times while we roll out a permanent solution. Thank you for your patience. (17:00 UTC — Jan 11) Investigating. We have a technician on-site and are diagnosing the issue further. (15:47 UTC — Jan 11)| Issues on sr.ht status
Service has been restored. Webhooks are now being processed and the backlog has been completed. (00:00 UTC — Oct 27) Service has mostly been restored. We are still running some migrations which must complete before we re-enable webhooks, but most services are functional. (13:00 UTC — Oct 26) Maintenance is underway. The maintenance window is now open. (09:00 UTC — Oct 26) This maintenance window has been rescheduled. It will now take place on Wednesday, October 26th, at 09:00 UTC.| Issues on sr.ht status
A brief issue with lists.sr.ht caused a delay in mail delivery: A problem with a recent lists.sr.ht deployment went unnoticed for about an hour and a half, during which time incoming emails were not processed or forwarded. Service has been restored and all of the missed emails are being automatically processed now. No emails have been dropped; it is not necessary to send them again. (14:48 UTC — Oct 20)| Issues on sr.ht status
A distributed denial of service (DDoS) attack on todo.sr.ht is underway: A denial of service attack against todo.sr.ht is currently underway. We suspect it is motivated by hate sentiments against a transgender maintainer on SourceHut. The attack is presently under control, but please let us know via IRC if you experience any problems. (11:00 UTC — Jul 23)| Issues on sr.ht status
Planned maintenance is complete. Normal service is restored. Thank you for your patience. (16:30 UTC — Feb 28) Planned maintenance is underway. All SourceHut services will be affected. Barring unforseen circumstances, the maintenance window is scheduled for two hours. (15:00 UTC — Feb 28) Phase three of our Alpine Linux 3.15 roll-out will cause major service disruptions. We will be rolling out Alpine Linux 3.15 to more of our hosts on Monday the 28th at 15:00 UTC.| Issues on sr.ht status
Maintenance complete. We have completed all maintenance operations planned for today and service is restored to normal. (17:15 UTC — Feb 21) Planned maintenance may exceed the scheduled maintenance window. We are still working on the roll-out and now expect to complete the migration before 18:00 UTC. (17:00 UTC — Feb 21) Planned maintenance is underway. Expect intermittent outages for all services. (15:00 UTC — Feb 21) Phase two of our Alpine Linux 3.15 roll-out will cause major service...| Issues on sr.ht status
Today’s scheduled maintenance is complete. We have completed our maintenance work for today, and all services are restored to normal operation. Thank you for your patience. (10:45 UTC — Feb 14) The migration is underway. We have started this migration work and expect to complete it no later than 12:00 UTC. (10:00 UTC — Feb 14) Phase one of our Alpine Linux 3.15 roll-out will cause minor service disruptions. We will be rolling out Alpine Linux 3.15 to some of hosts on Monday the 14th at ...| Issues on sr.ht status
An incomplete deployment caused a brief pages.sr.ht outage: pages.sr.ht suffered 13 minutes of downtime following an incomplete deployment which failed to apply a database migration correctly. (10:46 UTC — Feb 3)| Issues on sr.ht status
The issue has been resolved: The upstream network has corrected the problem. Normal service should be restored. Over the course of 19 minutes, we dropped 10 of our ~500 upstream chat.sr.ht connections. We are still experiencing internal issues, but they are not service impacting and we expect them to clear up shortly. (20:22 UTC — Dec 20) An ongoing issue with our upstream network provider is causing chat.sr.ht reliability problems: An issue with our network provider is causing some unrelia...| Issues on sr.ht status
A deployment issue caused a brief outage on meta.sr.ht: An updated for one of our dependencies was not correctly verified before being deployed to production, resulting in a 6 minute outage. Only meta.sr.ht was affected; other services remained available for the duration of the outage. (08:13 UTC — Nov 16)| Issues on sr.ht status
HTTP(s) clone access has been restored: The issue was narrowed down to a stale nginx configuration which directed authorization requests to the wrong port. The configuration has been corrected and normal service is restored. (08:35 UTC — Oct 27) HTTP(s) clone is currently unavailable for hg.sr.ht: A planned deployment ran afoul and has caused some problems with cloning hg.sr.ht repositories via https. An investigation is ongoing. SSH clone is still working correctly. (08:20 UTC — Oct 27)| Issues on sr.ht status
Planned maintenance is complete: We have finished upgrading all of the affected hosts and all services are restored to normal. (14:20 UTC — Aug 16) Planned maintenance is ongoing: Our maintenance window has opened and we have started our work. (13:00 UTC — Aug 16) Planned maintenance on August 16th will cause intermittent outages: We are planning the second (and last phase) of the maintenance which began on August 3rd. This will affect all services, causing intermittent outages that are e...| Issues on sr.ht status
Maintenance complete: The issue with pages.sr.ht has been resolved and all services are now available. (10:30 UTC — Aug 3) Maintenance mostly complete but pages.sr.ht still pending: We have completed maintenance on all services, which can now be expected to be stable, with the exception of pages.sr.ht. We have encountered an issue during the pages.sr.ht upgrades and are addressing it now. (10:00 UTC — Aug 3) Planned maintenance on August 3rd will cause intermittent outages: Planned mainte...| Issues on sr.ht status
Snapshot growth caused an outage on git.sr.ht git.sr.ht’s ZFS snapshots grew to consume all available disk space. This is normally an understood pathology of the server configuration, but due to a change in billing with Twilio, our paging script did not alert the operators to the imminent issue. It seems that there is not a grace period with Twilio; they reported the billing issue to us only yesterday. The issue with git.sr.ht has been resolved, the bill has been paid, and we are researchin...| Issues on sr.ht status
Planned maintenance has been completed.(16:30 UTC — Feb 8) Planned maintenance is now underway. (15:00 UTC — Feb 8) Planned maintenance on February 8th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. (15:00 UTC — Feb 8)| Issues on sr.ht status
One of our third-party DNSBL services, SpamCop, allowed their domain to expire, presumably as a mistake, and began to return “listed” for all DNSBL checks. We use a DNSBL as an early rejection for spam emails, and this caused 21 incoming emails to be wrongfully rejected. We have removed SpamCop from our list of DNSBLs and filed a ticket to improve our monitoring so that we may catch this sooner. Incoming emails are working now, but be advised that if your postmaster uses SpamCop, emails f...| Issues on sr.ht status
An issue with mail delivery caused some emails to be dropped. Upgrades to our mail server caused authentication to unexpectedly and silently fail, and it took some time to detect. During this period, ticket submission to todo.sr.ht failed, and emails were not forwarded from lists.sr.ht. You can thank Oracle for changing the license terms on Berkely DB, which had wider reaching consequences than we expected. (19:00 UTC — Jan 21)| Issues on sr.ht status
An issue with our upstream ISP caused an outage. Our ISP decided to ring in the new year by unplugging us for 12 minutes last night. Sorry about that. (09:03 UTC — Jan 8)| Issues on sr.ht status
The issued was addressed without disruption. While investigating the cause of disk growth, an error with our ZFS snapshot retention policy was found to be the cause. The disk space was trivially reclaimed without incurring an outage. (15:50 UTC — Oct 8) Advances in the pace of disk usage on git.sr.ht has increased the urgency of a planned migration. On September 18th, we were tracking the growth of git.sr.ht disk space usage up to an expected disk space exhaustion in November. However, in t...| Issues on sr.ht status
Maintenance complete. Depending on how soon your DNS server picks up the updates, you should find service restored shortly. (21:00 UTC — Jul 29) We’re still waiting on the latest data to transfer from the old server to the new. It is taking longer than we expected. The process is about halfway complete. We apologise for the delay. (19:00 UTC — Jul 29) Maintenance may require more time than expected. (17:50 UTC — Jul 29)| Issues on sr.ht status
An unexpected Redis failure caused a partial outage of git.sr.ht The git.sr.ht web service was partially interrupted today due to unexpected Redis errors. Some pages which rely on Redis for caching returned 500 errors, namely the summary and tree pages. The issue has been resolved. To prevent its recurrence, we have filed a ticket to make our caching system more tolerant of Redis outages, and added an alarm to detect this kind of issue earlier. (12:07 UTC — Jun 28)| Issues on sr.ht status
Planned maintenance is complete.(19:07 UTC — Jun 15) Planned maintenance is underway.(18:00 UTC — Jun 15) Planned maintenance on June 15th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. (18:00 UTC — Jun 15)| Issues on sr.ht status
Planned maintenance is complete. There may be some residual issues. (18:00 UTC — Mar 11) Planned maintenance is underway(16:00 UTC — Mar 11) Planned maintenance on March 11th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes. (19:00 UTC — Mar 5)| Issues on sr.ht status
Maintenance complete Service has been restored. (16:30 UTC — Jan 8) Maintenance has begun: This maintenance is now underway. (16:00 UTC — Jan 8) Planned maintenance on January 8th will cause a total service outage: We have scheduled maintenance for next Wednesday at 16:00 UTC to perform normal database maintenance. We expect a total outage of all services which should last no more than 2 hours. (16:00 UTC — Jan 8)| Issues on sr.ht status
A planned upgrade hit unexpected errors: We are working on a solution. (19:00 UTC — Dec 22)| Issues on sr.ht status
Migration complete pending DNS propegation: The migration is completed and all service is restored. However, you may not see service restored until your DNS provider finishes processing the updates. (17:45 UTC — Dec 20) git.sr.ht is now read-only: We’re in the thick of the migration now. Write access should be restored within an hour. (17:30 UTC — Dec 20) Planned upgrades to git.sr.ht will impact availability: We are planning on moving git.sr.ht to a new server today. The service may be...| Issues on sr.ht status
Deployments complete: The deployment has been completed for all services and no issues have been found. Service should be operating normally. (15:45 UTC — Nov 30) Ongoing deployment causing service disruption: We are currently deploying a major change in the Sourcehut architecture, which is likely to cause intermittent outages during deployment as services are rotated onto the new system and oversights are discovered and resolved. (15:00 UTC — Nov 30)| Issues on sr.ht status
Service restored: Service has been restored. I/O exhaustion flipped git.sr.ht’s filesystem into read-only mode and on reboot, filesystem corruption was found. The integrity of user data was not affected, and if they were, the integrity of our backups was found to be correct in an emergency integrity test (routine checks are also performed regularly and found to be in order). An investigation into the underlying causes of the failure is ongoing. (20:08 UTC — Nov 15)| Issues on sr.ht status
Resolved The issue has been fixed. It may take time for DNS updates to propegate to your local server. (21:45 UTC — Sep 29) Working on the issue We have reached support and they’ve rolled back the changes. We’re monitoring our services to make sure the trasition occurs smoothly. (21:00 UTC — Sep 29) No response We are still waiting for an update from our registrar. (20:00 UTC — Sep 29) Major DNS outage: While deploying IPv6 support across Sourcehut, we submitted a routine request to...| Issues on sr.ht status
Maintenance complete: The planned maintenance is complete. Write access should become available once your DNS cache is flushed. The TTL is 1 hour. (15:34 UTC — Jun 12) Planned maintenance: git.sr.ht is currently read-only as we complete planned maintenance on the service. (15:00 UTC — Jun 12)| Issues on sr.ht status
Upcoming planned maintenance: We have an upcoming planned database outage on May 10th. This will cause a temporary service disruption for all sr.ht services. The outage is expected to last less than 2 hours. (12:00 UTC — May 10)| Issues on sr.ht status
Maintenance complete: Everything appears to be in good working order. (20:00 UTC — Feb 18) Upcoming planned maintenance: We’re doing planned maintenance on Monday that will cause sporadic outages throughout the day as our hosts are moved into a larger rack. In exchange for your patience, we’ll be installing a much more powerful host for builds.sr.ht builds at the same time. (12:00 UTC — Feb 18)| Issues on sr.ht status
Full service restored DNS udpates should be well and propegated by now, and the host is confirmed to be in working order. Apologies for the disruption. (16:42 UTC — Feb 13) Rolling back migration There are other problems with the new host - DNS is being rolled back to the old host and another attempt to migrate will be taken at a later date. If you updated your /etc/hosts entry, please reset it to the old host: 45.56.100.217.| Issues on sr.ht status
Maintenance complete: We have completed this period of maintenance, sr.ht has been restored to full service. (03:00 UTC — Dec 29) Partial outage during planned maintenance: A planned service upgrade requires logins to be intermittedly for some sr.ht services. Users who are already logged in should not experience a disruption, though minor issues here and there may occur. (01:00 UTC — Dec 29)| Issues on sr.ht status
Debian & FreeBSD builds now available: Full service has been restored. (03:54 UTC — Nov 27) Arch Linux builds now available: Arch is available and Debian is coming. (18:01 UTC — Nov 25) Alpine builds now available: The next step is to start pulling build images down to the new build runner. Currently Alpine is available and Arch is being worked on next. (17:01 UTC — Nov 25) Restoring builds service: Work is ongoing to provision a new build host. Service is expected to be restored within...| Issues on sr.ht status
The services are online and working. We have been monitoring the mitigations we put in place last night and, though we’re still tying off some loose ends, all services should be available for users. (07:00 UTC — Apr 21) A distributed denial of service (DDoS) attack on sr.ht is underway. We have identified the source of the networking issues as a denial of service attack and we are deploying a mitigation. (19:10 UTC — Apr 20)| status.sr.ht
We have deployed Anubis to git.sr.ht. After some internal discussions we have ultimately decided that the best course of action to protect git.sr.ht from LLM crawlers is to deploy Anubis. This software presents some users with a proof-of-work challenge which is solved by the user’s browser with JavaScript. This challenge is automatically bypassed for logged-in users. If your browser does not support JavaScript (or you do not wish to enable it for any other reason), log in at meta.sr.ht to c...| status.sr.ht
Maintenance completed. Access to all services is restored. (11:40 UTC — Feb 11) Maintenance will take longer as expected. Due to unforeseen circumstances, the maintenance will take longer than anticipated. We are working to resolve the issue as quickly as possible. (11:00 UTC — Feb 11) Maintenance is underway. The maintenance window is now open. (10:00 UTC — Feb 11) Planned maintenance on February 11th will cause intermittent outages: Planned maintenance starting at 10AM UTC will affect...| status.sr.ht
SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations, instead of doing any of the other numerous things we have to do, like develop new features, upgrade our infrastructure, or respond to support emails in a timely manner. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users. If you are unable to use the SourceHut web UI:| status.sr.ht
Mitigations completed: We believe that we have the attack under control and that there is no need for end-user impacting mitigations to remain in place. (09:00 UTC — Jan 3) Attack still ongoing, but under control: Good morning. The attack is still ongoing, but we have been refining our mitigations this morning. We addressed some cases which were causing git and hg to crash under load, and narrowed our mitigations to better characterize the load. All of our mitigations have now been disabled...| status.sr.ht
Mitigations complete. We have finished mitigating the attack and have removed user-impacting mitigations. (17:00 UTC — Dec 4) Attack partially mitigated. We have partially mitigated the attack to restore service to git.sr.ht, but many features of the web UI are currently disabled to reduce load. (12:53 UTC — Dec 4) A distributed denial of service (DDoS) attack on git.sr.ht is underway: A denial of service attack against git.sr.ht is currently underway. We are working on deploying a mitiga...| status.sr.ht
Maintenance work is completed. All services should be operating normally. Please let us know if you notice any lingering problems. (13:26 UTC — Nov 25) The maintenance window is now open. (13:00 UTC — Nov 25) Maintenance window rescheduled for 13:00 UTC. We have advanced the schedule by one hour to 13:00 UTC. (08:00 UTC — Nov 25) Planned maintenance is scheduled for Monday, November 25th. On Monday at 13:00 UTC we will be rolling out some changes which will cause spurious errors and iss...| status.sr.ht
Upstream network issues appear resolved. We still know little of the cause but we know it broadly affected numerous providers in our service area. (14:00 UTC — Oct 30) Upstream network issues are causing intermittent availability problems for SourceHut. An issue with our upstream network provider has affected user access to SourceHut. We do not have any further details or an ETA on a solution at this time. We expect the network to be intermittent until the upstream issue is resolved. (11:30...| status.sr.ht
Some users are having problems with chat.sr.ht. There are some known issues with chat.sr.ht which are preventing a small number of users from accessing the service. The problem is known and being investigated, but it is complex and difficult to fully understand and address. We request your patience as it may require several days to identify a complete solution. (08:00 UTC — Aug 14)| status.sr.ht
All planned maintenance has been completed. The maintenance window is now closed. Please notify support if you encounter any unexpected issues. The maintenance window is now open. We will be opening our maintenance window in the next few minutes and begin rolling out upgrades. A planned maintenance window is scheduled for August 5th. The maintenance window originally scheduled for August 1st has been rescheduled for Monday, August 5th at 11:00 UTC. This is expected to cause intermittent parti...| status.sr.ht
The maintenance window has been cancelled. A new maintenance window will be planned in the next few days. A planned maintenance window is scheduled for August 1st. At 11:00 UTC we will be rolling out upgrades across our fleet. This is expected to cause intermittent partial outages between 15 and 30 minutes affecting all services. Follow this page or #sr.ht on IRC for updates.| status.sr.ht
Planned maintenance is complete. We have completed all planned upgrades and restored all services to normal working order. Please let us know if you notice any lingering issues. Thank you for your patience. (15:15 UTC — Mar 20) The maintenance window is open. Planned maintenance is now underway. (12:00 UTC — Mar 20) Planned maintenance this Wednesday will require service interruptions. We are planning upgrades across our fleet on Wednesday, March 20th, which will require service interrupt...| status.sr.ht
Update 2024-01-17 10:30 UTC: full service has been restored. Update 2024-01-16 19:45 UTC: builds.sr.ht is now running at full capacity. Update 2024-01-16 18:00 UTC: builds.sr.ht is now available for full service. We are running it at half of the planned capacity, with the remainder coming online shortly, so you may experience longer queue times while waiting for builds to be processed. Update 2024-01-16 10:30 UTC: Our priority for today is to bring builds.| status.sr.ht