Over the past few years, Apple seems increasingly willing to cooperate with authoritarian governments, uninterested in protecting its own users, and unwilling to actually standup for human rights in broad terms, as often portrayed by its marketing department or direct statements from CEO Tim Cook.
Over the past few years, Apple seems increasingly willing to cooperate with authoritarian governments, uninterested in protecting its own users, and unwilling to actually standup for human rights in broad terms, as often portrayed by its marketing department or direct statements from CEO Tim Cook.
Most recently, there was a dispute with ProtonVPN (the company that also makes ProtonMail) over an update for its app in the App Store. Proton Technologies claimed that Apple was intentionally blocking the update amid the ongoing crackdown in Myanmar. I agree with Gruber that there is little direct evidence to support this exact claim. While I am willing to give Apple the benefit of the doubt and consider this an inconvenient coincidence, I would not be surprised if this were a deliberate move. After all, Apple has pulled VPN apps from the App Store before. For now, we can assume (as Gruber highlights) that this is yet another issue with Apple’s poorly executed app review process where its so-called rules are applied arbitrarily.
\n\n
However, there is still reason to be concerned, because Apple does not have a laudable record when it comes to cooperating with authoritarian governments. Below is a brief history of events that I have been tracking so far. If you know of others, get in touch.
\n\n
* * *
\n\n
January 2017 — Apple removed NYT from the China App Store
Apple, complying with what it said was a request from Chinese authorities, removed news apps created by The New York Times from its app store in China late last month.
\n
\n\n
July 2017 — Apple removes VPN apps from China App Store
China appears to have received help on Saturday from an unlikely source in its fight against tools that help users evade its Great Firewall of internet censorship: Apple.
\n\n
Software made by foreign companies to help users skirt the country’s system of internet filters has vanished from Apple’s app store on the mainland.
We received notification from Apple today, July 29, 2017 — at roughly 04:00 GMT, that the ExpressVPN iOS app was removed from the China App Store. Our preliminary research indicates that all major VPN apps for iOS have been removed.
\n
\n\n
November 2017 — Apple removes Skype from China App Store
“We have been notified by the Ministry of Public Security that a number of voice over internet protocol apps do not comply with local law. Therefore these apps have been removed from the app store in China,” an Apple spokeswoman said Tuesday in an emailed statement responding to questions about Skype’s disappearance from the app store.
\n
\n\n
September 2019 — Apple adopts a “SIM canary”
\n\n
If you insert a Chinese carrier SIM, apps like TikTok & Apple News no longer function.
TikTok for iOS has an unique region (i.e. China) lock. Before that it was Apple News.\nNews detects iPhone’s SIM region, you can’t use the app only when ALL sims are from China.\nTikTok also checks SIM region, the diff is you can’t use it if ONE of the sims is from China.
\n\n
Having two sims inserted, one local and one roaming from China, TikTok can’t load a single video. Also failed on airplane mode. It works only after I take the China sim out.
I had no idea China-bound iPhones were shipping with physical dual SIM support. Clever tactic to region users.
\n\n
iPhone XS Max and iPhone XR. iPhone XS only supports dual SIMs with an eSIM.
\n\n
This is a pretty intense tactic, and I think it will be increasingly more common as the Chinese tech ecosystem diverges. It’s pretty concerning that Apple itself adopts it. The Great Firewall is moving to the metal; the VPNs won’t work in that future.
\n
\n\n
October 2019 — Apple removes Taiwan flag emoji in iOS 13
The Republic of China flag emoji has disappeared from Apple iPhone’s keyboard for Hong Kong and Macau users. The change happened for users who updated their phones to the latest operating system.
\n\n
Updating iPhones to iOS 13.1.1 or above caused the flag emoji to disappear from the emoji keyboard. The flag, commonly used by users to denote Taiwan, can still be displayed by typing “Taiwan” in English, and choosing the flag in prediction candidates.
Apple removed an app late Wednesday that enabled protesters in Hong Kong to track the police, a day after facing intense criticism from Chinese state media for it, plunging the technology giant deeper into the complicated politics of a country that is fundamental to its business.
This wasn’t a promotion for the Mac Pro or its assembly plant. It was a promotion for Trump. This video makes it look like Trump’s trade policies have been good for Apple and that Tim Cook supports Trump.
\n\n
[…]
\n\n
This is how Apple chose to unveil the packaging for the Mac Pro — in a poorly-shot overexposed propaganda video by the White House, scored with bombastic music that sounds like it came from an SNL parody of a Michael Bay film.
Admittedly, this event is a bit of an outlier in relation to the others I have listed so far. But it is important to mention. This ad revealed how much effort Tim Cook is willing to exert to debase not only himself, but his entire company, to avoid tariffs and please whoever wields political power.
\n\n
I find it difficult to reconcile. How can you claim to stand up for human rights while voluntarily participating in a re-election campaign ad for The Human-Rights-Violator-in-Chief? And to top it off, Tim Cook gifted Trump the ‘first’ 2019 Mac Pro. Gruber is adamant that Cook does not support Trump and that Cook personally engaging with Trump does not imply support. However, I fail to see how that is relevant, when Cook is clearly willingly to play the game, regardless of his personal views.
\n\n
This campaign ad made obvious the tacit acknowledgment that Tim Cook is merely a capitalist — not a human rights activist. Apple is ultimately beholden to shareholders and investors, which means the pursuit of profit takes priority over anything else — including and especially human rights violations.
\n\n
February 2020 — Apple removed Plague Inc. from Chinese App Store
Popular infection simulation game Plague Inc. has been removed from Chinese app stores, Apple and Xiaomi users noticed today, after enjoying renewed popularity during the Covid-19 outbreak.
\n\n
Why it matters: The removal shows just how serious the country’s authorities are in managing the public perception of the virus.
\n\n
Chinese authorities have been known to ban adult content and games with politically sensitive hidden messages. Plague Inc. has been praised for its educational value and scientific approach.
\n
\n\n
October 2020 — Apple forced Telegram to close channels run by Belarus protestors
Apple is requesting that Telegram shut down three channels used in Belarus to expose the identities of individuals belonging to the Belarusian authoritarian regime that may be oppressing civilians.
Anyone who believes Apple and the FBI are at an impasse over investigations into the iPhone maker’s criminal customers should think again. In Seattle, Apple has given the feds vital evidence from one of its iCloud users who was arrested for firebombing cop cars during the George Floyd protests in late May.
\n\n
The case shows how Apple is willing to help even where the context of the crime is controversial, namely the Black Lives Matter protests.
\n
\n\n
(Note: you may be wondering why I have included the US in a list of authoritarian governments. Let me remind you of the outrageouspolice brutalitythat we sawlast summer. And let’s not forget the orange elephant in the room.)
Apple Inc dropped plans to let iPhone users fully encrypt backups of their devices in the company’s iCloud service after the FBI complained that the move would harm investigations, six sources familiar with the matter told Reuters.
\n
\n\n
While this report is not confirmed, it does seem plausible. After years of incrementally advancing the security of iOS and touting its tough stance on privacy, why has Apple not delivered on end-to-end encryption for iCloud? I understand there are many technical challenges, but surely it should be a high priority.
At this meeting, defendant Apple supervisors stated that the Guo Media App is critical of the Chinese government and, therefore, should be removed from the App Store. Plaintiff Pham responded stating the Guo Media App publishes valid claims of corruption against the Chinese government and Chinese Communist Party and, therefore, should not be taken down. Plaintiff Pham further told his supervisors that the Guo Media App does not contain violent content or incite violence; does not violate any of defendant Apple’s policies and procedures regarding Apps; and, therefore, it should remain on the App Store as a matter of free speech.
\n\n
Defendant Apple became aware of plaintiff Pham’s criticism and defendant Apple’s managers responded by retaliating against plaintiff Pham and ultimately terminating plaintiff Pham.
\n
\n\n
February 2021 — Apple Removes Apps for Pakistani Government
Over the last two years, the government of Pakistan has forced Google and Apple to take down apps in the country created by developers based in other nations who are part of a repressed religious minority.
\n\n
[…]
\n\n
At issue are seven religious apps created by the Ahmadi community in the United States, published under the name “Ahmadiyya Muslim Community.”
\n
\n\n
March 2021 — Apple reportedly agrees to preinstall Russian software
Russian media are reporting that Apple has agreed to sell its gadgets in Russia with preinstalled Russian-made software to comply with a law that comes into force on April 1.
\n\n
[…]
\n\n
The list of Russian government-approved programs for mandatory preinstallation on smartphones and tablets includes the search engine Yandex, Mail.ru mail and news, ICQ messenger, social network VKontakte, payment system MirPay, and antivirus Kaspersky Lab, among others.
The entries below were added after this piece was first published. I will continue to keep this post up-to-date as new events occur. The date of this notice reflects the last time an update was published. Updates are in chronological order. Scroll to the bottom for the most recent update.
\n\n
\n\n
May 2021 — Seven Apple Suppliers Accused of Using Forced Labor From Xinjiang
The industrial park is surrounded by walls and fences with only one way in or out.
\n\n
And next to the park was a large compound identified by a satellite imagery researcher as a detention center where the factory workers lived. The researcher, Nathan Ruser, from an Australian think tank, said “almost no other factories in Xinjiang have these characteristics except for industrial parks where there is detainee labor.
\n\n
The Information and human rights groups have found seven companies supplying device components, coatings and assembly services to Apple that are linked to alleged forced labor involving Uyghurs and other oppressed minorities in China. At least five of those companies received thousands of Uyghur and other minority workers at specific factory sites or subsidiaries that did work for Apple, the investigation found.
\n\n
The revelation stands in contrast to Apple’s assertions over the past year that it hasn’t found evidence of forced labor in its supply chain.
\n
\n\n
May 2021 — Censorship, Surveillance and Profits: A Hard Bargain for Apple in China
Tim Cook, Apple’s chief executive, has said the data is safe. But at the data center in Guiyang, which Apple hoped would be completed by next month, and another in the Inner Mongolia region, Apple has largely ceded control to the Chinese government.
\n\n
Chinese state employees physically manage the computers. Apple abandoned the encryption technology it used elsewhere after China would not allow it. And the digital keys that unlock information on those computers are stored in the data centers they’re meant to secure.
\n\n
Internal Apple documents reviewed by The New York Times, interviews with 17 current and former Apple employees and four security experts, and new filings made in a court case in the United States last week provide rare insight into the compromises Mr. Cook has made to do business in China. They offer an extensive inside look — many aspects of which have never been reported before — at how Apple has given in to escalating demands from the Chinese authorities.
\n\n
[…]
\n\n
Mr. Cook often talks about Apple’s commitment to civil liberties and privacy. But to stay on the right side of Chinese regulators, his company has put the data of its Chinese customers at risk and has aided government censorship in the Chinese version of its App Store. After Chinese employees complained, it even dropped the “Designed by Apple in California” slogan from the backs of iPhones.
Apple Inc on Monday said a new “private relay” feature designed to obscure a user’s web browsing behavior from internet service providers and advertisers will not be available in China for regulatory reasons.
\n\n
Apple said it also will not offer “private relay” in Belarus, Colombia, Egypt, Kazakhstan, Saudi Arabia, South Africa, Turkmenistan, Uganda and the Philippines.
\n
\n\n
August 2021 — Engrave Danger: An Analysis of Apple Engraving Censorship across Six Regions
We analyzed Apple’s filtering of product engravings in six regions, discovering 1,105 keyword filtering rules used to moderate their content.
\n\n
Across all six regions we analyzed, we found that Apple’s content moderation practices pertaining to derogatory, racist, or sexual content are inconsistently applied and that Apple’s public-facing documents failed to explain how it derives their keyword lists.
\n\n
Within mainland China, we found that Apple censors political content including broad references to Chinese leadership and China’s political system, names of dissidents and independent news organizations, and general terms relating to religions, democracy, and human rights.
\n\n
We found that part of Apple’s mainland China political censorship bleeds into both Hong Kong and Taiwan. Much of this censorship exceeds Apple’s legal obligations in Hong Kong, and we are aware of no legal justification for the political censorship of content in Taiwan.
\n\n
We present evidence that Apple does not fully understand what content they censor and that, rather than each censored keyword being born of careful consideration, many seem to have been thoughtlessly reappropriated from other sources.
\n
\n\n
September 2021 — Apple removes Navalny app from stores before Russian elections
Earlier this month, when the Kremlin told multiple Big Tech companies to suppress political opposition amid nationwide elections in Russia, their answer was unequivocal: no. Yet just two weeks later, Apple and Google deleted from their app stores the Smart Voting app, opposition leader Alexey Navalny and his party’s primary tool for consolidating votes against Vladimir Putin’s regime. Then Telegram and Google-owned YouTube also restricted access to the recommendations for opposition candidates that Navalny was sharing on these platforms.
Apple has asked suppliers to ensure that shipments from Taiwan to China strictly comply with Chinese customs regulations after a recent visit by senior U.S. lawmaker Nancy Pelosi to Taipei stoked fears of rising trade barriers.
\n\n
Apple told suppliers on Friday that China has started strictly enforcing a long-standing rule that Taiwanese-made parts and components must be labeled as being made either in “Taiwan, China” or “Chinese Taipei,” sources familiar with the matter told Nikkei Asia, language that indicates the island is part of China.
\n
\n\n
It is certainly not a good look when you force your suppliers to print CCP state propaganda on Taiwanese-made parts.
\n\n
January 2023 — Apple Brings Mainland Chinese Web Censorship To Hong Kong
Apple quietly expanded the use of Chinese company Tencent’s website blacklist to users in Hong Kong — and no one will answer questions about it.
\n\n
When Safari users in Hong Kong recently tried to load the popular code-sharing website GitLab, they received a strange warning instead: Apple’s browser was blocking the site for their own safety. The access was temporarily cut off thanks to Apple’s use of a Chinese corporate website blacklist, which resulted in the innocuous site being flagged as a purveyor of misinformation. Neither Tencent, the massive Chinese firm behind the web filter, nor Apple will say how or why the site was censored.
\n
\n\n
December 2023 — Governments spying on Apple, Google users through push notifications
Unidentified governments are surveilling smartphone users via their apps’ push notifications, a U.S. senator warned on Wednesday.
\n\n
In a letter to the Department of Justice, Senator Ron Wyden said foreign officials were demanding the data from Alphabet’s Google and Apple. Although details were sparse, the letter lays out yet another path by which governments can track smartphones.
Apple on late Thursday into Friday removed the popular messaging and social media apps WhatsApp, Telegram, Signal, and Threads from its App Store in China at the request of the Chinese government […]
\n\n
Apple has complied with similar App Store removal orders from the Chinese government in the past for apps related to VPNs, news, and more.
\n
\n\n
July 2024 — Apple removes VPN apps from the App Store in Russia
Apple has caved to pressure from Russian authorities and removed a number of the best iPhone VPN apps from the App Store in the country.
\n\n
Apple’s App Store team has been notifying VPN developers that their apps are being removed “per demand from Roskomnadzor.” This the state media watchdog that previously forced both Apple and Google to remove a political app backed by the leader of the country’s opposition.
\n\n
[…]
\n\n
Reportedly, VPN usage increased dramatically in Russia following the start of the war with Ukraine. At that point, Russian authorities blocked access to a number of Western social media sites.
Apple has quietly removed DeICER, a civic-reporting app used to log immigration enforcement activity, from its App Store after a law enforcement complaint — invoking a rule normally reserved for protecting marginalized groups from hate speech.
Apple removed an app for preserving TikToks, Instagram reels, news reports, and videos documenting abuses by ICE, 404 Media has learned. The app, called Eyes Up, differs from other banned apps such as ICEBlock which were designed to report sightings of ICE officials in real-time to warn local communities. Eyes Up, meanwhile, was more of an aggregation service pooling together information to preserve evidence in case the material is needed in the future in court.
So what does any of this have to do with app developers? Why should we care? When it comes to the iOS App Store, Apple controls where we are allowed to distribute our apps. More importantly, Apple has the unilateral power remove our apps from any App Store region at any time to nurture its relationship with whatever unsavory government it is interested in pleasing in order to pursue its political motives or financial objectives.
\n\n
Apple’s centralized power over app distribution combined with its willingness to surrender to political pressures is incredibly concerning as ostensibly “democratic” governments across the globe (including the United Sates!) increasingly exhibit far-right, fascist behavior and implement fascist policies. What will happen when you need to build your own HKmap.live?
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2024/06/05/swift-concurrency-non-sendable-closures/#updated-05-june-2024",
"url": "https://www.jessesquires.com/blog/2024/06/05/swift-concurrency-non-sendable-closures/",
"title": "[Updated] Swift concurrency hack for passing non-sendable closures: Uncheck yourself before you wreck yourself",
"date_published": "2024-06-05T18:24:16-07:00",
"date_modified": "2024-06-05T18:24:16-07:00",
"summary": "
If you have attempted to adopt Swift Concurrency in your codebase, you have certainly needed to address dozens — likely, hundreds — of warnings and errors. Sometimes the issues can be resolved by addressing them directly. That is, your code was incorrect and you simply have to fix it to make it correct. In other scenarios, the resolution is not so straightforward. In particular, it is difficult to satisfy the compiler when working with APIs that you do not own that have not been updated for concurrency. Or, you may have found yourself in a situation where you know your code is correct, but the compiler is unable to verify its correctness — either because of a few remaining bugs in Swift Concurrency, or because you are using @preconcurrency APIs.
If you have attempted to adopt Swift Concurrency in your codebase, you have certainly needed to address dozens — likely, hundreds — of warnings and errors. Sometimes the issues can be resolved by addressing them directly. That is, your code was incorrect and you simply have to fix it to make it correct. In other scenarios, the resolution is not so straightforward. In particular, it is difficult to satisfy the compiler when working with APIs that you do not own that have not been updated for concurrency. Or, you may have found yourself in a situation where you know your code is correct, but the compiler is unable to verify its correctness — either because of a few remaining bugs in Swift Concurrency, or because you are using @preconcurrency APIs.
\n\n\n\n
One of the warnings you have probably seen multiple times is “Capture of ‘variable’ with non-sendable type in a @Sendable closure.” I was confronted with this warning in a project recently and I want to share the hack for how I worked around it. The issue was the result of a combination of factors I mentioned above. I was interacting with @preconcurrency APIs and I knew my code was concurrency-safe, but I was unable to accurately express that to the compiler.
\n\n
Background
\n\n
First, let’s discuss the context in which I was dealing with this concurrency warning. I have been working on a project that uses UICollectionViewDiffableDataSource and I am wrapping the UIKit APIs with something more user-friendly. For the purposes of this post, I have omitted much of the complexity to provide a clear, simple example.
\n\n
I have a custom diffable data source that performs the diffing on a background thread:
Per the documentation, the completion closure is always called on the main queue and you can call apply(_:animatingDifferences:completion:) from a background queue:
\n\n
\n
[…]
\n\n
completion \nA closure to execute when the animations are complete. This closure has no return value and takes no parameters. The system calls this closure from the main queue.
\n\n
[…]
\n\n
You can safely call this method from a background queue, but you must do so consistently in your app. Always call this method exclusively from the main queue or from a background queue.
\n
\n\n
Also note that UICollectionViewDiffableDataSource is annotated with @MainActor @preconcurrency.
\n\n
With the strictest concurrency checking enabled, the code above produces 2 warnings:
\n\n
funcapplyDiff(snapshot:Snapshot,animated:Bool=true,completion:SnapshotCompletion?=nil){\n self.diffingQueue.async{\n self.apply(snapshot,animatingDifferences:animated,completion:completion)\n // ^\n // Capture of 'completion' with non-sendable type 'DiffableDataSource.SnapshotCompletion?' (aka 'Optional<@MainActor () -> ()>') in a `@Sendable` closure\n // Converting function value of type '@MainActor () -> Void' to '() -> Void' loses global actor 'MainActor'; this is an error in Swift 6\n }\n}\n
\n\n
One solution would be to make the completion closure @Sendable:
(Note that we cannot use the defined typealias SnapshotCompletion in this case, and it must not be optional.)
\n\n
However, this will not work. This API updates the collection view to reflect the state of the data in the snapshot and, as mentioned, completion is called on the main thread. Callers of applyDiff(snapshot:) do additional UI updates and animations, or otherwise deal with @MainActor members and types in this completion closure. Those members and types cannot be marked as @Sendable. For example, the owner of the DiffableDataSource instance could be a view controller. Furthermore, I do not want to impose a @Sendable restriction upon callers.
\n\n
Thus, making the closure @Sendable produces the following kinds of errors at the call site:
\n\n
self.dataSource.applyDiff(snapshot:snapshot){\n\n // Call to main actor-isolated instance method 'someMethod()' in a synchronous nonisolated context\n\n // Main actor-isolated property 'someProperty' can not be referenced from a Sendable closure; this is an error in Swift 6\n}\n
\n\n
Callers could wrap the offending lines in Task { @MainActor in } or MainActor.assumeIsolated { } to silence these issues. But, that’s a burden for callers. Not to mention, the wrapper API does not accurately communicate what is happening here. We do not want a @Sendable () -> Void closure. We want a @MainActor () -> Void closure.
\n\n
So, we have situation where the Swift compiler is telling us that the closure being captured needs to be @Sendable but we cannot make it @Sendable. It is also telling us that the closure loses its @MainActor but we know that the closure will always be called from the main queue. Because of these two problems, we need to find a way to work around the warnings and coerce the compiler into doing what we want.
\n\n
Solution (It’s a hack)
\n\n
We can wrap the completion closure in another type that is @unchecked Sendable.
This will silence the warning about “capturing a non-sendable type in a @Sendable closure.” Again, UIKit guarantees that this completion closure will always be called on the main thread, and we can use a dispatchPrecondition() to verify this is happening.
\n\n
We can update our API to use this new UncheckedCompletion wrapper.
However, exposing UncheckedCompletion to callers is also not a great API. We should hide this detail. We can wrap this applyDiff() method with another that uses the original SnapshotCompletion typealias.
And now, the public API looks exactly the same as before to callers, but they can safely use @MainActor members and types in the completion closure without any warnings or errors.
\n\n
self.dataSource.applyDiff(snapshot:snapshot){\n // do anything here safely with @MainActor with no warnings or errors\n}\n
\n\n
Is this good?
\n\n
Is this a good idea? I am actually not sure! But, it seems like the best thing to do in this scenario. If you are facing a similar situation — namely, you know a captured closure is always called on the main thread and you cannot make it @Sendable — this might be a good solution for you too! However, this is probably a bad idea to attempt to generalize. Use wisely!
As anticipated, Matt Massicotte has come to the rescue, offering a simpler solution here. While my clever hack works, we can instead make the closure both @Sendableand@MainActor. After that, we can simply wrap calling the completion closure in MainActor.assumeIsolated { }. Here are the changes needed:
\n\n
// The addition of @Sendable is bizarre, but Swift 5.10 needs it.\n// Swift 6 (via SE-0434) will make it unnecessary.\ntypealiasSnapshotCompletion=@Sendable@MainActor()->Void\n\nfuncapplyDiff(snapshot:Snapshot,animated:Bool=true,completion:SnapshotCompletion?=nil){\n self.diffingQueue.async{\n // UIKit guarantees `completion` is called on the main queue.\n self.apply(snapshot,animatingDifferences:animated,completion:{\n // when you know its on the main actor, perhaps from documentation, but it isn't\n // encoded in the API, you can use dynamic isolation to make it work\n MainActor.assumeIsolated{\n completion?()\n }\n })\n }\n}\n
\n\n
This is great. It is much less code. I’m not sure why I didn’t try using @Sendable @MainActor for the closure. It’s probably because I assumed that @MainActorimplied@Sendable — which apparently it should and it will after SE-0434 in Swift 6.
\n\n
Anyway, the general idea of this hack might still be useful in other contexts or scenarios — especially if you cannot adopt Swift 6 and need to stay on Swift 5.10.
More and more Apple Platform developers are migrating away from CocoaPods in favor the Swift Package Manager, which is Apple’s first-party tool for managing and integrating dependencies. While it is still not quite a complete replacement for CocoaPods, it is getting closer. Unfortunately, SwiftPM’s integration with Xcode still has a number of shortcomings, even though it was introduced with Xcode 11 — 4 years ago. The worst bug is that Xcode frequently and randomly deletes the Package.resolved, which in turn produces dozens or hundreds of 'missing package product' errors. Here’s how I’ve worked around this bug on a team I work on.
More and more Apple Platform developers are migrating away from CocoaPods in favor the Swift Package Manager, which is Apple’s first-party tool for managing and integrating dependencies. While it is still not quite a complete replacement for CocoaPods, it is getting closer. Unfortunately, SwiftPM’s integration with Xcode still has a number of shortcomings, even though it was introduced with Xcode 11 — 4 years ago. The worst bug is that Xcode frequently and randomly deletes the Package.resolved, which in turn produces dozens or hundreds of 'missing package product' errors. Here’s how I’ve worked around this bug on a team I work on.
\n\n\n\n
The bug
\n\n
Xcode seems to specifically delete Package.resolved when changing branches in git. Then, depending on how large your project is and how many Swift packages are included, you’ll see dozens or hundreds or thousands of 'Missing package product <product_name>' errors.
\n\n
However, I have also experienced Xcode deleting the file for seemingly no reason at all. In fact, I’ve had moments where Xcode deletes the file immediately after I restore it via git restore — and I can just do that infinitely. I put the file back, then Xcode deletes it. In that beautiful scenario, the Xcode project must be closed and re-opened.
\n\n
Searching the Apple Developer Forums returns dozens of results for this bug. Here is a recent one that correctly describes the problem. Here is another post from 2021 that has new replies from 2 weeks ago. There is also this post from the Swift.org forums — from June 2020 — describing the issue. If you read through that thread, you will see the problem was briefly fixed around Xcode 12, but regressed in either the 12.5 or 13.0 release. This forum thread also has new comments from the past 2 weeks (including from yours truly). Unfortunately, the Swift forums are not the right place to report bugs in Xcode.
\n\n
Most “solutions” on the forums revolve around some magical combination of cleaning your project (cmd-shift-K), deleting Xcode’s DerivedData/, running File > Packages > Reset Package Caches, running File > Packages > Resolve Package Versions, and closing and re-opening Xcode. All of this is very heavy-handed and there’s a much lighter weight approach you can take to work around it.
\n\n
This bug is happening in the latest release, Xcode 15.4 (15F31d). It is a widespread problem reported across the various forums, as well as on Mastodon and other platforms. Any developer on a team that is using SwiftPM with Xcode is experiencing this bug daily. Furthermore, running xcodebuild -resolvePackageDependencies seems to have no effect and it does not regenerate Package.resolved.
\n\n
The Package.resolved file and git
\n\n
First, what is Package.resolved and why is it important? This file is equivalent to Podfile.lock in CocoaPods — similar to Gemfile.lock in Ruby/Bundler, or package-lock.json in Node.js/npm. This file is a manifest that describes the exact versions used for every package dependency. It allows SwiftPM to always install the exact same packages on multiple machines, for example the machines of each member on a team and CI machines.
\n\n
It turns out, it’s just a JSON file. Here’s an example of Package.resolved for an app that imports one library called Foil.
When working with only Swift packages, your Package.swift file defines all of your package dependencies. This plays a similar role to the Podfile in CocoaPods. With SwiftPM’s Xcode integration, the project file project.pbxproj — everyone’s favorite file — maintains the list of Swift package dependencies and their pinned versions. Again, this fulfills the same role as the Podfile.
\n\n
The project file is located at MyApp.xcodeproj/project.pbxproj. If you search the project.pbxproj file for XCRemoteSwiftPackageReference and XCSwiftPackageProductDependency, you will find all the Swift package references for your project. For the example Package.resolved file above, here is the project.pbxproj representation:
For small teams or individuals that already include their entire .xcodeproj or .xcworkspace bundles in git, there is nothing else you need to do. However, most teams explicitly ignore .xcodeproj and .xcworkspace — ironically, because of project.pbxproj, which is a nightmare to resolve when you have conflicts in git. These teams typically use Xcodegen or a similar tool to generate their project files, thus entirely avoiding merge conflicts on project.pbxproj. In this scenario, including the Package.resolved file but ignoring everything else requires a bit of work. Here are the git ignore rules needed:
Thanks to Kyle Bashour for pointing out a couple of details I got wrong about project.pbxproj. This section has been corrected.
\n\n
\n\n
Handling Package.resolved deletions
\n\n
Now that the Package.resolved file is saved in git, we can easily handle when Xcode deletes it by simply restoring it via git. For teams that ignore their project and workspace files and use Xcodegen (or similar), you also have the issue where Xcodegen deletes Package.resolved when it regenerates your project file. If you are also still using CocoaPods in conjunction with SwiftPM, then you have the issue where CocoaPods will delete Package.resolved when it regenerates your workspace file.
\n\n
When you notice the file gets deleted, you can restore it using git restore. And then all the 'Missing package product' errors will go away when you build your project.
\n\n
It is common for iOS and macOS projects to use a Makefile in order to bootstrap project setup, which often includes: generating the project file via Xcodegen, installing CocoaPods via Bundler, running pod install, etc. In addition, makefiles are also a great place to write “shortcut” commands that are relevant to your project, for example, linting files or running tests.
\n\n
Here’s a target you can add to your Makefile to make restoring a deleted Package.resolved file easier.
\n\n
PACKAGE_FILE:=\"MyApp.xcworkspace/xcshareddata/swiftpm/Package.resolved\"\n\n.PHONY:swiftpm\nswiftpm:\n @# Restore Package.resolved, which gets deleted when re-generating the project/workspace.\n@# Or, it gets deleted by Xcode.\n@# Only do this if the file was completely deleted.\n@# Otherwise, the user could be modifying packages which updates Package.resolved, so do not git restore it.\n@if[!-f\"$(PACKAGE_FILE)\"];then\\\n echo\"Restoring Package.resolved...\";\\\n gitrestore\"$(PACKAGE_FILE)\";\\\n xcodebuild-resolvePackageDependencies;\\\n fi\n
\n\n
With this target in your Makefile, you can simply run make swiftpm as needed.
\n\n
Note that this only calls git restore if the file is deleted, in case it has been (correctly) modified after updating packages. At the end, it calls xcodebuild -resolvePackageDependencies, which should regenerate the Package.resolved on its own, but that’s also broken. It will, however, download packages to the package cache if they are missing, which is what Xcode does when opening a project.
\n\n
To make this even better, you can include running this target as part of your bootstrapping process if you are using CocoaPods, Xcodegen, etc. Include this target part of your primary setup flow, and you should rarely have to run it manually.
\n\n
If you do not use a Makefile, then you can write a simple bash script instead with the contents above.
\n\n
Danger: accidental deletions
\n\n
Unfortunately (again), what we discovered on one of my teams is that the above is not foolproof. Because Xcode can (and literally does) randomly delete Package.resolved at any time, someone on your team might not notice and commit the file deletion. Oops! If you use a tool like Danger to automate code reviews (which, you should), then you can add a rule to catch this mistake.
\n\n
# Dangerfile\n\nifgit.deleted_files.include?(\"MyApp.xcworkspace/xcshareddata/swiftpm/Package.resolved\")\n fail(\"It looks like you deleted `Package.resolved`. Please don't do that.\")\nend\n
\n\n
This will fail a pull request if Package.resolved was deleted.
\n\n
Summary
\n\n
To recap: make sure Package.resolved is saved in git, use a Makefile to quickly and easily restore it, and add a Danger rule to catch mistakes, if needed. This should eliminate all 'Missing package product' errors. As a last resort, if you are still having trouble, you can run File > Packages > Resolve Package Versions. If you search for solutions on StackOverflow or elsewhere, you will find some really elaborate bash scripts that use fswatch to monitor the file system to workaround this problem. I think this is a bit overkill and prefer the simpler solutions I’ve offered here.
\n\n
This is all a bit ridiculous, and I would say it’s shocking that this bug still hasn’t been fixed after 4 years, but this kind of thing is not all that surprising for Apple. It is rather routine that many bugs remain unfixed and tools remain broken for years.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2023/12/16/macbook-notch-and-menu-bar-fixes/#updated-29-may-2024",
"url": "https://www.jessesquires.com/blog/2023/12/16/macbook-notch-and-menu-bar-fixes/",
"title": "[Updated] How to fix Mac menu bar icons hidden by the MacBook notch",
"date_published": "2024-05-29T18:20:52-07:00",
"date_modified": "2024-05-29T18:20:52-07:00",
"summary": "
Last week I wrote about setting up a new MacBook Pro — my first Apple Silicon Mac, and thus my first MacBook with a notch. I lamented how poorly macOS interacts with the notch, specifically how menu bar apps and icons simply get hidden if you have too many to display. Lots of folks on Mastodon offered various solutions, and some readers emailed me with options as well. I figured it was worth making a separate post about this specific issue to list all of the workarounds and alternatives. It is clear that this is a widespread problem that users are having.
Last week I wrote about setting up a new MacBook Pro — my first Apple Silicon Mac, and thus my first MacBook with a notch. I lamented how poorly macOS interacts with the notch, specifically how menu bar apps and icons simply get hidden if you have too many to display. Lots of folks on Mastodon offered various solutions, and some readers emailed me with options as well. I figured it was worth making a separate post about this specific issue to list all of the workarounds and alternatives. It is clear that this is a widespread problem that users are having.
I have gripes about the notch. There isn’t enough room to display all of my menu bar apps and icons, so… they just get hidden!? Apparently, everyone in Cupertino thinks the best solution to this problem is to hide them with zero indication that there are more that simply can’t be displayed because of the notch. I wasted so much time trying to figure out why Little Snitch and 1Password were not running on my new machine. Was there a compatibility issue with Apple Silicon that I didn’t know about? That couldn’t be. It turns out, they were running the whole time but they were hidden by the notch.
\n\n
[…]
\n\n
This “design” (or lack thereof) is so dumb. It is utterly ridiculous to me that this is still how it “works” two years after the introduction of the redesigned MacBook Pro with a notch. How hard could it be to add an overflow menu with a “«” (or should it be “»”?) button that shows the remaining apps and icons that can’t be displayed? This entire situation with the notch is ironic, because the iPhone notch and “dynamic island” are so thoughtfully designed with zero compromises regarding the functionality of iOS. In fact, they actually provide a better user experience. Yet on the Mac, how the notch interacts with macOS is laughably incompetent. It is shockingly lazy regarding attention to detail, and results in an outright disruptive and confusing user experience.
Aside from the problem of the icons being hidden, there’s no API for an app to tell whether its icon is hidden. NSStatusItem.isVisible tells you whether the app or user wants the icon to be visible, but it will return true if the icon is hidden in the notch—or even if it’s hidden behind a menu title.
\n
\n\n
First-party workarounds
\n\n
If you prefer not to install any third-party apps (like me), there are a number of steps you can take to alleviate your crowded menu bar and possibly remedy the issue entirely (which I’ve successfully done). My goal with the steps below is to get all of my apps and icons to display when Finder is active. Some applications have more menu items that span across the other side of the notch, and in this scenario there is not much we can do to prevent hidden menu bar icons — which was actually the case before the notch existed for applications with a very large number of menu items.
\n\n\n
\n
Move macOS system-provided icons into Control Center. A number of icons can be configured to display only in Control Center, which frees up space in your menu bar for icons and third-party apps that must be displayed in the menu bar. For example, you can move WiFi, Bluetooth, Battery level, AirDrop, etc. into Control Center. Additionally, some icons can be configured to only display when they are active, like Focus status or Screen Mirroring.
\n
\n
\n
Reorder your apps and icons from right to left to display the ones that are most important to you on the right and least important on the left. The result is that the least important icons are the ones that get hidden by application menus or the notch, while the most important icons are more likely to remain visible. For example, I like having Time Machine in the menu bar, but it doesn’t matter if it gets hidden while I’m working in an app with a large menu that spans to the other side of the notch, thus hiding Time Machine. Therefore, I have placed Time Machine in the left-most position. Furthermore, I place system icons that display only when active (like Focus status) in the right most position. This is important for me, because I want to make sure I always see when these things are active.
\n
\n
\n
Reduce the menu bar item spacing and padding via UserDefaults. (Thanks to Oliver Busch for the tip. Also see this Reddit post.) There are two defaults settings you can configure via Terminal, NSStatusItemSpacing and NSStatusItemSelectionPadding.
After some experimentation, I landed on the values above — 12 for spacing and 8 for padding fit my needs. You should experiment as well. The smallest tolerable values are probably around 6 or 8.
\n\n
Remove the values to restore the default behavior:
There are several third-party applications to help organize and manage all of your menu bar apps and icons. The apps I have listed below seem to capture the three broad categories of possible solutions. I know there are a few others out there with similar functionality, but I think these are the most representative.
\n\n\n
\n
Bartender. (Paid with Free Trial. Website). This one is very advanced and complex with elaborate customization options, including styling the entire menu bar. It works well, but I found it to be very heavy-handed and a bit cumbersome for my needs. It felt clunky and glitchy to me, in ways that I think the developer has tried hard to mitigate, but I imagine Apple does not make developing this sort of app easy. (I don’t think there are any public APIs for this stuff.) Bartender seems to rely on some hacks (like screen recording) to dynamically hide and show your overflowed menu bar apps. Like iOS, macOS now has a privacy feature where you are notified when apps are using the camera, microphone, screen recording, etc. On iOS there’s a little dot indicator that shows in the status bar, on macOS there’s an icon that displays in your menu bar. One specific issue I had with Bartender, was that the privacy indicator icon for screen recording appears very frequently, which was annoying.
\n
\n
\n
Hidden Bar. (Free. Mac App Store, GitHub). This one is extremely lightweight, providing a simple chevron “<” icon to expand and collapse the extra icons. You can customize which icons are always shown, and which are hidden when the menu is collapsed.
\n
\n
\n
Say No to Notch. (Free with IAPs, Mac App Store). This is another heavy-handed approach. This one adds a letterboxed-style black bar to the top of your screen and shifts the entire menu bar down below the notch — along with the entire contents of your screen. I do not like the reduced screen real estate that this creates.
\n
\n\n\n
All of these apps have pros and cons. Unfortunately, I was not quite satisfied with any of them. However, your experience and preferences might be very different!
As noted above, I have opted for the collection of first-party workarounds. For me, all of the third-party apps felt cumbersome, wonky, lacking, or simply did not match my aesthetic taste. That’s not to say they aren’t great apps and creative solutions — they just aren’t for me. In any case, I hope this post has been helpful if you’ve been searching for solutions to the MacBook notch problem.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2024/04/29/sdk-privacy-manifests/#updated-30-april-2024",
"url": "https://www.jessesquires.com/blog/2024/04/29/sdk-privacy-manifests/",
"title": "[Updated] The curious case of Apple's third-party SDK list for privacy manifests",
"date_published": "2024-04-30T09:53:30-07:00",
"date_modified": "2024-04-30T09:53:30-07:00",
"summary": "
At last year’s WWDC, Apple introduced privacy manifests. They recently sent out a reminder that the deadline for complying with these new requirements is May 1. Privacy manifests expand on the previously introduced privacy “nutrition labels” that are self-reported by developers and displayed on the App Store. Developers must start including a privacy manifest in their apps by the aforementioned deadline, but what’s more interesting is that Apple is, for the first time, imposing these new privacy rules on third-party SDKs as well. Even more interesting is the list of SDKs that Apple has published, which, upon inspection is quite bizarre.
At last year’s WWDC, Apple introduced privacy manifests. They recently sent out a reminder that the deadline for complying with these new requirements is May 1. Privacy manifests expand on the previously introduced privacy “nutrition labels” that are self-reported by developers and displayed on the App Store. Developers must start including a privacy manifest in their apps by the aforementioned deadline, but what’s more interesting is that Apple is, for the first time, imposing these new privacy rules on third-party SDKs as well. Even more interesting is the list of SDKs that Apple has published, which, upon inspection is quite bizarre.
\n\n\n\n
Historically, Apple has rarely, if ever, explicitly acknowledged any third-party SDK or library. It took years for them to even acknowledge community tools like CocoaPods in Xcode’s release notes (usually when they made a change that broke it). Thus, it is interesting to see which SDKs they have deemed important or concerning enough to explicitly mandate a privacy manifest. And, in typical Apple fashion, I’m pretty sure SDKs authors were not notified about this in advance. We all learned which SDKs need privacy manifests at the same time — when the list was published.
AppAuth, an SDK for communicating with OAuth 2.0 and OpenID Connect providers.
\n
AFNetworking and it’s successor Alamofire, networking libraries that wrap Apple’s APIs, which almost every iOS developer has encountered.
\n
BoringSSL, a fork of OpenSSL maintained by Google.
\n
\n\n
I can see how these libraries could be concerning with regard to user privacy, they are all dealing with networking, authentication, and security (except for Abseil) — these are common vectors for privacy-related issues. Abseil is the exception, but I could see an argument for why a low-level C++ library might be a concern. There are also a lot of SDKs from Google and Facebook on the list — neither of those companies have a particularly good reputation when it comes to user privacy. It makes sense for those to be included.
\n\n
But then… you see that the list contains UI libraries that haven’t seen significant updates or any activity for multiple years, like SVProgressHUD. Why does a library that provides a single UI component need a privacy manifest? Is it as concerning and as potentially privacy invasive as the Facebook SDK? Some of the UI-only SDKs on the list haven’t seen significant updates (or any updates at all) within the last 4-5 years. Furthermore, even AFNetworking hasn’t had an update in 4 years because it was deprecated long ago after being supplanted by Alamofire. The AFNetworking repo on GitHub has been archived and read-only for over a year! Who’s going to bother adding a privacy manifest to that?
\n\n
And then… there are some entries that are simply obscure and absurd: connectivity_plus, image_picker_ios, video_player_avfoundation, file_picker. What the hell are those?! They don’t even sound like SDK or library names. I have never heard of any of these, and I’ve been involved in the iOS open source community for a decade.
\n\n
And then… you know what’s even more bizarre about this list? There are no links! There are no links to the SDK project homepages or GitHub repos. It is a plain text list of names, and in some cases, seemingly random names like “file_picker”. Ok LOL. SDK and library names are not necessarily unique. How are you supposed to know exactly which SDKs they are referencing with so little information? Searching for “file_picker” or “image_picker_ios” or any of the other obscure names on both CocoaPods and the Swift Package Index returns no results!
\n\n
Finally, wouldn’t you expect some sort of reason or justification for each of these SDKs being on the list? We don’t need a 10-page essay but a brief explanation of a few sentences explaining why each of these SDKs is on the list would be helpful in understanding the logic and reasoning behind it.
As many readers have pointed out, there are also a number of popular SDKs that really should be on this list if Apple is concerned about privacy. For example, the TikTok SDK, GoogleAds, and the Unity Ads SDK are all missing from the list, just to name a few. How strange!
\n\n
And apparently, all of the obscure SDK names like “file_picker” are actually Flutter packages. Again, what an odd list!
\n\n
\n\n
* * *
\n\n
For a company that has positioned itself as a staunch privacy advocate, this list of SDKs is slapdash at best. The lack of attention to detail, like simply including links to SDK homepages, makes the list appear like it was assembled hastily and carelessly. It makes you wonder, how was this list compiled? What was the criteria for including or excluding an SDK from this list?
\n\n
I was venting about the list on Mastodon, and the general consensus is that it was most likely just a script dump from a static analysis of app binaries on the app store, with the sole criterion being “what are the most popular libraries” used across all apps, with some minimum threshold for inclusion. It is quite clear from the list that no one at Apple really put much thought into it. 🤡
\n\n
If we operate under the hypothesis that this list is merely the output of a script that someone at Apple wrote to check off the line item “determine which third-party SDKs should be required to included privacy manifests”, then it all starts to make more sense. This list is ultimately the result of a popularity contest, not a thoughtful analysis of SDKs that have meaningful implications for user privacy. They couldn’t even bother to link to the projects or provide brief explanations. There’s literally an entry titled “file_picker” with no other explanation. Did anyone at Apple even look into any of these libraries? Did anyone at Apple even read through this list after some script vomited it out?
\n\n
When Apple imposes new privacy regulations in such a slipshod manner, how are we, as developers and as users, supposed to take this seriously? This feels like more bureaucratic security and privacy theater. Let’s all take off our shoes and throw away sealed bottles of water we purchased at the airport before we proceed through the TSA security checkpoint — meanwhile doors and wheels are falling off the damn plane.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2023/12/06/xcode-shell-env/#updated-15-december-2023",
"url": "https://www.jessesquires.com/blog/2023/12/06/xcode-shell-env/",
"title": "[Updated] Xcode does not have access to your shell environment",
"date_published": "2023-12-15T15:50:32-08:00",
"date_modified": "2023-12-15T15:50:32-08:00",
"summary": "
I recently discovered, while setting up my first Apple Silicon Mac, that Xcode does not have access to your shell environment. But there’s one caveat to that. (Thanks to Boris for confirming!) This post will hopefully be a reminder to my future self when I encounter this issue again.
I recently discovered, while setting up my first Apple Silicon Mac, that Xcode does not have access to your shell environment. But there’s one caveat to that. (Thanks to Boris for confirming!) This post will hopefully be a reminder to my future self when I encounter this issue again.
\n\n\n\n
I realized the problem when an Xcode build failed because a Run Script Build Phase that runs SwiftLint failed because the SwiftLint binary could not be found. I have SwiftLint installed via Homebrew. I knew SwiftLint was in my PATH because I could run it successfully on the command line. This meant Xcode did not have access to my PATH (and, as I learned later, my entire shell environment).
\n\n
This only started to happen after switching from an Intel Mac to an Apple Silicon Mac — and that was the issue. Homebrew installs at different locations for Intel (/usr/local) and Apple Silicon (/opt/homebrew). Invoking SwiftLint from a Run Script Build Phase in Xcode on Intel Macs just happens to work because /usr/local is part of your default PATH. To fix the issue on Apple Silicon Macs, you need to tell Xcode where to look for Homebrew packages in your script.
\n\n
export PATH=\"$PATH:/opt/homebrew/bin\"\n
\n\n
* * *
\n\n
Anyway, the moral of this story is that Xcode does not have access to your shell environment. This is not necessarily an obvious thing, especially on Intel machines. Xcode only gets whatever the default PATH is, configured via launch services — unless you launch it through a terminal. What this means is that xcodebuild run from terminal and Xcode could end up with potentially different behavior in their Run Script Build Phases because they expose different environments. Good to know!
Don’t forget you can still set environment variables from your project’s scheme configuration. Edit your scheme from the Product menu, select the Arguments tab against the Run behaviour, and set Environment Variables.
Welcome to the third part of my going indie series! In the previous post, I discussed building a foundation, getting started, and finding clients. In this post, I am going to discuss many of the decidedly un-fun administrative aspects of being freelance and contracting like saving for retirement and — everyone’s favorite — taxes. Most folks consider these topics to be boring and tedious, but understanding them is critical to your success. The best approach is one of curiosity. As a software developer, you might find the task of optimizing (and minimizing!) your tax burden to be an interesting problem to solve — I definitely do!
Welcome to the third part of my going indie series! In the previous post, I discussed building a foundation, getting started, and finding clients. In this post, I am going to discuss many of the decidedly un-fun administrative aspects of being freelance and contracting like saving for retirement and — everyone’s favorite — taxes. Most folks consider these topics to be boring and tedious, but understanding them is critical to your success. The best approach is one of curiosity. As a software developer, you might find the task of optimizing (and minimizing!) your tax burden to be an interesting problem to solve — I definitely do!
\n\n\n\n
\n
Disclaimer: I am not an accountant, nor a financial advisor. I highly recommend you get both! All the information in this post is derived from my experience, conversations with my accountant, and advice from my financial advisor. Of course, your tax situation and financial situation differ from mine, so “your mileage may vary” as they say. For example, I do not have children, but if you do that has a significant impact on your taxes and finances.
\n\n
Also, as I noted in my first post, this series assumes you are based in the United States. While many aspects of this series are widely applicable, this post is very specific to the US.
\n
\n\n
What you may not realize is that going indie means starting your own business. Congratulations, you are now a small business owner as far as the IRS is concerned. Don’t worry, that does not mean some sort of formal business entity is necessary (as you will see below). It only means you need to shift your thinking a bit. My goal with this post is to give you a head start on learning how to structure your business, and what to expect regarding taxes. My hope is that you can begin your journey with more information than I had — which was literally zero.
\n\n
If you do not have an accountant or financial advisor, I highly recommend you get both. Find someone you trust. Their expertise will help guide you in making the correct decisions for yourself and they will save you a ton of time and potential headaches. Likely, someone in your network can make a recommendation for both.
\n\n
Business structure
\n\n
The first task after deciding to go independent and do freelance/contract work is deciding how to structure your self-employed business. You have a few options, each with its own pros and cons. The factors that differ between the options primarily center around taxes, reporting, paperwork, and bookkeeping. There are many resources on the internet that explain the differences, benefits, and drawbacks of each self-employed business entity — so I will be brief. I am not an expert on these topics, so you should ultimately consult your accountant. My goal here is to make you aware of your options and give you the gist of each.
\n\n
In all scenarios, you should open a new bank account for your business. It’s also a good idea to get a separate credit card that you use solely for business expenses. This is not strictly necessary as a sole proprietor (and the accounts can be normal accounts in your name, not actual business accounts), but it is a good practice to keep everything separate. This will make bookkeeping significantly easier for you.
\n\n\n
\n
Sole Proprietorship. [IRS, Wiki] A Sole Proprietorship is owned and run by one person and in which there is no legal distinction between the owner and the business entity. Being a sole proprietor is the simplest and easiest structure. You do not need to establish a formal legal entity. There is no paperwork or setup. You operate as yourself, as an individual, under your legal name. Similar to being a full-time W-2 worker, you file taxes using your social security number (SSN), etc. You can (optionally) “level-up” by getting a Fictitious Business Name (FBN), which allows you to operate under a different “business” name. However, as the name indicates, this is not a “real” business entity. It merely provides a facade to your sole proprietorship, which can be useful in some situations. If you opt to have an FBN, then you also need to get an Employer Identification Number (EIN) to associate with it. An EIN is basically an “SSN for your business” — you file your taxes using the EIN instead of your SSN and you provide your EIN to your clients for your 1099 forms. In addition to income tax, you also pay self-employment tax. You “pay yourself” simply by withdrawing or transferring funds from your business account.
\n
\n
\n
S Corporation (S-Corp). [IRS, Wiki] An S-Corp is a more formal structure that requires you to establish a payroll system and pay yourself a salary. You need an EIN and there is various paperwork to complete to establish the S-Corp entity. The process can take a couple weeks or up to 3 months, depending on your circumstances. It differs in that you can have multiple shareholders. An S-Corp is similar to a sole proprietorship in that it is a “pass-through entity” meaning the corporation’s income and losses are divided among and passed through to its shareholders (i.e., you). In this scenario, you are the singular shareholder so you more or less file taxes like an individual would. However, S-Corps have access to additional deductions and can yield higher tax savings. Notably, an S-Corp is not a “full” corporation and alleviates you from double taxation where you would otherwise pay corporate tax as well as individual income tax. You still pay self-employment tax like a sole proprietor, and because you have to run a payroll system you also pay payroll taxes. You “pay yourself” via your payroll system. S-Corps also come with some additional fees and specific states impose their own taxes on S-Corps. Also note that some states tax S-Corps as C-Corps, negating the pass-through benefit. Because of the additional fees and taxes, it usually does not make financial sense to establish an S-Corp until your annual earnings pass a threshold of around $100,000 — at which point you are better positioned to reap the tax benefits. (Again, consult your accountant here.)
\n
\n
\n
Limited Liability Company (LLC). [IRS, Wiki] An LLC is what I know the least about. It is an established business entity that you also have to file paperwork to legally setup, pay fees, get an EIN, etc. An LLC does not need to have payroll. It is not a corporation but it is a legal form of a company. It is a hybrid business structure that can combine the pass-through taxation of a sole proprietorship with the limited liability of a corporation. Regulation of LLCs varies from state to state. LLCs generally have fewer formalities and reporting requirements than S-Corps, which can make them easier and less expensive to manage.
\n
\n\n\n
My advice is: do the simplest thing first. If you are just getting started, operate as a sole proprietor. You do not need to complete any legal paperwork and you can start working immediately. This is beneficial because if you decide that you do not like freelancing and contracting, then you can stop and go back to full-time work without having to do anything else. At the same time, if you do decide to stay independent, then it is easy to transition to an S-Corp or LLC later. I have a friend that established an LLC once for a project that eventually fizzled out, and he said the process to dissolve it was a nightmare. He wished he had just gone with a sole proprietorship first.
\n\n
Finally, one important caveat is that sole proprietorships do not shield you from liability, whereas S-Corps and LLCs do. For example, say something terrible happens and a client files a lawsuit against you. With an S-Corp or an LLC, your personal assets are protected. That is, the client is suing the business entity and they can only go after the business entity’s assets. If you are a sole proprietor, then the client is suing you personally. Technically, that puts everything you own at risk of seizure and liquidation — bank accounts, investments, property. However — this is extremely unlikely and especially rare in our line of work. You are not going to get sued for a software bug. (I mean, maybe you could? But let’s be real.)
\n\n
If you were starting a construction business or a restaurant, that is a very different story with very real scenarios involving liability. So do not allow the issue of liability scare you away from starting with a sole proprietorship! Tons of people are sole proprietors without any problems. It is important to weigh the potential risks of your type of business when choosing the appropriate structure. For software development and design, that risk is extremely low. Still, ask your accountant!
\n\n
Taxes
\n\n
Taxes are terrible. Everyone knows this. And they are even worse when you are self-employed. However, you can mitigate your tax burden with deductions, which I’ll discuss below. Regardless of your business structure you will pay your normal income tax, which is the same as when you are a full-time worker at a company. The same federal and state income tax brackets apply.
\n\n
What differs when you are self-employed is that you must also pay self-employment tax, or SECA (named after the Self-Employed Contributions Act). SECA taxes include Social Security and Medicare. When you are a full-time worker at a company, you also pay these taxes (called FICA, per the Federal Insurance Contributions Act) and you can see the deductions taken from your paychecks. So what’s the difference? When you work at a company, the company pays half of these taxes (FICA) and you pay the other half. When you are self-employed, you pay the full amount (SECA) — because you are the employeeandthe employer. The total self-employment tax is 15.3 percent. When you work at a company, you and the company each pay 7.65 percent.
\n\n
There are a lot of details that go into these calculations that I will not cover in this post, but you are in luck — I made an app for that! Taxatio is tax calculator for freelancers that I released last year. It is available for iOS and macOS. It will help you with these calculations and tax estimations, as well as explain what’s going on with detailed breakdowns.
\n\n
* * *
\n\n
I would be remiss if I did not take this opportunity to highlight the injustices of our tax system. SECA/FICA taxes are regressive taxes — the more money you make, the less you pay as a percentage of your total income. This imposes an undue burden on the poor and lower wage workers, not to mention the rich usually pay nothing. SECA/FICA are taxes on earned income, that is, income you earn through labor. Because the rich do not work, but rather “earn” money through their pre-existing assets and investments, they pay nothing into Social Security and Medicare. Furthermore, their wealth — that is, their property, investments, inheritance, and other assets — is taxed at significantly lower rates. Thus, the rich get richer simply because they own things all while doing no labor whatsoever, yet they still reap the benefits of our decaying social safety net that real workers provide.
\n\n
The more you learn about our tax system, the more obvious it becomes that income taxes are designed to punish poor and working class people — and if you must work to pay your bills and survive, then you are working class. You may consider yourself middle class, but this is a facade propped up by our precarious and predatory credit system. Being “middle class” is a status that can be revoked at a moment’s notice, as we witnessed during the 2008 housing crisis. The middle class is an artificial bifurcation of working people manufactured by the rich via credit that deludes us into thinking there exists a ladder we can simply climb to join the wealthy elite at the top. It is a scam. No one earns a billion dollars — they steal it. (And before someone tries to lecture me about the “risk” of investments — there is no “risk” when you have a literal mountain of money and wealth at your disposal, not to mention a government that will bail you out after you make billions of dollars in derivatives evaporate overnight.)
\n\n
SECA taxes have a significant impact on independent workers. It is wrong that you must pay double as a single person, as if you have the same influence and wealth as an entire corporation. Even full-time workers should not have to split the burden of FICA taxes with corporations reap millions in profits while suppressing wages and destroying our environment with impunity.
\n\n
But, I digress.
\n\n
* * *
\n\n
Estimated taxes
\n\n
The most significant difference regarding taxes when you are self-employed and doing 1099 freelance/contract work is how and when you pay taxes. Your clients do not do any tax withholding. When you are a full-time W-2 worker, FICA taxes are withheld from each paycheck you receive. When you are self-employed, you must pay quarterly estimated taxes. This includes federal and state income tax (unless, of course, you live in a state that does not collect income tax). Instead of paying a small amount on each paycheck, you pay lump sum once per quarter.
\n\n
Take note — quarterly estimated taxes are no joke. As an example, suppose your adjusted gross income for the year is $200,000. Your quarterly estimated tax payments may be as high as $18,000 per quarter. That’s a lot of money to drop at once. This means you need to be prepared. When clients pay you, do not withdraw the full amount from your business account to pay yourself. You always need to leave money in your business account for tax payments. Again, you are in luck because Taxatio will help you here by showing you the complete quarterly payments breakdown.
\n\n
The hardest part about estimated taxes is… well, estimating them! When you first get started, this is a moving target. You are not sure how much you will make for the year. Do your best to make projections. Your accountant can help you here, too. After your first year of being independent is over, you can use the previous year’s income as your estimate for the next year. If you do not pay your quarterlies on time, in full, or at all, you will get hit with fees and penalties when you file your tax returns.
\n\n
One reason I recommend going independent at the beginning of the calendar year is to simplify your taxes and estimates. If you start your freelance/contract work during the same year that you quit your full-time job, then you will have mixed income: some W-2, some 1099. This makes your estimations a bit more of a moving target, and they will not stabilize until after your second year — that is, your first full calendar year of being independent. However, you should not worry about this too much.
\n\n
There is one nice thing about estimated taxes (and using Taxatio to plan). If you have accurate projections for your annual income, like me, that means you have an accurate understanding of your annual tax burden. This allows me to optimize my time and work. For example, once I have made enough money for the year and know I have enough to cover my taxes, I can stop working. Historically, this means I can not work for 2-3 months each year.
\n\n
Conveniently, you can pay (and automatically schedule!) your federal estimated tax payments by creating an account with EFTPS.gov. For state estimated taxes, you will need to figure out the equivalent for your state. If you live in California, like me, then you can create an account with the California FTB where you can similarly schedule and pay your estimated payments.
\n\n
Deductions
\n\n
Of course, we cannot discuss taxes without also discussing the available mechanisms we can employ to reduce them. In the example above, we discovered than an annual income of $200,000 could result in quarterly tax payments around $18,000. However, you can dramatically decrease this amount with deductions.
\n\n
It is important to understand the difference between your gross income (the total amount of money you earn) and your taxable income (the portion of your income that is subject to income tax). Deductions are subtracted from your gross income, which results in reducing your taxable income. Some of the most common deductions include health insurance premiums and a home office. Consult your accountant for advice.
\n\n
In part one I mentioned that you can purchase health insurance via the Affordable Care Act (ACA). Every state is different, but I have been happy with my experience in California. If you get health insurance (which, you probably should), the entirety of your premiums are tax deductible. Also note that failing to purchase insurance coverage will result in a tax penalty.
\n\n
Another substantial deduction is self-employment tax (SECA). You are allowed to deduct half of SECA — that is, the “employer” portion. While I think paying double is an unnecessary burden on sole proprietors, it does help to be able to deduct this.
\n\n
Finally, and most importantly, you should be keeping track of your expenses. Expenses are subtracted from your gross income, which again results in reducing your taxable income. I will discuss bookkeeping in detail in the next part of this series.
\n\n
And, of course, Taxatio does these calculations and will show you a breakdown. The current version is limited with a single field for your total amount of deductions, but a future update will expand this functionality to allow a full itemized list.
\n\n
Saving for retirement
\n\n
The last significant piece of this puzzle involving business structure, taxes, and deductions is how you can continue to save for retirement despite no longer working full-time at a company. Typically, most companies provide retirement plans and many often offer matching contributions. Most commonly, you will have the opportunity to open a 401(k) plan. If you are lucky, the company will match a portion of your contributions.
\n\n
In my experience, many folks are unaware that as a self-employed business owner you can open a Solo 401(k) plan. The rules are the same in terms of the maximum annual contribution amounts — except you are the employeeandthe company. This means you can contribute as an individual and then “match yourself” as the company. In 2023, the maximum individual contribution is $22,500. The maximum company contribution is $43,500 or 25% of your earned income — whichever is smaller. This means you can put up to $66,000 into your Solo 401(k) per year, assuming you make enough to maximize the company contribution. Yes — that is a sweet fucking deal. You might be missing out on stock options at a company, but you can dump a ton of savings into retirement.
\n\n
With a 401(k), your contributions can be traditional (pre-tax) or Roth (after-tax). The difference is that traditional contributions reduce your taxable income, thus reducing the taxes you pay now. However, you will then pay taxes on the funds you withdraw during retirement. For Roth, you pay taxes now before making contributions and then when you withdraw funds during retirement, they are not taxed. The question to ask yourself is: do you want to pay taxes now, or later?
\n\n
The other popular option is a Roth IRA. However, the maximum contribution for 2023 is only $6,500. You can have both a Solo 401(k) and an IRA, if you want. Be aware that there are many rules and regulations regarding contributions. I do not recommend trying to setup these accounts on your own. If you must choose one, a Solo 401(k) is often the better choice as it allows you to save significantly more. However, paperwork for an IRA can be much simpler, so that might be a good start depending on your circumstances.
\n\n
For all of these important decisions, you should consult your financial advisor to make the choices that work best for you and your financial goals.
\n\n
Finally, do I need to say it? Taxatio does these calculations for you. Currently, it only supports Solo 401(k) plans, but I would like to add support for Roth IRAs in a future update.
\n\n
Conclusion
\n\n
Today we covered some ugly, mundane — but nonetheless important — details of going independent. I hope this post was helpful if you are considering going independent. There is still more to come in this series. In the next part, I will discuss bookkeeping and invoicing! Stay tuned.
Many thanks to Michael Tsai and @ljmatkins for their feedback on this post regarding details about LLCs and retirement plans. I have updated this post to reflect their corrections and other notes.
UserDefaults is probably one of the most popular APIs on Apple Platforms. It is a highly-optimized key-value persisted store that is backed by a property list, and it is most commonly used for saving small pieces of data like user preferences. Despite its ease-of-use, there is one common anti-pattern I see developers use often.
UserDefaults is probably one of the most popular APIs on Apple Platforms. It is a highly-optimized key-value persisted store that is backed by a property list, and it is most commonly used for saving small pieces of data like user preferences. Despite its ease-of-use, there is one common anti-pattern I see developers use often.
\n\n\n\n
In dozens of iOS projects over the years, I find that developers are prefixing key names with the app’s bundle identifier. For example, instead of naming a key \"sounds-enabled\" it will be named \"com.mycompany.MyApp.sounds-enabled\". I am writing this post as a PSA to tell you that you do not need to do this!
\n\n
On iOS, in your application’s user data sandbox you will find the backing plist for UserDefaults at ~/Library/Preferences/com.mycompany.MyApp.plist. On macOS, for a non-sanboxed app you will find the plist at the same path and for an app that is sandboxed you will find the plist at ~/Library/Containers/MyApp/Data/Library/Preferences/com.mycompany.MyApp.plist.
\n\n
As you can see, your instance of UserDefaults is already namespaced by your app’s bundle identifier! Prefixing your keys is redundant. So please — do not prefix your key names with your app’s bundle identifier!
\n\n
* * *
\n\n
You might argue that prefixing key names is harmless, and that could be true sometimes. However, there are known issues with UserDefaultskey-value observing if key names have periods in the name — KVO does not work in that scenario. Furthermore, if you use debug tools (like FLEX) that allow you to inspect the values saved in UserDefaults or if you manually open the plist (which is just XML) to examine it, it gets really cumbersome to search and read through if every single key is prefixed with the same ~30 characters.
\n\n
Lastly, if you are convinced to drop the unnecessary prefixes, beware that you will need to write migration code to save your values from the old key to the new one, otherwise you will lose data.
A few folks on Mastodon have pointed out situations where prefixing might be warranted. Or rather, pointed out problematic scenarios to generally watch out for.
\n\n
Greg Heo mentioned that you should carefully consider your key names if you are using UserDefaultsshared domains. Typically, this scenario would be sharing data between your main app and an extension — so hopefully you are not colliding with your own key names.
\n\n
Matt Massicotte noted that some system frameworks store values in your app’s UserDefaults. For example, AppKit and SwiftUI on macOS store state restoration data, most commonly window size and location. I had forgotten about this. However, I highly doubt you’ll have naming collisions. For example, here are a couple of keys for one of my apps: NSWindow Frame com_apple_SwiftUI_Settings_window, com_apple_SwiftUI_Settings_selectedTabIndex.
\n\n
More importantly, Andy Ibanez warned that a third-party dependency might read and write to the same key name. That is very bad! If you are a library author, you should not be writing to UserDefaults.standard. This is why UserDefaults.init(suiteName:) exists — your library should initialize its own suite.
\n\n
All of these are valid concerns. Still, I think for the majority of use cases, prefixing is not necessary.
Xcode has a great UI for setting and editing breakpoints. I use breakpoints all the time while working and debugging, but I want to share another, unconventional way that I use them.
Xcode has a great UI for setting and editing breakpoints. I use breakpoints all the time while working and debugging, but I want to share another, unconventional way that I use them.
I’m happy to share that bookmarks are now a first-class feature in Xcode 15! We no longer have to hack breakpoints like I describe in this post. You can learn more about how they work in Sarun’s recent blog post.
\n\n
\n\n
One of the best features is being able to set and keep breakpoints in place, but quickly enable or disable them individually. In the sidebar, you can quickly see a list of all your breakpoints in the breakpoint navigator and toggle them on or off. Even better, Xcode preserves them across app launches for each project, but they do not affect your git working directory. This makes disabled breakpoints perfect “bookmarks” in your code, that only you can see.
\n\n
Any time I am exploring or getting familiar with a new codebase in Xcode, especially very large projects, I use disabled breakpoints as “bookmarks” to keep track of where I am, where I have been, and things I want to remember or need to revisit. Sometimes I even do this when debugging issues in codebases that I know well.
\n\n
This “bookmarking” is particularly useful when tracing through a long path of function calls to understand how something works, or if you are frequently jumping around (via “jump-to-definition”) to view the full declaration of new types that you encounter.
\n\n
When attempting to fix that first “starter bug” you get assigned, it is hard to know exactly which files you will need to edit to fix it. Perhaps a veteran maintainer on the project gets you started with a class name, or the name of a view controller. As you track down the bug, you confront dozens of new types for the first time. All of those different types are interacting together, and any of them may be relevant to fixing the issue. It is difficult to remember everything you have seen once you start digging around. Unobtrusively clicking in the gutter of the editor to add a breakpoint as you go along can be a huge help. And after browsing through dozens of files for the first time, you will have a list of relevant locations to investigate further, or examine tomorrow.
\n\n
Next time you are thrown into a new project, trying leaving some disabled breakpoint breadcrumbs so you can find your way back.
Epsilon. ε. The fifth letter of the Greek alphabet. In calculus, an arbitrarily small positive quantity. In formal language theory, the empty string. In the theory of computation, the empty transition of an automaton. In the ISO C Standard, 1.19e-07 for single precision and 2.22e-16 for double precision.
\n\n
The other day I was attempting to use FLT_EPSILON (which I later learned was laughably incorrect) when the Swift 4 compiler emitted a warning saying that FLT_EPSILON is deprecated and to use .ulpOfOne instead. What the hell is ulpOfOne? I read the documentation and then everything made sense — ha, just kidding. The FloatingPoint.ulpOfOne docs generously describe the static variable as the unit in the last place of 1.0 — whatever that means. Let’s find out.
Epsilon. ε. The fifth letter of the Greek alphabet. In calculus, an arbitrarily small positive quantity. In formal language theory, the empty string. In the theory of computation, the empty transition of an automaton. In the ISO C Standard, 1.19e-07 for single precision and 2.22e-16 for double precision.
\n\n
The other day I was attempting to use FLT_EPSILON (which I later learned was laughably incorrect) when the Swift 4 compiler emitted a warning saying that FLT_EPSILON is deprecated and to use .ulpOfOne instead. What the hell is ulpOfOne? I read the documentation and then everything made sense — ha, just kidding. The FloatingPoint.ulpOfOne docs generously describe the static variable as the unit in the last place of 1.0 — whatever that means. Let’s find out.
I gave a talk on the topics discussed in this post and more at iOS Conf Singapore. You can watch the video here!
\n\n
\n\n
\n \n
\n \n \n
\n
\n \n
\n
\n \n
\n \n
\n\n
Aside: new protocols in Swift 4
\n\n
The various numeric protocols got a facelift in Swift 4 (as well as in Swift 3). There’s a new, top-level Numeric protocol, some new integer proposals, numerous API additions to the FloatingPoint protocol (including ulpOfOne), as well as additions to the other numeric protocols.
SE-0113: Add integral rounding functions to FloatingPoint(Swift 3)
\n
SE-0067: Enhanced Floating Point Protocols(Swift 3)
\n
\n\n
ulpOfOne and ulp
\n\n
The discussion section for .ulpOfOne elaborates a bit more:
\n\n
\n
The positive difference between 1.0 and the next greater representable number. The ulpOfOne constant corresponds to the C macros FLT_EPSILON, DBL_EPSILON, and others with a similar purpose.
\n
\n\n
There’s also a computed property for Swift FloatingPoint protocol conformers called ulp. It’s a property, so you can write something like 1.0.ulp or 3.14159.ulp. According to the documentation, ulp is the unit in the last place of this value. So, it’s the same as .ulpOfOne but for any value. The discussion section explains:
\n\n
\n
This is the unit of the least significant digit in this value’s significand. For most numbers x, this is the difference between x and the next greater (in magnitude) representable number. There are some edge cases to be aware of […]
\n\n
This quantity, or a related quantity, is sometimes called epsilon or machine epsilon. Avoid that name because it has different meanings in different languages, which can lead to confusion, and because it suggests that it is a good tolerance to use for comparisons, which it almost never is.
\n
\n\n
For most numbers x, this is the difference between x and the next greater (in magnitude) representable number. This makes more sense — floating-point numbers are difficult to represent in computing because they are inherently imprecise due to rounding. Given the above, it’s clear why FLT_EPSILON is not intended to be used as a tolerance for comparisons. I’ll blame the terrible naming for my mistake there. 😉
NOTE: Rationale for “ulp” instead of “epsilon”:\nWe do not use that name because it is ambiguous at best and misleading at worst:
\n\n
\n
\n
Historically several definitions of “machine epsilon” have commonly been used, which differ by up to a factor of two or so. By contrast “ulp” is a term with a specific unambiguous definition.
\n
\n
\n
Some languages have used “epsilon” to refer to wildly different values, such as leastNonzeroMagnitude.
\n
\n
\n
Inexperienced users often believe that “epsilon” should be used as a tolerance for floating-point comparisons, because of the name. It is nearly always the wrong value to use for this purpose.
\n
\n
\n
\n\n
By contrast “ulp” is a term with a specific unambiguous definition. I’m not sure who wrote that, but “unambiguous” is a term I’d rarely use to describe anything regarding naming in computer science and programming. 😆 ULP is short for Unit in the Last Place, or Unit of Least Precision. It’s a unit, but what does it measure? It measures the distance from a value to the next representable value.
\n\n
Why we need ulp
\n\n
Consider integer values which can be represented exactly in binary. The distance from 0 to 1 is 1, the distance from 1 to 2 is 1, and so on. In fact, it’s always 1 — the distance between integers is uniform across all representable values and precisely equal to 1. In other words, for any value the next representable value is 1 away. This is intuitive if you imagine a number line with only integer values. We do not need any notion of “ulp” for integers. They are simple and easy to represent in a base-2 number system.
\n\n
Floating-point numbers, however, are more complex and difficult to represent in bits. In Number theory, this is the set of real numbersR, which includes the irrational numbers R/Q, the rationals Q, the integers Z, and the natural numbers N. Basically, all the numbers. 😆 While the set of all real numbers is uncountably infinite, the set of all rational numbers is countable, which means that almost all real numbers are irrational. Irrational numbers never terminate, so obviously need to be rounded. What’s more critical is that the rational numbers are densely ordered — for any two rational numbers there are infinitely many between them. In fact, there are more numbers (R/Q + Q) between 0 and 1 than there are in the entire set of integers Z.
\n\n
Computers obviously cannot represent all of the integers — from negative to positive infinity — which is why we have INT_MIN, INT_MAX, and other constants. Similarly, floating-point numbers are bound by FLT_MIN and FLT_MAX (or DBL_MIN and DBL_MAX). But more importantly, computers cannot represent all of the rational and irrational numbers between these bounds. While the representable integers are countable within their specified bounds of INT_MIN and INT_MAX, the floating-point numbers that occur between FLT_MIN and FLT_MAX are infinite. It’s clear that we need some kind of rounding mechanism, which means not all numbers are representable. Such a rounding mechanism will provide a way to determine the distance from one value to the next representable value — or, ulp. Thus, aside from the obvious limitations of silicon chips, this is why ulp needs to exist.
\n\n
\n \n
\n \n \n
\n
\n \n
\n
\n \n
\n \n
\n\n
Memory layout of floating-point numbers
\n\n
Before we define ulp, let’s briefly review the memory layout for floating-point numbers.
\n\n
First, integers are exact and can be represented directly in binary format. Signed integers are typically represented in two’s complement, but that’s an implementation detail. Conceptually, you have an integer which is stored as some bits. For example, 6 which is 0b0110. Floating-point numbers have three components: a sign, an exponent (which is biased), and a significand. For single precision, there’s 1 bit for the sign, 8 bits for the exponent, and 23 bits for the significand — 32 bits total. Double precision provides 11 bits for the exponent and 52 bits for the significand, which totals 64 bits. I won’t discuss the more elaborate details about how floating-point numbers work in this post, but you’ll find additional reading at the end.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n IEEE 754 single-precision binary floating-point format / Source \n
\n \n \n
\n
\n \n
\n \n
\n\n
Note that with this representation, positive and negative zero (+0 and -0) are two distinct values, though they compare as equal. If you ever wondered why computers have signed zero, this is why.
\n\n
Let’s look at an example in base-10. We can represent 123.45 with a significand of 12345 and exponent of -2: 123.45 = 12345 * 10e−2. In base-2, computing a value from these three components is a bit more complicated but conceptually the same. Swift provides a clear and expressive API that can really help us understand how all of this works.
Additionally, Swift’s APIs allow us to explore the memory layout. We can look at the bit patterns and verify them with this handy IEEE-754 floating-point converter.
We can also check the bit counts for each component:
\n\n
Float.exponentBitCount// 8, bits for the exponent\nFloat.significandBitCount// 23, bits for the significand\n\nDouble.exponentBitCount// 11, bits for the exponent\nDouble.significandBitCount// 52, bits for the significand\n
\n\n
Defining ulp
\n\n
Surprisingly, the IEEE standard doesn’t define ulpOfOne (or machine epsilon) explicitly, so there are a couple of varying definitions. However, most standard libraries provide constants for these values. The most prevalent is the ISO C Standard — 1.19e-07 for 32-bit values (Float) and 2.22e-16 for 64-bit values (Double). As expected, this is what we see in Swift:
Given these initial epsilon or ulpOfOne values, we can compute the ulp for any value v with an exponent exp as: epsilon * 2^exp, where 2 is the radix, or base.
Again, Swift provides a great, readable API for manipulating and exploring the properties of floating-point values. For example, for any value we can check .nextUp to see the next representable value. Given what we’ve learned so far, we can intuitively reason about what the next number (.nextUp) must be and verify our result.
letvalue=Float(3.14)\n// actually 3.14000010490417480468750 -- because rounding\n\nvalue// 3.14000010490417480468750\nvalue.ulp// 0.00000023841857910156250\nvalue+value.ulp// 3.14000034332275390625000\nvalue.nextUp// 3.14000034332275390625000\n
\n\n
Notice that the ulp of 1 is not the same as the ulp of 1,000. For each floating-point number ulp varies. In fact, the precision of a floating-point value is proportional to its magnitude. The larger a value, the less precise.
We can view the Swift standard library source, which lives in stdlib/public/core/ FloatingPointTypes.swift.gyb. If you’ve never seen .gyb (‘generate your boilerplate’) files, read Ole Begemann’s post. Per Ole’s instructions, we can generate the specific implementation for Float.
\n\n
publicvarulp:Float{\n if!isFinite{returnFloat.nan}\n ifexponentBitPattern>UInt(Float.significandBitCount){\n // self is large enough that self.ulp is normal, so we just compute its\n // exponent and construct it with a significand of zero.\n letulpExponent=\n exponentBitPattern-UInt(Float.significandBitCount)\n returnFloat(sign:.plus,\n exponentBitPattern:ulpExponent,\n significandBitPattern:0)\n }\n ifexponentBitPattern>=1{\n // self is normal but ulp is subnormal.\n letulpShift=UInt32(exponentBitPattern-1)\n returnFloat(sign:.plus,\n exponentBitPattern:0,\n significandBitPattern:1&<<ulpShift)\n }\n returnFloat(sign:.plus,\n exponentBitPattern:0,\n significandBitPattern:1)\n}\n
\n\n
Surprising at an initial glance, this is not the simple formula noted above (epsilon * 2^exponent). First there are some edge cases to handle, like Float.nan.ulp which is nan. Then it constructs a new Float after computing its constituent components — the sign, exponent, and significand. This code eventually calls into the public initializer: init(sign: exponentBitPattern: significandBitPattern:). Note that the final return is equivalent to Float.leastNonzeroMagnitude (or FLT_MIN).
This initializer eventually calls into init(bitPattern:), where it finally constructs a primitive LLVM 32-bit float (FPIEEE32) from the raw bit pattern.
We can break this down and observe each step in a Swift playground. For private APIs like Float._infinityExponent, we can look at the source and define them inline.
\n\n
// starting value\nletvalue=Float(3.1415)\n\n// 1. the first if is satisfied:\n// var ulp: Float {\n// if exponentBitPattern > UInt(Float.significandBitCount) {\n\nletulpSign=FloatingPointSign.plus// plus\nletulpExponent=value.exponentBitPattern-UInt(Float.significandBitCount)// 105\nletulpSignificandBitPattern=UInt32(0)// 0\n\n// 2. we call init with the 3 values above:\n// init(sign: FloatingPointSign,\n// exponentBitPattern: UInt,\n// significandBitPattern: UInt32) {\n\nletsignShift=Float.significandBitCount+Float.exponentBitCount// 31\nletsign=UInt32(ulpSign==.minus?1:0)// 0\n\nlet_infinityExponent=1&<<UInt(Float.exponentBitCount)-1// 255\nletexponent=UInt32(ulpExponent&_infinityExponent)// 105, or 2^-22\n\nlet_significandMask=1&<<UInt32(Float.significandBitCount)-1// 8388607\nletsignificand=UInt32(ulpSignificandBitPattern&_significandMask)// 0\n\nletsignMaskLeftShift=sign&<<UInt32(signShift)// 0\nletexponentMaskLeftShift=exponent&<<UInt32(Float.significandBitCount)// 880803840\nletbitPattern=signMaskLeftShift|exponentMaskLeftShift|significand// 880803840\n\n// 880803840 in binary is 00110100100000000000000000000000\n// Sign Exponent Significand\n// [0] [01101001] [00000000000000000000000]\n\n// 3. initialize with the computed bit pattern\nletfinalUlp=Float(bitPattern:bitPattern)// 2.384186e-07, or 0.00000023841857910156250\n
\n\n
This code is admittedly quite difficult to follow, but suffice to say it is equivalent to what we originally computed above.
So that’s ulp. There’s still a lot to unpack here. I’ve skipped over some details, but this rabbit hole gets deeper and deeper — and I had to end this post somewhere. Hopefully this was helpful to better understand ulp and a little about floating-point precision. To see how far the rabbit hole of floating-point numbers goes, check out the links below.
I recently discovered Xiaodi Wu’s excellent series of articles, Notes on numerics in Swift, which were written in 2018-2019. If you want to dive deeper into Swift’s numerics, I highly recommend reading these!
When I wrote about joining Mastodon, I said I would stay on Twitter for the moment and see what happens. Well, unsurprisingly, the service has continued to erode. It really is a shame, because I’ve found the software developer community there very helpful over the years. I met a lot of friends on Twitter, and later met them for the very first time in person at WWDC.
When I wrote about joining Mastodon, I said I would stay on Twitter for the moment and see what happens. Well, unsurprisingly, the service has continued to erode. It really is a shame, because I’ve found the software developer community there very helpful over the years. I met a lot of friends on Twitter, and later met them for the very first time in person at WWDC.
\n\n\n\n
I’ve been less active on Twitter for over a year, prior to the flippant billionaire crybaby takeover. Since then, I’ve been even less active on the site. I tried, briefly, to continue posting to both Twitter and Mastodon, but I simply don’t have time for that — especially as Twitter seems to be quickly deteriorating and increasingly unreliable. Am I seeing content I care about? Are my own tweets even being seen? Meanwhile, Mastodon continues to improve — and by improve, I mean more of my friends and connections are joining. Mastodon, as ever, is also free of ads, corporate surveillance, and manipulative algorithmic bullshit. So really, what choice did I have? I naturally gravitated away from Twitter and toward Mastodon.
\n\n
* * *
\n\n
Anyway, a recent announcement from the Twitter Dev account is why I’m writing today. Apparently, they plan to revoke free access to the Twitter API on February 9, in just a few days. (See also: Michael Tsai and Daring Fireball.) As I’ve written before, I use Zapier to automatically tweet blog posts (and IFTTT for Mastodon). I also use TweetDelete to periodically delete old tweets.
\n\n
It remains to be seen, but the expectation is that these services will stop working after Twitter makes this change. I suppose these companies could pay for Twitter API access to keep their own services running. If that happens, I expect the cost to be forwarded to users. Depending on how much they charge, I might do it — but given the progressively dismal state of Twitter, it’s hard to see how any fee would be worth it. Why automatically tweet blog posts when folks won’t even see the tweets? Why continue to invest in a platform that is clearly crumbling, especially when there is an exciting and growing alternative?
\n\n
And it gets worse. Twitter has blocked API access for a popular tool, Movetodon, which facilitates migrating from Twitter to Mastodon. You can follow this thread on Mastodon for updates from the author, Tibor Martini. Sadly, I doubt it will start working again. How petty and childish can Twitter get? I do not have time for this kind of weak-ass bullshit.
\n\n
What will transpire after February 9 is unclear at the moment, so I’m operating as if these things will simply stop working, or be too expensive for me to justify paying for them. I sure as hell will not be paying Twitter directly for API access. Had this API change happened prior to Twitter embarking on its journey into self-inflicted ruin, I would have gladly paid for these tools that save me so much time. But now? I gaze over at the state of affairs at this company and I think, you want me to pay… for this? Nah, I’ll pass.
\n\n
* * *
\n\n
Because of the announcement, I’ve run TweetDelete to delete all my content on Twitter and I won’t be participating there until we see what happens on February 9. If the automation services to tweet my blog posts continue to work, I will leave them intact. However, I won’t be active there much, if at all, going forward. Also, I will not be deleting my account. I’ll keep it for historical reasons — or just to auto-tweet blog posts, if that remains an option.
\n\n
I know deleting tweets is controversial. Many folks are interested in preserving content and preventing broken links, or saving links to tweets for reference later. My position is that social media is inherently ephemeral and should not persist indefinitely. That’s how I use it. Like in real life, I don’t record and keep transcripts of every conversation or thought that I have. If something is important, it should exist permanently outside of the guarded corporate walls of social media companies, because they could make any content inaccessible on a whim.
And if you want to join the party, you can use this invite link for Mastodon.social. As of this writing, it looks like Fedifinder, a similar tool to Movetodon, is still working. I would recommend using it while you can. It involves a more manual process than Movetodon, but it is better than nothing.
It looks like Twitter’s attempted API changes did not turn out well. It is truly comical how disorganized and chaotic the company seems to be. On February 8, I received an email from Zapier Support stating they anticipate that any Zap using the Twitter integration will stop working. Then on February 10, Zapier Support sent a second email confirming that Zaps will continue to work:
\n\n
\n
Our team has been working to ensure your Zaps continue working regardless of Twitter’s unexpected API changes. We’re happy to announce that we’ve found a solution to keep your Twitter Zaps running for the time being. No action is required from you at this time.
\n
\n\n
It’s unclear if Twitter conceded or came to an agreement with Zapier (and, presumably other services like IFTTT). Or, perhaps Zapier is using an unsanctioned workaround, like extracting the API keys from the Twitter mobile apps. (lolz)
\n\n
Anyway — for now, that’s good news! This means my blog posts should continue to be tweeted automatically. I still have no plans to engage with the service beyond this.
You may have noticed that my new blog posts and updated posts are still being tweeted automatically. A few readers have mistaken this automation as me returning to Twitter, so I want to be clear: I am not returning to Twitter. I have no plans to continue to use the service beyond the automation I have configured with Zapier, which is why I’m writing this update today.
\n\n
Twitter has officially announced their new API changes. See also: Michael Tsai’s roundup. It is unclear if my automation via Zapier will break at the end of the month, or if they will start charging me for Twitter integration. Either way, I don’t plan on fixing it nor paying for it.
\n\n
Let’s hope it keeps working. Otherwise, you know where else to find me.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2022/01/16/screen-time-is-drunk/#updated-14-february-2023",
"url": "https://www.jessesquires.com/blog/2022/01/16/screen-time-is-drunk/",
"title": "[Updated] Screen Time is drunk",
"date_published": "2023-02-14T10:28:59-08:00",
"date_modified": "2023-02-14T10:28:59-08:00",
"summary": "
Speaking of drunk software and not being in service to our possessions, Screen Time on iOS and macOS has been shockingly buggy for me lately. It reports that I spent over 22 hours on my devices in a single day last week, and nearly 10 hours on another day this week. In both instances, a significant portion of the usage is supposedly occurring after midnight.
Speaking of drunk software and not being in service to our possessions, Screen Time on iOS and macOS has been shockingly buggy for me lately. It reports that I spent over 22 hours on my devices in a single day last week, and nearly 10 hours on another day this week. In both instances, a significant portion of the usage is supposedly occurring after midnight.
\n\n\n\n
This has happened to be a number of times now, and it is always a web address that is the culprit for making up the excessive time spent. See the screenshot below. Apparently I was going wild, nonstop from midnight until just before 6pm.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n Buggy Screen Time usage report \n
\n \n \n
\n
\n \n
\n \n
\n\n
It is quite clearly a bug, as evidenced by the per-app breakdown that Screen Time provides. It reports that I visited http://hexedbits.localhost for over 18 hours, while simultaneously reporting that I used Safari for only 44 minutes. I was doing some local web development for hexedbits.com on my Mac that day using Safari, but certainly not for 18 hours.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n Screen Time showing inconsistent usage data \n
\n \n \n
\n
\n \n
\n \n
\n\n
For yesterday’s usage, Screen Time similarly reports that I visited one website for over 8 hours though used Safari for only 39 minutes. I remember the web page, which I only opened briefly on my phone. I’m not a “keep my tabs open” kind of guy. All tabs get closed by the end of every day, which makes this even more peculiar. I don’t know how Screen Time works under-the-hood, but I assume part of it is simply tracking the active processes. And so, my best guess is that Safari — on iOS and macOS — has a leak. In any case, you would think that Screen Time would be able to intelligently identity and throw out obviously bad data.
\n\n
It’s all quite unfortunate, because I feel like I can’t trust any of the Screen Time reports now. This feature has helped me catch when I’m spending too much time on my devices, which usually means I’m feeling shitty. I suppose it’s all kind of ironic — a device telling me when I’ve used it too much, even though my mental health is obviously deteriorating from said usage, and I should just fucking go outside.
I’m disappointed but not surprised to report that Screen Time is still drunk. Over a year later and this bug persists. Like before, the issue is with Safari and the web. This supports my hypothesis that there must be a memory leak or something in the tracking code.
\n\n
In this case, my Screen Time report is even more absurd than before. It thinks I spent over 24 hours each day on Friday, Saturday, and Sunday looking at a screen — specifically using github.com for a weekly total of 85 hours. See the screenshot below. There are 168 hours in a week, yet Screen Time thinks I was using a device for nearly 113 of them.
\n\n
It’s a shame, because this bug leaves Screen Time mostly useless for me. In particular, because these obviously erroneous reports skew historical data and weekly comparisons. If I’m trying to reduce my Screen Time, this bug prevents me from accurately tracking that. And isn’t that the entire point of Screen Time?
\n\n
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n UPDATE: Screen Time showing usage of over 24 hours in a day \n
Xcode 11 shipped with simctl status_bar, a tool to override the status bar values in the simulator so you can take perfect screenshots. I’ve written about this tool before here, here, and here. Unfortunately, simctl status_bar is broken in Xcode 14 with the iOS 16 simulators.
Xcode 11 shipped with simctl status_bar, a tool to override the status bar values in the simulator so you can take perfect screenshots. I’ve written about this tool before here, here, and here. Unfortunately, simctl status_bar is broken in Xcode 14 with the iOS 16 simulators.
\n\n\n\n
I noticed this bug in Xcode 14 when attempting to use my utility, Nine41 to prepare for taking screenshots for an app. The tool ran without reporting any issues, yet the status bars in the simulator did not update. I thought something might have changed in Xcode 14, so I invoked simctl status_bar manually — still nothing happened. As an extra sanity check, I also tried using Sim Genie, which also failed.
\n\n
This is a new regression with Xcode 14, but it only affects the iOS 16 simulators. I tested again with yesterday’s release of Xcode 14.2 and the issue has not been fixed. If you run your app using the iOS 15 or earlier simulators, then simctl status_bar (and Nine41) will work correctly. I have no idea what caused the regression, but perhaps it has something to do with the Dynamic Island on iOS 16?
\n\n
Luckily, you do not have to open Xcode 13 just to take screenshots. However, if your app’s minimum deployment target is iOS 16, then you currently have no choice but to take screenshots with ugly status bars.
Thanks to reader Neil Macy for letting me know that it is actually only the iOS 16.1 and 16.2 simulators that are broken. I’ve confirmed that using the iOS 16.0 simulator does work! This means you can still take perfect screenshots with a minimum deployment target of iOS 16. However, you will need to work around any features that are only available on 16.1 or 16.2 until this is fixed. This confirms my suspicion that the regression is related to the Dynamic Island — or at least the new devices released this fall.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2022/01/13/imessage-spam-and-reporting-abuse/#updated-08-november-2022",
"url": "https://www.jessesquires.com/blog/2022/01/13/imessage-spam-and-reporting-abuse/",
"title": "[Updated] iMessage spam and reporting abuse: How many taps does it take to block a bad actor?",
"date_published": "2022-11-08T11:11:16-08:00",
"date_modified": "2022-11-08T11:11:16-08:00",
"summary": "
iMessage is in the news again recently with a revival of years-old stories about “green bubbles” versus “blue bubbles” — and the social dynamics among teens who prefer blue bubbles while ostracizing their peers with green bubbles. There’s a lot to like and dislike about iMessage, but one thing that amazes me is that there is still no way to easily report abuse and the process for blocking spam is needlessly difficult.
iMessage is in the news again recently with a revival of years-old stories about “green bubbles” versus “blue bubbles” — and the social dynamics among teens who prefer blue bubbles while ostracizing their peers with green bubbles. There’s a lot to like and dislike about iMessage, but one thing that amazes me is that there is still no way to easily report abuse and the process for blocking spam is needlessly difficult.
\n\n\n\n
At this point, I get regular iMessage spam and junk on a weekly or monthly basis. It is nearly as bad as email, although all modern email clients have sophisticated blocking and filtering tools — so I rarely get bothered by email spam or phishing. Aside from the absence of tools to automate blocking and report abuse, iMessage is significantly worse than email ever was because I always receive a notification in real time. Since iMessage is my primary messaging app, I cannot turn off notifications.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n An iMessage spam message that I received this morning \n
\n \n \n
\n
\n \n
\n \n
\n\n
When you receive iMessage spam, you might think there’s a quick action available in the swipe menu to block or report. Sadly, no. You can only delete, silence notifications, or pin it — none of which prevent future junk from this spammer. Also, why would anyone want to pin this thread? Shouldn’t iMessage be smart enough to offer an option to report abuse here instead?
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n iMessage swipe actions \n
\n \n \n
\n
\n \n
\n \n
\n\n
Or, perhaps you could long-press to bring up the context menu options? Still, nothing.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n iMessage context menu options \n
\n \n \n
\n
\n \n
\n \n
\n\n
How can it be that in 2022 iMessage provides no way to report abuse like every other social media platform? (And yes, we should consider iMessage a social media platform. It has hundreds of millions of users that send billions of messages daily.) According to this Apple support document, you should see a “Report Junk” button when viewing messages from unknown senders. However, I have never seen that button. (See screenshots below.) That button never appears for unknown contacts that send me junk. Is this a bug or is that support document outdated?
\n\n
iMessage does offer a setting to “Filter Unknown Senders” but the only benefit is that it will silence notifications for unknown contacts, and it introduces additional drawbacks. In addition to making the UI even more clunky and difficult to navigate, it implements wholesale filtering of all unknown numbers. This is not what most people want or need. In today’s world there are dozens of notification and reservation systems that send texts — restaurant reservations, package delivery notifications, haircut appointment reminders, SMS-based 2-factor authentication systems, and so on. If you “Filter Unknown Senders”, then you don’t get notifications from any of these systems, most of which are very time-sensitive! This setting is too naive to be useful in any way.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n Apple Support claims there is a \"Report Junk\" option / Source \n
\n \n \n
\n
\n \n
\n \n
\n\n
The only recourse I have is to block bad actors manually every single time I receive junk. Thankfully Apple provides that functionality. The only problem is that the option to block a message is buried so deep in the UI, I bet most users never find it. How many taps does it take to block a contact on iMessage? Five! Also known as way too fucking many.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n Steps to block an iMessage sender \n
\n \n \n
\n
\n \n
\n \n
\n\n\n
Tap to view the message, even though you already know it is spam from the contact info and preview text. Note: there is no “Report Junk” button in the screenshot above.
\n
Tap the sender’s contact photo/info in the nav bar.
\n
Tap “Info” on the sender’s contact card.
\n
Tap “Block this Caller”.
\n
Tap “Block Contact”.
\n\n\n
In fact, claiming there are only 5 steps is actually quite generous. After you block a contact, you then have to unwind from all these views to get back to your iMessage inbox, which requires 3 more taps. And then you have to manually delete the message, which requires another 2 taps. In total, that’s 10 fucking taps to block a spammer and delete their message. Absolute madness is an understatement. Who designed this?!
\n\n
iMessage should offer a quick and easy way to block (and optionally report) bad actors. Rather than give me an option to pin a thread for an unknown (and obvious spammer) contact, it should offer the option to block the sender and report abuse to Apple. Apple could then maintain a list of known bad actors to help curb abuse of its platform.
Good news! Beginning in iOS 16, some of these issues have been addressed. Now when you delete a message thread from any unknown sender, there is a new option to “Delete and Report Junk” and the “Report Junk” button always appears in threads from unknown senders. (See the screenshot below.) This is great.
\n\n
Unfortunately, blocking a contact still requires the litany of steps listed above.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2022/03/25/my-website-disappeared-from-bing-and-duckduckgo/#updated-28-july-2022",
"url": "https://www.jessesquires.com/blog/2022/03/25/my-website-disappeared-from-bing-and-duckduckgo/",
"title": "[Updated] My website disappeared from Bing and DuckDuckGo",
"date_published": "2022-07-28T12:36:54-07:00",
"date_modified": "2022-07-28T12:36:54-07:00",
"summary": "
I discovered earlier this week that my website is no longer being indexed by Bing and DuckDuckGo. In fact, it appears that it has been deliberately removed from their search indexes. On Bing, rather than display a “no results” message, it displays a “Some results have been removed” message, which is very concerning. Notably, however, Google search is working fine.
I discovered earlier this week that my website is no longer being indexed by Bing and DuckDuckGo. In fact, it appears that it has been deliberately removed from their search indexes. On Bing, rather than display a “no results” message, it displays a “Some results have been removed” message, which is very concerning. Notably, however, Google search is working fine.
\n\n\n\n
I triple-checked all of my server and website configurations and settings — DNS records, robots.txt, occurrences of the noindex meta tag (there are none), etc. Everything looks fine. No issues. No errors. And again, Google search seems to be working great, which indicates that there are no problems on my end.
\n\n
* * *
\n\n
My website does not appear in any searches on Bing and DuckDuckGo, nor does a search for my name produce any relevant results. For contrast, my website is the first hit on Google when you search for my name. I’m not the kind of person who regularly searches my own name, but I discovered the issue when attempting to find one of my old blog posts. The search page on my site is implemented using a basic HTML form that is hooked up to DuckDuckGo’s site-specific search functionality. Much to my surprise, when I tried to search my site for a blog post there were zero results.
\n\n
Normally, when searching from my site you will be redirected to DuckDuckGo with the search automatically populated with <your search terms> site:jessesquires.com, and you will be able to see all pages on my site containing your search query. I know for certain that this was working at some point in the past.
\n\n
You can also use the site: operator to verify that your site is indexed. For example, site:jessesquires.com. You can see that this currently produces zero results, indicating the site is not indexed at all. To see a working example, we can check Michael Tsai’s blog using site:mjtsai.com. So, aside from the primary issue of no longer being indexed, this means search on my site is also broken.
\n\n
Similar to DuckDuckGo’s site: operator, Bing provides a url: operator which indicates whether or not your site is indexed. Again, we can try Michael’s blog using url:mjtsai.com, which produces the expected results. If you search for nonsense like url:thisIsNotARealWebsite.com, you’ll see a “There are no results for…” message. However, when you search for my website, you see a “Some results have been removed” message that links to this support page about how Bing results are delivered and why results might be removed. I don’t know, but that seems pretty bad, right? Like, my site was removed?!
\n\n
* * *
\n\n
You might be wondering what the hell does Bing have to do with any of this? (Also, do people actually use Bing?) Unlike other major search engines, DuckDuckGo does not provide any kind of webmaster tools, like Google Search Console and Bing Webmaster. Thus, you cannot can submit URLs and sitemaps directly to DuckDuckGo, nor can you request indexing of pages. Instead, DuckDuckGo gets results from various other sources, including Bing and specifically not from Google. According to various online forums, the best way to ensure your site gets indexed by DuckDuckGo is to submit it to Bing and Yandex.
\n\n
Thus, my current theory is that because my site was (seemingly) removed from Bing, it was therefore subsequently removed from DuckDuckGo. So now the question to answer is why was my website removed from Bing?
\n\n
I had a Bing Webmaster account at some point, which I used to submit my site. I remember testing it and seeing results on Bing, as well as DuckDuckGo, but this was years ago. However, I don’t know what happened to my account. I might have deleted it, but I don’t remember. Or, maybe Microsoft deleted it after a certain period of time with no activity. I basically never logged in to it after submitting my site. If my defunct Bing Webmaster account was in fact deleted, perhaps that resulted in my website being removed from the search index? Otherwise, this appears like a deliberate removal for some kind of violation of Bing’s terms? I have no idea.
\n\n
* * *
\n\n
So without a working Bing Webmaster account, I decided to create a new one. This time I logged in using my Google account. Bing Webmaster allows you to login with Google and will conveniently import all of your sites from Google Search Console. This had the effect of basically reinstating my old account, with my sites ready to go. I resubmitted my sitemaps, did a site scan, looked at the SEO reports, and checked on the search performance results in Bing Webmaster. Everything seemed ok.
\n\n
My traffic on Bing is rather low compared to Google. In the last month, Bing reports only a handful of clicks and impressions while Google reports over 10,000 clicks and over 368,000 impressions. I assume that’s because of popularity and preference amongst most software developers? I’m not sure. Anyway, all the crawl results look fine, seemingly indicating no problems with the site or Bing’s ability to index it. In fact, Bing reports that it has successfully indexed hundreds of pages on my site. The crawl information also dates back to 2017 — so I’m not crazy, this was working before. Furthermore, all results and metrics on Google Search Console look good and indicate there are no problems.
\n\n
But then, I noticed on Bing Webmaster that all search metrics dropped to zero starting March 11. Unfortunately, the Webmaster tools don’t provide any additional information or insight into why this happened. Again, the theory is that my website was deliberately removed from Bing. I also checked the “Copyright Removal Notices” page in Webmaster, which is where you can see “URLs have been flagged and are liable to be removed from Bing search results”, and there are zero notices listed.
\n\n
* * *
\n\n
This is quite the predicament. Seemingly, everything is fine. Yet, for some reason my website has been delisted from Bing. I’m perplexed. But to be honest, I’m not that concerned about Bing — I really just want my site indexed by DuckDuckGo again.
\n\n
Out of desperation to get my site back on DuckDuckGo, I debased myself and created a Yandex Search Webmaster account too. I submitted my sitemaps there, requested indexing, etc. Everything on Yandex looks good, and a site: search already produces results. I’m hoping the indexing data from Yandex will make it over to DuckDuckGo eventually.
\n\n
In the meantime, I’ve submitted a support request with Bing Webmaster to inquire about the apparent removal of my site. When I hear back, I will update this post. Otherwise, all I can do is wait and periodically check if results start to appear again.
\n\n
If you have any advice, please let me know!
\n\n
* * *
\n\n
There’s one last thing. The issue described above was also happening with another one of my websites, hexedbits.com. Google search was working, but not Bing and DuckDuckGo. However, Bing showed a standard “no results” message for HexedBits. Since submitting the URL and sitemap to Bing and Yandex, content from hexedbits.com is now appearing correctly on Bing, DuckDuckGo, and Yandex. This really has me wondering what could have gone wrong. Hopefully, the indexing issues for jessesquires.com will be resolved soon.
\n\n
* * *
\n\n
All of this emphasizes the frustrating lack of transparency around search engines and how they work. It is infuriating to be essentially helpless trying to debug and resolve issues with your site not being properly indexed. I will forever lament that search was not a core component of the Internet itself, but was instead a feature that private corporations had to bolt on top.
Good news! Well, sort of. I heard back from Bing Support. They confirmed that they received my report and have “escalated this request to the engineering team.” [Editor’s note: not April fools.]
Reader Nicolas Magand left a comment that they are also having the same problem, and shared this blog post, Bing Exploit Allows Website Owners to Deindex Competitors. The gist of the article is that Bing has an inferior search algorithm that is subject to negative SEO campaigns from malicious actors, a problem that Google does not have. The general issue of having your website de-indexed from Bing seems to be much more common than I expected.
\n\n
To reiterate, I really don’t give a shit about Bing, per se — I just want my site indexed by DuckDuckGo again. But, as noted above, it seems the best (only?) way to do that is through Bing. I sent a follow-up email to Bing Support today to check on the status of my support request.
This post is still being circulated and showing up in Google search for folks, so I want to highlight my recent follow-up post here.
\n\n
Also, another reader experiencing this issue emailed me after finding this post via Google. They also published a blog post here detailing their experience.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2022/04/22/docc-on-github-pages/#updated-25-april-2022",
"url": "https://www.jessesquires.com/blog/2022/04/22/docc-on-github-pages/",
"title": "[Updated] Using DocC on GitHub Pages: Pros and Cons",
"date_published": "2022-04-25T10:27:23-07:00",
"date_modified": "2022-04-25T10:27:23-07:00",
"summary": "
When I first wrote about DocC, I lamented the fact that it was incompatible with static hosting on GitHub Pages. Much has changed since my last post, so let’s take a fresh look. While there have been many welcome improvements to the tool, there are a few remaining issues that prevent me from adopting it for my open source projects.
When I first wrote about DocC, I lamented the fact that it was incompatible with static hosting on GitHub Pages. Much has changed since my last post, so let’s take a fresh look. While there have been many welcome improvements to the tool, there are a few remaining issues that prevent me from adopting it for my open source projects.
\n\n\n\n
My previous post mentioned a few community projects that preceded DocC, but the most popular one — and the one that I use — is Jazzy. Thus, much of my feedback here will be in relation to how DocC compares to Jazzy.
The generated docs are beautiful. I love the design and aesthetic. Dark mode is built-in, natch. The generated docs are nearly identical to Apple’s official SDK documentation. Take a look at the Swift DocC Plugin docs and compare it to UIKit. This is great for open source maintainers, because your documentation will fit right in and be immediately familiar to other Apple platform developers.
\n\n
I previously wrote: “The integration with Xcode’s Editor and Documentation Viewer is excellent. There is a command-line tool included, xcodebuild docbuild, which you can invoke from automation scripts. The overall workflow for generating and exporting documentation is built-in to Xcode and its build process, making it very easy to use.” This is still true, but now it also integrates directly with the Swift Package Manager via the aforementioned plugin.
\n\n
Adding the Swift DocC Plugin to your package is easy:
I setup an example project and repo to experiment with DocC and GitHub pages. You can find it here. For a more in-depth look at hosting on GitHub pages, I recommend Joseph Heck’s excellent post.
\n\n
Finally, you can easily preview your static site with the Swift DocC Plugin.
\n\n
swift package --disable-sandbox preview-documentation --target MySwiftPackage\n
\n\n
This runs a local web server, and you navigate to localhost:8000/documentation/MySwiftPackage to preview your docs.
\n\n
The Bad
\n\n
Below are the main issues and pain points I experienced while experimenting with DocC as a possible replacement for Jazzy.
\n\n
DocC URL schemes are… strange
\n\n
I strongly dislike the URL schemes that DocC produces. My first gripe is, admittedly, purely cosmetic. Consider my example project on GitHub at github.com/jessesquires/my-swift-package. Normally, a GitHub Pages site root is hosted at USERNAME.github.io/REPO_NAME. However, a DocC site changes this to USERNAME.github.io/REPO_NAME/documentation/PACKAGE_NAME. That is, the path appends /documentation and /PACKAGE_NAME. Typically, the repo name and package name are the same, so you end up with an ugly URL like jessesquires.github.io/MyLibrary/documentation/MyLibrary. I much prefer the shorter, cleaner URLs.
\n\n
Aside from being ugly, there are some usability issues. Firstly, navigating to the normal GitHub pages root at jessesquires.github.io/MyLibrary loads a completely blank HTML page (not a 404), which is very confusing. However, navigating to jessesquires.github.io/MyLibrary/documentation (without appending the final /MyLibrary) does result in a 404. Sure, you just need to know and link to the correct URL, but it’s weird — especially if you are familiar with GitHub Pages.
For this specific example, I should also note another quirk about DocC — it transforms dashes into underscores. Notice that my repo and package are titled my-swift-package (with dashes), but the final path component in the DocC URL is my_swift_package (with underscores). This isn’t terrible, but I was really confused when my site was not working at first. Then I realized that I was using the wrong URL with my-swift-package at the end instead of my_swift_package.
\n\n
Quirks with migrating from Jazzy
\n\n
The other issue with this URL scheme is migrating existing projects. Currently, all of my Apple platform projects use Jazzy and host docs at jessesquires.github.io/PROJECT_NAME. For example, Foil which has docs at jessesquires.github.io/Foil. If I immediately switched over to using DocC, I would break the documentation links for all of my projects — an Internet sin, to be sure. But it’s actually worse than that, because with DocC the root of the GitHub Pages site is just a blank page, as mentioned above.
\n\n
The obvious workaround here is to add a redirect. But because we’re in a static hosting environment, we can only accomplish this with meta tags. This is not ideal, but it works.
The above code should be added to the DocC generated docs/index.html file. However, there’s a major caveat. You will need to edit the root docs/index.htmlevery time you regenerate your documentation to add this redirect. You could automate adding the redirect as part of the script that generates your documentation, but that’s also not ideal.
\n\n
DocC generation is not stable
\n\n
This brings me to another issue with DocC. Its documentation generation is not stable. That is, it produces a diff on every single run. With Jazzy, if you generate your docs, commit the changes, and immediately run Jazzy again it will not produce a diff. In contrast, DocC always produces a diff. Specifically, the various JSON files output to docs/data/... change every single time.
\n\n
Aside from being a nuisance, this has implications for automation and scripting. Suppose you want to automatically generate documentation via GitHub Actions after every push to main, and have a bot commit the doc updates. You will quickly accumulate a bunch of unnecessary commits. The ideal scenario would be that documentation generation is stable, and only produces a diff if the documentation has actually changed. Then, in automation scenarios, doc generation could no-op and avoid all these extra commits.
\n\n
DocC local previews require a web server
\n\n
I mentioned above that DocC docs can be previewed easily via the Swift DocC Plugin, which I appreciate. It is easy and convenient enough. It works. However, it would be nice if it did not require running a web sever to do so. Isn’t this supposed to be a static site?
\n\n
This gets to the core of why DocC is a difficult drop-in replacement for Jazzy. While static hosting with DocC now works, it was not originally built with this in mind. Thus, it doesn’t behave like you would expect a static site to behave. The --transform-for-static-hosting flag is really just a hack, a workaround for static hosting environments like GitHub Pages. You cannot simply open docs/index.html or docs/documentation/MyLibrary/index.html because your local paths do not match the paths on the server. This is why a local web server is necessary for previews. The result is static hosting feeling like a second-class citizen.
\n\n
To compare with Jazzy, all you need to do is open docs/index.html directly in Safari and you can quickly preview your entire site. No web server needed.
\n\n
Other missing features
\n\n
DocC is missing a handful of useful features from Jazzy that I would like to see it adopt.
\n\n
Jazzy reports documentation coverage. If you look at the docs for Foil, you’ll see it is 100% documented. Similarly, the docs for Quick report it is currently 88% documented. Even better, Jazzy outputs a docs/undocumented.json file which specifies all undocumented symbols and their precise file and line location in the codebase. This is incredibly helpful for maintainers like me who strive for 100% documentation to avoid “No Overview Available”. Because Jazzy outputs structured data in undocumented.json, you can build automation around this, like a Danger plugin that reports if your pull request added new APIs without documenting them. One last benefit is that contributors can also see which symbols need documentation — often, an easy first-time contribution for folks wishing to get involved.
\n\n
Jazzy docs include a prominent “View on GitHub” link in the site header that takes you directly to the GitHub repo. This is particularly convenient if someone first lands on your documentation pages instead of your repo.
\n\n
Jazzy includes a client-side search. It’s a bit simplistic and basic — limited by the static environment of GitHub Pages — but it works pretty well in practice and is nice to have.
\n\n
Conclusion
\n\n
I think DocC offers a lot of value and has some really cool features like interactive tutorials, and it looks like the best option available if you do not need a static site. However, DocC static hosting imposes just enough inconvenience to my existing setup and workflow to make switching away from Jazzy not worth the effort — for now. I might consider using DocC for new projects, but migrating existing projects would be more trouble than it is worth.
\n\n
The main improvement I would like to see with DocC static hosting is making this feature a first-class citizen — fix the quirky URL scheme and don’t require a web server to make it a truly static site. As for DocC features in general, the most important one for me would be reporting documentation coverage.
To workaround the unstable JSON output, you can set the env variable export DOCC_JSON_PRETTYPRINT=\"YES\" as seen here.
\n
Turns out DocC is not transforming dashes into underscores. That’s Swift doing that, which makes sense now because you have to import my_swift_package. My mistake there!
According to various reports ([1], [2], [3], [4]), GitHub is suspending accounts of Russian developers and organizations linked to or associated with organizations sanctioned by the US government over Russia’s invasion of Ukraine. But it appears that GitHub did not think this through entirely, because these account suspensions are fucking up my projects.
According to various reports ([1], [2], [3], [4]), GitHub is suspending accounts of Russian developers and organizations linked to or associated with organizations sanctioned by the US government over Russia’s invasion of Ukraine. But it appears that GitHub did not think this through entirely, because these account suspensions are fucking up my projects.
\n\n\n\n
* * *
\n\n
First, some brief context and background.
\n\n
I recently took over as a lead maintainer for two popular projects in the Apple developer community, Quick and Nimble. I just releasedversion 5.0 of Quick a few days ago. During the week leading up to the release, I was reviewing and merging many pull requests. But when it came time to write the release notes, I noticed very bizarre behavior. Mysteriously, some pull requests were deleted. Poof. Gone. Then I realized that an entire contributor’s presence had disappeared — all of their comments on issues were missing, all of the issues they opened were gone, all of the pull requests they opened had vanished. Every piece of activity related to the user was gone. What the fuck?!
Both the user account and the pull request result in a 404. But you can find the merge commit here, which is all that’s left in terms of an historical accounting of this change. It’s particularly notable for this project because PR #1129 was a critical bug fix.
\n\n
Today, maintainer Rachel Brindle opened a pull request with another important bug fix, but lamented the fact that the original pull request that had introduced the bug had since been deleted:
\n\n
\n
The original PR that introduced it has since been deleted, so I’m unsure exactly of the intention of that contribution.
\n
\n\n
So here I am for the past few days wondering what the fuck is going on. Why are multiple users and pull requests disappearing from our project? My gratitude goes out to Tomasz Sapeta for helping us realize that all of these mysterious disappearances are due to GitHub flippantly suspending the accounts of Russian developers without any regard for the destructive side effects.
\n\n
There are multiple contributors to Quick whose accounts have been suspended, which means we have lost everything they contributed aside from raw commit history.
\n\n
* * *
\n\n
It is unclear to me what GitHub’s intended result was with these account suspensions, but it appears to be incredibly destructive for any open source project that has interacted with a now-suspended account. On a service like Twitter, you can visit the placeholder profile of a suspended account and see a message communicating that the account is suspended, and other users’ @mentions of the account still link to the suspended account’s profile. On GitHub, that’s not how it works at all.
\n\n
Apparently, “suspending an account” on GitHub actually means deleting all activity for a user — which results in (1) every pull request from the suspended account being deleted, (2) every issue opened by the suspended account being deleted, (3) every comment or discussion from the suspended account being deleted. In effect, the user’s entire activity and history is evaporated. All of this valuable data is lost. The only thing left intact is the raw Git commit history. It’s as if the user never existed.
\n\n
Again, at this time it is not clear to me if the data loss was GitHub’s goal or if this was a mistake. Either way, it’s a massive problem. Deleting this data with no notice is an abuse of trust. Should we continue to trust GitHub with important data?
\n\n
* * *
\n\n
This has been absolutely perplexing because there was no notification or communication from GitHub about this, aside from their generic statement in which they claimed “we are continuing to ensure free open source services are available to all, including developers in Russia.” So much for that. I’ve been working for only a week or so on this project that I inherited, trying to diligently keep track of changes like a good maintainer, and then all kinds of peculiar and unexpected weird shit started happening. Unbeknownst to me, GitHub was quietly joining the rest of the western world in its crusade to punish innocent Russian civilians whose only crime was being born in the wrong place and maybe being formerly associated with a bank that is now being sanctioned. Fuck Putin, but I don’t see how deleting GitHub accounts and causing food shortages for civilians is “winning” for anyone. As far as I could tell, the now-missing contributors were just ordinary iOS and macOS developers interested in contributing to a community open source project.
\n\n
These actions from GitHub are harmful and damaging to open source projects and the open source community. All of a sudden, I was seeing pull requests, issues, and comments disappear from users who were actively contributing to the project. We lost valuable contributions, information, context, and discussion history on issues and pull requests. We even lost pull requests that were open and under active review. That work is now entirely lost. Gone forever. For pull requests that did merge, we have the raw commit history — but that is not a substitute for a full code review and discussion.
\n\n
* * *
\n\n
It’s hard enough to maintain open source projects. It’s even harder to inherit an old, somewhat neglected project and try to get it back on track. In that scenario, every single pull request, issue, and comment is important for the long-term maintenance and success of the project. Comments, discussions, and code reviews provide valuable context that is not always captured in the commit history — especially for open source projects that have cycled through multiple maintainers over the years. I think a proper solution from GitHub would have been to leave all contributions intact, freeze the suspected accounts to prevent future activity, and clearly mark the account profile pages as suspended. And then, when possible, flip the switch to fully re-enable the accounts. But apparently GitHub thought the best thing to do was delete it all.
Good news! Martin Woodward, the Senior Director of Developer Relations at GitHub, reached out to let me know that GitHub has restored the missing pull requests, issues, comments, etc. from the Russian developers whose accounts have been suspended. User profiles are also restored, though they do not specifically mention that the accounts are suspended.
\n\n
According to him, the only mechanism GitHub previously had to suspend accounts was built to target spammers and other malicious actors — scenarios in which, usually, the best thing to is make the accounts and all activity entirely disappear. Clearly, that was not appropriate in this case. I really appreciate Martin contacting me and helping push for a better solution to this internally at GitHub.
\n\n
I still disagree with punishing ordinary individuals for atrocities perpetrated by The Russian State (and all States, for that matter), but I suppose most US corporations have little choice but to concede to governmental pressure. That’s a topic for another post.
Privacy-centered search engine DuckDuckGo has completely removed the search results for many popular pirates sites including The Pirate Bay, 1337x, and Fmovies. Several YouTube ripping services have disappeared, too and even the homepage of the open-source software youtube-mp3 is unfindable.
\n\n
[…]
\n\n
Update April 17: DuckDuckGo informs us that no domains were removed but they are having some issues and we still have questions. More details are at the bottom of the article.
\n\n
For example, searching for “site:thepiratebay.org” is supposed to return all results DuckDuckGo has indexed for The Pirate Bay’s main domain name. In this case, there are none.
Update April 17: DuckDuckGo has responded to our findings and says that no domains were removed, according to their records.
\n\n
Before publishing the article we searched for YouTube-dl and The Pirate Bay without the “site:” operator the official domains were not showing up at our end. They do now.
\n
\n\n
According to Gabriel Weinberg, the CEO and Founder, the company is having issues with their “site:” operator:
\n\n
\n
Similarly, we are not “purging” YouTube-dl or The Pirate Bay and they both have actually been continuously available in our results if you search for them by name (which most people do). Our site: operator (which hardly anyone uses) is having issues which we are looking into.
\n
\n\n
However, this doesn’t explain the fact that even a search for my name or website without using the site: operator yields no relevant results. For comparison, Google Search is very accurate in this regard. Searching for my name there shows my website, Twitter profile, and GitHub profile — in that order — as the top 3 results. And I know the results used to be the same on DuckDuckGo.
We don’t doubt that DuckDuckGo hasn’t intentionally removed any URLs but there still appear to be strange issues with pirate-related searches.
\n\n
Update 2: A DuckDuckGo spokesperson confirmed to TorrentFreak that the issues are related to Bing data.
\n
\n\n
Well, that seems to confirm my suspicion that this was ultimately caused by Bing removing my site (and apparently everything else about me) from its index. And, as previously mentioned I’m still waiting to hear back from Bing Support. Again, the lack of transparency into search engines in general is infuriating. I wonder, why doesn’t DuckDuckGo do more to build its own infrastructure to crawl the web and build its search index without relying solely on third parties?
\n\n
I used to be really excited about DuckDuckGo, but I’m quickly becoming less of a fan and losing confidence in their results, which seem to be degrading lately. I’ve had to switch back to Google Search for many queries in the past few months.
There’s another update from TorrentFreak. All of the pirate sites they were tracking have since been restored to the DuckDuckGo search index:
\n\n
\n
At this point, it became clear that the search engine wasn’t at all happy with what was happening. They never actively removed any of these sites. Instead, a third-party data provider ‘removed’ the results for them.
\n\n
Like many other smaller search engines, DuckDuckGo uses hundreds of data sources, including Bing. After some back and forths, DuckDuckGo’s spokesperson informed us that Microsoft’s search engine was the culprit.
\n
\n\n
My site still does not appear on DuckDuckGo, nor Bing. I’m still waiting to hear back from Bing Support. I wonder how many other sites are affected by this?
This post continues from my earlier post on label-based GitHub Actions workflows. A common and tedious task in a typical pull request workflow is updating your branch with changes from the default branch of your repo. In many scenarios, you may want to rebase your pull request on the main branch or merge the main branch into your pull request.
This post continues from my earlier post on label-based GitHub Actions workflows. A common and tedious task in a typical pull request workflow is updating your branch with changes from the default branch of your repo. In many scenarios, you may want to rebase your pull request on the main branch or merge the main branch into your pull request.
\n\n\n\n
Perhaps there are improvements from the main branch that you want incorporated into your pull request, or maybe you want to re-run the entire test suite with your changes and the latest changes from the main branch. It could also be the case that you require branches to be up-to-date before merging. With the way I work, this is a pain. After I submit a pull request for review, I typically start a new branch to start working on something else immediately. Later on, if I need to rebase an existing pull request, then I must stash or commit my current changes, switch branches and rebase, push the changes, then switch back to what I was working on before the interruption.
\n\n
One workaround to this is having multiple checkouts, where you do all the rebasing on the second checkout while leaving your current in-progress checkout untouched. This, however, is still tedious. If your branch has no conflicts with the main branch, then rebasing or merging can be fully automated. I’ve written two new workflows for each. This frees up my time to focus on things a machine can’t do for me.
\n\n
Automatic rebasing
\n\n
This workflow is triggered by adding a label called rebase to your pull request. It uses the automatic-rebase action to perform the rebase, and actions-ecosystem-remove-labels action to remove the label when complete. To use this, you’ll need to add the rebase label to your repo. If you’d like to use a different label name you can, just be sure to edit the workflow accordingly. You’ll also need to add a personal access token.
If you would rather merge the main branch into your pull request instead of rebase, I wrote a workflow for that too. I’ve titled it “Merge Main into PR”, but if your default branch is named differently, you can change this. You do not need to worry about updating the workflow based on what you’ve named your default branch, the workflow will detect the default branch name automatically. Similar to the rebase workflow, this one is triggered by adding a label named merge main. Again, you can adjust this as needed for your own configuration. And finally, this also removes the label when finished.
\n\n
Note that I’ve generously used git status during the merge to make debugging easier if something goes wrong.
When combined with the workflows from my previous post, I have started to accumulate a decent collection of general-purpose workflows. To keep track, I created a new repo to house them all and track changes, which you can find here. All you need to do is copy them to .github/workflows/ in your own repo, and edit them as needed. The current workflows are centered around automating tedious aspects of managing pull requests, but one can imagine all kinds of use cases for managing GitHub issues, releases, and more. As I write new workflows, I’ll be sure to include them in this project.
However, these label-based workflows can still be useful for performing these operations in large batches. For example, if you need to rebase many pull requests at once you could select them all and add the label rather than visit each pull request individually and click the “Update branch” button.
I have started using GitHub Actions for CI for my newer open source projects. I recently setup Danger for the first time on GitHub Actions and it was quite easy. Here is how to do it.
I have started using GitHub Actions for CI for my newer open source projects. I recently setup Danger for the first time on GitHub Actions and it was quite easy. Here is how to do it.
\n\n\n\n
You will need to follow Danger’s Getting Set Up guide, which involves creating a bot account and adding a new personal access token for it. After that, you will need to navigate to Settings > Secrets for your repo on GitHub, and add the personal access token you created as the value of the secret and name it DANGER_GITHUB_API_TOKEN. Technically, you could name this differently, but I think this is easiest to use this name to remember its purpose.
\n\n
With the setup complete, all you need to do is implement your workflow. Here is a template you can use for iOS or macOS projects, but with a few tweaks this can be used for any project.
This workflow will only be triggered on pull requests, which is convenient. The first four steps are just boilerplate for checking out the repo and setting up ruby. The final step executes danger and makes use of the DANGER_GITHUB_API_TOKEN secret that you created.
\n\n
If your Dangerfile does not require any additional data, you can use this workflow as-is. Otherwise, you can add additional steps after the ruby setup step and before the danger step. For example, suppose you need access to xcodebuild output for a specific Danger plugin, you can add a step that builds your project. For non-Xcode projects, the same applies. Add any additional steps you need before danger is called.
This workflow has been updated to reflect various changes and improvements to GitHub Actions, namely by using the more modern approach to setting up ruby and caching gems.
My current team has started using GitHub Actions to automate some tedious tasks for pull requests. In particular, we use labels on GitHub to categorize pull requests or highlight important metadata about them. Most of the time, a machine can figure out which labels are appropriate to add or remove. This is a great use case for GitHub Actions.
My current team has started using GitHub Actions to automate some tedious tasks for pull requests. In particular, we use labels on GitHub to categorize pull requests or highlight important metadata about them. Most of the time, a machine can figure out which labels are appropriate to add or remove. This is a great use case for GitHub Actions.
\n\n\n\n
Labeling stacked pull requests
\n\n
One common workflow (or problem, depending on how you view it) in git and GitHub is stacking pull requests. This is necessary when you have multiple pull requests that depend on each other and must merge into your main branch sequentially. For example, pull request A is opened to merge into main, pull request B is opened to merge into A, pull request C is opened to merge into B, and so on. After A is merged, B must be updated to merge into main, and so on.
\n\n
We like to highlight this and make it obvious. We have a brightly colored stacked PR label, which communicates to the reviewer “hey! this PR is part of a stack, just so you know.” It is tedious to repeatedly manually add and remove this label for every pull request in the stack, so I wrote this workflow to automate it.
This workflow contains two jobs that are nearly identical. The first checks if the pull request is merging into the main branch, and if so, it will remove the stacked PR label, if present. The second job does the opposite, it will add the stacked PR label to any pull request that is not targeting the main branch. It uses two actions from the Actions Ecosystem community project, add-labels and remove-labels.
\n\n
Do Not Merge
\n\n
Another important label we have is do not merge, which (unsurprisingly) indicates that a pull request should not be merged. Despite the label being a striking bright red, it could still be overlooked if all of our other CI status checks pass and you see that tempting, big green “merge” button on GitHub. It would be much safer if adding this label would fail the pull request to prevent you from merging it. We can write a workflow to do that using Michael Heap’s required-labels action.
This will check if the pull request has the do not merge label. If it does, the workflow will fail. Once you remove the label, the workflow will pass.
\n\n
To make this workflow actually prevent merging requires a few extra steps. You need to set up branch protection rules for your main branch and require this workflow to pass before a pull request can be merged. Once configured, the “merge” button will only be enabled if this workflow succeeds.
Thanks to my friend Ben Asher for sharing an improvement to the ‘Do Not Merge’ workflow, which can be rewritten without using third-party actions. You can find it below.
\n\n
Also, I’ve started maintaining a collection of useful workflows on GitHub. Check them out.
\n\n
\n\n
name:Do Not Merge\n\non:\n pull_request:\n types:[synchronize,opened,reopened,labeled,unlabeled]\n\njobs:\n do-not-merge:\n if:${{ contains(github.event.*.labels.*.name, 'do not merge') }}\n name:Prevent Merging\n runs-on:ubuntu-latest\n steps:\n -name:Check for label\n run:|\n echo \"Pull request is labeled as 'do not merge'\"\n echo \"This workflow fails so that the pull request cannot be merged\"\n exit 1\n
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2022/03/04/quickly-displaying-the-swift-version-that-ships-with-xcode/#updated-09-march-2022",
"url": "https://www.jessesquires.com/blog/2022/03/04/quickly-displaying-the-swift-version-that-ships-with-xcode/",
"title": "[Updated] Quickly displaying the Swift version that ships with Xcode",
"date_published": "2022-03-09T13:48:45-08:00",
"date_modified": "2022-03-09T13:48:45-08:00",
"summary": "
I previously wrote about writing a custom shell command to quickly switch between Xcodes. But recently, I needed to determine the version of Swift that is bundled with Xcode — specifically the version of Swift that is shipping with the current Xcode 13.3 beta. I was pretty sure that it is Swift 5.6, but I wanted to know for certain.
I previously wrote about writing a custom shell command to quickly switch between Xcodes. But recently, I needed to determine the version of Swift that is bundled with Xcode — specifically the version of Swift that is shipping with the current Xcode 13.3 beta. I was pretty sure that it is Swift 5.6, but I wanted to know for certain.
\n\n\n\n
Strangely, the Xcode beta release notes do not explicitly mention the Swift version included with the release. I expected this information to be available in the “Overview” section of the release notes, which is where all the SDK versions are listed, but the Swift version not present. If you are not following Swift development and Swift Evolution closely, then it isn’t immediately clear which version is shipping with Xcode.
\n\n
Luckily, we can find this by calling swiftc --version, but for the Xcode Beta we need to know the full path bundled with the app. It is located at /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc, which is a mouthful and difficult to remember.
Thanks to Sven Schmidt for offering improvements here to simplify the script and make it more robust. We can use DEVELOPER_DIR to specify which Xcode to use and then invoke xcrun to avoid having to use the exact path to the binaries. The script below has been updated.
\n\n
\n\n
The custom commands below build on what I previously wrote to switch between Xcode versions. I’ve added a new command, xcwhich that displays the currently selected Xcode, and then prints all the version information for both Xcode.app and Xcode-beta.app.
\n\n
# print current Xcode info\nfunction xcwhich(){\n _xcselect false\n echo\"\"\n echo\"RELEASE\"\n env DEVELOPER_DIR=/Applications/Xcode.app xcrun xcodebuild -version\n env DEVELOPER_DIR=/Applications/Xcode.app xcrun swift --version\n\n echo\"\\nBETA\"\n env DEVELOPER_DIR=/Applications/Xcode-beta.app xcrun xcodebuild -version\n env DEVELOPER_DIR=/Applications/Xcode-beta.app xcrun swift --version\n}\n\n# switch between release and beta Xcodes\nfunction xcswitch(){\n _xcselect true\n}\n\n# shared between `xcswitch` and `xcwhich`\n# determines current Xcode (beta or release)\n# pass true to switch, pass false to print current\nfunction _xcselect(){\n RELEASE=\"Xcode.app\"\n BETA=\"Xcode-beta.app\"\n\n CURRENT=$(xcode-select -p)\n NEXT=\"\"\n\n if[[\"$CURRENT\"=~ \"$RELEASE\"]]\n then\n NEXT=\"$BETA\"\n CURRENT=\"$RELEASE\"\n else\n NEXT=\"$RELEASE\"\n CURRENT=\"$BETA\"\n fi\n\n if[\"$1\"=true]\n then\n sudo xcode-select -s\"/Applications/$NEXT\"\n echo\"Switched to $NEXT\"\n else\n echo\"Current: $CURRENT\"\n fi\n}\n
\n\n
If you’d like to use this, you can paste the code above into your .zprofile (or similar). Here’s an example usage and output:
\n\n
> xcwhich\nCurrent: Xcode.app\n\nRELEASE\nXcode 13.2.1\nBuild version 13C100\nswift-driver version: 1.26.21 Apple Swift version 5.5.2 (swiftlang-1300.0.47.5 clang-1300.0.29.30)\nTarget: x86_64-apple-macosx12.0\n\nBETA\nXcode 13.3\nBuild version 13E5104i\nswift-driver version: 1.45.2 Apple Swift version 5.6 (swiftlang-5.6.0.323.60 clang-1316.0.20.8)\nTarget: x86_64-apple-macosx12.0\n
\n\n
An additional benefit of this is that you can see the exact Swift version that is bundled with the stable release of Xcode. Typically, point releases of Xcode bump the Swift version too. In this case, Xcode 13.2.x ships with Swift 5.5.2. This rarely matters for development purposes as you often only need to know which major and minor version of Swift that you are using — Swift 5.5 in this example. However, it is convenient to see the complete version information in one place.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2020/12/22/swift-self-executing-anonymous-closures/#updated-02-march-2022",
"url": "https://www.jessesquires.com/blog/2020/12/22/swift-self-executing-anonymous-closures/",
"title": "[Updated] What type is self in a Swift self-executing anonymous closure used to initialize a stored property?: The answer might surprise you",
"date_published": "2022-03-02T11:12:41-08:00",
"date_modified": "2022-03-02T11:12:41-08:00",
"summary": "
In JavaScript, this pattern is called an Immediately Invoked Function Expression (IIFE) or a Self-Executing Anonymous Function. Swift doesn’t have an “official” name for this, but IIFE works as well as “immediately executed anonymous closure” or “self-executing anonymous closure”. (Thanks to folks on Twitter for helping with this.)
In JavaScript, this pattern is called an Immediately Invoked Function Expression (IIFE) or a Self-Executing Anonymous Function. Swift doesn’t have an “official” name for this, but IIFE works as well as “immediately executed anonymous closure” or “self-executing anonymous closure”. (Thanks to folks on Twitter for helping with this.)
\n\n\n\n
Most Swift developers have seen and used this now-common approach to initialize properties for a type:
\n\n
classMyClass{\n\n letdateFormatter:DateFormatter={\n letdf=DateFormatter()\n df.timeStyle=.medium\n df.dateStyle=.long\n returndf\n }()\n\n // other members here...\n}\n
\n\n
It is a convenient and concise pattern that helps organize your code. This approach is especially popular when working with UIKit and defining custom views. It simplifies type initializers and is generally easy to read. Recently, I was using this approach and discovered a bug in my code, some very unexpected behavior in Swift, and maybe a bug in Swift.
\n\n
I was building a typical table view which contained cells that had a button subview. Tapping the entire cell and tapping the button within the cell performed different actions. You have probably built something similar before. Here is simplified sample code to illustrate:
I was building such a routine, mundane feature that I completed a large portion of the code before even running and testing it. I’ve written code like this thousands of times. So, you can imagine my surprise when the button inside the cell was not triggering the action. Tapping the button did nothing, but everything else seemed to work as expected. There’s a bug in that code above. Can you find it? I scoured line after line, trying to find what I missed. What I had overlooked out of habit?
\n\n
The problem was the call to addTarget(_:, action:, for:). Moving this line to the initializer fixed the issue, and everything worked as expected.
Upon discovering what the problem was, it was now time to ask why. My initial intuition was the following:
\n\n\n
The button is a stored constant property that gets initialized immediately via a self-executing anonymous closure
\n
Swift’s initialization rules mean that stored properties (with values, like this one) must be initialized before the type that owns them
\n
Thus, at the time that button is initialized (when the closure is executed), self must be nil
\n
The method addTarget(_:, action:, for:) accepts Any? for the target parameter, so passing nil would be ok
\n
Conclusion: self was just nil the whole time! What a goofy mistake!
\n\n\n
However, that was not the case. Specifically, selfwas notnil. Not only that, but self wasn’t the self I expected. Looking at this snippet, can you determine what type self is and why?
\n\n
classMyTableCell:UITableViewCell{\n\n letbutton:UIButton={\n letbutton=UIButton()\n button.setTitle(\"Title\",for:.normal)\n\n // what is self?\n button.addTarget(self,action:#selector(didTapButton(_:)),for:.touchUpInside)\n\n returnbutton\n }()\n}\n
\n\n
I paused execution in the debugger to check if po self printed nil. Instead, it printed (Function). In Swift, closures are first-class reference types. Thus, I concluded that self is referring to the closure, which is the type () -> UIButton. Right? Actually… no.
\n\n
Continuing the investigation, I printed type(of: self) which returned (MyTableCell) -> () -> MyTableCell. What?! This is a closure that receives a MyTableCell instance and returns a closure with the type () -> MyTableCell, which accepts nothing and returns a MyTableCell. I do not understand why this is the case.
\n\n
I think most devs would share my initial intuition that self refers to MyTableCell here, because that is true in other patterns that look similar to this one, like computed properties. Despite this, once realizing that self is a function, I think most devs would also intuit that the type must be () -> UIButton.
\n\n
This raises some interesting questions.
\n\n
First, I realize that the #selector construct, which was introduced by SE-0022, has limited abilities. As mentioned in the “Alternatives considered” section of the proposal, #selector is not type-safe. However, it is capable of determining if a selector is in scope. If you attempt to pass a selector from another class, the compiler will provide a warning. For example, considering the following:
This produces the expected error: Cannot find 'didTapButton' in scope. It is not clear to me how this is distinct from the MyTableCell example regarding the scope of the selector. How is didTapButton(_:) in scope for self if selfis not an instance of MyTableCell? Is this a bug with #selector?
This is not entirely accurate. You could write #selector(InnerClass.didTapButton(_:)), which successfully compiles and is consistent with SE-0022.
\n\n
\n\n
Second, why is self an instance of (MyTableCell) -> () -> MyTableCell and not() -> UIButton? Is this a bug in Swift?
\n\n
I’m hoping someone can answer these two questions and explain what is happening. If so, I’ll update this post!
\n\n
In any case, referencing self in self-executing anonymous closures for stored properties should be discouraged, or avoided entirely! That self is perhaps not the self you might have been expecting. Watch yourself.
If you declare button as lazy var instead of let, then the expected behavior occurs. That is, self is an instance of MyTableCell within the self-executing anonymous closure and the call to addTarget(_:, action:, for:) works. (Thanks @Geri_Borbas.) Also noteworthy, in this situation the initializer for MyTableCell gets called before the button closure is executed. However, this makes the situation more confusing — using let versus lazy var produces significantly different behavior that is not intuitive at all.
\n\n
The type (MyTableCell) -> () -> MyTableCell appears to refer to the initializer for MyTableCell. (Thanks @eneko.) No, this is not the case.
Thanks to @elliottwil and Noah Gilmore for uncovering and investigating the true underlying issues here. (Nerd sniped. 😄) The type of self resolving to (MyTableCell) -> () -> MyTableCell is the unfortunate result of the NSObject instance method -[NSObject self] and Swift’s curried functions. (Note that SE-0002 removed currying func declaration syntax but did not change the semantics of methods.)
\n\n
Update continued below.
\n\n
\n\n
There are multiple layers of confusion here. Let’s peel back each one. First, let’s revisit the problematic code:
\n\n
classMyTableCell:UITableViewCell{\n letbutton:UIButton={\n letbutton=UIButton()\n\n // what is self?\n button.addTarget(self,action:#selector(didTapButton(_:)),for:.touchUpInside)\n\n print(type(of:self))// (MyTableCell) -> () -> MyTableCell\n\n returnbutton\n }()\n}\n
\n\n
We know that self is of type (MyTableCell) -> () -> MyTableCell because of the -[NSObject self] instance method. And this satisfies the (lack of) type constraints on the target parameter, Any?. This answers the question: why is self an instance of (MyTableCell) -> () -> MyTableCell and not() -> UIButton? We can verify by printing to the console:
\n\n
print(type(of:NSObject.`self`))// prints (NSObject) -> () -> NSObject\n
\n\n
Note that NSObject.self (without the backticks) refers to the metatype typeNSObject, not the instance method-(instancetype)self. Thus, the expression type(of: NSObject.self) returns NSObject.Type. Because self is a reserved keyword, we must write it with backticks to reference the correct “self”. (You may have seen this before if you have ever tried to define an enum case called “default”.)
// Xcode's documentation popup correctly refers to the declaration let `self`: MyTableCell\nprint(type(of:MyTableCell.`self`))// let `self`: MyTableCell\n
\n\n
With this change, we can see that this is clearly wrong. Additionally, Xcode will syntax highlight “self” differently depending on if there are backticks or not.
\n\n
Next, which is now obvious, this only happens with classes that inherit from NSObject. Because I was using UITableViewCell, this was not something I considered at first. As an example, the following code will produce an error:
Now we can attempt to answer my two remaining questions.
\n\n
Is this a bug with #selector? No. This is the expected behavior of #selector as described in SE-0022. We could have written #selector(Self.didTapButton(_:)) instead, which is equivalent to #selector(MyTableCell.didTapButton(_:)). The #selector expression is not capable of checking itself against the target parameter.
\n\n
Is this a bug in Swift? In my opinion, yes. I think the correct behavior would be that using selfwithout backticks always references the enclosing type (which happens when using lazy var, as mentioned above) and using `self`with backticks references -[NSObject self] (which you almost never want). The fact that Xcode correctly syntax highlights each of these indicates that the correct information exists somewhere.
\n\n
classMyTableCell:UITableViewCell{\n letbutton:UIButton={\n\n print(type(of:self))// Bug: should be MyTableCell. Instead, same as `self` below.\n\n print(type(of:`self`))// (MyTableCell) -> () -> MyTableCell\n\n returnUIButton()\n }()\n}\n
\n\n
In other words, these two expressions are distinct in this context:
\n\n
// refers to instance of MyTableCell\nself\n\n// refers to func `self`() -> Self\nMyTableCell.`self`\n
\n\n
Yet, the Swift compiler treats them both as func `self`() -> Self.
\n\n
However, there is one last problem. Correcting the expression self (without backticks) to reference the enclosing type introduces another interesting question: what should be order of operations during initialization? When using let, the property is initialized before the enclosing type. When using lazy var, the property is initialized after the enclosing type. I am not a compiler expert, so I will not attempt to answer which is better. But if initialization order cannot be changed in the compiler to fix this, then I think the expected behavior would be to produce the same error as non-NSObject classes: “Cannot find ‘self’ in scope”.
As of September 2021, SR-4559 has been marked as resolved. The pull request is here and according to the description the fix is to emit a warning indicating that the self reference is likely not the one you intended.
\n\n
The fix is not yet included in Xcode 13.1 (13A1030d), nor Xcode 13.2 beta (13C5066c). However, the latest release of SwiftLint includes a new rule (self_in_property_initialization) to emit a warning for this exact issue.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2019/09/27/icloud-backup-using-rsync/#updated-21-february-2022",
"url": "https://www.jessesquires.com/blog/2019/09/27/icloud-backup-using-rsync/",
"title": "[Updated] Backing up your iCloud Drive files using rsync",
"date_published": "2022-02-21T18:52:38-08:00",
"date_modified": "2022-02-21T18:52:38-08:00",
"summary": "
Unfortunately, iCloud does not have a good reputation for being reliable, especially during beta releases of iOS and macOS. Yet a lot people still use it, often without any problems. I still use it, despite a few bad experiences in the past, because the best alternatives are questionable for other reasons. I’ve had good luck with iCloud Drive for the past few years, but I am terrified and paranoid of getting caught in the middle of an iCloud clusterfuck, so I backup what I have in iCloud periodically using rsync.
Unfortunately, iCloud does not have a good reputation for being reliable, especially during beta releases of iOS and macOS. Yet a lot people still use it, often without any problems. I still use it, despite a few bad experiences in the past, because the best alternatives are questionable for other reasons. I’ve had good luck with iCloud Drive for the past few years, but I am terrified and paranoid of getting caught in the middle of an iCloud clusterfuck, so I backup what I have in iCloud periodically using rsync.
\n\n\n\n
Exploring iCloud Drive
\n\n
The iCloud Drive folder on your Mac is tricky. The files are not what they seem. iCloud Drive displays as a special directory in the sidebar of Finder. Its actual location on disk is at ~/Library/Mobile Documents/. If you cd there and ls, you will see its entire contents, most of which is not viewable via Finder.
\n\n
There appear to be three general types of directories present. After some digging around, I could not discern a difference between them. They have the following formats:
\n\n
\n
Developer ID or “App ID Prefix” (I think?) followed by bundle identifier. Examples:\n
\n
8Z3V4F58RK~com~ustwo~monumentvalley/
\n
F3LWYJ7GM7~com~apple~mobilegarageband/
\n
F3LWYJ7GM7~com~apple~musicmemos~ideas/
\n
\n
\n
Apple first-party apps with specific iCloud support, prefixed with com~apple~. These correspond to the “special” app-specific folders that you see in Finder’s view of iCloud Drive. However, not all of these are present in Finder, like Notes and Mail. Examples:\n
\n
com~apple~Automator/
\n
com~apple~Keynote/
\n
com~apple~Preview/
\n
com~apple~TextEdit/
\n
com~apple~Notes/
\n
com~apple~mail/
\n
\n
\n
Bundle identifiers prefixed with iCloud~com~. Examples:\n
\n
iCloud~com~agilebits~onepassword-ios/
\n
iCloud~com~apple~iBooks/
\n
iCloud~com~ustwo~monumentvalley2/
\n
iCloud~com~apple~mobilesafari/
\n
\n
\n
\n\n
Aside from the com~apple~-prefixed directories, both first-party and third-party apps appear in both formats. My guess is that the difference is legacy vs modern naming conventions, especially considering MomumentValley (2014) and MonumentValley2 (2017).
\n\n
Backing up your documents
\n\n
Now that we sort of understand the layout of ~/Library/Mobile Documents/, where the hell are our iCloud Drive documents stored? Those live in com~apple~CloudDocs/. If you cd there, you should see all of the “custom”, non-app-specific files that you’ve stored in iCloud Drive. These should match what is viewable in Finder.
\n\n
This is the directory that we want to backup. We can use rsync to do that. (Side note: the way that rsync handles paths is a bit odd. It doesn’t like relative paths, or ~, or escaping spaces in directory names. Thus, this script uses absolute paths with spaces.) You just need to fill-in the USER and DEST variables.
--times: transfer modification times along with the files and update them on the remote system
\n
--exclude: exclude files matching PATTERN
\n
\n\n
You can use the --dry-run option to preview what would be transfered without actually executing rsync.
\n\n
This will backup only your iCloud Drive files. Any app-specific files (like Pages documents in com~apple~Pages/) will not be included. If you want to backup any of those files, you will need to write a similar script with those target directories as the source. If you want to backup everything, specify the root ~/Library/Mobile Documents/ directory — but beware, this might be a lot of data. For example, iCloud~com~apple~iBooks/ contains all of your synced iBooks that were not purchased from the iBook Store. I have a lot of those.
\n\n
That’s it! You can read the rsync docs (man rsync) for more details and options, but this should get you started.
I have been experiencing bizarre kernel panics with my Mac lately. I have a 2020 Intel MacBook Pro, the last Intel model before the M1 debuted. It has generally been working fine. Despite poor software quality and numerous bugs lurking around in macOS, I rarely see kernel panics anymore. In fact, I can’t remember the last time I had a kernel panic before this issue. There have been no major changes on my machine and I’m on the latest version of Monterey.
I have been experiencing bizarre kernel panics with my Mac lately. I have a 2020 Intel MacBook Pro, the last Intel model before the M1 debuted. It has generally been working fine. Despite poor software quality and numerous bugs lurking around in macOS, I rarely see kernel panics anymore. In fact, I can’t remember the last time I had a kernel panic before this issue. There have been no major changes on my machine and I’m on the latest version of Monterey.
\n\n\n\n
Here’s what happens. Suddenly the MacBook display turns off, the fans turn on full blast for a couple seconds (they are louder than compiling the largest Xcode project you can think of), and then the machine shuts down completely. I’m calling it the “sigh of death” because it’s as if the machine is simply giving up and exhaling in an exasperated, dramatic sigh of defeat — like an exaggerated cartoon character would do. It has happened to me 3-4 times within the past few weeks.
\n\n
\n \n
\n \n \n
\n
\n \n \n
\n macOS screen of death \n
\n \n \n
\n
\n \n
\n \n
\n\n
If you have been using a Mac for any amount of time, you have likely seen the “screen of death” at least once. If not, consider yourself lucky. Usually when this happens, it is preceded by some notable event — your machine freezes, your cursor starts beach-balling, everything becomes completely unresponsive, and then you’ll see the screen of death. It’s also usually silent, or at least, there’s no notable change with the fans.
\n\n
That’s why this is so peculiar. It seems to be random and unprovoked in terms of software — there are not any specific apps, scripts, tasks, etc. that seem to trigger it. Prior to the kernel panic, the machine is fully responsive and functioning like normal. In some cases, I am actively using it. However, there is one commonality across all incidents so far. Each time it has happened, it is after I unplug the machine from power. I typically work at a desk with the MacBook plugged in, and occasionally I’ll move to the kitchen table or couch and work off battery power for a bit. Each time the sigh of death has occurred, it is shortly after being unplugged. It makes me wonder if there’s some hardware issue?
\n\n
Upon rebooting, the “Problem Report for macOS” window appears as expected. Usually, the details section will contain a stack trace from which you can derive some kind of useful information about what happened — but not with the sigh of death. All it says is “Intel CrashLog recorded due to unexpected reset” — whatever that means.
\n\n
\n \n
\n \n \n
\n
\n \n
\n
\n \n
\n \n
\n\n
Finally, I vaguely remember reading another blog post from someone about this exact issue. It was awhile back, perhaps during a macOS beta, and they also called it the “sigh of death”. Unfortunately, I could not find the article anywhere on the web. If anyone recalls this post or can find it, please send me the link and I’ll update this post to link to it.
\n\n
And if you are experiencing this issue, or have any idea about what’s happening, please let me know.
Thank you Michael Tsai for finding the post I was referring to above, The Big Sur Sneeze. My memory of that post was not entirely accurate. The details are slightly different than what I describe in this post, but it seems likely that the underlying issue is the same.
\n\n
Also, a few folks on Twitter commented that they have experienced this issue.
I have the same problem after I unplug the dock with external screen, then later plug it back, external screen does not come back and then sigh and reboot
I’ve been on macOS Monterey for quite a few months now. I haven’t had any major issues with Monterey — only the thousands of paper cuts (like this error) representing the slow degradation of macOS that I’ve grown to expect in recent years.
\n\n
The error message is basically the same as before, “the backup disk image could not be accessed (error 35)” — however, now the error code is 35 instead of 45. Perhaps it is failing in a slightly different way, or perhaps the error code simply changed in Monterey. Like I mentioned before, I use Time Machine with my NAS (“Leviathan.local”) and haven’t had any problems with my setup until these recent errors. Sadly, I still have no idea what causes the error nor how to fix it given the absolute dearth of useful information in the error message.
\n\n
For now, rebooting my MacBook has “fixed” the issue and backups are working again — but this is likely only temporary. I expect to see the error again soon, and I supposed I’ll just have to reboot again. And again. And again.
Two readers have provided some tips on Twitter. David Humphrey mentioned a better possible workaround:
\n\n
\n
I had something like this with Time Machine backups to my Synology, and it turned out to be macOS holding open simultaneous connections, which is why rebooting fixed it. When it happens now, I know that I can go and force a disconnect of the user on the NAS and that fixes.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2021/06/29/apple-docc-great-but-useless-for-oss/#updated-30-december-2021",
"url": "https://www.jessesquires.com/blog/2021/06/29/apple-docc-great-but-useless-for-oss/",
"title": "[Updated] Apple's DocC is excellent, but unusable for open source projects",
"date_published": "2021-12-30T17:31:10-08:00",
"date_modified": "2021-12-30T17:31:10-08:00",
"summary": "
I was very excited at this year’s WWDC when Apple announced DocC, their new “Documentation Compiler” tool that generates documentation from comments in your source code. Unfortunately, it’s not going to work for the majority of open source authors.
I was very excited at this year’s WWDC when Apple announced DocC, their new “Documentation Compiler” tool that generates documentation from comments in your source code. Unfortunately, it’s not going to work for the majority of open source authors.
\n\n\n\n
A brief history
\n\n
Many folks are probably unaware that the company shipped and maintained a similar tool before, the now-defunct HeaderDoc, which had a final stable release in 2009. Eventually, Tomaz Kragelj wrote and released an open source replacement called AppleDoc, which became the standard way for Apple platform developers to publish documentation for their open source libraries. While AppleDoc began to languish from a lack of regular contributors, Swift was announced in 2014. AppleDoc was soon replaced by another community-built tool, Jazzy, authored by JP Simard and maintained by various contributors. There is also a lesser-known library called SourceDocs, authored by Eneko Alonso. More recently, Mattt took another approach at solving this problem with swift-doc. The community has mostly centered around Jazzy. After all these years of the open source community filling in the gaps and building their own tools, Apple finally announced a new first-party solution for generating documentation.
\n\n
Introducing DocC
\n\n
Overall, there are many things to like about DocC. The integration with Xcode’s Editor and Documentation Viewer is excellent. There is a command-line tool included, xcodebuild docbuild, which you can invoke from automation scripts. The overall workflow for generating and exporting documentation is built-in to Xcode and its build process, making it very easy to use. Finally, the documentation itself looks incredible. It uses the same styling as Apple’s official documentation, making it very familiar to any Apple platform developer. The design is clean and well-organized. You can even include supplemental resources, including interactive tutorials.
\n\n
The problem with DocC is hosting and distribution — arguably the most important aspect of this type of tool! What’s the point of generating amazing docs for your library or SDK if you can’t easily distribute or publish them for your users? When you run DocC, it produces a .doccarchive package that includes all the HTML, CSS, JavaScript, and other resources for your documentation website. According to the docs on distribution, you must host this .doccarchive on your web server and implement various rules to rewrite incoming URLs. The example provided defines rules in an .htaccess file for Apache.
\n\n
# Enable custom routing.\nRewriteEngine On\n\n# Route documentation and tutorial pages.\nRewriteRule ^(documentation|tutorials)\\/.*$ MyProject.doccarchive/index.html [L]\n\n# Route files and data for the documentation archive.\n#\n# If the file path doesn't exist in the website's root ...\nRewriteCond %{REQUEST_FILENAME}!-f\nRewriteCond %{REQUEST_FILENAME}!-d\n\n# ... route the request to that file path with the documentation archive.\nRewriteRule .* MyProject.doccarchive/$0[L]\n
\n\n
The docs suggest keeping it up to date using a continuous integration workflow that generates and copies your .doccarchive to your web server. For additional information on hosting, see this post on Hosting DocC Archives by Joseph Duffy. Examples provided include Netlify, Vapor middleware, nginx, and Apache.
\n\n
There is one last quirk. The default output location for the DocC archive is buried in Xcode’s DerivedData/ directory, which is… certainly a place to put it. Apple seems to think that’s a good directory for a lot of things, but I strongly disagree. Luckily, you can specify another derived data directory location when using xcodebuild docbuild.
\n\n
Ignoring prior art
\n\n
So far, this may not seem like an issue — unless you are an open source author and maintainer like me. There is a particularly vibrant community of open source iOS developers on GitHub. Most open source communities have standardized on GitHub, but especially the Apple developer community — everyone hosts their projects on GitHub. Because of that, we all host our documentation alongside our projects using GitHub Pages. It is an incredibly valuable product and service that is easy to use. First, you generate your docs — we all use Jazzy — then you place them in docs/ on your default branch. For an example, see the popular Alamofire library and its documentation. I also wrote about this in 2016.
\n\n
This is what nearly everyone does: host your project on GitHub, generate docs using Jazzy, and host your docs using GitHub Pages. In fact, no open source projects immediately come to mind that do not implement this workflow. I’m sure those projects exist, but they are the exception, not the norm. Even Apple hosts their open source projects on GitHub now and uses GitHub Pages for the Swift Evolution proposal site. More ironically, some teams at Apple actually use Jazzy and publish docs on GitHub Pages like I’ve described for some of their open source Swift packages, like SwiftNIO (see the docs here).
\n\n
DocC fails to deliver for this extremely popular use case — in my opinion, the most popular. DocC does not work with GitHub Pages — a significant barrier for adoption for essentially all open source projects in the Apple developer community. I experimented with some hacks, but was unable to make DocC do what I need. The GitHub Pages server environment is a bit opaque to the user, but it is intended for hosting static sites (Jekyll, for example), which DocC does not produce. DocC creates a Vue.js web app and requires that you run your own web server to dynamically serve it, as described above. The process feels clunky and overcomplicated compared to GitHub Pages.
\n\n
I badly want to use DocC. I would love to be able to use DocC for all my projects — but I cannot in its current form, nor can most anyone else. Unless, of course, you want to setup, maintain, and pay for a web server to host docs for your open source project that you work on for free. Who is going to do that!? Again, Joseph Duffy provided great examples for hosting, but how practical is it really? You would have to setup a new site for hosting the docs for each project you maintain. That is not tenable. Generating docs with Jazzy and hosting via GitHub Pages works great. Plus, it’s free.
\n\n
Unless something significant changes in DocC, I will continue using Jazzy. I do not have much of a choice, sadly. At the very least, it would be nice if DocC provided an option to generate a static site instead. Aside from working with GitHub Pages, a static site would hopefully require much less JavaScript, of which there is currently a seemingly excessive amount. I think Paul Hudson captured the core issue well:
\n\n
\n
As things stand, the generated documentation is effectively a sealed box […] The web pages are also very heavy on the JavaScript and honestly I’m not sure why – the reference documentation and articles are simple beasts, and if DocC could flatten them to plain HTML then I imagine the rewrite rules would just go away. At the same time, Apple’s own documentation system is completely inaccessible with JavaScript turned off, so I don’t hold out a great deal of hope here.
\n
\n\n
For whom is this intended?
\n\n
This brings me to the final question at hand — who is this tool for? I’m not being sarcastic. I can think of two main groups of users for DocC: individual open source library authors and bigger companies that provide closed-source SDKs. I’m curious — what did the DocC team envision for this tool? It certainly does not feel like it was written for our community of open source library authors, and that is beyond frustrating and disappointing.
\n\n
Could companies that distribute popular SDKs make use of DocC? I’m thinking of SDKs like Paddle, Segment, Firebase, and Facebook. However, it seems unlikely that any of these companies would use DocC. They (obviously) already have custom tooling and workflows to generate and host their own custom-branded documentation. They are not going to use DocC unless it can be heavily themed, and even then that’s probably not enough for them to switch from their current setup. Paul Hudson also points this out:
\n\n
\n
[…] the Apache rewrite rule references a theme-settings.json file, which suggest some customization is going to come, but I don’t know to what extent. I can imagine bigger companies wanting their corporate branding integrated, or their regular site navigation.
\n
\n\n
It is clear that the team behind DocC worked hard on this tool — it shows. Like I mentioned at the start, the breadth and depth of what DocC offers is impressive. Still, it feels like the creators were somewhat clueless about what third-party developers actually need, which seems to be a trend. It feels like Apple built DocC for Apple. We’ve been using Jazzy with GitHub Pages since 2014 — even CocoaDocs was using Jazzy. How could one overlook that when creating a new tool?
\n\n
This would not be so upsetting if Apple’s developer relations were not abysmal — not to mention, “the Elephant” at WWDC. For years, our community has built and maintained its own tools that work exactly how we need them. Then Apple (finally!) releases a first-party tool that is amazing for addressing 90% of the problem, but omits the most important 10% of the solution.
\n\n
And so, we are unable to use the could-have-been-so-great tool that the highly-paid software engineers at Apple built because it just barely misses the mark. We are left in stark contrast to continue maintaining our community-built tools, primarily with unpaid labor.
\n\n
To be open source
\n\n
Finally, Apple mentioned that DocC will be open sourced later this year. Some folks in the community seem optimistic about what that means for the future of DocC — maybe some determined, enthusiastic open source contributor will submit a patch that implements a “static site mode” for DocC. But at that point, what have we actually gained?
This post was updated to include mentioning SourceDocs, and that Apple actually uses Jazzy and GitHub pages for some of its Swift packages, like SwiftNIO.
\n\n
Additional updates:
\n\n
JP and friends maintaining Jazzy have opened an issue to track adding support for ingesting DocC archives as input, instead of relying on SourceKitten. So interestingly, what we may end up with is a community-built tool on top of DocC that provides the static site functionality we need.
\n\n
Sven Schmidt who maintains the Swift Package Index with Dave, suggested on Twitter that the Swift Package Index could conceivably generate and host DocC documentation. That would be pretty awesome in my opinion — it would be CocoaDocs 2.0!
\n\n
I think what I’m realizing is that community tooling is never going to be replaced by what Apple provides.
Good news! There’s a new thread on the Swift forums to discuss an official solution to support hosting DocC archives in static hosting environments, like GitHub pages.
There’s more good news to share! And hopefully this will be the final update to this post. A pull request (#44) to add static hosting support to DocC has merged into the main branch of the swift-docc repo. A second pull request (#56) is currently open to pick the changes into the release/5.6 branch, which matches the main Swift repo’srelease/5.6 branch. So, I believe we should expect an official release of DocC with support for static hosting alongside the release of Swift 5.6. However, there’s been no official post on Swift.org about the timeline or roadmap for Swift 5.6. It will likely be summer 2022.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2021/11/03/swift-package-ios-tests/#updated-17-november-2021",
"url": "https://www.jessesquires.com/blog/2021/11/03/swift-package-ios-tests/",
"title": "[Updated] How to test an iOS Swift package without an Xcode project",
"date_published": "2021-11-17T16:14:19-08:00",
"date_modified": "2021-11-17T16:14:19-08:00",
"summary": "
I came across a situation today where I needed to run an iOS test suite for a Swift Package. Previously, this required you to have an Xcode project but it no longer does.
I came across a situation today where I needed to run an iOS test suite for a Swift Package. Previously, this required you to have an Xcode project but it no longer does.
\n\n\n\n
I’m not entirely sure when this changed, but before, if you had a Swift package that supported iOS then the only way to build, run, and test that package was to have an Xcode project. Only having a Package.swift file would not suffice. The Swift Package Manager could generate a project for you.
Since it has been awhile since I’ve needed to do this, I wanted to see if there have been any changes. Luckily, there have and you no longer need an Xcode project for building and testing. This older thread on the Swift Forums explains. The generate-xcodeproj command has been deprecated, and you’ll get a warning when using it.
\n\n
If you have only a Package.swift, you can build and test via xcodebuild similar to how you would with a Xcode project. First, run xcodebuild -list to find valid schemes from the package. Then invoke xcodebuild with the desired scheme. Note that no project needs to be specified.
This makes working with iOS packages much nicer, especially for non-UI iOS libraries where you never need to actually run on a simulator. This also simplifies running tests on CI.
Reader Bernd Rabe has pointed out that testing packages this way prevents Xcode from collecting code coverage results, which I have also noticed. You win some, you lose some, I guess. I do not know of any workarounds at this time. If you need code coverage, you will need to keep using an Xcode project.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2018/07/19/blaming-homelessness-on-the-homeless-sf/#updated-06-november-2021",
"url": "https://www.jessesquires.com/blog/2018/07/19/blaming-homelessness-on-the-homeless-sf/",
"title": "[Updated] Hold it! Or, how to blame homelessness on the homeless in San Francisco: An essay on capital and compassion",
"date_published": "2021-11-06T11:47:05-07:00",
"date_modified": "2021-11-06T11:47:05-07:00",
"summary": "
The mayor of San Francisco called out feces on the sidewalks as a core problem to address in the city and wants homeless folks to “at least have respect” and “clean up after themselves”. It’s an unfortunate response, but I’m sure a lot of folks agree with the sentiment. No one relishes walking through the dirty streets in this city and it certainly is a concern, but this kind of rhetoric is actively harmful. It deliberately shifts responsibility for the problem onto the victims and away from the system that produced it. Homeless folks are among the most vulnerable in our society. In addition to their lack of housing, persistent precarity, mental health issues, and emotional struggles, the city is now going to ask them for respect and cleanliness?
The mayor of San Francisco called out feces on the sidewalks as a core problem to address in the city and wants homeless folks to “at least have respect” and “clean up after themselves”. It’s an unfortunate response, but I’m sure a lot of folks agree with the sentiment. No one relishes walking through the dirty streets in this city and it certainly is a concern, but this kind of rhetoric is actively harmful. It deliberately shifts responsibility for the problem onto the victims and away from the system that produced it. Homeless folks are among the most vulnerable in our society. In addition to their lack of housing, persistent precarity, mental health issues, and emotional struggles, the city is now going to ask them for respect and cleanliness?
“There’s more feces on the sidewalks than I’ve ever seen here. And we’re not just talking about from dogs, we’re talking about from humans.”
\n\n
“About 70 percent of the people who are estimated to be homeless in San Francisco were actually housed in San Francisco before they became homeless. We have to make sure people who live here, sadly, people who are homeless here, that they are also held accountable for taking care of our streets. This is our home.”
\n
\n\n
Shit on the sidewalks is unpleasant, but the human beings on the sidewalks are more tragic. Directing the discussion toward the dirty streets, rather than on the people who are forced to live on them is a distraction. To whom is the mayor trying to appeal? It is certainly not the homeless folks. (Do you remember “Tech Bro Justin”?) This language serves the privileged class.
\n\n
The interview continues:
\n\n
\n
“Ask them to at least have respect for the community, at least clean up after themselves, and show respect to one another and people in the neighborhood.”
\n\n
“I just solicited their help in trying to talk to their clients who, unfortunately, were mostly responsible for the conditions of the streets.”
\n
\n\n
Do you realize how much this is to ask of a homeless person? That they should respect a society that has given them none? They are denied the basic human right of housing. They are denied the dignity of free access to public toilets and showers. They hold little, if any, social or political power. They are vulnerable in every sense of the word. And the mayor suggests it is incumbent upon the homeless to clean up the streets, and that they are responsible for the conditions? Think for a moment how embarrassing and dehumanizing it must be to have no option but to defecate in public.
\n\n
What would it mean for someone on the street to “clean up after themselves”? How does a person without a home do this in a city with a dearth of public restrooms? With what resources will they clean the streets? It is an honest question. What do you want them to do? What do you expect them to do? The suggestion is not actionable by any measure I can imagine. What purpose, then, does this kind of language serve?\nIt is not offering a solution, but communicating to the homeless that their presence is unwelcome and should be concealed. It scapegoats the powerless and protects the brutality of state and capital by diverting public discussion away from critically examining and questioning the power structures that produced homelessness in the first place, a threat to current doctrine.
\n\n
The problem is not the shit on the sidewalks. The problem is that there are thousands of displaced and vulnerable people living on the street while corporations evade taxes and venture capitalists light money on fire to stay warm during a San Francisco summer.
\n\n
* * *
\n\n
What always comes to my mind is the song Land by Fifteen, an 80s-90s era punk band from Berkeley. East Bay punk will never let you down. Jeff Ott lived on the street for a number of years, and his lyrics get directly to the point.
\n\n
\n
The homeless are a problem for only one reason \nTheir presence raises the question \nWho owns the Land? \nWho owns the Land?
\n\n
[…]
\n\n
This morning I was awoken, by a man with a hand gun \nHe’s got a book of rules that says, I ain’t got no right to sleep
\n\n
This morning I was awoken, by a man with a hand gun \nHe’s got a book of rules that says, I ain’t got no right to be
\n\n
Who owns the Land? \nWho owns the Land?
\n\n
[…]
\n\n
Is there a connection \nWho owns the Land? \nTo my daily eviction \nWho owns the Land? \nAnd the extermination \nWho owns the Land? \nOf the Native American
\n\n
[…]
\n
\n\n
Homelessness is not a failure of capitalism, but a product thereof. It underscores the unequal distribution of wealth and exposes the imbalances in access to land and resources that are created by the construct of property rights. That is why it must be concealed. The city claims the homeless folks need to pull themselves up by their bootstraps, without acknowledging that the bootstraps have been pulled from their boots and tied around their hands and feet. The city refuses to install public restrooms or build public housing on the premise that there is not enough funding. But the idea that there is not enough money in Silicon Valley is absurd. The problem is never a lack of capital, but who controls it.
\n\n
While venture capitalists and banks possess most of the money, and landlords and corporations own most of the land, it is, of course, the homeless people who should be cleaning up their own shit off the sidewalks in San Francisco.
\n\n
* * *
\n\n
Resisting the atrocities of capitalism is a never-ending struggle. Aside from petitioning the city to install public restrooms, there are constructive actions you can take right now to help.
\n\n
\n
\n
Think with compassion and act with understanding. If you feel angry or disgusted by something you see on the street, think about how dehumanizing it would be to live there. Who should receive your grievances, the vulnerable or the powerful?
In the latest episode of Last Week Tonight, John Oliver’s main story was on homelessness. You should watch it. And it should be no surprise that circumstances have worsened rather than improved since I wrote this essay about 3 years ago. The pandemic certainly hasn’t helped, but it’s not the main culprit. The issues are the same and have existed for decades. The pandemic just exposed and exacerbated our problems in new ways.
\n\n
As discussed in the episode, the solution is relatively simple: to end homelessness, you need to give people homes. House people, no questions asked. And pay for it by taxing greedy, predatory billionaires. Everything else — treatment for addition, employment, etc. — comes second. There is no hope to address any other concerns unless you house people first.
\n\n
Capitalism has conditioned us to view a person experiencing homelessness as an individual failure, rather than our collective failure as a society to provide basic human needs, rights, and dignity to everyone. But capitalism is not capable, nor designed, to care for and meet the needs of everyone — if it were, we wouldn’t be discussing this.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2021/11/01/how-to-start-a-blog/#updated-05-november-2021",
"url": "https://www.jessesquires.com/blog/2021/11/01/how-to-start-a-blog/",
"title": "[Updated] How to start a blog or portfolio website, for developers",
"date_published": "2021-11-05T21:53:09-07:00",
"date_modified": "2021-11-05T21:53:09-07:00",
"summary": "
Ever so often an iOS developer asks me how to get started with making their own blog or portfolio website. Or, I’ll see a software developer from another community on Twitter ask the same thing. Often they are earlier in their career, or unfamiliar with web development, or unsure whether to build from scratch or use a platform, or all of the above. I find myself consistently making the same recommendations to folks. For this post, I want to share what I think is a great approach to get started, and how you can dive deeper once you master the basics.
Ever so often an iOS developer asks me how to get started with making their own blog or portfolio website. Or, I’ll see a software developer from another community on Twitter ask the same thing. Often they are earlier in their career, or unfamiliar with web development, or unsure whether to build from scratch or use a platform, or all of the above. I find myself consistently making the same recommendations to folks. For this post, I want to share what I think is a great approach to get started, and how you can dive deeper once you master the basics.
\n\n\n\n
Things to consider
\n\n
The first thing to consider is how much time and effort you want to commit to not only creating a site, but maintaining it. You’ll also need to decide what you want your site to be. Some people only want a single-page site that links to their various online profiles — GitHub, LinkedIn, Twitter, etc. Others prefer a multi-page portfolio site. Some want to start a full blog where they will be regularly writing articles. You may want some combination of these, or you might want to start with a single-page site that eventually grows into a blog.
\n\n
If you are only interested in a simpler portfolio site and don’t want to worry about maintenance, it’s probably worth considering one of the popular platforms: Squarespace, Wordpress, Wix, etc. Those will get you up and running pretty quickly. However, using these types of platforms is not the focus of this post. I don’t have much experience with these platforms, but I know that they work as advertised, more or less.
\n\n
However, not only are these platforms typically more expensive compared to building your own site, they give you much less control. I personally find them extremely frustrating to use as a software developer. They might be fine to begin with, but eventually you’ll start running into their limitations. If you are planning to start a blog with technical writing, I highly discourage them as they are terrible with formatting code snippets and syntax highlighting.
\n\n
The other question to ask yourself before starting a blog is: do you really want to start a blog? You might be unsure, and that’s ok! The reason I bring this up is to encourage you to start small and do the simplest thing first. You may write a couple of blog posts and decide that you actually hate blogging. (That’s also ok.) What you want to avoid is spending a lot of time (and/or money) setting up some complex website that you actually don’t want.
\n\n
Prerequisites
\n\n
This post assumes some basic working knowledge of HTML, CSS, and Javascript. You don’t need to be an expert. Being able to read and write simple HTML and CSS is all you need. Some Javascript experience is good too, but that’s less important for a static site. If you don’t have any experience with those, there are plenty of guides and tutorials online. These technologies lend themselves well to a “figure it out as you go” approach.
\n\n
The only other pre-requisite is having a GitHub account, which I think the vast majority of software developers do. In my experience, it’s the best platform out there for software projects — much better than BitBucket or GitLab.
\n\n
Getting started
\n\n
This post is primarily geared toward someone who is interested in making their own website that includes:
\n\n
\n
A handful of pages with information about yourself, your projects, your resume, your conference talks, etc.
\n
A blog where you can write and publish posts — or, at least the potential to start a blog later if you are currently undecided
\n
\n\n
What I highly recommend to folks is using GitHub Pages to get started. It’s a great place to experiment, get started quickly, and build a great-looking site. Maintenance is also near zero if hosting on GitHub. And — it’s free.
\n\n
About GitHub Pages
\n\n
In many ways, GitHub Pages a great middle-ground between building from scratch and choosing a platform like Squarespace. You can get your site up and running in a few minutes using their built-in templates and hosting on GitHub. The difference is that with GitHub Pages, you can keep advancing into the technical side of creating on your own site well beyond what any of these platforms offer. Better yet, you can move at your own pace and dive as deep as you like.
\n\n
Let’s step through the process, from the simplest options to the most advanced. Keep in mind that you can stop at any point when you reach a level with which you are comfortable. You can always progress to something more advanced later.
\n\n
A simple GitHub Pages site
\n\n
The main GitHub Pages landing page has a great getting started guide, complete with a video and links to additional guides and documentation. In short, it’s built on Jekyll, a static site generator. This means you can write some markdown and Jekyll will turn that into a webpage.
\n\n
After stepping through their guides to create your new repository and choose a template, you’ll have your site up and running at username.github.io. It should only take a few minutes. You can start adding new pages to your website or start writing blog posts right away. Better yet, this is all possible to do within the GitHub web UI. In fact, you can build an entire GitHub Pages site without ever opening terminal or cloning the repo to your own machine. In this way, it closely approximates how you would work with something like Squarespace. Of course, I would not necessarily recommend doing this. You may want to preview something before you publish it. In that scenario, you would need to build and preview your site locally, then git push the changes to be published.
\n\n
I use GitHub pages as a sort of homepage for my open source projects. If you want to see an example, you can find the repo here and the rendered site here. This is hosted on GitHub using one of their built-in templates. It’s just a simple, single-page website.
\n\n
If you are completely new to this, I would take some time to get acquainted with how GitHub Pages and Jekyll work. Experiment, publish, iterate. Again, both have great tutorials and documentation, so I won’t reproduce them here.
\n\n
Getting a custom domain
\n\n
Once you feel comfortable working with and updating your site, the next big step will be setting up a custom domain instead of using the .github.io subdomain. The good news is that the .github.io domain will automatically redirect to your new custom domain. So if you have already been sharing your site, don’t worry about broken links.
\n\n
I view this step as “committing” to your website, since this is where you’ll start to spend money. Registering a domain is typically inexpensive, but I think it still represents a commitment. You have decided “yes, this is for me” and you want to continue. If you are not enthusiastic about continuing, you can stop here. You can even delete your repo, and thus, delete your website.
\n\n
Any registrar will work. I use NearlyFreeSpeech.net. The hardest part will be configuring the DNS records correctly, but the GitHub guides should help. Whatever registrar you choose should also have guides, or at least easy-to-find settings to configure.
\n\n
A more advanced GitHub Pages site
\n\n
At this point, you have a fully working website with a custom domain. That’s awesome! The next big step will be implementing a custom design for your site instead of relying on the limited selection of GitHub Pages themes. This sounds much more intimidating than it is, but it can be tricky.
\n\n
So far, GitHub Pages has hidden a lot of implementation details from you. As I mentioned before, it is built on Jekyll. So diving deeper means “lowering” into Jekyll directly. I recommend going through Jekyll’s step by step tutorial. Don’t worry about your content, that will all be preserved. This step is only about abandoning your GitHub Pages theme and implementing a custom design.
\n\n
The tutorial explains everything quite well, but I want to call out one thing. You’ll be creating layouts, or templates. Note that layouts can inherit from other layouts. Another useful concept is includes, which are reusable snippets of HTML. These are useful, for example, for building navigating menus or footers that should appear on every page.
\n\n
If, like me, you are not a web designer, then designing a website sounds intimidating. However, there are many tools and frameworks to help. I’ve been using Bootstrap for years, and I really enjoy it. It is a well-establish and well-maintained frontend library. Think of it like UIKit, but for your website. There are many of these types of frameworks, like tailwind and gatsby. Bootstrap has excellent documentation and a great collection of examples. You’ll use one of these frameworks to make your layout templates, then your workflow will be similar to GitHub Pages where you simply write markdown, then git push to publish.
\n\n
If you would prefer to avoid a custom design from scratch, then I recommend checking out the vibrant collection of third-party themes at JekyllThemes.io.
\n\n
Diving deeper: a custom Jekyll site
\n\n
Now you have a custom-built, custom-themed website using Jekyll, which you host for free on GitHub Pages. This is what I used to have, but I eventually moved away from hosting on GitHub. I wrote about that in-depth here. Again, I highly recommend NearlyFreeSpeech.net for hosting, but they are very bare-bones and DIY. If using them, expect to spend some time learning about hosting, DNS, and general web technologies, as well as trouble-shooting issues. After a few years hosting on GitHub Pages, I feel like I simply outgrew it as I started to run into various limitations and desired more control. That’s why I decided to host my website elsewhere. You may never get to this point, and that’s fine.
\n\n
If you want to dive even deeper into Jekyll, you’ll need to learn some Ruby. Jekyll is written in Ruby and its templating language is called Liquid. One neat, more advanced feature of Jekyll is writing custom plugins, which are written in Ruby. This allows you to extend Jekyll itself.
\n\n
I should also note that if you prefer not to design your own site, you can keep using the GitHub Pages theme you originally selected, even if you are not hosting on GitHub. To do this, you’ll need to include the GitHub Pages gem instead of plain Jekyll.
\n\n
Other resources
\n\n
I occasionally write about Bootstrap and Jekyll on this blog. You may find those posts interesting as you advance.
\n\n
This entire website is open source if you need examples or want to see how all the pieces fit together. My only request is that you don’t reproduce an exact (or near-exact) copy of my site. The intent is to learn from it, not to duplicate it.
\n\n
Finally, I maintain a template repo on GitHub that you can use to get started on a more advanced site using Jekyll and Bootstrap.
\n\n
Alternatives
\n\n
As I mentioned before, the mainstream platforms might be a better fit for what you are trying to achieve. If that’s the case, you should not feel like have to do something more technical just because you’re a software developer. Good software developers choose the right tool for the job.
\n\n
One alternative that is similar to Jekyll would be using John Sundell’s Publish, which is a static site generator written in Swift. This might appeal more to iOS developers. You could write your entire site in Swift. I do not have any experience with this, but I’m sure you can find tutorials, examples, and guides online. Kaya Thomas used Publish for her personal website, and I think it looks great. And of course, Swift by Sundell uses Publish, if you want another example.
\n\n
However, if you have the time, patience, and interest then I highly encourage you to explore a new technology stack. If you are primarily an iOS developer, building your own website with Jekyll is great way to explore something completely different.
daiyi (chris) just published an interesting article, How to automate previews for Github Pages. It’s a pretty neat hack to get previews for website changes without having to run Jekyll locally or publish first. If you are going all-in on a GitHub pages site, this might be of interest to you.
\n"
} ,
{
"id": "https://www.jessesquires.com/blog/2018/02/25/replacing-google-with-duckduckgo/#updated-23-october-2021",
"url": "https://www.jessesquires.com/blog/2018/02/25/replacing-google-with-duckduckgo/",
"title": "[Updated] Replacing Google Search with DuckDuckGo: And how to add a custom DuckDuckGo search box to your site",
"date_published": "2021-10-23T10:49:17-07:00",
"date_modified": "2021-10-23T10:49:17-07:00",
"summary": "
I’m not interested in being an advertising product for Google to exploit. I’m also not interested in the company’s unsavory practices, in general. I’ve been using DuckDuckGo for over a year now, and I’m incredibly happy with it as a replacement for Google Search — not only for personal usage, but also for implementing a custom search component for this site.
I’m not interested in being an advertising product for Google to exploit. I’m also not interested in the company’s unsavory practices, in general. I’ve been using DuckDuckGo for over a year now, and I’m incredibly happy with it as a replacement for Google Search — not only for personal usage, but also for implementing a custom search component for this site.
Google no longer provides you with a simple list of ranked results. To become a major player in the attention economy, Google Search must track you across the web and mine your personal information to provide you with “personalized results” and to sell your information to advertisers. Google can now even track your offline purchases — that is, at physical stores in the real world. Pretty creepy. The company strongly emphasizes how they’ve done this securely, to protect users — but that’s beside the point for me. I do not want this intrusion into my daily, offline life.
\n\n
Google’s enormity gives it undueinfluence on content providers and publishers. In particular, Google AMP results, which are diplayed prominently in search, threatenthe web at a fundamental level.
\n\n
The lack of ethics codified into Google search are disappointing and irresponsible. Google has positioned itself as the digital arbiter of truth. However, this is undermined by its pursuit to grab a slice of the attention economy, which requires curating unique search results for each user. This combined with an absence of fact-checking produces outcomes that range from unfortunate to alarming to horrifying. Google’s curated results have the power to shape and distort people’s views. That’s an ethical problem.
\n\n
I’ll give them credit though. Google has worked to address some of the problems in the articles that I’ve linked to above. However, the company’s financial incentives remain the same as far as I can tell. As long as they are beholden to satisfying advertisers and scrounging for users’ attention, I’m not hopeful that the trolls won’t find new ways to turn the search engine into an alt-right propaganda machine again.
In completely unsurprising news, Google’s search results have gotten worse over time. And while I’m here — updating this post 3 years later — I have to say, DuckDuckGo has been great. I have no complaints.
\n\n
\n\n
Benefits of DuckDuckGo
\n\n
Similar to many Apple products and services, DuckDuckGo was established with privacy as a fundamental feature, not to mention a company value. While their revenue model is still based on advertising, the service does not mine your data to do so. DuckDuckGo also sometimes adds an affiliate code to some eCommerce sites, but this is done without using any personally identifiable information. All the details are clearly explained in their privacy policy, which you can compare with the Google privacy policy. Google collects and shares a staggering amount of your data that is personally identifiable. They obscure how they use it behind vague phrases like “improving your user experience”, and even after deleting your information from their services they reserve the right to retain a copy for “legitimate business or legal purposes”. DuckDuckGo provides a strikingly different policy, which puts privacy first and even includes an “Information Not Collected” section.
\n\n
\n
You can usually find out a lot about a person from their search history.
Gabriel Weinberg has a rather comprehensive list of reasons to use DuckDuckGo. I won’t bother reiterating each of them, but I will highlight one of my favorite features — !bangs, which let you search other sites directly. For example, you can type !giphy oh yeah into DuckDuckGo and land directly on Giphy’s search results for “oh yeah”.
\n\n
Quality of search
\n\n
After over a year of using DuckDuckGo, I think search quality is excellent. At times it could be better, but that’s a trade-off that I’m willing to make for privacy and a more ethical product. Overall, I’m incredibly happy using DuckDuckGo. If you aren’t satisfied with DuckDuckGo’s results, you can always !google to search again. I use this occasionally, though it’s increasingly rare.
\n\n
Implementing a custom search box
\n\n
Google allows you to create a free custom search component and embed it into your site, which I used here before switching to DuckDuckGo. You enter your site url, configure some settings, and then grab the code snippet to embed it. It sounds simple, but it is actually a heavy-handed and cumbersome script-based approach. It’s an exercise in pain to customize the appearance, and it won’t appear when using most ad-blockers.
\n\n
Creating a custom DuckDuckGo search is much simpler. They provide an <iframe> snippet, which is not ideal. Luckily, you can create and style your own search box with basic html forms — something Google doesn’t offer.
This adds a basic form with a few custom URL parameters and custom styling using bootstrap. You can add this to your own site by replacing jessesquires.com on line 5 with your own domain. You can see and use my custom search here.
\n\n
You can find the diff on GitHub for deleting Google custom search and replacing it with DuckDuckGo. It’s much more elegant, lightweight, and just looks better.
\n\n
Thanks to Stuard Colville for his helpful post on doing this without using iframes.
\n\n
Making the switch
\n\n
If you value privacy and the open web, it’s time to switch. By supporting DuckDuckGo, you will only help improve their service. Contrary to what most giant tech companies would have you believe, it is possible to build a great service without exploiting users’ privacy and personal information.
Bing was caught blocking results for Tiananmen Square protests in Hong Kong. Per Gabriel Weinberg: “China banned DuckDuckGo in 2014 and we have no plans to change that.”
\n\n
It is rare to see a tech company not cooperate with authoritarian governments.
A new report by Thomas Brewster at Forbes: “The U.S. government is secretly ordering Google to provide data on anyone typing in certain search terms, an accidentally unsealed court document shows. There are fears such ‘keyword warrants’ threaten to implicate innocent Web users in serious crimes and are more common than previously thought.”
\n\n
@DuckDuckGo: “Google complies with invasive ‘keyword warrants’ that identify anyone who searched for a term. DuckDuckGo doesn’t have any search histories by design and, bc of that, has had 0 search warrants (of any kind) since our founding in 2008.”
![\"dog](\"/img/blog/dogscience.gif\" "\\"dog")
\n ![\"Numbers\"](\"/img/blog/numbers.jpg\" "\\"Numbers\\"")
\n ![\"Floating-point](\"/img/blog/float32memory.png\" "\\"Floating-point")
\n ![\"Screen](\"/img/blog/screen-time-1.jpg\" "\\"Screen")
\n ![\"Screen](\"/img/blog/screen-time-2.jpg\" "\\"Screen")
\n ![\"Screen](\"/img/blog/screen-time-3.jpg\" "\\"Screen")
\n ![\"iMessage](\"/img/blog/imessage-spam-1.jpg\" "\\"iMessage")
\n ![\"iMessage](\"/img/blog/imessage-spam-2.jpg\" "\\"iMessage")
\n ![\"iMessage](\"/img/blog/imessage-spam-3.jpg\" "\\"iMessage")
\n ![\"iMessage](\"/img/blog/imessage-spam-5.jpg\" "\\"iMessage")
\n ![\"Steps](\"/img/blog/imessage-spam-4.jpg\" "\\"Steps")
\n ![\"iMessage](\"/img/blog/imessage-spam-6.jpg\" "\\"iMessage")
\n ![\"macOS](\"/img/blog/mac-sigh-of-death-1.jpg\" "\\"macOS")
\n ![\"macOS](\"/img/blog/mac-sigh-of-death-2.jpg\" "\\"macOS")
\n ![\"Time](\"/img/blog/time-machine-fail-again.jpg\" "\\"Time")
\n ![\"16th](\"/img/blog/16th_st_bart.jpg\" "\\"16th")
\n ![\"DuckDuckGo\"](\"/img/blog/duckduckgo.png\" "\\"DuckDuckGo\\"")
\n