Implement multi-patch support in patch-package

[roryabraham]

Problem

patch-package is a simple and useful package that we use to implement bug fixes and new features in 3rd-party libraries without creating a fork. However, it's limited in that each package can have only one monolithic patch file. This makes it difficult to keep track of separate changes in a 3rd-party package, which is one of the reasons we've opted for forks in the past. You can track pull requests in the fork in parallel with their upstream equivalent and use that to reconcile all the custom changes we've made to a 3rd-party dependency. This also helps indicate when our code has landed upstream and the fork/patch is no longer needed.

Solution

Upgrade patch-package to support multiple patch files per dependency. To ground this with an example, let's say we wanted to patch an image bug in React Native, and also wanted to build a new feature in the TextInput component.

For each of these features, we could have a separate patch file for each patch, and those patches would be applied in sequence:

patches/react-native+0.72.0+ImageBug.patch
patches/react-native+0.72.0+TextInputFeature.patch

We could have a separate PR in E/App to introduce each of these patches, and those PRs would link to their upstream equivalent.

Furthermore, we should have error reporting on a per-patch basis, and a way to resolve conflicts in specific patches while upgrading the base version.

Additional context

• Original slack thread
• Upstream issue
• Accounting note: Time spent on this should be considered R&D from a CAP SW perspective.

[melvin-bot]

Current assignee @mallenexpensify is eligible for the NewFeature assigner, not assigning anyone new.

[roryabraham]

@shannonmccallexfy I'm assigning you here so that you can help make sure this project gets counted as CAP SW. I guess the Accounting auto-assigner doesn't work in E/App

[ds300]

Hi! 👋🏼 I'm David Sheldrick, the author of patch-package.

I had a call with @roryabraham a couple of days ago to hash out a prospective scope for this feature. He asked me to write up the following proposal:

Scope

The project can broadly be broken down into three core features:

1. Patch sequence application.

   This would give the ability to load and apply multiple ordered patches to a single package, and for patch-package to convey to the user which patches failed, if any, during application. This would work the same way as cherry-picking a series of commits in git. The patch files should support semantic descriptions in the filenames, and might look something like:

   patches/react-native+0.72.0+01.ImageBug.patch
   patches/react-native+0.72.0+02.TextInputFeature.patch
   

   patch-package will attempt to apply all patches, even if one at the start or in the middle of the sequence fails to apply. And it will not leave merge conflicts in files. This matches the current single-patch behavior.

2. Patch sequence generation.

   This would give the ability to generate a new patch file on top of existing patch files, and to amend the 'latest' patch file. This would be achieved with an extra cli option, e.g.:

   • patch-package react-native --append 'FixPressableEventDispatch' to generate a new patch file with any extra changes added on top of existing patches.

     It would create a patch file named something like

     patches/react-native+0.72.0+03.FixPressableEventDispatch.patch
     

   • patch-package react-native to amend the latest patch file (this syntax mimics the current patch generation behavior)

   The existing workflow of simply editing the relevant files inside node_modules and running patch-package to generate the patch file, whether to amend or append, would remain unchanged.

3. Error recovery via --rebase

   Let's say you have a sequence of five patches for react-native which build on one another. If the second patch in the sequence fails to apply after upgrading react-native, but you don't want to simply delete it, you should be given the opportunity to resolve the merge conflicts manually. patch-package can enable this by allowing you to run a command like

   patch-package react-native --rebase
   

   Which will

   • Undo any patches that were previously successfully applied
   • Re-apply the patches in order until one of them fails
   • Insert merge conflict markers into any files where conflicts were detected, as git does during merge/rebase.
   • Print a list of the files with conflicts, and the name of the patch that caused them.
   • Allow you to run patch-package react-native --rebase-continue to update the failing patch with the resolved changes and continue the rebase.

All features will have appropriate unit and integration tests, in line with the existing patch-package codebase.

Integration with existing patch-package features

The --create-issue flag will not be supported.

The .dev.patch patch file suffix will be supported. This allows patch files created for transitive dev dependencies to be safely ignored if the package is not present in node_modules.

Budget allocation

The total budget for this project is $20,000, and I'd propose allocating it between two milestones as follows

1. Patch sequence application + generation: $15,000
   These two features are the bulk of the work, and it doesn't make sense to ship one without the other so I'd propose combining them.
2. Error recovery via --rebase: $5,000

This would include a two-month period of asynchronous support + bug fixing while you battle-test the features, starting on the day of final delivery of milestone 2. I can also provide up to 2 hours of synchronous zoom support in the EST time zone over the same two-month period if needed, scheduled ad-hoc and subject to availability.

Timeline

I expect to work on this in 1–2-hour chunks over the period of a month or so. I'll be working on it in my spare time during the height of summer so I can't commit to a fixed timeline, but I believe it's a realistic estimate. I'm obviously happy to keep you updated on my progress on a regular schedule (e.g. on Mondays and Thursdays, or whatever works for you), and I'll do the work in the open on the patch-package repo so you can follow along.

Payment method

If convenient you can pay me via one-time GitHub sponsors donations, which I already have set up on the patch-package repo. Otherwise Rory mentioned that you normally use Upwork, which I am happy to register for.

Thanks,
David

[roryabraham]

Patch sequence generation.

@ds300 What if you want to edit a specific patch file rather than the latest one? Maybe there should be three options:

• npx patch-package my-package – creates a new patch file if one does not exist, or edits the latest patch if one does exist
• npx patch-package my-package --append 'PatchName' – creates a new patch file to be applied after any existing patch files. Fails if there's already a patch w/ name PatchName
• npx patch-package my-package --update 'PatchName' – looks for any patch file matching PatchName. Fails if no such file exists, otherwise it overwrites that specific patch file.

The use-case for that might be that you introduced a patch associated with an upstream PR, but then you make changes to the upstream and want to apply the same changes to your patch. Rather than creating a second patch associated with the same upstream PR, you would update the existing one.

Integration with existing patch-package features

👍🏼 to both of the items there.

[roryabraham]

Overall, super excited for this feature, and thrilled that you're ready to build it for us and the whole community. I've assigned this issue to you, which means that we agree to your proposed scope and budget and you can start work whenever you're ready. Thanks! 🙇🏼

[ds300]

First Monday check-in! 🕐

I haven't broken ground yet, planning to get started tonight after dinner.

What if you want to edit a specific patch file rather than the latest one?

I think this would have to be part of the rebase tool, in order to make sure that it works for patches that build on one another sequentially.

Just like git's rebase it could target a particular patch file in the 'history' and roll back to that point.

patch-package react-native --rebase 'PatchName'

When you're done --rebase-continue would update the patch file you specified and then re-apply any subsequent patches.

[ds300]

The work is happening in this PR ds300/patch-package#474

[mallenexpensify]

Thanks for the help with this @ds300 , I'll be managing payment. Our normal process is to issue payment 7 days after PRs hit production, to ensure their aren't any regressions. I'm unsure if that makes sense here though. Once we get closer to completion we'll discuss payment options, it'd be great to minimize any fees you might have to pay.

[ds300]

Thursday check-in! 🕐

I have finished an initial implementation of patch sequence application and have done some testing for it. It seems to work well but in order to make it usable for patch generation and rebase it will need to be refactored to decouple the patch application bits from the error handling and console logging bits.

I plan to get started on that on Saturday morning.

[ds300]

Monday check-in! 🕐

I got an initial implementation of patch sequence generation working. I need to improve the error reporting a little and do some more integration testing for it. After that I'll do some QA on different OSes, node versions, and package managers. After which I'll merge this PR and ping you. Probably some time later this week. Then I'll get started on the rebase feature.

I published a canary release of this branch on npm. You can do npm install patch-package@canary to try out the --append flag.

[ds300]

No changes since Monday. I expect i'll be able to work on it next on Monday

[ds300]

Monday check-in! 🕐

I did some manual testing on all three platforms, then added and tested some sanity checks. I published a new canary and will merge the PR soon to begin work on the --rebase feature.

[ds300]

Nothing to report today. I'm planning to get started on --rebase on Saturday morning.

[roryabraham]

Thanks for the consistent updates @ds300!

[ds300]

Monday update 🕐 !

tldr; ran into some unexpected complexity with patch sequence application. It is resolved now.

While sitting down on Saturday to review the current state of the work and plan the --rebase feature I noticed that the new sequenced patch application code was no longer idempotent, and indeed would fail on subsequent attempts if patches build on top of one another in overlapping regions of code.

The patch application idempotency has so far been achieved by the following algorithm:

try to apply patch
  if successful, we're done
  if not successful:
    reverse the patch and see whether it applies (do a dry run)
      if it does the patch was already applied
      if it does not the patch is corrupt or outdated

With a sequence of patches, doing it in this fashion (i.e. trying to 'dryly' apply the sequence backwards) would introduce a good bit of extra complexity.

Also, while thinking about the --rebase feature I realised it would require storing some state on disk the same way git does. So I opted to store a 'state' file already, in the patched package directory, e.g. node_modules/react-native/.patch-package.json which records the hashes of any applied patches the last time patch-package ran. It now checks the state file and the current patch file hashes to see whether or not it needs to run again.

We still use the old algo above if there is only one patch to apply, since it is battle-tested and naturally more robust due to being stateless.

I just finished this off and am now planning to get started on --rebase in earnest tomorrow. I have the day off on Wednesday and I'm planning to get the feature alpha testable by the end of the week.

I also just published a new canary with the idempotency fix.

[roryabraham]

FWIW it looks like we are going to take a stab at using this feature tomorrow. So @fabioh8010 if you have any feedback on the feature feel free to leave it here.

[fabioh8010]

Hi 👋 I was able to use the append feature in this canary version 🎉 . I have one question, in case I want to "edit" a specific patch or the last one, is it possible to do it? I wonder if it's the case for the rebase option which is being developed, but I would want to confirm. Thanks!

[ds300]

@fabioh8010 that's great news! 🎉

I have one question, in case I want to "edit" a specific patch or the last one, is it possible to do it?

It's currently possible to edit only the last patch, by making changes inside node_modules and running patch-package react-native.

The --rebase feature will allow you to edit any of the previous patches, or insert a new patch into the sequence.

🕐 I have nearly finished implementing an initial version of --rebase. I expect to release it in a canary tonight or tomorrow night. I will post here when it's available.

[roryabraham]

Thanks for your patience @ds300, we're discussing the best way to pay you.

[mallenexpensify]

@ds300 the plan is:

• Pay $12,000 USD (max allowable) for a sponsorship via https://github.com/sponsors/ds300
• Pay $8,000 USD balance via GH sponsorship (assuming I'm able to after the initial payment)

That sound good? I should be able to handle this week, I'm waiting for my card limit to be raised

[ds300]

Works for me, thanks @mallenexpensify 🙏🏼

[mallenexpensify]

@ds300 , I've checked with out accounting team and we need to pay you through Expensify.

Contributor: @ds300 is due $20,000 via Expensify NewDot.

@JmillsExpensify and/or @anmurali can you please issue payment? What are the steps @ds300 needs to do in order to receive payment, it looks like he's in the UK.

[JmillsExpensify]

@mallenexpensify Can you add him to the Contributor Payments workspace owned by contributors@expensify.com? Then he'll need to Request money in his policyExpenseChat.

[mallenexpensify]

Thanks @JmillsExpensify I've added @ds300 via their gmail address we've communicated through before. I send these steps, let me know if anything doesn't look correct

• Big Green Plus button
• Request Money
• Manual > Amount ($20,000)
• contributors@expensify.com
• Enter Implement multi-patch support in patch-package #20770 into description

Anyone know what the best payment setup is for someone in the UK? Wise? Can we do direct transfer/payment to a bank account there?

[roryabraham]

Posted question in slack here

[anmurali]

1. Please make sure you have added them to the Global Reimbursements beta though and to the Expensify Contributors (Collect) workspace in OD as well as Expensify Contributors workspace in ND
2. Please make sure they have their default workspace set to the Expensify Contributors (Collect) policy on Old Dot before going to the next step
3. Instructions for a contributor plus to add their deposit bank account to get paid is here
4. Instructions for a contributor plus to request a payment is here

Anyone know what the best payment setup is for someone in the UK? Wise? Can we do direct transfer/payment to a bank account there?

They can add their UK bank account or a Wise account, it's their call.

[mallenexpensify]

I've been in touch with @ds300 and provided details. Once I've confirmed @ds300 is setup I'll comment here for payment.

I've updated internal documentation in this SO

[mallenexpensify]

Contributor: @ds300 is due $20,000 via NewDot.

@ds300 please confirm you've followed the steps to request payment that I sent you via email.

[JmillsExpensify]

$20,000 payment approved via NewDot based on BZ summary.

[ds300]

@ds300 please confirm you've followed the steps to request payment that I sent you via email.

I confirm

[mallenexpensify]

Thanks @ds300. Can you comment once you've received the payment then I'll close this.

[melvin-bot]

@ds300, @mallenexpensify, @roryabraham, @shannonmccallexfy Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[ds300]

I just received a notification saying that an ACH direct deposit was made, and I should have the money by Oct 4th.

Will post an update here once it's gone through.

[ds300]

The payment went through! Thanks for figuring this out @mallenexpensify and @roryabraham. It's been a pleasure to work with you both.

[roryabraham]

Great news, thanks for building this awesome feature for us. I hope the open-source community finds it as valuable as we do!

[melvin-bot]

@mallenexpensify @roryabraham @shannonmccallexfy Be sure to fill out the Contact List!

[mallenexpensify]

Thanks @ds300 , glad everything worked out!