I am trying to provide the users a way to generate their own workflow as part of the system. These workflows will be custom paths an order will take depending on customer requirements. For example: If a customer requires us to sign a set of terms and conditions, the order should not be able to be approved unless a T&C document has been uploaded.
I have been looking at using bpmn-js for the frontend and execute the output BPMN2.0 file every time something changes that is related to the workflow (i.e. hooks on when documents are uploaded in this case) but it doesn't look like the users will be able to select actual system functionality with that library out-of-the-box. Should I try to extend that library or is there something else I could use instead?
I have been looking into using Camunda as well, but it would be nice to not expect the users to use a second application.
Design your application in a way that the business process will be driven by the process engine. When a process is initiated it start a process instance. From there the process engine (embedded in your existing Java application or standalone back-end service) determines if business rules (DMN) need to be evaluated (all required data and approvals present?), services need to be called (invocation of your java code directly), automated (external task pattern) or user task need to be completed by technical workers (any language) or human process participants (based on assignment determined by engine).
If humans are involved, the UI/client queries the process engine for pending user tasks and updated those tasks when humans have performed the tasks.
The process engine will determine the next steps based on the interpretation (no code is generated) of the BPMN2-standard-based process model. Versioning for these is provided ootb. Newly started processes are automatically started on the latest version while running process instances continue their life-cycle on the version they were started on (unless they are migrated).
It is usually not an issue if users need to access a dedicated modelling environment at design time. They just should not have to work with two applications at runtime. Anyway, it is also easily possible to integrate the modelling part into the same application via the BPMN-js library you mentioned.
The "selection of system functionality" is done by selecting the implementation type and setting the corresponding attributes. The BPMN-JS library is generic in this regard. have a look at how this is done in the Camunda Modeler (https://camunda.com/products/camunda-bpm/modeler/):
Java: https://docs.camunda.org/get-started/java-process-app/service-task/ Spring: https://docs.camunda.org/get-started/spring/service-task/ External: https://docs.camunda.org/manual/latest/user-guide/process-engine/external-tasks/