ash/documentation/topics/bulk-actions.md
Zach Daniel 5967ed3a48 improvement!: 3.0 (#955)
* improvement!: use `%Ash.NotSelected{}` for unselected values

* improvement!: default `require_atomic?` to `true`

* improvement!: raise errors on unknown generic action arguments

* improvement!: default bulk strategy to `:atomic`

* improvement!: warnings on `require_atomic?` `true` actions
improvement!: revise `Ash.NotSelected` to `Ash.NotLoaded`
improvement!: errors on unknown action inputs across the board

* doc: clarify wording in notifiers.md

closes #889

* improvement!: default `api.authorization.authorize` to `:by_default`

* improvement!: require the api when constructing changesets

this commit also fixes some work from prior commits around
the default value for the `authorize` option

* improvement!: code_interface.define_for -> code_interface.api

`code_interface.define_for` is now `code_interface.api`. Additionally, it is set automatically if the `api` option is specified on `use Ash.Resource`.

* improvement!: remove registries

* improvement!: pubsub notifier default to `previous_values?: false`
improvement!: requires_original_data? callback defaults to false

* improvement!: rename Ash.Calculation -> Ash.Resource.Calculation
improvement!: improve `Ash.Query.Calculation.new` signature
improvement!: anonymous function calculations now take lists and return lists
improvement!: make callback contexts into structs
improvement!: pass context to builtin lifecycle hook changes
improvement!: calculation arguments are now in the `arguments` key of the context

* chore: fix build

* improvement!: remove `aggregates` and `calculations` from `Filter.parse` and `Filter.parse_input`

* improvement: update spark to 2.0

* improvement!: make picosat_elixir optional with `simple_sat`

* improvement!: rename api to domain

* docs: add more info to upgrading guide

* docs: tweak docs formatting

* improvement!: remove `Ash.Changeset.new!`

* docs: update docs for `Ash.Changeset.new/1`

* improvement!: deprecate `private?: false` in favor of `public?: true`

* doc: add upgrade guide for private -> public

* improvement: update reactor to 3.0

* improvement!: default `default_accept` is now `[]`

* improvement!: `Ash.CiString.new/1` returns `nil` on `nil` input

* improvement!(Ash.Reactor): Improve integration with Ash 3.0 changes.

* improvement!: clean up and reorganize `Ash` functions

this is in preparation of deprecating the functions that are defined
on the api

improvement!: remove context-based functionality

* chore: update docs references from `Ash.Domain` to `Ash`

* chore: fix bad merge

* chore: fix context access in atomic changes

* improvement!: Deprecate calling functions on (domain) api in favor of `Ash`

* improvement!: add `attribute_public?` and update `attribute_writable?` behavior

* improvement!: update atomic behaviors, default to invalid

* chore: update downcase docs

* improvement!: changeset.filters -> changeset.filter

* improvement!: remove deprecated functions

* improvement!: remove and simplify `Ash.Filter.TemplateHelpers`

* improvement: import Ash.Expr in modules where it is used
improvement: require Ash.QUery in modules where it makes sense

* fix!: keyword lists are no longer special cased in ash expressions

* improvement: add structs for more context implementations

* chore: small tweaks, finish `:all` -> `:*` conversion

* chore: update DSL docs for multitenancy.global?

* improvement: ensure selects are applied on destroys
chore: remove TODOs

* chore: some docs changes

* improvement!: introduce strict mode to calculations

* chore: update tests

* improvement: support custom expressions

* docs: document custom expressions

* chore: fix and test custom expressions and function fragments
docs: update relevant docs w/ the changes

* improvement!: reverse order of before action & before transaction hooks

* improvement!: default read actions are now paginatable

* improvement!: require explicit accept lists in default actions

* chore: update docs

* improvement!: remove Ash.Flow and Ash.Engine

* chore: unlock unused deps

* chore: don't use unused variable

* chore: include ash flow change in upgrade guide

* improvement!: standardize various exception keys and names

* improvement!: use `Splode` for errors

* improvement: update upgrade guide to include Splode

* feat: code interface on the domain

* improvement: only require primary key if resource has actions or fields
improvement: only build schema if resource has actions or fields
improvement: verify primary key in its own verifier

* improvement: add `resource/1` builtin check

* improvement!: move simple_notifiers to an option instead of a DSL builder
improvement!: update spark for better autocomplete, configure autocomplete for key functions
docs: replace `an domain` with `a domain`

* improvement: better code interface documentation

* fix: set tenant on query so that root calles to Api.aggreagte work as expected (#929)

* chore: fixes from previous improvements

* chore: update splode

* chore: update splode

* improvement!: swap position of sort order and arguments in calculation sorting

* improvement!: add `include_nil?` aggregate option, and default it to `false`

* improvement: support notifiers within actions

* improvement: support specifying multiple filters

* improvement: add `sortable?` flags to all fields
improvement: support multiple filters on relationships

* improvement: support sensitive? on calculations and arguments

* improvement: validate resources in inputs to code interface

* chore: don't require explicit accept lists when using `default_accept :*`

* chore: update spark

* chore: update public attribute handling per 3.0

* improvement: update reactor and tests

* chore: better error message

* chore: fix rebase issue

* chore: handle merge issues
improvement: don't require domain on relationships if destination has domain

* improvement!: errors on unknown inputs for calculations

* improvement: always choose to cast atomic

* improvement: support casting some embeds atomically

* improvement: various 3.0 updates, documented in upgrade.md

* chore: Add failing tests for loads with with explicit domains. (#948)

Co-authored-by: James Harton <james@harton.nz>

* improvement: ensure non-static dynamic domains works

* improvement: add Ash.ToTenant protocol

* chore: add docs for no ToTenant option

* fix: properly construct new query in `build/3`

* chore: update simple_sat dependency

* chore: don't reselect when missing primary keys

* chore: remove IO.inspect

* chore: update spark

* chore: update spark

* improvement: use `Keyword.put_new` in `Ash.Context.to_opts` (#953)

* improvement: support bulk and atomic operations in code interfaces

---------

Co-authored-by: James Harton <james@harton.nz>
Co-authored-by: WIGGLES <55168935+WIGGLES-dev@users.noreply.github.com>
Co-authored-by: Dmitry Maganov <vonagam@gmail.com>
2024-03-27 16:31:59 -04:00

2.9 KiB

Bulk Actions

Bulk actions are ways to create, update or destroy many records at once, backed by scalable patterns.

Currently, only bulk creates are implemented. Bulk updates and bulk destroys will come next.

Bulk Creates

Bulk creates take a list or stream of inputs for a given action, and batches calls to the underlying data layer. For example, with an action like this:

create :create do
  accept [:title, :subtitle]
end

You could then call Ash.bulk_create like so:

Ash.bulk_create([ %{title: "foo", subtitle: "bar"}, %{title: "baz", subtitle: "buz"}], Resource, :action)

Considerations

Generally speaking, all regular Ash create actions are compatible (or can be made to be compatible) with bulk create actions. However, there are some important considerations.

  • Ash.Resource.Change modules can be optimized for bulk actions by implementing batch_change/3, before_batch/3 and after_batch/3. If you implement batch_change/3, the change function will no longer be called, and you should swap any behavior implemented with before_action and after_action hooks to logic in the before_batch and after_batch callbacks.

  • Actions that reference arguments in changes, i.e change set_attribute(:attr, ^arg(:arg)) will prevent us from using the batch_change/3 behavior. This is usually not a problem, for instance that change is lightweight and would not benefit from being optimized with batch_change/3

  • If your action uses after_action hooks, or has after_batch/3 logic defined for any of its changes, then we must ask the data layer to return the records it inserted. Again, this is not generally a problem because we throw away the results of each batch by default. If you are using return_records?: true then you are already requesting all of the results anyway.

Returning a Stream

Returning a stream allows you to work with a bulk action as an Elixir Stream. For example:

input_stream()
|> Ash.bulk_create(Resource, :action, return_stream?: true, return_records?: true)
|> Stream.map(fn {:ok, result} -> 
  # process results
  {:error, error} ->
  # process errors
end)
|> Enum.reduce(%{}, fn {:ok, result}, acc -> 
   # process results
   {:error, error} ->
   # process errors
end)

Considerations

Because streams are lazily evaluated, if you were to do something like this:

[input1, input2, ...] # has 300 things in it
|> Ash.bulk_create(Resource, :action, return_stream?: true, return_records?: true, batch_size: 100) # the default is 100
|> Enum.take(150)

What would happen is that we would insert 200 records (assuming no errors were emitted). Because the stream would end after we process the first two batches. If you want to make sure that everything happens, just be sure you aren't using things like Stream.take or Enum.take to limit the amount of things pulled from the stream.