Bug Report

Description

A client locks table metadata for the entire duration of a make function call. When other clients attempt to drop or declare child tables, the call is blocked until the first client finishes make. This approach scales poorly with number of clients and number of child tables.

Reproducibility

Include:

• OS: Any
• Python Version: Any
• MySQL Version: Any
• MySQL Deployment Strategy: Any
• DataJoint Version: 0.14.1
• Minimum steps required to reproduce:
  • See the test case presented Table Metadata Lock by long populate calls LorenFrankLab/spyglass#1030

Proposed Solution

As an alternative to writing a Computed.make function, allow user to write three functions:

1. make_fetch for reading inputs
2. make_compute, which is not run in a transaction, and is passed the return value of make_fetch
3. make_insert, which inserts computed values using the same transaction semantics as make.

In pseudocode, these three functions will be used in the following routine as such:

if hasattr(table, "make"):
    return make()
else:
    assert hasattr(table, "make_fetch")
    assert hasattr(table, "make_compute")
    assert hasattr(table, "make_insert")
    input = make_fetch()
    conn.disconnect() # I assume this disconnect step is to ensure that make_compute cannot insert?
    result = make_compute(input)
    tx = conn.start_transaction()
    input2 = make_fetch()
    if hash(serialize(input2)) == hash(serialize(input)):
        result = make_insert(result)
        tx.commit()
        return result
    else:
        print("ERROR: inputs have changed")
        tx.abort()
        return None

Additional Research and Context

Related Issues

• Table Metadata Lock by long populate calls LorenFrankLab/spyglass#1030
• Allow disable transaction for select populates LorenFrankLab/spyglass#1067

cc: @dimitri-yatsenko @ttngu207 @CBroz1 @samuelbray32 @peabody124