Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: Deleted items reappear #2143

Open
sharphill2022 opened this issue Dec 30, 2024 · 0 comments
Open

[BUG]: Deleted items reappear #2143

sharphill2022 opened this issue Dec 30, 2024 · 0 comments
Labels
kind/bug Something is broken.

Comments

@sharphill2022
Copy link

What version of Badger are you using?

4.2.0 and 4.5.0

What version of Go are you using?

1.22.8

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, CPU, OS)?

128G RAM
Intel(R) Core(TM) i5-14600K
Ubuntu 22.04.5 LTS

What steps will reproduce the bug?

we have a btc indexer based on badger 4.5.0, https://github.com/sat20-labs/indexer
when we run indexer to build all index data from bitcoin block 0 to block 700000 (use a configuration file to set the block height, refer to max_index_height in file https://github.com/sat20-labs/indexer/blob/main/example.env.yaml), we got a base data, and all is fine now. but when we continue building data from block 700000 to block 720000, some items, that should be deleted before block 700000 appear again, then the building process will panic since it found index data has some problem. The indexer use this function to check the data:
func (b *BaseIndexer) CheckSelf() bool {
....
utxosInT1 := make(map[uint64]bool, 0)
go b.db.View(func(txn *badger.Txn) error {
defer wg.Done()

	var err error
	prefix := []byte(common.DB_KEY_UTXO)
	itr := txn.NewIterator(badger.DefaultIteratorOptions)
	defer itr.Close()

	startTime2 := time.Now()
	common.Log.Infof("calculating in %s table ...", common.DB_KEY_UTXO)

	for itr.Seek([]byte(prefix)); itr.ValidForPrefix([]byte(prefix)); itr.Next() {
		item := itr.Item()
		if item.IsDeletedOrExpired() {
			continue
		}
		var value common.UtxoValueInDB
		err = item.Value(func(data []byte) error {
			//return common.DecodeBytes(data, &value)
			return common.DecodeBytesWithProto3(data, &value)
		})
		if err != nil {
			common.Log.Panicf("item.Value error: %v", err)
		}

		sats := (common.GetOrdinalsSize(value.Ordinals))
		if sats > 0 {
			nonZeroUtxo++
		}

		satsInUtxo += sats
		utxoCount++

		for _, addressId := range value.AddressIds {
			addressesInT1[addressId] = true
		}
		utxosInT1[value.UtxoId] = true
	}

	addressInUtxo = len(addressesInT1)

	common.Log.Infof("%s table takes %v", common.DB_KEY_UTXO, time.Since(startTime2))
	common.Log.Infof("1. utxo: %d(%d), sats %d, address %d", utxoCount, nonZeroUtxo, satsInUtxo, addressInUtxo)

	return nil
})

....

common.Log.Infof("utxos not in table %s", common.DB_KEY_ADDRESSVALUE)
utxos1 := findDifferentItems(utxosInT1, utxosInT2)
if len(utxos1) > 0 {
	b.printfUtxos(utxos1)
}

........

INFO[12/30-11:10:54] calculating in av- table ...
INFO[12/30-11:10:54] calculating in b- table ...
INFO[12/30-11:10:54] calculating in u- table ...
INFO[12/30-11:11:08] b- table takes 13.924943325s
INFO[12/30-11:12:26] av- table takes 1m32.002112537s
INFO[12/30-11:12:26] 2. utxo: 83613987(83107919), sats 1923390854497096, address 43600825
INFO[12/30-11:15:10] u- table takes 4m16.335327894s
INFO[12/30-11:15:10] 1. utxo: 83151444(83107983), sats 1939759205330752, address 43600850
INFO[12/30-11:15:14] utxos not in table av-
INFO[12/30-11:15:21] 54d1e001340003 00ac8f758502fd21af32dde81dc7475098b7dfcad9c08f900c4be3b8803770e8:3 57190000000
INFO[12/30-11:15:22] 517e481f8c0006 016100c4613ffc292a052751f0ab4b5d44bfcbf461bb5e5da1c14adf31cef2a6:6 49999950000
INFO[12/30-11:15:22] 54d630098c0003 017e278c19c0dc5121c461fbfaef76231f25b7b5990444efee0830b64ce0fae9:3 100100000000
INFO[12/30-11:15:22] 54d00000740000 01f949b3cc0edd6c869e6ed6b14bd57540534886ae69dc5ed4df0f18aaa87821:0 900000000000
INFO[12/30-11:15:22] 54c81000080002 02209ab51490280c2beb90c42c49d033535502f2a5dcef7021b157bd9c323429:2 82977084274
INFO[12/30-11:15:24] 549f480aac0001 02eefd1c891d3e507be284d21d762c8eff7910a3a40fca6d90004f763a29c49d:1 215731921991
INFO[12/30-11:15:28] 54548008640001 09b908e3d326bdb4f41bd64d273ac49f85d901bd2ddd281ba9ff2eab75200801:1 63818966244

for example, the UTXO, 00ac8f758502fd21af32dde81dc7475098b7dfcad9c08f900c4be3b8803770e8:3 , should be deleted at block 695194, and can't be retrived when seek from table DB_KEY_UTXO.
in addition, we can try to retrive this utxo from database based on the block 700000, database return "key not found", that is fine. but when we continue building data from block 700000 to 720000, this utxo appear again and make the indexer broke down.

Expected behavior and actual result.

deleted item should not be retrived.

Additional information

No response

@sharphill2022 sharphill2022 added the kind/bug Something is broken. label Dec 30, 2024
@ryanfoxtyler ryanfoxtyler changed the title [BUG]: <Title>Deleted items reappear [BUG]: Deleted items reappear Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something is broken.
Development

No branches or pull requests

1 participant