Revisit sweep feature

[lassepe]

It may be worth re-evaluating the sweep feature. My understanding is that it was previously dropped due to segfaults but those seem likely to be fixed with #40

[RomeoV]

I'm also interested in this. Can you elaborate what you mean by "revisit"? Is there prior work or an overview what would need to get done?

[avik-pal]

It would entail checking if Wandb.wandb.sweep(...) works without segfaulting julia and if it does remove https://github.com/avik-pal/Wandb.jl/blob/main/src/sweep.jl with direct API calls.

[RomeoV]

There still seems to be problems with segfaults, although occasionally I am able to make it work (but not with running any complex code).

My basic setup is:

using Wandb, Wandb.PythonCall, Logging

cfg = @pyeval `
       {
           "name": "sweepdemo",
           "method": "grid",
           "metric": {"goal": "minimize", "name": "validation_loss"},
           "parameters": {
               "batch_size": {"values": [16, 32, 64]},
               "epochs": {"values": [5, 10, 15]},
           },
       }
`

function foo()
    # do nothing for now
 end

sweep_id = Wandb.wandb.sweep(cfg, project="Wandb.jl")

Wandb.wandb.agent(sweep_id, foo)

This worked once for me (but foo isn't doing anything), but after changing foo to try to include logging, it doesn't work anymore, and then even going back to this doesn't work anymore either. Probably some connection that is not cleaned up or something.

For now I will stick to the example in https://avik-pal.github.io/Wandb.jl/v0.5.6/examples/hparams.

[RomeoV]

In general though, I think something like this could work:

function foo()
    run = Wandb.wandb.run
    lg = WandbLogger(run, 1, 0, Info)
    global_logger(lg)
    @info "metrics" validation_loss=1
    close(lg)
end

[lassepe]

It would be worth filing an issue with PythonCall for any remaining segfaults

[RomeoV]

Yeah. To shine a bit more light, it seems that calling Wandb.wandb.agent(sweep_id, function) ultimately get's passed within python to multiprocessing.Process here.

So internally python is spawning a new process, which is then again calling the Julia function, where we try to access properties of the process such as logging through the run. It's quite a convoluted setup, so I'm not sure if PythonCall is really to blame here.