use NlOpt with GPU and Lux

[jakubMitura14]

Hello I tried to use NLopt like below

using Optimization, Lux, Zygote, MLUtils, Statistics, Plots, Random, ComponentArrays
using CUDA, LuxCUDA,  OptimizationNLopt

x = Float32.(rand(10, 10))
y = Float32.(sin.(x))
dev = gpu_device()

# data = MLUtils.DataLoader((x, y), batchsize = 1)|> dev
# data = MLUtils.DataLoader((x, y), batchsize = 1)
data = (x, y)

# Define the neural network
model = Chain(Dense(10, 32, tanh), Dense(32, 1))
ps, st = Lux.setup(Random.default_rng(), model)
ps_ca = ComponentArray(ps)
ps_ca = ps_ca |> dev


smodel = StatefulLuxLayer{true}(model, nothing, st)

function callback(state, l)
    state.iter % 25 == 1 && @show "Iteration: %5d, Loss: %.6e\n" state.iter l
    return l < 1e-1 ## Terminate if loss is small
end

function loss(ps, data)
    print("\n dataaaaaaa $(typeof(data)) \n")
    d1 = CuArray(data[1])
    d2 = CuArray(data[2])
    # print("\n typeof(d1) $(typeof(d1)) typeof(d2) $(typeof(d1)) \n")
    ypred = smodel(d1, ps)
    return sum(abs2, ypred .- d2)
end

optf = OptimizationFunction(loss, AutoZygote())
prob = OptimizationProblem(optf, ps_ca, data, lb=ps_ca.*(100000000*(-1)), ub=ps_ca.*100000000)


res = Optimization.solve(prob, NLopt.LD_LBFGS())

but it give warning and give information that arguments are invalid (Hovewer still return something)

┌ Warning: NLopt failed to converge: INVALID_ARGS
└ @ OptimizationNLopt /usr/local/share/julia/packages/OptimizationNLopt/YE3fr/src/OptimizationNLopt.jl:299
retcode: Failure
u: 385-element Vector{Float64}:
 -0.001771742245182395
 -0.20413166284561157
  0.47901201248168945
  0.5232356786727905
  0.19946818053722382
  0.9051346778869629
  0.5221430063247681
  ⋮
 -0.01594085991382599
 -0.15956547856330872
  0.22767578065395355
  0.18943265080451965
  0.2767360210418701
 -0.09060057252645493
 -0.12697920203208923

[odow]

Hi @jakubMitura14, I'm not sure what the cause is. I suggest you instead open an issue in https://github.com/SciML/Optimization.jl. This doesn't seem like a bug in NLopt.jl.

It would help to create a minimal reproducible example. Does this need the GPU? Does it need the lower and upper variable bounds? Does it need packages like Plots?

[jakubMitura14]

1. To be honest I was creating it as minimal working example
2. Yes it requires GPU
3. no it do not need upper and lower
4. indded it do not need i had added too mch packages - thanks

So what do you suggest here to change apart form removing unnecessary packages imports?

[blegat]

So what do you suggest here to change apart form removing unnecessary packages imports?

Even if the package is used in the example, try also to see if you can remove this dependency.
If the example requires Optimization.jl then it might be an issue with Optimization.jl, especially since the error INVALID_ARGS may suggests it was not used with the right arguments.
If you are able to still get the error without using Optimization.jl then it might indeed be a bug to report here.

[jakubMitura14]

Ok thanks !