Display custom type with PNG support

[jwortmann]

How can I use imshow to display an image of my custom type which has support for the image/png MIME type via Base.show?

Is it not supported, or is imshow not the appropriate function for this? Or does ImageView support some kind of AbstractDisplay, so that I could write display(ImageViewDisplay, Foo())?

Minimum non-working example:

julia> using ImageView, PNGFiles, TestImages

julia> struct Foo end

julia> Base.show(io::IO, ::MIME"image/png", x::Foo) = PNGFiles.save(io, testimage("mandrill"))

julia> imshow(Foo())
ERROR: MethodError: no method matching size(::Foo)

Closest candidates are:
  size(::Core.Compiler.StmtRange)
   @ Base show.jl:2774
  size(::Base.ExceptionStack)
   @ Base errorshow.jl:1018
  size(::Base.AsyncGenerator)
   @ Base asyncmap.jl:389
  ...

Stacktrace:
 [1] axes(A::Foo)
   @ Base .\abstractarray.jl:98
 [2] roi(A::Foo, dims::Tuple{Int64, Int64})
   @ ImageView C:\Users\jwortmann\.julia\packages\ImageView\Gg1GW\src\slicing.jl:45
 [3] imshow(img::Any; axes::Any, name::Any, aspect::Any)
   @ ImageView C:\Users\jwortmann\.julia\packages\ImageView\Gg1GW\src\ImageView.jl:277
 [4] imshow(img::Any)
   @ ImageView C:\Users\jwortmann\.julia\packages\ImageView\Gg1GW\src\ImageView.jl:274
 [5] top-level scope
   @ REPL[13]:1

[mkitti]

I think we're confusing two distinct things: Base.show and imshow. imshow wants a matrix of Colorant.

Are you thinking of some viewer that can render PNGs directly? For example, are you trying to show this image in a web interface or via VSCode directly?

[jwortmann]

Are you thinking of some viewer that can render PNGs directly?

Yes. Is ImageView.jl not the right tool for this?

Ideally I only want to implement Base.show for my type as described in https://docs.julialang.org/en/v1/manual/types/#man-custom-pretty-printing and then the environment where the code is executed decides how to display my type. But for development I would also like to have a simple way to display the image directly from the REPL by opening a window, e.g. via display(d::AbstractDisplay, x). So I think I'm searching for a package that provides an AbstractDisplay for rendering PNG.

I hoped that ImageView.jl could display any custom type directly, if the type can be displayed for example as image/png etc., but I think that this package doesn't provide an AbstractDisplay implementation for this, correct?
Or for example something like

function imshow(img::Any)
    if showable("image/png", img)
        # open a window and render the PNG data returned by Base.show
    elseif showable("image/jpeg", img)
        # open a window and render the JPEG data returned by Base.show
    else
        throw(ArgumentError("$(typeof(img)) can't be rendered as an image"))
    end
end

I could also imagine that it happens automatically if ImageView is loaded, like

julia> foo = Foo()
# `foo` is displayed as text here
julia> using ImageView
julia> foo
# now `foo` opens as an image in an external window

(or when using just display(foo) from within a source code file).
But perhaps this would be a bit too intrusive for some users if the new window opens automatically. Therefore having an explicit function like imshow to render the image is probably the better option.

[mkitti]

Typically what you want to do is provide a Matrix{RGB} or an indexed matrix via IndirectArrays. ImageView will take such a matrix and display it in a Gtk window.

Some displays may understand PNG such as a web browser or a VS Code.

See the Julia images documentation.

https://juliaimages.org/stable/examples/#demonstrations

[jwortmann]

While ImageView.jl currently doesn't provide an AbstractDisplay implementation, I saw that ImageInTerminal.jl does support it. However, I believe that its implementation is not entirely correct. It claims that it has support for image/png for all types which implement a corresponding Base.show method (https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L7), but in fact it only has specialized implementations for Vector{UInt8} and for AbstractArray{<:Colorant} (https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L9-L21).

So when I test the code from above with ImageInTerminal.jl, it still doesn't work:

julia> struct Foo end

julia> foo = Foo()
Foo()

julia> displayable("image/png")
false

julia> using PNGFiles, TestImages, ImageInTerminal

julia> displayable("image/png")
true

julia> showable("image/png", foo)
false

julia> Base.show(io::IO, ::MIME"image/png", ::Foo) = PNGFiles.save(io, testimage("mandrill"))

julia> showable("image/png", foo)
true

julia> foo
Foo()

Therefore, instead of the line

Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png") = true

at https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L7, it should actually be

Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png") = false
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png", ::Vector{UInt8}) = true
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png", ::AbstractArray{<:Colorant}) = true

I think there was a misunderstanding in that package about how Base.displayable is supposed to work. However, it shouldn't be too difficult to provide a proper imlementation with support for image/png for all types in ImageInTerminal.jl. It just needs to decode the PNG data from Base.show. Perhaps I can look into that later. Unfortunately even with that, ImageInTerminal wouldn't help me much in practice, because I'm on Windows and the resolution of the displayed images is so low (because it's restricted to use unicode characters and terminal colors on Windows Terminal). So I still hope that ImageView.jl could add support for an AbstractDisplay and open a Gtk window to render the PNG image.

Please also compare how my example code from above works in a Pluto notebook; here everything works as expected:

---

Update: I created a PR for the ImageInTerminal package at JuliaImages/ImageInTerminal.jl#76

[jwahlstrand]

Support for showing PNG's via Base.show in ImageView may be worth discussing further, but FYI here's a way to show a PNG in a Gtk window using Gtk4.jl:

using Gtk4, TestImages, PNGFiles

struct Foo end

Base.show(io::IO, ::MIME"image/png", x::Foo) = PNGFiles.save(io, testimage("mandrill"))

function displaypng()
    io=IOBuffer()
    show(io, "image/png", Foo())
    b=Gtk4.GLib.GBytes(take!(io))
    t=GdkTexture(b)
    win=GtkWindow("Image", size(t)[1], size(t)[2])  # set the window dimensions to be the same as the PNG
    win[] = GtkPicture(GdkPaintable(t))
end

[jwortmann]

FYI here's a way to show a PNG in a Gtk window using Gtk4.jl

Thank you for the suggestion. It works well with the example using the file from TestImages, however it turned out that this code randomly crashes (sometimes immediately, or after a few seconds, or sometimes if I resize the window) if it is used with my own implementation of Base.show. I assume that the bytes or texture object from Gtk4 only uses a reference to the memory where the data is stored, and it expects the user to handle its allocation and deallocation. This is pure speculation though, but I've already experienced a similar problem with SimpleDirectMediaLayer.jl (SDL) which also requires the user to handle the memory management when using certain functions. So I assume that a short time after running the displaypng() function, the Julia garbage collector cleans up the memory because there isn't any reference anymore to it outside of the function, and then the program crashes due to a memory access violation. A solution could be to permanently keep the data by assigning it to a global variable (this was what I used as a workaround for SDL), but this is maybe a bit awkward.

So I ended up using the approach from CairoMakie, which simply writes the PNG to a temporary file on disk and then it opens the file with the default image viewer:

using Scratch: @get_scratch!

function preview(x::Foo)
    filepath = joinpath(@get_scratch!("preview"), "preview.png")
    open(filepath, "w") do file
        show(file, "image/png", x)
    end
    command = @static Sys.iswindows() ? `powershell.exe start $filepath` : `open $filepath`
    run(command)
    nothing
end

[jwahlstrand]

OK, thanks for letting me know, I'll look into it -- the memory is supposed to be copied, but maybe there is a reference counting bug somewhere. I'm glad you found an alternative method that works.